Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
157891 stories
·
33 followers

How to customize your Windows Terminal dropdown

1 Share
From: kayla.cinnamon
Duration: 4:19
Views: 77

In this video, I explain how to customize Windows Terminal's dropdown menu.

Links:
Windows Terminal GitHub repository: https://github.com/microsoft/terminal
Windows Terminal Canary: https://github.com/microsoft/terminal?tab=readme-ov-file#installing-windows-terminal-canary

Socials:
👩‍💻 GitHub: https://github.com/cinnamon-msft
🐤 X: https://x.com/cinnamon_msft
📸 Instagram: https://www.instagram.com/kaylacinnamon/
🎥: TikTok: https://www.tiktok.com/@kaylacinnamon
🦋 Bluesky: https://bsky.app/profile/kaylacinnamon.bsky.social
🐘 Mastodon: https://hachyderm.io/@cinnamon

Disclaimer: I've created everything on my channel in my free time. Nothing is officially affiliated or endorsed by Microsoft in any way. Opinions and views are my own! 🩷

#windows #terminal #developer #development

Read the whole story
alvinashcraft
just a second ago
reply
Pennsylvania, USA
Share this story
Delete

Keep the gains: performance regression testing without fooling yourself

1 Share

TL;DR: Keep only the benchmarks that protect important hot paths. Use baseline/diff comparisons to catch regressions, but do not trust noisy shared runners blindly. Performance regression testing needs stable machines, clear thresholds, and restraint.

After a successful performance investigation, the benchmark folder looks valuable.

It contains the history of the work: experiments, false starts, before-and-after comparisons, warmup cases, exception cases, and small probes that helped explain the code. Keeping all of it feels safe, and deleting any of it feels strangely reckless.

That instinct leaves the team maintaining experiments long after they have answered their questions.

Benchmarks have a maintenance cost. They take time to run, understand, and fix when the production code changes. The useful measure is whether the suite protects important performance promises, not how many benchmark classes survive in the repository.

Most benchmark experiments should expire

Many benchmarks are investigation tools. They help answer a question during the improve-and-compare loop. Once the question is answered, the benchmark may no longer deserve permanent continuous integration time.

Keep the benchmarks that protect core hot paths. In the NServiceBus pipeline case, a benchmark for pipeline invocation can make sense because every message goes through that path. A regression there can affect many users across many transports.

Protecting that benchmark is worth the maintenance cost.

A one-off benchmark that tested an abandoned idea is different. Keep the lesson, not necessarily the code. Move the finding into the pull request, a decision record, or a short note near the optimized code. Future readers need to know why the code looks the way it does. They do not need every experiment to run forever.

  • Keep benchmarks for shared infrastructure hot paths.
  • Keep benchmarks that protect public contracts or expensive operations.
  • Delete or archive one-off experiments after the lesson is captured.
  • Document what was measured, what changed, and why the trade-off was accepted.

Catch regressions by comparing benchmark history

Performance regression testing usually compares two versions of the same benchmark. Run the benchmark at a known baseline commit. Store the artifacts. Move to the candidate commit or branch. Run the same benchmark again. Compare the results with a threshold.

The .NET performance repository includes a ResultComparer tool that can compare BenchmarkDotNet artifacts. The exact commands depend on your repository layout, but the workflow looks like this:

git checkout baseline-sha

dotnet run -c Release --artifacts "C:\results\before"

git checkout candidate-sha

dotnet run -c Release --artifacts "C:\results\after"

dotnet run --project C:\Projects\performance\src\tools\ResultsComparer \
  --base "C:\results\before" \
  --diff "C:\results\after" \
  --threshold 2%

The threshold is policy. Two percent might be reasonable for one benchmark and meaningless for another. Choose the threshold based on observed variance and business impact, not because the number looks tidy.

Thresholds are policy, not math magic

A benchmark result is a distribution, not a single truth. The same code can produce slightly different measurements from run to run. The machine, operating system, runtime, background processes, CPU frequency, temperature, and neighboring workloads all get a vote.

Before turning a benchmark into a failing gate, learn its natural variance on the machine that will run it. If a benchmark moves by three percent when nothing changed, a two percent regression threshold will create noise. The team will learn to ignore it, and then the gate has failed socially even if it works technically.

CPU-bound benchmarks are often more stable than memory-bound or disk-bound benchmarks, but stability is not guaranteed.

Measure the variance before deciding what the gate should enforce.

Shared runners can lie to you

Shared continuous integration runners are convenient. They are also shared. Another build on the same host can affect your measurements. The hardware can differ between runs. Power settings and virtualization layers can change the timing profile. That is the noisy-neighbor problem.

Two subsequent builds on the same revision can have ranges of 1.5..2 seconds and 12..36 seconds. CPU-bound benchmarks are much more stable than Memory/Disk-bound benchmarks, but the “average” performance levels still can be up to three times different across builds.

Andrey Akinshin, quoted in the BeyondSimpleBenchmarks talk material

That does not mean shared runners are useless. They can still compile benchmarks, run smoke checks, or provide a rough signal. But a flaky performance gate is worse than no gate.

Developers learn to rerun jobs until they pass, and performance work starts to feel like superstition. Eventually the team assumes every benchmark failure is noise.

Performance culture is social as much as technical. If people stop trusting the signal, the tooling has already lost.

Use stable hardware when the gate matters

If the benchmark is important enough to block a pull request, the machine should be stable enough to support that decision. That may mean a dedicated bare-metal runner, a controlled virtual machine, a lab machine, or a manually triggered benchmark pipeline for risky changes.

Not every team needs a performance lab. A team that is just starting can run benchmark experiments locally, review the results manually, and build shared knowledge. That is already progress. Automating noisy benchmarks too early can create more frustration than value.

Running every benchmark on every pull request is automation, but it is not necessarily maturity.

A mature team knows which performance promises are stable and important enough to defend automatically.

Regression tests still need human judgment

A regression is not always a bug. Sometimes a slower implementation fixes correctness, improves security, removes a dangerous shortcut, or makes the system easier to maintain. The benchmark should start a conversation, not replace one.

When a benchmark fails, ask what changed. Did the hot path slow down because of accidental allocations? Did a new feature add necessary work? Did the benchmark become invalid because the production code changed shape? Did the machine have a bad run?

The best performance gates make accidental regressions cheap to catch and intentional trade-offs explicit. They should not make teams afraid to improve the design.

Close the loop with continuous improvement

The full performance loop is still the foundation: profile with a profiling harness, improve a hot path, benchmark and compare, profile again, ship, and observe production. Regression testing is a maturity step that protects the gains after the team knows what matters.

This approach also pushes against rewrite culture. It is easy to look at old code and say, “This is slow. We should rewrite it.” Sometimes a rewrite is right. Most of the time, the team needs more knowledge first.

Otherwise, the rewrite repeats the old mistakes with newer code and better formatting.

Profiling and benchmarking build the missing knowledge. They show which paths matter, which assumptions were wrong, and which trade-offs paid off. After a few loops, the team may not need a rewrite. If it still does, the rewrite starts with evidence instead of frustration.

Performance knowledge accumulates. A small improvement can expose the next bottleneck, a useful benchmark protects the path, and production observations correct assumptions made in the lab. The team gradually has less reason to guess.

Start with one hot path and build a profiling harness around it. Take memory and CPU profiles, improve one thing, benchmark it, and profile again. Then write down what you learned.

The profile points at the work and the benchmark tests the change. Production then exposes whatever the lab missed.

Repeated often enough, this becomes ordinary engineering work rather than a rescue mission after performance has already collapsed.

Further reading

Common questions

Should performance benchmarks run on every pull request?

Only if the benchmark is stable, fast enough, and important enough. Otherwise, run it on demand, nightly, before release, or when a change touches the protected hot path.

What should I do if continuous integration results are noisy?

Measure variance, loosen or remove the gate, use dedicated hardware, or treat the result as a signal rather than an automatic failure. Do not keep a flaky performance gate just because it feels rigorous.

How many benchmarks should a team keep?

Keep the benchmarks that protect meaningful performance promises. If nobody can explain what decision a benchmark supports, it probably does not belong in the permanent suite.

Performance loop status

Read the whole story
alvinashcraft
27 seconds ago
reply
Pennsylvania, USA
Share this story
Delete

Spec-Driven Development for Teams: The Shared-Spec Workflow

1 Share
How a spec stops being personal discipline and becomes the contract your team ships from: shared specs in git, spec review by pull request, gates on a board, and why the bottleneck moves from writing code to integrating it.
Read the whole story
alvinashcraft
36 seconds ago
reply
Pennsylvania, USA
Share this story
Delete

The 10 Best Angular UI Grids

1 Share

TL;DR What’s the best Angular grid? For most enterprise Angular applications, Progress Kendo UI for Angular Grid is the strongest all-around choice. It combines high-performance data handling, virtualization, AI-assisted features, theming tools and integration with a broader UI component library.

On the internet, data is everything. Since the beginning of the web, showing lists has been the standard way to present information.

We started with simple HTML tables, but over the years, displaying data has become much more than just creating a list. Today, it involves complex interactions and the need for data to travel smoothly between many different components.

In any application, when a list is slow or difficult to use, it is not just a small problem. It is a major blocker. A grid that cannot handle real-time updates does more than frustrate users; it can cause serious business problems and angry users.

In 2026, choosing a grid component is a business decision, not just a technical one. Modern grids must handle thousands of rows, update every second, export data for reports, integrate with AI agents and look good on all screens. You need all of this without making your developers write thousands of lines of custom code.

To understand why this matters, the best way is with a real-world scenario.

The Scenario

You have been hired by a startup to build a trading dashboard with real-time market data, thousands of rows updating constantly, complex filters, frozen columns, and one-click export to Excel. The CTO wants a working minimum viable product (MVP) ASAP to compete with time-to-market.

In this situation, your grid is the most important part of the product. Every second of lag or every missing feature is a threat to your deadline. If the grid fails during a demo with investors, the rest of the app does not matter.

Consider: Many teams waste half their time building basic features like virtual scrolling. However, the biggest problem is communication. A grid must talk to the rest of your app. When you mix different libraries for your grid and your buttons, making them work together is a nightmare.

Let’s look at the best options available today and see how they can help you and your team.

1. Kendo UI for Angular Grid

No matter how small, large or complex your data is, the Progress Kendo UI for Angular Grid is the best choice when you need a grid that meets every requirement. It is built to handle everything from simple lists to massive enterprise datasets with real-time updates.

This component is not just powerful; it is also easy to customize. Because it is part of a complete suite, the grid integrates perfectly with other Kendo UI components like charts and dropdowns. This solves the “headache” of making different parts of your app talk to each other.

I have watched how building grids has evolved over many years. Based on that experience, here are the reasons why Kendo UI is the right choice for professional projects:

Why Kendo UI Grid is the Right Choice for Professional Projects

  • Smart Grid and AI Assistant: This is where Kendo UI really changes the game. It is not just a table; it is a “smart” component. With the AI Smart Box, users can use natural language to search, filter and manage data. Instead of looking for exact keywords, the Semantic Search understands what the user actually wants.

  • AI-powered insights: The grid can highlight important rows and provide smart data summaries automatically. This helps users find the most important information without manual work.

  • Virtual scrolling and row virtualization: The grid handles hundreds of thousands of rows easily. Scrolling is fast even when data changes constantly, which is essential for trading.

  • Real-time data binding: It works excellently with live data streams. You can update the grid without refreshing the entire dataset, keeping the UI fast.

  • Frozen columns and column management: You can pin important columns while users scroll horizontally through large datasets.

  • Built-in Excel and PDF export: This is a native feature, not a third-party plugin. It keeps your styles, filters and column settings.

  • MCP Server: The Kendo UI MCP Server helps AI tools (like Cursor or Copilot) write better code by providing the correct patterns for the component.

  • ThemeBuilder: You can import Figma design tokens to make the grid match your brand perfectly without writing complex CSS.

  • Expert support: If you have a problem during a busy week, you can get help directly from the engineers who built the grid.

Pro tip: You can try the Kendo UI for Angular Grid for free for 30 days. Test it with your real data before you decide.

 

If you have always worked within the official Angular ecosystem, the next option on this list will be very familiar to you.

2. Angular Material Table (CDK)

Angular Material is the official UI library from the Angular team. Most Angular developers have used its table component.

  • The Good: It is very stable, updated frequently and follows Material Design standards. The Component Dev Kit (CDK) gives you a lot of flexibility.

  • The Cons: You must build almost every feature yourself. Sorting, filtering, virtual scrolling and export are not included by default. For a project with a short deadline, this is too much work.

  • My Feedback: Angular Material is a good starting point, but it is only a foundation. In a high-stakes project, you should not waste time building features that professional libraries have already solved.

When you need a tool that focuses entirely on high-performance tables as a standalone component, you will likely look at the following library.

3. AG Grid

AG Grid is a very popular standalone grid library. It is known for high performance and many enterprise features.

  • The Good: If your app is mostly about tables, AG Grid is very fast. The community version includes many features for free.

  • The Cons: A real application needs more than just a grid. You will also need charts, date pickers and modals. AG Grid does not provide these. If you mix it with other libraries, it is difficult to keep the design and styling consistent.

  • My Feedback: AG Grid is powerful, but it is a standalone tool. Kendo UI gives you the same power within a unified system. This means your grid, charts and dropdowns all use the same design and the same support team.

For teams that prefer a large collection of components that are easy to implement, there is another very common choice in the community.

4. PrimeNG

PrimeNG is a very complete library. Its p-table component is good for most standard use cases.

  • The Good: It has good documentation and is easy to start using. It works well for standard CRUD applications with medium-sized datasets.

  • The Cons: For extreme scenarios, such as thousands of real-time updates, it can have performance issues. Virtual scrolling is available but requires more manual configuration.

  • My Feedback: PrimeNG is a good choice for standard business apps. But for a fintech dashboard where the grid is the main product, you need a tool built for high performance from the start.

As your project moves into the enterprise space with specific corporate requirements, you might consider an alternative that is widely used in large companies.

5. Syncfusion Angular Grid

Syncfusion offers an enterprise grid with many features like row grouping and PDF export.

  • The Good: It has a long list of features and handles large data well. It integrates well if you already use other Syncfusion tools.

  • The Cons: The API does not always feel like standard Angular code, so there is a learning curve. Changing the design to match a Figma file can be difficult and slow.

  • My Feedback: If your designers and developers need to work together daily, the difficulty in styling Syncfusion can be a problem. Progress ThemeBuilder makes this process much faster and more accurate in Kendo UI.

Teams coming from a background in traditional software development often find that the next grid fits their existing mental model perfectly.

6. DevExtreme Angular DataGrid

DevExtreme by DevExpress is a feature-rich grid often used in traditional enterprise software.

  • The Good: It is great for complex scenarios like master-detail views and multi-level grouping. It will feel familiar if your team has a background in .NET or WinForms.

  • The Cons: Some of the patterns feel old for modern web development. Customizing the CSS to look modern can be a difficult task.

  • My Feedback: Modern applications need to be fast and look current. DevExtreme’s older patterns can slow down a team. The Kendo UI library uses a modern, Angular-native architecture that is easier to maintain.

If your main priority is fast data visualization and specialized charts, there is a competitor that focuses heavily on those areas.

7. Ignite UI for Angular Grid

Ignite UI focuses on data visualization and high-performance grids.

  • The Good: The rendering engine is very fast and handles live updates well.

  • The Cons: The community is smaller than other options. If you find a complex bug, it is harder to find answers or documentation.

  • My Feedback: Performance is important, but support is also critical. The Kendo UI Grid offers similar performance but adds better documentation and a larger community to help you when you have problems.

For developers who value a very clean design and a modular architecture based on modern TypeScript principles, there is an elegant solution to consider.

8. Taiga UI Table

Taiga UI is a modular library with a focus on TypeScript.

  • The Good: It has a very clean design. Because it is modular, you only use the code you need.

  • The Cons: You have to build many features yourself. Virtual scrolling and export are not ready to use out of the box.

  • My Feedback: Taiga UI is a great project, but it requires too much manual work for a short deadline. The Kendo UI library gives you 120+ components that are already tested and ready to use.

Some projects still use libraries that were very popular during the early versions of Angular and remain in many existing codebases.

9. ng2-smart-table

ng2-smart-table was a very popular choice in the early days of Angular.

  • The Good: It is very simple to set up for basic tasks like sorting and filtering.

  • The Cons: The library is not updated frequently anymore. This creates a risk of bugs or compatibility issues with new versions of Angular.

  • My Feedback: Using a library that is no longer maintained creates “technical debt.” The Kendo UI library is supported by Progress Software, which means it receives regular updates and a clear roadmap you can trust.

Finally, if your application requires a grid that behaves exactly like a spreadsheet instead of a standard list, there is a specialized tool for that specific case.

10. Handsontable

Handsontable provides a spreadsheet-like experience (like Excel) in the browser.

  • The Good: It is the best choice if your users need to edit data like a spreadsheet (bulk edits, formulas, etc.).

  • The Cons: The grid does not fully support Angular, works like a wrapper and is very limited (it also has a separate commercial license for bringing in Excel formulas features).

  • My Feedback: Handsontable is perfect for spreadsheets, but a trading dashboard has different needs. The Kendo UI Grid provides the necessary Excel features while offering native Angular integration and real-time performance.

Summary

Today we learned how many good Angular grids exist, but when developers or agents in 2026 need to cover all the needs for your team and project, the list of options becomes much shorter. Mst grids on this list are good at one specific thing. However, for a project where the grid is the core of the product, you need a complete solution.

I pick the Kendo UI for Angular Grid as the best choice because it combines performance, features and design flexibility, and is also ready for the AI era with tools like the MCP Server.

When you have a short deadline, you need a tool that helps you work faster from day one and keeps the focus on building your product.

As I mentioned, Kendo UI for Angular comes with a free 30-day trial. So go ahead and poke around:

Download Free Trial

Read the whole story
alvinashcraft
45 seconds ago
reply
Pennsylvania, USA
Share this story
Delete

LLM Models Are Bullshit Engines

1 Share

Your LLM System Doesn’t Have to Be

Harry Frankfurt wrote a wonderful essay called On Bullshit. His central move is a distinction most people miss: the difference between a liar and a bullshitter.

The liar cares about the truth. He has to. Truth is what a liar wants to conceal, and you cannot conceal what you have not first located. Lying is a truth-tracking activity performed in reverse.

The bullshitter doesn’t care about the truth at all. Not for it, not against it. Truth is simply not a variable in his function. His goal is to persuade, to impress, to manage how he is perceived. Whether the words happen to be true is incidental.

That’s an LLM model.

I’m not the first person to make this connection. In 2024, three philosophers at the University of Glasgow, Michael Townsen Hicks, James Humphries, and Joe Slater, published a paper in Ethics and Information Technology titled, with admirable economy, ChatGPT is bullshit. They make the Frankfurtian case rigorously. They also argue that hallucination is the wrong word for what these systems do. It implies a perceptual faculty that misfired, when the reality is that there was never a faculty there to misfire. They’re right. I’m going to keep using ‘hallucinate’ for the rest of this piece anyway, because it’s the term of art my readers actually use and I’m actually using LLM systems so that term might be more accurate (but more on that later). Hold the objection in your head while you read. It matters for what comes next.

The physics of bullshit

An LLM model takes a token, runs it through a very large pile of mathematical weights, and predicts the next token. That’s not quite right. What it actually produces is a probability distribution over next tokens. Then something samples from that distribution.

The surprising part is that if you always pick the single most probable token, you get worse results, not better. Greedy decoding produces flat, repetitive, degenerate text. So we sample instead. Temperature is the knob that controls how sharp or how flat that distribution is before we sample from it. Turn it down and the mass concentrates on the top candidates and you approach, but never reach, deterministic output. Turn it up and the mass spreads out and you get more variety, which we experience as creativity. Two related knobs, top-k and top-p, truncate the candidate pool outright.

Now read that back. Nowhere in it is there a step called check whether this is true. There is no truth register. There is no fact table it consults, no assertion it defends. There is a distribution and there is a sampler.

The model is not lying to you. Lying would require it to have a concept of truth. It is bullshitting you. It produces plausible-shaped text with total indifference to whether the world matches. When it’s right, it’s right as a side effect of its training data, not out of conviction. When it’s wrong, it’s wrong with exactly the same confidence and exactly the same beautiful prose.

That’s not a bug in the model. That is the model.

This is physics, and physics doesn’t care how you feel about it. Nobody wrapping their car around a tree is a fan of F=ma. It happens anyway. Learn the physics and work around it.

The model is not the system

This is where a lot of the public conversation goes sideways. People conflate models with systems.

The early chatbots were a model with a REPL loop bolted on top. Ask what the date was and you got some date well in the past, or the thing announced it was trained on data up to a point that had already receded over the horizon. All it could do was reach into the weights. Pure engine of bullshit.

Then we got serious. Every modern chatbot you deal with has grounding mechanisms. Ask it for arithmetic and it recognizes the shape of the request and calls a calculator instead of rolling dice on the statistics of digits. Ask it about a current fact and it runs a web search and feeds the results back into context, which is retrieval-augmented generation. The answer is now conditioned on both what’s in the weights and on real retrieved evidence.

That is not magic. That is engineering. It’s what it looks like when somebody stops hoping and starts building.

Put on your big-boy engineering pants

So if LLM models are bullshit engines, and we are building systems on top of them, what then must we do?

Start by recognizing that most of our modern technology stack is built on crap.

IP (Internet Protocol) is best-effort. It will drop your packet, duplicate it, deliver it out of order, and tell you about none of it. So we layered TCP (Transmission Control Protocol) on top: sequence numbers, acknowledgments, retransmission, backoff. Reliable stream, unreliable substrate.

An individual disk drive is a spinning platter of rust that will betray you faster than a politician’s campaign promise. So we apply engineering, RAID (Redundant Array of Inexpensive Disks), and now we have reliable storage built on unreliable disks. But that reliable storage lives in a building, and buildings burn down and lose power and get their fiber cut by a backhoe. So we apply more engineering: geo-replication.

DRAM flips bits when a cosmic ray wanders through. So: Error Correcting Code (ECC) memory.

Every layer you trust is standing on a layer that does not deserve trust. None of this is new. It’s why we pay engineers a lot of money. They are alchemists, turning leaden, unreliable components into services with gold-plated SLAs. (Service Level Agreements).

Here is the alchemical formula for building a reliable system out of unreliable components.

  1. Recognize the problem. The component is unreliable. Say it out loud, in a design review, in front of witnesses.
  2. Characterize the flaws. How exactly does it fail? What are the failure modes, how often, under what conditions, and what does failure look like from the outside?
  3. Detect the flaws. Build a mechanism that can tell, at runtime, that this specific failure just happened.
  4. Address the flaws. Have a strategy for when detection fires: correct, retry, degrade, or escalate to a human.

Every one of those four steps is hard in the LLM domain. Characterizing the failure modes of a stochastic natural-language component is far nastier than characterizing a bad disk sector. But they are all engineering problems, and engineering problems have engineering answers. Nobody is coming by with a pixie dust dispenser.

I stepped on this rake

I ran into all of this while building the AI Rosetta Stone project, part of my fellowship at Harvard’s Berkman Klein Center.

The system runs debates. Different AI points of view argue a topic against each other. I designed them to argue like lawyers, and arguing like a lawyer means you don’t just say things. You respond to the opposing party through a set of well-defined dialectic moves. Fifteen of them, grounded in computational dialectics and argumentation theory.

I started running debates. The engine hallucinated a sixteenth move.

My first reaction was the wrong one. I assumed the model had found a real gap in my taxonomy. Maybe my fifteen weren’t good enough. So I added the new move to the list. Then it hallucinated another one. I added that. Then another. I added that too.

Each addition made me more uncomfortable, because the fifteen weren’t arbitrary. They came from theory. I was letting a semi-random word generator with an excellent vocabulary and a vastly overdeveloped sense of confidence edit my epistemology, one shrug at a time. And it didn’t even work. It just kept inventing new moves.

So I did what everybody does. I went back to the prompt. I told the model, in plain language, that these were the only fifteen dialectic moves it could use. It hallucinated a new one. So I added a self-check at the end: review your dialectic move, and if it is not one of these fifteen, do not proceed. It reviewed its move, concluded it was smarter than me, and hallucinated anyway.

Prompt engineering has its place and it can make things better. For this problem it was the wrong tool.

Now look back at the last five paragraphs. Hallucinated. Hallucinated. Hallucinated. Hicks and his co-authors would tell me that word smuggles in a faculty that was never there, and they’d be right. Nothing misfired. The sampler did exactly what samplers do. The word makes the failure sound like an accident, which is precisely why it’s so comfortable to reach for, and precisely why I kept fiddling with prompts instead of building a boundary.

LLM output is user input

Then it clicked.

We have to treat LLM output the way we have always treated user input.

Only the worst engineers take user input and use it directly. Good engineers know that people make mistakes constantly. They fat-finger, they paste garbage, they lean on the keyboard. Beyond honest mistakes, you have actual bad actors crafting input to run a SQL injection against you. So we learned, decades ago and at considerable cost: never trust user input. Validate it at the boundary, before it touches anything that matters.

An LLM sits on the far side of exactly that kind of boundary. It is an untrusted producer of strings. The fact that the strings are eloquent is not evidence of anything at all. Trusting model output because it reads well is the same category error as trusting a form field because it’s spelled correctly.

We absolutely have to do this for LLM models but the reality is that general purpose LLM systems (Gemini, Claude, DeepSeek, etc) also hallucinate even when they try to be truth tellers.

What I actually built

I changed the system so that every LLM output is a JSON document conforming to a well-defined schema. Then I validate it in two passes.

First the form: does it parse, does it match the schema. Then the content, through a series of what I call mini-fixers. Small, specific validators that each know exactly one thing about the domain. For this field, the mini-fixer asks one question: is this value one of the fifteen moves?

When it isn’t, we go into a repair strategy. Compute the cosine similarity between the invented move and each of the fifteen legitimate ones. If the best match clears a confidence threshold, remap it. Usually the model meant one of mine and just reached for its own vocabulary. If it doesn’t clear the threshold, retry.

Detect, then correct. It’s the RAID playbook with a weirder failure mode.

The irony here is not lost on me. I spent a big chunk of my career arguing that a shell should pass structured objects instead of doing fragile, prayer-based text parsing. Twenty-six years later I’m having the same fight with a different unreliable component. Similar problem. Similar answer. Constrain the interface, validate at the boundary, and stop praying that your masking tape holds the plane together.

There is no pixie dust

LLMs look like magic. You have a big hard problem, you sprinkle some pixie dust on it, and the problem goes away.

There is no pixie dust. My system only looked like it was working, right up until I did real QA on it. That gap, between it demos beautifully and it survives inspection, is where an enormous number of AI projects are quietly living right now.

LLMs are still magic. I mean that. What these things can do is astonishing and I’m not walking any of it back. But if you’re building a system where it actually matters whether it works, you have to throw away the rose-colored glasses and do the unglamorous work of engineering.

The sooner executives understand that and stop laying off the engineers who do it, the better off everyone will be.

Because that’s the whole argument in one line. The model is an engine of bullshit. Your system does not have to be.

That’s a choice, and it’s yours. Skip the engineering and you’re shipping a bullshit system. Do the engineering and you’re not.

Cheers!

Jeffrey

Notes

Harry G. Frankfurt, On Bullshit (Princeton University Press, 2005). Originally published as an essay in Raritan in 1986.

Michael Townsen Hicks, James Humphries, and Joe Slater, “ChatGPT is bullshit,” Ethics and Information Technology 26, 38 (2024). doi.org/10.1007/s10676-024-09775-5

The Hicks paper drew published pushback, which is what should happen to a good provocation. See David Gunkel and Simon Coghlan, “Cut the crap: a critical response to ‘ChatGPT is bullshit’,” Ethics and Information Technology 27, 23 (2025). doi.org/10.1007/s10676-025-09828-3

Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete

Agent swarms and the new model economics

1 Share
Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete
Next Page of Stories