There are lots of reasons to believe AI has shifted the bottleneck from coding to code review. That's wrong for two reasons, and we can prove it by answering one question.

For the system you currently work on, how many changes have passed code review but haven't been deployed and enabled for users?

The further this number is from zero, the less likely it is that coding or code review is your constraint.

The difficulty is that, within the software delivery industry, we've grown so used to certain practices that they now look like they belong. When you search for ways to improve software delivery capability, you don't see them. These practices are so deeply embedded that they are overgrown with moss and are indistinguishable from the hills.

When changes are collected in batches after code review, it proves that code review isn't your bottleneck, and coding wasn't either.

The desire for fast software delivery

Writing the code is a very small part of a long value stream. A value stream begins with an opportunity to provide something people want and ends when they get it.

Few organizations are building completely unique software. If the primary factor were speed, they would use something off-the-shelf that's "good enough". As someone has decided to spend vast sums of money creating something bespoke, you need to understand those other needs. What is it that makes it valuable enough to spend so much money on it?

To answer this question, you build the software, make sure it's releasable, and get it into the hands of the people who need to use it. Only then can you see if you're building the right thing.

The two errors

Since AI arrived, many people have proclaimed that the bottleneck has shifted from coding to code review. This isn't quite right, as coding wasn't the bottleneck in the first place (error 1), and it's not code review now (error 2). The reason we think either of these things is constraining the flow of value is that mossy hill we all stare past when we look at the mountains.

:::figure

:img{ src="/blog/img/code-review-not-bottleneck-even-with-ai/deployment-batch-size.png" alt="Chart showing 92% of teams have deployment batch sizes of more than 1 change." loading="lazy" }

::figcaption[Number of changes per deployment batch]

:::

Just 8% of teams deploy changes independently, while 92% deploy in batches. Only 8% are correct when they say the bottleneck moved from coding to code review. For the vast majority, this isn't true.

Batches are signposts

When you ask this batch size question, you peel back the moss and uncover what's beneath. This is how you find the true constraint in your value stream. You might have manual verification steps, a cumbersome change approval board meeting, or no easy way to deploy changes to development, test, and production. These are the things holding you back. Not coding and not code review.

You were likely working in batches before your AI initiative, and AI will certainly result in larger change size and larger batches if you don't pay attention to your constraints. Speeding up the coding stage only adds pressure to the real bottleneck.

Using the constraint to set the pace of your value stream will lead you to invest improvement efforts where they matter most. If you find your retrospectives fail to provide noticeable improvements, it's likely because you're missing that mossy mound. It's the reason some AI initiatives fail to deliver a return on investment, while others succeed.

But… the data

Some studies, like GitLab's 2026 AI Accountability Report, also say that bottlenecks have shifted from coding to code review, but they also overlook the mossy mound. Where they haven't captured the size or number of changes that haven't reached production users, they can't observe the true bottleneck.

Having invested in AI to increase coding speed, you'll now be tempted to invest in solving the code review problem, or to abandon it. If you're working in batches, you'll discover this makes little difference to your ability to deliver valuable software.

The reasons your organization is resisting solving the batch problem are the real problems you need to fix.

Look for the accumulation of changes throughout your process. Identify the true constraint, and apply the five focusing steps to manage it; and that includes making all other stages march at the pace of the constraint.

Happy deployments!

You’ve built an eval. You’ve established a baseline and found the gaps. Now you have hypotheses about how to fix them: maybe a documentation rewrite, or a different CLI output format. The natural next step is to ship the change and measure again. Don’t do that just yet.

Why most hypotheses fail

Every improvement you think of is a hypothesis. “If I add a warning callout to the release notes, the agent will use the CLI instead of attempting a manual upgrade.” It sounds reasonable, might even be obvious. But LLMs don’t reason the way you expect them to, and obvious changes produce counterintuitive results with uncomfortable regularity.

We’ve seen this first-hand working on a SharePoint Framework (SPFx) project upgrade scenario. SPFx projects upgrade incrementally, version by version, each step introducing its own configuration- and dependency changes. The agent needed to discover a CLI tool from documentation. The documentation included a tip recommending the CLI. Straightforward, right? The agent ignored it in every single run. The tip was there, the agent read the page, and it did nothing differently.

The intuition was that a recommendation in the docs would influence the agent’s behavior. The reality was that the agent had already formed a plan from its training data before it fetched the page. The documentation confirmed parts of that plan (package names, version numbers) and the tip got filed away as optional side information. Advisory content does not override an existing plan. The agent treats “you could also try X” as noise when it already has a strategy it believes will work.

What actually worked was not what we expected

We tested a dozen variations. We moved the tip to a different position on the page. Then, we reworded it. We even included the exact command to run. Nothing made a difference, and the most useful mental model for understanding why: the agent processes an entire page and gravitates toward actionable step-by-step content. A tip, by definition, is not a directive.

What finally worked was a warning that told the agent its current plan would fail. Not “here’s an alternative” but “your approach will result in build failures.” Five out of five runs flipped to using the CLI. The agent abandoned its plan because the documentation told it the plan was broken, not because a better alternative existed.

Nobody on the team predicted this. We assumed stronger wording in the tip would suffice. We were wrong, and we would have shipped a useless documentation change to production if we hadn’t tested it first.

The compounding problem

That was just one hypothesis. In the same engagement, we tested over a dozen more, and the hit rate was not encouraging:

• Removing links to a competing migration guide helped CLI adoption but destroyed overall quality (dependency accuracy dropped from 34/50 to 18/50) because the four out of five runs that still didn’t find the CLI no longer had the guide to work from.
• Adding structured JSON output to the CLI improved configuration correctness from 72/85 to 85/85, but switching to JSONL (which seems superficially similar) cost 2.6x more tokens with worse outcomes across every dimension.
• PowerShell syntax in code fences bled into the agent’s other work, degrading idiomatic code quality even in files unrelated to the upgrade task.
• Providing an explicit script to execute did not cause the agent to execute it. In zero runs out of ten did the agent pipe the script to a shell. It always decomposed and applied steps individually.

Each of these results contradicts a reasonable assumption. “JSONL is structured too, it should work as well as JSON.” “If I give the agent a script, it’ll run it.” “PowerShell is just a different shell syntax.” It turns out, that every one of these intuitions was wrong. Not slightly wrong: wrong to the point of tripling token costs or halving quality scores.

Why you need to emulate before you ship

If you’re changing documentation on a live site, every hypothesis means a pull request. Someone reviews it, someone merges it, it deploys, and then you measure. If the hypothesis was wrong (and many of them will be), you now have a live documentation change that’s either neutral or actively harmful, and you need another PR to revert it.

If you’re changing an MCP server response format, you’re deploying a new version to test something that might not work. If it’s a public API behind your tooling, you’re shipping changes that affect real users while you figure out whether your theory holds.

The iteration cycle for “change live infrastructure, measure, revert if wrong” is measured in days. The iteration cycle for “emulate the change locally, measure, ship only what works” is measured in minutes. When you’re testing a dozen hypotheses and most of them won’t pan out, the difference is between testing three ideas in a week and testing thirty.

What emulation looks like in practice

You intercept the requests your agent makes and return modified responses as if the change were already live. Your agent calls the same URLs, gets responses that reflect your hypothesis, and you measure whether behavior improves.

For the SPFx work, we used Dev Proxy to intercept requests to Microsoft Learn and return modified page content. The agent called https://learn.microsoft.com/en-us/sharepoint/dev/spfx/release-1.22 and got back a version of the page where we’d added the warning callout, without touching the live page. When the hypothesis worked, we shipped the change. When it didn’t (like the tip-only experiments), we moved on in minutes instead of days.

{
  "request": {
    "url": "https://learn.microsoft.com/en-us/sharepoint/dev/spfx/release-1.22",
    "method": "GET"
  },
  "response": {
    "statusCode": 200,
    "body": "@modified-release-notes.html",
    "headers": [
      { "name": "Content-Type", "value": "text/html" }
    ]
  }
}

The same pattern applies to MCP servers. If your agent calls an MCP endpoint and you want to test whether a different response format produces better outcomes, you can emulate the modified response without deploying a new version. Test locally, validate the hypothesis, ship only what’s proven.

No test environment to stand up, no PRs to merge and revert. The agent doesn’t know or care that a proxy is answering instead of the real server. You get deterministic, repeatable experiments against the same stable URLs your agent will use in production.

The experiment mindset

Improving agent experience requires an experimental discipline most teams aren’t used to. When you optimize a REST API, you can reason about the outcome: faster query, smaller payload. The relationship between change and outcome is predictable. With LLMs it isn’t. A change that seems cosmetic (switching from a tip to a warning) can flip behavior completely. A change that seems significant (providing a ready-to-run script) can have zero effect.

The only way to know is to measure. And measuring is only practical if the cost of each experiment is low enough to run many of them. This is why the teams that improve agent experience fastest are the ones that make experimentation cheap. They aren’t smarter about predicting LLM behavior. They just test more hypotheses per week because their iteration loop is shorter. The eval infrastructure is what gives you the ability to measure. Emulation is what gives you the ability to iterate.

Start with what you control

You don’t need a complex setup to begin. If your agent consumes documentation, you can emulate documentation changes. If it calls an API, you can emulate response changes. And you can also introduce new API endpoints, -operations and docs pages. Start with whatever surface your agent interacts with most, form a hypothesis about what would improve it, and test that hypothesis without shipping anything.

Here’s why this matters: every finding in this article came from our own experience of starting with a reasonable assumption and discovering through measurement that our assumption was wrong. We genuinely expected the tip to work. We expected the script to get executed. We were wrong on both counts, and on most of the others too. Models don’t have preferences, they have context. The context you provide determines what happens, but predicting exactly how is not something humans are good at yet.

Test your hypotheses. Most of them will be wrong. Make that cheap, and you’ll converge on what actually works.

The post How to test agent experience changes without shipping them appeared first on Microsoft for Developers.

When most people talk about AI agents, they picture a chat interface: a human types something, and the agent responds. That’s a real and useful thing. But it’s also only a small corner of the agent landscape.

There’s another category of agent — one that has no chat interface at all. No human types a prompt. No human reads the output. The agent wakes up in response to some event, does its work, and disappears. I call these headless agents, and I think they’re going to be far more common in business and enterprise settings than their chat-facing counterparts.

What Triggers a Headless Agent?

A chat agent is triggered by a human’s input. A headless agent is triggered by something else. That “something else” is the interesting part:

• A new row is inserted into a database table
• A record is updated and crosses some threshold
• A message arrives in a queue
• A file lands in a storage container
• A scheduled timer fires
• An API webhook delivers an event
• Another agent hands off a task

In each of these cases, the agent wakes up, examines the situation, applies some judgement, takes some actions, and shuts down. No human in the loop. No chat window. No one watching in real time.

That’s a headless agent.

Why These Will Dominate in Enterprise

Think about the actual day-to-day work that happens in most business roles. Plenty of it is genuinely interesting and requires human creativity, empathy, and deep domain expertise. But some significant fraction of it is… not.

How many people have a valuable job, but also need to check a specific inbox every morning and route some emails? Or stop their real work a few times a day to fill out some paperwork, send some notifications, or update a spreadsheet that feeds into some downstream process?

These tasks often require limited, but some judgement. Enough that a hard-coded rule engine tends to fail. Not enough that they couldn’t be handled by an agent with a well-crafted directive and access to the right tools and data.

These are the low-hanging fruit of AI automation. Not replacing people’s entire jobs — replacing the repetitive, peripheral tasks that orbit around people’s actual work. The tasks that fragment focus, consume time, and produce nothing that requires a human.

In my view, this is where agents will have the most immediate and measurable impact in business and enterprise settings. And these are all headless agents.

Same Architecture, Different Trigger

A headless agent is still an agent. As I’ve described before, every agent is:

Agent = Harness + LLM + Directive

The Harness is the code that orchestrates everything — the loop, the tool calls, the context management. The Directive is the system prompt that shapes the agent’s behavior and scope. The LLM does the reasoning.

For a chat agent, the Harness receives user input and returns a response to a UI. For a headless agent, the Harness receives an event — a queue message, a database notification, a webhook payload — and acts on it.

Structurally, these are the same thing. The difference is what initiates the work and what happens with the result.

A headless agent’s Directive is often narrower than a chat agent’s. It doesn’t need to handle arbitrary conversational context. It has a specific job: when this kind of event arrives, here is how to handle it. That narrowness is actually a feature. A tightly scoped Directive means the agent is more predictable, more testable, and less likely to go off in unexpected directions.

Headless Agents Absolutely Require Observability

This is not optional.

With a chat agent, there’s a human in the loop. If the agent produces bad output, the human notices. They can ask a follow-up question, try again, or escalate. The human is the safety net.

With a headless agent, there is no human watching. Nobody sees the output until some downstream consequence reveals that something went wrong — maybe days or weeks later, maybe never in a way that’s attributable to the agent.

I’ve written about observability for RockBot, and everything I said there applies here — with even higher stakes. For headless agents, you need:

• Structured logging for every invocation. What event triggered the agent? What did it do? What tools did it call? What was the final output?
• Metrics on token consumption and cost. Headless agents can run at high volume. A bad prompt that causes looping or excessive LLM calls can silently burn budget before anyone notices.
• Latency tracking. If an agent is supposed to process events in near-real-time but its p95 latency is climbing, something is wrong with the pipeline.
• Failure alerting. If the agent fails to process an event — or processes it incorrectly — someone needs to know. Not eventually. Now.

Without these things, you’re running automation that nobody can see, that nobody can debug, and that nobody will notice is broken until the damage is already done. That’s not automation — that’s a time bomb.

The rule I’d apply: if a human used to do this task and would have noticed when something went wrong, your headless agent needs to be able to surface that same signal through instrumentation.

Where Headless Agents Run

Headless agents are event-driven by nature, which makes them a natural fit for event-driven infrastructure.

Azure Functions are an obvious choice. A Function can be triggered by a queue message, a database change, a blob arriving in storage, a timer, an HTTP call — basically everything on the list above. The Function runs, does its work, and shuts down. You pay for what you use, not for idle time.

KEDA (Kubernetes Event-Driven Autoscaling) is another option, particularly if you’re already running on Kubernetes. KEDA can scale your agent workloads to zero when there’s nothing to process, and back up when events start arriving. The same economic principle: no idle resource cost.

Other event-driven platforms — AWS Lambda, Google Cloud Run, and similar — all apply the same model.

The key property all of these share is that the agent doesn’t sit idle waiting for work. It responds to demand. This matters because the “running cost” of a headless agent that processes 10 events a day is dramatically different from a long-lived service that holds a connection open around the clock.

Headless Agents Are Still Connected

Being headless doesn’t mean being isolated. These agents use the same ecosystem as any other agent.

A headless agent can call MCP servers to query databases, update records, send notifications, call external APIs, or retrieve documents. It can use A2A (agent-to-agent) protocols to delegate sub-tasks to specialized agents. It can enqueue messages to downstream systems or other agents as part of its processing.

In a mature enterprise agent ecosystem, you’d expect to see headless agents passing work to each other via queues — one agent processes an incoming order event, enriches the data, and publishes a message that triggers another agent to run a credit check, which in turn triggers another agent to update inventory. Each agent has a clear, bounded responsibility. Each can be deployed, scaled, and monitored independently.

This is the agentic equivalent of a well-designed microservices architecture: loosely coupled, independently deployable, observable at every boundary.

Where This Is Headed

I have no doubt that headless agents will quietly become the most important category of AI deployment in business. Not because they’re flashy — they’re the opposite of flashy — but because they’re where the actual automation happens.

Chat agents are visible and immediate. They’re easy to demo and easy to understand. But the work that changes how a business operates happens in the background: processing events, making decisions, routing tasks, updating records — all the repetitive, judgement-light work that currently fragments so many people’s days.

When those headless agents are running well, nobody notices. The inbox gets processed. The notifications go out. The spreadsheet gets updated. People get their time back.

When they’re running poorly — or not at all — somebody definitely notices. Which is exactly why observability isn’t an afterthought for these systems. It’s the whole game.

If you’re thinking about where AI agents can make a real difference in your organization, I’d encourage you to look past the chat interface. Look at the event-driven work that’s happening every day. Look at the peripheral tasks that fragment your team’s focus. That’s where headless agents belong.

This post was authored with the assistance of AI.

OpenClaw 2026.6.33