Log in
An eval score tells you whether the agent finished the task. To improve how agents work with your product, you need to see where they struggled, and that’s in the trajectory, the record of every step the agent took and why. Agents are persistent. They guess another URL after a 404 and try values until an error goes away, so the struggle rarely shows up in the score.
We saw this when we evaluated Dev Proxy, a local proxy that intercepts your app’s HTTP requests. We asked GitHub Copilot Chat with Claude Sonnet 5.5 and no extensions, five times, to write a CI script that uses the Dev Proxy API to record which GitHub APIs a Node.js tool calls. All five scripts worked. Yet the trajectories showed 15 problems, and each run took 29 to 39 requests and up to 1.2 million tokens.
What to look for in a trajectory
Agents can trip over your technology in many ways. We picked the patterns we think you’re most likely to find in your own trajectories, from the obvious to the surprising, and for each one show what it looked like with Dev Proxy and which fix it pointed to.
Agents guess your URLs before they search
Agents often skip the search. They fetch the URL where they expect your page to be, based on what they remember about your docs and how docs sites are usually organized. Every wrong guess costs a step and tokens, and after a miss or two, some agents give up on your docs entirely. From then on, they work from memory and trial and error, which is where outdated and invented answers come from.
Before reading any docs, four of the five Dev Proxy runs tried to fetch https://learn.microsoft.com/en-us/microsoft-cloud/dev/dev-proxy/technical-reference/api, which returned a 404. The model remembered where Dev Proxy keeps its reference and got only the page name wrong, since the real page is technical-reference/proxy-api. Two runs spent five to seven extra steps finding it, and the other two gave up on the docs altogether.
The guessing continued after the agents read the docs. Once one run found the real API page, it fetched technical-reference/plugins/executionsummaryplugin, likely because the docs’ navigation groups plugins in their own section. That page lives at technical-reference/executionsummaryplugin. In total, eight fetches across four runs returned a 404.
Check your product: A URL structure that feels logical to your team can still send agents to the wrong place, because agents predict paths from patterns, like the structure of your navigation. Look through your 404 logs for docs paths that almost exist. The logs might not tell you an agent sent them, but each one is a redirect worth adding. Print your docs URL in your CLI output too, so agents don’t have to guess.
Agents probe when your interface stops one step short
Teams often write help output and error messages for a person who can open the docs when something’s unclear. An agent, however, treats them as the spec. When they say what’s wrong without saying what’s right, the agent guesses a value, reads the next error and guesses again. Each round costs a request, and if no guess works, the agent either stops or builds around the gap.
Dev Proxy lists its API endpoints like this:
POST /proxy Update proxy status (e.g. start/stop recording)
GET /proxy/rootCertificate Get the root certificate
It doesn’t show request bodies or required parameters. So when one run called /proxy/rootCertificate exactly as listed, it got a 400:
{ "errors": { "format": ["The format field is required."] } }
The error doesn’t list valid values, so the agent tried pem, crt, cer and der. Only crt worked.
Check your product: Run your CLI help and trigger your most common errors. Make sure each one tells the agent what a valid input looks like, with an example or the list of allowed values.
Every run reinvents the same workaround
When your product lacks something the task needs, agents build it themselves. In a single run, that’s noise. When every run writes the same workaround, your product is missing a feature, and every developer whose agent hits that gap pays for the workaround in tokens and ends up with code that’s more fragile than the feature would have been.
Dev Proxy writes its reports shortly after recording stops, with no signal when they’re done. Four runs polled the output folder for up to 30 seconds. The fifth just waited two seconds:
await api('POST', '/proxy', { recording: false });
await sleep(2000); // let the reporters flush their files
On a busy CI runner, that will eventually fail. Polling is safer, but every run that polled had to guess which file to wait for.
The strangest workaround though, was about plugins. Dev Proxy has no command that lists them, so all five runs read the plugin DLL as raw bytes and searched it for class names:
$b = [IO.File]::ReadAllBytes("$d\plugins\DevProxy.Plugins.dll")
$s = [Text.Encoding]::Unicode.GetString($b)
[regex]::Matches($s, '\b[A-Z]\w*(Plugin|Reporter)\b') | % { $_.Value } | sort -u
It worked, so no score would ever show it, but it’s far from robust not to mention efficient. A trajectory does show it, because the same odd step shows up in every run.
Check your product: Every operation that finishes after the call returns needs a way to tell the agent it’s done. Everything an agent has to choose from, like plugins or templates, needs a command that lists it.
Agents fill gaps from memory
Developers use your tool together with other runtimes and frameworks, and your docs often stop at your tool’s edge. When the docs don’t explain how the pieces fit together, agents fill the gap with what they learned in training. If the model happens to know, the run succeeds and you never hear about it. If it doesn’t, or it learned an older version, the agent writes code that looks right and quietly does nothing.
For example, to route a Node.js app through Dev Proxy, the app’s fetch needs an opt-in setting that only recent Node versions support. Every run knew this from memory:
since undici doesn’t respect HTTPS_PROXY by default, I need NODE_USE_ENV_PROXY enabled, which requires Node 22.21.0+ or 24.0+.
This time, all five got it right, but none found it in the Dev Proxy docs. A model that knew less would have recorded nothing, without an error, and you can’t predict what a model knows about your product. In a trajectory, these gaps show up as reasoning that states a fact without citing a source.
Check your product: List the runtimes and frameworks your developers pair your tool with, and make sure your docs show the setup for each. For the ones they skip, run a task and see what the agent fills in.
Agents reject defaults built for a first run
Default configs and starter templates usually serve a first run. They let developers check that the setup works and see what the product can do within minutes. For that job, they’re the right default, but for a specific task, they often get in the way. When an agent rejects your default, it builds its own setup from scratch, and each run builds a different one, so developers get inconsistent results for the same task.
Dev Proxy’s default config simulates random errors on a sample API. It shows right away that requests are flowing through the proxy and what Dev Proxy can do with them, but for recording real traffic, it’s in the way. The agents avoided it immediately:
I should avoid the default config since it’ll inject random errors, so I need to set up my own configuration instead.
When they created a new config, it came out empty. So each run built its own, and they picked three different plugins for the same goal.
Check your product: Keep the default that makes a first run work, and next to it, offer a preset for each common task developers give agents, so every run starts from the same, correct place. List those presets in your CLI help, which agents read early.
Agents trust whatever search shows them
Agents often read your guidance through a search engine, and search indexes serve snapshots. A page you updated last month can still show up in search with its old content, and the same goes for your translations and the skills you ship for agents. Agents can’t tell an old snapshot from the current page, so after a breaking change, an outdated copy that ranks well keeps producing broken code long after you’ve fixed the source.
Since v3.3.1, the Dev Proxy API requires a token. One run’s search returned two results indexed before that change. One was the Ukrainian translation of the API reference, crawled in June. The other was a file from Dev Proxy’s own agent skill, which had been updated with auth six weeks before the run. Both showed calls without a token, which now return a 401:
curl -X POST http://localhost:8897/proxy \
-H "Content-Type: application/json" \
-d '{"recording": true}'
The run got auth right only because it had already read the CLI help. CLI help ships with the installed version, so it can’t go stale.
Check your product: Take a few tasks developers give agents and run the web searches an agent would run for them, like “ API authentication”. Check whether the results, including translations and your own skills, describe your current version. You can’t refresh someone else’s index, but you can state version requirements at the top of each page, where they show up in search snippets and tell the agent which version a snapshot describes.
Agents assume the defaults your help text doesn’t state
Agents learn your CLI from its help text, and they take it literally. When an option reads like a switch you turn on, agents assume it’s off unless they pass it. If it’s actually on by default, they never pass the value that turns it off, because nothing told them to. For most options, the cost is a run that behaves differently than the agent expected. For a default that changes something outside your tool, the agent ships a script with a side effect it doesn’t know about, and because the task still succeeds, your eval won’t catch it either.
Dev Proxy sets itself as the system proxy by default, which routes all of the machine’s traffic through it. On a developer’s machine, that’s convenient, because every app goes through the proxy without extra setup. But the help text reads like an opt-in, and it doesn’t mention that turning the option off takes an explicit --as-system-proxy false:
--as-system-proxy Set Dev Proxy as the system proxy
One run found out by accident, when a status command showed System proxy: Yes, and turned it off. Another turned it off in its config, and the other three never noticed. On a Windows CI runner, their scripts would reroute all the runner’s traffic, and their kill fallback would skip the cleanup that undoes it.
Still, all three passed, because the system proxy wasn’t part of this scenario’s evaluation criteria. Meanwhile, the judge wrongly failed the run that did turn it off. The score was wrong in both directions, and only the trajectories showed it.
Check your product: Read your help text the way an agent would. Every option should state its default, and every option that’s on by default should show how to turn it off. Pay extra attention to defaults that change anything outside your tool, like system settings or installed certificates, and consider turning those off when your tool runs unattended.
Fix your product and docs before you reach for a skill
Some problems need a change to your product, like a command that lists plugins or help text that states its defaults. Others only need better docs, like a redirect for a guessed URL or a setup guide for the runtime your developers use. The pattern you found in the trajectory usually tells you which one, and both are parts of the AX stack you control.
Shipping a skill is tempting, because it feels like less work than changing your product. But a skill moves the problem to your developers. Agents now get it right only for developers who find the skill and install it. Those developers also have to keep it current, because an installed skill is one more copy of your docs that falls behind your product. In comparison, a fix in your product or docs reaches every agent that uses them, with nothing for developers to install.
With SPFx, a trajectory showed the agent reading the release notes while it worked. So instead of shipping a skill, we fixed the release notes, and every run used the right tool without anything extra installed. Start with your product and docs, and reach for a skill when they can’t carry what the agent needs.
How to start reading trajectories
Read your trajectories yourself, including the ones from passing runs. You know your product inside out, so you’ll spot workarounds that an agent reviewer would take for normal steps. ATIF Preview turns a trajectory file into a step-by-step timeline in VS Code, and an agent can help you compare runs, as long as it cites the step behind each finding.
Even then, each finding is a hypothesis until you run the same tasks with and without the fix and compare quality and cost, as described in how to measure AI agent extension effectiveness. Every change you make should leave agents better off with your product than before, and the measurement is how you prove it did.
The post Your best AX fixes are hiding in agent trajectories appeared first on Microsoft for Developers.
