Have you ever made a small change to an API, shipped it with passing tests, and then heard from another team that you’d broken their integration?
Everything looked fine in your service. But somewhere else, a caller was relying on the field you changed, and you didn’t know that caller existed.
Now you’re searching for the endpoint path across repositories, following SDK imports, and reading client wrappers to figure out who else uses it. Some matches are actual callers. Others are examples, test fixtures, or outdated code. What you needed before merging was a list of consumers, including those not in the repo you checked out. Getting that list turns out to be its own engineering task.
A coding agent can help, but watch what it spends its time doing. It searches, opens files, follows references, and searches again. Much of that work is figuring out where the relevant code lives. And finding a client implementation only answers part of the question. To know whether it’s deployed and still making calls, you need deployment and runtime data too.
Think of a landlord changing the locks. They need to reach everyone who has a key, including the dog walker, the housekeeper, and the tenant’s cousin who waters the plants. Without a record, they have to knock on doors and ask around. Being faster at knocking helps, but it doesn’t help them remember someone in another building.
A key register gives them a starting list. It records who has a key and which door it opens. They still need to check whether each person uses it, and whether anyone is missing. But they can start by contacting the people on the list.
The Postman Context Graph does something similar for your API ecosystem. It indexes services, endpoints, and their relationships across connected sources. You ask which services depend on an API, and your agent gets a list of candidates to investigate. It still needs to check the evidence and establish whether each dependency matters to the change you’re making.
I wanted to know how much work that starting list saves. Does it help the agent reach a good answer with less searching, or does it only give the agent another list to work through?
So I built a harness and gave GPT 6.1 Terra the same 17 engineering questions twice: once with the code alone, and once with Context Graph results supplied before it started searching.
With the graph, Terra used 43% fewer input tokens, cost 43% less in model tokens, and spent 42% less time working. It took fewer steps and made fewer tool calls.
The headline numbers
Everything here is per question, averaged over 17 questions per arm (one run each).
|
Terra alone |
Terra + Context Graph |
Change |
| Cost (model tokens) |
$0.226 |
$0.128 |
43% less |
| Input tokens |
865,000 |
491,000 |
43% fewer |
| Output tokens |
6,850 |
5,930 |
13% fewer |
| Terra’s working time (average) |
11.3 minutes |
6.5 minutes |
42% less |
| Time to answer, including the graph query (average) |
11.3 minutes |
8.4 minutes |
26% faster |
| Steps per question |
13.4 |
10.2 |
24% fewer |
| Tool calls per question |
106 |
76 |
29% fewer |
| Services found per question (answer key) |
10.6 |
16.5 |
56% more |
Across the 17 questions, input tokens dropped from 14.71 million to 8.35 million, and total elapsed time dropped from about 3.2 hours to 2.4 hours.
At the same average model cost, 1,000 questions would cost about $128 instead of $226, roughly $98 saved, along with 374 million fewer input tokens. That’s an extrapolation from this run, and it excludes any charge for the graph query.
The efficiency wins held on most questions: the graph arm cost less on 15 of 17 and reduced Terra’s own working time on 16. The rest of the post shows where those wins came from, and where they didn’t.
What the Context Graph contains
The Postman Context Graph is a private, authenticated map of your APIs and the services that use them. It connects information from Postman, GitHub, and New Relic so you can trace a dependency without piecing everything together yourself.
Postman has the contracts and collections. GitHub has the implementation and call sites. New Relic has deployment and runtime information. Connecting those gives you a way to ask: who calls this API, are they running in production, and who owns them?
Say you’re changing the amount field returned by GET /claims/{id} from dollars to cents. A $125 claim currently returns 125.00. After your change, it returns 12500.
You update your implementation and tests, and everything passes. But what happens to a service that reads that value and still assumes it’s in dollars?
Here’s how I’d investigate that in a hypothetical system:
| Question |
What you find |
Why it matters |
| What does the API contract say? |
amount is documented in dollars, with 125.00 as an example. |
You’re changing what the value means. Keeping the same field name and type doesn’t make the change compatible. |
| Who uses that value? |
settlement-worker calls the endpoint and uses amount to calculate payments. |
You need to read that calculation. If it assumes dollars, it could calculate a payment 100 times too large. |
| Is that worker running? |
It has a production deployment, and runtime data shows calls to claims-service. |
This is a consumer you need to check before shipping. You still need to establish whether those calls reach the endpoint you’re changing. |
| Who should review the change? |
The payments team owns the worker. |
Now you know who to talk to about updating the caller and coordinating the rollout. |
| Where else does the amount go? |
Claims-service publishes a claim.updated event that a reporting service consumes. |
Check whether the event shares the same conversion or serialization code. You could change the event’s units accidentally while updating the HTTP response. |
A code search might find the worker’s client. The graph helps you connect that client to a running service, an owning team, and other dependencies worth checking. Your agent still needs to inspect the code and establish which consumers are affected.
You’ll also hear these relationships described as upstream and downstream. In this example, settlement-worker is a downstream consumer of claims-service. Claims-service is an upstream dependency of the worker.
When you’re changing claims-service, you want to know who depends on it. When you’re trying to run settlement-worker locally, you want to know what it depends on so you can run or simulate those services.
You connect your sources once, and the graph refreshes nightly. It can only work with what’s been connected and indexed, so its results still need checking. Finding a caller in code tells you how the integration works. Runtime data helps you establish whether it’s being used. Together, they give you a better basis for deciding what needs to change before you ship.
The Context Graph documentation covers the supported sources and relationships.
How we tested it: 17 questions, 802 repositories, 34 runs
The claims example is the kind of work this benchmark tries to measure: discovering candidate consumers, then verifying them against the code. We supplied graph results before Terra searched, then measured its token usage, tool calls, execution time, and answers. We did not use these runs to prove that every indexed relationship was correct or that the graph prevented a production incident. The benchmark evaluates repository discovery against code-derived answer keys.
We tested Terra on 17 engineering questions about one public GitHub organization with 802 repositories. We searched that code to build an answer key for each question, independently of the Context Graph. An answer key is the reference list of repositories the scorer expects for that question.
We then asked Terra each question twice:
- Without the Context Graph: Terra had to find the answer by searching the repositories.
- With the Context Graph: We asked the graph the same question first, then gave Terra its results as a starting list to check against the code.
Both arms used the same GPT 6.1 Terra model, the same pinned checkout, and the same read-only tools: read_file, list_dir, grep, glob, git_log, and git_blame. The prompts didn’t provide repository names as search hints or tell Terra how many results to find. The graph query is the same short question for every run of a given prompt, and it doesn’t name a repository, so we’re not handing Terra a search strategy. The baseline had no graph access. In the graph arm, the harness supplied the initial query results, and Terra still used the same code-search tools to investigate.
That gave us 34 runs: 17 questions, each tested with and without the graph. We ran each comparison once, so these results show what happened in this test. Repeated runs would be needed to measure how consistently Terra produces the same results.
Nine cross-repository dependency questions
Seven questions asked Terra to identify every repository integrating with a particular service, with a file and line as evidence.
The other two asked:
- Which repositories consume a Kafka topic published by a named service?
- Which external service has the most dependent repositories, and which repositories are they?
These are questions where the evidence is spread across repositories. There usually isn’t one file containing the complete answer, and some of it may not be in the checkout at all. Across the 17 questions, the graph arm named about 12 repositories per question that didn’t resolve to anything in the checked-out code, against about 2 for the baseline, which is the kind of reach a grep can’t give you.
Six inventory questions
Inventory questions don’t name a service to look for. They ask Terra to list everything in the organization that has a given property, such as every service that exposes HTTP endpoints or emits telemetry. The answers are large and open-ended. The answer keys range from 11 services (message consumers) to 161 (HTTP endpoints), so there’s no obvious point where Terra can stop searching and be done. The checkout can only show the repositories you have on disk. The graph can also name services that live in repositories you don’t have.
| Topic |
Question |
| HTTP endpoints |
Which services expose HTTP endpoints, and what are the paths? |
| Message consumers |
Which services consume from a message bus, and which topics? |
| Runtime calls |
Which services call each other at runtime? |
| Deployments |
Which services are deployed, and how? |
| Schemas |
Which services declare a machine-readable data or API contract? |
| Telemetry |
Which services emit traces, metrics, or errors about themselves? |
Two impact questions
We also asked which services call the organization’s own HTTP API, and which deployed services depend on an SDK facing a breaking change and would need redeploying.
None of the prompts told Terra where to look or how many answers to find.
How we graded: 70% answer key, 5% citations, 25% judge
The benchmark combines three checks, and the weights are fixed: 70% answer key, 5% citations, and 25% model judge.
1. The answer key (70%). The keys came from searching the checked-out code, never from the Context Graph. Grading the graph against its own output would be marking its own homework. For each answer, we report:
- Keyed recall: correct matches divided by the number of entries in the answer key. It’s the recall half of precision and recall.
- Precision: correct matches divided by the scored answer entries.
- F1: the harmonic mean of precision and recall,
2 × precision × recall / (precision + recall). A shotgun list can’t win on it.
- Services found: the absolute number of correct matches against the key. Matching is primarily at repository level, with component names normalized to parent repositories.
Recall tells us how much of the known dependency set Terra missed. Precision tells us how many reported results failed to match that set. Both matter: a list containing every correct service is less useful if it also includes hundreds of unrelated ones. On bounded questions, where there’s a fixed set of right answers, recall leads. On open-ended inventories, where volume and accuracy move together, services found leads.
2. Citations (5%). Every file path the answer cites gets checked against the repo at the pinned commit. Invent a path and you get caught. The check doesn’t, by itself, prove that the cited code supports the claim.
3. A judge (25%). A language model (Claude Opus 4.7) scores each answer against a rubric. It gets the smallest weight because it can’t inspect the repo the way the other two graders can.
A run passes at a combined score of 0.5. The benchmark also has a separate answer-key check that applies a precision factor to recall. Neither that check nor the combined score is interchangeable with recall or F1, so we don’t use them as accuracy measures here. Name matching allows underscores and hyphens to match, recognizes GitHub URLs, and credits a component like ingester-service to its parent repository.
What counts as the graph arm’s answer. For the answer-key part of the score, we score the graph arm on what the graph returned plus what Terra committed to, since both are part of what the system hands the user. The citation check and the judge see only Terra’s answer. We also score Terra’s answer separately, so you can see how much of any result comes from the graph and how much from Terra. The baseline has no graph output, so its score is its committed answer. That’s why we report recall in two versions later in the post.
Results: 43% cheaper, 42% less working time
Terra reads 43% fewer input tokens with the Context Graph
Input tokens accounted for about 99% of the baseline’s token volume. In this harness, each model step receives the accumulated conversation, including earlier tool results. Reading another file adds that file’s contents to every later request, and taking another step presents the existing transcript again. These input-token totals are cumulative across requests, rather than the size of a single prompt or a count of unique source-code tokens.
With the graph, average input fell from 865,000 to 491,000 tokens per question. Output fell from 6,850 to 5,930 tokens, including a drop in reasoning tokens from 924 to 740.
| Per question |
Alone |
+ Graph |
Change |
| Input tokens |
865K |
491K |
-43% |
| Output tokens |
6,850 |
5,930 |
-13% |
| Reasoning tokens |
924 |
740 |
-20% |
| Steps |
13.4 |
10.2 |
-24% |
| Tool calls |
106 |
76 |
-29% |
Whole-codebase grep |
52.5 |
32.7 |
-38% |
read_file |
44.9 |
35.2 |
-22% |
list_dir |
6.4 |
4.8 |
-24% |
The largest reduction was in whole-codebase grep calls, the blunt instrument you reach for when you don’t know where to look. That’s consistent with a narrower search: Terra starts with named candidates and inspects their files, rather than repeatedly searching the entire checkout. The aggregate counts support that interpretation, although they don’t establish that every remaining tool call was useful verification.
By question type, the nine dependency questions used 27% fewer input tokens, the six inventory questions used 67% fewer, and the two impact questions used 50% fewer. Some individual reductions were much larger. Runtime calls went from 1.30 million input tokens to 191,000, HTTP endpoints from 737,000 to 142,000, and API callers from 1.56 million to 419,000. Per question, the graph arm read fewer tokens on 15 of the 17.
Using fewer tokens matters for more than the bill. Chroma’s context rot research tested 18 frontier models and found every one degrading well before its context window filled, and the lost-in-the-middle study found accuracy dropping by 30% or more for information buried mid-prompt. We didn’t measure that effect here, but a shorter transcript is a smaller target for it. Postman’s post on why AI coding agents need context graphs covers the research in more detail.
The model bill falls by 43%
Here’s the cost across all 17 questions in each arm.
| Model token cost |
Terra alone |
Terra + Context Graph |
Saved |
| Uncached input |
$2.66 |
$1.32 |
$1.34 |
| Cached input |
$0.60 |
$0.35 |
$0.25 |
| Output |
$0.58 |
$0.50 |
$0.08 |
| Total |
$3.85 |
$2.18 |
$1.67 (43%) |
| Per question |
$0.226 |
$0.128 |
$0.098 |
| Per 1,000 questions |
$226 |
$128 |
$98 |
We calculated these figures using Terra’s published rates of $1.00 per million input tokens, $0.05 per million cached input tokens, and $5.00 per million output tokens. All input that wasn’t a cache read, including cache writes, was billed at the full input rate. The figures count model tokens only, and graph-query charges are excluded.
About 83% of input tokens were cache reads, which cost 20 times less than fresh input. That explains how a run processing hundreds of thousands of input tokens averages roughly $0.23 per question, and why the saving is smaller than a flat-rate calculation would suggest. Without the cache discount, the same token usage would cost $15.29 for the baseline and $8.85 with the graph.
The graph arm cost less on 15 of 17 questions, by as much as 74% on API callers ($0.535 to $0.138), and 57% on runtime calls.
Terra spends 42% less time working with the Context Graph
We measured both Terra’s working time and the total time to answer. The 42% reduction refers to model working time, which leaves out the wait for the graph. End-to-end answers were 26% faster on average.
| Minutes per question |
Terra alone |
Terra + Context Graph |
| Terra’s working time, average |
11.3 |
6.5 |
| Terra’s working time, median |
6.4 |
2.6 |
| Wait for the graph query, average |
none |
1.8 |
| Wait for the graph query, median / longest |
none |
0.6 / 7.0 |
| Time to answer, graph wait included, average |
11.3 |
8.4 |
| Time to answer, median |
6.4 |
8.1 |
| Time to answer, slowest quarter begins at |
19.1 |
13.3 |
| Time to answer, 95th percentile |
24.8 |
14.6 |
| Slowest single question |
25.3 |
15.1 |
| Total across the 17 runs |
3.2 hours |
2.4 hours |
Results: what happened to the answers
On dependency questions, same recall, less than half the time: 18.8 minutes → 10.3 minutes
A dependency question asks Terra to find every repository that relies on one specific thing. Seven of the nine ask which repositories integrate with a named service: Datadog, Google Cloud Storage, Slack, AWS S3, PagerDuty, Vercel, and OpenRouter. Each answer needs a file and line that proves the integration. The Slack question, for example, has an answer key of 12 repositories, and Terra has to find which 12 of the 793 they are. One question asks which repositories consume a Kafka topic published by a named service, and the last asks which single external service the most repositories depend on.
These are the questions closest to the “who else uses this?” problem from the start of the post. The code that answers them is spread across many repositories, with nothing listing it in one place.
Across these nine questions, recall was already around 90% in both arms. Terra could usually assemble the answer from the code. The graph reduced the work needed to get there: average working time fell from 18.8 minutes to 10.3 minutes, about 45% less.
Terra's working time, dependency questions (9) minutes
alone ████████████████████████████████████████ 18.8
+ Graph ██████████████████████ 10.3
The combined benchmark score, including the answer key, citations, and judge, rose from 0.822 to 0.858. It improved on seven questions, stayed level on the Datadog question, and fell on the most depended-on service question.
Each row below is one of the nine questions. The score runs from 0 to 1, higher is better, and blends the three checks described earlier: 70% answer key, 5% citations, and 25% judge.
| Question |
Alone |
+ Graph |
| Slack integrations |
0.78 |
0.94 |
| PagerDuty integrations |
0.94 |
1.00 |
| OpenRouter integrations |
0.80 |
0.85 |
| Google Cloud Storage integrations |
0.81 |
0.86 |
| Kafka topic consumers |
0.84 |
0.88 |
| AWS S3 integrations |
0.80 |
0.82 |
| Vercel integrations |
0.75 |
0.76 |
| Datadog integrations |
0.87 |
0.87 |
| Most depended-on service |
0.80 |
0.75 |
Slack is a useful concrete example. Terra alone found 11 of the 12 repositories that integrate with Slack. With the graph, it found all 12. Google Cloud Storage went from 16 of 18 to all 18. For the other named integrations, Terra was already finding everything or almost everything. There wasn’t much room to improve recall, but there was plenty of room to reduce the search.
This is the kind of question where the answer is spread out. A dependency like Slack or S3 shows up in dozens of repos, each in a different file, and nothing lists them in one place. File search has to assemble that list one grep at a time, while the graph already has the relationships indexed.
On inventory questions, recall nearly doubled: 0.22 → 0.42
On the six inventory questions, Terra alone found about 22% of the answer key on average. These were large inventories, including 161 services exposing HTTP endpoints and 137 emitting telemetry, and the baseline often stopped after a couple of minutes with a short list. With the graph, Terra’s own answer covered about 42% of the key.
| Question |
In the key |
Recall, Terra alone |
Recall, Terra + Graph |
| HTTP endpoints |
161 |
18% |
40% |
| Deployed services |
42 |
31% |
60% |
| Message consumers |
11 |
27% |
55% |
| Telemetry |
137 |
14% |
34% |
| Schemas |
28 |
32% |
39% |
| Runtime calls |
50 |
12% |
28% |
Recall improved on every inventory question, while input token usage fell by 67% across the group. The graph found what Terra couldn’t find by searching, and Terra got there reading a third of what it read before.
But Terra also carried more incorrect candidates into its answers. The graph hands Terra a long candidate list (240 names for endpoints, 277 for deployed services, 199 for telemetry), and on HTTP endpoints Terra’s final list grew from 36 names to 139. Across the inventory questions, precision on Terra’s own answers fell from 0.71 to 0.46.
That’s why I wouldn’t describe the inventory result as a clean accuracy win. Terra missed fewer real services and used fewer tokens, but a reader would have more false positives to check. The next thing to improve is how thoroughly the agent validates and prunes the graph’s candidates before presenting the answer.
Where the Context Graph earns its place
After running these tests, I’d start with questions that send an agent searching across repositories: who uses this service, who subscribes to this topic, or what depends on this API.
On the nine dependency questions, Terra already found most of the right answers. With the graph’s candidate list, it spent about 45% less time working and earned a higher combined score. Across all 17 questions, using the graph cost less on 15 and reduced working time on 16. Terra could already do the work. The graph helped it get there with less searching.
I’d be more careful with questions like “list every service.” The graph helped Terra find much more, but it also gave it more candidates to rule out. The relationships need to be clear, too. An app that sends errors through the Sentry SDK is doing something different from a service that calls Sentry’s REST API. Sorting that out can take time.
For an API change, I’d ask specifically for direct HTTP callers, SDK users, or topic subscribers. Then I’d have the agent check the code, show its evidence, and distinguish confirmed consumers from ones it still needs to verify. If I wanted to understand production impact, I’d also check what’s deployed and actually running. A dependency in the code doesn’t tell you that on its own.
The graph gives the model more facts and fewer places to search. The model still has to check whether those facts answer the question. These runs showed a substantial drop in search work, with room to improve how candidates are filtered and relationships are interpreted.
How an agent queries the Context Graph
The agent can query the graph through the HTTP API, the Postman CLI, or the Model Context Protocol (MCP). Authentication scopes access to the team’s graph. You can keep your existing coding agent and model, and the graph provides additional context at inference time.
The HTTP interface uses POST /context-graph/asks to submit a question and GET /context-graph/asks/{askId} to retrieve its result. Completed results can include an answer, structured entities, citations, and query provenance. The API reference defines the response fields. Through MCP, the corresponding tools are submitContextGraphAsk and getContextGraphAsk.
For the hypothetical claims change, I would ask something specific: “Which services directly call the claims HTTP endpoint, and what evidence identifies each caller?” That’s narrower than “what depends on claims?”, which might also include SDK dependencies, message consumers, or other relationships.
The agent then needs to inspect each candidate’s call site, check which endpoint or SDK operation it uses, and determine whether it relies on the response field being changed. A service-level dependency establishes a place to investigate. It doesn’t establish that every endpoint change will break that service.
Try the comparison on your own repositories
Pick a dependency question you already know the answer to, such as which services call billing-api. That gives you a way to check what the agent missed and what it incorrectly included.
First, make sure the graph has your data. In Postman, go to Home > Agent Context, click Manage Data Sources, then click Index. Indexing runs in the background, so you can keep working while it finishes. For a repository-level comparison, make sure the repositories you’re testing are included. The setup documentation covers source configuration. Then authenticate the Postman CLI with postman login and submit the question:
postman context-graph ask "Which services directly call billing-api over HTTP?" --wait --interval 5 --timeout 300
You can also connect an MCP-capable agent to the Context Graph server. For Claude Code, the configuration command is:
claude mcp add --transport http postman-context-graph https://mcp.postman.com/context-graph
Complete authentication for the MCP connection before querying. The US remote endpoint supports OAuth or an API key, and the MCP setup docs describe client configuration and regional options.
Run the same question once with the graph’s candidates in the prompt and once without. Record model tokens, tool calls, and total elapsed time for each, then compare both answers against evidence you trust, such as a grep you’ve verified. Expect the graph’s list to be longer than the true answer, and have the agent confirm each name by reading the code before it commits.
Ask both runs to emit the same final structure: repository, service, relationship type, evidence path, and verification status. Score the confirmed list separately from unresolved candidates. Repeat the comparison enough times to measure variation, and record the checkout revisions and graph revision where available. That makes it easier to separate a useful retrieval result from a grading artifact or a difference in source coverage.
Resources
The post Context Graph Makes GPT 6.1 Terra 42% Faster and Uses 43% Less Tokens appeared first on Postman Blog.
Log in
