If you’re evaluating AI SRE products, you probably own most of one already. The coding agent your engineers use, whether that’s Claude Code, Codex or Cursor, can already investigate a Kubernetes cluster. What it usually lacks is context: what’s failing right now, and what changed before it broke.

We measured how much that gap matters. The same agent and model worked through 50 broken clusters twice, once with a shell and raw kubectl, and once with read-only MCP tools that return that context:

Claude Code, same model, same 50 faults	Shell + kubectl	Read-only cluster context
Correct	45	46
Mean judge score (0 to 1)	0.905	0.936
Correct within 2 minutes	60%	86%
Correct within 5 minutes	78%	90%
Mean time to submit a diagnosis, right or wrong	191 s	64 s
Median	87 s	44 s
90th percentile	632 s	94 s
Mean tool calls per fault	16.3	9.8

With context the agent was slightly more accurate: 46 correct against 45, and a mean judge score of 0.936 against 0.905. Both arms were right on 44 of the same faults.

The bigger difference was time, which is what counts during an active outage: nobody wants to wait fifteen minutes for an AI investigation while users are getting errors. With context, the agent had a correct diagnosis within two minutes on 86% of faults, against 60% with a shell. With a shell, one fault in ten took more than ten minutes. The gap was widest on the 14 faults where every pod reported Ready but the application was broken: 326 seconds on average with a shell, 79 with context. Fewer tool calls also means fewer model turns, which usually means lower token spend.

So before you evaluate an AI SRE, give the agent you already have that context, and measure it. That gives you a baseline for what the product adds.

How we ran it

We ran 50 faults from SREGym, the open Kubernetes fault benchmark from Microsoft and UIUC, on a live 3-node Amazon EKS cluster. Both arms used Claude Code on Claude Sonnet 5 through Amazon Bedrock. One had a shell and raw kubectl. The other had read-only MCP tools from Radar, an open-source Kubernetes UI and MCP server that we maintain, with kubectl blocked so it couldn’t fall back to a shell. SREGym’s judge, Claude Opus 5 at temperature 0, graded the first diagnosis each run submitted.

Context didn’t fix the model’s judgment. Both arms missed the same three faults. In one, a mutating webhook rewrote the pod’s memory limits at admission. In another, a PriorityClass mis-scoped as the global default changed the priority of newly created pods and set off preemption. Neither agent named the object responsible in either case. In the third, both found a real but unrelated misconfiguration and reported that instead of the DNS fault. Same model, same mistakes.

The two arms also differed in permissions, since only the shell could write, so this run doesn’t say which kind of context did the work. Every tool call from both arms is published in a per-fault replay.

What we mean by context

Radar’s read-only MCP tools add four things on top of what kubectl returns:

• Current failures, already detected. A ranked list of what’s broken right now, so the first call starts from what’s failing instead of a full inventory.
• A change timeline. Spec changes and Helm release history, kept after Kubernetes events expire. Events last an hour by default (the API server’s –event-ttl flag), and many changes never produce one.
• Resources joined to their relationships. Owners, topology and related issues come back with the object, so the agent doesn’t fetch them one by one.
• Logs filtered to the errors. Diagnostic lines first, with secrets redacted.

Since we maintain Radar, read the results with that in mind. None of the four depends on Radar specifically: any MCP server that returns the same context can be tested the same way.

What the agent asked for

One fault shows the difference. In the OpenTelemetry Astronomy Shop demo, the frontend Deployment had lost its CART_ADDR environment variable. Every pod reported Ready and nothing restarted. Both arms got the right answer:

Shell + kubectl: 58 tool calls, 701.6 s (excerpt)
  2.1s  kubectl get pods -n astronomy-shop -o wide
  2.2s  kubectl get events -n astronomy-shop --sort-by=.lastTimestamp
 10.3s  kubectl logs -n astronomy-shop product-catalog-77497fdc44-f5nx9 --previous
   ...
423.6s  kubectl run catsrc ...; kubectl cp astronomy-shop/email...
665.8s  helm get values astronomy-shop -n astronomy-shop
674.6s  kubectl get secret sh.helm.release.v1.astronomy-shop.v1 ... | base64 -d | base64 -d | gunzip
701.6s  submitted: "The frontend Deployment in astronomy-shop is missing the
        CART_ADDR environment variable..."

Read-only cluster context: 7 tool calls, 34.8 s
  1.0s  ToolSearch         (loads the MCP tool definitions)
  4.6s  issues             namespace=astronomy-shop
  4.9s  get_changes        namespace=astronomy-shop
 11.8s  get_resource       deployment/frontend
 12.3s  get_workload_logs  frontend
 19.8s  get_workload_logs  frontend
 20.4s  diagnose           deployment/product-catalog
 34.8s  submitted: "The frontend Deployment in astronomy-shop is missing its
        CART_ADDR environment variable, which was removed from the container spec..."

With a shell, the agent spent nearly twelve minutes on it, and ended by decoding Helm’s release Secret to diff the deployed manifest by hand. With the cluster’s recent changes one call away, it had the answer in 35 seconds.

Across all 50 faults the transcripts show the same thing. Without context, the agent opened every fault with kubectl get pods and kubectl get events. On 24 of them it then went looking for history, in ReplicaSets, rollout history or Helm release values, because nothing it could read told it what had changed.

With context, the agent asked for the same two things first, nearly every time. Current failures were its first call on all 50 faults, and the change timeline was its second on 35. Those are the two things the shell agent had to piece together from get pods, get events and rollout history. Much of the shell agent’s time went into rebuilding that context before it could start on the diagnosis.

What you’re actually buying

An AI SRE is several products sold as one:

Part	What it does	What most teams already have
Trigger	Starts an investigation when an alert fires	Alerting, but nothing that starts an agent
Context	What the agent can read: live state, recent changes, current failures	kubectl and RBAC. Change history is the usual gap
Reasoning loop	The model, plus the harness that picks the next call	Claude Code, Codex, Cursor
Delivery	Puts the answer in Slack, the ticket or the incident channel	Partly
Governance	Whose identity it uses, what it may change, what gets recorded	Kubernetes RBAC, and API audit logs if you’ve configured an audit policy

If your team pays for a coding agent, it already has a reasoning loop, and the results above measure the context row on its own. When you evaluate a product’s context, check whether it:

• keeps change history after Kubernetes events expire
• starts from what’s already failing

Context also isn’t tied to one agent. An MCP server can give the same read-only context to your coding agent or to an SRE agent: HolmesGPT, a CNCF Sandbox project, accepts remote MCP servers as toolsets. So you can test the context and the agent separately.

If your coding agent with good context diagnoses as well as the AI SRE you’re trialling, judge the product on the other rows of that table, starting with the trigger. A coding agent investigates when someone asks it to. An AI SRE can start when the alert fires, while you’re asleep, and have an answer waiting by the time you open Slack. That’s a real advantage, and worth weighing on its own.

What it runs as

The shell transcripts also show why the governance row matters. The agent launched its own pods on 13 faults and ran exec, cp, port-forward or debug on 21. On 4 it pulled Secret data into its context, decoding it on 3. Read-only access doesn’t prevent that: the built-in view role excludes Secrets, but many “read-only” roles don’t. On one fault in the Social Network app it launched a pod that ran drop() on two of the application’s MongoDB collections to test a theory, then submitted a wrong diagnosis. SREGym’s prompt allowed all of this: it tells the agent a fix stage follows, says its kubectl can “inspect and modify resources”, and tells it not to ask for confirmation. An agent given write access and told to fix things will use that access while it’s still investigating.

So check what an AI SRE runs as. It should:

• work with a read-only identity, with no exec and no access to Secrets
• record every call it made, so you can see what it read

Test it yourself

Give every contender the same model and the same read-only identity. Kubernetes’ built-in view role leaves out Secrets and grants no exec. This writes a kubeconfig that holds only a service account bound to it, so no contender can fall back to your admin credentials:

kubectl create serviceaccount ai-eval -n default
kubectl create clusterrolebinding ai-eval-view --clusterrole=view --serviceaccount=default:ai-eval
TOKEN=$(kubectl create token ai-eval -n default --duration=8h)
ADMIN_USER=$(kubectl config view --minify -o jsonpath='{.users[0].name}')
kubectl config view --minify --flatten > ai-eval.kubeconfig
export KUBECONFIG=$PWD/ai-eval.kubeconfig
kubectl config set-credentials ai-eval --token="$TOKEN"
kubectl config set-context --current --user=ai-eval
kubectl config delete-user "$ADMIN_USER"
kubectl auth can-i get secrets   # should print "no"

Run the same faults. SREGym has drivers for Claude Code, Codex, Copilot, Cursor, Gemini CLI and OpenCode, and its Lite suite of 17 problems runs on kind with 8 vCPU and 16 GB of memory (setup):

uv run main.py --suite sregym-lite --agent claudecode --model claude-sonnet-5 \
  --judge-model <a model none of your contenders use> --stages diagnosis --n-attempts 3

For an AI SRE product SREGym has no driver for, --use-external-harness --problem <id> injects the fault and leaves the cluster broken for you to point the product at.

Score at a fixed time limit, and repeat. Use a limit that matches your response target; we report two and five minutes. Run each fault more than once, because a gap of two or three faults in a single run can be noise. Record token spend per fault too.

Then replay two or three of your own incidents on staging, with the same identity. Benchmarks inject faults into a freshly deployed app, and your incidents won’t look like that.

What this doesn’t show

• We maintain Radar, and the context arm used Radar’s MCP server. The per-fault data is public so you can check our reading of it.
• Each arm ran each fault once.
• SREGym injects each fault minutes after deploying the app. That favours anything that tracks recent changes, Radar included.
• We measured diagnosis, not remediation.

Start with context

Give the agent you already have read-only access to the cluster’s current failures and recent changes, and measure it at a fixed time limit. Use that as the baseline for anything you trial: compare diagnosis first, then what the product adds around it. Before you replace the agent, measure what happens when you improve what the agent knows.

Radar is an open-source (Apache-2.0) Kubernetes UI with a read-only MCP endpoint. The source and install instructions are at github.com/skyhook-io/radar.

Every enterprise company is currently trying to figure out how to have its AI tools work with its data. Meanwhile, every major enterprise vendor that works with those enterprises (think SAP, Microsoft, Salesforce, Workday, etc.) is making a claim on becoming the central hub for managing and securing that data.

But what about the companies that want to do it themselves?

Nvidia, on Wednesday, announced how it wants to provide enterprise AI agents with the data they need to make sense of company data — all without competing with the data platforms that already store it.

At its GTC Berlin conference Wednesday, the company added Agent Knowledge Foundation to its Agent Toolkit. This new collection of open-source components bundles the company’s Nemotron Retriever models, a new Auto-Ontology library, the Kumo family of models for structured data, and its CUDA-X data processing libraries.

With this system, Nvidia says, the data can stay where it is and remain managed under the access rules these companies already have in place.

Today, as Adel El Hallak, Nvidia’s vice president of product for agentic AI, tells The New Stack, enterprises may be able to connect an agent to a vendor like Snowflake, but that doesn’t help the agent understand the overall enterprise context.

“I wish it was as easy as connected to my Snowflake system, connected to my email exchange, connected to my knowledge base, and then all of a sudden, magically, the AI understands all the data. But the reality is different,” he says.

Large language models are “great at predicting next tokens,” El Hallak argues, but when presented with a single spreadsheet that has “a whole bunch of tabs,” the agent will have a hard time finding something specific. “It’s gonna struggle,” he says.

Better retrieval, fewer lookups

“In the world of agents, you want the most accurate embedding models or reranking models,” El Hallak says, even if that adds a bit of latency to each lookup. The agent then gets “much more pertinent information. So the number of lookups at the end, the overall latency actually goes down.”

Nvidia says its Nemotron Retriever embedding model raised search relevance by 65% in tests on an open source OpenSearch setup, needing fewer search steps and adding hardly any latency. SAP’s fine-tuned version of the reranker ran nearly three times faster, Nvidia says, and ServiceNow got better reranking quality with close to 25% less latency.

From RAG to ontologies

“RAG was awesome, but is it good enough?” El Hallak asks. Knowledge graphs, he says, represent concepts and the relationships between them better than retrieval-augmented generation, which pulls in chunks of text based on similarity.

Nvidia’s Auto-Ontology library builds those graphs from a company’s own tables and maps raw data to business concepts like customers, products, and orders. Its listing in Nvidia’s NGC software catalog shows a REST API and a Model Context Protocol (MCP) server that agents can use to query it.

“You and I, it’s easy for us to understand. A tier is the same thing as a plan when you’re looking at a mobile carrier,” El Hallak says, but LLMs “mess those up.”

Nvidia bought Illumex, an Israeli startup that automated the creation of semantic layers and ontologies for enterprise data, for a reported $75 million in February. At the time, an Nvidia spokesperson told Globes that the team’s ontology expertise would help it build “libraries and models that will enable the adoption of Agentic AI in organizations.”

Predictions from tables

Kumo, the startup Nvidia bought in June in a deal The Information valued at more than $400 million, built foundation models for relational data. Nvidia has since released several Kumo models as open weights on Hugging Face, including Kumo Tabular in September.

El Hallak says those models replace the per-task pipelines data teams used to build “for every classification, for every churn, for every sentiment analysis.”

“The whole point of the ontology … is to bring harmonization as you create those knowledge graphs, so there’s [consistency] across the tables, and then you could use Kumo to drive predictive analytics,” he says.

Partners, not rivals

“I’m not looking to compete with, for example, a Snowflake. On the contrary, I want to partner with them,” El Hallak says.

Nvidia’s own supply chain team uses Agent Knowledge Foundation through Palantir’s Foundry platform, building on earlier work between the two companies. Databricks is pairing a Kumo anomaly detection model with Genie Ontology, the context layer it introduced in June, and Snowflake customers can run Kumo models in Snowflake Notebooks.

“I’m not thinking about ACLs and data access and all those things. That’s why I need the partners,” El Hallak says, referring to access control lists and other security features that its enterprise partners offer.

Owning the stack

Not every vendor is taking that route. SAP closed its acquisition of Prior Labs, the company behind the TabPFN tabular models, in July and committed more than €1 billion to it over four years. It now offers TabPFN-3.5 Plus in SAP AI Core alongside its own RPT models.

Nvidia, meanwhile, distributes its models through Hugging Face, which it agreed to buy for $12.9 billion in September. “We just build little weights and models, and we publish them on Hugging Face,” El Hallak says.

That leaves Nvidia building the models, while the likes of Snowflake and Databricks keep control over the data and the permissions around it.

The post Nvidia bets open models can give enterprise AI agents the context they’re missing appeared first on The New Stack.

W3C's VP Technical Strategy, Philippe Le Hegaret wrote "W3C and the State of the Web in the AI Era", a document putting together the observations and findings of the W3C Strategy Team about how the W3C technical agenda for the web needs to evolve based on the rise of AI agents. It is part of building a consistent, coherent shared vision for evolving the web, starting with AI. 

W3C needs to position itself as the venue where AI and web stakeholders discuss the risks and opportunities of this evolution of web user agents to develop responsible technology standards that align with W3C’s vision for the web. W3C’s core vision is to develop the web for all humanity: we have to keep humanity at the center of the web while allowing AI agents to act on their behalf.

This paper is intended to foster discussion in the technical community at TPAC — the most important W3C community event of the year. 

In addition to outlining our understanding of how current and emerging AI technologies intersect with the W3C's roadmap this document can help TPAC 2026 participants prepare for the many discussions on the agentic web that will take place in group meetings, a mini-Workshop, and breakouts. This paper does include some early recommendations that we expect to socialize during TPAC and we seek community feedback.

This report will be presented during the Web and AI Interest Group session on 29 October and feedback is also welcome to public-webai@w3.org and the w3c/webai-roadmap repository.