An AI agent pursues a goal on its author's behalf over minutes, hours, or days. It calls tools, writes files, keeps state between steps, and makes its own decisions within the guardrails it was given. At different moments it leverages a request/response service, a batch job, a serving endpoint, and other capabilities.

Running this type of workload safely requires a solution architecture of its own. In this article, we describe that solution as a blueprint to use for your agentic workload delivery needs.

A new workload class

The model is a stateless inference service. The harness turns it into an agent: it owns the loop, the memory, the context window, and the tool calls. The sandbox runtime decides what the agent may touch: which files, which binaries, which network endpoints. These three components have separate lifecycles, scale differently, and carry different security requirements, so we keep them separate.

Red Hat AI, the enterprise platform for any model and any agent across the hybrid cloud, is a natural host for that separation. We write this blueprint mainly from an Agent-as-a-Workload perspective. This means each agent is one sandboxed pod and one individual, independent value proposition. A second deployment pattern, Agent-as-a-Service, moves the loop out of the agent software into a shared runtime. Its adoption will follow business and operational needs, and we contrast the two later in this article.

We describe the solution blueprint as an open architecture of interchangeable components. Every component in Figure 1 is a role that several implementations can fill. AI accelerator vendors ship implementations for many of them; NVIDIA currently has the most visible set with NemoClaw, OpenShell, Nemotron, and CUDA-X. They appear in our blueprint on those merits, though they are not required. The architecture stands on open interfaces: an OpenAI-compatible inference access API and the open source Open Responses specification for models, Model Context Protocol (MCP) for tools, the agent-sandbox API for isolation, and SPIFFE (Secure Production Identity Framework For Everyone) for workload identity. This architecture gives each agent a verifiable, short-lived credential instead of a static API key.

We wrote this article for platform engineers and AI infrastructure teams who need to run autonomous agents safely on open, scalable AI platforms such as Red Hat AI. This article is not a tutorial for building an agent. Instead, it serves as a blueprint for operating agents as governed cloud-native workloads.

The solution architecture

In Figure 1, we lay out the complete blueprint in a single view. The rest of the article walks you through the blueprint component by component. The figure reads as three horizontal bands with one checkpoint between them, and the legend colors mark the planes.

The control band across the top (blue) creates and governs agents. Read it from left to right to understand the lifecycle:

• An agent blueprint declares what each agent is made of.
• The sandbox control plane turns that blueprint into a Sandbox resource.
• A controller turns the resource into exactly one pod.
• The workload identity service issues every pod and gateway a verifiable credential.

The dashed arrows dropping out of this band are control traffic, and nothing else about the figure moves until they've done their work.

The middle band is where the magic happens. It contains three zones from left to right:

• Skill backends: These backends can run anywhere (represented by a dashed border because location is not a commitment).
• Agent pod: This pod hosts the agent, harness, supervisor, and thin skill clients.
• Inference plane: This plane serves models inside the cluster and contains the gateway, semantic routing, scheduler, and vLLM workers.

Two gray boxes on the right edge sit apart from that flow on purpose: an optional external large language model (LLM) that is reachable only with masked personally identifiable information (PII), and audit and tracing, where every hop in the architecture eventually lands.

The checkpoint between the bands is the MCP Gateway. It's drawn in control-plane blue because of its function: every tool call leaves the pod through this gateway, is authorized by token claims rather than by the model itself, and then reaches a skill backend.

The platform bars at the bottom carry everything above them: Red Hat OpenShift, the Linux kernel, and the silicon. The silicon bar lists several options deliberately because the architecture assumes no single vendor.

Components

Each solution component in and around Figure 1 has a role, a duty, and a set of implementation options. We list these options in Table 1. Maturity varies across these options, and we separate production-grade implementations from technical preview features later in this article (see Table 4).

Table 1: Solution components, roles, and implementation options.

Solution components	Role/duty	Options
Agent blueprint / GitOps	Declare, deploy, version agents	Helm & GitOps · NemoClaw · OpenShell (planned)
Agent orchestration	Multi-agent coordination, discovery	A2A & signed AgentCards · AGNTCY Agent-Exchange
Sandbox runtime (supervisor)	Constrain agent process: files, network, syscalls	OpenShell (planned) · Kata & K8s agent-sandbox
Harness	Loop, memory, context, tool use	OpenClaw · Hermes Agent · LangChain Deep Agents · LangGraph · custom
Agentic loop runtime (Agent-as-a-Service)	Server-side loop: tool schema advertisement, execution, iteration, trace store	OGX (Responses API) · custom
Model	Reasoning service	Nemotron · Llama · gpt-oss · DeepSeek · Qwen · Kimi · GLM · Mistral
AI gateway (MaaS)	Auth, quota, OpenResponses/OpenAI API	OGX · LiteLLM · Red Hat AI gateway
Semantic routing	Pick best price/performant model	vLLM Semantic Router (upstream) · Red Hat AI gateway (roadmap)
Inference routing	Pick replica: KV cache, load	llm-d Router / EPP (Gateway API ext.), Praxis (Early)
Model workers	Token generation on accelerators	NIM / TensorRT-LLM · vLLM (CUDA/ROCm/XPU/CPU; TPU/Gaudi via plug-ins)
Skills	Domain capabilities as tools	CUDA-X (cuDF, cuOpt, and so on) · Polars · DuckDB · OR-Tools · JAX · Qiskit
Workload identity	Cryptographic agent identity	SPIFFE/SPIRE (JWT-SVID)
Tool governance	Authorize tool calls by token claims	MCP Gateway

Agent packaging

The sandbox layer on Kubernetes consists of two pieces, with one on each side of the pod. Outside the pod sits a sandbox control plane, and inside the pod runs a supervisor process. In our blueprint OpenShell supplies both, and any implementation can fill either role because the contract between them is a Kubernetes custom resource.

The delivery path has two hops, with a custom resource in the middle. The control plane receives a request for an agent and writes a Sandbox custom resource; in our blueprint the OpenShell Gateway is the target implementation of this role as its Kubernetes path matures. The Agent Sandbox controller, a Kubernetes Special Interest Group (SIG) Apps subproject that implementations target, reconciles that resource into exactly one pod. One sandbox maps to one pod in the current API.

Inside the pod, the supervisor runs as PID 1 and starts before the agent does. It fetches policy from the control plane, installs the kernel filters, injects credentials, and then launches the harness as its child process. All egress leaves through the supervisor's proxy.

Policy enforcement splits along this same boundary. Kubernetes governs the pod from the outside using the following primitives:

• Pod security standards
• SELinux
• NetworkPolicy
• An optional Kata microVM through RuntimeClass
• Confidential containers beneath those layers

The supervisor governs the process from inside, regulating which binaries the process can run, which paths it can touch, and which endpoints it can reach. Stacked together, these layers provide three nested isolation rings: hardware or virtual machine (VM), pod, and process. Every primitive in all three rings relies on standard Linux and Kubernetes technologies, such as seccomp, Landlock, namespaces, SELinux, AppArmor, or Kata Containers. None of these primitives depend on a GPU vendor.

One discipline from our 12-Factor Agent article carries over directly: the agent workload is long-lived, but its sessions are not. Sessions remain ephemeral, while workflow state lives outside the pod in a durable data store, such as a database, an object store, or a durable-execution engine. Consequently, each new session starts with a clean context.

Agent-as-a-Service

Everything above assumes the loop ships inside the pod with the harness. Red Hat OpenShift AI 3.5 (Early Access at the time of writing) ships a second pattern through the open source OGX project (formerly Llama Stack): Agent-as-a-Service. Here the agentic loop runs in a shared runtime behind the Responses API. The client sends one request naming the goal and the MCP servers it requires. The loop runtime advertises the tool schemas to the model, executes the calls the model selects, feeds results back, and iterates to a final answer. Every step lands in a durable data store. Figure 2 contrasts the two patterns side by side.

The shared runtime inverts the pod pattern's trade-offs. It gains:

• Enforcement in one place: every action transits the loop runtime.
• One audit trail across all tenants.
• Credential isolation: clients never hold tool credentials.

It pays with:

• Shared blast radius: the loop service is a single point of failure.
• API mismatch: a request-scoped API fits long-running, goal-holding agents poorly.
• Blurred identity: per-agent identity survives only if the runtime propagates caller identity downstream, a gap SPIFFE does not yet address; the active standards work is in the IETF Workload Identity in Multi-System Environments (WIMSE) working group.

Most harnesses today are thick clients that own their loop, so adoption starts with operations teams standing up a shared runtime for request-scoped workloads: assistants, copilots, high-tenancy tools. Autonomous agents follow later.

The two patterns also compose. A sandboxed agent pod can delegate tool execution to a shared loop while keeping its own identity and rings, and the MCP Gateway's claims-based checks preserve per-agent policy inside the shared runtime. Duration and ownership pick the pattern, the same way you choose between a standard Deployment and a Knative service.

The governance

Governance in this blueprint consists of two mechanisms working as one system: workload identity establishes who the agent is, and tool authorization decides what that identity can do. Neither mechanism consults the model, and that's the point. While the agent's behavior can be manipulated, its credentials and permissions cannot.

Workload identity

SPIFFE/SPIRE gives each agent an automatically rotated cryptographic identity that is scoped to its namespace and service account. An authentication sidecar uses that identity for standards-based exchange with an identity provider such as Keycloak, using token exchange (RFC 8693) or the JWT bearer grant (RFC 7523), depending on the provider's support for externally issued tokens. 

This process trades the identity for short-lived access tokens to downstream services, to help verify no static API key exists anywhere in the path. The payoff is clear: a leaked prompt or workspace file contains nothing durable to steal. Lifecycle tooling planned as part of Red Hat AI injects the sidecar and the identity plumbing at deploy time, with no agent code changes.

Tool governance

The MCP Gateway, built on Envoy with Kuadrant/Authorino for policy, fronts every MCP tool server behind a single endpoint. The agent is configured with one MCP_URL variable and sees an aggregated tool catalog. Token claims minted during the workload identity exchange then decide which tools the agent can actually call. The gateway never reads the prompt. A prompt-injection attack that forces the model to call an unauthorized tool fails at the infrastructure layer because authorization is not the model's decision to make. 

What the gateway cannot stop is misuse of tools the agent is entitled to call, such as an injected instruction that exfiltrates data through a permitted endpoint. That risk belongs to the inference guardrails and egress policy described in the security model, which is why this checkpoint is one control among several rather than the whole defense. The same claims-based check covers every tool backend equally: a CUDA service, an OR-Tools container, or a Software-as-a-Service (SaaS) endpoint. MCP itself now lives under neutral governance, donated to the Agentic AI Foundation under the Linux Foundation in late 2025, with an authorization model that makes each tool server an OAuth 2.1 resource server, which is precisely the claims-checking this gateway enforces. This is the checkpoint from Figure 1 doing its job.

Inference routing

An inference request crosses three routers on its way to an AI accelerator. Three tiers exist because the three decisions have different owners: where a query may go is a security decision, which model serves it is a cost decision, and which replica executes it is an efficiency decision. Collapsing them into one router would put one component in charge of three concerns that change at different speeds and answer to different teams. Table 2 summarizes the three tiers, and Figure 3 follows one request across them.

Table 2: The three inference routing tiers.

Tier	Component	Question it answers	Decision inputs
1	Sandbox egress router (in supervisor)	Where may this query go?	Policy, privacy class, PII masking, local vs. cloud
2	Semantic routing (vLLM Semantic Router or AI gateway)	Which model should serve it?	Intent and complexity classification, cost, reasoning on/off
3	llm-d Router (EPP or Gateway API InferenceeExtension)	Which replica serves it?	KV cache locality, load, prefill/decode disaggregation

Tier 1 is the security tier and lives client-side in every sandbox, inside the supervisor. Confidential context never leaves the cluster, and anything routed to an external endpoint is disguised to protect personally identifiable information (PII) first. For in-cluster traffic, the same router directs requests to the AI gateway, so even local inference enters through the governed front door rather than reaching model endpoints directly. The masking is detection-and-redaction at the proxy, so residual false-negative leakage is the control's main failure mode and should be monitored.

Tier 2 is the cost tier: a lightweight classifier selects the cheapest model that can handle the request. The classifier itself adds latency and can misroute, so this tier is a measured trade-off rather than free capacity. The system needs a fallback that escalates to a more capable model on low-confidence or low-quality signals.

Tier 3 is the efficiency tier: once the model pool is fixed, llm-d's endpoint picker maximizes KV-cache hits and balances load across vLLM workers. Red Hat's integration work describes the split the same way: the semantic router picks the model, llm-d schedules replicas within the pool.

Two properties make this path operable, and they're the same two demands you would place on any request path in production. It is traceable end to end when instrumented: OpenTelemetry follows a request across all three tiers, and MLflow Tracing records tool calls, prompts and outputs, latencies and costs. MLflow Tracing itself is production-ready; the OpenTelemetry GenAI semantic conventions it emits are still in development, so plan for attribute changes rather than interface rewrites (see Table 4). And it's decoupled at every hop: every hop speaks OpenAI-compatible or OpenResponses APIs, so the model behind the path can change without the agent noticing.

Job placement

A common design mistake we see is assuming the skills execute inside the agent pod. This habit carries over from library-based development, where importing a capability means hosting its compute resources. In this blueprint, the skill components in Figure 1 represent definitions and thin clients: instruction files and API bindings rather than AI workloads. The distinction keeps the agent pod cheap and the blast radius small. Heavy compute tasks follow one of three patterns, and only pattern 1 runs in the pod: execute in place, call a service, or dispatch a job (Table 3).

Table 3: Skill execution patterns and their enforcement points.

Pattern	Where compute runs	Enforcement point	Typical use
1: In-pod	Inside the sandbox (the agent-sandbox SIG's Sandbox custom resource must request a GPU explicitly)	Kernel: seccomp, Landlock	Light DataFrame ETL (cuDF, Polars, DuckDB)
2: Service call	Separate serving pods, any cluster, or SaaS	MCP Gateway & egress policy	Optimization REST (cuOpt, OR-Tools), RAG services
3: Job dispatch	Batch jobs dispatched via the Kubernetes Job API or Slurm	Scoped role-based access control (RBAC) & admission	Physics simulation (PhysicsNeMo, JAX), quantum programs

Pattern 2 serves as the default model at scale. For this reason, Figure 1 illustrates skill backends as a location-agnostic column outside the platform bars. The optimization service can be cuOpt on a GPU node or OR-Tools on CPU; the agent's client and the gateway policy do not change. Because authorization is endpoint-specific and claims-based, relocating or swapping a backend is a policy change rather than an architecture change. Pattern 3 needs the most attention in security review: an agent that can create Jobs can create arbitrary workloads, so its RBAC must be scoped tightly.

Getting this split right also sets the economics. Agent pods stay resource-efficient, dense, long-lived, and CPU-only in the common case, with the in-pod GPU of pattern 1 as the deliberate exception, while expensive accelerators sit in pooled backends that scale independently. The fleet then scales on two independent axes: agent count on cheap cores, accelerator capacity on utilization, and neither purchase forces the other.

To see the blueprint in action, follow one request end to end; every component named so far makes exactly one appearance. 

A field-service agent receives a dispatch-optimization goal and initializes inside its sandboxed pod. The process then unfolds across the layers:

• The harness calls the optimization skill; the MCP Gateway checks the agent's token claims and forwards the call to the cuOpt or OR-Tools backend.
• The planning prompt carries customer data, so tier 1 keeps it on in-cluster models; a follow-up summarization request is low-risk, so tier 2 routes it to a small model and tier 3 picks the replica with the warmest cache.
• Every tool call, authorization decision, latency, and cost lands in the OpenTelemetry and MLflow trail. 

At no point does the architecture depend on the model performing predictably; the path maintains the same shape even against a hostile model.

Security model

The distinctive threat with a long-running agent is that the workload itself can be turned against you. A prompt-injection attack can redirect its goals, a poisoned tool result can corrupt its memory, and a compromised skill dependency can act with everything the agent is allowed to reach. The design response is to assume the model's behavior cannot be trusted and to route every consequential action through enforcement the model cannot influence.

Figure 4 shows how the architecture layers that enforcement: three nested isolation rings around the agent process, and four cross-cutting controls that apply across all of them. The rings nest deliberately: each one assumes the ring inside it has already failed.

• Ring 1 (process): The sandbox supervisor enforces per-binary, per-path, per-endpoint policy via kernel primitives (seccomp, Landlock, network namespaces). The agent cannot override this policy even if compromised because the policy lives outside the process it constrains.
• Ring 2 (pod): Kubernetes pod security standards, SELinux, AppArmor, NetworkPolicy, and resource quotas bound what the pod can reach and consume. Optionally, Kata Containers add a microVM boundary per sandbox.
• Ring 3 (hardware): Confidential containers extend the trust boundary into silicon, protecting an agent's memory at runtime even if a neighboring workload or the node itself is breached, with attestation proving the sandbox runs the code it claims before any secret is released. This ring is the precondition for multi-tenant agent designs. Confidential computing on GPUs is now generally available on current accelerators but still carries runtime overhead, concentrated in host-to-device transfers, so treat it as the ring you plan for rather than the one you start with.

Four controls cut across all three rings, and they carry the same names in Figure 4:

• Workload identity: SPIFFE SVIDs replace static keys, so a leaked workspace contains nothing durable to steal.
• Tool authorization: The MCP Gateway authorizes every tool call from token claims, never from prompt content.
• Inference guardrails: Input and output screening at the inference boundary (TrustyAI Guardrails Orchestrator, NeMo Guardrails).
• Pre-production testing: Adversarial scanning in CI/CD pipelines catches jailbreak and injection weaknesses before an agent is promoted.

The rings and the controls answer different questions. The rings limit what a compromised agent can do; the controls reduce the chance it gets compromised and remove the rewards if it does. Containment economics follow from the single-sandbox, single-pod, and single-identity mapping: revoking, throttling, or terminating a misbehaving agent is a single targeted operation instead of an incident-wide hunt. The MLflow/OpenTelemetry trail turns every enforcement decision into reviewable evidence, and the posture holds unchanged at the edge as in the data center.

Vendor gravity and escape hatches

Vendor gravity in this stack lives in the libraries and the silicon, one layer below where most lock-in audits look. The license layer is clean: several core building blocks are open source under permissive terms (OpenShell and NemoClaw are Apache-2.0), so no single vendor holds exclusive rights to the architecture, while commercial packaging, NIM profiles, CUDA libraries, and certified support remain vendor-specific. What binds you, when something does, is an unported library or a proprietary interconnect. Here is our layer-by-layer audit:

• Interface-portable: The model layer is any open model behind the same API. The agent blueprint is YAML, portable by construction. The sandbox runtime enforces policies with stock Linux primitives, and Kata Containers plus the agent-sandbox SIG covers the same three rings on any conformant cluster. The whole inference plane is portable: semantic routing and llm-d are vendor-neutral, and vLLM runs on CUDA, ROCm, XPU, and CPU directly, with TPU, Gaudi, and other accelerators supported via hardware plug-ins. The serving architecture therefore survives a full silicon swap untouched at the interface level. Achieving operational parity in performance, tuning, and accelerator coverage is engineering work you should plan for.
• Sticky: The CUDA-X libraries are the real gravity: cuDF, cuOpt, and PhysicsNeMo are CUDA underneath, and NIM containers run only on NVIDIA silicon. Substitutes exist with trade-offs: Polars, DuckDB or Spark for DataFrames, OR-Tools or HiGHS for optimization, and JAX-based surrogates for physics. What the architecture controls is the price of the swap: skill backends sit behind neutral REST/MCP interfaces, so replacing cuOpt with OR-Tools is an endpoint and policy change rather than a redesign.

The rule we enforce throughout the blueprint: every vendor component sits behind a neutral interface. That means OpenAI-compatible and OpenResponses for models, MCP for tools, REST for skill backends, the agent-sandbox API for isolation, and SPIFFE for identity. Under that rule, vendors compete on price/performance per component instead of owning your architecture, and procurement decisions stay reversible for the lifetime of the deployment. NVIDIA then becomes a per-component choice rather than a structural dependency.

Maturity and open items

Not everything in Figure 1 is equally ready. In Table 4, we separate the layers you can run in production today from the layers to treat as preview, where interfaces will still change. The reading rule: build your base decisions on the mature column, adopt from the preview column behind pinned versions and feature gates, and revisit at each release.

Table 4: Maturity map across the blueprint layers.

Layer	Mature today	Emerging / preview
Workload identity	SPIFFE/SPIRE · JWT-SVID	Agent-specific identity injection (Kagenti) · identity chaining (IETF WIMSE)
Tool governance	Gateway patterns (Envoy, Kuadrant)	MCP Gateway · claims-filtered tool catalogs
Agent sandbox	Kata · kernel primitives	agent-sandbox CRD (SIG Apps, v1beta1) · OpenShell (alpha)
Inference	vLLM · OpenAI-compatible APIs · llm-d Router (Gateway API Inference Extension GA)	Semantic routing
Agentic loop runtime	Client-side loops in harnesses	OGX (OpenShift AI 3.5 EA) · caller-identity propagation
Agent lifecycle	GitOps · Helm · operators	NemoClaw (pre-1.0) · OGX Operator (OpenShift AI 3.5 EA)

And a blueprint earns trust by stating what it does not solve yet. Note that each of the following open items belongs to a different discipline, which is itself the message: closing them takes products, standards bodies, researchers, and operators, and no single vendor closes the list.

• Human approval workflows for high-risk actions still need product-level patterns.
• Multi-agent coordination standards are converging (A2A reached 1.0) but interoperability in practice is not settled.
• Agent memory poisoning remains an open research and engineering problem.
• Sandbox policy authoring is operationally complex at fleet scale.
• Cost modeling for long-running agents needs real workload data.
• Confidential computing protects memory at runtime; it does not stop prompt-layer or tool-layer compromise.

Closing notes

In this article we walked the agentic stack inside out. We defined the agent as model, harness, and sandbox runtime, and laid the solution out in Figure 1 as interchangeable components behind open interfaces. 

Then we followed it through delivery: an agent packaged as one sandboxed pod, or delegating its loop to a shared runtime where the workload profile fits, governed by workload identity and claims-checked tool calls, served through three routing tiers to vLLM workers, with skill compute placed in governed backends and a defense-in-depth security model wrapping it all. 

The result is a blueprint you can begin assembling today from open source parts, treating several layers as preview, with vendor choices that stay reversible at the interfaces you kept neutral. Managed sandbox services offer parts of this blueprint as SaaS; what the open architecture adds is that the isolation, identity, and governance layers stay on infrastructure you control, with interfaces you can take to any conformant cluster. We verified the component landscape in this article as of July 2026; in a stack where several layers are pre-1.0, check the linked projects for current status before you build.

The agentic stack is beginning to look like the microservices stack one abstraction up: blueprints instead of charts, sandboxes instead of containers, inference routing instead of service mesh, skills instead of client libraries, and share d loop runtimes instead of application servers. The same disciplines apply: least privilege, declarative policy, separate control and data planes, pooled expensive resources behind schedulers, and keeping the interfaces under your own control. The difference is that this workload makes its own decisions for days at a time. The stack above is what it takes to let it do that safely, observably, portably, and at a cost per task you can measure.

Where to go next

Architecture arguments are easier to judge with running code, so we are rebuilding our Telecom AI Experiments workloads on this blueprint at agent-school; the 101 NOC Assistant runs offline on a laptop in two commands. To run the blueprint on your own infrastructure, we recommend exploring Red Hat AI, Red Hat OpenShift AI, and Red Hat AI Inference Server.

The post Architect an open blueprint for cloud-native AI agents appeared first on Red Hat Developer.

"Hey, can you take a photo of me using Photo Booth with a filter… also, open Chess and move some of the pawns up."

That little exchange shows computer use in action, one of the coolest things happening in agentic AI right now.

What is computer use?

Computer use lets an AI use software the same way you do, so it can view the screen, click, and type—all without you touching the keyboard or mouse. This approach is invaluable for automating tedious, manual tasks. For example, one of my previous jobs required me to move data from a CRM into Jira tickets. Computer use can also help with enterprise-level tasks where custom integrations are not available.

Pretty neat, right? But you almost certainly have private data on your own machine that you'd rather not hand to a third-party AI provider. If you pay per token via an LLM API, these tool calls can get quite expensive. You might want to look at running your own LLM and wiring it up to an agent running locally.

With that in mind, I'll show you how to run an open-weight model (such as Qwen or Gemma) with an open source AI harness (Hermes) so that you can implement computer use on your own terms.

Step 1: Pick a model (and don't overpay for hardware)

When you choose a model, it's worth checking out Red Hat AI on Hugging Face, where an entire library of compressed models can dramatically cut your hardware requirements (in some cases, halving the RAM requirements or more). The core technique is quantization, which squeezes a model down to fewer bits per weight (Figure 2).

Take the Qwen 3.6 35B model shown in Figure 3. The native released weights in full precision are roughly 72 GB. Red Hat's 4-bit version? About 25 GB. This reduced footprint means that you can run this model on your own MacBook Pro with stable performance. Thanks to modern compression algorithms, the quantized version performs almost identically to the original on most benchmarks. The compressed version retains 99% of the original model accuracy while using a fraction of the hardware footprint.

Step 2: Deploy it (either locally or on Kubernetes)

For this example, we use Qwen 3.6 with 35 billion parameters. It's a sparse Mixture-of-Experts model, meaning it features 35 billion total parameters with roughly 3 billion active parameters per token, and it's natively multimodal. This architecture makes it a good fit for computer use because the agent passes a screen capture to the model and then uses tools to interact with user interface elements like check boxes.

Now, let's get it running.

Deploying the model locally

If you have the hardware to run it (I like to use the llmfit project to check compatibility before downloading anything) you might be surprised by how quickly you can serve a model locally. As a macOS user, I've tried most of the local inference tools, from Ollama to LM Studio and upstream llama.cpp. These days, I'm reaching for oMLX, a simple open source server and menu-bar application that manages models and exposes an OpenAI-compatible API endpoint (Figure 4). This tool usesApple's MLX framework to accelerate inference through unified memory.

If you are developing on Windows or Linux, the same model runs through llama.cpp. Download a model (Unsloth, shown in Figure 5, makes some good quantized ones), point llama-server at it, and configure the same OpenAI-compatible endpoint to connect your agent.

Deploying the model with vLLM on OpenShift AI

On Red Hat OpenShift AI, you can deploy models directly from the model catalog onto a Kubernetes environment. You can then create a route to make the model reachable from outside the cluster, authenticated with a custom API key.

Inside the cluster, the platform optimizes throughput and reduces latency by intelligently routing traffic and disaggregating the inference steps. This process splits the compute-heavy prefill phase from the memory-bound decode phase by using the llm-d project. In Red Hat's published benchmarks, intelligent inference scheduling delivered up to 57 times faster time-to-first-token and roughly two times the throughput compared to naive round-robin routing on the same hardware.

Note

To learn more about the differences between llama.cpp and vLLM, read our guide on choosing the right local LLM inference engine.

Step 3: Set up the local agent

On my local machine, I've installed Hermes, a popular open source AI agent that runs on macOS, Windows, and Linux (Figure 7). Hermes can write code, manage your calendar, diagnose your Kubernetes cluster, and more. Because it connects to any OpenAI-compatible endpoint, I can point it directly at the model I just served—whether that's oMLX on my macOS environment or vLLM on OpenShift—with no code changes. The configuration requires only a base URL and an API key.

To interact with your desktop, Hermes performs computer use through a computer use agent (CUA) driver, which is an open source tool that lets the agent control the screen. The driver also includes configurable, multi-layer guardrails that protect windows from closing unexpectedly and require user approval before the agent types text or takes sensitive actions. You stay in the loop.

Step 4: Profit? Well, yes

So, let's give it a try: "Hey, can you use my browser to go to the vLLM repo and give it a star?"

Just kidding, you don't have to. But that's what computer use is all about. The model reads the screen, finds the browser, locates the star button, and selects it automatically. This architectural configuration differs because it runs entirely within a sovereign environment, keeping the local model, the local agent, and your private data on your machine.

To explore more capabilities, view the available open source tools and options from Red Hat AI.

The post Computer use: How AI agents can automate almost anything appeared first on Red Hat Developer.

For most PyTorch users, "distributed" still means c10d. That's the runtime surface they use today, and it will remain the surface for a while. The real question is what should replace the old backend layer underneath it as training and inference systems demand more than host-driven process groups and standard collectives. TorchComms is the clearest answer PyTorch has presented so far.

Why replace c10d

The c10d library gave PyTorch a ProcessGroup-centered distributed runtime. That interface is still what most users touch, and it's not going away overnight. But the old back-end layer underneath it was designed around host-driven collectives and a smaller communication design space than modern LLM training and inference now need. TorchComms is the replacement path for that back-end layer.

That's the right scope for the claim. TorchComms doesn't need you to rewrite your code and throw away torch.distributed. Instead, it allows you to replace the old c10d backend substrate with a communicator-centric layer that can still plug back into torch.distributed.

The official launch material makes that roadmap explicit: The old c10d::Backend interface is expected to be retired gradually as TorchComms stabilizes underneath PyTorch Distributed.

That distinction matters. The replacement target is the back-end layer, not the entire distributed surface area. ProcessGroup, DeviceMesh, FSDP, TorchTitan, and higher-level training code can keep their existing structure while the communication implementation underneath them becomes more explicit, more extensible, and easier to evolve.

Communicator-centric API

The core API shift is simple: TorchComms starts from explicit per-device communicators.

import torch
import torchcomms

device = torch.device("cuda")
backend = "ncclx"
comm = torchcomms.new_comm(backend, device, name="global")

x = torch.ones(1024, device=device)
comm.all_reduce(x, torchcomms.ReduceOp.SUM, async_op=False)

That looks small, but it is a real model change. Collectives, point-to-point operations, hooks, finalize, and window operations all hang off the communicator object. TorchComms also exposes pluggable backends, eager communicator setup, DeviceMesh integration, and communication surfaces that go beyond classic ProcessGroup collectives.

The practical payoff is lifecycle control. With a communicator object, setup, split, finalize, and backend-specific extensions live in one place. That makes it easier to add semantics such as windows, RMA, transport-specific hints, or future fault-tolerance operations without overloading a single generic ProcessGroup interface.

That back-end surface is also becoming easier to prototype against. A recent PyTorch distributed devlog shows Python bindings for TorchComms backends and two pure-Python prototypes, one on top of nccl4py and one on top of SymmetricMemory plus Triton, both still plugging into torch.distributed.

Those prototypes are not production-ready, but they reinforce the same point: TorchComms is not just a replacement for an old back-end interface, it's becoming the place where new communication ideas can be tried without first rewriting the C++ comms stack.

Migration path into PyTorch and Torchtitan

TorchComms is not only a standalone API. PyTorch already contains a wrapper path in torch/distributed/distributed_c10d.py that instantiates a TorchComms communicator, wraps it in _BackendWrapper, and attaches it to the existing ProcessGroup machinery when dist.config.use_torchcomms is enabled. That keeps the migration small:

import torch.distributed as dist
import torch.distributed.config as dist_config

dist_config.use_torchcomms = True
dist.init_process_group(backend="nccl")
dist.all_reduce(tensor)

The important detail in this local checkout is that the safe wrapper example is backend="nccl". The equivalent backend="ncclx" wrapper path still fails locally with AssertionError: Unknown backend type ncclx, so it must not be presented as already working everywhere.

Even with that caveat, the direction is clear. TorchComms is not being built as an isolated alternate runtime. It is being wired into the same torch.distributed and DeviceMesh ecosystem that large PyTorch training stacks already use today.

The same pattern shows up in Torchtitan. Torchtitan already exposes --comm.mode torchcomms, and its distributed initialization path sets dist.config.use_torchcomms = True before normal process-group and DeviceMesh setup.

Architecturally, that is the point: The model and parallelism code stay in the usual TorchTitan and DeviceMesh world. The communication layer underneath is what changes. In this bundle, the command is illustrative rather than a locally green end-to-end run, because the current environment still hits a tyro/typeguard mismatch before TorchComms initialization.

Figure 1 shows the clearest startup result. Default process-group initialization drops from 14.5 seconds to 3.97 seconds at 8,000 GPUs, from 55.71 seconds to 11.89 seconds at 32,000 GPUs, and from 265 seconds to 24 seconds at 96,000 GPUs.

That's the strongest argument for replacing the old back-end layer with something built for large communicator counts, faster control-plane work, and topology-aware setup.

Those numbers matter because startup is not a cosmetic metric at this scale. Large jobs create many communicators, restart after faults, and repeatedly pay control-plane costs that are negligible on a small cluster but dominant at tens of thousands of GPUs. Faster initialization directly changes how practical a communication stack is for real training systems.

Figure 2 shows the inference side. Decode-time improvement rises with communication pressure: for k=1, batch=128, it grows from 19% at 4 hosts to 56.5% at 16 hosts. For k=4, batch=256, it grows from 45.87% at 4 hosts to 83.3% at 16 hosts. The baseline here is the token-shuffling path built from two AllGather operations and one AllToAll with the same compute kernels, so this is a back-end layer result rather than a generic c10d against TorchComms chart.

This is also why the TorchComms story is not only about training collectives. The back-end layer must handle both bulk-synchronous training and more dynamic, metadata-sensitive inference patterns. GPU-resident metadata, transport flexibility, and lower small-message overhead are central to that problem.

Figure 3 shows the network backdrop. Cross-rack, cross-zone, and cross-data-center traffic sees roughly 7x, 15x, and 30x the latency of same-rack communication. At that point, initialization, topology handling, registration, and small-message overhead are system constraints, not cleanup details.

Current status

The open source stack already has the key pieces in place: Direct communicators, a torch.distributed wrapper path, DeviceMesh integration, and a TorchTitan communication mode.

The safe wrapper example to show today is still the nccl route. The ncclx wrapper path is not something to present as universally wired through init_process_group yet.

That's a good way to read the current state of the project overall. The core architecture is already visible in the open tree. What is still uneven is the completeness of specific integration paths, not the direction of the design

Conclusion

The c10d library will remain the user-facing interface for a while, but the back-end layer underneath it is moving. TorchComms gives PyTorch explicit communicators, a wrapper path back into existing torch.distributed code, DeviceMesh integration, and communication surfaces such as windows and RMA that do not fit naturally inside the old back-end abstraction. The startup and inference results show why that matters at scale. Architecturally, the migration path is already visible in the open tree. That makes TorchComms the replacement path for the old c10d back-end layer, not just another back-end toggle.

References

• Official launch blog
• Official docs
• Python-first backend prototyping
• Official paper

The post PyTorch distributed is changing and TorchComms is why appeared first on Red Hat Developer.

Gratuitous AI related Image of  Max Headroom by juzmental on DeviantArt

So, I made some minor editorial changes, built out an MCP to demonstrate its use, and submitted my draft The prompt URI Scheme for AI Agent Sessions and Prompts to the IETF.  We'll see where this goes, it's my first time through this process.

At the same time, if you look at AudaciousInquiry/prompt-url: The prompt: URL scheme for AI agent session provenance project on GitHub, you'll see a readily installable MCP and skill for using it in that repo.  The skill documents one use of the MCP, to create a URL for annotating commits, and another use enabling resolution of the prompt via the MCP so that the prompt truly is a traceable link on
the machine and user account where it was run from.

There should be a second skill that describes the use of the MCP to actually find the right Prompt-URL: value to use for a github checkin.  Both the agents I use already know how to do this so I should have one of them write it up.  By the time you see this, it might already be present in the .claude folder.

For what it's worth, GitHub Copilot reads CLAUDE.md and .claude/skills folders so you only have to write things once for Claude and Copilot will use the tooling.  

The MCP framework is highly reusable, and eventually I will extract it from the base MCP APIs.  It runs in node.js in three modes, as a standalone HTTP server, in stdio mode launching a server if one is not already running and connecting to it over HTTP when commands are sent to it, or via CLI where it basically does the same kind of thing.  This makes standalone MCP testing very easy, because the framework handles transport and the rest, so you can just implement commands and get three different ways to activate them.

     Keith