I was at a ThoughtWorks “The Future of Software Engineering” event in Switzerland some weeks ago. What a braintrust. Lots of conversations about AI generally, agentic AI in particular and harness engineering.

Here’s a distinction I’ve been mulling for a while and keep reaching for in conversations about AI systems in the enterprise: build-time AI versus run-time AI. One is artifact-oriented, the other is execution-oriented.

Run-time AI isn’t only transactional, either. Generation of movies, songs, images, and slide-shows is run-time too. Those outputs are files, which makes them look artifact-like, but they’re products of execution, not engineering artifacts. Say a generated slide-show doesn’t specify the behavior of any system, and nobody’s putting it through code review as it might be going straight into GoogleDocs or Office 365. That’s the real test for which side of the line something sits on: build-time artifacts specify (later) behavior; run-time outputs are the consequences of wished-for behaviors.

Cross that with the conversational -> workflow -> agentic ladder that has become the common way of classifying AI systems, and you get a tidy 3×2:

Harness engineering is a build-time discipline

The interesting consequence: harness engineering is predominantly a build-time discipline, even though it exists entirely to influence run-time behavior.

The build-time harness looks like source code, because it is source code:

AGENTS.md
tools.yaml
memory_policy.py
planner.py
evals/
guardrails/
prompts/

Well, it is today. It gets committed to Git, reviewed in pull requests, and deployed via CI/CD - the full software delivery treatment.

Run-time is a different world entirely. Steps in a workflow:

1. Customer asks for a refund
2. Agent plans
3. Looks up CRM
4. Issues refund
5. (Audit log written)
6. (Database updated)

The last two are in parentheses because the agent isn’t doing them directly - the existing application or service (AI made or not) it invoked does those as a matter of course.

No Git commit occurs during that interaction. The agent changed the state of several systems of record, but it didn’t change itself.

An orthogonal axis, not a maturity stage

It’s tempting to present build-time vs run-time as another rung on the maturity ladder, after conversational, workflow, and agentic. I’d resist that. It’s an orthogonal axis. There are two independent questions to ask of any AI system:

1. What kind of AI system is this? - Conversational → workflow → agentic.
2. Where does the “engineering” happen? - Build-time artifacts ↔ run-time execution.

Every cell in the 3×2 above is a legitimate place to be, with the non-agentic ones included in that claim. A prompt-engineered conversational system with its templates under version control is not “less mature” than an agentic one. Indeed, it may be exactly the right amount of machinery for the job.

What a “harness” actually is

This framing also gives “harness” a satisfyingly software-engineering flavor. Edward Mangini’s recent Harness Engineering: The Devil is in Your Details defines a harness very broadly - essentially “everything except the model.”. He credits others (Trivedy & Birgitta Böckeler) for that. I think there’s a cleaner separation: the harness is the version-controlled, deployable specification of agent behavior. The live conversation, the database state, and the external side effects are not the harness - they are the consequences of the harness executing.

That gives harnesses a lifecycle human software engineers will recognize:

**author -> review -> commit -> deploy -> execute -> observe -> revise

Everything before “execute” is build-time. “Observe” is where run-time telemetry feeds back into the next build-time iteration. Which is to say: harness engineering is just engineering, and all the disciplines we already have for source code - code review, trunk-based development, continuous delivery, configuration as code - apply to it directly.

Well of course, build-time vs run-time isn’t common language in the AI field, so I’ll keep on trying to update my mental model.

An AI agent pursues a goal on its author's behalf over minutes, hours, or days. It calls tools, writes files, keeps state between steps, and makes its own decisions within the guardrails it was given. At different moments it leverages a request/response service, a batch job, a serving endpoint, and other capabilities.

Running this type of workload safely requires a solution architecture of its own. In this article, we describe that solution as a blueprint to use for your agentic workload delivery needs.

A new workload class

The model is a stateless inference service. The harness turns it into an agent: it owns the loop, the memory, the context window, and the tool calls. The sandbox runtime decides what the agent may touch: which files, which binaries, which network endpoints. These three components have separate lifecycles, scale differently, and carry different security requirements, so we keep them separate.

Red Hat AI, the enterprise platform for any model and any agent across the hybrid cloud, is a natural host for that separation. We write this blueprint mainly from an Agent-as-a-Workload perspective. This means each agent is one sandboxed pod and one individual, independent value proposition. A second deployment pattern, Agent-as-a-Service, moves the loop out of the agent software into a shared runtime. Its adoption will follow business and operational needs, and we contrast the two later in this article.

We describe the solution blueprint as an open architecture of interchangeable components. Every component in Figure 1 is a role that several implementations can fill. AI accelerator vendors ship implementations for many of them; NVIDIA currently has the most visible set with NemoClaw, OpenShell, Nemotron, and CUDA-X. They appear in our blueprint on those merits, though they are not required. The architecture stands on open interfaces: an OpenAI-compatible inference access API and the open source Open Responses specification for models, Model Context Protocol (MCP) for tools, the agent-sandbox API for isolation, and SPIFFE (Secure Production Identity Framework For Everyone) for workload identity. This architecture gives each agent a verifiable, short-lived credential instead of a static API key.

We wrote this article for platform engineers and AI infrastructure teams who need to run autonomous agents safely on open, scalable AI platforms such as Red Hat AI. This article is not a tutorial for building an agent. Instead, it serves as a blueprint for operating agents as governed cloud-native workloads.

The solution architecture

In Figure 1, we lay out the complete blueprint in a single view. The rest of the article walks you through the blueprint component by component. The figure reads as three horizontal bands with one checkpoint between them, and the legend colors mark the planes.

The control band across the top (blue) creates and governs agents. Read it from left to right to understand the lifecycle:

• An agent blueprint declares what each agent is made of.
• The sandbox control plane turns that blueprint into a Sandbox resource.
• A controller turns the resource into exactly one pod.
• The workload identity service issues every pod and gateway a verifiable credential.

The dashed arrows dropping out of this band are control traffic, and nothing else about the figure moves until they've done their work.

The middle band is where the magic happens. It contains three zones from left to right:

• Skill backends: These backends can run anywhere (represented by a dashed border because location is not a commitment).
• Agent pod: This pod hosts the agent, harness, supervisor, and thin skill clients.
• Inference plane: This plane serves models inside the cluster and contains the gateway, semantic routing, scheduler, and vLLM workers.

Two gray boxes on the right edge sit apart from that flow on purpose: an optional external large language model (LLM) that is reachable only with masked personally identifiable information (PII), and audit and tracing, where every hop in the architecture eventually lands.

The checkpoint between the bands is the MCP Gateway. It's drawn in control-plane blue because of its function: every tool call leaves the pod through this gateway, is authorized by token claims rather than by the model itself, and then reaches a skill backend.

The platform bars at the bottom carry everything above them: Red Hat OpenShift, the Linux kernel, and the silicon. The silicon bar lists several options deliberately because the architecture assumes no single vendor.

Components

Each solution component in and around Figure 1 has a role, a duty, and a set of implementation options. We list these options in Table 1. Maturity varies across these options, and we separate production-grade implementations from technical preview features later in this article (see Table 4).

Agent packaging

The sandbox layer on Kubernetes consists of two pieces, with one on each side of the pod. Outside the pod sits a sandbox control plane, and inside the pod runs a supervisor process. In our blueprint OpenShell supplies both, and any implementation can fill either role because the contract between them is a Kubernetes custom resource.

The delivery path has two hops, with a custom resource in the middle. The control plane receives a request for an agent and writes a Sandbox custom resource; in our blueprint the OpenShell Gateway is the target implementation of this role as its Kubernetes path matures. The Agent Sandbox controller, a Kubernetes Special Interest Group (SIG) Apps subproject that implementations target, reconciles that resource into exactly one pod. One sandbox maps to one pod in the current API.

Inside the pod, the supervisor runs as PID 1 and starts before the agent does. It fetches policy from the control plane, installs the kernel filters, injects credentials, and then launches the harness as its child process. All egress leaves through the supervisor's proxy.

Policy enforcement splits along this same boundary. Kubernetes governs the pod from the outside using the following primitives:

• Pod security standards
• SELinux
• NetworkPolicy
• An optional Kata microVM through RuntimeClass
• Confidential containers beneath those layers

The supervisor governs the process from inside, regulating which binaries the process can run, which paths it can touch, and which endpoints it can reach. Stacked together, these layers provide three nested isolation rings: hardware or virtual machine (VM), pod, and process. Every primitive in all three rings relies on standard Linux and Kubernetes technologies, such as seccomp, Landlock, namespaces, SELinux, AppArmor, or Kata Containers. None of these primitives depend on a GPU vendor.

One discipline from our 12-Factor Agent article carries over directly: the agent workload is long-lived, but its sessions are not. Sessions remain ephemeral, while workflow state lives outside the pod in a durable data store, such as a database, an object store, or a durable-execution engine. Consequently, each new session starts with a clean context.

Agent-as-a-Service

Everything above assumes the loop ships inside the pod with the harness. Red Hat OpenShift AI 3.5 (Early Access at the time of writing) ships a second pattern through the open source OGX project (formerly Llama Stack): Agent-as-a-Service. Here the agentic loop runs in a shared runtime behind the Responses API. The client sends one request naming the goal and the MCP servers it requires. The loop runtime advertises the tool schemas to the model, executes the calls the model selects, feeds results back, and iterates to a final answer. Every step lands in a durable data store. Figure 2 contrasts the two patterns side by side.

The shared runtime inverts the pod pattern's trade-offs. It gains:

• Enforcement in one place: every action transits the loop runtime.
• One audit trail across all tenants.
• Credential isolation: clients never hold tool credentials.

It pays with:

• Shared blast radius: the loop service is a single point of failure.
• API mismatch: a request-scoped API fits long-running, goal-holding agents poorly.
• Blurred identity: per-agent identity survives only if the runtime propagates caller identity downstream, a gap SPIFFE does not yet address; the active standards work is in the IETF Workload Identity in Multi-System Environments (WIMSE) working group.

Most harnesses today are thick clients that own their loop, so adoption starts with operations teams standing up a shared runtime for request-scoped workloads: assistants, copilots, high-tenancy tools. Autonomous agents follow later.

The two patterns also compose. A sandboxed agent pod can delegate tool execution to a shared loop while keeping its own identity and rings, and the MCP Gateway's claims-based checks preserve per-agent policy inside the shared runtime. Duration and ownership pick the pattern, the same way you choose between a standard Deployment and a Knative service.

The governance

Governance in this blueprint consists of two mechanisms working as one system: workload identity establishes who the agent is, and tool authorization decides what that identity can do. Neither mechanism consults the model, and that's the point. While the agent's behavior can be manipulated, its credentials and permissions cannot.

Workload identity

SPIFFE/SPIRE gives each agent an automatically rotated cryptographic identity that is scoped to its namespace and service account. An authentication sidecar uses that identity for standards-based exchange with an identity provider such as Keycloak, using token exchange (RFC 8693) or the JWT bearer grant (RFC 7523), depending on the provider's support for externally issued tokens. 

This process trades the identity for short-lived access tokens to downstream services, to help verify no static API key exists anywhere in the path. The payoff is clear: a leaked prompt or workspace file contains nothing durable to steal. Lifecycle tooling planned as part of Red Hat AI injects the sidecar and the identity plumbing at deploy time, with no agent code changes.

Tool governance

The MCP Gateway, built on Envoy with Kuadrant/Authorino for policy, fronts every MCP tool server behind a single endpoint. The agent is configured with one MCP_URL variable and sees an aggregated tool catalog. Token claims minted during the workload identity exchange then decide which tools the agent can actually call. The gateway never reads the prompt. A prompt-injection attack that forces the model to call an unauthorized tool fails at the infrastructure layer because authorization is not the model's decision to make. 

What the gateway cannot stop is misuse of tools the agent is entitled to call, such as an injected instruction that exfiltrates data through a permitted endpoint. That risk belongs to the inference guardrails and egress policy described in the security model, which is why this checkpoint is one control among several rather than the whole defense. The same claims-based check covers every tool backend equally: a CUDA service, an OR-Tools container, or a Software-as-a-Service (SaaS) endpoint. MCP itself now lives under neutral governance, donated to the Agentic AI Foundation under the Linux Foundation in late 2025, with an authorization model that makes each tool server an OAuth 2.1 resource server, which is precisely the claims-checking this gateway enforces. This is the checkpoint from Figure 1 doing its job.

Inference routing

An inference request crosses three routers on its way to an AI accelerator. Three tiers exist because the three decisions have different owners: where a query may go is a security decision, which model serves it is a cost decision, and which replica executes it is an efficiency decision. Collapsing them into one router would put one component in charge of three concerns that change at different speeds and answer to different teams. Table 2 summarizes the three tiers, and Figure 3 follows one request across them.

Tier 1 is the security tier and lives client-side in every sandbox, inside the supervisor. Confidential context never leaves the cluster, and anything routed to an external endpoint is disguised to protect personally identifiable information (PII) first. For in-cluster traffic, the same router directs requests to the AI gateway, so even local inference enters through the governed front door rather than reaching model endpoints directly. The masking is detection-and-redaction at the proxy, so residual false-negative leakage is the control's main failure mode and should be monitored.

Tier 2 is the cost tier: a lightweight classifier selects the cheapest model that can handle the request. The classifier itself adds latency and can misroute, so this tier is a measured trade-off rather than free capacity. The system needs a fallback that escalates to a more capable model on low-confidence or low-quality signals.

Tier 3 is the efficiency tier: once the model pool is fixed, llm-d's endpoint picker maximizes KV-cache hits and balances load across vLLM workers. Red Hat's integration work describes the split the same way: the semantic router picks the model, llm-d schedules replicas within the pool.

Two properties make this path operable, and they're the same two demands you would place on any request path in production. It is traceable end to end when instrumented: OpenTelemetry follows a request across all three tiers, and MLflow Tracing records tool calls, prompts and outputs, latencies and costs. MLflow Tracing itself is production-ready; the OpenTelemetry GenAI semantic conventions it emits are still in development, so plan for attribute changes rather than interface rewrites (see Table 4). And it's decoupled at every hop: every hop speaks OpenAI-compatible or OpenResponses APIs, so the model behind the path can change without the agent noticing.

Job placement

A common design mistake we see is assuming the skills execute inside the agent pod. This habit carries over from library-based development, where importing a capability means hosting its compute resources. In this blueprint, the skill components in Figure 1 represent definitions and thin clients: instruction files and API bindings rather than AI workloads. The distinction keeps the agent pod cheap and the blast radius small. Heavy compute tasks follow one of three patterns, and only pattern 1 runs in the pod: execute in place, call a service, or dispatch a job (Table 3).

Pattern 2 serves as the default model at scale. For this reason, Figure 1 illustrates skill backends as a location-agnostic column outside the platform bars. The optimization service can be cuOpt on a GPU node or OR-Tools on CPU; the agent's client and the gateway policy do not change. Because authorization is endpoint-specific and claims-based, relocating or swapping a backend is a policy change rather than an architecture change. Pattern 3 needs the most attention in security review: an agent that can create Jobs can create arbitrary workloads, so its RBAC must be scoped tightly.

Getting this split right also sets the economics. Agent pods stay resource-efficient, dense, long-lived, and CPU-only in the common case, with the in-pod GPU of pattern 1 as the deliberate exception, while expensive accelerators sit in pooled backends that scale independently. The fleet then scales on two independent axes: agent count on cheap cores, accelerator capacity on utilization, and neither purchase forces the other.

To see the blueprint in action, follow one request end to end; every component named so far makes exactly one appearance. 

A field-service agent receives a dispatch-optimization goal and initializes inside its sandboxed pod. The process then unfolds across the layers:

• The harness calls the optimization skill; the MCP Gateway checks the agent's token claims and forwards the call to the cuOpt or OR-Tools backend.
• The planning prompt carries customer data, so tier 1 keeps it on in-cluster models; a follow-up summarization request is low-risk, so tier 2 routes it to a small model and tier 3 picks the replica with the warmest cache.
• Every tool call, authorization decision, latency, and cost lands in the OpenTelemetry and MLflow trail. 

At no point does the architecture depend on the model performing predictably; the path maintains the same shape even against a hostile model.

Security model

The distinctive threat with a long-running agent is that the workload itself can be turned against you. A prompt-injection attack can redirect its goals, a poisoned tool result can corrupt its memory, and a compromised skill dependency can act with everything the agent is allowed to reach. The design response is to assume the model's behavior cannot be trusted and to route every consequential action through enforcement the model cannot influence.

Figure 4 shows how the architecture layers that enforcement: three nested isolation rings around the agent process, and four cross-cutting controls that apply across all of them. The rings nest deliberately: each one assumes the ring inside it has already failed.

• Ring 1 (process): The sandbox supervisor enforces per-binary, per-path, per-endpoint policy via kernel primitives (seccomp, Landlock, network namespaces). The agent cannot override this policy even if compromised because the policy lives outside the process it constrains.
• Ring 2 (pod): Kubernetes pod security standards, SELinux, AppArmor, NetworkPolicy, and resource quotas bound what the pod can reach and consume. Optionally, Kata Containers add a microVM boundary per sandbox.
• Ring 3 (hardware): Confidential containers extend the trust boundary into silicon, protecting an agent's memory at runtime even if a neighboring workload or the node itself is breached, with attestation proving the sandbox runs the code it claims before any secret is released. This ring is the precondition for multi-tenant agent designs. Confidential computing on GPUs is now generally available on current accelerators but still carries runtime overhead, concentrated in host-to-device transfers, so treat it as the ring you plan for rather than the one you start with.

Four controls cut across all three rings, and they carry the same names in Figure 4:

• Workload identity: SPIFFE SVIDs replace static keys, so a leaked workspace contains nothing durable to steal.
• Tool authorization: The MCP Gateway authorizes every tool call from token claims, never from prompt content.
• Inference guardrails: Input and output screening at the inference boundary (TrustyAI Guardrails Orchestrator, NeMo Guardrails).
• Pre-production testing: Adversarial scanning in CI/CD pipelines catches jailbreak and injection weaknesses before an agent is promoted.

The rings and the controls answer different questions. The rings limit what a compromised agent can do; the controls reduce the chance it gets compromised and remove the rewards if it does. Containment economics follow from the single-sandbox, single-pod, and single-identity mapping: revoking, throttling, or terminating a misbehaving agent is a single targeted operation instead of an incident-wide hunt. The MLflow/OpenTelemetry trail turns every enforcement decision into reviewable evidence, and the posture holds unchanged at the edge as in the data center.

Vendor gravity and escape hatches

Vendor gravity in this stack lives in the libraries and the silicon, one layer below where most lock-in audits look. The license layer is clean: several core building blocks are open source under permissive terms (OpenShell and NemoClaw are Apache-2.0), so no single vendor holds exclusive rights to the architecture, while commercial packaging, NIM profiles, CUDA libraries, and certified support remain vendor-specific. What binds you, when something does, is an unported library or a proprietary interconnect. Here is our layer-by-layer audit:

• Interface-portable: The model layer is any open model behind the same API. The agent blueprint is YAML, portable by construction. The sandbox runtime enforces policies with stock Linux primitives, and Kata Containers plus the agent-sandbox SIG covers the same three rings on any conformant cluster. The whole inference plane is portable: semantic routing and llm-d are vendor-neutral, and vLLM runs on CUDA, ROCm, XPU, and CPU directly, with TPU, Gaudi, and other accelerators supported via hardware plug-ins. The serving architecture therefore survives a full silicon swap untouched at the interface level. Achieving operational parity in performance, tuning, and accelerator coverage is engineering work you should plan for.
• Sticky: The CUDA-X libraries are the real gravity: cuDF, cuOpt, and PhysicsNeMo are CUDA underneath, and NIM containers run only on NVIDIA silicon. Substitutes exist with trade-offs: Polars, DuckDB or Spark for DataFrames, OR-Tools or HiGHS for optimization, and JAX-based surrogates for physics. What the architecture controls is the price of the swap: skill backends sit behind neutral REST/MCP interfaces, so replacing cuOpt with OR-Tools is an endpoint and policy change rather than a redesign.

The rule we enforce throughout the blueprint: every vendor component sits behind a neutral interface. That means OpenAI-compatible and OpenResponses for models, MCP for tools, REST for skill backends, the agent-sandbox API for isolation, and SPIFFE for identity. Under that rule, vendors compete on price/performance per component instead of owning your architecture, and procurement decisions stay reversible for the lifetime of the deployment. NVIDIA then becomes a per-component choice rather than a structural dependency.

Maturity and open items

Not everything in Figure 1 is equally ready. In Table 4, we separate the layers you can run in production today from the layers to treat as preview, where interfaces will still change. The reading rule: build your base decisions on the mature column, adopt from the preview column behind pinned versions and feature gates, and revisit at each release.

And a blueprint earns trust by stating what it does not solve yet. Note that each of the following open items belongs to a different discipline, which is itself the message: closing them takes products, standards bodies, researchers, and operators, and no single vendor closes the list.

• Human approval workflows for high-risk actions still need product-level patterns.
• Multi-agent coordination standards are converging (A2A reached 1.0) but interoperability in practice is not settled.
• Agent memory poisoning remains an open research and engineering problem.
• Sandbox policy authoring is operationally complex at fleet scale.
• Cost modeling for long-running agents needs real workload data.
• Confidential computing protects memory at runtime; it does not stop prompt-layer or tool-layer compromise.

Closing notes

In this article we walked the agentic stack inside out. We defined the agent as model, harness, and sandbox runtime, and laid the solution out in Figure 1 as interchangeable components behind open interfaces. 

Then we followed it through delivery: an agent packaged as one sandboxed pod, or delegating its loop to a shared runtime where the workload profile fits, governed by workload identity and claims-checked tool calls, served through three routing tiers to vLLM workers, with skill compute placed in governed backends and a defense-in-depth security model wrapping it all. 

The result is a blueprint you can begin assembling today from open source parts, treating several layers as preview, with vendor choices that stay reversible at the interfaces you kept neutral. Managed sandbox services offer parts of this blueprint as SaaS; what the open architecture adds is that the isolation, identity, and governance layers stay on infrastructure you control, with interfaces you can take to any conformant cluster. We verified the component landscape in this article as of July 2026; in a stack where several layers are pre-1.0, check the linked projects for current status before you build.

The agentic stack is beginning to look like the microservices stack one abstraction up: blueprints instead of charts, sandboxes instead of containers, inference routing instead of service mesh, skills instead of client libraries, and share d loop runtimes instead of application servers. The same disciplines apply: least privilege, declarative policy, separate control and data planes, pooled expensive resources behind schedulers, and keeping the interfaces under your own control. The difference is that this workload makes its own decisions for days at a time. The stack above is what it takes to let it do that safely, observably, portably, and at a cost per task you can measure.

Where to go next

Architecture arguments are easier to judge with running code, so we are rebuilding our Telecom AI Experiments workloads on this blueprint at agent-school; the 101 NOC Assistant runs offline on a laptop in two commands. To run the blueprint on your own infrastructure, we recommend exploring Red Hat AI, Red Hat OpenShift AI, and Red Hat AI Inference Server.

"Hey, can you take a photo of me using Photo Booth with a filter… also, open Chess and move some of the pawns up."

That little exchange shows computer use in action, one of the coolest things happening in agentic AI right now.

What is computer use?

Computer use lets an AI use software the same way you do, so it can view the screen, click, and type—all without you touching the keyboard or mouse. This approach is invaluable for automating tedious, manual tasks. For example, one of my previous jobs required me to move data from a CRM into Jira tickets. Computer use can also help with enterprise-level tasks where custom integrations are not available.

Pretty neat, right? But you almost certainly have private data on your own machine that you'd rather not hand to a third-party AI provider. If you pay per token via an LLM API, these tool calls can get quite expensive. You might want to look at running your own LLM and wiring it up to an agent running locally.

With that in mind, I'll show you how to run an open-weight model (such as Qwen or Gemma) with an open source AI harness (Hermes) so that you can implement computer use on your own terms.

Step 1: Pick a model (and don't overpay for hardware)

When you choose a model, it's worth checking out Red Hat AI on Hugging Face, where an entire library of compressed models can dramatically cut your hardware requirements (in some cases, halving the RAM requirements or more). The core technique is quantization, which squeezes a model down to fewer bits per weight (Figure 2).

Take the Qwen 3.6 35B model shown in Figure 3. The native released weights in full precision are roughly 72 GB. Red Hat's 4-bit version? About 25 GB. This reduced footprint means that you can run this model on your own MacBook Pro with stable performance. Thanks to modern compression algorithms, the quantized version performs almost identically to the original on most benchmarks. The compressed version retains 99% of the original model accuracy while using a fraction of the hardware footprint.

Step 2: Deploy it (either locally or on Kubernetes)

For this example, we use Qwen 3.6 with 35 billion parameters. It's a sparse Mixture-of-Experts model, meaning it features 35 billion total parameters with roughly 3 billion active parameters per token, and it's natively multimodal. This architecture makes it a good fit for computer use because the agent passes a screen capture to the model and then uses tools to interact with user interface elements like check boxes.

Now, let's get it running.

Deploying the model locally

If you have the hardware to run it (I like to use the llmfit project to check compatibility before downloading anything) you might be surprised by how quickly you can serve a model locally. As a macOS user, I've tried most of the local inference tools, from Ollama to LM Studio and upstream llama.cpp. These days, I'm reaching for oMLX, a simple open source server and menu-bar application that manages models and exposes an OpenAI-compatible API endpoint (Figure 4). This tool usesApple's MLX framework to accelerate inference through unified memory.

If you are developing on Windows or Linux, the same model runs through llama.cpp. Download a model (Unsloth, shown in Figure 5, makes some good quantized ones), point llama-server at it, and configure the same OpenAI-compatible endpoint to connect your agent.

Deploying the model with vLLM on OpenShift AI

On Red Hat OpenShift AI, you can deploy models directly from the model catalog onto a Kubernetes environment. You can then create a route to make the model reachable from outside the cluster, authenticated with a custom API key.

Inside the cluster, the platform optimizes throughput and reduces latency by intelligently routing traffic and disaggregating the inference steps. This process splits the compute-heavy prefill phase from the memory-bound decode phase by using the llm-d project. In Red Hat's published benchmarks, intelligent inference scheduling delivered up to 57 times faster time-to-first-token and roughly two times the throughput compared to naive round-robin routing on the same hardware.

Note

Step 3: Set up the local agent

On my local machine, I've installed Hermes, a popular open source AI agent that runs on macOS, Windows, and Linux (Figure 7). Hermes can write code, manage your calendar, diagnose your Kubernetes cluster, and more. Because it connects to any OpenAI-compatible endpoint, I can point it directly at the model I just served—whether that's oMLX on my macOS environment or vLLM on OpenShift—with no code changes. The configuration requires only a base URL and an API key.

To interact with your desktop, Hermes performs computer use through a computer use agent (CUA) driver, which is an open source tool that lets the agent control the screen. The driver also includes configurable, multi-layer guardrails that protect windows from closing unexpectedly and require user approval before the agent types text or takes sensitive actions. You stay in the loop.

Step 4: Profit? Well, yes

So, let's give it a try: "Hey, can you use my browser to go to the vLLM repo and give it a star?"

Just kidding, you don't have to. But that's what computer use is all about. The model reads the screen, finds the browser, locates the star button, and selects it automatically. This architectural configuration differs because it runs entirely within a sovereign environment, keeping the local model, the local agent, and your private data on your machine.

To explore more capabilities, view the available open source tools and options from Red Hat AI.

For most PyTorch users, "distributed" still means c10d. That's the runtime surface they use today, and it will remain the surface for a while. The real question is what should replace the old backend layer underneath it as training and inference systems demand more than host-driven process groups and standard collectives. TorchComms is the clearest answer PyTorch has presented so far.

Why replace c10d

The c10d library gave PyTorch a ProcessGroup-centered distributed runtime. That interface is still what most users touch, and it's not going away overnight. But the old back-end layer underneath it was designed around host-driven collectives and a smaller communication design space than modern LLM training and inference now need. TorchComms is the replacement path for that back-end layer.

That's the right scope for the claim. TorchComms doesn't need you to rewrite your code and throw away torch.distributed. Instead, it allows you to replace the old c10d backend substrate with a communicator-centric layer that can still plug back into torch.distributed.

The official launch material makes that roadmap explicit: The old c10d::Backend interface is expected to be retired gradually as TorchComms stabilizes underneath PyTorch Distributed.

That distinction matters. The replacement target is the back-end layer, not the entire distributed surface area. ProcessGroup, DeviceMesh, FSDP, TorchTitan, and higher-level training code can keep their existing structure while the communication implementation underneath them becomes more explicit, more extensible, and easier to evolve.

Communicator-centric API

The core API shift is simple: TorchComms starts from explicit per-device communicators.

import torch
import torchcomms

device = torch.device("cuda")
backend = "ncclx"
comm = torchcomms.new_comm(backend, device, name="global")

x = torch.ones(1024, device=device)
comm.all_reduce(x, torchcomms.ReduceOp.SUM, async_op=False)

That looks small, but it is a real model change. Collectives, point-to-point operations, hooks, finalize, and window operations all hang off the communicator object. TorchComms also exposes pluggable backends, eager communicator setup, DeviceMesh integration, and communication surfaces that go beyond classic ProcessGroup collectives.

The practical payoff is lifecycle control. With a communicator object, setup, split, finalize, and backend-specific extensions live in one place. That makes it easier to add semantics such as windows, RMA, transport-specific hints, or future fault-tolerance operations without overloading a single generic ProcessGroup interface.

That back-end surface is also becoming easier to prototype against. A recent PyTorch distributed devlog shows Python bindings for TorchComms backends and two pure-Python prototypes, one on top of nccl4py and one on top of SymmetricMemory plus Triton, both still plugging into torch.distributed.

Those prototypes are not production-ready, but they reinforce the same point: TorchComms is not just a replacement for an old back-end interface, it's becoming the place where new communication ideas can be tried without first rewriting the C++ comms stack.

Migration path into PyTorch and Torchtitan

TorchComms is not only a standalone API. PyTorch already contains a wrapper path in torch/distributed/distributed_c10d.py that instantiates a TorchComms communicator, wraps it in _BackendWrapper, and attaches it to the existing ProcessGroup machinery when dist.config.use_torchcomms is enabled. That keeps the migration small:

import torch.distributed as dist
import torch.distributed.config as dist_config

dist_config.use_torchcomms = True
dist.init_process_group(backend="nccl")
dist.all_reduce(tensor)

The important detail in this local checkout is that the safe wrapper example is backend="nccl". The equivalent backend="ncclx" wrapper path still fails locally with AssertionError: Unknown backend type ncclx, so it must not be presented as already working everywhere.

Even with that caveat, the direction is clear. TorchComms is not being built as an isolated alternate runtime. It is being wired into the same torch.distributed and DeviceMesh ecosystem that large PyTorch training stacks already use today.

The same pattern shows up in Torchtitan. Torchtitan already exposes --comm.mode torchcomms, and its distributed initialization path sets dist.config.use_torchcomms = True before normal process-group and DeviceMesh setup.

Architecturally, that is the point: The model and parallelism code stay in the usual TorchTitan and DeviceMesh world. The communication layer underneath is what changes. In this bundle, the command is illustrative rather than a locally green end-to-end run, because the current environment still hits a tyro/typeguard mismatch before TorchComms initialization.

Figure 1 shows the clearest startup result. Default process-group initialization drops from 14.5 seconds to 3.97 seconds at 8,000 GPUs, from 55.71 seconds to 11.89 seconds at 32,000 GPUs, and from 265 seconds to 24 seconds at 96,000 GPUs.

That's the strongest argument for replacing the old back-end layer with something built for large communicator counts, faster control-plane work, and topology-aware setup.

Those numbers matter because startup is not a cosmetic metric at this scale. Large jobs create many communicators, restart after faults, and repeatedly pay control-plane costs that are negligible on a small cluster but dominant at tens of thousands of GPUs. Faster initialization directly changes how practical a communication stack is for real training systems.

Figure 2 shows the inference side. Decode-time improvement rises with communication pressure: for k=1, batch=128, it grows from 19% at 4 hosts to 56.5% at 16 hosts. For k=4, batch=256, it grows from 45.87% at 4 hosts to 83.3% at 16 hosts. The baseline here is the token-shuffling path built from two AllGather operations and one AllToAll with the same compute kernels, so this is a back-end layer result rather than a generic c10d against TorchComms chart.

This is also why the TorchComms story is not only about training collectives. The back-end layer must handle both bulk-synchronous training and more dynamic, metadata-sensitive inference patterns. GPU-resident metadata, transport flexibility, and lower small-message overhead are central to that problem.

Figure 3 shows the network backdrop. Cross-rack, cross-zone, and cross-data-center traffic sees roughly 7x, 15x, and 30x the latency of same-rack communication. At that point, initialization, topology handling, registration, and small-message overhead are system constraints, not cleanup details.

Current status

The open source stack already has the key pieces in place: Direct communicators, a torch.distributed wrapper path, DeviceMesh integration, and a TorchTitan communication mode.

The safe wrapper example to show today is still the nccl route. The ncclx wrapper path is not something to present as universally wired through init_process_group yet.

That's a good way to read the current state of the project overall. The core architecture is already visible in the open tree. What is still uneven is the completeness of specific integration paths, not the direction of the design

Conclusion

The c10d library will remain the user-facing interface for a while, but the back-end layer underneath it is moving. TorchComms gives PyTorch explicit communicators, a wrapper path back into existing torch.distributed code, DeviceMesh integration, and communication surfaces such as windows and RMA that do not fit naturally inside the old back-end abstraction. The startup and inference results show why that matters at scale. Architecturally, the migration path is already visible in the open tree. That makes TorchComms the replacement path for the old c10d back-end layer, not just another back-end toggle.

References

• Official launch blog
• Official docs
• Python-first backend prototyping
• Official paper