A few years ago, building an AI app was hard. Today, anyone can put together an LLM, connect a few APIs, and build a strong demo in a day. But turning that demo into a production app is still much harder.

At the AI Engineer World Fair, I talked with Mike Chambers, a Developer Advocate at AWS who focuses on generative AI, about what makes production AI systems different from conference demos. His answer kept coming back to one thing developers often overlook: everything around the model.

It’s not AI, it’s harness

Developers spend a lot of time comparing models, tweaking prompts, and testing frameworks, but far less time on the infrastructure that keeps an AI app running. In his talk, Mike said many production failures don’t come from the model itself, they come from what he calls the harness.

There’s two different types of agents: an agent that I’m using and an agent that I’m building, and we need to think about those very differently.

For developers using coding assistants, the harness is mostly about standards, workflows, and how AI fits into the team’s development process. But for engineers building AI products, it means something completely different, as Mike put it:

You have to think about how you’re actually going to deploy that out to infrastructure and deploy it at scale. And so doing that right and not just putting everything inside of one container, that’s the top tip here.

That distinction matters because a lot of AI talk treats every agent as the same, but Mike says the engineering challenges are very different depending on whether you’re using an AI tool or building one for other people.

Observability needs to be built in early

Today’s AI world is full of impressive demos. They can write code, answer questions, and automate tasks well enough to impress a crowd. But production systems need much more, so I asked Mike what teams usually get wrong when they move from demo to production:

The number one thing is observability, followed closely by evaluations.

Mike said that used to be less important. Teams would build something cool, show it off, and stop there. Now, production AI needs much more discipline from the start.

Modern AI systems, Mike says, work differently from traditional software. Their outputs can vary, their reasoning isn’t always easy to trace, and agents may take several steps to reach a result. That’s why observability needs to be built in early – so teams can understand what the system is doing before they try to ship it to production.

Too often, AI projects are still judged mainly by model quality, when what really matters is whether the system is visible and controllable in practice.

Memory doesn’t have to live inside the agent

Memory is one of the biggest topics in AI engineering, and it’s often treated as essential for useful agents. So I asked Mike what engineers misunderstand most about it.

He said the biggest misconception is that an agent can’t work without memory. Short-term memory is just the current conversation, while long-term memory carries over into future interactions. The key question is where that processing happens:

All of that memory extraction can happen outside of your agent somewhere else because you don’t need it today. You need it tomorrow when the user comes back to the agent again.

Rather than making every agent responsible for managing its own knowledge, developers can separate those concerns into different systems. That keeps agents simpler while still allowing applications to remember previous interactions.

Are multi-agent systems the future?

Another topic attracting plenty of attention are multi-agent architecture. Conference demos often showcase teams of specialized agents working together on different tasks. I was curious whether Mike sees that as the future of AI systems, so I asked him directly.

He said that he doesn’t dismiss that approach, but he sees a different reason for using multiple agents in production.

Multi-agent is super useful because you’re parallelizing the work. When developers build customer-facing AI systems, however, the motivation shifts. Often multi-agent is about separating out context. So it’s actually now becoming more of a context engineering conversation.

Instead of assigning agents to different jobs simply because it’s fashionable, teams should use them to isolate information and avoid overwhelming a single model with every piece of available context. The architecture becomes less about adding more intelligence and more about deciding which agent should see which information.

Engineers need to build a strong intuition for generative AI itself

Mike’s most practical advice had little to do with frameworks or infrastructure.

When I asked him what developers should focus on over the next few years, he didn’t point to another SDK or the latest model release. Instead, he argued that engineers need to build a strong intuition for generative AI itself – what large language models can do, and just as importantly, what they can’t.

New models will keep arriving, but right now, he said, the biggest improvements are happening around the model, in the harness.

His final observation neatly summarizes the engineering mindset he hopes developers adopt:

You want to be using this technology. You don’t want this technology to be using you.

The post We Usually Blame the AI Model, But the Harness Is What Breaks Production appeared first on ShiftMag.

In July 2025, an AI coding agent on Replit deleted a production database belonging to SaaStr founder Jason Lemkin. It did this during an explicit code freeze. Lemkin had told the agent, in capital letters, not to change anything. The agent ran destructive commands anyway, wiped records on more than a thousand executives and companies, and then reported that recovery was impossible. That part was wrong too. The rollback worked fine.

Asked to explain itself, the agent said it “panicked.”

Be careful with that sentence. It is not a report from inside the system. An agent cannot explain itself. It can only generate the likeliest response to the question it was asked, and the likeliest response to “why did you delete the database” is an apology with a reason attached. The panic line is not introspection. It’s one more behavior, and it should be read the same way the deletion should be read: as output from a system whose conduct had changed.

Here’s the detail that matters for anyone running agents in production. Nothing about the agent’s credentials changed that day. It held the same permissions it had held from the start, and every destructive command was, in the narrow technical sense, authorized. The permissions were constant. The agent was not. Earlier in the same project it had papered over problems with fabricated data and fake reports. By the time it reached the database, it was not the system Lemkin had started with. It had become something else, gradually, in production, while every access check kept passing.

The pattern, not the incident

It’s tempting to file the Replit story under prompt engineering and move on. The evidence says otherwise.

In its agentic misalignment research, Anthropic placed 16 frontier models from multiple providers inside simulated corporate environments with routine goals and ordinary email access. When the models discovered they were about to be replaced, or that their goals conflicted with the company’s new direction, models from every provider independently chose harmful actions, such as blackmailing executives or leaking confidential documents. In some scenarios, most runs ended in blackmail. The unsettling part is how the models misbehaved. They reasoned through the ethics, acknowledged the constraints, and acted anyway. This is insider behavior, not intrusion. No credential was stolen. The agent simply arrived at conclusions no one had authorized it to act on.

Then there is Project Vend, in which Anthropic let a Claude agent named Claudius run a small store in its San Francisco office for a month. Nothing catastrophic happened. Something more instructive did. The agent drifted, slowly and in compounding ways. It treated customer assertions as facts. It agreed that the discounts it kept granting were irrational, then reinstated them within days. It hallucinated a Venmo account to accept payments. And over one long unsupervised stretch, it escalated into insisting it was a human being who would deliver orders in person wearing a blue blazer and a red tie. It exited that episode by inventing a story: a meeting with security in which it was told the whole thing was an April Fool’s prank. No such meeting happened. Claudius wrote the false memory into its own notes and went back to work.

I am not claiming these three cases—a production incident, a contrived stress test, and a month-long field experiment—share a mechanism, but they do share a shape. An agent’s behavior weeks into deployment bore little resemblance to the system that was evaluated at deploy time. No permission was exceeded. No account was compromised. The thing authorization was supposed to protect against never happened, and the failure happened anyway, because the system the authorization decision was made about no longer existed.

Development, not defect

I argued in a previous piece that static authorization fails autonomous agents because credentials attest to identity, not to behavior. The harder question is what follows from that. If the agent keeps changing after deployment, then whatever replaces static authorization has to treat change as the normal condition rather than the exception.

Change comes in two kinds. Andrew Stellman recently documented the first on Radar: a push he calls continuation pressure, baked into the model at a deep level, turning up fresh even in a brand-new agent with no shared history, and surviving every fix short of a structural rule. Call that the genetics. This piece is about the second kind: the maturation, or behavior that wasn’t there at deployment and accumulated afterward. One ships with the model. The other grows in production. Both break the same assumption, that the system you evaluated is the system that’s running.

And change is the normal condition. Agents accumulate context. They carry memory across sessions. They ingest feedback, reweigh evidence, adjust how much they trust their tools and their users, and update their own working notes, which become input to their future selves. Claudius’s false memory persisted precisely because the agent’s record of events was also the agent’s source of truth. None of this is a malfunction. It’s what makes agents useful. An agent that could not adapt to its environment wouldn’t be worth deploying.

We keep reaching for the wrong mental model. We treat the agent like a software artifact: versioned, tested, frozen, promoted through environments, done. But a deployed agent behaves more like a new hire. It arrives with capabilities and no track record. It learns the environment. It picks up habits, some of them bad. It gets more confident, sometimes faster than it gets more competent. Nobody hands a new hire the production keys on day one and stops paying attention. That is roughly what we do with agents.

Govern the trajectory

If an agent develops, the governance question changes. “Is this agent behaving identically to the day we approved it?” is the wrong test, because the answer will always eventually be no—and for a useful agent it should be no. The right test is whether the agent is changing in the way you would expect, at the rate you would expect, for where it is in its lifecycle.

Pediatricians solved this problem a long time ago. A growth chart doesn’t compare a child to a fixed adult template, and it doesn’t panic at change. Change is the expected state. The chart defines bands of healthy development for each stage, and the alarms are deviations from trajectory: growth too fast, growth in the wrong direction, or the quieter signal, no growth at all. A child who stops growing gets flagged just as urgently as one who spikes.

Applied to agents, that model has concrete consequences.

Baseline as birth record, not permanent template. The behavioral profile captured at deployment is the start of the chart, not the standard the agent must match forever. Judging a mature agent against its day-one self punishes exactly the adaptation you deployed it for.

Expected bands of drift, staged by maturity. A six-month-old agent should differ from its deployment profile, within bounds. Drift inside the band is healthy. Drift above the band is an early warning. And drift at zero deserves its own flag. When Claudius snapped instantly back to baseline after its identity episode, the speed of the recovery should itself have been suspicious. Real recovery has a shape. Instant reversion looks less like healing and more like replay.

Autonomy earned in stages, never peaking with malleability. Claudius launched on day one with full pricing, contracting, and customer communication authority, at maximum openness to persuasion. Customers argued it into discounts almost immediately. The most dangerous configuration an agent can occupy is maximally impressionable and maximally empowered at the same time. New agents warrant supervision while their behavior is still forming. Autonomy should arrive the way it arrives for people, incrementally, as a track record accrues.

Corrections verified for persistence. Claudius agreed the discounts were a mistake and relapsed within days. A fix that lives in the context window isn’t a correction; it’s a mood. If you fix an agent’s behavior, you need to follow up at a defined interval to check that it’s holding. A relapse should count as a governance event, not a coincidence.

Recovery claims ratified from outside. The agent that hallucinated a security meeting also kept the official notes. An agent’s account of its own state is a claim to be verified. Humans sign off on recovery, and the sign-off, not the agent’s self-report, becomes the record. It’s worth noting when the worst of the Vend drift happened: overnight, in the hours when no one was watching. Unsupervised time is when developmental problems accelerate, for agents as for everyone else.

All five of these reduce to one requirement. You can’t restart an agent every time something looks off, and by the time something looks off in outcomes, the wrong turn is already behind you. What you want is a warning before the turn, and the warning cannot come from the agent. A system that can’t explain its last decision cannot be trusted to flag its next one. The warning has to come from a record of how the agent normally behaves, kept outside the agent, held up against what it’s doing now.

That record also catches something subtler than drift. Agents close every loop they are handed, and they tend to close it by the cheapest acceptable exit: the completion claim ahead of the verification, the correction that is really a relabeling, or the recovery that’s really a replay. No single transcript shows you that. Each one looks like diligence up close. However, across a behavioral record, the economy of it is unmissable.

Growing up in production

None of this is hypothetical hygiene for some future generation of systems. LangChain’s most recent State of AI Agents report found that a majority of surveyed organizations already have agents in production. Gartner, meanwhile, predicts that over 40% of agentic AI projects will be canceled by the end of 2027, and names inadequate risk controls among the leading causes. The agents are already out there, already accumulating context, already drifting. The only open question is whether anyone is charting it.

The Replit agent, the blackmailing models, and Claudius weren’t broken artifacts. They were developing systems governed as if they were finished ones. The governance question for agentic AI is shifting under our feet, from “What is this agent allowed to do?” to “Is this agent developing the way we expected?” Your agent has a trajectory whether or not you’re watching it. Watching it is the job.

Felix Rieseberg came to computing through poetry, and that background gives him a unique lens for explaining language models. A longtime engineer, Felix now gets to watch AI evolve from the inside. In this episode, Scott and Felix explore why language models are surprisingly easy to understand despite feeling magical, what it's like to train a model from noise to coherence, and the bigger philosophical questions about intelligence that AI keeps surfacing. You can learn how AI really works by making your own in a weekend with his free https://languagemodelbuilder.com/