Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
160128 stories
·
33 followers

Debunking the hotfix pipeline myth

1 Share

Hotfix pipelines are like an extended warranty; they make sense in principle, but in reality, they aren’t necessary. And just like an extended warranty, you spend a lot of money (and time) building a hotfix pipeline that you should use only rarely. This article will first walk through the fundamental flaws of hotfix pipelines. Then it will explain why it is better to focus on improving the standard Production deployment pipeline.

Defining hotfix pipelines

Hotfix pipelines are a "fast lane" to the normal Production deployment pipeline. Their purpose is to skip specific steps to get changes into Production as soon as possible. Typically, they are:

  • Production deployment pipeline: Development → Test → Staging → Production
  • Hotfix pipeline: Staging → Production

They exist because of this scenario:

  1. A critical bug appears in Production hours or days after a deployment.
  2. A rollback is impossible, as that will cause more problems than it solves.
  3. There is work in flight in lower environments.
  4. The hotfix pipeline skips the lower environments.

The core idea behind hotfix pipelines is that they allow normal work to continue while providing a path to fix a critical bug as soon as possible. Someone, either within the development team or the business, determines that a bug is important enough to fix and can skip use the hotfix pipeline.

Hotfixes in the real world

From experience, hotfix pipelines are created in response to a suboptimal Production deployment pipeline. There is a step, or several steps, that either add little value or take so much time that everyone is fine skipping them when certain conditions are met.

I experienced a suboptimal process before joining Octopus Deploy. Before Octopus Deploy, I was a software engineer for almost 15 years. This was a deployment process at a company I worked for before we fully automated it (using Octopus Deploy, of course!).

  • Four environments: Development, QA, Integration, and Production.
  • Shared database for the same application for all in-flight work.
  • The build server only monitored the main branch.
  • The build server automatically deployed to Development and QA.
  • Deploying to Integration and Production used different tooling and was a mix of manual and automated.

As you can see, deploying to Integration and Production was completely different from Development and QA. Some notable new steps include placing code build artifacts in "hot folders" to automatically deploy code, placing database delta scripts in designated network folders, and hand-typing instructions for the Web Admins and DBAs.

Deployments to Production took hours due to missing database and configuration changes, and occasional failures to update a server with the latest code. Because deployments to Production took hours and were error-prone, the application I was responsible for deployed new functionality and most bug fixes to Production once a quarter. Those deployments were always off-hours.

Integration was slightly better; we could deploy to that in the middle of the day, but it was still just as error-prone. We didn’t want to deal with that headache. Integration was often updated only days before a Production deployment. As a result, Integration and Production were almost always running the same version. This created a clear hotfix path for us.

Generally, but not always, hotfixes could be placed into one of four buckets. In each case, we could run the proposed fix in Integration before Production.

  • Fix the issue by changing a record in the database to "trick" the code into the right path.
  • Solve the problem by making a change to the database schema or a stored procedure. Sometimes it was as simple as adding an index.
  • Make a configuration change, either in a web.config file or the operating system host.
  • If it required a code fix, we could configure the build server to create the build artifacts but not deploy to Development and QA. We’d copy the artifacts to the "hot folders" for a deployment to Integration.

The common workflow when a bug was reported was for the on-call engineer to triage it. Once they isolated the root cause, they’d pair with a DBA or Web Admin until the issue was resolved. While issues were resolved, it led to other problems.

  1. Database schema hotfix changes were rarely "copied down" to lower environments. The delta in the database schema between Dev/QA and Integration/Production was significant.
  2. Configuration changes sometimes didn’t make it into version control. The next deployment to Integration and Production would often overwrite the hotfix, requiring another hotfix.
  3. Because of the once-a-quarter releases, finding an appropriate point in time in version control that just had what was in Integration/Production took a lot of effort.

Impact of automation on hotfix pipelines

Automating a terrible process doesn’t make it any less terrible. The core problems must be addressed. Our core problem was two different processes to deploy software from Development through to Production. Those two processes were required because the build server was used for deployments and shared application database. The build server lacked the appropriate RBAC and configuration controls to satisfy security and audit requirements. The shared database encouraged the merging of unfinished code.

Using a combination of local database development, git, Redgate's tooling, Octopus Deploy, and Azure DevOps (or Visual Studio Team Services as it was called back then), we rebuilt the entire deployment pipeline.

  • All changes, code, and database are made in a branch.
  • Changes are merged to the main using a Pull Request.
  • Azure DevOps / VSTS builds the code and packages database changes.
  • Octopus + Redgate deploys those build artifacts to Development.
  • Automated Verification, then Octopus + Redgate, promotes those artifacts to QA.
  • QA verifies.
  • Promote those artifacts to Integration using Octopus + Redgate.
  • Final sign-off.
  • Promote those artifacts to Production using Octopus + Redgate.

Rebuilding the deployment pipeline had numerous benefits.

  1. Consistent deployments: No more surprises when deploying to Integration and Production, as it was the same process for Development and QA
  2. Eliminated post-deployment emergency fixes: Automation stopped the dumb mistakes that led to many hotfixes.
  3. Reduced manual work: No one had to copy files to specific network folders and write up instructions.
  4. Increased release cadence: Production deployments increased from once a quarter to once a week.
  5. Smaller changeset: Instead of dozens or 100s of changes, each deployment had fewer than a dozen. That meant there was a lower chance that something could go wrong.

The most surprising thing to us was the more tolerant users. When it took two or three months to get a bug fix into production, users would classify most bugs as high/must-fix because they couldn’t wait that long. With the new pipeline, unless the bug was critical, most users were willing to wait a day or two for a fix. With smaller change sets, the number of critical bugs dropped significantly.

While we were deploying once a week, we still had major features that would take over a month to develop. All that work would occur in a branch. When we merged in those major changes, it would sometimes take a week or two to fully test it (our actual average was deploying every 10 days). During that time, there was "no clear path to production" for hotfixes.

  • For major features, we’d often do a few deployments to Integration as a final test with Production-like data.
  • Once major features were pushed to Integration, there was no easy way to roll back those changes.
  • We used a Service-Oriented Architecture (SOA) to achieve loose coupling, but there was still some coupling. Whenever anyone rolled back Integration to match Production, other applications would break because they were expecting a specific version.

Because we knew there was a chance of bugs after deploying a major feature to Production, we would implement a merge freeze, except for critical bugs, for a week after the Production deployment.

The merge freeze was the best solution we could come up with at the time. We looked into creating a hotfix pipeline, but with deployments to Production occurring once a week (or so), we kept running into the same roadblocks:

  • How do we qualify a bug to justify using the hotfix pipeline?
  • What if a pending change is in Integration?
  • How do we configure the build server to tell Octopus to push to Integration instead of Development?
  • How do we know which branch would be in Production when another bug occurred with a hotfix pipeline?

The decision tree of if/then/elseif/else for when to use a hotfix pipeline became extremely confusing. In addition, fewer critical bugs were released. The work to create a hotfix pipeline for my team was de-prioritized. Until I left, I never saw that becoming a priority. Every once in a while, when we had a merge freeze, someone would bring it up. But it was more of a passing comment. Not a dictate to make a change.

Hotfixes don’t fit in modern software delivery

The story from above took place over 10 years ago. It involved deploying .NET applications to static environments hosted by Windows Servers. There were many Continuous Delivery techniques we didn’t implement at the time. In addition, many technologies and techniques have been introduced.

Below are the core principles every software delivery pipeline should follow. See Continuous Delivery and Achieving Continuous Delivery with TPF, and Trunk-Based Development for more details.

  • The main branch must always be in a deployable state.
  • Create the build artifacts from the main branch once and promote them through the necessary environments to production.
  • How you deploy to Production should be exactly the same as how you deploy to Development, Testing/QA, and Staging/Integration.
  • Automate as much testing as possible, including unit testing, integration testing, and post-deployment smoke and soak tests.
  • Store environment configuration in version control and use that to keep all environments similar. Compute resources and external access can differ by environment.
  • Separate deploying new code from releasing new functionality by using feature flags.

With those core principles in mind, the delivery pipeline is the following:

  1. Make any changes in a short-lived branch.
  2. Checking in changes to a branch creates a pre-release artifact and deploys to an ephemeral environment or a static Development environment.
  3. Merge those changes into the main branch via a pull request.
  4. The pull request should also be verified on an ephemeral environment or a static PR environment.
  5. After merging into the main branch, create the release artifacts.
  6. Promote those release artifacts through any static testing environments (Test, QA, Staging, Pre-Production, etc.) to Production.
  7. Once a feature is ready, enable the new functionality for a subset of users in Production via feature flags. Start with internal teams, then slowly add users until you have enabled it for all users.

:::figure

:img{ src="/blog/img/hotfix-myth/branching-diagram-with-ephemeral-environments.png" alt="Diagram demonstrating when ephemeral environments will be used in a trunk-based or GitHub Flow based branching strategy" loading="lazy" }

:::

With the appropriate guardrails around the main Production deployment pipeline, a hotfix pipeline for just Staging → Production raises a lot of questions.

  • Branching and Deploying
    • Will the hotfix branch use an ephemeral environment for testing before going to Staging?
    • How will the build server know to skip the Test environment and move to Staging → Production?
    • What will the version number be for the hotfix release? If Production is 2026.8.1, does that mean the hotfix is 2026.8.1-Hotfix, 2026.8.1.1, or 2026.8.1.1-hotfix?
    • The main branch is supposed to represent production. How will the appropriate hotfix be communicated to the rest of the engineering team?
    • When will the hotfix changes merge into the main branch? How much of a delta is there between what is in the main branch and production? Can the fix even be merged into the main branch without serious modifications?
  • Testing and Risk
    • What is preventing the main branch from being deployed to Production?
    • If there is new functionality that’ll likely have many edge cases and potentially show-stopping bugs, why wasn’t it behind a feature toggle?
    • Were there changes already in Staging that were overwritten by the hotfix? Will that impact other teams or applications?
    • What steps and tests are being skipped in the Test environment?
    • How much time is really being saved by skipping the Test environment?
    • What if the hotfix requires a hotfix? How long is it acceptable to block the normal pipeline from deploying to Staging → Production?
    • How often is the hotfix pipeline tested and verified?

Hotfix pipelines no longer make sense

All of the challenges listed above are solvable with enough time and money. But to quote Ian Malcolm from Jurassic Park, "Your scientists were so preoccupied with whether they could, they didn't stop to think if they should."

To put it bluntly:

  • Is it worth spending time to work through all those issues to create a hotfix pipeline?
  • How often do you need to push a hotfix?
  • If it is a regular occurrence, is the hotfix process masking a suboptimal process like the one I described earlier?

The time spent creating a hotfix pipeline is better spent making the primary software delivery pipeline as efficient as possible. A good goal is to take less than an hour from pull request acceptance to being ready to deploy to Production. That includes builds, testing, linting, scanning, deploying to lower environments, and verification.

Achieving that requires addressing some hard problems:

  • If the main branch is regularly in an undeployable state, what testing and verification should be moved earlier in the pipeline to ensure it is always in a deployable state?
  • Is the branching strategy Trunk Based Development or GitHub Flow (not to be confused with GitFlow)? If not, why not? Even the creator of GitFlow has said not to use it for most applications.
  • Showstopping bugs, ones that typically require a hotfix, are typically the result of new features and functionality. How can Feature Flags be introduced into the pipeline to separate deploying new code from releasing new features?
  • If manual review and approval processes are the primary bottleneck, which steps in that process are prime candidates for automation to speed up approvals?

The primary advantage of focusing on the items above is that they have a net positive for any change. New features, security patches, and bug fixes will be deployable faster.

**Disclaimer:*- The one hour is a goal to aim for. It isn’t a hard rule. One hour for some applications, like monoliths, is impossible. That doesn’t mean you shouldn’t try. Small improvements add up over time. Improving a monolith's pipeline from one day to two hours is a huge accomplishment.

Configuring Octopus Deploy

By this point, you might be asking yourself, how does this impact my configuration of Octopus Deploy?

Unfortunately, lifecycles (and channels) are among the most misconfigured constructs within Octopus Deploy. To make onboarding easier, the default lifecycle is built using conventions. If you were to create Development, Test, Staging, and Production environments, the default lifecycle automatically becomes:

  • Development → Test → Staging → Production

That default lifecycle encourages bad behavior, necessitating a hotfix lifecycle.

  • It doesn’t represent how developers work in branches. Because it includes Production, it has to represent the main branch. But it also includes Development. To get feedback, developers are forced to use their local machine or merge unfinished changes into main.
  • With unfinished changes in the main branch, it is currently undeployable. It could remain there for days and sometimes weeks.
  • Enforcing SemVer versioning rules becomes nearly impossible. For example, releases that are not ready for Production typically receive a pre-release tag because the lifecycle includes Development and Production; that rule cannot exist.
  • There isn’t a clear path to push a fix to Production. To prevent that from happening, teams will implement merge freezes for a period of time after a major release (like I did at a previous job). Or create a hotfix lifecycle that skips Development and Test.

The root cause of that is having the static Development environment included in the same lifecycle as Production. Development is for unfinished changes, Production is for finished changes. A Production lifecycle must never include Development.

If a static Development environment is required, my recommended lifecycles are:

  • Default: Development
  • Release: Test → Staging → Production

:::figure

:img{ src="/blog/img/hotfix-myth/recommended-octopus-lifecycles.png" alt="Screenshot of Octopus Deploy interface showing the recommended lifecycles of default and release." loading="lazy" }

:::

**Disclaimer:*- The release lifecycle should include all static testing environments required to reach Production. You might only need Test → Production, or Staging → Production. I included Test → Staging → Production because, as an industry, we have coalesced around four environments.

The subsequent Project Channels are:

  • Default (uses the default lifecycle or an ephemeral environment): build artifacts require a pre-release tag and can only be created from non-main branches.
  • Release (uses release lifecycle): build artifacts cannot have a pre-release tag and can only come from the main branch.

:::figure

:img{ src="/blog/img/hotfix-myth/recommended-octopus-channels.png" alt="Screenshot of Octopus Deploy interface showing the recommended default and release channels for a specific project." loading="lazy" }

:::

GitHub Actions (or really any build server) doesn’t make dynamically selecting channels based on branches any easier. They require using a hard-to-decipher if/then/else command in the build definition. For example, ${{ github.ref == 'refs/heads/main' && vars.OCTOPUS_RELEASE_CHANNEL || vars.OCTOPUS_DEFAULT_CHANNEL }}.

Running multiple versions in Production

Occasionally, REST APIs and other backend services must run multiple versions in Production for backward compatibility. For the recommendations below, my example application has three versions: v1.x, v2.x, and v3.x.

  • The main branch represents the latest version (v3.x)
  • Separate branches for each version (v1.x and v2.x)
  • Each version branch is treated like a "trunk"
    • Changes are made in short-lived branches that were branched off the version branch.
    • Merging into those version branches requires a pull request.

Within Octopus, you’ll only need one lifecycle:

  • Release: Test → Staging → Production

But the Project will have four Channels:

  • Default
    • Uses an ephemeral environment
    • Build artifacts require a pre-release tag and can only come from non-version or main branches.
  • vCurrent
    • Uses release lifecycle
    • Build artifacts cannot have a pre-release tag
    • Build artifacts must come from the main branch
    • Build artifacts version must be <= 3.x
  • V2
    • Uses release lifecycle
    • Build artifacts cannot have a pre-release tag
    • Build artifacts must come from the v2 branch
    • Build artifacts version must be between 2 and 2.999999
  • V1
    • Uses release lifecycle
    • Build artifacts cannot have a pre-release tag
    • Build artifacts must come from the v1 branch
    • Build artifacts version must be between 1 and 1.999999

Ephemeral environments make this significantly easier, as you can spin up a sandbox for a change for any version and verify it before merging into the appropriate branch. If you cannot use ephemeral environments, I’d recommend setting up a couple of static development environments and configuring a lifecycle that lets you deploy to any of them. The downside is that a person must determine which static development environment to use.

Conclusion

A Production incident is not the time to improvise a deployment pipeline. Steps shouldn’t be skipped to "go faster." All too often, a "simple change" that isn’t properly vetted causes a bigger issue. But that is essentially what a hotfix pipeline is designed to do. It skips important steps in the normal Production deployment pipeline to save time. To resolve a Production incident, you want a well-tested and well-used pipeline, so you know you aren’t introducing even more risk. The time required to create and improve a hotfix pipeline is better spent improving the Production deployment pipeline. Once the Production deployment pipeline takes less than an hour to be ready for deployment to Production, the need for a hotfix pipeline will be all but eliminated.

Read the whole story
alvinashcraft
just a second ago
reply
Pennsylvania, USA
Share this story
Delete

Your agent’s guardrails have a bypass

1 Share

The deployments are real now: agents with tools, credentials, and the autonomy to act on an organization’s behalf. Every agent carries the same requirement: policies must be enforced, approvals must actually gate actions, and there must be evidence of what happened. And every framework answers that requirement differently, with mechanisms that were designed for observability, not governance. As agents grow more capable, their execution paths multiply: subagents, retries, batch entry points, background tasks. The governance question moves from “did we write a guardrail?” to “is it enforced on every path, and can we prove it?” 

Here is an example of this gap in practice. 

A team builds a customer support agent. It can look up accounts, draft replies, and issue refunds. Compliance sets two rules: refunds above a threshold need human approval, and account data never reaches the reply channel unredacted. The team does what their framework documentation suggests, adding a guardrail callback on tool calls and tool outputs. 

While performing quarter-end financial closing, it was discovered that there is a large gap exceeding the discretionary customer refund budget. The system has been issuing refunds erroneously. The incident review finds three things. The approval guard threw an exception on a malformed refund request; the framework dispatcher caught the error, logged a warning, and executed the refund anyway, which is its documented default. The output scanner never saw one egress path, because a batch entry point emitted no callback at all: the guard was attached to the interactive path, and the batch path simply never fired it. And nobody could produce evidence of what either guard actually evaluated, because callbacks observed values without recording anything that binds them to what executed. 

Nothing here is exotic. The team followed the instructions. The instructions were the problem.

The failure class: enforcement attached to one path, while the runtime grows paths. The fix is structural, not more callbacks.
The failure class: enforcement attached to one path, while the runtime grows paths. The fix is structural, not more callbacks.

The hooks you have are not a governance surface

We catalogued the interception surfaces of the mainstream agent frameworks from their primary documentation and source. LangChain’s BaseCallbackHandler defines 20 lifecycle events, and the dispatcher discards handler return values, so a callback cannot block or rewrite anything; a handler exception is caught and swallowed unless the author opts into raise_error, which defaults to false. CrewAI’s event bus registers 78 typed event kinds, all observe-only. LlamaIndex’s instrumentation module is telemetry by design: no return value, exception, or mutation reaches the underlying action. The OpenAI Agents SDK exposes lifecycle hooks that observe and guardrails that can block, but its input guardrails race the first model call unless you set a flag. Semantic Kernel’s filters genuinely block; but when registered through dependency injection, execution order is documented as not guaranteed and ordering decides whether redaction runs before egress. 

Count the lifecycle surfaces alone: LangChain exposes 20 callback events, CrewAI 78, the OpenAI Agents SDK seven, and Semantic Kernel three, while LlamaIndex ships two coexisting observability surfaces. Payloads range from untyped dictionaries to typed contexts. Control semantics range from none to full block-and-modify. Failure behavior ranges from silently swallowed to propagated. And not one of these frameworks ships a conformance suite that a controls author can run to verify that a deny stops the action. Every guarantee your control depends on is framework-specific folklore.

Why builders should care

If you run agents in production, you inherit two problems: controls behave differently across frameworks, and you cannot reliably prove that they ran. 

Control builders must create and maintain a separate adapter for each framework. Worse, each framework answers the most important question differently: when a control denies an action, does the action always stop? 

Framework builders face the other side of the same problem. Enterprise customers need approvals, policy checks, audit records, and data controls, so each framework must build and maintain these features itself. A shared contract changes the practical outcomes for all three: write controls once and reuse them across frameworks, verify enforcement instead of assuming it, get audit evidence by construction, and stop paying the per-framework integration tax on every governance requirement.

Agent Hooks: One governance contract for the ecosystem, testable on both sides

Today we’re publishing Agent Hooks, specified as AGENT-HOOKS-0.1: an open, framework-neutral governance contract for AI agents, and a common interoperability layer that any framework can implement and any control can target. It ships with SDKs in Python, TypeScript, .NET, Rust, and Go, a 47-scenario conformance kit that makes “supported” a testable claim, and a first-class implementation merged into Microsoft Agent Framework’s core. The contract is deliberately small: eight interception points that bracket the agent loop, one context payload, three verdicts, and normative obligations on the host. Controls integrate against it once. Frameworks implement it once. The M×N adapter matrix becomes M+N.

Twenty bespoke adapters, or one contract with a conformance kit on each side of it.
Twenty bespoke adapters, or one contract with a conformance kit on each side of it.

What it looks like

An interceptor is a few lines. This one enforces the refund rule from the opening incident:

from agent_hooks import Interceptor, Verdict class RefundGuard(Interceptor): def intercept(self, context) -> Verdict: if context["interception_point"] != "pre_tool_call": return Verdict.allow() call = context["tool_call"] if call["name"] == "issue_refund" and call["args"]["amount"] > 500: return Verdict.escalate(reason="refund_over_limit", message="requires human approval") return Verdict.allow()

And installing the full contract in Microsoft Agent Framework is one factory call:

pip install agent-framework-core[agent-hooks] agent = Agent(client=client, tools=[issue_refund], middleware=[create_agent_hooks_middleware([RefundGuard()])])

That single call installs enforcement at every point of the loop: it is deliberately impossible to install part of the contract and believe you have all of it.

How it works: Emitting and enforcing

The eight points bracket the loop: agent_startup, input, pre_model_call, post_model_call, pre_tool_call, post_tool_call, output, agent_shutdown. At each, the host builds an AgentContext, a tiered JSON payload with a small required core (agent, session, sequence, timestamp, and the target under evaluation), per-point required fields, and namespaced extensions. The target is the one value a transform may rewrite, pinned per point. 

Interceptors return a verdict with one of three decisions; allow, deny, transform. On the wire, an escalation looks like this:

{ "decision": "deny", "reason": "refund_over_limit", "message": "requires human approval", "approval": { "resolver": "host", "context_identity": "sha256:11f8bab5…" } }

In earlier iterations, we also had warn and escalate verdicts. Warn verdict was removed since a warning is an allow carrying warnings, because warning is metadata, not control flow. An escalation is now modeled as a deny carrying an approval block, denied as-is unless the approval seam lifts it. That construction removes a failure mode outright. An unresolved escalation used to be a state the host had to remember not to proceed on; now it is simply a deny. Fail-closed is a property of the type system, not a code path someone has to maintain. 

Host obligations are the half that frameworks usually leave undefined, and they are normative here. A deny at pre_tool_call means the tool is not invoked. A deny at post_tool_call means the result is discarded and never enters agent state. A host that cannot build a valid context, cannot reach an interceptor, times one out, or receives a malformed verdict must synthesize a deny with a reserved machine-readable reason. The opening incident cannot occur on a conformant host: the crashing guard becomes a deny, and the record says so. 

Every emission also produces an InterceptionRecord, which is payload-free by design. It carries the verdict projection, which interceptor decided, the composition profile, the sequence number, and content identities computed before and after enforcement. It never carries customer content. You can export the full audit trail of an agent’s decisions without exporting a single prompt.

The approval that can’t be replayed

The approval block above carries a context_identity: a SHA-256 over the canonical JSON of exactly what the approver was shown. The resolution must echo that identity byte-for-byte. This is the difference between approving an action and approving a session.

Approval binds to content, not to a session. Change the content, and the approval does not transfer.
Approval binds to content, not to a session. Change the content, and the approval does not transfer.

In our demo suite, a support agent tries to refund $840 against a $500 cap. The guard escalates; a human sees the exact call (tool, arguments, identity) and approves. The refund executes, and the record binds the approval to that identity. Then the demo replays the approval against a mutated call: issue_refund for $8,400, claiming the earlier authorization. Different content, different identity, and the deny stands. We run this same scenario across eight frameworks: LangGraph, the OpenAI Agents SDK, Microsoft Agent Framework, Semantic Kernel, LlamaIndex, CrewAI, the Claude Agent SDK, and a bare reference host. Every one of them produces the identical 20-row decision stream, with the identity probe byte-identical across the Python, .NET, and TypeScript SDKs. One contract, one behavior, provable.

Doesn’t a hook layer slow everything down?

The enforcement seam itself is cheap. During the Microsoft Agent Framework integration review, the per-run overhead of the contract’s machinery (identity allocation, gate consultation, scope management) was measured at roughly a microsecond per run and a microsecond per streamed update on commodity hardware: noise against any model call. The honest cost lives elsewhere, and we will name it: fully fail-closed streaming means buffering. A host that guarantees no token egress before the output verdict cannot also give you first-token latency during enforcement. The spec supports a declared bounded-exposure incremental mode for hosts that need streaming, with the exposure bound stated in the conformance claim; buffered is the default, because it is the only mode with zero exposure. 

Policy evaluation is also cheap when it is in-process. The first policy runtime built on the contract originally shelled out to an external policy engine per decision: 26.8 milliseconds per evaluation, dominated by process spawn. Moving Rego evaluation in-process brought a warm evaluation to 0.32 milliseconds, measured as best-of-five over the same policy pack on the same hardware, with activation amortized after roughly twenty decisions. Your policy engine shouldn’t be the slow part of your agent, and it doesn’t have to be.

What Agent Hooks doesn’t protect against

Agent Hooks is a cooperative contract, not a security boundary. The host framework is fully trusted: interceptors run in-process with full data access, and registering an interceptor is equivalent to granting it write access to every action the agent takes. A hostile or buggy host can skip points or ignore verdicts, and the conformance kit can’t detect that. There’s no complete-mediation claim: a framework may expose direct tool execution or background paths that never reach pre_tool_call, so the contract makes coverage testable, not automatic. Server-side tool execution (hosted code interpreters, service-managed tools) can’t be intercepted at the tool seam at all; it surfaces at post_model_call, and conformant hosts document exactly that. The threat model in the spec says all of this in normative language. A hook layer governs what a cooperating framework does; containing hostile or untrusted code is a sandbox’s job, and Agent Hooks isn’t a sandbox. If someone tells you otherwise, they are selling something.

Proof it survives contact

A contract is worth what its enforcement survives. The Microsoft Agent Framework integration went through five maintainer review rounds, and the maintainers didn’t take our claims on faith: they reproduced real fail-open paths, including a retrying middleware that defeated persistence ownership and a drained-and-discarded attempt that persisted before any verdict existed. Every finding was fixed with a regression test that fails when the fix is reverted. We’ll be honest: the hardest part of this project wasn’t writing the spec; it was watching skilled reviewers falsify our own “this is fail-closed” claims, twice, and rebuilding until they couldn’t. 

That discipline is what the conformance kit packages. Forty-seven scripted scenarios drive a host through the contract: denies that must stop actions, transforms that must be applied, crashes that must become denies, approvals that must bind to content. A conformance claim is a declared surface plus the report. There is no certification theater, just results you can re-run. Two hosts hold certified claims today: the Agent Control Specification policy runtime and Microsoft Agent Framework’s core implementation, which passes 47 of 47 applicable scenarios and discloses its one non-default behavior (terminating the run on enforcement-layer failure) as a declared posture rather than papering it over. The .NET implementation is in review with the same test discipline: 84 tests, every enforcement property pinned.

Try it

pip install agent-hooks-sdk # Python npm install @responsibleai/agent-hooks # TypeScript (napi, prebuilt) cargo add agent-hooks-sdk # Rust dotnet add package ResponsibleAI.AgentHooks # .NET go get github.com/responsibleai/agent-hooks/sdk/go/agenthooks

The spec, the conformance kit, the documentation, and all five SDKs live in the agent-hooks repository at github.com/responsibleai/agent-hooks. If you’re on Microsoft Agent Framework, the feature is in core behind the agent-hooks extra today. The eight-framework demo suite, including the replay scenario above, ships with runnable, deterministic scripts that need no API keys.

An open contract needs more than one author

AGENT-HOOKS-0.1 is versioned, the schemas are published, changes go through public proposals, and the conformance kit is the arbiter of what “supported” means. We built reference implementations in five languages so that no single runtime defines the contract, and we encourage other implementations. If you maintain a framework and want the conformance report with your name on it, the harness interface is four methods, and we’ll do the integration work with you. If you build controls and the contract is missing a seam you need, the proposal process is open. We’d love to hear from both sides.

FAQs

Why not just use each framework’s middleware?

Because middleware answers “where can code run,” not “what must happen when it says no.” The contract’s value is the normative half: deny stops the action, crashes become denies, approvals bind to content, records are payload-free. Middleware is how hosts implement it; the contract is what makes the result verifiable.

Three verdicts seems small. Where are warn and escalate?

They’re encoded, deliberately: warn is allow plus warnings; escalate is deny plus a liftable approval. Five verdicts means five states hosts can mishandle; three with fail-closed composition means an unresolved anything is a deny.

What happens under streaming?

By default, everything buffers until the output verdict: zero exposure, at an honest latency cost. Hosts that need incremental release declare a bounded-exposure mode in their conformance surface, with the bound stated. What no conformant host may do is stream first and enforce later while claiming otherwise.

Can a malicious host just lie?

Yes. See the threat model section: the host is trusted, and this is a contract, not a sandbox. What the contract changes is that a cooperative host’s claims become testable, and a gap becomes a conformance finding instead of an incident.

Is this Microsoft only?

No. The spec and SDKs are MIT-licensed under an open organization; the first certified consumer is an independent policy runtime, and the same scenario suite runs on eight frameworks from six vendors. Microsoft Agent Framework is the first framework to ship it in core; the contract is written, so it will not be the last.

The post Your agent’s guardrails have a bypass appeared first on Command Line.

Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete

How Uno Platform uses .NET, MCP, and AI to build high quality apps

1 Share

This is a guest post by Sam Basu. Sam is a technologist, author, speaker, Microsoft MVP and Developer Advocate for Uno Platform.

If you’ve spent any time building software in the last couple of years, you’ve felt the shift. AI is no longer a novelty sitting on the sidelines – it’s right there in the editor, the terminal, the build pipeline. And for .NET developers, this moment is particularly exciting. The ecosystem is deep, the tooling is stellar, and AI just keeps getting better at navigating both.

But raw AI power and grounded, contextual AI are two very different things. An AI agent will happily write you a settings page for a cross-platform .NET app. It will compile. It will pass review if you only read it. And it will still be wrong in ways you cannot see until the app is running in front of you. Closing that gap is the problem we set out to solve at Uno Platform – you can now build cross-platform .NET apps in browser with AI; give it a try @ https://platform.uno/.

The developers who will get the most out of this era aren’t the ones prompting the hardest – they’re the ones giving AI the right context to actually do the job well. The focus is on quality – how can we provide AI all the guardrails to be successful and be able to validate its own work, and tooling that makes .NET developers productive from the start. Let’s unpack.

Why MCP, and why we ended up with two servers

The obvious first move is context stuffing: shovel the docs into the prompt, add a long instructions file, hope for the best. It fails for a reason that is clear in hindsight. Documentation is large, the useful slice is small and query-dependent, and no amount of prompt real estate substitutes for the agent being able to look something up at the moment it has the question.

Model Context Protocol solves the lookup problem. It does not solve the verification problem. Knowing what the API should be does not tell an agent whether the layout it just wrote actually renders. Those are two different jobs with two different lifetimes, and that distinction is why we ended up with two servers rather than one.

The split we landed on maps to those two lifetimes.

The docs server: grounding

The docs server is publicly hosted at https://mcp.platform.uno/v1, speaks HTTP, and is stateless. It answers what is true about this framework right now – a question whose answer changes when we ship, not when your app runs.

  • uno_platform_docs_search – search official documentation and return the most relevant results
  • uno_platform_docs_fetch – fetch a full documentation page as markdown
  • uno_platform_agent_rules_init – initialize the agent session with rules for working against a running app
  • uno_platform_usage_rules_init – load common API usage rules

It also ships two prompts: /new to scaffold an app with current best practices, and /init to prime an existing conversation before adding a feature to an existing codebase.

The design property that matters is that this server is versioned with our documentation, not with the developer’s SDK. Correct a doc page and every agent everywhere gets the correction on its next call. That is a very different maintenance story from shipping guidance inside a NuGet package, and it is the main reason we host it rather than distribute it.

The app server: eyes and hands

The app server is the opposite in every dimension. It ships as a .NET tool launched over stdio, runs on the developer’s machine as a bridge to the Uno DevServer, is stateful, and belongs to exactly one session. It answers what is actually happening right now.

It gives an agent four capabilities. It can run the app – uno_app_start launches in debug mode with Hot Reload enabled, so the agent controls the whole lifecycle rather than waiting for a human to press F5. It can seeuno_app_get_screenshot for pixels, and uno_app_visualtree_snapshot for an XML snapshot of the visual tree. It can actuno_app_pointer_click, uno_app_key_press, uno_app_type_text, and uno_app_element_peer_action to invoke automation peers directly. And it can check itselfuno_health reports the status of the bridge and its connection, because an agent that cannot tell “the app is broken” from “my connection dropped” will confidently debug the wrong thing.

MCP tool list showing Uno app and docs servers registered in an IDE

Both servers, side by side, as the agent sees them.

The visual tree tool is the one that earns its keep. Screenshots tell a model that something looks wrong; the XML tree tells it which element is at fault and what its properties are. Pixels are for detection, structure is for diagnosis, and an agent needs both.

There is one detail in that tool list worth calling out: read the description on uno_app_pointer_click and it says prefer uno_app_element_peer_action. That preference lives in the tool description itself rather than in documentation nobody loads, because coordinate clicking is brittle across window sizes and DPI while automation peers are stable. More on why that matters below.

Building it: the MCP C# SDK in production

Both servers are written in C# on the official MCP C# SDK, which Microsoft maintains in collaboration with the community. Two things we would tell any .NET team starting the same work.

Pick your transport from your topology. The docs server is HTTP because it is a hosted multi-tenant service that needs OAuth. The app server is stdio because it is a child process on one developer’s machine talking to one running app. The topology decides the transport; there is not much of a choice to agonize over once you have written the constraints down.

Your tool definitions are a permanent tax on the context window. Every tool name, description, and input schema is loaded before the model does any work. Our docs server costs about 6.4k tokens and the app server about 1.5k – for comparison, the built-in GitHub MCP server in the same session costs about 5.2k. That is real budget spent before a single question is answered, and it is why terse, high-signal tool descriptions are not a style preference.

Copilot CLI listing MCP servers with transport, token cost, and auth

The same servers in GitHub Copilot CLI. Note the token cost per server.

That second point has a corollary: tool descriptions are prompts, not documentation. A tool the model never selects may as well not exist, and the only lever you have over selection is the wording. This is why uno_app_pointer_click explicitly tells the model to prefer the automation-peer tool instead – that is not documenting a preference, it is steering a decision at the moment it is made.

Generating code and functionality verification are different problems

Here is the hot take: AI can write UI code faster than any human team, and it cannot tell whether what it wrote is correct. As agentic workflows become normal, that asymmetry is the bottleneck. Generation got cheap. Verification did not.

Web developers already solved their half of this. Playwright drives a real browser, so an agent working on a web app can check its own work. There has been no equivalent for a native cross-platform .NET app running on Windows, macOS, Linux, iOS, Android, or WebAssembly – the app is a black box the moment it launches.

The app server is our answer to that: Playwright-style UI automation for .NET apps. The agent writes a change, the app hot reloads, the agent takes a screenshot, reads the visual tree, clicks through the flow, and decides for itself whether the change did what was asked. When it did not, the agent fixes it before handing anything back.

Code is cheap. Software is not. This is how you hold both truths at once.

Skills: giving the agent the “how”

MCP tools give an agent the what. They do not say when to reach for which one, or in what order, or what “done” looks like. That is what Skills are for.

The cooking analogy holds up well here. MCP tools are ingredients – atomic, each does one thing. Skills are recipe cards – the reusable instructions for combining ingredients into something worth eating. The agent is the cook, choosing a recipe and adapting it to what is actually in the kitchen.

Our Skills library is organized by the thing you are actually doing: MVUX state and feeds, navigation, theming, the Uno Toolkit controls, and testing. The one that closes the loop is uno-testing-ui, which automates UI testing through the app server – the Skill knows the order to drive the tools in, so the agent does not have to work it out from first principles every session.

Grounded documentation, a live app it can inspect, and curated procedure for the workflows that matter: that combination is what we mean by contextual AI.

Uno Platform Skills listed as toggleable plugins inside an AI agent session

Skills install as plugins, available to any MCP-compatible agent.

What it adds up to

The most interesting thing we built with all of this is not a feature list, it is a compiler running where a compiler has no business running.

Uno Platform Studio 3.0 generates a full cross-platform .NET app entirely in the browser. Behind the prompt box, a specialized agent orchestrated by Microsoft Agent Framework plans and executes the work across parallel steps and multi-turn conversations. A full Roslyn workspace then compiles what the agent writes, loads the generated assemblies, resolves NuGet changes, and hot reloads the result into the running app – all in the browser, while you watch. The docs server keeps the agent’s knowledge current. The app server lets it check its own work. The Skills keep it on the rails.

That is Roslyn, Microsoft Agent Framework, and the MCP C# SDK doing work that would have been a research project a few years ago, and the entire stack is .NET.

Uno Platform Studio generating a CRM dashboard app in a browser with an agent panel

Prompt on the right, compiled and running .NET app on the left. Not a mockup.

The practical consequence for a team is that the agent stops being a fast typist. It knows your design system, it validates its own output against a running app, and it follows workflows you chose. That is a different proposition from writing code faster.

The generated .NET app is fully interactive in the browser, along with page navigation and Previews to work on app UI in isolation. Developers can iterate on app UI with the Agent or manually with Hot Design in the browser – the changes are immediately visible with Hot Reload. There is no barrier to entry – developers can start in the browser, iterate on app UI with Agent or Hot Design, and drop down to local IDE/CLI with same tools, when ready.

Uno Platform Studio in edit mode

Why we work upstream

None of this would be buildable on a foundation we could not influence, and that is the honest reason we invest where we do.

We co-maintain SkiaSharp alongside Microsoft’s .NET team. SkiaSharp is the 2D graphics API underneath a large share of .NET charting, custom controls, and data visualization – it is built on Google’s Skia, the same engine in Chrome and Android – and it is what Uno Platform renders with. Becoming a co-maintainer formalized years of investment ahead of SkiaSharp 4.0, the largest release the project has had in years.

We also work directly on the .NET runtime through a formal collaboration with the Microsoft .NET team, contributing to .NET for Android and .NET for iOS bindings and to AOT in .NET 10.

The pattern is the same one this whole post describes: the further upstream you fix something, the more people never have to think about it again.

Wrap up

If you are building an MCP server for your own .NET stack, the two things we would pass along are these. Split your servers by lifetime, not by feature – knowledge that changes when you ship does not belong in the same process as state that changes when the app runs. And spend real time on your tool descriptions, because they are prompts, and a tool the model never selects may as well not exist.

The rest is ordinary .NET. The MCP C# SDK, Roslyn, Microsoft Agent Framework, and a graphics stack we help maintain, doing work that is anything but ordinary.

Try the Uno Platform MCP servers at aka.platform.uno/mcp.

The post How Uno Platform uses .NET, MCP, and AI to build high quality apps appeared first on .NET Blog.

Read the whole story
alvinashcraft
1 minute ago
reply
Pennsylvania, USA
Share this story
Delete

Global Secure Access – Now with Windows Update support!

1 Share

Last Updated on August 27, 2026 by Michael Morten Sonne Introduction An interesting improvement is making it´s way…

The post Global Secure Access – Now with Windows Update support! first appeared on Blog - Sonne´s Cloud.
Read the whole story
alvinashcraft
2 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

VSLive! @ Microsoft HQ: Developer Takeaways and Must-Watch Sessions

1 Share

VSLIVE MSHQ recap blog banner image

VSLive! @ Microsoft HQ 2026 was one of the most energizing VSLive! events I’ve attended. What stood out was the level of engagement over five days of learning and conversation from July 27 through July 31. Developers came ready to dig into the future of Visual Studio, .NET, and AI-assisted software development, and many Microsoft engineers and product managers were there to have those conversations directly. 

If you weren’t able to join us in Redmond, there’s still plenty you can take away from the event. The Microsoft-led sessions were recorded, so you can catch up on many of the topics that generated some of the biggest conversations during the week. 

AI is becoming part of the entire developer workflow 

If there was one theme that ran through VSLive! @ Microsoft HQ, it was AI. 

The conversation has moved well beyond code completion. Developers are thinking about how AI can help with planning, debugging, modernization, testing, code review, agents, and workflow automation. 

What I found especially important was how practical these conversations were. Developers want to know what works today, what can be trusted in real development environments, how teams can adopt AI responsibly, and where human judgment still matters. 

Great AI experiences still need great developer fundamentals 

One of the biggest takeaways from the week was that developers aren’t asking us to choose between AI innovation and the fundamentals of a great development environment. 

They want both. 

Speed, stability, predictability, debugging, testing, and usability still matter enormously. AI becomes more valuable when it builds on a development experience developers already trust. 

That reinforces why Visual Studio remains so relevant as development changes. Sessions covering GitHub Copilot in Visual Studio, debugging, productivity, MCP, and AI workflows showed how the IDE can increasingly serve as an orchestration point for modern software development while continuing to support the fundamentals developers depend on every day. 

.NET modernization is happening alongside AI adoption 

The excitement around AI doesn’t mean developers are leaving their existing applications behind. Quite the opposite. 

We saw strong interest in AI-enabled .NET applications, Aspire, Blazor, .NET MAUI, WinForms, SQL integration, and C#. But many developers are also starting from applications built on .NET Framework 4.7 or 4.8, Web Forms, WinForms, or codebases that have grown over many years. 

For those teams, the challenge is practical: How do you move toward .NET 8 or .NET 10 when time and staffing are limited and disruption needs to be kept under control? 

That makes modernization guidance and tooling an important part of the same conversation. Developers want to explore what’s next while continuing to improve the applications their businesses depend on today. 

VSLive! @ Microsoft HQ 2026 sessions: hot takes 

If you missed VSLive! @ Microsoft HQ, or attended and want to revisit what you learned, the recorded sessions are a great way to keep on learning. 

You can explore select VSLive HQ 2026 sessions on the Visual Studio YouTube channel, including these already trending videos: 

Improving Performance in .NET Applications

Performance remains one of the fundamentals developers care deeply about. If you’re looking to improve how your .NET applications perform, this is a good session to add to your watch list. 

Explore the Future of ASP.NET Core & Blazor in .NET 11

For developers building modern web applications, this session explores what’s ahead for ASP.NET Core and Blazor in .NET 11. 

Modernizing .NET Applications

Modernization was a strong, recurrent theme during VSLive! @ Microsoft HQ 2026. If you’re working with an established .NET codebase and thinking about what comes next, this session is especially relevant. 

Everything You Need to Know About the Latest in C#

C# continues to evolve alongside .NET. This session is a chance to catch up on the latest developments and think about how they apply to the applications you’re building. 

SQL MCP Server: Bringing AI Agents to Your SQL Data

This session brings together two themes that generated a lot of interest: AI and MCP. It’s a practical example of how developers are beginning to think beyond AI-assisted coding and toward agents that can work with the systems and data behind their applications. 

Whether you’re exploring AI, modernizing an existing application, improving performance, or staying current with .NET and C#, there’s a lot here you can bring back to your own projects and teams. 

Keep learning with VSLive! 

There are two more opportunities this year to connect with developers, learn from experts, and get hands-on with the technologies shaping modern development. 

VSLive! San Diego 

September 14-18, 2026 

Bahia Resort Hotel, San Diego, CA 

VSLive! San Diego is another opportunity to step away from the daily backlog, sharpen your skills, and spend time beachside in the warm Southern California sun with developers tackling many of the same challenges you are. 

VSLive! / Live! 360 Tech Con 2026 

November 15-20, 2026 

Royal Pacific Resort – Universal Orlando 

VSLive! / Live! 360 Tech Con 2026 offers another opportunity to close out the year with practical learning, technical conversations, and connections across the developer community. 

If you’re a Visual Studio subscriber and considering joining us at one of these events  Sign in to my.visualstudio.com to access your VSLive! discount code. 

Thanks to everyone who joined us in Redmond, asked questions, shared feedback, and spent time with our teams. Those conversations matter, and they help us shape what comes next. 

 

The post VSLive! @ Microsoft HQ: Developer Takeaways and Must-Watch Sessions appeared first on Visual Studio Blog.

Read the whole story
alvinashcraft
2 minutes ago
reply
Pennsylvania, USA
Share this story
Delete

What is cloud database vendor lock-in? The 3 types explained (and how to audit yours)

1 Share

Cloud database vendor lock-in is one of the most expensive surprises in enterprise infrastructure — not because the dependency is hidden, but because most teams never quantify it before committing to a platform. Each distinct type – engine lock-in, feature lock-in, and operational lock-in – accumulates differently, costs a different amount to exit, and requires a different conversation before you migrate.

This article breaks down exactly what each type means, which cloud providers create the deepest dependencies, and introduces a four-question framework for measuring your own lock-in — before a pricing change, service deprecation or organizational shift forces the issue.

Every time you move a database workload to a managed cloud service, you (and your team) are accepting some form of vendor dependency (‘lock-in’). However, what normally isn’t accepted is exactly what you are committed to, how deep it runs, and what it would actually cost to undo. It’s a conversation so many teams never have before committing to a cloud migration – and it’s an important, potentially costly, one to miss.

Why does this happen – and what are the consequences?

That conversation gets deferred because the managed service is genuinely useful, so there’s pressure to migrate to it. Plus, this vendor lock-in problem you’re hearing about seems so insignificant – maybe even hypothetical – next to the operational problem you and the team are currently struggling through.

However, it very much stops feeling hypothetical when your provider raises prices, changes a service behavior your application depended on or, worse still, the needs of your organization change enough that the cloud service is no longer required.

The key stat

Gartner’s 2025 Magic Quadrant for Cloud Database Management Systems (November 2025) found enterprise cloud database adoption accelerating, with distributed SQL among the most actively evaluated categories.

Redgate also reported similar findings in their special Cloud Migration Divide report (part of the 2026 State of the Database Landscape survey).

This article makes the lock-in conversation specific, explaining the three distinct types of cloud database dependency. Understanding which type you are accumulating, and how much of it, is the kind of clarity you need before making a big migration decision – not after, when it’s too late.

What are the 3 types of cloud database lock-in?

Engine lock-in, feature lock-in, and operational lock-in. These are the three distinct types of cloud database lock-in, but what do they each mean? What are the differences between them? Let’s find out.

Engine lock-in (the least dangerous type)

Engine lock-in means dependency on a specific database product, not on a specific vendor’s infrastructure. The migration is operationally painful – it involves downtime planning, data transfer, connection string updates, and validation. However, it’s technically straightforward, because the engine runs the same code wherever it runs. 

It’s also, in practice, often the least expensive to exit. If you’re running PostgreSQL on Amazon RDS, the database engine is standard, community-maintained PostgreSQL. Your schemas, stored procedures, queries, and application connection strings work against any PostgreSQL instance, anywhere.

So, if you decide to migrate to Azure Database for PostgreSQL, Google Cloud SQL for PostgreSQL, or a self-hosted instance, the technical barriers are low. 

The reason this matters less than it appears is that organizations rarely remain at engine-level dependency once they begin using a managed service. The managed service features that make it worth choosing over self-hosted are almost always provider-specific.

Feature lock-in (where the actual costs accumulate) 

Feature lock-in occurs when you build application behavior that depends on capabilities specific to one provider’s implementation of a database engine. The data is portable, but the behavior is not. 

Amazon Aurora is the clearest example. Aurora presents itself as MySQL-compatible and PostgreSQL-compatible, and at the protocol level, that is largely accurate. But Aurora’s underlying storage layer is not MySQL or PostgreSQL storage – it’s a distributed log-based storage system engineered by AWS that exists nowhere outside of AWS.

This architecture delivers genuine performance advantages: up to five times the throughput of standard MySQL on equivalent hardware, up to 15 read replicas (compared to five for standard RDS), and lower replication lag under write-heavy workloads. 

Those advantages come from Aurora-specific design. For example, Aurora’s exclusive backtrack capability, which lets you rewind the database to a previous point without restoring from a backup. Then there’s Aurora Global Database’s cross-region replication, which has different semantics from standard PostgreSQL streaming replication.

And finally, Aurora Serverless v2’s autoscaling behavior, which responds to traffic changes in fine-grained increments that have no equivalent on other platforms. 

The point is simple: if you build operational runbooks, application logic, or disaster recovery procedures that depend on any of these behaviors, you have feature lock-in. Migrating your data from Aurora to standard PostgreSQL is technically straightforward.

Replicating Aurora’s performance characteristics and operational behaviors on a different platform, however…that’s a different engineering project entirely,  and one that often takes longer than the data migration itself. 

Operational lock-in (the most underestimated type) 

Operational lock-in is the accumulation of integrations, monitoring configurations, automation scripts, IAM policies, VPC configurations, credential management approaches, alerting setups, and institutional knowledge that builds up around a specific managed service over time. It’s both the least discussed type and the most expensive to exit. 

Let’s take, for example, a database that has been running on Amazon RDS for three years. Typically, it’ll have CloudWatch dashboards tuned to its specific metrics, AWS IAM roles configured for access control, automated backup jobs defined in RDS parameter groups, performance insights reports that the team references daily, and staff who know how to interpret RDS console outputs.

None of that is the database itself. Instead, it’s all the operational context around the database – and crucially, it doesn’t transfer to another platform. When organizations underestimate migration costs, this is nearly always where the discrepancy comes from.

There’s also the time cost to consider. The data migration may have only taken two weeks, but the operational infrastructure rebuild takes three months. Retraining the team to work with a different management interface and toolset? That’s another two months. These costs are practically invisible until the migration is actually underway, so they rarely surface in initial vendor comparison exercises.

Simple Talk is brought to you by Redgate Software

Take control of your databases with the trusted Database DevOps solutions provider. Automate with confidence, scale securely, and unlock growth through AI.
Discover how Redgate can help you

Cloud vendors and lock-in: a guide

Let’s now look at some cloud vendors and how each handles lock-in. Here, we’re focusing on Amazon Aurora, Azure SQL Managed Instance, and Google Cloud Spanner.

Amazon Aurora 

Amazon Aurora genuinely delivers on its performance claims before it creates its dependencies. AWS announced Aurora DSQL at re:Invent 2024, positioning it as a serverless distributed SQL database with 99.999% multi-region availability and active-active architecture.

By Q1 2026, Aurora DSQL had reached general availability across four AWS regions, in direct competition with Google Cloud Spanner and CockroachDB Dedicated

The technical claims are substantiated. Aurora DSQL uses optimistic concurrency control rather than the traditional pessimistic locking approach, which eliminates lock contention in distributed write scenarios and reduces cross-region write latency compared to traditional multi-version concurrency control (MVCC) implementations. These are major benefits for organizations running globally (geographically) distributed applications with high write throughput.

The lock-in comes from what you give up to get these capabilities. Aurora DSQL’s documentation is explicit: the service doesn’t support explicit database locks (because it uses optimistic concurrency control), foreign keys, temporary tables, or certain PostgreSQL extensions.

Additionally, if your application contains SELECT FOR UPDATE statements or advisory locks, it needs to be refactored before using Aurora DSQL. Every application pattern you build to work around these constraints is application logic specific to Aurora DSQL’s behavior. 

AWS’s own documentation describes Aurora as creating ‘a different vendor lock-in by providing unmatched ROI.’ That’s a remarkably candid statement, but it’s true: the return on investment is real. In turn, so is the lock-in, which only deepens with every Aurora-specific feature you adopt.

Azure SQL Managed Instance

Azure SQL Managed Instance (MI) occupies a different position in the lock-in landscape, marketed on proprietary performance features rather than just compatibility. 

Microsoft’s documentation positions it as being nearly 100% compatible with on-premises SQL Server, designed specifically for lift-and-shift migration of existing workloads. That compatibility is real and meaningful for organizations migrating complex SQL Server environments to the cloud. 

The lock-in story here is ecosystem lock-in rather than feature lock-in. Instead of offering exotic capabilities that can’t be replicated elsewhere, Managed Instance integrates tightly with Microsoft Entra ID for authentication, Azure Monitor for observability, Azure Blob Storage for backup destinations, and the Azure networking stack for connectivity. For organizations already invested in the Microsoft ecosystem, this makes it a no-brainer to use MI.

There are also documented gaps between Managed Instance and on-premises SQL Server worth understanding before migration. Microsoft’s T-SQL differences documentation lists specific behaviors around linked servers, which in MI are limited to a small set of Azure targets and SQL Server instances, with no support for external file systems or other relational databases, backup and restore operations, certain replication scenarios, and distributed transactions. These gaps have narrowed with each release but remain relevant for complex on-premises environments. 

For organizations coming from SQL Server, Managed Instance is often the most viable cloud path because the alternatives require more extensive re-platforming. Said organizations just need to remember that choosing Managed Instance means committing to a deepening integration with the Azure services stack that surrounds it.

Google Cloud Spanner 

Google Cloud Spanner is a globally-distributed relational database built on Google’s proprietary TrueTime infrastructure, atomic clocks, and GPS receivers distributed across Google’s data centers. It represents the most complete form of cloud database lock-in in commercial use – and is quite honest about it.

Cloud Spanner provides external consistency across regions – delivering horizontal scaling with strong consistency without the consistency compromises that typically accompany distributed systems. 

The SQL dialect Spanner uses is not standard ANSI SQL. It diverges from PostgreSQL and MySQL on data types, functions, and query semantics in ways that make application code written for Spanner non-portable to other databases without meaningful rewriting.

Furthermore, Spanner doesn’t support foreign keys in the same way traditional RDBMS products do. Its pricing model, based on nodes or processing units, is Spanner-specific, so doesn’t align with other database services. Broadly speaking, organizations using Spanner know and accept this. The distributed consistency properties Spanner delivers aren’t available at the same operational simplicity as any other platform. 

For example, CockroachDB and YugabyteDB offer comparable consistency models but require more operational involvement. Similarily, Aurora DSQL is competitive for certain workloads but is newer, with more constraints. Spanner’s specific – and prominent – technical position is chosen by organizations choose because the alternative is building the distributed consistency infrastructure themselves. 

The organizations that run into problems with Spanner are the ones who adopted it for a workload that didn’t actually require its distributed consistency capabilities. Before realizing this, they’d already built application logic specific to the platform. The result is SQL dialect differences and the absence of standard foreign key behavior – issues that compound over time.

How do you measure cloud lock-in? The four-question framework you should use

Before adopting a cloud database service or auditing one you already rely on, these four questions make the dependency concrete enough to inform a real decision.

1. What does a complete data export look like – and can you run it today? 

For standard PostgreSQL on RDS, you can run pg_dump and restore the output anywhere PostgreSQL runs. For Aurora DSQL with optimistic concurrency control behavior embedded in your application code, the data export is straightforward, but the behavioral compatibility is not. Running through this exercise before you need to do it tells you the actual complexity.

2. How much application code depends on provider-specific behaviors? 

Stored procedures that reference Aurora-specific system variables. Every monitoring integration that reads CloudWatch-specific metrics rather than standard database views. Each authentication flow that uses AWS IAM database authentication rather than standard credentials. These are all examples on lock-in ‘surfaces.’

Why is this important? Well, put simply, listing them all gives you an estimation of migration cost – even if migration isn’t a current priority. You’ll know if the list is too long before it becomes relevant…and costly.

3. How many other services does this database integrate with, and are those integrations portable? 

What’s the significance of a database that feeds an event stream to a cloud-native message queue, triggers cloud-native functions on changes, and is monitored through a cloud-native observability platform? Answer: it has accumulated operational lock-in well beyond the database itself.

Moving the database to a different provider doesn’t move any of that integration infrastructure, either. Understanding the full integration surface is necessary for any realistic migration cost estimate.

4. You need to move in 12 months. How long will it take, and how much will it cost?

You need to be concrete here. Estimate the data migration time, the operational infrastructure rebuild, the integration rework, and the time spent on team retraining. If, once added up, the number is 18 months and many engineers, you have useful information regardless of whether migration is currently on the roadmap. Information that should inform every significant cloud database platform decision. 

In conclusion: navigating cloud database lock-in

The goal of this framework is not to argue against cloud database lock-in categorically. For many organizations, deep integration with a single provider’s services is the right decision.

It might be that Aurora’s performance and operational simplicity are a huge asset to you. Or perhaps it’s Azure SQL Managed Instance’s SQL Server compatibility, removing (expensive) migration friction for your organization. And then there’s Spanner’s global consistency capabilities, which are unique. 

In short, it’s not a mistake to choose a ‘locked-in’ service. It is a mistake, however, to choose one without making the trade-off explicit and understanding what exit actually looks like. Organizations with a full understanding of the depth of their lock-in are in a much better position to negotiate with their provider, plan for scenarios that require changes, and evaluate new options as they emerge.

On the other hand, organizations that discover their lock-in during an emergency simply do not have these options. 

Measure before you build. The cost of that clarity is a few hours of architecture discussion – significantly less hefty than the cost, and consequences, of discovering it during an unplanned re-platforming event.

How to use Redgate Flyway as a multi-database migration system

Learn how to use Flyway to do a single-batch, multi-database migration, comprising SQL Server, Oracle Cloud, PostgreSQL, MySQL and SQLite databases.
Read the guide

FAQs: Cloud database vendor lock-in

1. What are the three types of cloud database lock-in?

The three types are engine lock-in (dependency on a specific database product, such as PostgreSQL), feature lock-in (dependency on provider-specific capabilities like Aurora’s backtrack or Spanner’s TrueTime consistency), and operational lock-in (dependency on the surrounding infrastructure — IAM policies, monitoring dashboards, automation scripts, and institutional knowledge — that accumulates over time).

2. Which type of cloud database lock-in is hardest to exit?

Operational lock-in is the most expensive and underestimated to exit. While a data migration may take two weeks, rebuilding the operational infrastructure — monitoring, access control, automation, alerting — on a new platform typically takes months. Team retraining adds further time. These costs rarely appear in initial vendor comparison exercises.

3. Does Amazon Aurora create vendor lock-in?

Yes. Although Aurora presents itself as MySQL- and PostgreSQL-compatible, its underlying storage layer is proprietary to AWS. Features like Aurora Serverless v2 autoscaling, Aurora Global Database cross-region replication, and the backtrack capability have no direct equivalents on other platforms. AWS’s own documentation acknowledges Aurora creates “a different vendor lock-in by providing unmatched ROI.”

4. How do you measure cloud database lock-in before migrating?

Use a four-question framework: (1) Can you run a complete data export today, and is the output portable? (2) How much application code depends on provider-specific behaviours? (3) How many integrations surround the database, and are they portable? (4) If you had to migrate in 12 months, what would it realistically cost in time and engineering resource? Answering these concretely before migration surfaces hidden costs.

5. Is Google Cloud Spanner more locked-in than other cloud databases?

Spanner represents the most complete form of cloud database lock-in commercially available. Its SQL dialect diverges from PostgreSQL and MySQL on data types, functions, and query semantics. It is built on Google’s proprietary TrueTime infrastructure (atomic clocks and GPS receivers) that exists nowhere outside Google’s network. Organisations typically choose Spanner knowing and accepting this, because its global consistency properties are not available at the same operational simplicity elsewhere.

6. Can you avoid cloud database lock-in entirely?

Avoiding lock-in entirely usually means forgoing the features that make managed cloud databases valuable in the first place. The more practical goal is to make the trade-off explicit: understand which type of lock-in you are accumulating, how deep it runs, and what exit would realistically cost — so that decision is made deliberately rather than discovered during an unplanned re-platforming event.

The post What is cloud database vendor lock-in? The 3 types explained (and how to audit yours) appeared first on Simple Talk.

Read the whole story
alvinashcraft
2 minutes ago
reply
Pennsylvania, USA
Share this story
Delete
Next Page of Stories