This week, Microsoft took a huge ax to its Xbox business. The company announced that it would be laying off 1,600 workers now, 1,600 more over the next fiscal year, and that it would be shedding four studios. Xbox CEO Asha Sharma hasn't been shy about why she's making such dramatic cuts, saying in a memo that the business is "not healthy." Speaking to Fortune, she said that "we simply spread ourselves too thin."
AI models have reached a threshold where they exhibit expert-level capabilities in vulnerability discovery, exploit chaining, and proof-of-concept generation. As AI-powered vulnerability discovery matures, every organization that builds or runs software at scale needs continuous proactive evaluation to ensure security controls are correctly implemented, layered effectively, and working as intended in production.
At Microsoft we encompass these security requirements, along with threat knowledge and operational frameworks in our Secure Future Initiative (SFI), to guide what a well-defended cloud service looks like. But defining the requirements is only the start. Meeting them means continuously evaluating our live services against them, at AI speed.
That is why Microsoft built a multi-agent AI system that proactively evaluates and hardens our cloud infrastructure—matching the speed, scale, depth, and quality needed for our unique hyper-scale production environments. This system is purpose-built to evaluate Microsoft’s own cloud services against our stringent security requirements and make our infrastructure harder to compromise. While this is an internal capability and not available as a customer-facing product or service, the insights and patterns we develop through this work will inform how we improve our products over time. This system complements existing tools in Microsoft’s security ecosystem. For example, this system incorporates code-level vulnerabilities, including from systems like codename MDASH and adds configuration, identity, network, and runtime context, to assess overall service security posture.
A modern AI architecture for proactive defense
Vulnerabilities don’t just live in code. They emerge from the interplay between how a service is built, configured, deployed, and connected. Consider a cloud service where the application code passes every security review, the identity configuration follows least-privilege policy, and the network rules restrict inbound traffic as designed. Individually, each component is compliant. The system evaluates the service as a whole and may find that a combination of a permissive service-to-service trust relationship, a token scope that grants broader access than the service requires, and a deployment configuration that exposes an internal API to an adjacent network tier creates a composite vulnerability that no single-component review would surface.
At its core, the system employs a multi-tier agent hierarchy: orchestration agents for workflow management, analysis agents that specialize in security reasoning and are grounded in Microsoft’s threat intelligence—including emerging patterns and threat actor activity—and evidence-gathering agents that investigate across code repositories, infrastructure definitions, identity configurations, runtime settings, network topologies, and live resource states.
The result of this multi-stage analysis is a comprehensive security understanding of each service that goes beyond what any single analysis method can provide on its own. Compared to traditional human-led security reviews that take weeks, the system compresses the same depth of analysis into hours.
How it works: The system follows a multi-stage analysis pipeline, where each stage builds on the one before it:
Profiles each service architecture to understand components, data flows, trust boundaries, risk exposure, and more.
Enumerates applicable security controls based on SFI requirements across identity, network, tenant isolation, engineering systems, and detection domains.
Verifies control implementations against real-world code, configurations, and cloud resources.
Evaluates defense-in-depth coverage to help ensure layered protections exist across all control domains.
Identifies where controls are missing, misconfigured, or brittle, and maps the compensating controls that determine whether a gap is exploitable in practice.
Produces compensating controls and durable fix recommendations for immediate-risk reduction while driving lasting remediation.
Continuously learns and improves by incorporating feedback from security reviewers and service teams, and by tapping into Microsoft’s evolving threat intelligence to adapt to new patterns.
Core design principles
The analysis pipeline is shaped by four principles that determine how the system reasons about security:
1. Frontier-ready architecture
The system is built with modular model interfaces that can take advantage of new frontier capabilities as they emerge. New models, enhanced planning, and execution capabilities can be integrated behind stable agent interfaces—preserving existing tooling, orchestration, knowledge, pipelines, reporting, and governance.
2. Compositional risk reasoning
The system uses “what-if” agentic ideation to reason compositionally about risk. It explicitly explores how individual security gaps can chain together into multi-step attack paths. For example, a minor misconfiguration in identity, combined with a seemingly unrelated network exposure, and a missing data encryption control, might together enable a serious breach. Modern attacks are often complex sequences rather than single bugs, and the system is designed to help identify and analyze them. By running diverse models and large-scale reasoning trials in parallel, the system explores an expansive space of scenarios that traditional static analysis or single-scan tools would miss.
3. Service-specific adaptation
Cloud services aren’t one-size-fits-all, so security analysis shouldn’t be either. Rather than applying a fixed checklist, the system builds a service-specific understanding of each service it analyzes. It profiles the service in depth—identifying its components, mapping data flows, locating trust boundaries, and determining which security controls should apply given that service’s unique architecture and risk profile. If a service uses a novel pattern, a microservices architecture spanning multiple codebases, or an agent-to-agent communication model, the system adapts its analysis to account for those patterns. This adaptive approach, guided by current SFI requirements, means that the system can tackle emerging cloud paradigms that don’t fit traditional security checklists.
4. Defense-in-depth evaluation
A key focus area for SFI is layered defense. The system asks two questions: “What vulnerabilities exist?” and “Where does this service lack multiple lines of defense?”. It evaluates whether critical security domains have overlapping, robust controls, and it flags any missing or brittle layers—even if no immediate exploit is identified.
For example, the system will highlight a scenario where a service might have a weak network segmentation or an overly permissive admin role—even in the absence of a known attack—because those gaps mean a single failure could lead to a compromise.
This forward-looking, “assume breach” analysis embodies the Zero Trust and defense-in-depth principles reinforced by SFI. In an era when AI-assisted attackers can enumerate systems faster and chain together weaknesses more systematically, ensuring redundant safeguards is increasingly critical.
The assurance tree: SFI in action
At the core of the system are the SFI engineering and security principles: a structured body of security requirements shaped by years of hardening the Microsoft infrastructure. These requirements guide what the system evaluates, how it reasons about risk, and the recommendations generated. When security expectations evolve—whether to address a new class of threats or incorporate lessons from remediation—the system’s reasoning evolves with them. The assurance tree is how we express these requirements: a structured, hierarchical map of security controls that the system expects a service to have in place, tailored to that service’s usage and design.
As the system profiles a cloud service, it generates an assurance tree tailored to that service. At the top level of the tree are the fundamental security domains, that map to the SFI pillars. Each of these domains is recursively decomposed into more granular controls and sub-controls tailored to the service. For instance, Identity security decomposes into controls for password policies, OAuth token handling, and MFA enforcement—down to verifying that the service’s code correctly validates a JSON Web Token’s issuer and expiration. The assurance tree guides the system’s evidence-gathering agents to verify that thousands of expected controls are in place and effective—or to identify where something is missing.
This approach turns security from an open-ended hunt into a systematic verification of the SFI requirements: the system is essentially asking, “Have all the security measures that should protect this service been properly implemented?”. Crucially, it goes further—considering how individual gaps might combine, helping to ensure that even combinations of missing controls are identified and addressed.
Proven results: From theory to practice
Within a few months, the system has enabled Microsoft security engineering teams to proactively harden our cloud services. It generates findings and recommendations which our security engineering teams then validate and implement. Because the system evaluates the whole service in context and reasons about the severity and exploitability of each issue before surfacing it, its findings have proven high quality and actionable: more than 90% have been confirmed as genuine security issues by our security engineers, enabling proactive action to improve security posture. Just as important as the volume and precision of findings is their nature. Many issues the system discovers are nuanced, cross-domain vulnerabilities that wouldn’t have been caught by traditional methods. For example, the system has uncovered security gaps that only become apparent when considering code, configuration, and cloud resources together—the kind of issue that isolated scans or compliance checklists could overlook.
This capability allows us to enhance how we do security reviews. Traditionally, a deep security review of a complex service might span weeks of effort by multiple domain experts. The system can achieve a thorough review in a matter of hours—allowing teams to assess more services, more frequently.
The path forward: Applying these principles in your environment
If you are responsible for security at your organization, the key question is whether your defenses are keeping pace. AI models will continue to evolve. The organizations that are hardest to compromise will be the ones that have layered, verified controls already in place—not the ones that react fastest after something is found.
Based on what we have learned from building and operating this system, here are three principles any organization can apply now:
Go beyond code scanning to system-level discovery. The most consequential issues emerge not from a single bug, but from how factors including code, configuration, identity, and network interact in production. Collect rich signals across these domains and evaluate your services as composed systems, not isolated components. Prioritize composite attack paths over individual findings.
Move beyond known vulnerability patterns to proactive defensive controls. Traditional scanning asks, “Is there a known bug here?” Proactive hardening asks, “Does this service have comprehensive controls and layered defenses?” Reason about not just vulnerabilities, but controls, and how defense-in-depth coverage can improve protection before a specific exploit is discovered.
Integrate AI to drive proactive prevention at machine speed. The same AI capabilities that accelerate vulnerability discovery can be applied to continuously evaluate whether security controls are correctly implemented, layered effectively, and working as intended. Organizations that adopt AI-powered proactive evaluation will identify and close gaps faster than those relying solely on periodic manual review.
For deeper guidance on implementing AI-powered defense for an AI-accelerated threat landscape, customers can review Secure Now guidance for AI‑powered security and proactive defense. Any customer with a Microsoft Entra ID can access it. Microsoft Security customers will also have access to capabilities that enable them to assess their exposure and take action.
Moving forward, we will share more about how we are scaling our response operations to match machine speed and how SFI’s engineering practices are evolving for this new reality.
To learn more about Microsoft Security solutions, visit our website. Bookmark the Security blog to keep up with our expert coverage on security matters. Also, follow us on LinkedIn (Microsoft Security) and X (@MSFTSecurity) for the latest news and updates on cybersecurity.
Polished charts from simple specs. Flint allows AI agents to reliably generate expressive, visually polished charts from simple, human-editable specifications.
Semantic types guide design. Flint leverages semantic data types to express meanings of data. They help the compiler choose appropriate scales, baselines, formatting, and color schemes.
Layouts adapt to the data. Flint automatically manages sizing, spacing, labels, and layout so charts remain readable as cardinality and density change, without explicit user configurations.
One spec can target multiple backends. A single Flint specification can compile to Vega-Lite, Apache ECharts, or Chart.js without rewriting the chart from scratch.
Built for agent workflows. The open-source project includes the flint-chart library and the flint-chart-mcp server, so agents can create, validate, and render charts directly in chat or coding environments.
Figure 1. Flint supports a diverse collection of visualizations with its simple spec, which can be rendered with visualization libraries like Vega-Lite, Echarts, and Chart.js.
Creating a good chart requires many design decisions: how dates should be parsed, whether a scale should start at zero, how values should be formatted, how much room labels need, and which colors make the data easier to read. Modern visualization libraries such as Vega-Lite, Apache ECharts, and Chart.js expose these controls, but there is a trade-off: Short specifications that rely on system defaults often produce uninspiring charts, while polished visualizations require detailed specifications with purposely chosen parameters that are often verbose, fragile, and error-prone.
This trade-off becomes sharper as large language models (LLMs) and AI agents take on more visualization work. Agents are especially prone to errors when they must manage complex, low-level specification details, and the resulting fragile code can be difficult for people to inspect, repair, or reuse. Ideally, we need something in between: a compact specification that agents can produce reliably, people can edit directly, and a system can compile into a well-designed chart.
To address this challenge, we introduce Flint (opens in new tab), a visualization intermediate language for AI-driven chart creation. Flint helps AI agents create expressive, attractive charts from simple, human-editable chart specs. Instead of requiring verbose low-level parameters for scales, axes, spacing, and layout, the Flint compiler derives optimized chart settings from the data, semantic types, chart type, and encodings. The same Flint spec can render through multiple backends, including Vega-Lite, Apache ECharts, and Chart.js.
Figure 2. Flint compiles a compact, human-editable chart specification into a complete backend-native specification and rendered visualization. In this heatmap example, the Flint spec names semantic types (period as YearMonth, newUsers as Profit) and maps fields to visual channels. The compiler derives the Vega-Lite details, including temporal parsing, axis formatting, color scale, cell sizing, legend configuration, and layout.
How Flint works
Figure 2 illustrates the how the Flint compiler turns a compact chart specification into a refined heatmap.
To produce a high-quality heatmap, traditionally, we need to explicitly tell the system with low-level chart properties about how to process the period field, how to properly label MonthYear values, size individual heatmap cells, and choose a color scale that appropriately represents positive and negative newUsers values. Without these configurations, visualization libraries must guess from field names and raw values, which can lead to charts that are technically valid but potentially misleading. While they are important, hard-coding these details can be difficult and error-prone, and they make specification fragile and hard for users to understand or adapt.
In Flint, these low-level details are systematically managed, where the compiler infers them from high-level data and chart specifications. Here, the data specification captures semantic types and optional metadata, and the chart specification defines the chart type and maps fields to visual channels such as x, y, color, size, or facet. From this information, the compiler derives the parsing rules, scales, axes, aggregations, formatting, color schemes, layout, and generates the backend-native specification, which is used to render the final polished visualization. This frees users from explicitly setting fragile and error-prone low-level details.
Furthermore, because the intermediate representation is separate from any single rendering library, Flint can target backends with very different APIs and programming models. Users can keep the same compact chart intent while compiling to Vega-Lite, ECharts, or Chart.js, and choose the backend whose capabilities best fit the visualization.
PODCAST SERIES
The AI Revolution in Medicine, Revisited
Join Microsoft’s Peter Lee on a journey to discover how AI is impacting healthcare and what it means for the future of medicine.
Flint is well suited to LLM-based chart generation because semantic types are often easier for models to infer than the full set of low-level visualization parameters. Field names, value patterns, and common data knowledge can help an agent recognize whether a column represents a date, price, percentage, country, ranking, or correlation. Once those meanings are explicit, the compiler can handle many design decisions that would otherwise appear as brittle, library-specific code.
In our research study, we compared Flint with DirectVL, a baseline that asks the model to directly generate full (more complex) Vega-Lite specifications in a LLM self-evaluation pipeline. Across three tested models based on testing data from Tidy Tuesdays, Flint received higher overall LLM-judge scores: 16.27 vs. 15.91 with GPT-5.1, 16.16 vs. 15.60 with GPT-5-mini, and 15.91 vs. 15.34 with GPT-4.1. In fact, Flint has been so powerful and reliable that it is now used to power Data Formulator (opens in new tab), a Microsoft Research project for AI-assisted data analysis and visualization.
To make Flint easy for your agents to access, we also release flint-chart-mcp, a Model Context Protocol (MCP) server that allows agents to create, validate, and render charts inside a chat or coding environment. MCP calls can embed data inline or read configured local files, and the server can open an interactive chart view so users can inspect and refine the results.
Figure 3. Once you set up the flint-chart-mcp with your favorite AI client, the agent can generate interactive visualizations powered by Flint to answer your data exploration questions.
Flint points toward a shared semantic layer for visualization, where people and AI agents can work with compact chart intent while a compiler handles the careful low-level details. We invite the community to explore the project and build on it.
Custom domains make a project feel real. But for many developers, DNS, the last mile, is also the most frustrating: A records, CNAME entries, TTLs, and that long wait where you’re never quite sure if the internet is broken or you are.
In this post, I’ll walk through how I took a project from an empty repository to a live website on a custom domain, secured with HTTPS, in about 14 minutes without manually editing a single DNS record. The trick is to let GitHub Copilot CLI drive the work, with a community Namecheap skill handling the DNS automation through the registrar’s API.
A Namecheap account, for buying the domain and using its API
No prior DNS expertise required. That’s the whole point. Let’s get started. ⤵
Step 1: Publish a site with GitHub Pages
Every deployment needs something to deploy, so start with a home for the site: a new public repository.
With the repository in place, you don’t have to hand-write an index.html, commit it, and then click through the pages settings yourself. Instead, describe the outcome you want to Copilot CLI and let it create the landing page and enable GitHub Pages for you.
The site is now live on a github.io URL. That’s a solid start. Now let’s give it a proper address.
Step 2: Register an inexpensive domain
You don’t need a premium .com to ship a side project. For this walkthrough I chose one of the cheapest top-level domains available, .click, and searched for an available name.
The total came to USD $2.00, or about CAD $2.46. That’s a low-risk price for trying a custom domain on a side project.
Step 3: Connect the domain to GitHub Pages
This is the step developers tend to dread. Here, an AI assistant does the repetitive work while you stay in control of the decisions.
Enable Namecheap API access
Before Copilot CLI can update your DNS, you need to turn on Namecheap’s API. In your Namecheap account, go to Profile → Tools, scroll to Business & Dev Tools, and select Manage under Namecheap API Access.
You can also navigate directly to the API access settings page (note that this URL may change over time).
On that page, complete three steps:
Toggle the API to ON.
Add the public IP of the machine that will call the API to the IP allowlist (Namecheap labels this field Whitelisted IPs).
Copy the API Key and store it somewhere safe. You’ll need it shortly.
Next, give Copilot CLI the ability to talk to Namecheap by installing the Namecheap skill. It’s a single command:
gh skill install github/awesome-copilot namecheap --scope user
The first time you ask Copilot to do something like “list my Namecheap domains, it confirms the skill is configured and prompts you for your username.
Then it asks for the API key you copied earlier.
With credentials in place, Copilot returns the list of domains in your account. It’s a quick way to confirm everything is wired up correctly before making any changes.
Point the domain at GitHub Pages
Now connect the domain to the site. Ask Copilot to configure the custom domain using the skill.
A good automation asks before it acts. The skill pauses to confirm the change before touching any records.
Once you approve, it replaces the existing parking records with the GitHub Pages A records and a CNAME for the WWW subdomain, which is the exact configuration GitHub Pages expects. This matches GitHub’s documented steps for configuring a custom domain for your GitHub Pages site.
It also handles the repository side, committing a CNAME file that tells GitHub Pages which custom domain the site should answer to.
Not using Namecheap? The same approach works with any registrar that offers an API. You don’t need a purpose-built skill: point Copilot CLI at your registrar’s API documentation and ask it to read, understand, and use that API to set the GitHub Pages records for your domain. The registrar changes; the workflow doesn’t.
Step 4: Verify the deployment
Rather than assuming success, Copilot CLI checks its own work. First, it confirms the domain resolves.
Then it confirms that the site returns a healthy HTTP 200 response.
Now for the timeline. The domain was purchased at 11:21:27 a.m. ET.
The site was live on the custom domain, served over HTTPS, at around 11:35 a.m. ET. That’s roughly 14 minutes from owning nothing to a fully deployed site, including API setup, skill installation, DNS configuration, propagation, and verification.
Wrapping up
DNS isn’t hard, exactly, but it’s fiddly, easy to get wrong, and slow to give feedback. By pairing GitHub Pages with GitHub Copilot CLI and the Namecheap skill, the repetitive parts of a custom-domain deployment fade into a short conversation: you make the decisions and approve the changes, and the tooling handles the plumbing.
The following article originally appeared on Markus Eisele’s newsletter, The Main Thread, and is being republished here with the author’s permission.
There’s a mental model spreading through the developer community right now that goes something like this: Agents are smart enough to figure things out, so heavy upfront specification is bureaucratic overhead you don’t need anymore. Just describe the goal loosely, let the agent explore, and correct as you go. Fast. Flexible. Modern.
It’s wrong. Not because agents aren’t capable—they often are—but because the accounting is off. You’re not eliminating cost. You’re deferring it, fragmenting it, and making it harder to see.
Let’s run the actual ledger.
Two poles, two hidden costs
At one extreme: minimal specification. You describe intent loosely, agents interpret freely, and work begins immediately. The upfront cost in human effort is near zero. What you don’t immediately see is what accumulates downstream: correction loops, each carrying token cost plus human reengagement time. Review cycles where a human acts as the oracle for every output—deciding whether what the agent produced is what was actually meant. Rework when it wasn’t.
At the other extreme: full formal specification. TDD, BDD, Gherkin scenarios, acceptance criteria locked down before a single line of code runs. The upfront human effort is real and visible. But the downstream verification cost looks fundamentally different, because the tests are the oracle. Pass or fail. The human doesn’t need to personally evaluate every output—the spec does it automatically, repeatedly, without fatigue.
What you’re actually trading off is when you pay and in what currency. Minimal spec front-loads token cost and back-loads human judgment. Heavy spec front-loads human effort and back-loads almost nothing—automated verification doesn’t scale with runs.
The total cost of both approaches traces a U-shaped curve when you plot it against specification completeness. The minimum of that curve—the sweet spot—sits somewhere around well-structured acceptance criteria or BDD scenarios. Not at zero specification, and not at a 40-page formal requirements document.
The trap is visible once you plot the whole ledger. Minimal specification looks cheap only before downstream rework enters the chart. Multi-agent work pushes the minimum further right because drift compounds across handoffs.
The old problem was always the spec
The real challenge in software engineering has always been specification.
Not typing. Not syntax. Not even architecture in the abstract. The hard part was agreeing what should exist, what should never happen, which trade-offs matter, what the system is allowed to forget, and what “done” means when the world is messier than the ticket.
Agents don’t remove that problem. They make it more visible.
For decades, we hid the specification problem inside meetings, backlogs, code reviews, QA cycles, incident retrospectives, and the private mental models of senior engineers. A lot of software engineering was never “writing code.” It was dragging an underspecified idea through enough friction that the missing pieces were forced into the open.
Agents reduce the friction of producing code. That is wonderful. It also means the missing pieces surface later, because the system can now produce a plausible implementation before anyone has really decided what the implementation is supposed to mean.
In the old world, vague requirements ran into human slowness. In the agent world, vague requirements run into machine speed.
When implementation gets cheaper, the bottleneck doesn’t disappear. It moves into specification and verification.
But writing the spec is only half the problem
Here’s what almost every framing of this trade-off leaves out: A spec needs to be validated before you hand it to an agent.
This sounds obvious stated plainly. In practice, it’s systematically ignored.
When you write a spec—even a careful one—it can fail in ways that are invisible until the agent executes against it. It can be internally inconsistent: two requirements that contradict each other, neither obviously wrong in isolation. It can be incomplete: It covers the happy path thoroughly and says nothing about what happens when the third-party API returns a 429. It can be technically correct but untestable: The spec describes behavior that can’t be mechanically verified. And most insidiously, it can be precisely what you wrote but not what you meant.
An agent executing faithfully against a flawed spec produces something that is difficult to debug. It passed every check it was given. The problem isn’t in the implementation—it’s upstream, in the spec itself. And now the correction loop is more expensive, because you have to unwind not just code but reasoning.
Spec validation is therefore a distinct cost category that lives between “write spec” and “run agent.” It asks: Is this spec internally consistent? Is it complete enough to constrain the agent usefully without over-constraining valid solutions? Does it actually describe the thing we intend to build?
That validation work is human time, or it’s agent time, or ideally it’s both—but it isn’t zero. The moment you add it to the ledger honestly, the picture changes.
How agents can write specs
There’s a third strategy this two-pole framing systematically ignores: use agents to write and validate the spec, then use implementation agents to execute against it.
This changes the cost structure of the spec side of the curve. Instead of heavy human effort to produce acceptance criteria or BDD scenarios, a spec-drafting agent produces a first version from rough intent. A spec-validation agent—with a different role and system prompt, possibly with search access or domain knowledge—stress-tests that draft for consistency, completeness, and testability. A test-writing agent translates the surviving claims into executable checks. You review the result, which is faster than writing it from scratch.
The important detail is that the agent should not merely “write requirements.” That produces polished fog.
A useful spec-writing agent behaves less like a stenographer and more like a skeptical product engineer. It should name assumptions. It should separate goals from nongoals. It should produce examples and counterexamples. It should say which requirements are mechanically testable and which ones still depend on human judgment. It should identify the failure modes a lazy implementation would probably miss. It should ask what must be invariant across valid solutions.
The best prompt isn’t “write me a spec.” It is closer to this:
Draft the smallest spec that would let another agent implement this safely. Include assumptions, nongoals, acceptance criteria, edge cases, observable outcomes, and open questions. Then mark which parts can become automated tests and which parts require human review.
Then you run a different agent against the output:
Attack this spec. Find contradictions, ambiguous terms, hidden dependencies, untestable claims, missing failure modes, and places where an implementation could pass the written criteria while still violating the intent.
The sweet spot is not agent-written prose. It’s human-approved, agent-drafted, adversarially reviewed specification with as much of the oracle made executable as the domain allows.
Agents don’t remove the need for a spec. They can lower the cost of moving toward the useful part of the curve, where the spec is complete enough to guide implementation but still reviewed by a human.
This doesn’t make spec validation disappear. It changes who does it and at what cost. The structural requirement—that the spec be validated before the implementation agents run—remains. What changes is that agents are now doing part of that work.
How BDD partially solves this
Behavior-driven development, when done well, collapses spec writing and spec validation into the same artifact. A Gherkin scenario is simultaneously a description of intent and an executable test. You can run the spec against a skeleton implementation immediately and observe whether the description produces coherent behavior. The act of making the spec executable forces a kind of validation that prose acceptance criteria don’t—some kinds of ambiguity have to be resolved before the scenario can even run.
This is why the minimum of the total cost curve doesn’t just reflect reduced rework. It reflects the structural advantage of a format where validation is built into the medium.
BDD earns its keep when it moves judgment out of repeated human review and into an executable oracle. That is why its sweet spot appears around behavior that is stable enough to test.
The catch is that someone still has to write the scenarios well. Gherkin can be written badly. Business-language specs can be ambiguous in ways that the BDD framework doesn’t catch because ambiguity lives in semantics, not syntax. The format helps, but it isn’t a substitute for discipline.
Multi-agent pipelines break everything
If you’re running a single agent on a well-bounded task, underspecification is recoverable. The feedback loop is tight, correction is local, and the cost is bounded.
Multi-agent pipelines are a different class of problem entirely.
When Agent A produces output that becomes Agent B’s input, any interpretive drift from A compounds into B’s execution. B doesn’t know that A went slightly off-course. B works hard and confidently on the wrong foundation. By the time the output surfaces to a human, the error has been amplified and obscured through multiple layers of apparently coherent work.
This shifts the breakeven point decisively toward specification. In a multi-agent system, a spec isn’t just guidance for a single execution—it’s a coordination contract between agents. The less precise that contract, the more each agent’s interpretive freedom introduces variance that accumulates. You want a strongly typed interface between agents, not a loose conversational handoff.
For multi-agent work, the x-axis is no longer just “How much did we specify?” It’s “How strong is the handoff contract?” The minimum moves toward typed contracts and executable validators.
Validation of that contract matters correspondingly more. If the spec that coordinates your agents is flawed, you don’t have one agent doing the wrong thing—you have all of them, in parallel, doing differently wrong things.
What survives from methodology
So does this make everything we learned about coordinating software teams obsolete?
No. But it does change which parts were load-bearing.
Agile as theater is in trouble. Standups where people recite status into the air, estimation rituals that produce fictional precision, ticket ceremonies whose main function is to reassure management that uncertainty has been domesticated—agents do not need those. Honestly, humans didn’t either.
Agile as a feedback philosophy survives. Short cycles survive. Working software over abstract progress survives. Customer collaboration survives. The insistence that plans should bend when reality speaks survives. If anything, agents make this more important, because they can generate a lot of convincing wrongness very quickly. The feedback loop has to get tighter, not looser.
XP survives even better. Test-first thinking survives because executable oracles are more valuable when implementation gets cheaper. Pair programming mutates into human-agent pairing, but the underlying idea remains: keep design judgment close to code production. Continuous integration survives because every agentic change needs a fast, impartial gate. Refactoring survives because agents can produce working code that is locally correct and structurally mediocre. Small releases survive because large invisible deltas are where both humans and agents lose the plot.
What probably fades is methodology as coordination theater for large groups of humans. What survives is methodology as a set of constraints that make ambiguity cheaper to discover.
Methodology survives where it creates fast feedback. It fades where it only creates status artifacts.
The interesting question is not whether Agile or XP “wins” in the agent era. The interesting question is which practices still reduce the cost of discovering that the spec was wrong.
Where to actually invest
The practical takeaway from this analysis is not “always write full BDD specs” and it’s not “always let agents roam.” It’s that the optimal investment point is task dependent, and the honest calculation includes spec validation as a real cost.
There is no universal optimum. The sweet spot moves with the work.
For a single agent on a small, well-bounded task, the sweet spot is usually structured intent: a goal, examples, nongoals, and a few acceptance criteria. BDD may be overkill. Zero spec is still lazy accounting.
For deterministic, well-understood work—API integrations, CRUD services, data transformations—the breakeven point sits further right. More specification pays off faster because the domain is constrainable and the tests are automatable. Skimping on spec here is just deferring rework.
For exploratory or creative work—architecture decisions, novel problem approaches, research synthesis—over-specification constrains exactly what the agent’s flexibility is good for. The breakeven sits further left. Use the agent’s interpretive freedom deliberately, but put boundaries around the exploration.
For multi-agent systems, the sweet spot shifts right again. The handoff is the product. Every agent boundary needs a contract: schema, invariants, allowed ambiguity, validation checks, and failure behavior. Otherwise you’re not orchestrating agents. You’re compounding interpretations.
In all cases: Validate your spec. Whether that’s a human review, an agent stress-test, or an executable format like BDD that forces structural consistency, the cost of skipping it is paid later, at higher interest, with worse diagnostics.
The seductive promise of zero-spec agent work is real, but the ledger it ignores is also real. The agents are getting better. The accounting problem is still ours.
Posted by Zoe Lopez-Latorre, Senior Developer Relations Engineer, Android
Back in March, we introduced Android Bench—our LLM leaderboard for real-world Android development tasks. Our goal was to provide transparency around model capabilities in Android development and to encourage model improvements, to give you more helpful AI options for your everyday workflow. Since then, we have enhanced the benchmark based on your feedback, including evaluating open-weight models and adding cost and efficiency dimensions to the leaderboard.
But AI capabilities are ever-evolving, and measurement needs to follow suit. As part of our July release, we have adopted the Harbor framework, which includes an updated version of the benchmarking agent used to evaluate models.
Along with this change to our evaluation, in this July release we’re adding 8 new models (Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus and Qwen 3.7 Max) to the leaderboard. We’re also sharing opportunities for you, the Android developer community, to contribute to the benchmark.
Upgrading our methodology with the Harbor framework
When we designed Android Bench, we anchored our methodology on leading industry standards available at the time. We used mini-swe-agent v1, a general-purpose benchmarking agent, and adapted it to the nuances of Android development to provide a baseline measurement for the capabilities of models for common Android development tasks.
To continue providing you with state-of-the-art evaluations that accurately measure the latest model capabilities on Android development, we are standardizing our benchmark to the Harbor framework. Harbor defines standards and integrations that make it easy for anyone to run the benchmark, evaluate their preferred set-up, or share results – providing you with additional transparency and visibility.
This upgrade enables us to more rigorously evaluate models and their capabilities, and we re-ran the benchmark on all models to establish an updated baseline. This means there is a minor shift in scoring, but you will still be able to view historical scores within the archive on our website.
We want to ensure Android Bench is helpful for you, so we will continuously update it as our evaluations and the industry mature.
Expanding the leaderboard with 8 new models
As part of our commitment to keeping the leaderboard fresh, we have added Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus and Qwen 3.7 Max to the Android Bench leaderboard.
You will see that Claude Fable 5 is at the top of the leaderboard with a score of 84.5, followed by GPT 5.5 with 80.2, with Claude Sonnet 5 in 3rd with a score of 76.2.
When just comparing Open-weight models, GLM 5.2 is at the top with 72.2, followed by Kimi K2.7 Code with a score of 70.4.
You can check out model performance and efficiency metrics on the updated leaderboard to see how these new and previous models navigate Android-specific challenges like Jetpack Compose migrations, wearable networking, and platform API updates.
Opening Android Bench to community contributions
From the beginning, we’ve valued an open and transparent approach, which is why we made our original methodology and test harness publicly available on GitHub. You’ve asked for a way to provide feedback on our dataset, so now we’re taking collaboration a step further by giving you, the Android developer community, a chance to shape Android Bench.
Starting today, you can contribute to Android Bench in two ways:
We will be reviewing the submitted tasks and will be assessing if they get added to the benchmark. We hope to build a benchmark that truly reflects the diverse, day-to-day realities of the global Android developer community.
Looking ahead
With more and more options for agentic development, maintaining a cutting-edge benchmark ensures that the AI assistance you rely on keeps getting smarter, more helpful, and more effective. Head over to our GitHub repository to check out the tasks. We invite you to submit a task to our team for review, and you can check out Harbor Hub to explore the dataset or submit evaluations.
Log in
