Read more of this story at Slashdot.
Read more of this story at Slashdot.
Consider a composite of a failure pattern that’s becoming increasingly common on teams that have scaled AI-assisted development past a handful of enthusiasts.
A staff engineer named Priya opens a pull request. The PR is 340 lines and adds an endpoint that writes to the customer table directly, bypassing the internal customer service API. The code is clean. The tests pass. The AI coding agent that wrote it has been the team’s most productive contributor for six months. The reviewer, three months into the team, approves. The PR ships that afternoon.
Two weeks later, during an integration debug, someone notices that customer records written by that endpoint are missing audit-log entries. The audit hooks live in the customer service API. The team banned direct database access two years ago for exactly this reason and wrote an architectural decision record (a versioned markdown document, one of dozens the team has accumulated, that captured what was decided, why, and what was superseded) to memorialize the rule. The engineer who wrote the ADR has since left. Nobody on the current team remembered the decision. The document was sitting in a directory the current workflow never touches.
The endpoint gets rewritten. Audit gaps get backfilled. The team spends most of a sprint on cleanup. Nobody calls this a failure of AI-assisted development. The AI wrote functional code. It just wrote code that violated a decision the team had already made, in a document the AI had no view into.
This is a specific failure mode. It is not a hallucination since the output was grounded, syntactically valid, and idiomatic. It’s not a model-quality problem since a better model on that same prompt wouldn’t necessarily help if the decision remained absent from its context. It’s a memory problem. Not the model-internal sense of context window, but the organizational sense. The ADR was available in the repository. It was never surfaced to the agent, and the reviewer had not read it.
This piece is about naming what would need to exist for that PR to have been caught, or better, never written in the first place.
Priya’s PR illustrates one source of a broader rework problem. Faros AI, an engineering analytics platform, published a report in 2026 based on telemetry from more than 22,000 developers across 4,000+ teams. AI-code acceptance rates had risen from 20% to 60% between periods of low and high AI adoption, while code churn (i.e., lines deleted within days of being added) had increased 861% over the same interval.
Faros is careful in how it frames the churn number. The increase may include not only rework but productive refactoring, previously unaffordable cleanup, or faster iterative improvement. But the number still exposes a gap between code entering the repository and code that survives there. A gap that has widened, at scale, alongside AI adoption. Anecdotally, engineers at teams running these tools describe returning to code they had already approved to fix issues that were not obvious at review time. Architectural drift, i.e., code that individually looks fine but collectively pulls the codebase away from where the team agreed it should go, is one plausible contributor.
I’ll call the discipline of preventing this failure mode architectural drift prevention: keeping generated code aligned with the architectural decisions a team has already made.
The pattern-matching explanation for that gap has been that the AI isn’t good enough at writing code yet, and a better model will close it. Watch enough review cycles at a team running AI-assisted development for a year, though, and a different pattern emerges. The generated code isn’t obviously bad. It compiles. It passes tests. What it doesn’t do is respect decisions the team has recorded but the AI has never seen.
Teams commonly reach for several existing mechanisms to close this gap. Most of them are the wrong shape for the problem.
The closest attempts are files like Cursor Rules and CLAUDE.md, i.e., free-text markdown documents dropped into the project root so agents read them as standing instructions. These are the right instinct at the wrong resolution. Free text has no precedence rules, no versioning, no lifecycle. When one rule contradicts another, nothing arbitrates. When a rule is violated, nothing catches it. These are documents in the shape of configuration.
Linters and code formatters operate a layer below. They enforce that a function has a return type annotation or that a variable name follows a convention. They can’t enforce that customer-data writes must go through the customer service API, because that’s not a syntactic property. It’s a semantic decision recorded in a document the linter has no reason to read.
Dependency scanners, SCA tools, and lockfile audits close a related gap; they catch known vulnerable libraries, license violations, and outdated versions. They would’ve flagged nothing about Priya’s PR. Every dependency in it was current, had no known vulnerabilities, and was approved. The violation was a routing choice inside the team’s own architecture, not a library problem.
LLM-assisted code review is the fashionable answer. A second AI reads the pull request and comments on it. This catches surface issues like a swallowed exception, an off-by-one bug. It doesn’t catch drift, however, because the second AI has the same problem as the first: no durable access to the team’s recorded decisions. Two probabilistic passes over the same blind spot are not one deterministic pass with sight.
Human review is the last line, and it works when the reviewer knows the history and has enough time to inspect the change. Agentic development changes both conditions. Agents can produce multiple implementations, pull requests, and revisions in the time it takes a human reviewer to assess one. Code output scales; review attention does not. Asking humans to compensate by reviewing more and harder simply moves the constraint downstream.
None of these tools are bad. They’re just the wrong layer for the drift problem.
The missing layer connects recorded engineering decisions to the tools that generate, review, and merge code. Its job is to make the team’s accumulated architectural decisions machine-readable, injectable, and enforceable. I’ll call this layer engineering governance, borrowing the term from adjacent categories like data governance and security governance, where it means the same thing: A structured way for an organization to make explicit and enforceable the rules it already implicitly follows.
At the shape level, the layer needs to do four things.
The critical property that ties those four together is a discipline about where probabilistic reasoning is allowed. Probabilistic systems may retrieve or recommend. They shouldn’t independently determine an enforcement verdict. Every block or warning has to reconstruct from artifacts on disk. That’s the code, the ADR, the retrieval log, the rule text. Then, when a developer asks “why did this fail?” or an auditor asks “on what basis?” the answer isn’t “the AI said so.” That’s what makes the layer defensible in the situations where defensibility matters: regulated environments, compliance review, incident retrospectives, and the everyday conversation where an engineer has to justify a blocked merge to the person whose code was blocked.
AI may help surface relevant decisions, but it shouldn’t be the final authority. The enforcement path must remain deterministic: every verdict should resolve to explicit rules, observable evidence in the code, and a result that another person can reproduce.
Naming a category clearly requires naming what falls outside it. The engineering governance layer, done right, is not any of the following:
The AI coding stack is assembling itself in the open, without anyone architecting it. Each of the major coding assistants is specializing on a different piece of the loop; editing, autonomous execution, review, planning. The layer this stack doesn’t yet have is engineering governance.
The need isn’t limited to regulated industries. Any team that values reliable engineering, long-term product quality, and the trust of its customers needs to know that generated code respects the decisions the system depends on. In higher-risk environments, that requirement becomes formal and auditable. Elsewhere, it’s simply part of building software responsibly. In both cases, the enforcement path should be deterministic and traceable, with the rules, evidence, and verdict open to inspection rather than hidden inside another model or a proprietary black box.
Priya’s PR is a pattern, not an incident. Three things worth doing this quarter, regardless of tooling choices.
The productivity gains from AI-assisted development are real and worth having. So is the architectural coherence teams spent years building. Engineering governance is the layer that lets you keep both.
Imagine plugging two LLMs from different frontier labs into the same coding agent and finding that they solve exactly the same number of benchmark tasks. If the evaluation stopped there, you might conclude that the models are interchangeable and simply choose the cheaper one.
This is what also happened on one of our private benchmarks: Claude Opus 4.7 and Gemini 3.5 Flash solved the same number of tasks. But the tie concealed two very different execution profiles. Opus used an average of 184 steps and cost USD 2.79 per run, while Gemini took an average of 271 steps but cost only USD 1.24. The final result was identical, but the way each model reached it was not.
This difference is invisible in the metric most often used to compare coding agents, which is the resolve rate. It measures how many tasks the evaluation tests pass, expressed in percent. While resolve rate answers an important question, namely whether the agent solved the task, it says little about how the solution was reached.
A coding agent such as Junie is more than the LLM behind it. Given an issue and a repository, Junie lets the model inspect files, search for symbols, edit code, run commands, and execute tests. These observable actions form the agent’s trajectory. A trajectory does not reveal the model’s private reasoning, but it does show how the model worked with the repository. We can see whether it localized the problem before editing, repeated the same searches, tested its assumptions, and kept the final patch focused.
We built an evaluation pipeline that analyzes both the result and the path that produced it. It combines four perspectives: functional outcome, execution efficiency, patch quality, and process quality. Functional correctness remains the starting point, while additional metrics explain what lies behind the final score.
A recent JetBrains Research post describes the benchmark meaning gap, identified in a recent research paper: a benchmark measures performance under a particular setup, but its score is often treated as evidence of a much broader coding ability. Performance gains may not transfer to other tasks, even within the same codebase, and model rankings can change with the task type.
Our work looks at a related gap within individual agent runs. Passing tests does not fully describe the patch quality. Two patches may implement the required behavior while differing greatly in scope, complexity, and fit compared to the existing architecture. For example, one may change a single relevant function. Another may add helpers, state, branches, or unrelated files – and still pass the same tests.
A failed outcome is equally ambiguous. The agent may never find the relevant code, may instead misunderstand the cause, edit the wrong layer, implement only part of the fix, or stop without adequate validation. These failures all require different actions. For example, a repeated search may call for better repository navigation or more focused prompting. Another example is a correct diagnosis of the issue followed by an incomplete patch. This suggests a problem in implementation or task completion.
Cost and latency add another dimension. As mentioned above, two successful runs can differ substantially in tokens, runtime, model calls, and tool use. A long trajectory is not necessarily bad if the task requires broad investigation. The important distinction is whether the extra work contributed to the solution, or it came from repeated and unproductive actions.
For model selection, the more useful questions are which model suits a particular kind of task, where it spends its effort, and how it tends to fail. This can be answered by a fine-grained analysis of both the trajectory and the patch through our pipeline.
For each benchmark task, the pipeline combines the issue, repository context, generated patch, test result, and execution trace. It then evaluates the run from four perspectives, which ask the following questions:
We propose a pipeline that combines deterministic metrics with semantic evaluation. The deterministic layer derives reproducible measurements from logs and repository data. These include test outcomes, runtime, token use, tool calls, modified files and symbols, code complexity changes, repeated file reads, unchanged command retries, and tool failure loops.
Rules alone cannot interpret every action. Opening a file twice may be wasteful, or it may be necessary after a related edit. A large patch may be unfocused, or it may be appropriate for a change that spans several components. For these questions, LLM judges receive structured evidence from the issue, patch, trajectory, and bounded repository context. They assess milestones such as finding the relevant code, reproducing the defect, identifying the root cause, addressing it in the patch, introducing unnecessary complexity, and validating the result. This combination gives us a clearer account of progress. It shows not only whether a run failed, but whether it failed during localization, implementation, or validation. Below figure serves as an illustration of the aforementioned components, inherent in our evaluation pipeline.

We used the pipeline to compare Claude Opus 4.7 and Gemini 3.5 Flash in Junie across four benchmark datasets containing 523 tasks. The results are shown below:

As you can see in the figure above, Claude Opus resolved 267 tasks, or 51.1 percent, while Gemini Flash resolved 254, or 48.6 percent. The models produced the same outcome on 430 tasks: both solved 214 and both failed 216. Only 93 tasks separated them. The overall scores were close, but the trajectories and patches showed different behavioral profiles.
A same-task comparison makes the different behavioral profiles concrete. One Opus run and one Gemini run both solved the same task. Both first opened a relevant file at step 15, were judged to have identified the root cause, and performed thorough validation. However, they had progressed in varying increments d by that point. Opus used a targeted search within the file and began implementation 13 steps later. Gemini initially inspected the large module more broadly. It ran its first executable check at step 30, but did not make its first production edit until step 88. Opus finished in 53 steps, moving between exploration, implementation, and validation six times; Gemini needed 192 steps and thirty-four such switches. The following figure depicts the different paths.

Gemini’s additional investigation was partly useful, but it also widened the scope and led to an unrequested change. Both runs passed the evaluation tests, and both changed the same file and symbols that the reference solution changed. Opus touched nothing else. Gemini’s patch also reached four further files, making edits there. It was assessed as sprawling, with significant redundancy and moderate hallucination.
This single example is illustrative rather than statistical. It shows how the same benchmark success can come from a direct, contained run or a longer path with unnecessary expansion.
A successful run usually progresses through four stages: locating the relevant code, identifying the root cause, implementing the complete fix, and validating the result. Resolve rate compresses this entire process into a single binary outcome, whereas trajectory analysis shows where the agent succeeded and where it fell short.
As trajectory analysis separates them, we can better analyze the 216 tasks that both models failed. We can see the results of the analysis in the figure below.

For both models more than 85 percent were assessed as having at least partially identified the root cause. For example, in one task, both agents recognized that text exceeding a token limit caused the error, but truncated the text instead of splitting it into valid chunks. In another, both corrected a faulty download parameter in one code path and missed the same problem in a companion path. A binary failure treats these runs like cases in which the agent never found the relevant component, although they were much closer to a correct solution.
The models were not completely lost. They had reached the relevant mechanism but implemented the fix incompletely, changed the wrong layer, or missed the task’s exact contract.
This is not simply a question of matching the golden patch. The reference solution is useful, but it is not the only possible valid implementation. A candidate may change a different file or architectural layer and still address the same mechanism. Structural comparison therefore needs to be combined with semantic assessments of diagnosis, completeness, and validation.
By using these outcomes it is possible to construct model-specific profiles that give more information about their strengths and weaknesses. In the following we list exemplary ones for Claude Opus 4.7 and Gemini 3.5 Flash.
Opus was more likely to identify the underlying cause of ambiguous defects. It often reached the correct mechanism or architectural layer and solved 53 tasks that Gemini missed. These results make Opus a useful starting point when the main challenge is understanding an unfamiliar repository or separating a visible symptom from its source.
The main weakness appeared after localization. Some runs found the right mechanism but stopped with a reproduction test, missed a companion branch or call site, or implemented a plausible custom solution instead of following an existing repository pattern. In 123 runs, Opus performed no executable validation, including 68 runs that still resolved the task. Skipping executable validation means a patch’s correctness is never actually confirmed, so even a resolved task carries undetected risk of regressions or edge-case failures that only running the code would surface.
Overall, Opus’ profile suggests a strong diagnostic model that benefits from an explicit transition to implementation, completion, and testing.
Gemini was more likely to run an executable check and use its output to refine the solution. These features were useful when the expected behavior was explicit, the responsible component was reasonably clear, and feedback was readily available.
The main risks we found with Gemini were convergence and repository grounding. Gemini often continued searching after reaching relevant code, repeated equivalent commands, or spent many steps on build infrastructure. It was also more likely to rely on unverified APIs, dependencies, paths, or test fixtures: 195 of its runs, or 37.3 percent, were assessed as containing moderate or severe hallucination, against 130 runs for Opus. Some patches expanded beyond the issue or included unrelated artifacts, and 80 runs, or 15.3 percent, showed significant or severe redundancy, more than twice the Opus rate of 6.5 percent.
Overall, Gemini benefits from precise task contracts, symbol verification, clear stopping rules, and a final review of the diff.
We also ran the pipeline over a wider set of models. We evaluated GPT-5.5, Claude Opus 4.7, Gemini 3.5 Flash, and Qwen 3.6 27B FP8 on the same four benchmark datasets, and the table below compares them on the 522 tasks all four of them share. The same four perspectives separate them as well: GPT-5.5 reached the highest resolve rate at 51.5 percent and was the only model that always ran an executable check, Opus led every patch quality metric, and Qwen 3.6 27B FP8 resolved 38.9 percent of the tasks at three percent of GPT-5.5’s cost per run.

GPT-5.5 and Opus finish four tenths of a point apart on resolve rate and within a cent of each other per run, so a leaderboard would treat them as interchangeable. Their patches are not: Opus was assessed with moderate or severe hallucination in 24.7 percent of its runs against 33.7 percent for GPT-5.5, and with significant or severe patch redundancy in 6.3 percent against 13.2 percent, while producing the shortest trajectories of the four models. What GPT-5.5 offers in return is process discipline, since it never ended a run without an executable check while Opus skipped validation in 23.6 percent of its runs.
Qwen 3.6 27B FP8 is a third kind of trade-off: 12.6 points behind on resolve rate and the weakest of the four at identifying the root cause, but inexpensive enough that a failed run costs little. Which model is preferable therefore depends on whether the expensive part of the work is diagnosis, patch review, or the run itself.
In this post, we inferred profiles for Claude Opus 4.7 and Gemini 3.5 Flash. These inferences are based on a specific Junie scaffold used in this evaluation, and they should not be used to generally describe the model themselves. Moreover, The LLM judge assessments are diagnostic signals rather than ground truth, and are heavily based on a single golden patch, which in most cases, as typical in the coding domain, is not the only viable solution. The judges may be therefore inclined to score negatively valid solutions if they differ from the reference one.
Resolve rate remains the foundation of coding agent evaluation, but it becomes more useful when paired with evidence about efficiency, patch quality, and process. Our overall goal is not to replace a leaderboard with another aggregate score. We would like to understand what produced each result and use those patterns to improve model selection, prompting, and agent design. From an industry perspective we can better refine agent design, by moving beyond aggregate success rates to fine-grained scores and behavioral profiles. On the other hand, from a user perspective, we can now empower Junie users to choose the right model for the job.
After careful consideration, we’ve decided to sunset the JetBrains Teacher Pack for Bootcamps.
If you’re planning to run a bootcamp and would like support from JetBrains, you can submit one final application by September 30, 2026, 11:59 pm CET. Starting October 1, 2026, we will no longer accept new applications.
Although the bootcamp program is coming to an end, we continue to offer a range of JetBrains education programs designed for different teaching and learning needs.
Read on to learn more about the timeline and available alternatives, as well as to find answers to frequently asked questions.
Over the past few years, we’ve expanded our education programs to better support teachers, students, classrooms, and independent learners.
Thanks to these expanded offerings, educators can now choose from dedicated programs for different teaching and learning scenarios, making a separate bootcamp program unnecessary.
September 30, 2026, will be the last day to apply for a JetBrains Teacher Pack for Bootcamps.
Bootcamps that submit an application on or before this date will receive one final set of student coupons, valid for up to six months.
Starting October 1, 2026:
The bootcamp program will be fully retired on April 1, 2027. By then, all student coupons provided via the bootcamp program will have expired, and the graduation discount will no longer be available.
Depending on how you teach or learn, there are several other ways to access JetBrains products.
If you teach at an accredited educational institution, the JetBrains Teacher Pack is the best place to start.
Depending on your role and needs, you can apply for:
If you publish your programming courses on platforms like Coursera, Udemy, Moodle, edX, or LinkedIn Learning, or even your own platform, the JetBrains Course Creators Program is a great option for you.
Students at accredited educational institutions can apply for the JetBrains Student Pack.
The Student Pack includes free access to all JetBrains IDEs, in-IDE courses, a two-year graduation discount after your student license expires, and more.
If you don’t qualify for an educational license, you still have the following options:
No matter how you learn or teach, we’re committed to making professional development tools accessible to you.
We’d like to thank every bootcamp instructor and learner who has been part of the JetBrains Teacher Pack for Bootcamps program.
While the bootcamp program is coming to an end, our commitment to education remains the same. We look forward to continuing to support educators, students, course creators, and educational organizations through our evolving educational programs.
Your JetBrains Academy team
Yes. Student coupons already issued through the bootcamp program will remain valid until their expiration dates. However, after October 1, 2026, you will not be able to request additional student coupons beyond the quantity originally approved with your application.
It will not be possible to request student coupons or free IDE access for bootcamp instructors after October 1, 2026.
Yes. Your access will remain active until its expiration date.
Bootcamps that apply and are approved before October 1, 2026, can receive a final pack of coupons for their students. This should allow eligible organizations to complete courses that begin before the application deadline.
If your course runs beyond the validity of the final license, please review the alternative JetBrains programs listed above.
Students remain eligible for the Graduation Discount until April 1, 2027.
For questions about an existing bootcamp program license or the transition to another JetBrains offering, please contact bootcamps@jetbrains.com.
AI coding is quickly becoming a foundational skill for knowledge workers, not just software engineers. NLW breaks down how to identify software-shaped problems in your work, choose between automating, upgrading, and inventing, and find a practical first project worth building.
The AI Daily Brief helps you understand the most important news and discussions in AI.
Subscribe to the podcast version of The AI Daily Brief wherever you listen: https://pod.link/1680633614
Get it ad free at http://patreon.com/aidailybrief
Learn more about the show https://aidailybrief.ai/