One of the video game studios impacted by Xbox’s layoffs in July has successfully reclaimed its independence, as well as control over its intellectual property.

Compulsion Games, headquartered in Montreal, was founded as an independent studio in 2009 and acquired by Xbox in 2018. Its one release as a member of the Xbox Games Studio network was 2025’s South of Midnight, an action/adventure game set in a magical Deep South.

In July, Microsoft announced the first wave of a planned 3,200 job cuts throughout its Xbox department, alongside plans to spin out or shut down five of its studios. Compulsion Games was one of those five, alongside Double Fine Productions (Psychonauts), Ninja Theory (Hellblade), Undead Labs (State of Decay), and Arkane Studios (Deathloop, Dishonored).

Subsequently, on Aug. 20, Compulsion CEO Guillaume Provost revealed in an interview with GamesBeat’s Dean Takahashi that Compulsion’s management had successfully reacquired the studio, its staff, and the South of Midnight IP on Aug. 11.

South of Midnight is still available via its previous storefronts, including Steam and the PlayStation Network, but is currently self-published by Compulsion.

Provost told GamesBeat that no layoffs had been made at Compulsion as it transitioned to independence, and most of the team elected to stay together.

As for the other studios affected by Xbox’s July 6 layoffs:

• Double Fine Productions, headquartered in San Francisco, confirmed on July 28 that it had laid off 23 employees to return the studio to a “sustainable size.” It is once again fully independent and has control of its IP, such as Psychonauts, and will be exhibiting in Seattle on Labor Day weekend as part of the Penny Arcade Expo.
• Ninja Theory, from Cambridge, England, was one of the more unexpected shutdowns, as it had debuted Senua, a third entry in its Hellblade series, only a few days before the layoffs announcement. It has reportedly been spun off from Microsoft and will continue work on Senua under an unspecified new owner.
• Likewise, Seattle’s Undead Labs is currently under unidentified new ownership and still plans to release the long-anticipated third entry in its zombie survival series State of Decay at some point next year.
• Finally, Arkane’s status has yet to be firmly established. It formerly consisted of two studios, in Austin, Texas and Lyon, France, but its Austin office was closed down as part of a wholly separate wave of Xbox layoffs in May 2024. Several of the affected employees in Texas, including former studio head Harvey Smith, announced on Aug. 19 that they’d founded a new company, Black Pony Immersive, with plans to create new games in the same “immersive sim” subgenre as Dishonored.

Xbox is currently exhibiting at the Gamescom conference in Cologne, Germany.

GitHub Copilot's coding agent can now use Microsoft Teams conversations as context to investigate software tasks, change code, and create pull requests.

The post GitHub Copilot Coding Agent Can Now Use Microsoft Teams Conversations appeared first on TechRepublic.

A language model can perform well on a clean benchmark and still struggle with the cases that matter in production.

Benchmarks and curated datasets are useful when prototyping an LLM-based system. They help teams compare models, test an initial prompt, and determine whether an idea is technically plausible.

But as a system moves closer to production, the evaluation problem changes.

Real inputs are often ambiguous. Labels may be inconsistent. Important context may be missing or truncated. The evaluation set may not reflect the production distribution. Edge cases that rarely appear in benchmarks can become common sources of failure. Even when offline metrics improve, those results may not translate cleanly into production behavior.

We encountered these challenges while evaluating an LLM-based system designed to reduce false positives in GitHub secret scanning.

Secret scanning identifies credentials such as tokens and keys that may have been committed to a repository. Because some candidate strings resemble secrets, but don’t actually represent real credentials, developers may spend time investigating alerts that don’t require remediation.

Rather than determine whether an LLM could classify a string correctly, we needed to understand whether the system could reduce noisy alerts while preserving enough recall to remain safe for a security workflow.

In this post, we share the practices that helped us move from promising prototype results to production. The lessons apply broadly to LLM-powered systems in code analysis, developer tools, security, data analysis, and other production workflows.

1. Start with the product decision, not the model

When an LLM system doesn’t perform as expected, the first instinct is often to adjust its technical components.

Teams may rewrite the prompt, add context, introduce another reasoning step, adjust the surrounding pipeline, or switch models. Before making any of these changes, they should define the decision the evaluation is meant to support.

For our secret-scanning work, we asked:

Can the system reduce false positives while preserving enough recall to be safe in a production security workflow?

To answer this question, teams must decide which mistakes are acceptable, which metrics should drive the product decision, and which guardrails must remain within their defined thresholds.

In secret scanning, incorrectly suppressing a real credential can be more consequential than asking a developer to review an additional alert. We therefore did not treat precision and recall as equally interchangeable metrics.

Our primary objective was to reduce false positives and improve precision. Recall served as a safety constraint: an experiment could advance only if any decrease remained within a predefined acceptable range. This gave us a clear way to evaluate tradeoffs. We selected the configuration that achieved the strongest false-positive reduction while satisfying the recall requirement and meeting our operational guardrails.

We organized the evaluation criteria into three levels:

Primary outcome

This measured the user benefit we were trying to improve:

• False-positive reduction
• Precision

Safety constraint

This prevented an apparent improvement from introducing unacceptable security risk:

• Recall

Operational guardrails

These determined whether the result was practical to deploy:

• Latency
• Cost
• Reliability
• Production compatibility

This distinction prevented us from treating every metric as interchangeable. A change that reduced false positives but significantly lowered recall wasn’t automatically an improvement. Neither was a change that improved quality while making the system too slow, expensive, or difficult to integrate.

Consider two hypothetical experiment results:

Experiment A may look stronger if precision is viewed in isolation. Experiment B is more aligned with the product goal because it improves the developer experience without violating the recall guardrail.

Before evaluating an LLM system, decide what success means for the user and which guardrails the system must respect. We want to generate evidence that supports a product decision.

2. Treat offline evaluation like integration testing

An LLM-based system continues to change after its first successful evaluation, so evaluation should not be a one-time exercise. Teams revise prompts, adopt new models, change how inputs and context are constructed, and refine the surrounding business logic.

Any of these changes can improve the system, introduce a regression, or shift its behavior in an unexpected way.

For that reason, we treated offline evaluation similarly to an end-to-end integration test. We reran it whenever we made a meaningful change to the prompt, model, input construction, or broader system logic.

The evaluation also needed to be repeatable enough that each new result could be compared against a known baseline. For every run, we recorded the prompt, model, dataset version, and system configuration.

This made it possible to answer questions such as:

• Did the new prompt improve precision without reducing recall?
• Did the model upgrade help across the dataset or only within certain categories?
• Did a change to the input or context fix one error pattern while introducing another?
• Did a change to the surrounding logic improve the result consistently, or simply shift where errors appeared?

Without this discipline, teams can easily compare results generated under different conditions and attribute an improvement to the wrong change.

Change one major variable at a time

Repeatability alone is not enough. Experiments also need to be designed so that the cause of a result is clear.

We changed one major variable at a time and compared each run against a known baseline. For example, we evaluated a prompt revision separately from a model upgrade before testing the two together.

This mattered because even small prompt changes could shift model behavior, while a model upgrade could affect quality, cost, latency, or output consistency. If both changed in the same experiment, we would not know which one caused the improvement or regression.

We treated prompts and evaluation configurations like code. We versioned them, recorded what changed, kept previous configurations reproducible, and made rollback possible.

The values in the evaluation run tracking table above shown are hypothetical and included only to illustrate how evaluation runs can be tracked and compared.

Test model upgrades regularly

When an LLM system underperforms, developers often respond by adding more instructions to the prompt. Sometimes that helps, but not always. For example, the prompt may be carrying complexity that comes from the model itself.

A stronger model may perform better with a simpler prompt than an older model does with extensive tuning. Simpler prompts are also easier to understand, test, and maintain.

Model upgrades still need careful evaluation. A new model may improve performance in one category while introducing regressions elsewhere. It may also affect cost, latency, output formatting, or compatibility with the existing pipeline.

The evaluation process should be inexpensive and repeatable enough that testing a new model becomes routine. Any meaningful change to the prompt, model, or pipeline should go through offline evaluation before reaching production.

3. Keep offline evaluation close to production

An offline evaluation is only useful when it resembles the task the system will perform in production.

In a secret-scanning workflow, the model is rarely evaluating one clean, isolated value. It may need to assess a specific candidate alongside surrounding code and other information that is relevant, incomplete, or potentially distracting. Differences in how that information is presented can materially affect the result.

Our offline evaluation therefore needed to preserve the important characteristics of the production task, including:

• The candidate being evaluated
• The surrounding context available to the model
• Relevant supporting information
• The way inputs are formatted and constrained
• The broader system logic around the model

Even small differences can skew the results. A cleaner dataset may exclude ambiguous cases, provide more complete context, or remove nearby values that could distract the model.

Consider a simplified example:

example_token = "sample_value_for_documentation" 
production_api_key = get_secret_from_environment() 
candidate_value = "flagged_value"

Suppose candidate_value is the value the system is expected to assess. The model may instead focus on example_token because its variable name appears more security-relevant, producing a plausible explanation about the wrong value.

This kind of failure is easy to miss when evaluation examples contain only one obvious candidate. It surfaced because the offline evaluation preserved some of the ambiguity and distractions found in real secret-scanning workflows.

The closer the offline pipeline is to the production pipeline, the more useful the evaluation becomes. When the two differ, a strong offline score may simply reflect an easier problem than the one being deployed.

4. Treat production labels as signals, not unquestionable truth

Production data can make an evaluation more representative, but its labels often capture workflow outcomes rather than reliable ground truth. A dismissed or resolved secret-scanning alert, for example, does not necessarily represent a false positive.

A developer might resolve an alert because:

• The credential was rotated
• The risk was accepted
• The alert needed to be cleared to unblock a workflow
• The alert was incorrectly classified

These outcomes may look similar in product data while representing different ground-truth states.

Before using production labels, ask:

• How was the label created?
• Does it match the question the evaluation is trying to answer?
• Are different workflow outcomes being grouped into the same category?

For important or ambiguous subsets, you may need to complete a manual review. You’re not trying to eliminate every imperfect label, but you need to make sure the evaluation data is accurate enough to support the decision being made.

5. Use synthetic and open datasets to fill coverage gaps

Representative production data may be limited, sensitive, or unavailable early in development. Synthetic examples, academic benchmarks, and open datasets can help developers bootstrap an evaluation and expand coverage, but these examples should supplement rather than stand in for production-like data.

With that in mind, synthetic examples can greatly help fill in the gaps for testing cases that are rare or difficult to collect, such as ambiguous inputs, missing context, unusual formatting, and underrepresented failure patterns. A list of credential strings, for example, can test whether a model recognizes common formats, but it cannot fully evaluate how the model reasons about a candidate within real code.

We adapted external examples to match our task and reviewed labels that did not align with our product definition. We also used realistic failure patterns to create targeted synthetic cases involving nearby credential-like values, test code, placeholders, indirect references, and missing context.

6. Use error analysis to find what aggregate metrics hide

Aggregate metrics tell you whether a system improved overall. Error analysis tells you what to change next.

A higher precision score doesn’t reveal whether the remaining errors come from ambiguous inputs, poor prompt framing, missing context, noisy labels, or a narrow dataset.

To understand those problems, inspect the failures.

We reviewed samples of false positives and false negatives and grouped them by their likely source: the model, prompt, input, pipeline, dataset, or label. The recurring issues included several already discussed, such as reasoning about the wrong candidate, missing context, and labels that did not match the evaluation definition.

Each category suggested a different response. Reasoning about the wrong value pointed to prompt or input framing, missing evidence pointed to context construction, and incorrect labels required data cleanup. Repeated domain-specific ambiguity could indicate the need for a clearer product policy or a dedicated evaluation category.

Manually reviewing dozens or hundreds of examples takes time, but it often leads to faster progress. Once a recurring failure pattern is clear, the team can make a targeted change and measure whether it solved the problem.

A useful question for each error is: Did this failure come from the model, prompt, input, pipeline, dataset, or label?

That classification turns a vague quality problem into a concrete engineering task.

7. Use LLM-as-judge to focus human review

Reviewing every evaluation example manually may not scale. LLM-as-judge can reduce that burden by classifying clear cases, identifying potentially mislabeled examples, and prioritizing ambiguous cases for human review. Because the judge can also make mistakes or agree with another model for the wrong reason, its output should be treated as another prediction rather than ground truth.

A safer pattern is to use the judge for triage:

1. Automatically process clear, low-risk cases.
2. Route low-confidence, conflicting, or high-impact cases to human reviewers.
3. Periodically sample high-confidence cases to check for systematic errors.
4. Track disagreement between the judge, the evaluated system, and human reviewers.
5. Version and evaluate the judge prompt like any other model component.

Used this way, the judge concentrates human attention on the cases where review is most likely to change the outcome.

8. What secret scanning taught us

Our goal was to reduce false positives while preserving recall in a security-sensitive workflow. Offline evaluation gave us a controlled way to compare prompt, model, input, and pipeline changes before beginning online experimentation.

Through repeated evaluation and targeted error analysis, we reached a 95% reduction in false positives on the evaluated offline dataset while keeping recall within our defined guardrail. More importantly, we understood how the result had been produced: the evaluation reflected the production task more closely, changes were measured against reproducible baselines, and the remaining failure patterns were documented.

Offline evaluation did not prove how the system would behave in every production scenario. It provided enough structured evidence to justify moving to online experimentation with clearly understood risks and guardrails.

Checklist: Before moving an LLM system toward production

Use this checklist to assess whether your evaluation provides enough evidence to move the system forward. Work through each section to confirm that the goals, data, experiments, and remaining production risks are clearly understood.

Product Goals

• Is the product decision and primary success metric clear?
• Are the safety and operational guardrails defined?

Data and Labels

• Does the evaluation data resemble the production workflow and include difficult cases?
• Do we understand how the labels were created and where human review is needed?

Evaluation Rigor

• Are the prompt, model, dataset, and pipeline versions recorded?
• Are major changes isolated and compared against a known baseline?

Error Analysis and Production Readiness

• Have false positives and false negatives been reviewed by category?
• Can we rerun the evaluation and explain where offline results may differ from production?

Evaluate before you trust

As LLM-based systems move into production, evaluation should become part of the regular engineering workflow. A strong offline evaluation can show whether the product goal has been met under representative conditions, where uncertainty remains, and whether the system is ready for a controlled production rollout.

Production uncertainty is unavoidable. Evaluation makes it visible, measurable, and manageable.

The post How to evaluate LLMs before production appeared first on The GitHub Blog.

AI-assisted development has made it easier to generate frontend code quickly. A developer can ask for a form, a dashboard widget, a settings page, or a modal flow and get a working first draft in seconds. That speed is useful, especially when teams are moving through routine UI work.

But speed creates a problem that’s easy to miss at first. If every AI-generated feature introduces its own components, styling choices, interaction patterns, and accessibility decisions, the frontend can become inconsistent very quickly. A product may end up with forms that handle errors differently, modals that behave differently, buttons that look almost right but don’t behave the same way, and small interaction differences that slowly become expensive. This is where design systems become much more important.

A design system is often described as a way to keep visual design consistent. It provides shared colors, typography, spacing, components, and usage rules. That still matters. But in an AI-assisted workflow, a design system can do more than make interfaces look consistent. It can become the control plane for AI-generated UI.

By control plane, I mean the layer that guides how interfaces are created, what patterns are allowed, and which decisions should not be reinvented every time a new screen is built. A strong design system can encode accessibility, interaction behavior, content guidance, component boundaries, and safe defaults. It gives both developers and coding agents a shared set of rules to work from. Without that layer, AI tools have too much freedom.

AI can generate UI faster than teams can standardize it

Frontend teams already struggle with consistency. Even without AI, it’s common to find several versions of the same pattern inside a product. Some of this happens because teams move fast. Some of it happens because older code stays around for years. Some of it happens because people solve local problems without seeing the whole system.

AI can accelerate that problem. When a coding agent is asked to build a new feature, it usually tries to satisfy the immediate request. If the prompt says “build a filter panel,” it may create a solution that works in isolation but doesn’t match how the rest of the product handles filtering, validation, loading states, or keyboard behavior.

That’s the risk. AI-generated UI can look reasonable in a single pull request while quietly increasing inconsistency across the product. Design systems help by reducing the number of decisions that need to be made from scratch. The question should not be, “Can the AI generate a working dropdown?” The better question is, “Should this feature use the existing dropdown pattern, and does that pattern already handle the behavior we need?” When the answer is yes, the AI should compose the existing pattern rather than inventing a new one.

Design systems are not only component libraries

Many teams treat the design system as a component library. That’s a good start, but it isn’t enough. A component library gives developers reusable building blocks. A design system should also explain when to use those building blocks, how they behave, what content they require, and what constraints they carry. This becomes especially important when AI tools are involved because the agent needs context, not just code.

A button component, for example, is not only a styled element. It carries decisions about hierarchy, states, labels, disabled behavior, loading behavior, and focus visibility. A modal carries decisions about focus movement, escape behavior, headings, accessible names, background interaction, and what happens when it closes. A form field carries decisions about labels, helper text, validation, error messages, required state, and programmatic relationships.

If these rules live only in people’s heads, AI tools will not know them. If they live in the design system, they can be reused, documented, tested, and referenced. The design system becomes a source of truth for both humans and agents.

The design system gives AI safer defaults

AI-generated code is shaped by context. If the agent has no project context, it will rely on general patterns and whatever the developer includes in the prompt. Sometimes that works. Often, it produces code that’s close but not quite aligned with the product.

A design system gives the agent safer defaults. Instead of asking an AI tool to “create a confirmation modal,” the team can instruct it to use the existing modal component, the standard button variants, the approved alert pattern, and the documented content structure for destructive actions. The agent still helps assemble the feature, but the riskiest decisions are already handled by the system.

This matters because many UI decisions aren’t just visual preferences. They affect whether people can use the product. A custom modal might forget to manage focus. A custom button might lose visible focus styles. A custom form field might show an error visually but fail to connect it to the input. These details are easy to miss when a generated interface looks polished, and they are exactly the kind of details that good design-system components can carry by default.

Project instructions should point agents to the design system

Prompts are useful, but they aren’t the whole workflow. If developers have to repeat every design-system rule in every prompt, the process becomes fragile. Someone will forget. Someone will write a shorter prompt. Someone will assume the tool already knows the standard.

A better approach is to make design-system expectations part of the agent’s persistent project context. For some teams, that might mean a CLAUDE.md file, an agent startup file, or another project-level instruction source. The exact mechanism will vary by tool, but the principle is the same: The agent should know the standing rules before it starts generating feature code.

Those rules might include instructions to use existing design-system components before creating new ones, prefer native HTML elements when possible, avoid custom controls without a clear reason, follow documented form and modal patterns, include meaningful loading and error states, and follow the project’s accessibility expectations.

Then the feature prompt can stay focused on what’s unique about the task. The persistent instructions describe how the team builds UI. The task prompt describes what this particular feature needs to do. That separation makes AI-assisted development less dependent on prompt quality alone and more dependent on shared engineering standards.

A design system can reduce review burden

Code review becomes harder when AI generates large amounts of plausible-looking code. Reviewers may see a clean diff and assume the obvious decisions were handled correctly. But frontend quality is full of details that don’t always show up in a quick scan.

A design system can reduce the number of things reviewers need to check manually. If the feature uses the approved modal component, the reviewer doesn’t need to reevaluate focus handling from scratch every time. If the form uses the standard FormField component, the reviewer can have more confidence that labels, descriptions, and error messages are connected properly. The review can shift from “Did the generated code invent this pattern correctly?” to “Did the generated code use the right pattern in the right way?”

That’s a much better question. It also helps teams avoid the slow drift that happens when every feature is slightly different. Small differences may not matter in a prototype. In a production product, they add up. They make the UI harder to maintain, harder to test, and harder for users to learn.

The design system should include behavior

For AI-generated UI, the most useful design systems are the ones that document behavior clearly. A visual example of a component is helpful, but it isn’t enough. Agents and developers also need to know how the component should behave in real situations.

A modal page in the design system should not only show what a modal looks like. It should explain when to use a modal, when not to use one, how focus should behave, what kind of heading is required, and how destructive actions should be confirmed. A form pattern should explain labels, helper text, validation timing, error recovery, and submit behavior.

The more clearly these patterns are documented, the easier they are to use as AI context. That context does not have to be perfect. It just has to be better than asking an agent to guess.

The harder part is discipline

The technical side is only part of the story. Design systems fail when people don’t use them, don’t trust them, or can’t find what they need. AI adds another version of that problem. If an agent can’t discover the right component or doesn’t have enough context to use it correctly, it may generate something new. That doesn’t always mean the agent failed. Sometimes it means the system wasn’t easy enough to follow.

Teams need to make the right path easier than the wrong one. Components should be discoverable. Documentation should be readable. Examples should be realistic. Usage guidance should be specific. Deprecated patterns should be clearly marked. If a component shouldn’t be used anymore, both the agent and the developer should be able to see that.

But there is also a human discipline problem. AI tools don’t automatically know which inconsistencies matter to a product, which patterns are worth protecting, or when a small UI change may affect users who have built habits around the existing interface. Those decisions require people to care about consistency before it breaks. If a team hasn’t already defined that discipline, AI tools are unlikely to supply it on their own. They may make it easier to generate slightly different versions of the same idea unless the team gives them clearer boundaries.

That doesn’t mean the design system should block every new pattern. Sometimes a new pattern is necessary. But new patterns should be intentional, reviewed, and eventually folded back into the system if they become reusable. Without that discipline, AI-generated UI can lead to many almost-standard components. They look close to the system but behave differently, which often makes them harder to clean up than obviously custom code.

The frontend engineer’s role becomes more architectural

As AI tools write more code, frontend engineering becomes less about producing every line by hand and more about shaping the environment in which code is produced.

That includes defining component APIs, documenting patterns, setting accessibility expectations, creating project-level agent instructions, reviewing generated code, and deciding when a new pattern belongs in the design system. These are architectural decisions that influence many features over time.

This is where experienced frontend engineers become even more important. They understand the difference between a component that works once and a component that can be reused safely. They know when a custom interaction is worth the cost. They know where accessibility issues usually hide. They can see when a generated solution works for one feature but doesn’t fit the broader frontend system.

AI can generate code quickly. It can’t decide, on its own, what kind of frontend system a team should have.

The control plane for generated interfaces

AI-generated UI will only become more common, and many teams are already using AI to build interfaces. But will these generated interfaces become more consistent, accessible, and maintainable, or will they just add another layer of drift?

Design systems can help teams choose the better path. When a design system includes clear components, documented behavior, accessibility expectations, tested patterns, and persistent instructions for coding agents, it becomes a control plane for generated UI. It gives AI tools boundaries. It gives developers a shared language. It gives reviewers something concrete to enforce. Most importantly, it gives users a more consistent experience.

The future of AI-assisted frontend development won’t be shaped only by better prompts. It will be shaped by the systems we give those prompts to work within.

. . .

AI use acknowledgment

AI assistance was used lightly for phrasing, editing, and tightening parts of this draft. The article’s ideas, structure, examples, and final review are my own.

Microsoft released PowerToys 0.101 today, and it's a big update with a new Window Hopper utility and numerous updates to existing utilities.

The post PowerToys 0.101 Arrives With Window Hopper, Numerous Updates appeared first on Thurrott.com.