WinApp CLI v0.5.0 greatly expands the UI automation toolkit (recording, touch, pen, and keyboard injection), lets you call Windows Runtime APIs straight from JavaScript and TypeScript with zero native addons, and makes WinUI crashes far easier to diagnose. Get the update by running winget install Microsoft.WinAppCLI or check the repo for other install options.

Here’s what the new version has to offer in depth:

An expanded UI automation toolkit

# Record a session to MP4 (great for demos and bug reports)
winapp ui record -a myapp --duration-sec 10 --output demo.mp4

# Synthesize real touch gestures
winapp ui touch img-map-9f8e -a myapp --gesture pinch --distance 200
winapp ui touch -a myapp --at 100,300 --gesture swipe --to-point 400,300

# Draw and erase with a simulated pen/stylus
winapp ui pen -a myapp --path "100,100 150,120 210,140 260,120"
winapp ui pen -a myapp --at 320,240 --pressure 0.8 --tilt-x 30

# Send real keystrokes, including chords and typed text
winapp ui send-keys "ctrl+a delete" -a myapp
winapp ui send-keys "Hello world" --target txt-name-a1b2 -a myapp

# Hover to reveal a tooltip, then screenshot it
winapp ui hover btn-info-a1b2 -a myapp
winapp ui screenshot -a myapp --capture-screen

# Drag one element onto another (or use raw screen coords)
winapp ui drag itm-card-9f8e itm-slot-2c1a -a myapp
winapp ui drag 120,200 480,200 -a myapp

# Mouse-wheel scroll over an element
winapp ui scroll img-map-a1b2 --wheel -1 -a myapp

Call Windows APIs Directly from JavaScript & TypeScript

npx winapp init . --use-defaults --add-js-bindings

const {
  AppNotificationBuilder,
  AppNotificationManager,
} = require('#winapp/bindings');

const notification = AppNotificationBuilder
  .create()
  .addText('Hello from WinRT — no native addon required')
  .build();

AppNotificationManager.getDefault().show(notification);

We’ll be releasing a full blog post on using these bindings soon, so stay tuned for that. In the meantime, you can check out our guide on calling Windows APIs from JavaScript.

Better WinUI Crash Diagnostics

# Auto-triage on crash; add --symbols for fully-resolved names
winapp run .\build\Debug --debug-output --symbols

Other Notable Changes

• Native PowerShell winget cmdlet: The README now documents installing via the native PowerShell winget cmdlet alongside the classic CLI.
• screenshot auto-creates output directories: Point ui screenshot at a path that doesn’t exist yet and winapp creates it instead of erroring out.
• Claude Code plugin: Added a Claude Code plugin, kept automatically in sync with the GitHub Copilot CLI plugin.
• WinApp VS Code extension: Added AppxManifest editor support; the extension now lives in its own microsoft/WinAppVSCE repository and is available on the Visual Studio Marketplace. Read more about the new manifest editor in the latest WinApp VSCE blog post.

Bug Fixes

• Fixed a certificate bug with multi-component publisher validation
• Fixed ui send-keys --via send-input silently dropping characters on long text
• Fixed ui send-keys --via post-message silently dropping input
• Fixed a dropped \n escape in send-keys text=
• ui touch/ui pen now warn (instead of rejecting) on out-of-window coordinates
• Fixed ui set-value on RichEditBox / TextPattern-only edit controls via a LegacyIAccessible fallback
• Fixed @-prefixed arguments being swallowed by response-file expansion
• Fixed a silent failure in --debug-output and added a warning when --symbols is passed on its own

Breaking Changes

• UI coordinate terminology standardized (v0.5.0): “app” coordinates are now called “screen” coordinates across the ui commands. If your scripts reference the old terminology, update them to match. See the 0.5.0 release notes.

Get started today

The Windows App Development CLI is available now in public preview. Visit our GitHub repository for documentation, guides, and to file issues. We would love to hear your feedback!

Install via WinGet:

winget install microsoft.winappcli

Install via npm:

npm install --save-dev @microsoft/winappcli

Happy coding!

The post Windows App Development CLI v0.5.0 – expanded UI automation, JS/TS bindings, and more appeared first on #ifdef Windows.

At endjin, we maintain Corvus.JsonSchema, and in the OpenAPI series we applied V5's code generation engine to HTTP APIs. Now let's do the same for event-driven messaging.

The messaging serialization problem

If you've built a producer for Kafka or NATS or Service Bus, you've probably written the same boilerplate many times. You serialize a payload to JSON, construct a channel or topic name from a template, set the right headers, publish, and hope the consumer on the other end agrees about the shape of the message.

The problem is that this agreement is informal. The producer serializes a TurnOnOffCommand class. The consumer deserializes into its own copy of what it thinks that class looks like. If someone adds a field on one side but not the other, nothing fails at compile time. You find out at runtime. Worse, the consumer silently ignores the extra field and operates on incomplete data.

AsyncAPI specifications exist to make this contract explicit, in the same way OpenAPI does for HTTP. They describe channels, message payloads with JSON Schema, channel parameters, and security schemes. But until now, there hasn't been a .NET code generator that enforces that contract with schema validation and typed models.

That's what corvusjson asyncapi-generate provides: typed producer classes that validate payloads against their schema before publishing, with the same zero-allocation models and pooled memory from the rest of V5.

Getting started

The full reference documentation is on the Corvus.JsonSchema AsyncAPI guide, and there's an interactive AsyncAPI playground where you can paste a spec and see the generated code. This post focuses on the producer side and the design choices behind it.

dotnet tool install --global Corvus.Json.Cli
dotnet add package Corvus.Text.Json.AsyncApi
dotnet add package Corvus.Text.Json

Then add a transport package for your broker. All transports implement the same IMessageTransport interface, so your producer code doesn't change when you switch brokers:

dotnet add package Corvus.Text.Json.AsyncApi.Nats
dotnet add package Corvus.Text.Json.AsyncApi.Kafka
dotnet add package Corvus.Text.Json.AsyncApi.Amqp
dotnet add package Corvus.Text.Json.AsyncApi.Mqtt
dotnet add package Corvus.Text.Json.AsyncApi.WebSocket
dotnet add package Corvus.Text.Json.AsyncApi.AzureServiceBus

Generating a producer

Given a Streetlights AsyncAPI spec (the canonical AsyncAPI example, equivalent to Petstore for OpenAPI):

corvusjson asyncapi-generate streetlights.json \
    --rootNamespace Streetlights.Client \
    --outputPath ./Generated \
    --mode producer

The generator reads the spec's send operations (or subscribe in AsyncAPI 2.6) and produces a typed producer class, message metadata types, and model types in a .Models sub-namespace. It also writes a lock file that tracks the spec hash for incremental regeneration. If your spec hasn't changed, the next generation run is a no-op.

Publishing with type safety

Here's what using the generated producer looks like. We'll use InMemoryMessageTransport so the example is self-contained, but the API is identical with any real broker:

using System.Text;
using Corvus.Text.Json.AsyncApi;
using Corvus.Text.Json.AsyncApi.Testing;
using Streetlights.Client;
using Streetlights.Client.Models;

await using InMemoryMessageTransport transport = new();
TurnOnProducer producer = new(transport, ValidationMode.Basic);

await producer.PublishTurnOnOffAsync(
    payload: new TurnOnOffPayload.Source((ref TurnOnOffPayload.Builder b) =>
    {
        b.Create(command: "on"u8, sentAt: DateTimeOffset.UtcNow);
    }),
    streetlightId: "lamp-42");

PublishedMessage msg = transport.PublishedMessages[0];
Console.WriteLine($"Channel: {msg.Channel}");
Console.WriteLine($"Payload: {Encoding.UTF8.GetString(msg.PayloadBytes)}");

A few things to notice here.

The streetlightId parameter comes from the channel address template in the spec (smartylighting.streetlights.1.0.action.{streetlightId}.turn.on). The generated code constructs the full channel address using zero-allocation UTF-8 byte manipulation with pooled buffers, avoiding both string concatenation and intermediate allocations. You pass the parameter as a typed argument, and the generator handles the rest.

The payload uses the same Source and Builder pattern from the rest of V5 (if you're not familiar with this approach to constructing JSON objects without allocations, the Corvus.JsonSchema documentation covers it in detail). Required properties are mandatory; optional ones have defaults. The schema says command must be "on" or "off". With validation enabled, the producer checks that before the message leaves your process.

Validation happens before the wire

This is the same philosophy as the OpenAPI client: catch contract violations immediately, with a clear exception, rather than letting a malformed message propagate through your system.

TurnOnProducer producer = new(transport, ValidationMode.Basic);

try
{
    await producer.PublishTurnOnOffAsync(
        payload: new TurnOnOffPayload.Source((ref TurnOnOffPayload.Builder b) =>
        {
            b.Create(command: "invalid-command"u8, sentAt: DateTimeOffset.UtcNow);
        }),
        streetlightId: "lamp-001");
}
catch (ArgumentException ex)
{
    // "Message payload validation failed for 'payload'."
    Console.WriteLine(ex.Message);
}

In Basic mode, validation is a fast boolean schema check. In Detailed mode, you get full evaluation diagnostics with JSON Pointer locations - useful during development. In None mode, validation is skipped entirely for maximum throughput on trusted internal services where you control both ends.

Authentication

When your AsyncAPI spec defines security schemes, pass an authentication provider to the producer. The generated code calls AuthenticateAsync before each publish, so credentials are attached consistently without you wiring it up per-message:

IMessageAuthenticationProvider auth = new UserPasswordAuthenticationProvider(
    username: "service-account",
    password: "kafka-secret");

TurnOnProducer authenticatedProducer = new(transport, ValidationMode.Basic, authProvider: auth);

Swapping transports

Because all transports implement IMessageTransport, switching from the in-memory test transport to a real broker is a one-line change. Your producer code, payload construction, and validation behaviour stay exactly the same:

// NATS
await using NatsMessageTransport transport = await NatsMessageTransport.CreateAsync(new()
{
    Url = "nats://broker.example.com:4222",
});

// Kafka
await using KafkaMessageTransport transport = new(new()
{
    BootstrapServers = "kafka.example.com:9092",
    GroupId = "streetlights-producer",
});

// Azure Service Bus
await using AzureServiceBusMessageTransport transport =
    await AzureServiceBusMessageTransport.CreateAsync(new()
    {
        ConnectionString = "<connection-string>",
        QueueName = "streetlights",
    });

All transports implement IMessageTransport. Your producer code doesn't change.

What's next

In the [ref slug=asyncapi-code-generation-with-corvus-typed-consumers text=next post], we'll look at the consumer side - implementing handler interfaces, error policies, and dead-letter routing for reliable message processing.

The CNCF landscape has hundreds of logos, dozens of categories, and no shortage of people telling you what belongs in your platform. Joep Piscaer, Field CTO at Portainer, joined me on Inside Platform Engineering with a take I don't hear often enough: that the best platform decision is frequently to add nothing at all.

Joep calls the CNCF landscape a candy shop, and it's a comparison that stuck with me. Just because something's on the shelf doesn't mean it belongs in your cart, and the cost of a bad choice doesn't show up at checkout, it shows up months or years later when someone has to support it.

Watch the episode

You can watch the episode with Joep below.

Inside Platform Engineering with Joep Piscaer

The candy shop problem

Joep's argument is simple but easy to forget in practice, which is that every tool you add to your platform is a tool you now have to operate, secure, and explain to whoever inherits it. He's not against new tooling on principle, he's against choosing it reflexively because it's popular or well-marketed. His rule of thumb is that the best choice is often no choice at all, and that resisting the landscape is itself a skill worth developing, not a sign you're falling behind.

Small teams will build lean platforms

One of the more provocative points Joep made was that a platform team with a budget and an SLA will naturally start building for its own survival, not just for its users. His suggested fix leans further than most people may be comfortable with, keep the team small, ideally under eight people (this is very contextual to the organization), so there's only time for the basics. I liked how he framed a bloated platform as a freight ship rather than a speedboat. Once it's big, you can only change course by a single degree at a time, no matter how good your intentions are.

Talk to your users before you build

Joep was adamant that understanding why you're building something matters more than the build itself, going as far as to say a good developer might spend as little as 20% of their time actually writing code. The rest goes into figuring out what's actually needed. This came up multiple times throughout our conversation. His advice for platform teams is to get out of meetings and sit next to the people doing the work, which he only half-jokingly compared to Fisher-Price's old "soul-crushing meeting" toy.

:::figure

:img{ src="/blog/img/inside-platform-engineering-joep-piscaer/soul-crushing-meeting.png" alt="An image of a satirical Fisher-Price toy box parody designed by Daniel Picard." loading="lazy" }

:::

Vibe coding is changing who the platform needs to support

We spent time on how AI-assisted coding is reshaping who your platform needs to serve. Joep's read is that business users are increasingly vibe-coding their own tools because commodity software rarely fits the way their teams actually work. Once they've got something that works, they just want a URL, not a ticket in your backlog or a crash course in Kubernetes. What struck me was Joep's parallel back to Platform Engineering itself. Just as we're told to go and understand what our users actually need rather than guessing, these business users are doing exactly the same thing for themselves, they just build it rather than ask for it. The job for a platform then becomes hiding all of that complexity so those tools can be deployed simply, while staying lean, secure, and compliant underneath. Whether that's a threat to platform teams or an opportunity probably depends on how ready your platform already is to support something it didn't design.

Happy deployments!

:::div{.hint}

Inside Platform Engineering is a series of conversations with Matt Allford and a guest, bringing their own experience and perspective from the world of Platform Engineering.

You can find more episodes on YouTube.

:::

You can follow every enterprise best practice. You can use Clean Architecture, some type of layered architecture, event-driven architecture, microservices, or whatever else is popular.

It can still end up in the same place.

You have a system that is really hard to change.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

When you do make a change, you are afraid you are going to break something. Eventually, the same comment keeps coming up:

We would be better off rewriting this whole thing.

The codebase might not even look that bad. It might be organized and relatively easy to follow. Except it is not easy to follow when you actually need to change it. That is when everything becomes convoluted and complicated.

One of the reasons is probably one of these five architectural decisions. In each case, you are making an expensive decision without having enough information.

That is the common thread between all five. You are making expensive decisions before you truly understand the business or the problems you are trying to solve. Rather than solving an actual problem, you are adding technical nonsense as a solution to a problem you might not even have.

The shift is in the question.

It is not:

What architecture should we use? Should we use Clean Architecture or event driven architecture?

The question should be:

What do we understand about the problem that justifies the architectural decision we are about to make?

1. Choosing an Architecture Before Understanding the Domain

The first bad decision is choosing your architecture, tooling, frameworks, and infrastructure without really understanding the business domain.

If the start of a project conversation is about how you need microservices, event sourcing, CQRS, Kafka, and Kubernetes, but nobody can explain the business processes or workflows, you are doing it backwards.

Can anybody explain the constraints? What consistency issues might exist? What are the possible failure modes? Which parts of the system change frequently? Where are delays acceptable?

If nobody can answer those questions, why are you already deciding on the solution?

Architectural patterns and tooling are not a bingo card. They are not a checklist. Every one of them comes with tradeoffs and added complexity.

You want independent deployability? Now you have distributed operations.

You want to scale parts of the system independently? Now you have network failures to deal with.

You want team autonomy, with different teams managing their own services and deployments? You are going to have cross service and cross team communication. Even if that communication is asynchronous and uses events, there is still a contract that needs to be managed.

You want isolation between boundaries? You are going to have consistency challenges across those boundaries.

There is always a tradeoff. There is always added complexity. You might be getting a solution, but that solution has a cost.

The conversation should not start with whether microservices are good or bad. That is irrelevant. The focus should be on the forces acting against your system and the problems you are trying to solve.

Do you have consistency issues that need to be addressed? Is there a part of the system where the rules change quickly and should be isolated? Does a workflow contain delays that downstream processes need to account for?

Those are the questions you should be asking.

Do not start with, “Should we use this pattern or that pattern?” Use the pattern that fits as a solution to the problem you actually have.

If you are making technical decisions first, you are guessing that you are eventually going to have the problem those decisions are supposed to solve.

2. Building Entity Services Driven by CRUD

The second bad decision is what I call entity services. This is where the system is completely driven by its data model rather than its behavior.

At first glance, the code can look great. It might be well organized. Maybe you have Clean Architecture or some type of layered architecture. You have controllers, services, and repositories that interact with the database.

But the entities are really just tables or object graphs representing customers, orders, products, and other records.

Ultimately, you just have CRUD.

Consider a simple order with an ID, a customer ID, a status, and a total. What does that model tell us?

It tells us that some data exists. It does not tell us anything about how an order behaves.

Can you cancel an order? You would assume so, but how does that happen? Do you just change the status? At what point can the order be shipped? Does it need to be in a particular status? Can it be changed after inventory has been reserved?

The model tells us none of that. It only gives us data.

Where does all the business logic go when the system is driven by CRUD and entity services?

Often, it stays in the end users’ heads. They are the ones who actually understand the processes and workflows.

Some of that logic will make its way into the system, but it is usually sprinkled everywhere. It might be in controllers, message handlers, stored procedures, or increasingly, frontends that contain most of the actual business logic.

CRUD itself is not bad. There are always parts of a system that are inherently CRUD. Referential data and simple lists might have no meaningful behavior behind them. They are CRUD, and they should be treated that way.

The issue is assuming everything is CRUD. It is developing everything as if it were just an updatable record when it is not.

There is a significant difference between an operation called UpdateOrder, where you change properties on an order, and an operation called CancelOrder, where you explicitly communicate that the user intends to cancel an order and provide the reason.

That is not simply CRUD. It expresses behavior and business intent.

The question you need to ask is:

Does the operation I am performing communicate business intent?

In this example, the intent is to cancel the order. Once that intent is explicit, you can start asking useful questions.

Can the order still be cancelled based on how long ago it was placed? Has inventory already been reserved? Has the order been shipped? Does the customer need a refund?

Those questions are derived from the intent of the operation.

When everything is treated as a generic update, you lose that intent. You also lose the natural place where business rules, validation, and workflow decisions should exist.

3. Using the Database Schema as an Integration Point

The third bad decision follows directly from the second. It comes from thinking only about the data model and using the database schema as an integration point.

Imagine two parts of a system: sales and warehouse.

They use a single database instance, but the database is somewhat separated into data related to sales and data related to the warehouse.

Sales interacts with the data it owns. But then it also reaches over and queries data that appears to belong to the warehouse. Potentially worse, it writes directly to warehouse data.

Who actually owns that data?

Is it sales, or is it the warehouse?

At that point, there is no real separation and no clear ownership. The database schema has become the integration point.

Some people will say they do not care. They have one large database, and the different parts of the system need consistency constraints. They want to use the database directly.

That can be fine, as long as you understand the implications. You have no explicit contract. You are treating the database as shared storage.

The problems usually start when you need ownership.

When data is written, who controls how it is written? If a table, collection, or stream needs to change, who owns that change? Who needs to be involved? Which other parts of the system could break?

Without clear ownership, you do not know.

One solution is to have the warehouse expose an API. That API becomes a contract that sales can use to send requests or retrieve information.

A database view can also be a contract. A view can explicitly define how other boundaries are allowed to access data. If you create a view for that purpose and treat it as a contract, it can be a completely valid alternative.

The important distinction is that sharing a database instance is not automatically a problem. Different boundaries can own separate schemas within the same database instance.

The problem is the lack of ownership.

When the database becomes a free for all and anything can read or change any data, you eventually start breaking things.

Suppose an order gets into an invalid state. How did it get there?

You have no idea.

Anything could have changed it. The update might not have gone through the actual process, business rules, and validation required to enter that state correctly.

The order ended up in an invalid state because nobody clearly owned the process of changing it.

4. Creating Abstractions Before You Know What Varies

The fourth bad decision, and arguably my biggest pet peeve, is building abstractions when you do not know what varies.

It usually starts with a reasonable sounding idea.

Maybe we will need to swap something out later.

You begin with a shared service. Then you notice some patterns, so you create a generic workflow engine. After that, you start thinking about what happens if you change the database or messaging library, so you create abstractions around those as well.

Only after all of that do you start building the actual application.

It sounds like it makes sense. We are all taught to value reuse. There is also the constant “what if” question.

What if the underlying provider changes? We will have an abstraction. We can create a new implementation behind it without rewriting large parts of the application.

For example, maybe we want to replace our messaging library. If everything is behind an abstraction, that should be easy.

I understand the reasoning. The problem is that if you only have one implementation of the abstraction you are creating, you probably do not have an abstraction.

You do not have enough information.

You do not understand the other concrete implementations, where they overlap, or where they differ. You are trying to generalize something when you only understand one example.

Messaging libraries are a good example. RabbitMQ and Azure Service Bus have significant differences. Kafka is different again. They might all appear to involve sending and receiving messages, but they have different semantics.

If you start with one of them and immediately build an abstraction around it, that abstraction will be shaped entirely by the only implementation you know.

You did not remove the coupling. You hid it behind an interface.

Are abstractions bad? Of course not.

But a bad abstraction pretends to remove coupling when it does not. It often makes the system harder to understand because developers now need to understand both the abstraction and the concrete technology hidden behind it.

You should create abstractions based on actual variation that you understand, not variation you are imagining might exist someday.

5. Building for Scale You Do Not Have Yet

The fifth bad decision is building for scale you do not have yet.

“Yet” is the important word.

This means paying all the upfront complexity to build for a level or type of scale that you do not know you will ever have.

That does not mean you should design a system that cannot scale if the need arrives. It means you should not pay the entire cost upfront based on hypotheticals.

We might have millions of users.

We might need to replace the database.

This might eventually become a global system.

What do any of those statements actually mean?

When you say the system needs to scale, are you talking about more users, more data, more transactions, or more geographical regions?

“It needs to scale” is not a requirement.

This is not about ignoring scale. It is not about assuming growth will never happen. It is about defining clear boundaries and managing the coupling between those boundaries.

You also need to understand that logical boundaries and physical boundaries are not the same thing.

When you define logical boundaries and avoid coupling them unnecessarily, you give yourself a much better chance of scaling different parts of the system if that need actually appears.

This is also why the conversation about modular monoliths versus microservices is often misleading.

People assume microservices automatically scale better because everything is independent. But a modular monolith and microservices can often be scaled in the same ways.

Once you understand that logical and physical boundaries are different things, the idea that these architectural styles are complete opposites starts to fall apart.

You can define meaningful boundaries without immediately turning every boundary into a separately deployed service.

Start with the boundaries. Decide on the physical deployment model when you have enough information to justify it.

Expensive Decisions Require Real Information

These five decisions cause a lot of pain in software systems.

You make technical decisions first without understanding the domain.

You treat your data model as if it were a domain model, while the actual workflows remain in users’ heads or are scattered throughout the system.

You have no clear ownership, turning the database into a free for all where anything can read or change data anywhere.

You create abstractions without understanding what they are supposed to abstract.

You build for hypothetical scale and pay the cost of complexity before you know whether you will ever need it.

In every case, the problem is the same. You are making an expensive decision without enough information.

Architecture should not start with patterns, frameworks, or infrastructure. It should start with understanding the business, the workflows, the constraints, and the actual problems the system needs to solve.

Then you can make the architectural decision that fits.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Resilience Patterns Can Make Your System Less Resilient
• Stop Blaming Event-Driven Architecture
• Building Mutlitenant Systems with CosmosDB

The post 5 Software Architecture Mistakes That Make Systems Hard to Change appeared first on CodeOpinion.

You asked the AI to write a function. It gave you something that looks right. It even runs. But is it actually good?

Most engineers stop there. They see green and move on. That habit will quietly cause you problems.

AI coding tools like GitHub Copilot, Cursor, and Claude are genuinely useful. But they're non-deterministic, meaning the same prompt can produce different outputs on different days.

They can produce code that's plausible-looking but subtly wrong, or code that works for the happy path but falls apart on edge cases. Without a system for evaluating what the AI gives you, you're essentially shipping untested third-party code and hoping for the best.

This guide walks you through a practical, beginner-friendly approach to evaluating AI-generated code, so you can use these tools with confidence instead of crossed fingers.

What We'll Cover:

• Why AI Code Needs Its Own Evaluation Discipline

• Step One: Define Correctness Before You Generate

• Step Two: Build a Golden Dataset

• Step Three: Measure Reliability, Not Just Correctness

• Step Four: Review for What Tests Can't Catch

• Step Five: Treat Prompt Changes Like Code Changes

• The Mindset That Makes This Work

Why AI Code Needs Its Own Evaluation Discipline

When a human colleague writes code, you can ask them questions. You can read their commit history. You have context.

When an AI writes code, you have none of that. The output arrives fully formed, often with confident-sounding comments, and it's easy to assume competence where there may be none.

The other problem is that AI models are trained on vast amounts of public code, including bad public code. They can reproduce anti-patterns fluently. They can write code that passes a quick read but fails under real-world load, unusual inputs, or security scrutiny.

Evaluating AI code isn't about distrusting AI. It's about applying the same engineering discipline you would to any code that enters your codebase.

Step One: Define Correctness Before You Generate

The single most effective thing you can do is write your tests before you ask the AI to write the implementation. This is the spirit of test-driven development (TDD), and it maps perfectly onto AI-assisted workflows.

When you define correctness upfront, you give yourself an objective measure the moment the code arrives. You're not eyeballing it. You're running it against a contract you wrote yourself.

Here's a simple example. Say you want an AI to write a function that parses a price string like "$12.99" and returns a float. Before prompting the AI, write this:

def test_parse_price():
    assert parse_price("$12.99") == 12.99
    assert parse_price("$0.00") == 0.0
    assert parse_price("$1,299.99") == 1299.99
    assert parse_price("") is None
    assert parse_price("free") is None

Now prompt the AI: "Write a Python function called parse_price that takes a price string like $12.99 or $1,299.99 and returns a float. Return None for invalid input."

Run your tests immediately. The AI might pass four out of five. Now you know exactly what to fix and you didn't have to read a single line of implementation to find the gap.

Step Two: Build a Golden Dataset

A golden dataset is a small collection of inputs with known correct outputs. Think of it as a permanent test suite for any AI feature you build. You start with five or ten examples. You add to it whenever something breaks in production.

This becomes your regression set. Every time you tweak a prompt, upgrade a model, or refactor a pipeline, you run the golden dataset first. If anything breaks, you know immediately.

Here's what a golden dataset might look like for the price parser. A simple JSON file works fine:

[
  { "input": "$12.99",    "expected": 12.99,  "note": "basic case" },
  { "input": "$1,299.99", "expected": 1299.99, "note": "thousands separator" },
  { "input": "12.99",     "expected": 12.99,  "note": "missing dollar sign" },
  { "input": "$ 12.99",   "expected": 12.99,  "note": "space after symbol, from prod bug #142" },
  { "input": "€12.99",    "expected": null,   "note": "unsupported currency" },
  { "input": "free",      "expected": null,   "note": "non-numeric text" },
  { "input": "",          "expected": null,   "note": "empty string" }
]

Each entry is just an input, the correct output, and a short note on why it's there. A script loads the file, runs each input through your function or prompt, and compares results.

For a code-generation use case, the same idea scales up: a folder of input prompts paired with expected output files, diffed by a script. For data extraction, a CSV of sample inputs alongside expected parsed values.

So how do you decide what goes in? Three sources cover most of it.

First, the representative cases: the ordinary inputs your feature handles ninety percent of the time. Second, the boundary cases you can predict upfront, like empty strings, unusual formats, and inputs that should be rejected. Third, and most valuable, real failures.

Notice the $ 12.99 entry above tagged with a production bug number. A user hit that input, the parser choked, and now it's in the dataset forever. That's the test: if an input broke something once, or plausibly could, it earns a permanent spot. If it's just a minor variation of a case you already cover, skip it and keep the dataset small enough to run on every change.

The key discipline is this: don't just fix the failing case. Add it to the golden dataset, fix it, and verify everything else still passes. This is how you stop the whack-a-mole problem where fixing one AI failure silently breaks three others.

Step Three: Measure Reliability, Not Just Correctness

AI outputs aren't deterministic. Correct once doesn't mean correct always. This is especially important if you're embedding AI into a product: a prompt that works 80% of the time will fail your users 20% of the time, and that's not acceptable in production.

The fix is to run your evaluation across multiple samples. Run the same prompt ten times and check how many outputs pass your tests. Tools like promptfoo make this easy to automate. You define your test cases in a config file, point it at your prompt, and it runs the evals and reports pass rates.

Here's what a simple promptfoo config looks like:

prompts:
  - "Parse the following price string and return only a float: {{input}}"

providers:
  - openai:gpt-4o

tests:
  - vars:
      input: "$12.99"
    assert:
      - type: equals
        value: "12.99"
  - vars:
      input: "$1,299.99"
    assert:
      - type: equals
        value: "1299.99"
  - vars:
      input: "free"
    assert:
      - type: equals
        value: "null"

Run this across ten iterations and you'll quickly see if your prompt is brittle. A 100% pass rate across ten runs gives you real confidence. A 70% rate tells you the prompt needs tightening before it goes anywhere near production.

Step Four: Review for What Tests Can't Catch

Tests tell you if code is correct. They don't tell you if it's readable, maintainable, or secure. After your automated checks pass, do a focused human review on three things.

The first is security. AI models can produce code with real vulnerabilities like SQL injection via string concatenation, missing input sanitization, and hardcoded credentials in examples it then forgets to flag. Run AI-generated code through a static analysis tool like Bandit for Python or ESLint with a security plugin for JavaScript as a baseline check.

The second is edge cases the AI didn't consider. Look at the test cases you wrote and ask: what did I not cover? Empty lists, null values, very large inputs, concurrent calls, and so on might not be handled by AI. You need to push it on the edges.

The third is over-engineering. AI sometimes produces elaborate solutions to simple problems. If you asked for a function that checks whether a number is even and got back a class with three methods and a configuration object, that's a red flag.

Complexity is a cost. Prefer simple code you understand over clever code you do not.

Step Five: Treat Prompt Changes Like Code Changes

If you're using AI in a repeatable way like an internal tool, a product feature, or a script you run regularly, your prompts are part of your codebase. Version control them. Review changes to them. Don't just edit a prompt and hope for the best.

The practical habit is to store prompts in files rather than hardcoding them inline, commit them to Git alongside your code, and re-run your golden dataset any time a prompt changes. This takes maybe ten minutes to set up and saves hours of debugging later.

LangSmith and Weights & Biases both offer prompt versioning and eval tracking if you want a more structured solution. For most small projects, a prompts folder in your repo and a simple test runner is enough.

The Mindset That Makes This Work

Every technique in this guide comes down to one shift: treat AI outputs like external inputs, not trusted code.

You wouldn't deploy an API response to production without validating its shape. You wouldn't accept a file upload without checking its contents. AI-generated code deserves the same skepticism, not because the AI is unreliable, but because all external inputs are unreliable, and good engineering accounts for that.

The engineers who get the most out of AI tools aren't the ones who trust them most. They are the ones who verify fastest. Write the tests first. Build the golden dataset. Measure reliability. Review what automation misses. Version your prompts.

Do those five things consistently and you'll ship AI-assisted code with the same confidence you bring to anything else in your stack.

Hope you enjoyed this article. You can connect with me on LinkedIn.