The software code review process is a systematic, peer-driven quality assurance procedure that scrutinizes code when a developer submits a pull or merge request. 

Although poor code review procedures are bemoaned for their inevitable delays and the potential for people to create bottlenecks over inconsequential niggles, good code review catches bugs early, fosters mentoring relationships, and is seen as a democratic way of sharing responsibility.

That’s all there is to it, except that it’s not.

Senior software engineer Mark Dominus tells The New Stack that although many companies do obviously perform code reviews, they don’t “articulate what the result of the review is supposed to be”, and that’s a problem.

“Say you’re a junior engineer, and you’re told for the first time to do a code review. What’s your deliverable? In many places I’ve worked, the deliverable has been left unspecified,” Dominus says. 

In a Mastodon post on this subject, Dominus wrote that “anyone who depends on code review to find bugs is living in a fool’s paradise”, primarily because it is “not in general possible” to find bugs by examining code. He underlined his comment, saying that the “primary purpose of code review” is to find code that will be hard to maintain in the future.

A sure path to code review bad results

He suggests that in scenarios where a software engineer is expected to take a leisurely walking tour through the change set and call out things they don’t like, that leads to bad results, i.e., it encourages people to try and enforce their own preferences about how things should be done, and arguments generally ensue.

“Finding bugs, just from examining a change set, is extremely difficult and requires a certain amount of luck.” – Mark Dominus. 

“Sometimes the boss gives the junior engineer a little more to go on, such as: ‘see if you can find any bugs,’ and this puts the junior in an unpleasant position,” clarifies Dominus. “Maybe they do find a bug, great! But what if they don’t? Finding bugs, just from examining a change set, is extremely difficult and requires a certain amount of luck.”

He suggests a different approach in which software team project leaders drive the code review process by saying: “Look at this for two hours, write a note about anything you don’t understand, and if you don’t get all the way through, make a note about where you stopped.”

A route to delivering real software engineering value

“Now, if the reviewer doesn’t have time to look at everything, you have learned something valuable: the changes are too big or complex to be understood in two hours. Perhaps the change set should be broken into two or more smaller submissions that should be reviewed separately. No matter how junior, inexperienced, or hungover the reviewer is, they can deliver what was asked, and what they deliver will have real engineering value,” explains Dominus. 

That real engineering value is a positive negative in this case. The team has now identified code that another team member can’t understand, or that the change set is too unwieldy for the current code review process… or both.

Code review is theatre, playing soon, near you

Agreeing broadly with these sentiments, Mikhail Golikov, a QA engineer at a high-load e-commerce platform company, tells The New Stack that “code review is theatre” if a team is leaning on it to catch bugs.

“When a team treats review as its bug filter, it ships with a warm feeling and no evidence, then finds out in production,” – Mikhail Golikov. 

“A human skimming a diff cannot see a race condition or a discount that goes negative under load. Review is for code you will hate maintaining later; tests are for code that is broken now,” Golikov underlines. “A reviewer reads a clean diff, sees sensible names and tidy functions, and clicks approve. None of that tells you a key part of the app malfunctions and delivers a broken service to the user.”

Golikov, who is also a maintainer of open-source Python testing tools, explains that the kind of bugs that cause these errors don’t live in the code developers can read. Instead, they live in the states the code gets into at runtime. 

Don’t ship with a warm feeling and no evidence

“When a team treats review as its bug filter, it ships with a warm feeling and no evidence, then finds out in production,” Golikov confirms. “Review is for the question ‘will I hate maintaining this in six months’ and ‘does it actually work’ when a test is run against real inputs, not for a human skimming a pull request at 5 pm.”

There’s little argument among developers and vendors alike that code review processes need to change, especially in the era of AI with the increasing presence of agentic coding tools.

Judah Taub, managing partner at Hetz Ventures, agrees with the narrative here and tells The New Stack that for years, engineering teams have treated code review as the last line of defense against bugs, but that was never really their strength.

“The role of the engineer continues to move further away from the actual code and toward validating architecture, intent and business logic. In the future, even that may be automated,” Judah Taub.

“Code review catches style, architecture, readability and maintainability,” Taub says. “Tests catch bugs. Production catches everything else. The best engineering organizations don’t rely on another developer spotting a subtle edge case buried in hundreds of lines of code – they build systems that automatically verify correctness long before a pull request reaches another human. Code review is to ensure the next engineer can work on the code seamlessly.”

The future for code review

As AI-generated code now enters the fray, Taub thinks the inevitable new equilibrium will be one in which humans review less code directly and increasingly review AI reviews of code instead. 

“The role of the engineer continues to move further away from the actual code and toward validating architecture, intent, and business logic. In the future, even that may be automated,” he predicts.

We can surely extract key trends here and say that software engineering teams certainly need to get out of the nineties, if that’s where they currently reside. 

The combined revolutions of cloud-native, platform engineering, and vibe coding onward to agentic coding (you can add in big data, DevOps, and the standardization towards enterprise open source if you wish) have changed the way software developers work, so there should arguably be a commensurate change in the way they review what just happened.

The post The code review bug hunt is dead. Here’s what developers get wrong. appeared first on The New Stack.

Google was ordered to pay almost $2 billion this week to Pricerunner, reports Bloomberg: The Patent and Market Court in Stockholm, which issued the judgment on Wednesday, dismissed most parts of the claim in which Pricerunner sought 80 billion Swedish kronor, or roughly $8.2 billion, in the wake of a European Union antitrust crackdown... The Swedish price-comparison website argued that Google has been abusing its dominant position as a search engine by favoring its own comparison shopping service over competing portals for more than a decade. Wednesday's award compensates for lost revenue caused by Google's preferential treatment of its own comparison-shopping service over independent price-comparison services, conduct that also drives up costs for consumers, [Pricerunner owner] Klarna said in a statement after the judgment... A Google spokesperson said the company doesn't agree with the court's decision and will consider its legal options. [The ruling can be appealed.] Changes implemented in 2017 to Google's platform are working and generating growth and jobs for hundreds of comparison shopping services operating more than 1500 websites across Europe, according to the statement. The litigation is linked to a 2017 decision by the European Commission to fine Google €2.4 billion for illegally leveraging its search dominance to give its own shopping service an edge. The EU decision unleashed a wave of so-called follow-on suits, which were delayed for years as Google appealed the EU fine. Two years ago the EU's top tribunal confirmed that the company did violate antitrust laws — meaning EU-based plaintiffs no longer have to prove that in court. A Berlin court last year ordered the tech giant to pay €573 million in damages to two German price-comparison websites, a ruling Google appealed. Similar cases are pending across Europe.

Read more of this story at Slashdot.

Every business runs on a database, but not everyone who needs an answer from the database speaks SQL. Data Analysts wait on engineers, and stakeholders wait on analysts, and by the time the query runs, the decision window has passed.

LangChain's SQL integration fixes this, translating plain English questions like "Which product category had the highest revenue last year' into valid SQL, executing it, and returning a human-readable answer.

Building a database deployment pipeline with AI assistance sounds straightforward…until the AI starts treating temporary security shortcuts as permanent architecture.

In part three of his series on building an app and database almost entirely through an LLM, Grant Fritchey walks through using GitHub Copilot and VSCode to configure Redgate Flyway for PostgreSQL on Azure Flexible Server, provision a self-hosted GitHub Actions runner inside a private VNet, and wire it all together into an automated CI/CD pipeline.

The result is impressively functional — but the process surfaces a critical lesson: AI coding assistants are fast, capable, and dangerously confident about security decisions they have no business making alone.

Find the full repo, including every prompt, on GitHub.

Since I’m starting development on the database side of dbRosetta, and since I’m much more comfortable with databases than with code, I’m going to stay in the database sphere for a bit.

My next step is to work with the AI to get a pipeline in place to deploy our database code to Azure Flexible Server. I’m using Redgate Flyway as the database automation tool because I’m comfortable with it (also, because it’s the best DB deployment tool out there — fight me).

Setting up Flyway

I went straight to Microsoft Copilot – my main AI companion for this experiment – and asked it to generate the prompt I’d need for VSCode. However, I did a bad job on my own prompt. See if you can spot the problem:

Can you give me a prompt for VSCode, using the agreed prompt structure that you helped me build, to create a Flyway Enterprise baseline and initial set of migrations based on reverse engineering using Flyway Compare technology, to capture the existing database we created earlier through the Flyway CLI?

It might not be obvious why this prompt is bad. In fact, Copilot generated a perfectly reasonable VSCode prompt from that – versioned migration naming, non-destructive baseline, CI-friendly output, validation step. So, what’s wrong?

Well, I didn’t specify what I wanted for development versus production environments – so it’s treating them all the same. This is something we’ll need to fix later. Isn’t it great that, even with AI, we can still introduce code debt!

Deploy with confidence with Redgate Flyway

The best-in-class database migration and version control tool – designed for for stable, governed database changes.

Learn more & try for free

Giving the bad prompt to VSCode

I fed the prompt into VSCode. It went to work. It knows how to connect up to my Azure Flexible Server, so it ran `pg_dump` to get the SQL file for the baseline (I’d assumed it would use Flyway for this – it surprised me). And it then built out the `*.toml` file for controlling Flyway, and ran a baseline command against my database using the migrations it had created.

It didn’t work flawlessly. It made a mistake in the configuration. Then, since it had provided multiple test steps, it fixed it. I’m not kidding. So, yeah, I’ve got a Flyway migration harness ready to go.

There may be more that’s needed down the road to automate this fully, but I’m not going to lie – I’m surprised at what I’ve got so far. Pleasantly so.

Creating a pipeline

When I’ve set these up myself, I get Flyway configured and functional first, then figure out how to get it to run inside the pipeline I create. Talking with Copilot, it wanted me to prompt it, to enable it to build the GitHub Actions. It is weird talking to one to talk to the other, but it’s working gangbusters so far!

I did have to remind it that, only earlier, it had told me we’d need to set up a VM (virtual machine) inside my VNet (virtual network) to be a self-hosted GitHub runner…

I’ve noticed these things don’t do a flawless job of maintaining context. It falls back on us – at least in my experience so far – to be extremely thorough in our prompts in order to get the best results. However, it remembers again that we’re behind a firewall.

(And yes, before anyone asks, I’m using the paid tier for this adventure. I have to imagine the free tier may be even more forgetful.)

What Copilot eventually produced

The prompt Copilot eventually produced was a self-hosted runner spec: cheapest viable Azure VM (Standard_B1s, ~$7.50/month), no public IP, system-managed identity, secrets fetched from Key Vault at runtime via OpenID Connect (OIDC). Sensible. Over to VSCode…

This time it chose to create everything as bash scripts. The first one ran cleanly and provisioned the VM, security group, network interface and VNet. On to the next script…

This one involved a lot of setup within the VM, where VSCode can’t help me as an agent — so I did a lot of typing from VSCode prompts. Now I’m the AI. Ha!

The interesting thing was the troubleshooting. The AI didn’t get everything right the first time, although I may have introduced a typo or three. Troubleshooting, however, was actually pretty outstanding. I’d feed it errors and it would feed me solutions (or additional checks followed by solutions.)

Most of the rest of this was very interactive. Some work was readily done by GitHub Copilot acting as an agent inside VSCode. And some was me getting certificates and other things that couldn’t readily be retrieved through Copilot.

When temporary becomes permanent (the security issues triggered by the AI)

It was interesting how much I had to remind the AI of things. For example, it wanted to test what we’d done, but only supported a `Development` or `Production` branch. Its first inclination was to add the branch we were working on, but that branch wouldn’t be permanent, so I had to remind it that we could do it for a test but it had to be easily undone.

Then, we hit a security problem on our PostgreSQL cluster.

It wanted to use the public IP address we had temporarily added to our VM. It was like, once a thing was done and available, that thing was permanently part of the project – despite the stated purpose being a temporary thing for setup.

The conversation went something like this:

Me: Instead of the public IP address, shouldn’t we do something within Azure to let the resource access it since the IP address is temporary for the setup?

Copilot: Excellent point! You’re absolutely right. Since both the VM and PostgreSQL are in Azure, we should use VNet integration or Azure service endpoints instead of relying on the public IP.

[Copilot then investigated and pivoted to suggesting the `0.0.0.0` “AllowAzureServices” firewall rule.]

Copilot: Perfect! The firewall rule `0.0.0.0` is a special Azure rule that allows all Azure services (including VMs in the same subscription) to connect to the PostgreSQL server. This is the Azure-native way to allow internal Azure resources to communicate.

The latter is the kind of suggestion you want to pay close attention to – but why?

Well, the 0.0.0.0 rule allows access from *any* Azure subscription, not just yours. It’s one of the more commonly cited misconfigurations in cloud security write-ups.

For a hobby project with no production data, the risk is small. For an engineer in their first job, however, copying this pattern into a real codebase because the AI called it the “Azure-native way,” the risk is most certainly not small.

In this instance, I pushed back, but only because I knew to push back. We ended up with a sensible VNet-based path instead.

This was actually a common pattern across the whole step. Once the AI had something working – a temporary IP rule, a test branch in a deployment config, a one-off setup script – its instinct was to fold it into the architecture rather than flag it for removal.

That’s simply because scaffolding and architecture look identical to an LLM (large language model). Both are *code that exists in the project*. The distinction lives in your head, not the model’s.

Summary: I’m very impressed – but security continues to be an issue

Overall, I’m blown away. Most snags were not because the AI hallucinated, or was horribly wrong, but instead from licensing, permissions, and stuff I didn’t define while I was writing prompts to both AIs. The silly thing was pretty darn helpful.

I let it do troubleshooting I was perfectly capable of doing myself, because I wanted to see it work things out. And it did. Quite well.

AI is coming for your job… IF you’re not real good at your job. I could have done every bit of this work, but using GitHub Copilot made it a lot faster and easier.

However, I had to be on top of it. I had to track everything, because it didn’t always. It was more than ready to compromise security, multiple times. I couldn’t have just let it run. And this is a point I keep coming back to. The AI is a particularly brilliant, but extremely junior, developer on my team.

It’s lightning fast. It’s often correct. But it’s confidently wrong in exactly the places that matter most – and especially around the things that are “temporary for now”, but that no junior developer has the experience to flag.

You don’t let a junior developer near production secrets without supervision. You don’t let them make the call on networking topology. The same goes here, with the AI. That’s why humans are still so valuable (at least for the moment). We’re not just employed – we’re necessary!

In the next article, we’ll begin actually building the application. See you there.

Enjoying this article? Subscribe to the Simple Talk newsletter

Get selected articles, event information, podcasts and other industry content delivered straight to your inbox.

Subscribe now

FAQs: AI can build the pipeline, but don't trust it for security (building an app with an AI LLM, part three)

1. What is Redgate Flyway and why is it used for database deployments?

Redgate Flyway is a database migration and version control tool. It manages schema changes through numbered migration scripts, making deployments repeatable and auditable. In this project, Flyway Enterprise is used alongside Azure PostgreSQL Flexible Server to create a baseline from an existing database and apply controlled migrations through a CI/CD pipeline.

2. Can GitHub Copilot set up a GitHub Actions pipeline automatically?

Partially. Copilot can generate the YAML, bash scripts, and configuration files needed for a pipeline, and can troubleshoot errors when you feed them back. However, it requires careful human oversight — in this project it needed reminders about VNet constraints, self-hosted runner requirements, and security configurations it was getting wrong.

3. What security risks can AI assistants introduce in cloud infrastructure setup?

A common pattern is treating temporary scaffolding as permanent architecture. In this case, Copilot suggested using the 0.0.0.0 PostgreSQL firewall rule — which grants access to all Azure subscriptions, not just your own — and described it as the “Azure-native” approach. A developer without prior cloud security experience could easily copy that pattern into production. The correct solution was a VNet-based private connection.

4. Do AI coding tools maintain context across a long conversation?

Not reliably. In this project, Copilot would forget earlier constraints (such as the self-hosted runner requirement or the temporary nature of a public IP) and needed to be re-prompted. Writing thorough, explicit prompts and tracking decisions yourself remains essential.

5. Is AI going to replace database engineers?

No – not experienced ones. AI functions like a fast, confident junior developer — useful for acceleration, but dangerous without supervision. It can’t distinguish scaffolding from production architecture, and it lacks the instinct to flag security or design decisions for review. Human expertise remains the necessary check.

The post AI can build the pipeline, but don’t trust it for security (building an app with an AI LLM, part three) appeared first on Simple Talk.

In this episode, Alex Zenla (CTO/Co-founder, Edera) challenges the "laissez-faire" attitude toward modern infrastructure. She promotes "spite-driven development", building software to solve genuine technical pain points rather than passively accepting flawed abstractions, as a philosophy of improving the world of software.

By Alex Zenla

Ask most people what an eval is for and you’ll get some version of “testing”. You run it before you ship to check the app works, like a unit test. That’s true, and it’s also about a third of the story.

Evals do three different jobs, and the testing one is just the first. The same basic machinery - judge an output, score it - gets pointed at three completely different problems depending on when it runs and what happens next with the result. Treat all three as “testing” and you’ll under-use the other two, which is where a lot of the value actually is.

Want it to hand? Download the infographic as a PDF.

The three run left to right from before release to in production. The eval itself can be near enough identical across all three - same prompt, same judge. What changes is where it sits in the lifecycle and what you do with the score. That’s the bit worth getting straight.

Pre-release testing

This is the one everyone already knows. Before you ship, you run your evals over a golden dataset and check the scores clear the bar. It’s eval-driven development, and it works exactly like the unit tests you already write, just with a fuzzier kind of assertion at the end.

The value here is the same value tests have always had: you catch the regression while it’s cheap. A prompt change that quietly broke the tone of every reply, a model upgrade that made answers less grounded - you find it on your own dataset, before a single user is exposed to it. It’s the cheapest possible place to catch a problem, because the only thing that’s been harmed is a test run.

Production monitoring

Here’s where it gets interesting, and where most teams stop too early. Once the app is live, the evals don’t retire. They keep running, scoring real traffic as it flows through.

This matters because real users don’t behave like your test set. They ask things you never thought of, in combinations you never tried, and the world drifts out from under your carefully tuned golden dataset the moment you ship. Pre-release testing tells you the app worked on the inputs you imagined. Monitoring tells you whether it’s working on the inputs users actually bring, right now. You score live traffic continuously and you alert when quality drops, so you find out a model’s having a bad day from a dashboard rather than from an angry post going round. Same eval as the testing step, pointed at production instead of a fixed dataset.

Inline guardrails

The third job is the most aggressive. Here the eval runs inside the request path, checking a response in real time before it’s allowed out the door. And critically, the score doesn’t just get logged - it gets acted on, immediately.

If the check fails, you can block the response, regenerate it, or fall back to something safe, all before the user sees a thing. This is the eval as a bouncer rather than an auditor. The trade-off is that it has to be fast and it has to be cheap, because now it’s sitting in the critical path of every single request, adding latency to real user traffic. So you reserve guardrails for the things that genuinely can’t be allowed through - leaking personal data, going wildly off-topic, saying something unsafe - rather than every quality nicety you’d happily check after the fact.

Same tool, three jobs

The thing to take away is that “eval” isn’t one activity. It’s one tool doing three jobs, and the difference between them is timing and consequence. Pre-release, you test against a fixed set and block the release if it fails. In production, you monitor live traffic and alert if it dips. In the request path, you guard each response and act on it in real time.

Most teams build the first, occasionally build the second, and forget the third exists. But an app you can actually trust in front of real users usually needs all three - the test to ship with confidence, the monitor to know it’s still working, and the guardrail to catch the things that must never get out. It’s the same reason evals belong at every stage of the lifecycle and not just before launch: the work isn’t done when you ship, it just changes job.