Sr. Content Developer at Microsoft, working remotely in PA, TechBash conference organizer, former Microsoft MVP, Husband, Dad and Geek.
165053 stories
·
33 followers

Run AI Agents in TeamCity

1 Share

AI coding agents don’t have to run only on a developer’s machine. You can also make them part of your CI/CD workflows to review code, investigate failed builds, update documentation, or handle other repository tasks.

The new experimental AI Agents plugin for TeamCity lets you run Codex and Claude Code directly in TeamCity builds and connect them to the LLM provider of your choice.

The plugin installs the coding agent on the TeamCity build agent, provides credentials safely for the duration of the build, and collects usage statistics and agent trajectories.

When a build starts, the plugin checks whether the selected client is available and installs it if necessary. You can then call codex or claude from a regular TeamCity Command Line build step. 

Install the plugin

What does the AI Agents plugin do?

The plugin handles much of the setup needed to run coding agents reliably in CI.

Connect to your preferred LLM provider

The plugin stores your provider credentials as a TeamCity connection. It currently supports:

  • JetBrains Air
  • OpenAI
  • Anthropic
  • LiteLLM
  • OpenRouter

This is not limited to models provided directly by these services. Gateways such as OpenRouter and LiteLLM give you access to a much wider range of models, so you can choose models and providers that make sense for your team.

Run Codex and Claude Code without preinstalling them

You don’t have to install Codex or Claude Code manually on every TeamCity build agent. When a build starts, the plugin checks whether the selected client is available and installs it if necessary. You can then call codex or claude from a regular TeamCity Command Line build step.

Keep provider credentials out of your build agents

TeamCity build agents are often reused across builds, so leaving provider credentials on an agent can be a security risk. The plugin provides the necessary credentials to the TeamCity agent only for the duration of the build and removes them afterward. 

Your provider credentials are managed through the TeamCity connection rather than permanently stored on individual build agents.

Track token usage

The plugin collects usage statistics from agent runs and publishes them as TeamCity statistical values, including prompt, completion, cached, and total token counts.

Because these are regular TeamCity statistics, you can use them to track AI usage over time and create charts. This can also help you spot unexpected changes in token consumption and investigate possible cost anomalies.

Inspect and analyze agent trajectories

When an AI agent does something unexpected, seeing its final response is often not enough. You need to understand what happened along the way.

The plugin records the agent trajectory and makes it available in two formats:

  •  HTML gives you a readable view of the agent’s messages, tool calls, and token usage.
  •  JSON makes trajectories available for programmatic analysis.

The JSON format is especially useful when you want to analyze multiple agent runs together, like to understand why an agent behaves a certain way or how to adjust the original prompt to get better results.

You can also use a local coding agent together with the TeamCity CLI to retrieve trajectories from multiple builds and analyze them in batches. This gives you a practical feedback loop: run an agent in CI, collect its trajectories, analyze what happened across multiple runs, adjust the prompt, and try again.

This makes trajectories useful both for debugging individual builds and improving the agent workflows you run in CI.

Set up the plugin

Install the plugin from JetBrains Marketplace and add it to your TeamCity server.

1. Create a connection 

First, you need to set up a connection. The plugin supports various LLM providers and gateways. Open Project Settings | Connections. 

Supported providers:

  • OpenAI
  • Anthropic
  • OpenRouter
  • LiteLLM
  • JetBrains Air

Enter the credentials required for your connection and save it.

If you’re using JetBrains Air, enter your service account client ID and client secret. Each team can use its own service account, which makes it possible to track shared AI usage and spending.

2. Add the AI Agent build feature

Open the build configuration where you want to use the coding agent and add the AI Agent build feature.

Select Codex or Claude Code, then select the provider connection.

3. Add your prompt

Add a Command Line build step and call codex or claude as usual. 

With Codex:

codex exec --ephemeral --skip-git-repo-check "summarize this repository"

With Claude Code:

claude -p "summarize this repository"

The plugin finds or installs the selected client and prepares its configuration for the build. You can replace these example prompts with whatever task you want the agent to perform as part of your build.

Review what the agent did

After the build finishes, the plugin publishes telemetry and agent trajectory files as hidden build artifacts under: .teamcity/ai-agents/.

It also publishes statistical values, such as prompt, completion, cached, and total token counts.

For a more convenient view, the plugin generates an HTML report showing the agent’s messages, tool calls, and token usage.

You can make this report available directly from the TeamCity build results. 

Go to Project Settings | Report Tabs, create a build report tab, and use: .teamcity/ai-agents/trajectory.html as the start page and “AI Agent Trajectory” as the name. This gives you a tab where you can inspect what the agent did during the build.

A note about sensitive data

The generated artifacts may contain prompts, source code, and model output. Treat them the same way you would other sensitive build artifacts and make sure access is appropriately restricted.

Try the plugin

The plugin is currently experimental, and its functionality may change as we continue developing it.

You can install it from JetBrains Marketplace and try running Codex or Claude Code in your TeamCity builds.

Install the plugin

We’d love to hear how you’re using it and what you’d like us to improve. Share your feedback in the TeamCity Slack community.

Read the whole story
alvinashcraft
just a second ago
reply
Pennsylvania, USA
Share this story
Delete

Beyond the Hype: Will AI make legacy a thing of the past? by Colin Eberhardt

1 Share

In this episode, I’m joined by James Camilleri and Alan Froggatt to ask whether AI could make legacy software a thing of the past. We explore how AI is changing the economics of modernisation, making previously expensive and time-consuming migrations dramatically faster.

But writing code is only part of the challenge. AI can help document unfamiliar systems, generate tests and accelerate migration. However, it can’t easily capture the unwritten knowledge of the people who understand how those systems really work or why they matter.

Together, we consider a future where legacy becomes much easier to escape, while asking an equally important question: just because AI makes it possible to replace an ageing system, does that mean you should?

Read the whole story
alvinashcraft
16 seconds ago
reply
Pennsylvania, USA
Share this story
Delete

The Agent Knows. It Just Doesn’t Always Do It

1 Share

The more I work with agents, the more I understand how big the task of making sure they work is. Like, Jupiter big.

And how much of our practices need to change in order to at least have some idea of what we ship to production.

Here’s an example straight from my TDD agent. I taught it well. It knows how to generate the tests first, run them, and see them fail. Only then should it move on to generating the code.

Well, it knows. What it does depends on how it’s feeling that day.

So the bug in this case is not working in the right order. Only, it’s hard to see the bug. Because the agent created working software, only not just how I wished it to.

And how do I know there was a bug? I needed to get into the console logs to see what it was actually doing.

The second example came from the same agent. As we all know, in classic TDD (and my agent is really up there with the classics), you implement just enough code to pass the tests. Anything extra is a no-no.

Again, depending on how the genie felt that day, it did what I asked. Or not.

How did I know? I started running a targeted coverage check as a step after the Green phase. It told me if there was some extra untested code added.

These are just two examples of how to make sure the agent “works”. We need to rummage through logs, looking for evidence. Or set traps – like my coverage tool – to see if it gets out of line.

And these have to run constantly, with every run. And even that isn’t enough. Because I can set traps and look for information – if I know what to look for, and where it will appear.

What about the rest?

In classic testing it’s called exploration. We learn what the app does, and how it behaves. And if we find something we like, or something we want to keep out, we put automated tests around it so we’ll know when it comes back.

In our language – set traps based on our exploration.

But wait, that’s not enough either.

Because, as we know, things can change from under us. Models drift, interpretation changes. The equivalent of automation – pass/fail – may not be enough.

So we need to track trends of behavior.

For example – with my coverage trap, I know when extra code is added. The problem is, in reality, sometimes it’s ok to add extra code. So the event is fired – extra code was added – and I decide if it’s ok or not.

But does it happen a lot? Because if it does, then it’s a bug. If not, it’s a tolerated exception.

I track the behavior over time to see if it’s a real problem.

So where a simple pass/fail check was enough, I need more. More checks, more traps and more trackers.

Our testing practices need to change. We can’t continue to test agents the old way.


If your team is building agents and wondering how to know they work, that’s where I come in – mentoring and training for your team.

Mentoring by topic – API testing, unit testing, testing with AI, testing AI features. On your own code and your own decisions. Or training delivered where your team needs the ground covered.
Tell me what you’re dealing with. A couple of lines is enough.

The post The Agent Knows. It Just Doesn’t Always Do It first appeared on TestinGil.
Read the whole story
alvinashcraft
27 seconds ago
reply
Pennsylvania, USA
Share this story
Delete

Random.Code() - Adding Generics Support to SaferAccessors, Part 5

1 Share
From: Jason Bock
Duration: 1:15:12
Views: 49

In this stream, I hope to wrap up this feature by getting constraints into the extension definitions, and then maybe move on to other features.

https://github.com/JasonBock/SaferAccessors/issues/4

#dotnet #csharp

Read the whole story
alvinashcraft
6 hours ago
reply
Pennsylvania, USA
Share this story
Delete

Critter Stack Roadmap Update for November 2026

1 Share

I haven’t done one of these posts for awhile, and the year is winding down fast. I personally have the Thanksgiving week, a big trip down under for the Yow Conference in Brisbane and Melbourne, and a fervent wish to take a lot more time off during the holidays than I have for years this year.

The big ticket items are major version releases of the major Critter Stack tools to add .NET 11 and drop .NET 9:

  • Weasel 10.0 – a bit of modernization, some .NET 11 related optimizations, Npgsql 11 when it lands
  • Marten 10.0 – lots of optimizations, some tidying up of the public API that’s kind of bloated, and flipping some defaults for improved observability. A big, unexpected win is going to be significant performance optimizations to the DCB support
  • Fisher 2.0
  • Polecat 6.0 – really the same new things as Marten
  • Wolverine 7.0 – lots of optimizations, using the Dekaf library for Kafka, flipping some defaults for improved observability.
  • CritterWatch 2.0 – using the new major versions above
  • Bobcat 1.1 – enabling .NET 11 mostly, but there are some significant improvements that got missing in 1.0

I don’t have a perfect timeline for when this will all be done, but we have alpha releases out for everything. We’re waiting a bit on some upstream dependencies like Npgsql 11 before we finish the official releases. I’m also abusing my poor MBP doing a lot of performance testing and optimization work that will continue for awhile.

For all of the tools, we’re trying to take advantage of new features in .NET 11 like the runtime asynchronous compilation to save on memory usage (which we’ve proven to benefit Wolverine at least) and some improvements for serialization. We’re investing much more this time in performance testing as we go than we were able to in the last round of major releases. We’re also doing a lot of work to try to lift duplicated code from Marten, Fisher, and Polecat into shared libraries as a way of easing support over time.

My big initiative for JasperFx right now is writing a new version of our internal CRM portal application to try to automate much more of JasperFx’s business. This is partially because I despise paperwork of all kinds and JasperFx has fortunately gained enough business that spreadsheets are no longer enough. I am also using this work to do more dogfooding of our greater “Event Modeling” story across the Critter Stack. We’ll put most of this code and work into a public repository and use this to rebuild a new set of tutorials on Event Modeling, Spec Driven Development, and using AI agents to scaffold Critter Stack applications.

Now, some bigger themes for new work:

Improvements to Wolverine’s Saga support and really its ability to support long running workflows all the way around. To be blunt, I’m looking to make Wolverine the absolute best server side platform for long running workflows out there, so we’re planning some improvements to the Saga support for diagnostics, error handling, compensation action support, more visualization and user interface or MCP management in CritterWatch, and even finally dipping our toes into a routing slip capability. I’m experimenting a little bit with an alternative syntax to define sagas through something like the Azure Durable Functions syntax that still functions as a Wolverine Saga behind the scenes. We will absolutely have an event sourced Saga capability that I think will pair very well with CritterWatch. I’m also going to experiment with using Wolverine’s virtual actor capability for long running tasks like file imports that don’t necessarily fit well into the existing messaging.

For the event stores, I think there’s an opportunity to expand our support for “cold storage” of event data to formal ways to “un-thaw” archived data on demand. A pair of JasperFx clients have now expressed interest in that.

Adding 2nd level caching to our event sourced command handlers. See this issue that’s been in our backlog quite awhile (that I would heavily consider dogfooding in CritterWatch). This will have to pair up with our global partitioning feature in Wolverine. More or less, the general idea is in some cases, we’ll basically have a very highly optimized architecture for event streaming within a cluster where operations are emitting events to persist in Marten or Polecat.

I guess I need to just come out and say that I’m somewhat interested in extending Wolverine into actor-based architectures. I think we have the foundation to do that already, especially after we had to harden our leadership election and agent assignment subsystem this year.

We need to break down and have an easy recipe for people to designate roles to different nodes in what’s otherwise the same Wolverine application. The need here is people who want only some nodes to run the transactional inbox/outbox agents or only some nodes to run messaging while maybe some nodes only handle HTTP traffic. Today you really need to do that with two logical Wolverine applications with isolated Wolverine node tracking storage.

I’m interested in adding formal support for the latest version of Quartz.Net in Wolverine and CritterWatch. Especially after Quartz.Net adopted Weasel for their own database schema management.

I don’t really know what to say about Stoat. I think the “AI Agent Wrangler” tool market is already way too crowded and it’s probably a fool’s game to play too much in that pool. The one thing that Stoat probably does earn its keep is managing and scheduling agent behavior and quality gates on a single box. Stoat might settle into just being a JasperFx internal tool, or it might grow into something that sits between the local work and newer AI planning tools like Linear.

Anyway, I’m thinking out loud here. I’m always interested in any feedback!



Read the whole story
alvinashcraft
6 hours ago
reply
Pennsylvania, USA
Share this story
Delete

AIStoryBuilders 4.0 is in the Microsoft Store: everything that's new for Windows

1 Share
AIStoryBuilders 4.0 for Windows is a free update in the Microsoft Store. It brings a Knowledgebase for every story, a writing voice the AI follows, Story Chat changes you approve before they happen, local models with Ollama, and a redesigned workspace. Your stories carry over unchanged.
Read the whole story
alvinashcraft
6 hours ago
reply
Pennsylvania, USA
Share this story
Delete
Next Page of Stories