A small team at GitLab spent the past few weeks running an experiment: Could we use AI agents to migrate part of our legacy rate-limiting system without dropping the safety bar?

Short answer: yes. AI agents do work. They can also expose weaknesses in how you usually work. The pod, the loop, and the observability mattered more than the agents. What follows is how we structured the work using GitLab, GitLab Duo Agent Platform, and other tools — what worked, what the loop is and where it missed, and how you can copy what we did.

The setup

GitLab has had two rate-limiting paths in production for years: an application-level Gitlab::ApplicationRateLimiter with 121 keys, and a separate Rack-level system. The goal was to unify them on a single implementation in labkit-ruby. Observable, testable, and operated the same way everywhere. Every request to the monolith touches it, so its failure modes have to be visible and reversible.

The pod comprised three GitLab team members and a handful of AI agents. Max Woolf, a Staff Backend Engineer on the API Platform team, owned the monolith side and ran most of the rollouts. Bob Van Landuyt, who works on Scalability, owned the gem and shaped the architecture. I held scope and wrote some of the early labkit code. A couple other engineers floated in to absorb context and contribute code and reviews.

Agents read context, drafted specs, implemented bounded changes, wrote tests, and pre-reviewed merge requests. GitLab Duo Code Review kept code quality high on merge requests. Humans owned scope, architecture, rollout, and final review.

We ran a strict loop: read the epic, write the spec, run adversarial review on the spec, implement only after blockers cleared, verify with explicit evidence, run adversarial review on the merge request, escalate to human review, merge. Adversarial review was capped at two resolution rounds before a human had to weigh in. Across the project we shipped 14 numbered specs and somewhere north of 30 merge requests into labkit-ruby. In practice, the loop ran tighter or looser depending on the person. Bob often did several spec/review cycles privately before producing a shared artifact.

That loop sounds like a lot. On legacy code, it’s a loop I can trust an agent to execute.

What worked

Cohort 1 was the high-stakes test: five heavily-trafficked keys including pipelines_create, notes_create, and user_sign_in. We rolled it 1% → 10% → 50% on May 4, 2026, 100% on May 5, 2026. Bob’s running commentary from that day is the operating model in miniature:

“All rollouts complete. Up to now, all rate limits from the applimiter and the labkit implementation agree. But I suspect this is because there’s not a lot of traffic there. I’m going to see if I can generate some traffic exceeding the limit.”

That’s what a good rollout looks like, and it’s the kind of judgement no agent should make for you. The new system agreeing with the old isn’t success, it might just mean nothing tripped.

Cohort 2 collapsed the next 95 call sites, 83 in the monolith, and 12 in Enterprise Edition (EE), under a single feature flag pair. Without that consolidation, the rollout would have meant something like 95 individual flag flips and ~190 YAML edits. Agents are very good at this kind of mechanical fan-out across a codebase. Humans are very bad at it.

Where the loop missed

The loop missed the following things.

One was a shadow-mode miss. Cohort 2 had been running in shadow mode for days, agreeing with the old implementation. Switching it from observe to enforce should have been uneventful. There was a small hiccup.

The new adapter quietly dropped an identifier on one unauthenticated code path. Three String values were being squeezed into two primitive slots, and the wrong value overwrote the identifier. A tiny portion of users saw a generic failure for a short period of time.

Shadow comparison had actually flagged that key as diverging. We just hadn’t built our label set to distinguish a structural collision from a normal disagreement, so the signal sat in the dashboard while we ramped to 100%.

We immediately turned the enforcement flag off. Bob pinned the structural problem in one sentence:

“I think we should make this better once we clean up this mess and call the ApplicationLimiter only with named characteristics, no more array scopes.”

The immediate fix shipped two days later. The structural cleanup is on the list for the next pass.

It went through every step of the loop: spec, adversarial review, implementation, GitLab Duo Code Review, gradual rollout. The loop both did and didn’t catch it. The lesson wasn’t “agents are dangerous.” It was that we had observability, but not observability that distinguished the failure modes that mattered.

On May 15, Max ran an audit against master, pinged me in Slack, and opened Cohort 6:

“I’ve added a Cohort 6 to the migration: bits and bobs that got missed (not you, Bob).”

We had planned five cohorts. We needed six.

The diagnosis came a few days later: Claude had missed a handful of EE-only rate limits: notification_emails, some EE registry entries, three webhook keys, three sub-second partner_* keys, a few orphaned adapter rows. 17 keys out of 121 had slipped past the earlier cohorts. Each had a reason it didn’t fit cleanly into one of them. None had a reason to be invisible.

We hadn’t asked the agents, or ourselves, to keep a running count against the full key inventory.

Another was Redis. The redis-cluster-ratelimiting service runs as a 4-shard cluster. Bob’s read at the start was honest: “There’s headroom, but not enough to double utilization entirely.”

By early May the constraint we’d hit before came back:

Bob: “The bottleneck we came across before, that wasn’t new for this project, is an actual bottleneck. This means we need to do an infra change to get around that.”

Max: “Uh oh.”

We bumped maxclients in stages and halted at 75,000 connections instead of pushing to 100,000, once it became clear that more connections were going to tip the primaries’ CPU into saturation. One primary per shard, one core for command execution. No vertical lever to pull.

What the agents actually changed

They moved the bottleneck. With agents drafting specs and implementing inside a tight loop, code generation stopped being the slow part. Review capacity, rollout judgement, and operator attention became the slow parts. That’s a much better problem to have, but today it still consumes human capacity.

It also wasn’t always pleasant. Mid-project, Max wrote:

“Mixed bag, ended up in circles with an agent. Had one of those ‘I could’ve done this faster myself’ moments, which was irritating.”

A few weeks earlier he’d called the project “one of my steeper learning curves at GitLab, for sure.” Working with agents is a skill, and the cost of building it is days where you’d have made more progress alone.

The other shift was being honest about what “done” meant. Bob’s note at the end of Cohort 1 (“the feature flag per rate-limit is overkill, we shouldn’t do these for the next migrations”) is a small example of the kind of judgement no agent makes for you. They will happily generate 95 flag flips if you ask. The human judgement was deciding not to.

Where we are now

By mid-June, all six cohorts are at 100%. All 121 keys in the ApplicationRateLimiter run through the new framework, an audit confirmed the legacy path is down to near-zero, and we added a guardrail so no future rate limit can silently bypass it.

That’s the application-level migration done. RackAttack is next, the higher-volume layer at roughly 4 billion requests a day. Its shadow-and-enforce middleware is in development; the first merge request is approved and queued for merge.

If you want to copy this, you can use GitLab Duo Agent Platform to help you write your specs, Duo Developer to implement your issues, and Duo Code Review to help you merge your MRs. But that’s the easy part. I’d ask whether you have a Bob. Someone who’ll deliberately try to break the new system at 1% before letting it run at 50%. And whether you have a Max. Someone who’ll run an audit when everyone else thinks the migration is done. The workflow matters; the people more. If you want to try this on your own legacy code, try it out today..

AI agents work. So does changing how we work alongside them.

Yesterday, I wrote about using Cursor to review my blog for performance. Cursor (and again, for full disclosure I work here now ;) did a dang good job of finding performance issues with my site, handling the huge size and various languages, platforms, and so forth. As I said, it worked well, but I wanted to look into making this more of a repeatable process.

I'm saying "repeatable", not "automated", on purpose here. I can automate the process, but a full performance review feels much more like something I should run when I think it makes sense, for example, when I know I've tweaked my blog at the 'code' level versus writing cat-related blogs. I do plan to look into automations with Cursor later, but for now, what I've done is something I'll run when I think it makes the most sense.

What's a skill again?

At the simplest level, a skill simply gives direction to an AI agent on how to perform a task. It also describes itself in such a way that your AI agent knows when to invoke the skill. This is done in a simple Markdown file that can, and probably should, be checked into your repo.

So for example, if I had a skill related to checking the health of my cat, my AI agent should be smart enough to recognize when it should use the skill based on my prompt. So for example:

• I'm worried about Donut's health, anything I can do?
• Sir Fluffalot is doing fine, but what steps can I do to ensure she's healthy?

That's a pretty quick summary but you get the idea. Where the benefit comes in is helping document and describe a process that you need to run again and again, or a process your organization/team/herd of cats needs to standardize on and wants to be consistent.

Turning my performance review into a skill

When I first did my performance review, I kept the prompt pretty simple. I did this on another machine so I don't have it in front of me, but I believe it was literally as simple as:

Review this site and tell me of any performance improvements you can find.

That generated the review I shared in my last post and guided my development.

Cursor provides a few ways to make skills:

• Since it's just a Markdown file, you can just make a file, old school like we did in the old days.
• Cursor has a skill to make skills: /create-skill i need a skill to help with cat care
• You can also just ask Cursor to do this in a prompt.

The last option is what I did, asking Cursor to make a skill like my prepublish one.

Cursor responded with:

A note on the last paragraph of the response. I had started my conversation with Cursor initially thinking I was going to automate it with a hook, but changed my mind. That's why it's mentioned here. The last bit "frontmatter trigger phrases" goes back to what I said earlier - a skill is both a guide on how to do something as well as a description of when it should be run.

The result was pretty astounding I think, specifically in how Cursor recognized the "engine" parts of my blog versus the "content" bits. It did this well in the one off so I'm not surprised, but I love having the various parts spelled out specifically in the skill itself. Like yesterday, I'll share the entire skill at the end, but let me talk a bit about what it did when I tried running it.

Running the Skill

When I ran the skill, I didn't expect it to find anything as my last commit was just a blog post. Cursor responded as such mentioning there was nothing staged. It noted two files that were dirty but not important:

---

I asked it to review the last two and it also noticed they didn't have anything relevant either - but what got me excited was that it went ahead and looked at one more and noted relevant changes there. Here's how that response ended:

Want me to precheck 1117ccc2 (or 1117ccc2 + 06626459 together)? That is where the meaningful review would be.

As I mentioned, I didn't do everything in the initial performance check, so I figured, why not, and I approved this scan. It noted two places I had used rss-parser and not yet added the timeout change so I went ahead and let Cursor make those changes as well.

The Skill

Here's the skill created by Cursor. Let me know what you think! Also, check out Cursor's docs on Skills - it's not just specific to Cursor and is a great reference in general.

---

---

Learn how to balance control and security using the four stages of Auth0 customization, from standard Universal Login to embedded authentication.

Most “AI agent” demos forget everything the moment the process exits. That’s fine for a toy project, but useless for anything real. An agent that helps you write, triage, or support needs two things a language model alone can’t give it: durable state and the ability to recall the right context by meaning. This post shows how to build exactly that by integrating two pieces that fit together surprisingly well:

Eve — Vercel’s filesystem-first agent platform. Drop a file in agent/tools/, and it becomes a tool the model can call.

Azure Cosmos DB JavaScript SDK — the official, promise-based client for Cosmos DB NoSQL, and the piece that does the heavy lifting here. It’s a modern, fully typed TypeScript SDK

We’ll build an agent that authors a Next.js blog and keeps a long-term semantic memory in Azure Cosmos DB. Every code sample below is production-shaped, not pseudocode.

Meet eve: agents as a filesystem

If you’ve built with the Vercel AI SDK, eve will feel immediately familiar it’s the same team’s take on agents, and its core idea is refreshingly literal: your agent is a folder. There’s no graph DSL to learn and no orchestration boilerplate. You describe the agent with plain files, and eve wires them into a running agent:

 

• Tools are just files. Drop a defineTool({…}) in agent/tools/ and it’s available to the model — no registration, no wiring. Inputs are described with a Zod schema, and eve generates the JSON schema the model sees.
• Instructions are Markdown. md is your system prompt. Editing behavior means editing prose, not code.
• Bring your own model. eve runs on the AI SDK’s provider ecosystem, so you can point it at OpenAI, Anthropic, or as we do here Azure OpenAI

agent.ts

import { createAzure } from "@ai-sdk/azure";
import { defineAgent } from "eve";
 
const azure = createAzure({
  baseURL: `${process.env.AZURE_OPENAI_ENDPOINT}/openai`,
  apiKey: process.env.AZURE_OPENAI_API_KEY,
  apiVersion: "preview",
});
 
export default defineAgent({
  model: azure.chat("gpt-4.1"), // your Azure deployment name
});

• Runs locally, ships anywhere. eve dev gives you an interactive chat in the terminal; eve build produces a deployable server. The agent talks to the outsideworld (your blog’s API, Azure Cosmos DB) through the tools you

What eve deliberately doesn’t give you is persistence. A model call is stateless; when the process restarts, the conversation is gone. That’s the gap we fill with Azure Cosmos DB — turning a stateless agent into one that genuinely remembers.

Why Azure Cosmos DB for agent memory?

Agent memory has an awkward shape for traditional databases: you write a lot of small items, you read them back by meaning (not exact keys), and you want it fast and global. Azure Cosmos DB NoSQL fits this well:

• Single-digit-millisecond point reads when you know the id and partition key.
• Native vector search — store embeddings and query with VectorDistance(); no separate vector database to run.
• Native TTL — expire ephemeral memories with zero cleanup jobs.
• One SDK, one bill — the same container holds your app data and your vector

Step 1: A singleton client (the #1 SDK rule)

A CosmosClient owns a connection pool and routing caches. Creating one per request exhausts sockets and adds TLS latency. Create it once and reuse it:

// src/lib/cosmos.ts    
    import { CosmosClient } from "@azure/cosmos";    
    import { DefaultAzureCredential } from "@azure/identity";    
         
    type CosmosGlobal = typeof globalThis & { __client?: CosmosClient };    
         
    export function getCosmosClient(): CosmosClient {    
      const g = globalThis as CosmosGlobal;    
      if (!g.__client) {    
        g.__client = new CosmosClient({    
          endpoint: process.env.COSMOS_ENDPOINT!,    
          // Prefer Entra ID over keys. Falls back to a key only if provided.    
          aadCredentials: new DefaultAzureCredential(),    
          connectionPolicy: {    
            // Let the SDK absorb transient 429s with backoff.    
            retryOptions: { maxRetryAttemptCount: 9, maxWaitTimeInSeconds: 30 },    
          },    
        });    
      }    
      return g.__client;    
    }    

Caching on globalThis matters in dev: frameworks like Next.js reset module state on hot-reload, which would otherwise leak a new client on every save.

Step 2: Model for cheap reads

Pick a partition key that is high-cardinality and immutable, and make the item id equal to it where you can — then single-item fetches are 1 RU point reads instead of queries:

    // A blog post: id === slug, partition key is /slug.    
    const { resource: post } = await container    
      .item(slug, slug) // (id, partitionKey)    
      .read(); // ~1 RU, single-digit ms    

For lists, use continuation tokens, never OFFSET/LIMIT:

    const iterator = container.items    
      .query({ query: "SELECT c.slug, c.title FROM c WHERE c.status = 'published'" })    
      .getAsyncIterator();    
    // ...or .query(spec).fetchNext() and pass resource.continuationToken to the client.    

Step 3: Turn the container into a vector memory store

Here’s where it gets fun. To store embeddings and search them, define a vector embedding policy and a vector index at container-creation time (the policy is immutable afterward):

         
    import {    
      VectorEmbeddingDataType,    
      VectorEmbeddingDistanceFunction,    
      VectorIndexType,    
    } from "@azure/cosmos";    
         
    await database.containers.createIfNotExists({    
      id: "memory",    
      partitionKey: { paths: ["/sessionId"] }, // one conversation per partition    
      defaultTtl: -1, // TTL enabled; set per-item `ttl` for ephemeral memories    
      vectorEmbeddingPolicy: {    
        vectorEmbeddings: [    
          {    
            path: "/embedding",    
            dataType: VectorEmbeddingDataType.Float32,    
            dimensions: 1536, // text-embedding-3-small    
            distanceFunction: VectorEmbeddingDistanceFunction.Cosine,    
          },    
        ],    
      },    
      indexingPolicy: {    
        includedPaths: [{ path: "/*" }],    
        // CRITICAL: keep the raw vector out of the regular index (saves write RUs).    
        excludedPaths: [{ path: "/embedding/*" }],    
        vectorIndexes: [{ path: "/embedding", type: VectorIndexType.QuantizedFlat }],    
      },    

Storing a memory — embed the text, then create the item:

    import { randomUUID } from "node:crypto";             
    async function appendMemory(sessionId: string, content: string) {    
      const embedding = await embedText(content); // Azure OpenAI embeddings → number[]    
      await container.items.create({    
        id: randomUUID(),    
        sessionId,    
        type: "memory",    
        content,    
        embedding,    
        createdAt: new Date().toISOString(),    
      });    
    }             

Recalling by meaning — VectorDistance() with ORDER BY, a literal TOP, and a parameterized query vector (so the query plan caches):

    async function recall(sessionId: string, query: string, k = 5) {    
      const vector = await embedText(query);    
      const { resources } = await container.items    
        .query(    
          {    
            query:    
              `SELECT TOP ${k} c.content, VectorDistance(c.embedding, @vec) AS score ` +    
              `FROM c WHERE c.type = "memory" AND c.sessionId = @sessionId ` +    
              `ORDER BY VectorDistance(c.embedding, @vec)`,    
            parameters: [    
              { name: "@vec", value: vector },    
              { name: "@sessionId", value: sessionId },    
            ],    
          },    
          { partitionKey: sessionId }, // single-partition → cheap and fast    
        )    
        .fetchAll();    
      return resources; // nearest memories, most similar first    
    }    

Because we filter on the partition key, this stays a single-partition query — exactly what you want on a hot path.

Step 4: Expose it to the agent as an eve tool

eve’s model is delightfully simple: every file in `agent/tools/` is a tool. You describe its inputs with Zod, and eve hands the model a JSON schema automatically.

         
    // agent/tools/recall.ts    
    import { defineTool } from "eve/tools";    
    import { z } from "zod";    
    import { blogApiUrl, authHeaders } from "../lib/blog-client.js";    
         
    export default defineTool({    
      description:    
        "Recall the most relevant saved memories for a query using vector search.",    
      inputSchema: z.object({    
        query: z.string().min(1).describe("What to search your memory for."),    
        sessionId: z.string().optional(),    
        k: z.number().int().min(1).max(20).default(5),    
      }),    
      async execute({ query, sessionId, k }) {    
        const url = new URL(blogApiUrl("/api/memory")); // resolves BLOG_API_URL    
        url.searchParams.set("q", query);    
        url.searchParams.set("k", String(k));    
        if (sessionId) url.searchParams.set("sessionId", sessionId);    
        const res = await fetch(url, { headers: authHeaders() });    
        const { results } = await res.json();    
        return { ok: true, memories: results };    
      },    

Now the agent can decide on its own, to call recall before answering — and because the memories live in Azure Cosmos DB, they persist across sessions, deploys, and regions.

The full loop

That’s a complete, durable, semantically searchable memory in well under 200 lines no extra infrastructure, no second database.

Try it

The complete, runnable source for everything in this post lives in the AzureCosmosDB/eve-cosmos-memory repository clone it and follow along:

    npm install    
    npm run seed     # creates the DB, blog container, and vector memory container    
    npm run dev      # the blog (Next.js on Vercel)    
    npm run agent    # chat with the eve agent — ask it to remember and recall    

Give your agent a memory. The Azure Cosmos DB JavaScript SDK makes the durable, global, vector-searchable part almost boring which is exactly what you want from infrastructure.

About Azure Cosmos DB

Azure Cosmos DB is a fully managed and serverless NoSQL and vector database for modern app development, including AI applications. With its SLA-backed speed and availability as well as instant dynamic scalability, it is ideal for real-time NoSQL and MongoDB applications that require high performance and distributed computing over massive volumes of NoSQL and vector data.

To stay in the loop on Azure Cosmos DB updates, follow us on X, YouTube, and LinkedIn.  Join the discussion with other developers on the #nosql channel on the Microsoft Open Source Discord.

The post Building on Vercel’s eve + Azure Cosmos DB: An Agent That Remembers appeared first on Azure Cosmos DB Blog.

You start off with what seems like the obvious solution to a multi-tenant SaaS application.

We have tenant A. We have tenant B. We have one application and one database. Within that database, for every structure, whether that is a table, collection, or stream, we segregate things by a tenant ID.

It is simple. It is easy. It works.

Until it does not.

What happens when one tenant imports five million records? Or they run a bunch of reports, and some of those reports are massive? That shared infrastructure which seemed simple is also what is coupling everything together.

YouTube

Check out my YouTube channel, where I post all kinds of content on Software Architecture & Design, including this video showing everything in this post.

https://youtu.be/4_pxeE1Xv3Q

Shared Infrastructure Means Shared Problems

Let us use another basic example. We have multiple instances of our application in a scaled out environment, but now we need to make a schema change.

The first thing that happens is we make our schema change to the database. Then we do a rolling deployment where we update one of the application instances first. That instance is now updated. Its code matches what the schema is. That is what it is expecting.

Everything is fine.

Uh oh.

What about the second instance that has not been updated yet? It is still making requests at runtime while everything is being deployed. Of course, it is going to fail. It has zero expectation of what that schema change is. It does not even know about it.

If your schema changes are not backwards compatible, your old code and your new code are looking at the database in a different way. They are expecting a certain shape, and the database is not giving both of them what they expect.

Can you go down the road of blue green deployments? Yes.

Could you have canary deployments? Sure.

But you are still under the same circumstance. You need to understand the coupling involved and the trade offs being made.

The First Question Should Not Be Database Per Tenant

Almost always, when people talk about multi-tenancy, they immediately ask:

Should we have a shared database?
Should we have a database per tenant?
Should we have a schema per tenant?

Those are valid questions, but they should not be the first question.

A better question is this:

What are you actually trying to isolate?

Are you trying to isolate data? Compute? Deployments? Schema changes? Performance? Different types of failures?

Because multi-tenancy is not just about databases. It is about creating isolation through boundaries. A database is just one of those boundaries.

Every Shared Resource Creates Coupling

Every time tenants share something, they are coupling through it.

In the database example, yes, you are sharing the schema. You have to deal with shared migrations. You are also sharing the performance of that database.

With your API, tenants are sharing the compute that is actually running it. They are sharing deployments. They are sharing failures happening within that instance or those instances.

The same thing applies if you are using queues and background processing. The same thing applies to cache, what is in memory, invalidation, and the keys used in that cache. It applies to regions, where things are deployed, residency, and latency.

If you are sharing something between tenants, they are ultimately coupled through it.

A Shared Database Can Be Perfectly Fine

Back to the simple example: one application, one shared database, and tenant ID on the rows.

There is nothing inherently wrong with this.

If we have a single instance and a single shared database with segregation by tenant ID, it is simple. It works. When we add a new tenant, it is easy. We do not have to set up a new database. We do not have to roll out some new infrastructure.

We are sharing the schema, and that is a lot easier to manage. It is simpler to deal with. It is probably more cost effective.

Within our code, every time we deal with queries, we have some type of filtering. Depending on the libraries and frameworks you are using, that might be handled by query filters or something similar.

Again, there is nothing inherently wrong with this.

As long as you understand the trade offs.

The Tenant ID Has To Leak Everywhere

The trade off is that this information has to leak everywhere through your system.

It is not a bad thing. It is just the trade off.

If you are doing database access at the row level, the tenant ID has to be in every possible query, or you have to force it through some type of query filter.

Every report that you have anywhere in your system has to filter by tenant. It might not even be application code. It might be somewhere else entirely, but it still has to filter.

If you are using queues and messaging, tenant ID has to be part of every message because you are not segregating everything physically.

Your tenant ID has to be part of everything.

Again, not bad. Just the trade off.

Shared Schema Means Shared Migrations

If you are sharing the database, that means you are sharing the schema. You are sharing migrations.

That means if you upgrade your database, make some type of schema change, and are doing a rolling deployment, it might work for one instance, but it might not work for another if everything is not backwards compatible.

So you need to think about that.

The trade off is that you have to make schema changes backwards compatible. That means having a process. You expand the schema, make the change in a compatible way, deploy the new code, then backfill the data.

If you add columns that are nullable, and the new application code is going to start sending that data, you still have to assume that existing data needs to be backfilled.

That becomes part of your deployment pipeline.

Is it more complexity? Yes.

Is it more work? Yes.

Is it a trade off? Yes.

That is the whole point. You get the simplicity of maintaining one physical database, but the complexity moves into dealing with shared schema changes and rolling deployments.

Deployment Isolation Is Not Schema Isolation

There is also isolation at the application level.

You could have tenant A using version one of your application and tenant B using version two. That gives you isolation at the deployment level, almost like a canary deployment.

This can be totally fine.

Maybe you have a change coming in that you are unsure about. You give it to tenant B because they are a little more relaxed about it. Once you know it is solid and there are no issues, you roll it out further to tenant A or to more tenants.

But if they are still using the same underlying schema, you still have to make everything backwards compatible.

You are providing deployment isolation. You are not necessarily providing schema isolation.

Multi Tenant Architecture Is A Spectrum

When you hear somebody talk about multi tenant architecture, it is really about isolation.

But what kind of isolation?

It is not just the database.

At the database level, you have data isolation. At the API level, you have compute and deployment isolation. If you are using queues and messaging, you have background processing isolation. You might have different queues by priority or by tenant. The same thing applies to caching.

If you are using cloud regions, now you are also talking about residency and latency depending on where things are deployed relative to your tenants.

It is not just about data isolation.

All of these choices are on a spectrum. Depending on your needs, you can be on one side, in the middle, or on the other.

For data isolation, you could have shared tables with a tenant ID. You could have a shared database, but separate schemas. You could have different pools where some tenants are on one database instance and some tenants are on another. Or you could go as far as having a database per tenant.

The same goes for compute. You could have shared instances where everybody is using everything. You could have pooled instances where a subset of tenants uses a subset of compute. Or you could have completely dedicated compute.

You do not necessarily have to fit on one side or the other. It is often a mix and match depending on your needs.

Efficiency Versus Control

The trade off is efficiency versus control.

On one side, you have efficiency. A shared database is simple to maintain. You have one instance. You have one place for backups, migrations, upgrades, and operational concerns.

On the other side, you have control. A database per tenant gives you more control, but that control is not free. There is cost and complexity in maintaining multiple databases, different pools, dedicated compute, or dedicated infrastructure.

The same thing applies to compute.

Shared compute can be very efficient and cost effective. But if you have pooled or dedicated compute for particular tenants, now you are dealing with every particular instance for every tenant or group of tenants.

There is just more complexity involved.

Noisy Neighbors

Here is one reason you might not want to share compute.

Let us say we have tenant A and tenant B.

Tenant A is making requests. Everything is working fine. Happy path. Everything is good.

Tenant B starts making a massive number of requests. Maybe they are importing data. Maybe they are running requests against the database that are really heavy. Maybe what they are doing is CPU or memory intensive.

Now tenant B is affecting the database. They might also be affecting the API.

That is the noisy neighbor problem.

Tenant B is flooding requests to the app or to the database, and tenant A is affected by what tenant B is doing.

Is that a reason to isolate or pool compute differently? It could be.

But there are also alternatives.

Rate Limiting Is One Tool

One alternative is rate limiting.

That means every tenant has their own scope. It does not necessarily mean we have to separate compute and have some compute sitting idle, not doing much.

Both tenants could still use the exact same underlying instance of the app or API, but we rate limit by tenant ID as part of the request.

That can help solve the noisy neighbor problem, but it does not necessarily solve it entirely. Depending on the request being made, you might save the API, but that does not mean you are saving the database.

Rate limiting is just one tool.

You could do the same kind of thing with database connection pools and group those by tenant. Again, they are just tools.

What you really need to identify is what resources are being exhausted.

Tenants Are Not All The Same

The type of system you are building and the customer base you have matters.

At the beginning, tenants might seem like they are all the same, but they probably are not.

Some tenants might have a small number of users. Larger tenants might have a much larger set of users. Some tenants might run heavier workloads. Some might import more data. Some might run more reports.

Over time, you might realize that you need to pool small tenants together, pool large tenants together, or move very large tenants into dedicated infrastructure.

Again, it goes back to mix and match.

At first, you might think all tenants are the same, but they are not.

When Should You Isolate?

This leads to the actual decision you need to make: what should you isolate, and when?

You isolate when you need a smaller blast radius. You isolate when you do not want noisy neighbors at a given level, whether that is compute, database, background processing, or something else.

You isolate when you have tenant specific requirements. That could be compliance. You might have a requirement that a tenant needs to be isolated at the database level for compliance reasons.

You isolate when a tenant needs dedicated capacity. Maybe they do not want to share resources because they care about availability or performance guarantees.

You isolate when you want safer rollouts. Maybe you want to provide one version to a subset of tenants, and then roll it out later to more sensitive tenants or a larger pool.

It is isolation at various levels because you want to avoid noisy neighbors, reduce the blast radius, and have protection against specific types of failures.

Multi-Tenancy Is About Boundaries

Multi-tenant architecture is not a database pattern.

It is about isolation.

The real question is: what should be isolated?

Sharing gives you efficiency. It is usually simpler and more cost effective.

Isolation gives you more control. But that control comes with more cost and more complexity.

So if you are building or living in a multi-tenant system, do not start with “shared database or database per tenant?”

Start with what you are actually trying to isolate.

Then decide where sharing makes sense, where isolation makes sense, and what trade offs you are willing to take.

Join CodeOpinon!
Developer-level members of my Patreon or YouTube channel get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

Related Links

• Resilience Patterns Can Make Your System Less Resilient
• Stop Blaming Event-Driven Architecture
• Change Your Architecture During Deployment

The post Multi-Tenancy Isn’t About Databases appeared first on CodeOpinion.