Solving the LLM quota monitoring paradox with zero-overhead local Connect RPC agent hooks.

Abstract

Google Antigravity CLI users using Google OAuth face abrupt task failures when API quota hits 0%, while account switching triggers unrecoverable signature errors. Querying quota via LLM tool calls creates a paradox by consuming the very tokens being monitored. We resolve this with antigravity-cli-check-usage-plugin, a CLI Agent Hook running outside the LLM execution turn. Directly querying local Connect RPC endpoints, it monitors quota with zero token overhead and injects proactive warning banners when threshold limits are reached.

1. Introduction

Developers relying on Google Antigravity CLI for autonomous pair programming frequently encounter a frustrating barrier: running out of API quota mid-session. When using Google OAuth authentication, your quota can silently hit 0%, causing task execution to halt abruptly with an unrecoverable quota error:

⚠ Individual quota reached. Please upgrade your subscription to increase your limits. Resets in 1h00m00s.
Error ID: 49a81c0f

To bypass this roadblock, developers often attempt to log out and switch to a paid Google Cloud project billing account. However, in Antigravity CLI v1.1.12, attempting to resume an active agent session after switching accounts triggers a critical signature mismatch failure:

⚠ Invalid thought signature.
Error ID: e2901f4c

This error prevents the session from continuing, forcing you to wait until the quota resets. While future CLI updates may resolve this session state issue, waiting for a patch is not a viable strategy when shipping code today.

The architectural divergence between standard tool-based monitoring and our agent hook model is illustrated in Figure 1. While developers can manually run the /usage slash command to view quota, AI agents executing multi-step autonomous tasks cannot trigger /usage programmatically. In traditional CLI workflows, invoking quota checks via LLM tool calls requires passing context back and forth through the inference API, depleting active model tokens. Conversely, the zero-overhead agent hook interceptor executes locally prior to prompt dispatch, querying the process socket silently and injecting status alerts only when remaining quota breaches configured safety bounds.

In this article, to overcome the limitation of agents being unable to trigger /usage, we walk through the engineering journey of building antigravity-cli-check-usage-plugin. By combining local Connect RPC inspection with proactive lifecycle hooks, this plugin automatically performs external quota checks with Zero Quota Consumption (0 LLM tokens), completely preventing mid-session crashes.

2. Repository

The plugin developed and discussed in this article is open-sourced and available on GitHub:

• GitHub Repository: tanaikech/antigravity-cli-check-usage-plugin

This repository contains the dual-runner entrypoint (entrypoint.sh), Python script (check_quota.py), pure Bash fallback script (check_quota.sh), lifecycle hook manifest (hooks.json), and default threshold configuration (config.json), allowing instant one-command installation as an Antigravity CLI plugin across any developer environment.

3. Core Motivation

While Antigravity CLI provides the /usage slash command for developers to manually inspect quota limits, AI agents executing autonomous task loops cannot invoke /usage programmatically.

If we attempted to solve this by equipping the AI agent with a custom tool to query the internal RPC endpoint (/exa.language_server_pb.LanguageServerService/GetUserStatus), the tool invocation and context turns would consume LLM API tokens. This creates a fundamental paradox: using LLM context tokens to check remaining quota consumes the very quota you are trying to preserve.

In addressing this challenge, the solution built upon our previously published article, A Developer’s Guide to Agent Hooks in Antigravity CLI. Recalling the out-of-band execution mechanics of CLI Agent Hooks explored in that guide, we leveraged lifecycle events (PreInvocation and PostInvocation) to run local process checks completely outside the LLM inference turn—guaranteeing zero API token quota overhead.

• Zero Token Overhead: During normal operation, quota checking runs entirely outside the LLM context (via local Python/Bash scripts) without invoking LLM tool calls.
• Local RPC Interception: It automatically queries the CLI's internal status endpoint on 127.0.0.1 without external network calls.
• Proactive Threshold Alerting: It notifies both the developer and the AI agent before quota hits 0%, preventing session corruption and hard crashes.

4. Connect RPC

Through reverse-engineering the Antigravity CLI local process architecture (originally explored in the antigravity-usage repository by skainguyen1412), we discovered that the running agy process hosts a local HTTPS server using the gRPC / Connect Protocol on 127.0.0.1.

By querying the internal endpoint /exa.language_server_pb.LanguageServerService/GetUserStatus, we can retrieve real-time model quota fractions and reset timestamps directly from the local process.

Because the agy process may open multiple listening sockets on 127.0.0.1 for IPC and WebSockets, a shell loop that probes each detected port until it receives a valid userStatus response is required:

# Scan listening sockets for the active 'agy' process on loopback (127.0.0.1)
for PORT in $(ss -tulpn 2>/dev/null | grep agy | awk -F'127.0.0.1:' '{print $2}' | awk '{print $1}' | sort -u); do
  # Post a Connect Protocol request to the internal GetUserStatus RPC endpoint
  RES=$(curl -k -s -X POST https://127.0.0.1:${PORT}/exa.language_server_pb.LanguageServerService/GetUserStatus \
    -H "Content-Type: application/json" \
    -H "Connect-Protocol-Version: 1" \
    -d '{"metadata":{"ideName":"antigravity","extensionName":"antigravity","locale":"en"}}')

  # Verify if the response contains the userStatus JSON key
  if echo "$RES" | grep -q "userStatus"; then
    echo "$RES" | jq .
    break
  fi
done

To execute this logic seamlessly and rapidly inside an agent hook outside the LLM invocation turn, we implemented a Python script using standard library components, alongside a pure Bash fallback script (check_quota.sh) and an entrypoint runner (entrypoint.sh) that automatically selects Python when available or Bash on systems without Python installed.

[!IMPORTANT]

Note on Scope: The GetUserStatus endpoint returns the Five Hour Limit Remaining fraction (remainingFraction) and ISO reset timestamp (resetTime) for active model pools. The long-term Weekly Limit Remaining is not exposed through this RPC endpoint.

5. Complete Agent Hook Workflow

Building upon the lifecycle concepts detailed in A Developer’s Guide to Agent Hooks in Antigravity CLI, the plugin integrates into the Antigravity CLI by registering PreInvocation and PostInvocation agent hooks in hooks.json. Because PreInvocation fires after the user submits input but before the prompt payload is dispatched to the LLM backend, it inspects local process state and dynamically injects steps prior to model inference.

As detailed in Figure 2, the final agent hook operates under two distinct execution patterns based on the configured warning threshold (default: 20%):

Pattern A: Normal Operation (Quota > Threshold)

When remaining quota is above the warning threshold, the hook outputs an empty step injection payload:

{
  "injectSteps": []
}

• Impact: Zero Quota Consumption (0 Token Overhead). The hook executes silently in less than 50 milliseconds. No messages or extra context are injected into the LLM session, consuming absolutely zero model quota.

Pattern B: Warning State (Quota <= Threshold)

When remaining quota drops to or below the threshold, the hook injects a transient system message with mandatory agent directives:

{
  "injectSteps": [
    {
      "ephemeralMessage": "⚠️ [SYSTEM QUOTA WARNING] Model quota is below threshold (20%) (Active: gemini-3.6-flash-medium):\n - GEMINI Models [ACTIVE MODEL]: 20.0% remaining (Refreshes in 3h 00m)\n\n[MANDATORY INSTRUCTION FOR AGENT]: The model quota has dropped below the threshold. You MUST display a prominent Quota Warning banner at the very top of your response for THIS TURN ONLY! Do NOT display a warning banner on subsequent turns unless another quota warning is explicitly injected. In the warning banner, you MUST also inform the user that they can run the '/usage' command at any time to inspect detailed quota status."
    }
  ]
}

• Impact: The AI agent immediately prepends a prominent Quota Warning banner to its response, advising the developer to run /usage or pause heavy multi-step automation before encountering a hard crash.

6. Installation & Dual Runtime

The complete implementation is published as an open-source Antigravity CLI plugin: antigravity-cli-check-usage-plugin.

Installation

Install the plugin directly via the Antigravity CLI:

agy plugin install https://github.com/tanaikech/antigravity-cli-check-usage-plugin

Dual Runtime Architecture: Python Primary + Pure Bash Fallback

The plugin features a multi-environment entrypoint (entrypoint.sh) producing 100% identical JSON outputs across both runtimes. The engineering rationale behind this dual design includes:

• Python (Primary Runner): Requires zero external dependencies like jq, absorbs OS-specific syntax differences across Linux, macOS, and Windows, and guarantees type-safe date math.
• Pure Bash (Fallback Safety Net): Ensures instant execution in minimal or containerized environments where Python is not pre-installed.

Configuration and Disabling

You can customize or completely disable the warning threshold (default: 20.0%) using environment variables, configuration files, or hook arguments.

Set Custom Threshold (e.g., 25%):

export QUOTA_THRESHOLD=25.0

Disable Quota Check Completely:
Setting QUOTA_THRESHOLD to -1 instructs the hook to skip all RPC queries immediately:

export QUOTA_THRESHOLD=-1

7. Real-World Testing & Verification

After installing the plugin, setting export QUOTA_THRESHOLD=80.0 and executing a live session test in Antigravity CLI v1.1.12 demonstrates the hook in action, as captured in Figure 3:

When the user enters a simple greeting (hello), the agent hook instantly detects that the active model's remaining quota (71.0%) has dropped below the configured threshold (80.0%). A prominent yellow Warning banner (Quota Warning: GEMINI Models quota is at 71.0% remaining...) is dynamically prepended at the top of the AI's response, alerting the developer and providing a reminder to inspect detailed limits via /usage.

8. Updating & Uninstalling

To update the plugin to the latest version or remove it from your environment:

• Check installed plugins:

  agy plugin list

• Uninstall the plugin:

  agy plugin uninstall antigravity-cli-check-usage-plugin

• Reinstall the updated version:

  agy plugin install https://github.com/tanaikech/antigravity-cli-check-usage-plugin

Summary

In this article, we presented a zero-overhead solution to eliminate mid-session quota crashes and account-switching signature errors in Google Antigravity CLI. Drawing upon foundational concepts from A Developer’s Guide to Agent Hooks in Antigravity CLI and resolving the paradox where using LLM tool calls to query internal RPC endpoints consumes quota, we built native CLI Agent Hooks (PreInvocation / PostInvocation) running completely outside the LLM execution turn. Featuring a dual Python primary and pure Bash fallback architecture, the hook probes internal local Connect RPC endpoints with absolute zero token consumption during normal operation. By proactively injecting warning banners and /usage reminders when quota drops below threshold, it guarantees universal environment compatibility and eliminates task interruptions cleanly at the root.

I'd been waiting for more than 30 minutes. The terminal just sat there, blinking, without returning a single word. I'd launched Gemma2 in its 9-billion-parameter version on my laptop (a regular Mac, the kind any professor or student would use) and the model simply wasn't responding.

It wasn't a bug. It was the most honest answer the experiment could have given me.

That frustrating wait ended up being, without exaggeration, the most interesting finding of the whole process. Because the question that brought me there wasn't "how big can a model get?" — it was a much more practical one: what actually happens when an agent you built in a tutorial has to survive in production?

I've been working with Gemma as a case study to understand that jump — from an educational prototype to something that can hold up under long conversations, limited hardware, and real users. This post is the honest summary of that process: what worked convincingly, what didn't work the way I expected, and why that "didn't work" turned out to be more useful than a clean result would have been.

The real problem: why tutorials are a little dishonest

Almost every conversational agent tutorial does the same thing, without saying so out loud: on every turn, it sends the model the entire previous history, all over again.

Imagine that every time you added a sentence to a conversation, you had to repeat everything said before it — every message, every reply — before you could say the new one. At first you don't notice. But if the conversation runs 30 or 50 turns, you're repeating an entire novel just to add one sentence.

This pattern is called linear context stacking, and it causes three concrete problems:

1. Memory saturation — every call to the model processes an increasingly large context.
2. Risk of hitting the token limit — every model has a maximum context window; sooner or later, you hit it.
3. Quality degradation — there's a documented phenomenon in NLP literature called "lost in the middle": when context gets very long, models pay less attention to information sitting in the middle of it, versus the beginning or end. In other words, it's not just slower — it gets worse.

This problem isn't unique to any one model, but it weighs differently depending on context. If you're using a closed API with a massive context window and pay-per-token billing, the cost of this problem is financial — you just pay more. But if you're running an open model locally, as is common in universities and research labs across Latin America, the cost is infrastructure: limited RAM, no dedicated GPU, no room to "just pay for more compute." An unbounded context isn't a minor optimization detail there — it's the difference between the agent working at all or not.

The experiment: design and decisions

To avoid staying purely theoretical, I ran a simple but controlled comparative experiment using Gemma 2 (2B), running locally with Ollama — no dependency on any paid external API.

The idea: simulate a typical technical conversation (a microservice troubleshooting case, where each turn adds new information) and run it against two different architectures:

• Pipeline A (Naive): accumulates the entire history with no compression at all. This is, literally, what a tutorial-style agent looks like.
• Pipeline B (Optimized): applies history pruning — instead of sending the whole conversation, it sends a compact summary of the latest state.

# Pipeline A — accumulates everything, no pruning
conversation_history += f"\nPrevious text {i+1}: {chunk}\n"
full_prompt = f"{conversation_history}\n{TASK_PROMPT}\n{chunk}"

# Pipeline B — only a compact summary of the latest state
full_prompt = f"Previous compact context: {compact_context}\n{TASK_PROMPT}\n{chunk}"

Three methodological decisions I almost overlooked, and which turned out to be key to making the results trustworthy:

1. The "cold start" nearly ruined everything.
In my first run, the first step of each pipeline came out suspiciously slower than the ones after it — several seconds off. It wasn't the prompt size: it was the cost of loading the model into memory the first time it's called. The fix was adding a throwaway "warm-up" call before starting to measure each pipeline, so both started on equal footing.

2. Real tokens, not estimated ones.
At first I was estimating tokens by counting words and applying an approximate conversion factor — a completely avoidable loss of precision. Ollama returns the real, exact count in every response (prompt_eval_count). Switching to that number made the charts far more defensible.

3. A single run isn't enough.
I ran each pipeline 3 times and averaged the results, with error bars included in the charts. This is what honestly revealed that one of my early results wasn't as solid as it first looked — more on that below.

Results: what held up cleanly, and what didn't

Tokens: the result that actually holds

The token pattern was consistent across all 3 runs, with no ambiguity. The naive pipeline grows linearly — from 107 to 266 tokens in just 4 steps, nearly tripling. The optimized pipeline flattens into a plateau, around 104 tokens.

That's a 61% reduction in input tokens by the final step. Active context management delivers exactly what it promises: it keeps the conversation's memory footprint from growing unchecked.

Latency: the result that forced me to rethink the hypothesis

This is where the experiment got genuinely interesting. The intuition says: fewer input tokens, faster response. The real data didn't back that up — at least not clearly. The error bars for the naive and optimized pipelines overlap in almost every step.

Why? Because with a 2B model, on relatively short conversations, total response time is dominated by how much the model has to generate as output — not by how much it has to read as input. Shrinking the context doesn't automatically speed up the generation of the response.

It's a "negative" result in the sense that it doesn't confirm the initial hypothesis, but it's honestly the most valuable finding of the whole experiment: context management and latency are related problems, but they're not the same problem, and optimizing one doesn't guarantee improving the other.

The failed attempt with Gemma2 9B (and why I'm not hiding it)

I wanted to push one step further and repeat the comparison with Gemma2's 9B version, to see whether a larger model would show a clearer latency advantage — the hypothesis being that processing a long prompt weighs more when the model itself is bigger.

I never got that data. Over 30 minutes running on my laptop, without a single complete response. I had to cancel it.

I could have left this out of the post. But it's a relevant data point in its own right, and honestly the one closest to my reality as a researcher in the region of Latin America: the barrier to experimenting with larger models isn't just a software optimization problem, it's a hardware access problem. If I, with intent and dedicated time, struggle to run a 9B model on a consumer laptop, that's exactly why this kind of work — optimizing efficient agents with small, accessible models — matters for universities, labs, and teams in the region that don't have dedicated GPUs on hand.

What this means in practice

If you're building, or thinking about building, an agent on a local open model, here's what I'm taking away from this experiment:

• Measure before you optimize. My initial intuition about latency was not the correct one, and I only found out because I measured rigorously (3 runs, warm-up, real tokens) instead of trusting a single run.
• Saving tokens doesn't automatically buy you latency. Depending on model size and conversation length, the real bottleneck might be somewhere else entirely.
• Context pruning has trade-offs — it's not magic. My current implementation trims by length, not semantic relevance, which means there's real risk of losing important historical information. That's a limitation I'm naming, not hiding.
• A failed experiment on real hardware is data, not a failure. I couldn't run 9B on my laptop. That data point ends up being as useful to the argument of this work as any chart.

Wrap-up

This experiment started from a simple question — how do you take a tutorial-style agent and make it survive production? — and ended up giving me a more nuanced answer than I expected: context management matters, a lot, but it doesn't solve every performance problem on its own, and hardware constraints are a legitimate part of the technical conversation, not just a logistics footnote.

All the code is available in the repository for anyone who wants to reproduce or adapt it — including both the successful results with Gemma2 (2B) and the documented limitation with the 9B model, because I believe transparency about what didn't work is as valuable as what did.

If you're working with open models in the region, I'd genuinely love to hear about your experience — what hardware you're running, what you've hit, what context management strategies have worked for you. Reach out on LinkedIn.

This work was also presented as a poster at the Second South American NLP School (Buenos Aires, August 2026).

Most people don't need more meetings—they need meetings that lead to action. At their best, meetings drive decisions and keep work moving forward. But poorly managed meetings can actually slow progress. In fact, our Work Trend Index report found that inefficient meetings are the biggest productivity disruptor at work. Common challenges include staying on track, catching up after joining late, and leaving without clear next steps.

Microsoft 365 Copilot helps teams get more out of every meeting—from spending less time preparing to having more productive discussions and staying aligned on follow-up. The impact can be significant. Microsoft commissioned Forrester Consulting to conduct a Total Economic Impact (TEI) study on Microsoft 365 Copilot in Teams. The study, New Technology: The Projected Total Economic Impact™ Of Microsoft Teams With Microsoft 365 Copilot, projects that companies could realize a potential ROI of 400% over three years.  

Here are some of the latest Microsoft 365 Copilot innovations designed to make meetings and collaboration more effective.

Before the meeting: Plan and prepare faster

Planning a meeting often means juggling calendars, drafting agendas, and gathering context before the conversation even begins. Copilot helps simplify every step.

Scheduling is often the first hurdle. In Copilot Chat, you can simply ask Copilot to set up a meeting, and it handles the coordination—checking availability, suggesting times, and preparing the invite details. It also shows how each option fits into your Outlook calendar, making it easy to choose the best one.

Copilot can also help manage scheduling conflicts proactively. Users can define which 1:1 meetings and personal events are flexible, and when conflicts arise, Copilot can automatically reschedule them and notify attendees of any changes.

Every productive meeting starts with a clear objective. Copilot can create a personalized agenda based on meeting details, attendees, and relevant work. It can also recommend topics from your emails, chats, and recent meetings, giving you a strong starting point that you can easily review and refine before sending the invite.

To help you arrive prepared, Copilot can proactively generate meeting insights directly in the invite. It surfaces relevant context, highlights important information, and suggests useful materials so you can contribute from the start.

During the meeting: Turn discussions into action

Great meetings stay focused, encourage participation, and lead to clear outcomes.

Facilitator works alongside your team in real time to answer questions, track agenda progress, capture notes, and turn conversations into shared tasks and documents. For a more personal experience, Copilot gives you a private space to ask questions and get answers grounded in the meeting, your work, and the web.

The best conversations happen when everyone can participate naturally. Interpreter provides real time speech-to-speech translation, helping teams communicate across languages without breaking the flow of conversation. Live translated captions make it easy for everyone to follow along.  

After the meeting: Keep work moving forward

The real value of a meeting often comes from what happens next. Copilot helps ensure important decisions and action items don't get lost once the meeting ends.

Meeting recap provides AI-generated notes, suggested action items, and personalized highlights, making it easy to catch up on what matters most. It also gives you new ways to revisit the conversation, including audio and video recaps and custom AI summary templates, so you can stay informed in the format that works best for you.

After the meeting, Copilot remains available to help you follow up, explore ideas, and keep work moving forward.

The Meeting Recaps app brings your intelligent recaps together in one convenient, pinned app in the Teams sidebar, making it easier to find and catch up across your meetings.

Get more out of every meeting

Microsoft 365 Copilot customers can start using Copilot in Outlook and Teams today to manage the mechanics of meetings before, during, and after—so they can spend more time moving work forward. Learn more about Microsoft 365 Copilot and explore the resources to dive deeper into each of the features highlighted above.

By Guest Blogger Rafael Ferreira, Microsoft MVP in Azure, Cloud & DevOps engineer based in Florianópolis, Brazil. Community organizer with Azure User Groups Brasil, speaker at DevOpsDays, TDC and KCD, and volunteer mentor in cloud and DevOps programs. He writes in Portuguese at orafaelferreira.com.

From prompting agents to designing reliable loops

I have been working with LLMs and coding agents for about two years, most of that time with GitHub Copilot. Over the last few months, the shape of the work changed: I stopped writing prompts and started designing the thing that writes them.  

That shift has a name now: loop engineering. You replace yourself as the person who prompts the agent and design the system that does it instead. But the interesting part is not the autonomy. It is that a loop only works when something outside the model can say "this is wrong." Agents need ground truth from the environment, not from their own opinion. 

My context makes that impossible to skip. I work on multi-tenant platform engineering on Azure: Terraform, AKS, GitHub Actions. When an agent is wrong in application code, a test turns red. When an agent is wrong here, an environment goes down. 

I use AI agents this way because the stakes in my work are real: a mistake here does not just fail a test; it can take a shared production environment down. Letting the agent handle the repetitive, error-prone checking, and keeping every judgment call for myself, is what lets me move faster without handing away the decisions only a person should make. 

That matters beyond my own setup. Loop engineering gives us a practical way to use AI agents safely in high-stakes engineering. Lightweight hooks, explicit verification gates, restricted tools, and human approval make repetitive work easier to automate while keeping judgment and accountability with the engineer. 

My story: four guardrails that held up in practice

The cheapest verifier in my setup has no AI in it: a thirty-line shell hook catches a mistake and feeds the correction back into the loop. When the hook exits with code 2, whatever it writes to stderr is added to the model's context as an error. The model receives the correction at the exact moment of the mistake, expressed in words I chose. That is the feedback path from the theory, implemented in Bash. 

Guardrails before generation, not after. The standard pattern is generate, then verify. I moved part of it earlier: my rules force the agent to validate provider versions and resource schemas through an MCP server before it writes the first line of Terraform. The cheapest verifier in the world is the rule that keeps the error from existing. 

Order the gates by cost and risk. In infrastructure, each later check runs closer to the real environment and carries a greater potential blast radius. So, the verification ladder is explicit: lint first, validate the configuration and schema, run the plan, then perform health checks after apply. The agent never skips a rung. It is the test pyramid, priced in operational risk rather than execution time. 

Separate the writer from the checker through tools, not instructions. Telling an agent that it is read-only relies on compliance; withholding write tools enforces the boundary. My reviewer agents cannot edit, real clusters give agents read-only access, and only a local cluster accepts agent-initiated writes. That asymmetry is the guardrail. 

One more hook I would write first if I started over: every destructive operation stops and asks for confirmation, in every permission mode, including the one where I told the tool to stop asking. I have already told the story of the Friday I took production down with my name on the log; I am not curious about the version where an agent typed the command. In twenty days, that hook intercepted 100 destructive operations, about five a day. A simple confirmation gate can prevent an automated action from proceeding unchecked.   

Impact and insights 

Two habits changed my results more than any model upgrade. 

The first is an anti-reward-hacking rule: when the agent adds a regression test, it has to prove the test fails if you revert the fix it protects. Agents are excellent at writing tests that pass. "Write a test and show me it catches the bug" is a different request. 

The second is letting the agent read CI results directly, but only the failed steps. The command gh run view --log-failed returns the broken portions of a GitHub Actions run rather than the complete workflow log. That keeps the relevant error visible and avoids filling the agent's context with unrelated output. 

The piece that surprised me most is a weekly scheduled routine that reviews my working sessions and turns recurring patterns into written procedures. On its first real run, it proposed one new procedure worth keeping, improved two existing ones, rejected four and recorded why, and corrected one of my notes that was simply wrong. The loop does not just execute the process; it improves the memory that guides future runs.   

Start with guardrails, not autonomy

The part I will not delegate is the outer loop. The objective comes from a work item, and the merge is mine. Delegating the inner loop is leverage; delegating judgment is abdication. 

If you want to start, do not start with autonomy. Write one hook that blocks destructive commands and forces you to confirm, especially in the mode where you already told the tool to stop asking. It protects against you on autopilot, not against the model, and it costs an afternoon. 

Key learnings

Safer AI agent workflows begin with clear boundaries: ground truth from the environment, verification gates ordered by risk, tools that enforce permissions, and human control over objectives and final decisions. Together, these guardrails turn repetitive automation into dependable engineering practice without giving away accountability.

What is one guardrail you could add today to make your next AI agent workflow safer? Let us know in the comments.

Want to Learn More About the MVP Program?

To find an MVP and learn more about the MVP Program visit the MVP Communities website and follow our updates on LinkedIn or #mvpbuzz.

Join us for a future live session through the Microsoft Reactor where we walk through what the MVP program is about, what we look for, and how nominations work. These sessions are designed to help you connect the dots between the work you’re already doing and the impact the MVP Program recognizes — with time for questions, examples, and real conversations. 

Niki Manoledaki is a Staff Platform Engineer at Grafana Labs, A CNCF Ambassador and Green Software Foundation Champion, and a core maintainer of Project Kepler. We explore the recent rewrite of Project Kepler and the challenges of measuring sustainability in the era of AI.

Do you have something cool to share? Some questions? Let us know:

- web: kubernetespodcast.com

- mail: kubernetespodcast@google.com

- twitter: @kubernetespod

- bluesky: @kubernetespodcast.com

News of the week

• Confidential Containers becomes a CNCF Incubating Project

• KEP-5972 Dynamic Pod Mutation: Optimistic Execution & Hierarchical Resource Delegation

• RedHat - Introducing asago: Open source AI safety and governance orchestration

• Asago on GitHub

• Asago website

• KubeCon NA 2026 Schedule

• CNCF Blog: Welcome CoHDI to the CNCF: Evolving Kubernetes into composable disaggregated infrastructures

Links from the interview

• Project Kepler on GitHub

• CNCF Blog: Kepler, re-architected: Improved power accuracy and a community call to action!

• Green Software Foundation

• "Building Green Software" by Anne Currie, Sarah Hsu, Sara Bergman

• Grafana Labs Blog: How Grafana Labs switched to Karpenter to reduce costs and complexities in Amazon EKS

• GreenIO New York Conference

• AI Energy Score Initiative on Hugging Face

• Dr Sasha Luccioni's:  Power hungry processing: Watts driving the cost of AI deployment?

The release of new large language models (LLMs) continues to accelerate. Thankfully, the vLLM community has worked hard to keep pace with the rapid release of new model architectures, often providing Day 0 support for newly released models.

This leaves users asking the question, "What version of vLLM do I need to run my model?"

Determining compatibility comes down to matching your model's underlying architecture against vLLM release capabilities in three practical steps.

Red Hat validated models

Before inspecting model configuration files manually, check whether your model has already been verified end-to-end through the Red Hat validated models program.

Red Hat engineers test models in the validated models program with official Red Hat vLLM images to verify they run.

You can browse the catalog of validated models on the RedHatAI Hugging Face page (Figure 1).

Alternatively, you can look up specific models on the RedHatAI page and look for the Model validated by Red Hat badge (Figure 2).

In addition to the Model validated by Red Hat badge, the model card specifies which versions of Red Hat OpenShift AI and Red Hat AI Inference were used to validate the model.

Red Hat OpenShift AI users can also find validated models in the Models section of the AI Hub. Models in the AI Hub include performance insights data and let users easily deploy the model using a ModelCar image.

Additionally, Red Hat publishes a support matrix for validated models in the official Red Hat AI Inference documentation, listing the minimum vLLM version alongside the corresponding Red Hat AI Inference and OpenShift AI releases.

vLLM model support fundamentals

While the validated models program can help provide customers confidence in supporting models Red Hat has already tested, users might still find themselves trying to understand if vLLM supports a specific model Red Hat hasn't validated.

To find out, it helps to understand how vLLM handles model support.

vLLM generally doesn't support specific models directly. Instead, it supports model architectures.

For example, in the config.json file for Llama-3.3-70B-Instruct, you can find the architectures attribute:

  "architectures": [
    "LlamaForCausalLM"
  ],

The model's architecture is a named representation of the specific features and structures the model uses. Many different models, even of different sizes, can use these architectures. For example, Llama-3.1-8b-instruct also uses the LlamaForCausalLM model architecture. While Meta created this specific model architecture, other publishers can use it when building their own models.

In most cases, if a model uses a supported architecture and doesn't introduce unsupported customizations, it should run on a vLLM release supporting that architecture. If a publisher releases a new model using that architecture (for example, Meta creating Llama 3.4), that model should run on any vLLM version that already supports the architecture.

Checking the vLLM supported models page

The vLLM supported models documentation is generally the easiest way to determine if vLLM supports a specific model or architecture.

After identifying a model architecture, you can search for that architecture on the supported models page. While the supported models list might not explicitly list a specific model such as Llama-3.3-70B-Instruct, since we know the model architecture is supported, we can confidently assume Llama-3.3-70B-Instruct will run successfully (Figure 3).

Keep in mind that the supported models documentation defaults to the latest release of vLLM, and not all models are backward compatible (Figure 4).

For example, gemma-4-31b-it uses the Gemma4ForConditionalGeneration model architecture, which the latest release of vLLM supports, but older vLLM versions such as v0.18.0 (shipped in Red Hat OpenShift AI and Red Hat AI Inference 3.4) do not support.

Finding Red Hat supported vLLM images

Red Hat distributes vLLM under the Red Hat AI Inference product name. Red Hat AI Inference issues regular releases of vLLM that customers can deploy. Additionally, Red Hat OpenShift AI makes the same Red Hat AI Inference images available through the OpenShift AI platform.

You can find Red Hat AI Inference images in the rhaii namespace on the Red Hat Container Catalog, where Red Hat publishes a unique image depending on the accelerator you use. For example, you can find the NVIDIA CUDA image at rhaii/vllm-cuda-rhel9.

Understanding the vLLM version shipped in each Red Hat AI Inference release is critical to making informed decisions on which Red Hat AI Inference version you might require to run your desired model.

The easiest way to determine which vLLM version ships in each Red Hat AI Inference release is by checking the release notes.

Red Hat AI Inference releases a new version about once a month as either a general availability (GA) release or an early access (EA) release. EA releases aren't supported, but you can use them for testing newer models, while GA releases have a 7-month support window. Because upstream vLLM moves rapidly, developers needing immediate access to newer model architectures can utilize preview images or early access releases between GA cycles.

Day 0 model support

Additionally, Red Hat publishes preview releases such as rhaii-preview/vllm-cuda-rhel9, which you can use to test the latest releases of vLLM and models—often with Day 0 support for new models.

Conclusion

Determining whether your model will run on vLLM comes down to a few practical checks. If you're deploying on Red Hat platforms, start with the validated models program or AI Hub in OpenShift AI to see if Red Hat has already tested your model end-to-end. For everything else, look up the architecture in the model's config.json file, confirm the architecture appears on the supported models page for the version you plan to use, and cross-reference the Red Hat AI Inference release notes if you're running an Red Hat AI Inference image.