Paying a cloud API per million tokens is straightforward until the invoice arrives. But when you move to self-hosted large language models (LLMs), the economics flip upside down: you pay for raw compute regardless of whether your GPUs are processing requests or idling.
For platform engineers and machine learning operations (MLOps) leads, tokenomics, the economics of how tokens are produced and what they effectively cost, requires looking at both what you spend and how much you serve.
Calculating cost per token for self-hosted LLMs comes down to controlling 2 main levers: spending less and serving more.
A simple formula
At its core, cost per token over a period of time is:
Cost per token = operating cost ÷ tokens processed
To improve that metric, you can reduce the overall cost, increase the number of tokens processed, or both.
Suppose your LLM infrastructure costs about US$50,000 per month and you process 500 million tokens in that month:
US$50,000 ÷ 500M tokens ≈ 10 cents per million tokens
If you cut monthly costs to US$40,000 without changing usage, cost per million tokens drops to about 8 cents. If, instead, you keep the US$50,000 spending but grow usage to 1,000M tokens, cost per million tokens drops to about 5 cents. Both paths improve the metric.
Self-hosted LLM cost breakdown
Cost can be broken down into a few high-level categories:
- Hardware/infrastructure
- Software
- People
- Other
Hardware and infrastructure costs
In a cloud model, hardware and infrastructure costs are generally straightforward because they appear on a monthly provider bill. You pay a fixed hourly rate for your instance type, plus additional fees for storage and network egress.
However, if you are self-hosting your own hardware, things can get a little more complicated. Let's say you purchased a node for US$500,000 as a capital expenditure. To understand the cost of that node for a single month, you would need to understand the expected lifespan of that node and depreciate the cost of it over that period of time. In simpler terms, if you expect to use that node for 4 years, you can take US$500,000/48 to get a monthly cost of about US$10,400. In addition to the nodes themselves, you probably also need to factor in other hardware such as networking hardware.
For both scenarios, you might also want to factor in the cost of other hardware in the cluster, such as your control planes. If you are running your GPU workloads in a multi-tenant environment with other use cases, you might wish to spread these costs across all of your use cases.
Software and personnel costs
Software is another area to factor in. If you are using Red Hat OpenShift AI, the most common stock-keeping unit (SKU) purchased by customers is Red Hat OpenShift AI Enterprise, which includes entitlements for Red Hat OpenShift, Red Hat OpenShift AI, and accelerators on that node. While some customers might choose to pay for the individual pieces of software separately, these are the main software costs from Red Hat.
The cost of the people needed to deploy, support, and manage your LLM infrastructure is an often overlooked aspect of running your own LLMs. Organizations frequently have a dedicated team managing multiple OpenShift clusters, and a portion of that team's time can be allocated and tracked toward the cost of running your own LLMs. Enterprise AI platforms reduce engineering labor by standardizing model delivery. Instead of developers building custom container images and updating vLLM runtimes manually, automated platform catalogs handle runtime maintenance, but people remain part of the true operating cost.
For a complete total cost of ownership (TCO), also account for facility overhead: power consumption, cooling, datacenter staffing, and physical rack footprint.
Optimizing costs to reduce LLM cost per token
To lower the US$50,000 monthly spending from our example, start by targeting your biggest infrastructure costs.
Platform teams often default to top-tier GPU instances out of caution. But locking up an 8xH100 node for a low-traffic internal chatbot drains budget unnecessarily. Matching GPU capacity to actual usage profiles, such as swapping a p5.48xlarge for a budget-friendly g6e.48xlarge with eight L40S GPUs, keeps costs grounded.
Autoscaling both the model server replicas and the nodes in the cloud environment can also have a dramatic effect on the cost of running an LLM. Letting instances scale down to a minimum number of replicas while usage is low, and automatically scaling them back up when traffic is heavier, is an effective way to reduce overall costs while still maintaining required service level objectives (SLOs).
For self-hosted environments, you can still take advantage of autoscaling to reduce how much hardware a specific model is using and free up those resources for other use cases, such as overnight batch training jobs, to spread the cost of the hardware across multiple workloads. Additionally, while you are committed to the hardware you have purchased, spending more time up front to understand what models you plan to deploy, and how many requests or tokens you need to serve, can help you right-size before making hardware purchases.
Software choices also affect people cost. Platforms like OpenShift AI make it easier to source models and run supported vLLM releases from a container registry, which reduces the time teams spend building and maintaining custom inference images, and this time savings helps to improve operating costs.
Tokens processed
Tokens processed can be evaluated from two different lenses: your theoretical maximum (how many tokens the system could process) and your actual tokens processed (how many you did process over a period of time).
You can derive the theoretical maximum by performing load testing with tools such as GuideLLM, which simulates realistic workloads and ramps up concurrent connections until performance starts to degrade. Based on that result, you can estimate the theoretical maximum number of tokens you can process in a period of time. For example, if you were able to process 1M tokens per minute, you could theoretically process 1,440M tokens per day.
In most scenarios, you won't be able to sustain that level of maximum load 24x7. However, the theoretical maximum is still useful for right-sizing and capacity planning. For example, it helps you decide how many replicas you need to meet real-world demand. If you have built a system that can process 1,440M tokens per day, but you are only processing 50M, you might be over-provisioned relative to demand, which keeps the cost side of the formula high relative to the tokens side.
Measuring real-world token processing is a better metric for deriving cost per token. Returning to our example: US$50,000 ÷ 500M tokens is your true cost per token for that month, not US$50,000 divided by the theoretical maximum you could have served if the system ran flat-out.
Red Hat OpenShift AI helps make it easy to measure your real-world token usage through its integration with Red Hat OpenShift metrics and Prometheus, and helps track usage by individuals and teams with Models-as-a-Service.
Optimizing tokens processed
The biggest effect on cost per token usually comes from increasing overall usage, not from squeezing more peak throughput out of an underutilized system. If your system can process 1,440M tokens per day but you are only processing 500M, consolidating internal AI workloads onto a shared cluster fills idle GPU cycles. Running background batch processing alongside interactive chat directly increases token volume without adding hardware cost. In our monthly example, growing from 500M to 1,000M tokens at the same US$50,000 spending halves the cost per million tokens.
It also helps to better use the system in non-peak hours. Shifting batch workloads to off-peak hours can raise the number of tokens processed without adding hardware, while keeping performance high for peak interactive traffic.
Teams often turn next to optimizing the model deployment itself by tuning serving configuration, batching, and related settings to raise throughput. That work primarily increases theoretical maximum capacity, not actual tokens processed. If you optimize a deployment from 1,440M to 1,800M tokens per day of capacity (+25%) but real-world usage stays at 500M, the cost per token doesn't change.
Those optimizations still matter as enablers: higher capacity can absorb additional use cases without new hardware, or let you scale down to fewer resources if demand is already met. Treat them as a way to either grow the tokens side or shrink the cost side, not as an automatic win on cost per token by themselves.
Tuning with vLLM and llm-d
By default, vLLM balances throughput against request latency with zero setup. However, tuning settings such as batch size and block allocation for your specific workload unlocks significantly higher throughput while staying within your SLO thresholds.
Llm-d also offers a number of performance tuning options that can positively affect cost per token. Capabilities such as intelligent routing create more opportunities to take advantage of key-value (KV) cache hits when you have multiple vLLM replicas. Additionally, KV cache offloading allows you to use CPU memory to expand the KV cache beyond what can be stored in vRAM, and prefill and decode disaggregation can help reduce bottlenecks in vLLM for larger-scale deployments.
The impact of model choice
The model you choose pulls on both sides of the formula. Larger models usually raise the cost side through more GPUs per replica. They can also change how many tokens you can serve per dollar on the tokens side through throughput and latency characteristics.
A model such as Llama-3.1-8B-Instruct is significantly smaller than Llama-3.3-70B-Instruct. The 8B model can easily fit on a single H100, or a more budget-friendly GPU like an L40S, while the 70B model might require up to four H100s to serve a single instance. Choosing the smaller model when quality is "good enough" for the use case can cut hardware costs sharply while also enabling higher total throughput, and, if that frees budget or capacity for more traffic, the cost per token improves further.
Quantization is another lever. An FP8 (8-bit floating-point) version of the 70B model can be deployed on four H100s instead of four, reducing the required hardware while aiming to preserve much of the model's quality.
Quantization cuts GPU memory requirements—for instance, compressing FP16 weights to FP8—allowing you to double batch sizes on identical GPUs or host models on fewer accelerator cards.
Input tokens, cached tokens, and output tokens
When dealing with cloud providers, you will often see different prices depending on the type of tokens being processed. Even when you self-host, those same token types still have different effective costs because they consume the GPU differently. Your mix of input, cached, and output tokens therefore shapes the real cost per token, even when you are not paying a published per-token rate.
Evaluating input and cached tokens
Input tokens are generally processed in a single pass. Model servers like vLLM can take advantage of continuous batching here, so input-heavy workloads are often highly parallelized and relatively efficient per token.
Cached tokens are tokens already computed and available in the GPU's KV cache (the key-value cache vLLM uses to avoid recomputing values for repeated context). vLLM can reuse cached tokens when queries share common system prompts, or in multi-turn conversations where chat history is resent with each request. Cached tokens are generally inexpensive because the value is looked up rather than recomputed on the GPU.
Projects such as llm-d can increase the chance of a KV cache hit in multi-replica deployments through intelligent routing that sends related requests to replicas more likely to already hold the relevant cache, which improves total throughput and can reduce the effective cost when your traffic has shared context.
Output tokens are highly iterative: each token is predicted one at a time. That process generally makes output tokens more expensive in GPU time than input or cached tokens, so output-heavy workloads push the effective cost per token higher for the same headline token count.
Optimizing output tokens with speculative decoding
Why are output tokens so expensive? GPUs must generate them sequentially, one by one. Speculative decoding solves this bottleneck by letting a lightweight draft model guess ahead while the primary model verifies predictions in parallel. Accepted tokens are kept, and rejected ones are regenerated. The result is faster output generation with the same final text, raising the number of useful tokens served per unit of GPU time when verification succeeds often enough.
Final thoughts
Self-hosted LLM pricing is less about a sticker price per million tokens and more about understanding what you spend and how much you serve. Cost per token is operating costs (hardware, software, people, and whatever else you include) divided by tokens processed over the same period. You improve that number by cutting spending through right-sizing and autoscaling, by increasing real-world usage so idle capacity is put to work, or by choosing models and serving techniques that better match your workload.
The most useful takeaway is to measure both sides of the equation. Track actual tokens processed, not just the theoretical maximum throughput, and account for how input, cached, and output tokens behave differently. With that visibility, model choice, quantization, speculative decoding, and routing for cache hits become deliberate levers rather than guesswork, and you can compare self-hosted economics against managed APIs on equal footing.
The post
The tokenomics of self-hosted LLMs appeared first on
Red Hat Developer.
Log in
