The Cost of One Token

🚧Memory-Bound Decode#

It waits on bandwidth, not compute

Every time a model produces a token, it rereads nearly all of its weights from memory. A 70B model loaded at 16 bits has about 140GB of weights alone, so when requests are handled one at a time, memory bandwidth hits its limit before the compute cores do. That is why most ways of cutting inference cost come down to getting more tokens out of each read, or reducing the bytes that have to be read.

Prefill and Decode#

Two phases, two different bottlenecks

Prefill, which reads the prompt, processes the input tokens all at once, so its bottleneck is compute. Decode, which writes the answer, produces tokens one by one and rereads the weights and KV cache each time, so its bottleneck is memory. Time to the first token is set mainly by prefill, and the gap between later tokens by decode. That is why research such as DistServe and Splitwise proposed running the two phases on separate GPUs.

🛠️Serving Techniques#

Three ways around the same bottleneck

There are three main ways to lower cost per token on the serving side without changing the model. Batching produces tokens for many requests from a single read of the weights, quantization shrinks the bytes to be read, and speculative decoding confirms several tokens with a single verification pass. Batching and speculative decoding leave the model's output distribution unchanged. Quantization gives up a little quality to free up memory.

Continuous Batching and the KV Cache#

One read, shared across many requests

Continuous batching does not wait for a request to finish. At every decode step it slots new requests into free places, producing tokens for many users from a single read of the weights. What limits the batch size is the KV cache that builds up for each request. The 2023 vLLM paper reported that existing systems wasted 60 to 80% of the memory set aside for the KV cache, and by managing the cache in pages it raised throughput 2 to 4 times at the same latency.

Quantization#

At 8 bits, half the bytes to read

Shrinking weights from 16 bits to 8 halves what has to be read from memory for each token, and 4 bits cuts it to a quarter. Weight-only methods such as GPTQ and AWQ pay off immediately in decode, where memory is the bottleneck, while W8A8, which also brings activations down to 8 bits, speeds up prefill, where compute is the bottleneck. Quality can drop as bits are removed, so the bit width is chosen by re-evaluating on the actual tasks the service handles.

Speculative Decoding#

A small model drafts, the large model checks in one pass

A small draft model proposes a few tokens first, and the main model verifies them together in a single forward pass. Because only accepted tokens are kept, the output distribution is the same as if the main model had worked alone. In 2023 Google researchers reported a 2 to 3 times speedup on T5-XXL without changing outputs, and the real gain depends on how often draft tokens are accepted.

💸Working Out Token Cost#

Double the throughput, halve the cost

If you serve a model yourself, the cost of one token is the GPU's hourly cost divided by the number of tokens produced in that hour. Serving techniques and model designs lower that cost by raising throughput, not by changing what the GPU costs. Even as the unit price of a token falls, though, a product that calls the model many times per task, as agents do, can see its call volume grow faster, so the total bill can still rise.

The Token Tax#

Inference cost lands in cost of revenue

For traditional SaaS, serving one more user costs little, but an LLM product spends GPU time on every request. So inference cost goes into cost of revenue, and on a flat-rate plan the heaviest user becomes the most expensive one. You need to know the input tokens, output tokens and cache-hit rate of a single request before you can tell whether a pricing plan makes money.

Cache-Hit Discounts#

A prompt that isn't reread costs less

When every request carries the same system prompt or long document up front, reusing the KV cache for that opening part lets the model skip that much prefill. This is why major model APIs charge less for cached input tokens than for regular input. A cache only hits on a part that matches exactly from the beginning, so simply putting fixed content first and changing content last can change the hit rate.

📐Model Design#

Cutting what must be read at design time

If serving techniques are ways to run a given model cheaply, model design is a way to reduce, from the outset, how much has to be read and computed for each token. Mixture of experts (MoE) cuts the parameters used per token, and MLA cuts the KV cache that builds up per request. DeepSeek-V3 uses both, was trained with FP8 mixed precision, and released its weights in FP8 as well.

Mixture of Experts (MoE)#

37B of 671B per token

DeepSeek-V3 activates only 37B of its 671B total parameters to process each token. Compute per token and the weights to be read shrink to the active parameters, but because there is no way to know in advance which experts will be called, all 671B must stay loaded in memory. MoE saves compute at the price of more memory capacity, and designs that spread experts across multiple GPUs follow.

MLA#

Compressing the KV cache to grow the batch

Multi-head latent attention (MLA) does not store keys and values as they are. It compresses them into small latent vectors and caches those instead. The DeepSeek-V2 technical report states that this model, which uses MLA, has a KV cache 93.3% smaller than DeepSeek 67B's. When each request takes up less memory, the same GPU can hold more requests and longer contexts.

🧑‍💻Serving Engineer#

The role that cuts the cost of a token

As inference cost came to drive product margins directly, running models cheaply and quickly became important work in its own right, separate from building them. The measure of this role is choosing and tuning serving engines such as vLLM, SGLang and TensorRT-LLM to get more tokens out of the same GPU. Most branches of this map are settings this job adjusts every day.

The Serving Engineer's Work#

From measuring the workload to decode settings

The first step is to measure the workload and work out whether prefill or decode is the bottleneck, and whether the latency target is time to the first token or the gap between tokens. Next come adjustments to batch size, KV cache memory allocation, quantization bit width and whether to use speculative decoding, followed by a check that quality holds. Each time a setting changes, throughput, latency and quality are measured again together.