If you have sat in a meeting about running language models in production, you will have heard some of these: KV cache. Prefill and decode. Quantization. Continuous batching. FlashInfer. Tensor parallel. Goodput.
They arrive together, at speed — normally from somebody asking for a budget. They are good words. The problem is that nobody explains what sits underneath them, so they get memorized as a list instead of understood as consequences.
In this series, I will try to answer what the thing is, how it works, why you would care, and what it looks like on a system taking real traffic. Start here, because the rest of it is downstream of one division.
The picture above is the series. From the bottom: silicon is the GPU, the chips that calculate and the memory that feeds them. Kernels are the small programs that run on it. Memory and precision covers how the model’s numbers are stored, and how much of your card each conversation eats. The engine is what you actually install — almost always vLLM — and it decides whose request runs next. Distributed serving is what happens when one machine is not enough. Workload and SLOs at the top is your real traffic: prompt lengths, how many people are asking at once, how fast you promised to answer.
Requests enter at the top. The work happens all the way down.
One token, six layers. What each layer decides is on the right, in the order it happens to you. Any one of those six can be the thing you are actually waiting for, which is why there is a part for each.
One thing does not read in that order, and that is deliberate. It goes: this part, then the memory each conversation eats, then the kernels underneath it, then how the numbers are stored, then the engine, then more than one machine, then how to tell whether any of it worked. That is not bottom to top. It doubles back once, and two of the parts share a layer, because the memory band carries both the cache and the precision and each is big enough to need its own article.
The order is by what you need first rather than by what sits where. The cache is the constraint you will actually hit, so it comes early. Kernels are the layer you are least likely to ever touch yourself, so they wait until part two has given you a reason to care about them.
Training builds the model. It happens once, costs a fortune, and somebody else has usually done it. Inference is using the model, and it is what you pay for forever. Inference is two jobs that feel like one:
Prefill is the model reading your prompt, all of it at once. Decode is the model writing the answer one token at a time — a token being roughly three quarters of a word, each one waiting for the last. They look like one job and behave nothing alike.
Eight tokens of prompt cost one read of the model. Eight tokens of answer cost eight. That asymmetry is the rest of this article.
A GPU spec sheet gives you two figures. The first is arithmetic speed: an H100 does about 990 TFLOP/s, which is 990 trillion operations a second. Take it as unimaginably fast. The second is memory bandwidth, how quickly the chip can pull data out of its own memory — about 3.35 TB/s. Also fast, but not in the same way. Divide them:
990 trillion operations per second ÷ 3.35 trillion bytes per second= about 296 operations per byte
Both numbers belong to one card. The left half is the arithmetic you bought; the right half is everything the model has to be fetched from; and the arrow is the only way between them. The ridge point is the ratio of the two, which is why it is a property of your hardware and not of your model.
Picture a very fast chef and a pantry down a long corridor. Each trip costs the same whether you carry one ingredient or an armful. Make one omelette per trip and you spend the day walking; the chef’s speed is irrelevant. The chef only becomes the limit at roughly 300 dishes per trip.
That ratio, dishes per trip, is arithmetic intensity. The turning point is the ridge point, and the chart of it is a roofline.
Left of the dashed line, fetching is your limit. Right of it, arithmetic is. Prefill lives on the right. Decode does not.
Prefill is comfortable. Each chunk of the model gets loaded once and multiplied against your whole prompt. A 4,000-token prompt is around 4,000 operations per byte, far past 296.
Decode is the problem. To produce one token the model reads every weight it has, and how much that is depends entirely on how the numbers are stored:
70B parameters × 2 bytes (BF16, the usual default) = 140 GB70B parameters × 1 byte (FP8) = 70 GB70B parameters × 0.5 byte (INT4) = 35 GB
Seventy gigabytes of memory traffic for three quarters of a word. That is the sentence I would tattoo on this series.
Here is the fact that makes any of this fixable: the weights do not depend on who is asking.
Your next token and a stranger’s next token come out of exactly the same 70 GB. Different conversations, different histories, identical matrices. So the GPU does not fetch them once for you and then again for them. It fetches them once, and multiplies them against both requests in the same pass.
That is batching, and it is worth being precise about what it is not. It is not a queue. It is not waiting for a group to fill up. It is some number of requests going through the model together, in one sweep, each of them riding on a read that was happening anyway.
Which answers the objection you should be having. Adding a fiftieth person does not make the other forty-nine wait their turn, because nobody was taking turns. The fetch was the expensive part and they share it. What that fiftieth person costs you is arithmetic — and arithmetic is precisely what you had spare.
The read is the same in every frame. Seventy gigabytes for three quarters of a word is the batch-of-one price, and this is where it stops being true.
So: one person, and that 70 GB produces a single token, which is as wasteful as it sounds. Fifty people, and the same read produces fifty. One trip, fifty dishes. Which gives a tidy result:
In decode, arithmetic intensity is just the number of users you serve at once.
Batch of 32 puts you at a tenth of what the card can do. Batch of 296 reaches the ridge, which almost nobody sees in practice.
Batch 1 to 512. Gold is the bulk of the model, climbing as users share each trip. Blue is the part that never moves.
All of that assumes a dense model, where every parameter works on every token. Many current models are not. A mixture-of-experts model splits its bulk into sub-networks and a router picks a few per token: Qwen3.6–35B-A3B holds 35 billion parameters and uses about 3 billion on any given one.
Attention is identical in all three. What changes is how much of the feed-forward half you have to fetch.
Two things follow, and the second one is the one that gets missed. It cuts traffic, not capacity. Every expert still has to sit in GPU memory, because the router can send the next token to any of them. Active parameters govern what you read; total parameters govern what you buy.
Batching takes the saving back. One user touches a few experts. Two hundred users scatter across all of them, so you read almost everything anyway — except now each expert got a thin slice of work. The fraction that matters here is not the parameter count: it is how many of the experts fire. Qwen3.6–35B-A3B routes 8 of its 256 experts per token, so:
dense intensity = BMoE experts intensity ≈ B × 8/256 = B × 0.031
Reaching the ridge would need a batch near 9,500. Nobody serves that, so MoE decode stays bandwidth-bound at any concurrency you will really run. It is excellent for one user on one card, which is why it owns local inference. At serving scale it moves the problem rather than solving it.
Batching lifts the weight matrices up the roofline. It does nothing at all for attention — not at batch 8, not at 512.
Most of the model is shared weights that every request multiplies against, so batching helps. But the model also remembers your conversation, and that memory — the KV cache — is yours alone. Fifty users means fifty separate histories fetched, with nothing to spread the cost over.
There is one thing spreading it, and it is worth naming because part two is about it. Several query heads share each stored key-value pair, so the same fetched bytes do serve more than one head: eight of them, on the models in this series. That puts decode attention at an intensity of about 8 rather than 1. Against a ridge of 296 it makes no practical difference, you are still deep in bandwidth-bound territory, but 8 is the honest number, and the mechanism that produces it is the same one part two credits with making long context possible at all.
This one takes a while to properly absorb, and it is the first thing worth checking when a long-context feature is slow. You cannot batch your way out of a cost that is per-user by definition — which is why the engineering went into making that cache smaller rather than the maths faster.
Work out your own ridge point. Peak arithmetic rate divided by memory bandwidth, both off the spec sheet of whatever you actually run on. One division, and the answer is not a constant. (Drag the batch slider here if you would rather watch it move.)
H100 SXM 990 TFLOP/s ÷ 3.35 TB/s = 296A100 80GB 312 TFLOP/s ÷ 2.04 TB/s = 153L40S 362 TFLOP/s ÷ 0.864 TB/s = 419
The L40S is the one to stare at. It is the cheaper card, and it has the harder number to reach — not because it has more arithmetic (it has far less than an H100) but because it has much less bandwidth per unit of arithmetic. Moving a workload there to save money means you need a bigger batch to break even, not a smaller one. That does not come up when someone quotes you the hourly rate.
Find the batch size you are really getting. Not --max-num-seqs, which is only a ceiling. The number the scheduler actually reaches under your traffic: vllm:num_requests_running on the metrics endpoint, or the Running: N reqs vLLM prints as it works. If that number is 8 on an H100, you are at 8 against 296, which is under 3% of the arithmetic you are being billed for.
And treat any tokens-per-second figure quoted without a batch size as unfinished. Here is why, using only numbers already on this page.
Take a 70B model stored at one byte per parameter. Every single token it writes means reading all 70 GB. An H100 moves 3.35 TB every second, so one of those reads takes about 21 ms. Call it 48 tokens a second, or roughly 36 words a second — already faster than anyone can read — and that is what one person alone on that card gets.
Now sit 32 people down at it, each holding a short conversation of about 2,000 tokens. The weights are still a single read, which is the entire point of batching. But you are now also fetching 32 private caches, another 21 GB, so the step costs 27 ms instead of 21:
Three things happened at once. The card’s output went up twenty-four times. Each individual person’s went down. And how far down was decided entirely by how long their conversations were, because the cache is the only term in this that grows.
So when a vendor tells you their system does a thousand tokens a second, they have told you what their machine adds up to and nothing whatsoever about what any one of your users will see. They also have not told you the context length, which is the variable that pulls those two numbers apart. Ask for all three. I will keep saying that in every part.
Language models feel slow in a way that does not match the hardware bill because decoding is a memory problem wearing a compute problem’s clothes. The GPU is not struggling to do the arithmetic. It is struggling to fetch the things it does arithmetic on — every number the model is made of, and every token of the conversation so far, once for each new token it writes.
Everything after this — the cache, the kernels, the precision, the settings, the second machine — is a different answer to that one sentence. Which is why the most useful number in the whole stack costs you one division and almost nobody has it.
** Part two is the KV cache**: what that private memory costs, and why your context length quietly decides how many customers fit on a card.
Hardware figures are H100 SXM specifications, and the 990 TFLOP/s is the dense BF16 rate — NVIDIA’s page prints 1,979 with sparsity, which is double. A100 figures are the SXM part; the PCIe card has less bandwidth and a different ridge. Every number here is one division you can repeat, llm-inference-arithmetic is the calculations as a Python package, MIT, no dependencies.
Start Here: The Words Everyone Uses About LLM Inference was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.