{"slug": "prefill-vs-decode-the-two-halves-of-inference", "title": "Prefill vs Decode: The Two Halves of Inference", "summary": "An engineer explains that the pricing difference between input and output tokens in LLM inference stems from the distinct hardware bottlenecks of prefill and decode phases. Prefill is compute-bound, while decode is memory-bound, and the break-even intensity can be derived from chip specifications like the NVIDIA H100. The post also discusses how batching decode requests improves efficiency and influences serving economics.", "body_md": "Every provider charges more for output tokens than input tokens, usually three to five times more. That is not a pricing preference. It falls out of the fact that the two halves of inference hit different hardware limits, and you can derive the ratio yourself from a datasheet.\n\n**Prefill** processes the prompt. Every token in it is already known, so all of them go through the model together, as one large matrix multiplication per layer. Output: the KV cache for the whole prompt, plus the first token’s logits.\n\n**Decode** produces the answer. Each step handles exactly one new token, attends to everything cached so far, and cannot start until the previous step finished. Output: one token, and one more column in the KV cache.\n\nSame weights, same layers, same arithmetic per token. The difference is how many tokens are in flight at once, and that difference is everything.\n\nArithmetic intensity is the ratio of floating-point operations performed to bytes moved from memory. It decides which of the two hardware limits you hit: a low ratio means the chip is idle waiting for weights to arrive (memory-bound), a high one means memory is idle waiting for the maths to finish (compute-bound).\n\nTake a dense model with `P`\n\nparameters stored in 2-byte precision. A forward pass over `N`\n\ntokens costs roughly `2·N·P`\n\nFLOPs — the factor of two is one multiply and one add per parameter per token — and it must read `2·P`\n\nbytes of weights, once, regardless of `N`\n\n. So:\n\n```\nintensity(N) = FLOPs / bytes\n             = (2 * N * P) / (2 * P)\n             = N            FLOPs per byte\n\nprefill, N = 4000 tokens   ->  ~4000 FLOP/byte\ndecode,  N = 1 token       ->  ~1 FLOP/byte\n```\n\nThat is the whole argument, and it is exact to within the terms we dropped (attention itself, which adds work that grows with sequence length, and the KV cache reads, which matter at long context). Prefill reads the weights once and does thousands of operations with each byte. Decode reads the same weights and does one.\n\nA chip has a break-even intensity: its peak FLOP/s divided by its memory bandwidth. Below that ratio you are memory-bound, above it you are compute-bound. NVIDIA’s published H100 SXM specification quotes roughly 990 TFLOP/s of dense BF16 throughput and 3.35 TB/s of HBM3 bandwidth. Divide:\n\n```\nbreak-even = 990e12 FLOP/s / 3.35e12 byte/s\n           ~= 295 FLOP per byte      (NVIDIA H100 SXM datasheet figures)\n```\n\nCompare with the two intensities above. Prefill at a few thousand tokens sits far above 295 and is compute-bound: it uses the expensive part of the chip properly. Decode at one token per step sits at roughly 1, which is more than two orders of magnitude below the break-even point — during decode a card of that class is spending essentially all of its time streaming weights out of HBM and almost none of it computing.\n\nThat is why single-stream decode speed is, to a good approximation, `bandwidth / model_bytes`\n\ntokens per second, and why it barely improves when you move to a chip with more FLOPs but similar memory bandwidth.\n\nThe fix for a memory-bound kernel is to do more work per byte read. Decode has an obvious source of extra work: other users. If `B`\n\nsequences decode in the same step, the weights are read once and used `B`\n\ntimes, so intensity becomes roughly `B`\n\ninstead of 1. At `B`\n\naround 256 you are back near the break-even ratio and the hardware is being used well.\n\nThis is why serving economics are what they are. Decode is cheap per token only when it is shared, and sharing means your request sits in a batch with strangers, which is precisely the source of the latency variance in [continuous batching](https://multigrid.ai/learn/continuous-batching). Prefill needs no such rescue — it is already compute-bound with a batch of one — which is also why long prompts from different users compete with each other for the same scarce resource.\n\nThere is a further consequence that is reshaping how large deployments are built. If the two phases want opposite things from hardware — prefill wants FLOPs and is happy alone, decode wants bandwidth and needs company — then running them on the same device forces a compromise on both. *Disaggregated* serving splits them onto separate pools: prefill workers compute the prompt’s KV cache, ship it over a fast interconnect, and decode workers do nothing but generate at high batch. Each pool can then be sized, scheduled and even specified in hardware for the bottleneck it actually has.\n\nYou cannot see any of this from the API, but it explains an otherwise odd observation: on some endpoints, time to first token and tokens per second appear to move independently under load rather than degrading together. That is what two separately-scheduled pools look like from the outside.", "url": "https://wpnews.pro/news/prefill-vs-decode-the-two-halves-of-inference", "canonical_source": "https://dev.to/multigrid/prefill-vs-decode-the-two-halves-of-inference-4mc8", "published_at": "2026-08-12 16:37:00+00:00", "updated_at": "2026-08-12 16:48:01.200771+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-products"], "entities": ["NVIDIA", "H100"], "alternates": {"html": "https://wpnews.pro/news/prefill-vs-decode-the-two-halves-of-inference", "markdown": "https://wpnews.pro/news/prefill-vs-decode-the-two-halves-of-inference.md", "text": "https://wpnews.pro/news/prefill-vs-decode-the-two-halves-of-inference.txt", "jsonld": "https://wpnews.pro/news/prefill-vs-decode-the-two-halves-of-inference.jsonld"}}