# You're Not Paying for Compute. You're Paying for Memory Bandwidth > Source: > Published: 2026-07-11 13:00:52+00:00 TL;DR—Inference cost conversations obsess over FLOPs and token prices, but the real constraint on LLM serving is memory bandwidth— specifically the cost of moving the KV cache in and out of GPU memory on every decode step. Teams that optimize for compute utilization instead of memory traffic end up overpaying for capacity they never use. The fix is architectural: disaggregating prefill from decode, right-sizing batch and context, and treating bandwidth as the scarce resource it actually is. Every infrastructure conversation about running large models starts with FLOPs. GPU spec sheets lead with them. Cost calculators multiply them against price-per-hour. Vendors brag about them. And almost none of it matters for the workload most teams actually run: autoregressive decoding at scale. The thesis here is simple and under-discussed: for the majority of production LLM inference, the bottleneck isn't compute. It's memory bandwidth. Specifically, it's the cost of shuttling the key-value cache in and out of high-bandwidth memory on every single decode step. If you're sizing infrastructure, negotiating vendor contracts, or designing a serving stack around FLOPs, you're optimizing the wrong variable, and you're probably overpaying. Prefill— the pass where the model processes your prompt— is genuinely compute-bound. It's a big matrix multiply, GPUs are extremely good at big matrix multiplies, and utilization can climb close to peak. This is the phase FLOPs benchmarks measure well, and it's the phase that makes GPU spec sheets look impressive. Decode is a different animal. Generating each new token requires reading the entire KV cache accumulated so far, plus the model weights, to produce a single vector. The actual arithmetic per token is tiny relative to prefill. What dominates is data movement: pulling gigabytes of cached keys and values off memory for every single token you emit. Do this one request at a time and your expensive accelerator spends most of its cycles waiting on memory, not computing anything. The roofline model gives you the right lens: arithmetic intensity, the ratio of compute operations to bytes moved. High-intensity workloads are compute-bound and benefit from more FLOPs. Low-intensity workloads are bandwidth-bound and get no benefit from a faster chip— they need faster memory, or they need fewer bytes moved per unit of useful work. Decode-heavy inference sits firmly in low-intensity territory. This is why doubling a GPU's advertised compute throughput barely moves the needle on decode latency or cost, while a memory-bandwidth-limited chip with mediocre FLOPs can be a much better economic fit for the same workload. Buying decisions made off a FLOPs-per-dollar spreadsheet are, for this reason, systematically wrong for decode-dominated traffic. Context length makes this worse, and it makes it worse nonlinearly. The KV cache grows linearly with sequence length and with the number of concurrent sequences you're serving. Every token of context you keep alive is bytes you must move on every subsequent decode step, for the lifetime of that request. A chat session with a long history isn't just "a bigger prompt"— it's a permanent tax on every future token generated in that conversation, paid in memory bandwidth, not compute. This is the uncomfortable part of the long-context pitch. Longer context windows are marketed as a capability upgrade. Economically, they're a bandwidth liability that compounds with every additional token retained. Teams that adopt aggressive context lengths without changing their serving architecture often see cost-per-token climb even as the price sheet stays flat, because the KV cache traffic— not the token count— is what's actually billed against their hardware. The most consequential infrastructure shift in serving architecture over the past couple of years has been separating prefill and decode onto different hardware pools, sometimes literally different machines connected by a fast interconnect. This isn't an aesthetic choice. It's a direct response to the fact that these two phases have opposite resource profiles: one wants raw compute, the other wants bandwidth and cache locality. Colocating them on the same GPU means you're constantly compromising: either you underutilize compute during decode, or you starve prefill of memory bandwidth during bursts. Disaggregation lets you provision each phase against the resource it actually consumes, and it lets you batch decode requests across many concurrent sequences to amortize the fixed cost of memory reads— batching decode is, in effect, the only lever that meaningfully improves arithmetic intensity for that phase. The tradeoff is added system complexity: request routing, cache transfer between pools, and coordination overhead that a monolithic design avoids. That complexity is the real price of matching hardware to workload shape, and it's why disaggregated serving remains an infrastructure investment, not a config flag. If memory bandwidth is the real constraint, several conventional instincts need correcting. None of this means compute is irrelevant— prefill-heavy workloads, batch scoring jobs, and training all live in genuinely compute-bound territory, and FLOPs comparisons are fair there. But the dominant production pattern for most teams— many users, long conversations, continuous decode— lives on the other side of the roofline. Pricing, capacity planning, and hardware selection built on the wrong side of that line don't just cost more. They cost more in a way that scales invisibly with every additional token of context your users generate. The FLOPs number on the spec sheet is a compute story. The bill you actually pay is a memory story. Treat them as the same thing and the economics of scale will keep surprising you, always in the same direction.