KV Cache and PagedAttention: How to Get More Throughput From the GPU You Already Have

VLLM's KV cache and PagedAttention techniques reduce latency and cost in large language model inference by storing and efficiently managing key-value matrices, enabling higher throughput on existing GPUs. The KV cache avoids recomputing previous tokens' keys and values during autoregressive generation, while PagedAttention addresses memory fragmentation. These methods are implemented in the open-source inference engine vLLM and can be enabled with a simple flag like use_cache=True in Hugging Face Transformers.

You’ve got a large language model running. With one user, time to first token is super quick. At 10 users, latency starts to climb. But at a hundred, you’re watching GPU memory spike and throughput tank — and every wasted cycle is wasted money. Your model is most likely not the culprit here; rather, it’s how your memory is being used during inference. Specifically, it’s how the model stores and retrieves the context it’s building token by token. Today, we’re going to unpack two mechanisms for reducing the wait time and cost associated with inferencing models: KV cache and PagedAttention . They both come out of the open-source inference engine vLLM, and they’ve helped us make strides in what’s possible for LLMs that we infer at scale. By the end of this article, you’ll know exactly how these two techniques work, why they work, and how you can configure them to get dramatically more throughput out of a GPU that you already have. If you’ve used an LLM API, you’ve noticed this. You submit a long prompt and then wait for a bit. Then, suddenly, tokens stream back fast. That lag before your first token is the model processing your prompt. This is the prefill phase — one of the two phases in LLM inference, and it’s highly compute-bound. This is where the model has to run your input through every transformer layer to build a mathematical representation of everything you said before it can produce a single output token. Now imagine you’re doing this for those 100 simultaneous users. Every request has its own growing context, and at every step of token generation the system has to reach back into GPU memory and retrieve that context. This is the decode phase . It’s memory-bound, and it’s using the KV cache. If memory is fragmented, if it’s full, or if it’s being recomputed, it becomes latency you can see. Now that we understand the two phases of LLM inference — prefill and decode — let’s discuss KV cache and PagedAttention. When a transformer generates a new token, every layer computes three things for each token it’s attending to: a query, a key, and a value. The query is the current token asking, “What’s relevant to me?” The key and the value are answering that question. The problem is that in autoregressive generation, you’re producing one token at a time, but you have to rerun attention over all previous tokens on every single step. Without a cache, generating a thousand-token response means the thousandth token has to recompute the keys and values for all 999 tokens before it. The KV cache simply stores those key and value matrices from previous steps so they don’t recompute what you already know. Each new token only needs to compute its own query, key, and value; then it attends over the cached KV history. KV cache is a memory-for-compute trade-off, but it’s proven to be worth it for long sequences. In practice, you rarely wire this up by hand — you just flip on caching and let the framework reuse the past keys and values: python import torchfrom transformers import AutoModelForCausalLM, AutoTokenizermodel id = "meta-llama/Llama-2-13b-hf"tok = AutoTokenizer.from pretrained model id model = AutoModelForCausalLM.from pretrained model id, torch dtype=torch.float16, device map="auto", inputs = tok "Explain KV caching in one line.", return tensors="pt", .to model.device use cache=True reuses previously computed K/V states instead of recomputing them during autoregressive decoding.output = model.generate inputs, max new tokens=128, use cache=True, print tok.decode output 0 , skip special tokens=True That single use cache=True flag is the difference between recomputing 999 tokens on every step and reusing them — which is exactly why memory, not compute, becomes the next thing to worry about. Memory is the real bottleneck in LLM serving, so let’s look at how naive serving allocates GPU memory. A 13-billion-parameter model, something like Llama 13B, takes about 26 gigabytes of GPU memory just for its weights on an A100 40-gigabyte card. That’s already 65% of our available memory — just for the weights. That’s VRAM gone before the user hits a single endpoint. The remaining 35% has to support the KV cache for every active request, and here’s where the traditional system starts to fall apart. These systems pre-allocate a fixed, contiguous block of memory for each request based on the maximum possible output length. So if your max context is 2,048 tokens, but the average user sends in 200 tokens and gets 300 back, that’s 1,500 tokens of reserved memory just sitting empty per request. Research shows that these traditional systems waste about 60 to 80% of the 35% used for KV cache memory, leaving only a small bit actually usable. PagedAttention treats GPU memory the way an OS treats RAM. The KV cache is powerful, but it creates a new problem: how do you store it efficiently for many concurrent requests of wildly different lengths? Traditional systems store each request’s KV cache as a giant contiguous block — like reserving an entire hotel floor for a single guest. So if your max sequence length is 2,048 tokens, that’s how much memory you have on reserve. Even if the user only generates 200 tokens, the rest is wasted and unavailable for anyone else. This is called internal fragmentation . It can be the culprit for KV cache memory waste, along with external fragmentation , where requests of varying lengths leave large gaps between allocations. So if a new request needs 500 tokens, you may very well have enough memory, but no contiguous region big enough for it to work. Also, look out for redundant duplication , where the system prompt is stored separately for every concurrent request. PagedAttention eliminates each one of these issues and applies the same insight that operating systems use for RAM: virtual memory paging. Instead of one contiguous block, it breaks the KV cache into small, fixed page sizes — by default, 16 tokens each. These pages can live anywhere in GPU memory: non-contiguous, allocated on demand. A lightweight block table maps logical page addresses — this is what the model sees — to physical page addresses where they actually are in VRAM. So your GPU memory allocation looks more like this: you still have 65% allocated to the weights, but the rest of that 35% is now packed efficiently instead of sitting reserved and empty. Finally, here are three things you can tune on your deployment to get the most out of your GPU. First, tune GPU memory utilization. This controls what fraction of remaining VRAM goes to the KV cache. The default is 0.9. Push it to 0.95 on stable workloads to pack in more concurrent requests, and pull it back to 0.8 if you’re seeing OOM errors under load bursts. And you can benchmark your specific model before you commit here. If you need something to do that, you can check out GuideLLM — it’s open source and part of the vLLM project. python from vllm import LLMllm = LLM model="meta-llama/Llama-2-13b-hf", gpu memory utilization=0.95, pack more KV cache on stable workloads; lower to 0.8 if OOM errors appear under bursts Raising this fraction hands more of that idle 35% to the KV cache, which is precisely the reclaimed space PagedAttention was designed to fill with concurrent requests. Second, enable prefix caching. PagedAttention hashes each KV block by its token sequence, so requests sharing a system prompt point to the same physical memory — vLLM computes and stores it once. In RAG pipelines, multi-turn chat, and coding agents, hit rates of 75 to 95% are common, with time to first token dropping dramatically because shared prefill is skipped entirely. llm = LLM model="meta-llama/Llama-2-13b-hf", enable prefix caching=True, shared system prompts are computed once and reused This directly attacks the redundant-duplication problem from earlier: instead of storing the same system prompt per request, every request points back to one shared copy. Third, enable chunked prefill. For throughput-heavy workloads, vLLM by default runs prefill to completion before resuming decode, which causes streamed tokens to stutter when long prompts arrive. Chunked prefill batches the decode requests first, then fills the remaining compute budget with prefill chunks. Production deployments have seen a 50% throughput improvement. You can also set max num batched tokens greater than 2,048 alongside it. llm = LLM model="meta-llama/Llama-2-13b-hf", enable chunked prefill=True, max num batched tokens=4096, Example; tune for your latency/throughput target With chunked prefill, vLLM can prioritize decode work and use the remaining token budget for prefill chunks. max num batched tokens becomes an important tuning knob: lower values generally favor decode latency, while higher values can improve prompt processing. Finally, a bonus feature for latency-sensitive workloads is enabling a speculative decoding model. During decode, your GPU has spare compute sitting idle between memory reads. A small draft model proposes a series of output tokens, and then a larger model verifies these in one forward pass. If they’re good to go, they get sent forward; if not, the wrong ones get corrected. Output quality here is mathematically identical to running the large model alone. At very high concurrency, the gains shrink, since the batch is already keeping the GPU busy — so reach for this when interactive latency matters more than raw throughput. vLLM also supports lightweight n-gram speculation, which doesn’t require a separate draft model and can be useful for prompts with repetitive patterns. vllm serve meta-llama/Llama-2-13b-hf \ --speculative-config '{ "method": "ngram", "num speculative tokens": 5, "prompt lookup max": 4 }' vLLM also supports n-gram speculation, which generates draft tokens by matching patterns in the prompt and doesn’t require a separate draft model. The lesson here is one worth carrying into every deployment: Better LLM serving rarely starts with a better model or a bigger GPU. It starts with using the memory you already have more efficiently. KV cache eliminates redundant compute, and PagedAttention makes that cache practical at scale. Everything after that is tuning the serving stack to fit your workload, not the other way around. So before you reach for new hardware, profile your inference, find where memory is actually being wasted, and fix that first. The throughput you were looking for is usually already sitting on the card. I hope this was helpful. I’d love to know: what’s the biggest memory-related problem you’ve hit in production AI development? Feel free to drop it in the comments. If you found this helpful, consider clapping👏 so others can find it too and follow me for more amazing technical AI content KV Cache and PagedAttention: How to Get More Throughput From the GPU You Already Have https://pub.towardsai.net/kv-cache-and-pagedattention-how-to-get-more-throughput-from-the-gpu-you-already-have-a29b74acf10e was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.