Why LLMs Run Out of VRAM: KV Cache Fragmentation and How PagedAttention Fixes It

A technical explainer details how KV cache memory, not model weights, dominates VRAM during LLM inference: for a 13-billion-parameter model with 40 layers, 40 KV heads and 128-dimension heads at 16-bit precision, the KV cache costs roughly 800 KB per token, so 8 concurrent requests at 4,096-token context exceed the 24 GB of an NVIDIA RTX 4090 before model weights or CUDA overhead are counted. The piece attributes up to 60% to 80% of that VRAM in naive serving setups to memory fragmentation and credits PagedAttention, an approach borrowed from 1960s operating systems, with fixing it.

Why LLMs Run Out of VRAM: KV Cache Fragmentation and How PagedAttention Fixes It You deploy a 13-billion parameter model quantized to 4-bit weights. The static model weights consume roughly 7.5 GB of VRAM. You put it on an NVIDIA RTX 4090 with 24 GB of memory, confident you have more than 16 GB of headroom for traffic. Then you run a batch of 8 concurrent requests with 4,000-tok You deploy a 13-billion parameter model quantized to 4-bit weights. The static model weights consume roughly 7.5 GB of VRAM. You put it on an NVIDIA RTX 4090 with 24 GB of memory, confident you have more than 16 GB of headroom for traffic. Then you run a batch of 8 concurrent requests with 4,000-token context windows. Within seconds, your server throws torch.cuda.OutOfMemoryError: CUDA out of memory or your throughput collapses from 80 tokens per second to single digits. Where did those 16 gigabytes go? Most developers assume GPU memory during inference is dominated by model weights. In reality, once an LLM starts serving concurrent users, the Key-Value KV Cache quickly swallows more memory than the model itself. Even worse: in naive serving setups, up to 60% to 80% of that VRAM is pure waste, locked away by memory fragmentation. Here is what is actually happening inside GPU memory during inference, why naive allocators fail, and how an idea borrowed from 1960s operating systems PagedAttention fixed it. To understand why memory explodes, look at how autoregressive token generation works. Generating text is an iterative loop. When generating token $t$: The token is embedded and passed through $N$ transformer layers. In each self-attention layer, the model computes Query $Q$ , Key $K$ , and Value $V$ projections. Attention is calculated: $$\text{Attention} Q, K, V = \text{softmax}\left \frac{QK^T}{\sqrt{d k}}\right V$$ To compute the attention of the new token $t$ against all previous tokens $ 1 \dots t-1 $, the model needs the $K$ and $V$ vectors of every previous token. Step 1: Prompt: "The", "quick", "brown" - Generates "fox" Step 2: "The", "quick", "brown", "fox" - Generates "jumps" Step 3: "The", "quick", "brown", "fox", "jumps" - Generates "over" Without caching, generating token 1,000 would require recomputing the Key and Value matrices for all 999 previous tokens across every single transformer layer. That would turn inference complexity into $O n^2 $ compute. To avoid this quadratic recalculation, inference engines store the $K$ and $V$ tensors of all previous tokens in GPU High-Bandwidth Memory HBM . On every step, we only compute $Q$, $K$, and $V$ for the single new token, append its $K$ and $V$ to the cache, and calculate attention against the stored history. Compute drops to $O 1 $ per step. But your memory requirement becomes dynamic and strictly increasing. How big is a KV cache in physical memory? For every generated token, at every transformer layer, we must store two vectors $K$ and $V$ . The exact formula for KV cache memory per token is: $$\text{Bytes per Token} = 2 \times n {\text{layers}} \times n {\text{kv heads}} \times d {\text{head}} \times b$$ Where: $2$ accounts for both the Key and Value tensors $n {\text{layers}}$ is the number of transformer layers $n {\text{kv heads}}$ is the number of Key-Value attention heads $d {\text{head}}$ is the hidden dimension per head $b$ is the precision in bytes 2 bytes for FP16/BF16, 1 byte for FP8 Layers $n {\text{layers}}$ : 40 KV Heads $n {\text{kv heads}}$ : 40 standard Multi-Head Attention Head Dimension $d {\text{head}}$ : 128 Precision: 16-bit float 2 bytes $$\text{Bytes per Token} = 2 \times 40 \times 40 \times 128 \times 2 = 819,200 \text{ bytes} \approx 800\text{ KB per token}$$ For a single request processing a 4,096-token context: If you serve 8 concurrent users at 4k context: The KV cache alone exceeds the total VRAM of an RTX 4090 before you even account for model weights or CUDA overhead. Modern architectures use Grouped-Query Attention GQA where multiple query heads share a single KV head. Layers: 32 KV Heads: 8 sharing across 32 query heads Head Dimension: 128 Precision: 16-bit float 2 bytes $$\text{Bytes per Token} = 2 \times 32 \times 8 \times 128 \times 2 = 131,072 \text{ bytes} = 128\text{ KB per token}$$ At 4,096 tokens, one request takes 512 MB. A batch of 32 requests takes 16 GB. If a request only needs 512 MB at 4k tokens, why did naive serving engines run out of memory with only a handful of short requests? The problem lies in how standard PyTorch memory allocators manage dynamic tensors. Standard attention kernels like basic PyTorch MultiheadAttention or cuDNN expect the Key and Value tensors of a sequence to be stored in contiguous virtual and physical memory. Naive Allocator Contiguous Slot per Sequence : Request A: Token 1 Token 2 Token 3 ... Reserved Empty Space up to 4096 Request B: Token 1 Token 2 ............ Reserved Empty Space up to 4096 Because the inference engine cannot predict when a model will emit the <|endoftext| token, it faces three severe structural inefficiencies: To prevent constant memory reallocation and costly CUDA memory copies as sequences grow, naive engines pre-allocate a contiguous memory buffer sized for max model len e.g., 4,096 tokens . If a user prompt only generates 350 tokens and terminates, the remaining 3,746 token slots over 90% of the allocated buffer sit completely unused, yet locked. Even if an engine attempts dynamic chunking, it must reserve memory for future tokens that have not arrived yet. That reserved memory cannot be assigned to other concurrent requests. Requests finish at different times. As short requests finish and free their memory chunks, the GPU memory pool becomes a checkerboard of small, non-contiguous free memory holes. Even if you have 8 GB of total free VRAM, if the largest contiguous block is only 200 MB, a new incoming request requiring a 500 MB contiguous buffer will immediately crash with an Out of Memory error. In research published by UC Berkeley Kwon et al., SOSP 2023 , measurements showed that in traditional serving systems like FasterTransformer and Orca, only 20% to 40% of the allocated KV cache memory actually held useful token states. The remaining 60% to 80% was lost to fragmentation. Operating systems solved the exact same problem for system RAM in the 1960s with Virtual Memory Paging. A process sees a continuous virtual address space $0 \dots N$ , but the OS kernel splits physical RAM into fixed-size 4 KB pages. A CPU Page Table maps each virtual page to any available non-contiguous physical memory frame. PagedAttention the core engine powering vLLM, SGLang, and modern inference backends applies this exact concept to GPU VRAM. Logical KV Cache Sequence 0 : Block 0: Tokens 0-15 - Block 1: Tokens 16-31 - Block 2: Tokens 32-47 │ │ │ ▼ ▼ ▼ Physical GPU Memory Non-contiguous : Physical Block 84 Physical Block 12 Physical Block 903 Fixed-Size KV Blocks: The KV cache of a sequence is partitioned into fixed blocks of tokens typically 16 or 32 tokens per block . Centralized Physical Block Pool: At startup, vLLM pre-allocates all available GPU memory into a pool of physical blocks. Block Tables: For each active sequence, the engine maintains a Block Table analogous to an OS page table . The block table records the physical block ID where each logical block resides, along with how many slots in the latest block are filled. On-Demand Allocation: When a request starts, the engine allocates only one physical block 16 tokens . As the model generates tokens 1 through 15, it writes into the existing block. When it hits token 16, the engine grabs another physical block from the pool and records its address in the Block Table. Standard cuBLAS and FlashAttention kernels require contiguous memory pointers. PagedAttention replaces the attention operation with a custom GPU kernel. During the decoding step, the kernel reads the query token $Q i$, looks up the sequence's Block Table, and dynamically fetches $K$ and $V$ vectors directly from scattered physical memory addresses across GPU HBM during the matrix multiplication loop. Internal Fragmentation: Limited strictly to the last uncompleted block of a sequence less than 4% memory waste with block size 16 . External Fragmentation: 0%. Any free block anywhere in memory can be assigned to any sequence immediately. Effective Batch Size: Increases by 2x to 4x on the exact same hardware. Because memory is managed through indirection block tables pointing to physical blocks , PagedAttention unlocks memory optimizations that are impossible with contiguous arrays. Suppose you want the model to generate 4 candidate completions for the same prompt e.g., beam search, voting, or speculative draft trees . In naive systems, you must duplicate the prompt's KV cache 4 times in memory. With PagedAttention, all 4 sequences point their block tables to the same physical prompt blocks. The physical blocks are marked with a reference count of 4. Request Prompt Tokens 0-31 : Physical Block 7, Physical Block 14 Ref Count = 2 Branch A Tokens 32-47 - Writes to new Physical Block 89 Private Branch B Tokens 32-47 - Writes to new Physical Block 102 Private Only when a branch generates its first new token does the engine allocate a new private block for that specific branch Copy-on-Write . Memory consumption for the prompt is reduced by 75%. In agentic workflows, system prompts, tool schemas, and multi-turn conversation history are reused constantly across requests. With prefix caching enabled, when a new request arrives containing a system prompt that matches a previously computed sequence, vLLM checks its hash table of physical blocks. If the tokens match, it skips the prefill compute phase entirely and simply maps the existing physical blocks into the new request's block table. First-token latency Time-to-First-Token drops from hundreds of milliseconds to near zero, and VRAM usage for the shared prefix drops to 0 additional bytes. If you are running self-hosted LLMs using vLLM, SGLang, or Ollama, here is how to tune these memory parameters for maximum throughput: By default, vLLM sets gpu memory utilization=0.90. This reserves 90% of total VRAM for model weights and the KV block pool, leaving 10% for PyTorch activation buffers and CUDA contexts. For dedicated serving nodes with fixed models, raise utilization to 0.95 vllm serve meta-llama/Llama-3.1-8B-Instruct \ --gpu-memory-utilization 0.95 \ --max-model-len 8192 If you experience random CUDA initialization crashes, lower it to 0.85. The default block size in vLLM is 16. Block size 16: Lowest internal fragmentation. Best for workloads with short, unpredictable outputs. Block size 32: Better GPU memory coalescing and slightly higher arithmetic throughput on high-end GPUs A100/H100 , with slightly higher tail waste. vllm serve meta-llama/Llama-3.1-8B-Instruct --block-size 32 If you run coding assistants, customer support bots, or agent loops that share large system prompts: vllm serve meta-llama/Llama-3.1-8B-Instruct \ --enable-prefix-caching If you are memory-bound rather than compute-bound, you can quantize the KV cache from FP16 2 bytes to FP8 1 byte . This cuts your KV memory usage in half with negligible perplexity degradation. vllm serve meta-llama/Llama-3.1-8B-Instruct \ --kv-cache-dtype fp8 Feature Naive Contiguous Allocation PagedAttention vLLM Memory Layout Contiguous virtual & physical DRAM Non-contiguous fixed-size blocks Memory Waste 60% – 80% Internal & External fragmentation < 4% Last-block tail only Max Batch Size Constrained by peak pre-allocation Dynamic, maximizes hardware saturation Prompt Sharing Requires full memory duplication Zero-copy pointer sharing Copy-on-Write Prefix Caching Complex tensor slicing & copying Instant block hash lookup The next time your local or production LLM runs out of VRAM, remember: the bottleneck is rarely just the model parameters. Understanding how the KV cache grows and how paging eliminates memory waste is the single most important lever for scaling LLM inference throughput. Key Takeaways - •You deploy a 13-billion parameter model quantized to 4-bit weights - •This story was reported by Dev.to , covering developments in the dev space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage. 📖 Continue reading the full article: Read Full Article on Dev.to → https://dev.to/syed anzar/why-llms-run-out-of-vram-kv-cache-fragmentation-and-how-pagedattention-fixes-it-fle