High-Throughput LLM Inference & Training: A Deep Dive into vLLM An engineer at g factor detailed how vLLM's PagedAttention and continuous iteration-level batching solve the memory-bandwidth bottleneck in production LLM inference, drawing on benchmarks run on dedicated NVIDIA H100 and H200 clusters. The writeup explains how naive PyTorch and Hugging Face pipelines suffer external memory fragmentation and CUDA out-of-memory failures under multi-tenant load, and how paging the KV cache into fixed-size blocks mirrors classic OS virtual memory. The author argues text generation is fundamentally a memory bandwidth problem disguised as a compute problem. Editor's Note: Originally published on the g factor engineering blog https://www.g-ftech.com/blog/vllm-throughput-deep-dive . All benchmarks and telemetry in this article were conducted on dedicated NVIDIA H100 and H200 clusters on gft-studio https://studio.g-ftech.com/ . If you have ever stared at nvidia-smi during a production inference run and felt your heart sink seeing 12% GPU compute utilization while users complained about sluggish generation, you have run headfirst into the central reality of modern LLMs: text generation is a memory bandwidth problem disguised as a compute problem . When a transformer generates text token-by-token, your multi-thousand-dollar GPU spends almost none of its time flexing its tensor cores. Instead, it spends virtually all its time acting like a high-speed forklift in a warehouse—shuffling tens of gigabytes of model weights and historical Key-Value KV cache tensors back and forth across High Bandwidth Memory HBM for every single emitted word. The moment you push beyond single-user toy demos into multi-tenant production or high-throughput reinforcement learning RL rollouts, naive PyTorch stacks hit a wall: memory fragmentation chews up your VRAM, static batching leaves GPUs idling in massive "bubbles," and host-side driver overhead leaves silicon starving. That is why vLLM took over the inference world—not through arcane black magic, but through elegant, battle-tested operating systems engineering: PagedAttention and continuous iteration-level batching . Autoregressive transformers do not generate a paragraph all at once. When a prompt arrives, the model processes all input tokens in parallel during the prefill phase . This is dense, compute-heavy matrix multiplication GEMM —the kind of workload GPUs were born to do. But the moment generation begins the decode phase , everything changes. To generate token 501, the model needs self-attention over the preceding 500 tokens. Recalculating all 500 token representations from scratch on every single step would be an O N2 computational nightmare. So, we cache the intermediate Key and Value vector representations in GPU VRAM: The first factor of 2 accounts for both Keys and Values. L is the number of full-attention layers, HKV is the number of KV heads per layer, dhead is the head dimension, T is the active sequence token length, and b is the precision in bytes e.g., 2 bytes for BF16 . Crucially, modern open-weight architectures like Qwen3.6-27B use Grouped-Query Attention GQA to keep memory sane: 16 full-attention layers with 4 KV heads and a head dimension of 256. At an 8,192-token sequence in BF16, that single sequence commands: Half a gigabyte sounds manageable—until you realize what happens under load. In standard PyTorch or basic Hugging Face generate pipelines, memory management is primitive. Dynamic containers like DynamicCache allocate buffers on the fly. In a multi-user service where one user asks for a 20-line bash script and another submits a 6,000-token legal document, allocating and reallocating memory turns your GPU VRAM into Swiss cheese. This is external memory fragmentation : you might have 15 GB of total free VRAM reported, but because it is shattered into non-contiguous fragments, the next request asking for a contiguous 2 GB block crashes with a catastrophic CUDA Out of Memory OOM . Back in the 1960s, operating system pioneers realized that requiring programs to live in contiguous physical RAM was madness. Their solution was virtual memory paging: chop memory into fixed pages usually 4 KB and let the hardware map arbitrary virtual addresses to scattered physical pages via a page table. vLLM brought this exact insight to GPU memory with PagedAttention . Instead of reserving a giant contiguous chunk of VRAM for each sequence's worst-case length, it chops the KV cache into fixed-size physical blocks typically holding 16 or 32 tokens . Below is a comparison of memory allocation policies when serving 3 concurrent requests holding 3, 5, and 2 tokens respectively with a maximum length of 12 tokens: Figure 1: Contiguous reservation baseline — Each request reserves room for 12 tokens: 36 slots reserved, 10 used, 26 unused 72% memory wasted . Figure 2: Paged block allocation — Four-token blocks reserve 4 + 8 + 4 = 16 slots: 10 used, only 6 unused. Blocks live anywhere in physical memory, freeing 20 slots for other user requests. A centralized Block Table maps logical token positions to physical blocks: Imagine a city bus that refuses to let any new passengers board until every single person on the bus has reached their final destination, even if three people got off at the first stop and one person is riding all the way to the airport. That is exactly how traditional static batching operates. If you batch four requests together that produce 50, 120, 240, and 1,024 tokens respectively, the GPU compute cores sit completely idle on three out of the four slots for hundreds of iterations, waiting for that single 1,024-token straggler to finally emit its