vLLM Architecture, Memory and Benchmarks Deep Dive

VLLM's PagedAttention and continuous iteration-level batching address the KV cache memory bottleneck that limits LLM inference throughput, according to a technical deep dive on the inference engine's architecture. The article details how a Qwen3.6-27B model with 16 full-attention layers, 4 KV heads, and a head dimension of 256 consumes roughly half a gigabyte of KV cache for a single 8,192-token sequence in BF16, and how external memory fragmentation can cause CUDA Out of Memory errors even with 15 GB of free VRAM. PagedAttention applies 1960s-era virtual memory paging to GPU VRAM by splitting the KV cache into fixed-size physical blocks typically holding 16 or 32 tokens.

If you have ever stared at nvidia-smi during a production inference run and felt your heart sink seeing 12% GPU compute utilization while users complained about sluggish generation, you have run headfirst into the central reality of modern LLMs: text generation is a memory bandwidth problem disguised as a compute problem . When a transformer generates text token-by-token, your multi-thousand-dollar GPU spends almost none of its time flexing its tensor cores. Instead, it spends virtually all its time acting like a high-speed forklift in a warehouse—shuffling tens of gigabytes of model weights and historical Key-Value KV cache tensors back and forth across High Bandwidth Memory HBM for every single emitted word. The moment you push beyond single-user toy demos into multi-tenant production or high-throughput reinforcement learning RL rollouts, naive PyTorch stacks hit a wall: memory fragmentation chews up your VRAM, static batching leaves GPUs idling in massive “bubbles,” and host-side driver overhead leaves silicon starving. That is why vLLM took over the inference world—not through arcane black magic, but through elegant, battle-tested operating systems engineering: PagedAttention and continuous iteration-level batching . 1. The KV Cache Bottleneck: Why Memory Bites Back Autoregressive transformers do not generate a paragraph all at once. When a prompt arrives, the model processes all input tokens in parallel during the prefill phase . This is dense, compute-heavy matrix multiplication GEMM —the kind of workload GPUs were born to do. But the moment generation begins the decode phase , everything changes. To generate token 501, the model needs self-attention over the preceding 500 tokens. Recalculating all 500 token representations from scratch on every single step would be an O N ² computational nightmare. So, we cache the intermediate Key and Value vector representations in GPU VRAM: The first factor of 2 accounts for both Keys and Values. L is the number of full-attention layers, H <sub KV</sub is the number of KV heads per layer, d <sub head</sub is the head dimension, T is the active sequence token length, and b is the precision in bytes e.g., 2 bytes for BF16 . Crucially, modern open-weight architectures like Qwen3.6-27B use Grouped-Query Attention GQA to keep memory sane: 16 full-attention layers with 4 KV heads and a head dimension of 256. At an 8,192-token sequence in BF16, that single sequence commands: Half a gigabyte sounds manageable—until you realize what happens under load. In standard PyTorch or basic Hugging Face generate pipelines, memory management is primitive. Dynamic containers like DynamicCache allocate buffers on the fly. In a multi-user service where one user asks for a 20-line bash script and another submits a 6,000-token legal document, allocating and reallocating memory turns your GPU VRAM into Swiss cheese. This is external memory fragmentation : you might have 15 GB of total free VRAM reported, but because it is shattered into non-contiguous fragments, the next request asking for a contiguous 2 GB block crashes with a catastrophic CUDA Out of Memory OOM . 2. PagedAttention: Borrowing a 50-Year-Old OS Masterpiece Back in the 1960s, operating system pioneers realized that requiring programs to live in contiguous physical RAM was madness. Their solution was virtual memory paging: chop memory into fixed pages usually 4 KB and let the hardware map arbitrary virtual addresses to scattered physical pages via a page table. vLLM brought this exact insight to GPU memory with PagedAttention . Instead of reserving a giant contiguous chunk of VRAM for each sequence’s worst-case length, it chops the KV cache into fixed-size physical blocks typically holding 16 or 32 tokens . How much cache do three requests reserve? Try this: Switch the allocation policy. The same requests still contain 3, 5, and 2 tokens. What changed: Each request reserves room for 12 tokens: 36 slots reserved, 10 used, 26 unused. This is a deliberately simple fixed-reservation baseline; other caches need not behave this way. Numbered lime cells = stored tokens. Dark cells = reserved but unused capacity. One small block holds four tokens; the diagram groups each request’s blocks for readability. A centralized Block Table maps logical token positions to physical blocks: 1. On-Demand Allocation: Blocks are handed out only when tokens are actually generated. Only the very last block of a running sequence has any unused slots. 2. Slashing Memory Waste to < 4%: Internal fragmentation virtually disappears. Because you aren’t hoarding empty memory buffers for worst-case prompts, the exact same GPU hardware can suddenly host 2x to 4x more concurrent user streams without breaking a sweat. 3. Copy-on-Write CoW Branching: This is a game-changer for reinforcement learning RL and parallel tree search. In algorithms like GRPO, the model generates 8 or 16 candidate rollouts from the exact same prompt. With PagedAttention, all 16 candidate rollouts physically share the prompt’s KV memory pages. Physical memory is cloned only when individual candidate completions diverge. 3. Continuous Batching: Ending the Tyranny of the Slowest Token Imagine a city bus that refuses to let any new passengers board until every single person on the bus has reached their final destination, even if three people got off at the first stop and one person is riding all the way to the airport. That is exactly how traditional static batching operates. If you batch four requests together that produce 50, 120, 240, and 1,024 tokens respectively, the GPU compute cores sit completely idle on three out of the four slots for hundreds of iterations, waiting for that single 1,024-token straggler to finally emit its <eos token. These wasted cycles are known as GPU bubbles . vLLM implements continuous iteration-level batching an architecture pioneered by Orca : - The scheduler makes decisions at the boundary of every single forward pass , not at the boundary of whole requests. - The millisecond a sequence emits an end-of-sequence token, its physical memory blocks are freed back to the pool. - A waiting request from the queue steps into that vacated slot on the very next token iteration. The GPU cores stay continuously saturated, and throughput jumps dramatically. 4. Squeezing the Hardware: CUDA Graphs, Chunked Prefill, and FP8 PagedAttention solves the memory footprint, but getting raw throughput out of modern NVIDIA Hopper silicon H100/H200 requires tackling kernel dispatch overhead: - The Python Tax and CUDA Graphs enforce eager: false : In standard PyTorch eager mode, generating a single token requires the Python runtime to launch dozens of individual GPU kernels in rapid succession across 60+ transformer layers RMSNorm, QKV projection, RoPE, attention, SwiGLU, down-projection . On Hopper, a single-token GEMV kernel finishes in just 3 to 8 microseconds But the CPU driver call to dispatch that kernel takes 10 to 15 microseconds. The GPU ends up spending more time waiting for Python to hand it work than actually doing the math. CUDA Graphs solve this by recording the entire sequence of operations into a static execution graph during warmup, allowing the GPU to replay the whole pipeline in a single dispatch. - Chunked Prefill: A massive 8,000-token prompt arriving during active generation used to cause a massive latency spike for everyone else. Chunked prefill breaks long prompts into manageable bites e.g. 512 tokens and interleaves them smoothly alongside decode tokens, keeping inter-token latency steady. - FP8 Tensor Core GEMMs: Running weights and activations in 8-bit floating point doubles effective memory bandwidth and unlocks Hopper’s specialized Cutlass FP8 matrix cores. - Hot-Swapping LoRA Adapters: In multi-task serving or RL training, you don’t want to reboot your inference engine every time weights update. vLLM allows syncing LoRA adapter weights directly into the running worker processes over NCCL in milliseconds. 5. Battle Scars from the Lab: Real Benchmarks from gft-studio Synthetic benchmark charts on Twitter are easy to fake. We wanted to see what happens when you push real engineering workloads through this stack. Below is empirical telemetry gathered on our research platform gft-studio running Qwen3.6-27B on dedicated NVIDIA H100 and H200 SXM clusters. Benchmark A: Hugging Face vs. vLLM on H100 Controlled 1x H100 Smoke Test In Group Relative Policy Optimization GRPO , models generate groups of rollouts against external environments. To cleanly isolate the engine, we ran a controlled 3-step test on an identical NVIDIA H100 80GB SXM GPU, keeping the prompt, base weights, random seed, and SQL task strictly identical: | Sampling Engine & Setup | Hardware | Steps Completed | Avg Generation Time / Step | Avg Total Step Time | Observed Speedup | |---|---|---|---|---|---| | Hugging Face Baseline | 1x H100 80GB | 3 / 3 SUCCESS | 283.7 s | 338.9 s | 1.00x Baseline | | vLLM + CUDA Graphs | 1x H100 80GB | 3 / 3 SUCCESS | 78.9 s | 143.5 s | 3.59x Gen Speedup 2.36x Step | On identical silicon, vLLM cut generation wallclock from 283.7 seconds down to 78.9 seconds per step—a 3.59x raw generation speedup . Total step time dropped by 2.36x . A Realistic Caveat: This was a 3-step execution smoke test. While it cleanly isolates engine mechanics on identical hardware, it does not evaluate long-horizon policy convergence over 500 steps. Benchmark B: How Much Do CUDA Graphs Actually Matter on H200? To measure host-side Python dispatch overhead in practice, we tested Qwen3.6-27B on NVIDIA H200 hardware with eager execution enforce eager: true versus captured CUDA Graphs enforce eager: false : | Experimental Arm | Execution Mode | enforce eager Flag | Steps & Status | Avg Generation Time / Step | Observed Ratio | |---|---|---|---|---|---| | L0 Latent Think Deterministic | PyTorch Eager Mode | enforce eager: true | 5 / 5 SUCCESS | 437.9 s / step | 1.00x Baseline | | L0 Latent Think Deterministic | CUDA Graphs Replay | enforce eager: false | 5 / 5 SUCCESS | 101.3 s / step | 4.32x Speedup | | L2b Latent GRPO Policy Gradients | PyTorch Eager Mode | enforce eager: true | 5 / 5 SUCCESS | 391.8 s / step | 1.00x Baseline | | L2b Latent GRPO Policy Gradients | CUDA Graphs Replay | enforce eager: false | 50 / 50 SUCCESS | 132.7 s / step | 2.95x Speedup | The numbers speak for themselves: on single-token decode iterations, replaying pre-recorded CUDA graphs reduced generation time by 2.95x to 4.32x simply by removing host-side driver stalls. Supplemental Telemetry: Downloadable Run Observations For engineers interested in per-step telemetry across training configurations, the interactive table below presents verified arithmetic means from recorded training rows in our gft-studio environment: | Historical gft-studio multi-hop routing runs, Qwen3.6-27B, vLLM generation. Arithmetic means of recorded training rows. | | | | | | |---|---|---|---|---|---| | Run ID | Latent variant | Eager enforced? | Steps | Generation / step | Total / step | |---|---|---|---|---|---| | 1788615027 | L0: answer-only objective | Yes | 5 | 437.9 s | 582.7 s | | 1788635340 | L0: answer-only objective | No | 5 | 101.3 s | 205.6 s | | 1788615660 | L2b: sampled latent objective | Yes | 5 | 391.8 s | 447.8 s | | 1788700311 | L2b: sampled latent objective | No | 50 | 132.7 s | 171.2 s | Source: recorded metrics and resolved configurations from the latent-grpo-recovery worktree at 94de10c, extracted September 8, 2026. Download the selected per-step data and provenance https://www.g-ftech.com/blog-data/vllm-latent-observations.json . Final summary rows are excluded. “Eager not enforced” is a configuration setting, not proof that every operation used a captured graph. Hardware and full environment equivalence are not established by this extract; these rows are observations, not a controlled speedup or quality comparison. Benchmark C: Production Serving and the Hard Truth About Interconnects In an inference serving pilot using standard AIPerf workloads 564 input tokens, 128 output tokens , we pushed Qwen3.8-27B under increasing concurrency: | Hardware & Precision | Client Concurrency | TTFT p50 / p99 | Output Throughput | Scaling Factor | |---|---|---|---|---| | 1x H100 BF16 | 1 | 100.7 ms / 106.8 ms | 48.4 tok/s | 1.0x | | 1x H100 BF16 | 4 | 222.2 ms / 263.5 ms | 179.5 tok/s | 3.71x | | 1x H100 BF16 | 8 | 406.7 ms / 487.1 ms | 307.4 tok/s | 6.35x | | 2x H100 FP8 Cutlass DP2 | 8 | 172.9 ms / 274.5 ms | 483.2 tok/s | 9.98x | | 2x H100 FP8 Together TP2 | 8 | 263.2 ms / 387.4 ms | 982.6 tok/s | 20.30x | A Crucial Engineering War Story: Notice the 982.6 tok/s peak under Tensor Parallelism TP2 . That speed was achieved on a single node connected via ultra-high-speed NVLink. Earlier in our testing, we attempted a custom cross-node TP2 setup over a standard VPC network interconnect. The result? Throughput collapsed to 75.8 tok/s Unless your GPUs share high-bandwidth NVLink, do not run tensor parallelism across physical machines; network latency will decimate your throughput. Use data parallelism independent workers instead. 6. Production Checklist: Running vLLM Without 3 AM Pages If you are deploying vLLM in enterprise infrastructure, here is the architecture pattern we rely on: 1. Two-Tier Ingress Architecture: Place a resilient gateway like LiteLLM in front to handle authentication, team quotas, and audit logging. Route traffic across backend vLLM workers using active queue-depth health checks. 2. Immutable Local Storage: Never make your workers download 50 GB weight files from public object storage on startup. Pre-mount model checkpoints on local NVMe or high-speed read-only PVCs so pods boot in seconds. 3. Watch Your Colocated Memory Budget: If you run RL post-training where the trainer and the vLLM inference worker live on the same GPU, set gpu memory fraction: 0.35 . This reserves ~28 GB for vLLM while leaving ~50 GB for PyTorch gradient activations and optimizer states. Neglecting this balance will trigger immediate CUDA OOM crashes the moment your training loss runs over a long trajectory. Bottom Line: vLLM does not magically make models smarter, but it transforms LLM serving from a brittle, memory-starved script into a predictable, rock-solid engineering system. When you respect the hardware, the hardware delivers.