{"slug": "high-throughput-llm-inference-training-a-deep-dive-into-vllm", "title": "High-Throughput LLM Inference & Training: A Deep Dive into vLLM", "summary": "An engineer at g factor detailed how vLLM's PagedAttention and continuous iteration-level batching solve the memory-bandwidth bottleneck in production LLM inference, drawing on benchmarks run on dedicated NVIDIA H100 and H200 clusters. The writeup explains how naive PyTorch and Hugging Face pipelines suffer external memory fragmentation and CUDA out-of-memory failures under multi-tenant load, and how paging the KV cache into fixed-size blocks mirrors classic OS virtual memory. The author argues text generation is fundamentally a memory bandwidth problem disguised as a compute problem.", "body_md": "**Editor's Note:** Originally published on the [g factor engineering blog](https://www.g-ftech.com/blog/vllm-throughput-deep-dive). All benchmarks and telemetry in this article were conducted on dedicated NVIDIA H100 and H200 clusters on [gft-studio](https://studio.g-ftech.com/).\n\nIf you have ever stared at `nvidia-smi` during a production inference run and felt your heart sink seeing 12% GPU compute utilization while users complained about sluggish generation, you have run headfirst into the central reality of modern LLMs: **text generation is a memory bandwidth problem disguised as a compute problem**.\n\nWhen a transformer generates text token-by-token, your multi-thousand-dollar GPU spends almost none of its time flexing its tensor cores. Instead, it spends virtually all its time acting like a high-speed forklift in a warehouse—shuffling tens of gigabytes of model weights and historical Key-Value (KV) cache tensors back and forth across High Bandwidth Memory (HBM) for every single emitted word.\n\nThe moment you push beyond single-user toy demos into multi-tenant production or high-throughput reinforcement learning (RL) rollouts, naive PyTorch stacks hit a wall: memory fragmentation chews up your VRAM, static batching leaves GPUs idling in massive \"bubbles,\" and host-side driver overhead leaves silicon starving. That is why vLLM took over the inference world—not through arcane black magic, but through elegant, battle-tested operating systems engineering: **PagedAttention** and **continuous iteration-level batching**.\n\nAutoregressive transformers do not generate a paragraph all at once. When a prompt arrives, the model processes all input tokens in parallel during the *prefill phase*. This is dense, compute-heavy matrix multiplication (GEMM)—the kind of workload GPUs were born to do.\n\nBut the moment generation begins (the *decode phase*), everything changes. To generate token #501, the model needs self-attention over the preceding 500 tokens. Recalculating all 500 token representations from scratch on every single step would be an \nO(N2)\n computational nightmare. So, we cache the intermediate Key and Value vector representations in GPU VRAM:\n\nThe first factor of 2 accounts for both Keys and Values. L is the number of full-attention layers, HKV is the number of KV heads per layer, dhead is the head dimension, T is the active sequence token length, and b is the precision in bytes (e.g., 2 bytes for BF16).\n\nCrucially, modern open-weight architectures (like `Qwen3.6-27B`) use Grouped-Query Attention (GQA) to keep memory sane: 16 full-attention layers with 4 KV heads and a head dimension of 256. At an 8,192-token sequence in BF16, that single sequence commands:\n\nHalf a gigabyte sounds manageable—until you realize what happens under load. In standard PyTorch or basic Hugging Face `generate()` pipelines, memory management is primitive. Dynamic containers like `DynamicCache` allocate buffers on the fly. In a multi-user service where one user asks for a 20-line bash script and another submits a 6,000-token legal document, allocating and reallocating memory turns your GPU VRAM into Swiss cheese.\n\nThis is **external memory fragmentation**: you might have 15 GB of total free VRAM reported, but because it is shattered into non-contiguous fragments, the next request asking for a contiguous 2 GB block crashes with a catastrophic `CUDA Out of Memory` (OOM).\n\nBack in the 1960s, operating system pioneers realized that requiring programs to live in contiguous physical RAM was madness. Their solution was virtual memory paging: chop memory into fixed pages (usually 4 KB) and let the hardware map arbitrary virtual addresses to scattered physical pages via a page table.\n\nvLLM brought this exact insight to GPU memory with **PagedAttention**. Instead of reserving a giant contiguous chunk of VRAM for each sequence's worst-case length, it chops the KV cache into fixed-size physical blocks (typically holding 16 or 32 tokens).\n\nBelow is a comparison of memory allocation policies when serving 3 concurrent requests (holding 3, 5, and 2 tokens respectively) with a maximum length of 12 tokens:\n\n*Figure 1: Contiguous reservation baseline — Each request reserves room for 12 tokens: 36 slots reserved, 10 used, 26 unused (72% memory wasted).*\n\n*Figure 2: Paged block allocation — Four-token blocks reserve 4 + 8 + 4 = 16 slots: 10 used, only 6 unused. Blocks live anywhere in physical memory, freeing 20 slots for other user requests.*\n\nA centralized **Block Table** maps logical token positions to physical blocks:\n\nImagine a city bus that refuses to let any new passengers board until every single person on the bus has reached their final destination, even if three people got off at the first stop and one person is riding all the way to the airport.\n\nThat is exactly how traditional **static batching** operates. If you batch four requests together that produce 50, 120, 240, and 1,024 tokens respectively, the GPU compute cores sit completely idle on three out of the four slots for hundreds of iterations, waiting for that single 1,024-token straggler to finally emit its `<eos>` token. These wasted cycles are known as **GPU bubbles**.\n\nvLLM implements **continuous iteration-level batching** (an architecture pioneered by Orca):\n\nPagedAttention solves the memory footprint, but getting raw throughput out of modern NVIDIA Hopper silicon (H100/H200) requires tackling kernel dispatch overhead:\n\n`enforce_eager: false`):\nSynthetic benchmark charts on Twitter are easy to fake. We wanted to see what happens when you push real engineering workloads through this stack. Below is empirical telemetry gathered on our research platform (`gft-studio`) running `Qwen3.6-27B` on dedicated NVIDIA H100 and H200 SXM clusters.\n\nIn Group Relative Policy Optimization (GRPO), models generate groups of rollouts against external environments. To cleanly isolate the engine, we ran a controlled 3-step test on an identical NVIDIA H100 80GB SXM GPU, keeping the prompt, base weights, random seed, and SQL task strictly identical:\n\n| Sampling Engine & Setup | Hardware | Steps Completed | Avg Gen Time / Step | Avg Total Step Time | Observed Speedup | \n|---|---|---|---|---|---|\n| **Hugging Face Baseline** | 1x H100 80GB | 3 / 3 (SUCCESS) | 283.7 s | 338.9 s | 1.00x (Baseline) | \n| **vLLM + CUDA Graphs** | 1x H100 80GB | 3 / 3 (SUCCESS) | **78.9 s** | **143.5 s** | **3.59x Gen Speedup (2.36x Step)** | \n\nOn identical silicon, vLLM cut generation wallclock from 283.7 seconds down to 78.9 seconds per step—a **3.59x raw generation speedup**. Total step time dropped by **2.36x**.\n\n**Caveat:** This was a 3-step execution smoke test. While it cleanly isolates engine mechanics on identical hardware, it does not evaluate long-horizon policy convergence over 500 steps.\n\nTo measure host-side Python dispatch overhead in practice, we tested `Qwen3.6-27B` on NVIDIA H200 hardware with eager execution (`enforce_eager: true`) versus captured CUDA Graphs (` enforce_eager: false`):\n\n| Experimental Arm | Execution Mode | enforce_eager Flag | Steps & Status | Avg Gen Time / Step | Observed Ratio | \n|---|---|---|---|---|---|\n| **L0 Latent Think (Deterministic)** | PyTorch Eager Mode | `enforce_eager: true` | 5 / 5 (SUCCESS) | 437.9 s / step | 1.00x (Baseline) | \n| **L0 Latent Think (Deterministic)** | **CUDA Graphs Replay** | `enforce_eager: false` | 5 / 5 (SUCCESS) | **101.3 s / step** | **4.32x Speedup** | \n| **L2b Latent GRPO (Policy Gradients)** | PyTorch Eager Mode | `enforce_eager: true` | 5 / 5 (SUCCESS) | 391.8 s / step | 1.00x (Baseline) | \n| **L2b Latent GRPO (Policy Gradients)** | **CUDA Graphs Replay** | `enforce_eager: false` | 50 / 50 (SUCCESS) | **132.7 s / step** | **2.95x Speedup** | \n\nThe numbers speak for themselves: on single-token decode iterations, replaying pre-recorded CUDA graphs reduced generation time by **2.95x to 4.32x** simply by removing host-side driver stalls.\n\nHistorical multi-hop routing runs (`Qwen3.6-27B`, vLLM generation):\n\n| Run ID | Latent Variant | Eager Enforced? | Steps | Generation / Step | Total / Step | \n|---|---|---|---|---|---|\n| `1788615027` | L0: answer-only objective | Yes | 5 | 437.9 s | 582.7 s | \n| `1788616147` | L0: answer-only objective | No | 5 | 101.3 s | 247.9 s | \n| `1788626786` | L2b: sampled latent objective | Yes | 5 | 391.8 s | 567.9 s | \n| `1788656686` | L2b: sampled latent objective | No | 50 | 132.7 s | 291.6 s | \n\nIn an inference serving pilot using standard AIPerf workloads (564 input tokens, 128 output tokens), we pushed `Qwen3.8-27B` under increasing concurrency:\n\n| Hardware & Precision | Client Concurrency | TTFT (p50 / p99) | Output Throughput | Scaling Factor | \n|---|---|---|---|---|\n| 1x H100 BF16 | 1 | 100.7 ms / 106.8 ms | 48.4 tok/s | 1.0x | \n| 1x H100 BF16 | 4 | 222.2 ms / 263.5 ms | 179.5 tok/s | 3.71x | \n| 1x H100 BF16 | 8 | 406.7 ms / 487.1 ms | 307.4 tok/s | 6.35x | \n| 2x H100 FP8 (Cutlass DP2) | 8 | 172.9 ms / 274.5 ms | 483.2 tok/s | 9.98x | \n| **2x H100 FP8 (Together TP2)** | 8 | 263.2 ms / 387.4 ms | **982.6 tok/s** | **20.30x** | \n\n**An Engineering War Story:** Notice the 982.6 tok/s peak under Tensor Parallelism (TP2). That speed was achieved on a single node connected via ultra-high-speed NVLink. Earlier in our testing, we attempted a custom cross-node TP2 setup over a standard VPC network interconnect. The result? **Throughput collapsed to 75.8 tok/s**! Unless your GPUs share high-bandwidth NVLink, do not run tensor parallelism across physical machines; network latency will decimate your throughput. Use data parallelism (independent workers) instead.\n\nIf you are deploying vLLM in enterprise infrastructure, here is the architecture pattern we rely on:\n\n`gpu_memory_fraction: 0.35`. This reserves ~28 GB for vLLM while leaving ~50 GB for PyTorch gradient activations and optimizer states. Neglecting this balance will trigger immediate CUDA OOM crashes the moment your training loss runs over a long trajectory.\nvLLM does not magically make models smarter, but it transforms LLM serving from a brittle, memory-starved script into a predictable, rock-solid engineering system. When you respect the hardware, the hardware delivers.\n\n*Interested in specialized RL environments and post-training infrastructure? Explore our platform at [g-ftech.com](https://www.g-ftech.com/) or launch a cluster on [gft-studio](https://studio.g-ftech.com/).*", "url": "https://wpnews.pro/news/high-throughput-llm-inference-training-a-deep-dive-into-vllm", "canonical_source": "https://dev.to/g_factor/high-throughput-llm-inference-training-a-deep-dive-into-vllm-5818", "published_at": "2026-09-21 18:33:06+00:00", "updated_at": "2026-09-21 19:01:52.994580+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "mlops", "ai-research"], "entities": ["vLLM", "NVIDIA", "H100", "H200", "PyTorch", "Hugging Face", "PagedAttention", "g factor"], "alternates": {"html": "https://wpnews.pro/news/high-throughput-llm-inference-training-a-deep-dive-into-vllm", "markdown": "https://wpnews.pro/news/high-throughput-llm-inference-training-a-deep-dive-into-vllm.md", "text": "https://wpnews.pro/news/high-throughput-llm-inference-training-a-deep-dive-into-vllm.txt", "jsonld": "https://wpnews.pro/news/high-throughput-llm-inference-training-a-deep-dive-into-vllm.jsonld"}}