{"slug": "vllm-0-34-reclaims-gpu-memory-and-boosts-cache-capacity-by-9-5", "title": "vLLM 0.34 Reclaims GPU Memory and Boosts Cache Capacity by 9.5%", "summary": "VLLM 0.34.0 now sizes its KV cache from memory measured after graph capture and warmup instead of pre-warmup estimates, giving most models 4–10% more cache capacity, with Qwen3-32B on GB200 rising from 103.1 GiB to 113.0 GiB (9.5% more cache tokens). The release, which ships 1,061 commits from 412 contributors and adds 10 models including Mistral Large 4, changes the gpu_memory_utilization default to 1.0, removes Model Runner V1, and speeds Step5 NVFP4 decode by up to 1.95× on 4×GB200. Operators must review old memory flags, since an explicit --gpu-memory-utilization 0.92 now yields a smaller cache than before, and the release also sets the default block size to 64, drops ROCm 7.2, and requires huggingface_hub 2.0+.", "body_md": "# vLLM 0.34 Reclaims GPU Memory and Boosts Cache Capacity by 9.5%\n\nvLLM v0.34.0 now sizes the KV cache from memory it measures after warmup, removes the old V1 runner, adds 10 models and speeds up Blackwell decode by as much as 1.95×.\n\n- [vLLM v0.34.0](https://github.com/vllm-project/vllm/releases/tag/v0.34.0?utm_source=alphasignal) ships with 1,061 commits from 412 contributors, plus 10 new models including Mistral Large 4.\n- The extensible KV cache is now on by default and sizes memory after warmup, giving most models 4–10% more cache.\n- The `gpu_memory_utilization` default is now 1.0, so an explicit 0.92 now gives you a smaller cache than before.\n- Model Runner V1 is removed, and setting `VLLM_USE_V2_MODEL_RUNNER=0` now fails at startup.\n- Step5 NVFP4 decode is up to 1.95× faster on 4×GB200, and GLM-5.3-Flash throughput is 1.6× higher.\n- Breaking changes: default block size is 64, ROCm 7.2 is dropped, and huggingface_hub 2.0+ is required. Install via [PyPI](https://pypi.org/project/vllm/?utm_source=alphasignal) .\n\nvLLM 0.34.0 replaces conservative KV cache estimates with post-warmup memory measurements, allowing dedicated inference servers to use GPU capacity previously held in reserve. The release contains 1,061 commits from 412 contributors, enables the extensible KV cache by default, removes Model Runner V1, and changes the default cache block size. Operators should review old memory flags because an explicit limit such as `--gpu-memory-utilization 0.92` can erase part of the capacity gain.\n\n## The allocator waits for real numbers\n\nThe KV cache stores attention keys and values for tokens being processed. Its capacity determines how many tokens and concurrent requests the engine can hold without recomputing or preempting work. Earlier vLLM releases created this cache before CUDA graph capture, so they had to estimate the memory that graphs, workspaces, and samplers would later consume. The default `gpu_memory_utilization` of `0.92` left 8% of GPU memory outside the engine’s budget to absorb estimation errors.\n\nVersion 0.34.0 measures available memory after graph capture and warmup. CUDA graphs record GPU operations for efficient replay, which requires memory addresses to remain stable. The new allocator preserves those addresses through virtual memory:\n\n1. vLLM reserves the KV cache’s complete virtual address range without immediately backing it with physical memory. NVIDIA GPUs use the CUDA driver’s `cuMem*` APIs; AMD GPUs use`hipMem*` .\n2. Graph capture and model warmup run, with physical memory committed only for pages they touch.\n3. Each worker records its peak temporary allocation and measures the memory still available.\n4. The engine keeps a safety margin of `max(1 GiB, 0.75 × transient peak)` , then maps the remaining physical memory into the reserved cache range.\n\nStable virtual addresses keep captured graphs valid while the post-warmup measurement gives each model a cache sized around its observed memory use. The implementation is documented in the [KV cache PR](https://github.com/vllm-project/vllm/pull/56492?utm_source=alphasignal).\n\n## Capacity gains depend on the workload\n\nBenchmarks from the pull request covered eight models on NVIDIA GB200 systems. Every tested configuration gained KV cache capacity, although the increase varied with graph and workspace requirements. TP4 denotes tensor parallelism across four GPUs.\n\n| Model | Configuration | Reported gain | \n|---|---|---|\n| Qwen3-32B | GB200 | 103.1 GiB to 113.0 GiB, or 9.5% more cache tokens | \n| Qwen3-8B | GB200 | 7.5% more cache tokens | \n| Qwen3-235B-A22B-FP8 | GB200, TP4 | 8.2% more cache tokens | \n| DeepSeek-V4-Flash | GB200, TP4 | 4.1% more cache tokens | \n\nDeepSeek-V4-Flash gained less because its graphs and workspaces already consumed most of the old reserve. The [vLLM 0.34 release](https://github.com/vllm-project/vllm/releases/tag/v0.34.0?utm_source=alphasignal) gives a larger example: MiniMax-M3 on H200 increased from 433,000 to 672,000 cache tokens.\n\nEngine initialization excluding weight loading improved by 21% to 34% because vLLM no longer runs a disposable graph-capture pass for memory estimation. Qwen3-235B initialization fell from 76 seconds to 50 seconds, reducing the time required to add replicas during autoscaling.\n\n## Old flags can erase the gain\n\nThe larger cache can increase decode latency per token when the scheduler fills the added capacity with more sequences. Release benchmarks report 6% to 18% higher per-token decode latency on some H200 models at concurrency 128. GPU memory dashboards will also remain near full because the allocator deliberately assigns available memory to the cache.\n\nAn explicit `--gpu-memory-utilization 0.92` produced caches 0.6% to 3.7% smaller than before in the project’s tests. The new allocator subtracts measured graph memory from a budget already capped at 92%, and vLLM now prints a startup warning when an explicit value withholds memory.\n\n| Operational goal | Setting | Effect | \n|---|---|---|\n| Use available memory on a dedicated GPU | Omit the memory flag | Lets the extensible cache size itself after warmup | \n| Reserve memory for another process | `--gpu-memory-utilization 0.9` | Keeps the extensible allocator while limiting its budget | \n| Restore the previous allocation path | `--no-enable-extensible-kv-cache` | Disables the extensible cache | \n| Limit decode contention | `--max-num-seqs N` | Caps concurrent sequences without shrinking the cache | \n\nExplicit memory limits remain appropriate when several engines share one GPU, another process allocates GPU memory after startup, or KV connectors and reinforcement-learning weight-transfer jobs maintain separate buffers. Unified-memory systems such as DGX Spark also need room for the host operating system and CPU workloads because they share the same physical memory pool.\n\n```\n# Dedicated GPU: remove old memory limits\nvllm serve \"$MODEL\"\n\n# Preserve the previous cache block size\nvllm serve \"$MODEL\" --block-size 16\n```\n\nVersion 0.34.0 raises the default `--block-size` from 16 tokens to 64. Prefix caching reuses only complete blocks, so a shared 40-token system prompt yields no reusable block at size 64. A 16-token block size can reuse 32 tokens from the same prompt. The pull request does not include a 16-versus-64 comparison on a production serving workload, so deployments with short shared prefixes should benchmark both settings.\n\n## V2 becomes the only GPU runner\n\nModel Runner V2 now handles all GPU execution. Setting `VLLM_USE_V2_MODEL_RUNNER=0` causes startup to fail, and custom logits processors written for V1 must migrate to the V2 sampler module. Dual-batch overlap, which lets two micro-batches share GPU execution, now works on V2 alongside speculative decoding.\n\nSuffix decoding also runs directly on V2 without the separate Arctic Inference package. This speculative technique drafts tokens by matching the current sequence against earlier output, then verifies those candidates with the model.\n\n## Faster kernels favor newer accelerators\n\nReported kernel improvements concentrate on NVIDIA Blackwell GPUs and AMD MI355X accelerators. Results vary by model shape, batch size, context length, and parallelism configuration.\n\n| Workload | Hardware or path | Reported result | \n|---|---|---|\n| Step5 NVFP4 MoE | 4× GB200, TRTLLM-Gen kernels | Up to 1.95× faster decoding | \n| GLM-5.3-Flash | DeepGEMM FP8 MegaMoE | 1.6× higher output throughput | \n| GLM-5.3 long prompts | GB200, 32K to 128K context | 1.24× to 1.40× faster time to first token | \n| Kimi-K3 | GB300, opt-in KV-balanced MLA decoding | 1.34× to 1.59× faster decode attention | \n| Qwen3.5 | Breakable CUDA graphs | 3% to 8.8% lower decode latency | \n\nModel support expands to Mistral Large 4, MiniMax-M3.1-Flash, Nemotron-3.5-Transcribe, Liquid AI’s LFM3 family, Zyphra ZAYA1, and several speech models. Mistral Large 4 is a mixture-of-experts model with 1 trillion total parameters and 52 billion active parameters per token. Preliminary Kimi K3.1 code landed in the release, but K3.1 cannot yet be served.\n\n## Upgrade checklist and removals\n\nVersion 0.34.0 is available under the Apache 2.0 license from [PyPI](https://pypi.org/project/vllm/?utm_source=alphasignal). The main Docker image uses CUDA 13.0, with separate ROCm 10.0, CPU, and XPU variants.\n\n```\npython -m pip install \"vllm==0.34.0\"\ndocker pull vllm/vllm-openai:v0.34.0\n```\n\nSeveral removals and dependency changes can block an in-place upgrade:\n\n- ROCm 7.2 images have been removed; deployments must move to ROCm 10.0 artifacts.\n- `huggingface_hub >= 2.0.0` ,`tokenizers >= 0.23.3` , and`timm >= 1.0.30` are required.\n- Marlin W4A8 support has been removed. Use `--quantization humming` with`VLLM_HUMMING_INPUT_QUANT_CONFIG` instead of`VLLM_MARLIN_INPUT_DTYPE` .\n- Support for QuTLASS `fp_quant` checkpoints and Keye-VL models has been removed.\n\nCommunity Windows builds report that the extensible KV cache works on Windows. A GPU that also drives the desktop may expose less free memory during warmup, producing a smaller cache than earlier releases. Dedicated inference GPUs avoid that source of variability.\n\nOn a dedicated GPU, remove legacy memory limits, deploy a canary with representative prompts, and compare throughput, time to first token, inter-token latency, `vllm:kv_cache_usage_perc`, and preemption counts. High raw GPU memory use is expected under the new allocator. Deployments with strict latency targets can use `--max-num-seqs` to control concurrency while retaining the larger cache.", "url": "https://wpnews.pro/news/vllm-0-34-reclaims-gpu-memory-and-boosts-cache-capacity-by-9-5", "canonical_source": "https://alphasignal.ai/news/vllm-0-34-reclaims-gpu-memory-and-boosts-cache-capacity-by-9-5", "published_at": "2026-11-18 17:15:22+00:00", "updated_at": "2026-11-19 16:42:47.351688+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-tools"], "entities": ["vLLM", "Mistral Large 4", "Qwen3-32B", "DeepSeek-V4-Flash", "MiniMax-M3", "NVIDIA GB200", "H200", "ROCm"], "also_reported_by": [], "takeaway": "vLLM 0.34.0 sizes the KV cache after warmup with a safety margin of max(1 GiB, 0.75 × transient peak), so removing an explicit --gpu-memory-utilization 0.92 flag recovers 4–10% more cache tokens.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/vllm-0-34-reclaims-gpu-memory-and-boosts-cache-capacity-by-9-5", "markdown": "https://wpnews.pro/news/vllm-0-34-reclaims-gpu-memory-and-boosts-cache-capacity-by-9-5.md", "text": "https://wpnews.pro/news/vllm-0-34-reclaims-gpu-memory-and-boosts-cache-capacity-by-9-5.txt", "jsonld": "https://wpnews.pro/news/vllm-0-34-reclaims-gpu-memory-and-boosts-cache-capacity-by-9-5.jsonld"}}