{"slug": "when-the-kv-cache-spills-over-pcie", "title": "When the KV Cache Spills Over PCIe", "summary": "A single 8-GPU DGX H100-class server running Llama 3.1 70B on vLLM can hold only about 37 of 200 long-context sessions in HBM, because each 32K-token session needs roughly 10.7 GB of KV cache at about 320 KB per token in BF16, according to an analysis by @Srinivas Kodali. The remaining 163 sessions must either recompute prefill, which takes roughly 1.5 to 2.5 seconds on an idle server for 32K tokens, or reload a parked cache over PCIe at about 50 GB/s measured one-way on an H100 versus 64 GB/s on paper, roughly 67 times narrower than HBM. Reloading from CPU memory is more than 50 times faster than recomputing, and even a Gen4 SSD path measured at 6.2 GB/s by Red Hat is at least 7 times faster, while FP8 halves the cache to 1.07 TB and removes the need for SSDs.", "body_md": "Nov 14, 2026 · @Srinivas Kodali\n\nOne model, one 8-GPU server, 200 long-context users. Where the KV cache goes when GPU memory runs out, what PCIe does to response time, and how to keep it under control.\n\nIn my last article, [PCIe in AI Inference: Where It Shines and Where It Lags](https://pub.towardsai.net/pcie-in-ai-inference-where-it-shines-and-where-it-lags-ad862bd9e1ec), PCIe sat mostly idle while the GPU wrote tokens. It mattered only when data had to move in bulk.\n\nIn long-context serving, the biggest bulk mover is the KV cache. Every chat, document and agent step leaves one behind. When GPU memory fills up, the cache spills into CPU memory and SSDs, and PCIe becomes the only road back.\n\nThis article follows one production setup end to end: how big the cache gets, where it lives, how long it takes to bring back, and what breaks under real load.\n\nOne example runs through the whole article: Llama 3.1 70B served by vLLM on a DGX H100-class server. The hardware numbers come from public spec sheets, and the software is open source.\n\nThe workload is a typical coding or support assistant. Users paste documents, keep long chat histories, run agent steps, and pause to think between turns.\n\nFor every token in the context, each layer keeps one key and one value vector per KV head. For Llama 3.1 70B, that comes to:\n\n**KV per token = 2 (key + value) × 80 layers × 8 KV heads × 128 dims × 2 bytes = 327,680 bytes**\n\nThat is about 320 KB per token in BF16, or 160 KB in FP8. Multiply it by the context length to get each user’s cache.\n\nWhy 400 GB? Start with 640 GB of HBM. Keep 10% back as a safety margin, then subtract 141 GB of weights and some working memory for activations.\n\nOur 200 sessions average 32K tokens each, so they need 200 × 10.7 GB = 2.15 TB of KV cache. HBM holds about 37 of them.\n\nThat leaves 163 sessions with two choices:\n\nRecompute costs GPU time. Parking costs PCIe time. The rest of this article is about that trade.\n\nFP8 changes the picture. It halves the total to 1.07 TB, which fits in HBM plus CPU memory with no SSD needed.\n\nA parked KV cache moves down a ladder. Each step away from the GPU adds capacity and loses bandwidth.\n\nTwo numbers matter most. The road from CPU memory back to the GPU runs at about 50 GB/s in practice. The [vLLM team measured](https://vllm.ai/blog/2026-01-08-kv-offloading-connector) that one-way peak on an H100, against 64 GB/s on paper. That is about 67 times narrower than HBM.\n\nSSDs are slower again. A Gen4 data center drive reads about 6 to 7 GB/s, and [Red Hat measured](https://developers.redhat.com/articles/2026/11/02/benchmarking-tiered-kv-cache-offloading-h100-gpus-openshift-ai-3-5) 6.2 GB/s during its H100 inference tests.\n\nA user comes back after five minutes, and their 32K-token KV cache was parked. What does the next answer cost?\n\nWith tensor parallelism across 8 GPUs, each GPU holds one-eighth of the cache. So each GPU pulls 1.34 GB over its own PCIe link, and all eight pull at the same time.\n\nReloading from CPU memory is more than 50 times faster than recomputing. Even the SSD path is at least 7 times faster.\n\nStart with the work. Prefill costs about 2 FLOP per parameter per token, plus attention, which grows with the square of the context:\n\n**Prefill work ≈ 2 × parameters × tokens + 2 × layers × hidden size × tokens²**\n\nFor 32K tokens on Llama 3.1 70B, that is about 4.6 + 1.4 = 6.0 PFLOP.\n\nEight H100s peak at 7.9 PFLOP/s in dense BF16, or 989 TFLOP/s each. (The 1,979 TFLOP/s on the datasheet assumes sparsity, which dense models don’t use.) At that peak, recompute would take 0.76 s. Real prefill never runs at peak, because each of these steps needs compute or time beyond the raw math:\n\nTogether, these put recompute at roughly 1.5 to 2.5 s on an idle server. FP8 roughly halves it. To get the real number for your own setup, measure a single 32K-token prefill:\n\n```\nvllm bench latency --model meta-llama/Llama-3.1-70B-Instruct -tp 8 --input-len 32768 --output-len 1 --batch-size 1 --no-enable-prefix-caching\n```\n\nUnder load, recompute gets much slower. It waits behind other users’ prefills and shares the GPUs with their decode steps. [LMCache measured](https://blog.lmcache.ai/en/2026/04/22/lmcache-on-amazon-sagemaker-hyperpod-accelerating-llm-inference-with-managed-tiered-kv-cache/) Llama 3.1 70B on 8× H100 with 100 chat sessions of only 12K tokens each and a small CPU cache. Time to first token was 4.5 s at P50 and 8.3 s at P90.\n\nRecompute also has a hidden cost: every second of it is GPU time taken from users who are already generating tokens.\n\nThe 27 ms above is the best case: one user, an idle bus and well-tuned code. Production adds traffic, topology and wear.\n\nUsers don’t come back one at a time. A meeting ends, a batch of agents wakes up, or the morning rush starts. If 20 users return together, each GPU must pull 20 × 1.34 GB = 27 GB before the last of them sees a token.\n\nAt the full 50 GB/s, about 18 users fit inside a 500 ms budget. At 15 GB/s, the sustained rate [one study](https://arxiv.org/html/2601.19910v1) measured on an H100 server, only 5 do. FP8 doubles each of these numbers.\n\nThe GPU’s PCIe link doesn’t belong to the KV cache alone. In many 8-GPU servers, each GPU sits behind a PCIe switch with its network card, and some designs put two GPUs on one switch. Network traffic, storage reads and KV reloads then compete for the same lanes.\n\nOffload writes new KV to CPU memory while reloads pull old KV back. PCIe is full duplex on paper, but host-side limits can make the two directions slow each other down. Test both directions at once with nvbandwidth before you trust one-way numbers.\n\nvLLM stores KV in 16-token blocks. In its older layout, one Llama 3.1 70B block was only 8 KB per layer, and copy engines are poor at small copies. The [newer layout](https://vllm.ai/blog/2026-01-08-kv-offloading-connector) packs all layers into one 1.25 MB block, which raised offload throughput about tenfold.\n\nWith tensor parallelism, a reload finishes only when all eight GPUs finish. One link running at half width or an older PCIe generation slows down every user. Check link speed and width on all eight GPUs, not just one.\n\nIf a GPU’s pinned host buffer sits on the far CPU socket, every copy crosses the link between the sockets. nvidia-smi topo -m labels that path SYS, the slowest kind of GPU path.\n\nKV offload writes all day. Read-intensive drives such as the [Micron 7450 PRO](https://www.storagereview.com/news/micron-7450-ssd-announced) are rated for one full drive write per day, so eight 3.84 TB drives can absorb about 31 TB a day. A steady 1 GB/s of KV writes is 86 TB a day, which wears the drives out in under 2 years instead of 5.\n\nOn DGX H100, the data drives also ship as [RAID 0](https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html). One failed drive empties the whole tier, and a restart means a cold cache.\n\nWith several replicas, a returning user can land on a replica that never saw their KV cache, and the system recomputes anyway. With 4 replicas and plain round-robin routing, a returning user lands on the right server only 1 time in 4, so about 75% of returning turns recompute for nothing. Sticky or KV-aware routing fixes this by sending each turn back to the replica that holds its cache.\n\nThe KV cache is a working copy of the user’s conversation. On disk, it needs the same encryption and deletion rules as chat logs. DGX H100’s data drives are self-encrypting, which helps, but retention rules are still your job.\n\nOffload pays off when the same context comes back. When it doesn’t come back, offload just burns bandwidth.\n\n*Table 4. Where the KV cache should live, by workload.*\n\nReuse is the deciding number. The longer the context, the more a recompute costs, so long contexts pay back the offload even when only some sessions return.\n\nFP8 halves nearly every number in this article. In the next article, I look at quantization: shrinking weights and the KV cache so one GPU node serves more tokens per second.\n\n[When the KV Cache Spills Over PCIe](https://pub.towardsai.net/when-the-kv-cache-spills-over-pcie-996781bd7658) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/when-the-kv-cache-spills-over-pcie", "canonical_source": "https://pub.towardsai.net/when-the-kv-cache-spills-over-pcie-996781bd7658?source=rss----98111c9905da---4", "published_at": "2026-11-17 21:01:01+00:00", "updated_at": "2026-11-17 21:15:16.590568+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-chips"], "entities": ["Llama 3.1 70B", "vLLM", "DGX H100", "H100", "Red Hat", "Srinivas Kodali"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/when-the-kv-cache-spills-over-pcie", "markdown": "https://wpnews.pro/news/when-the-kv-cache-spills-over-pcie.md", "text": "https://wpnews.pro/news/when-the-kv-cache-spills-over-pcie.txt", "jsonld": "https://wpnews.pro/news/when-the-kv-cache-spills-over-pcie.jsonld"}}