Your 128K Context Fits on a 4090 — If You Quantize the KV Cache

A developer has published sizing math showing that vLLM's fp8 KV-cache flag halves KV-cache memory, letting a Llama 3.1 8B model with a 64K context fit on a 24 GB RTX 4090 at roughly $0.35/hr instead of requiring an 80 GB card at $1.50+/hr. The writeup, based on public model cards and vLLM documentation rather than original benchmarks, notes that at 128K the FP8 cache alone still exceeds 24 GB for that model and must be paired with FP8 weights or a GQA-heavy architecture like Qwen2.5 7B. The developer recommends E4M3 over E5M2, dataset calibration via llm-compressor for production, and retrieval-focused evals before and after enabling the flag.

Researched October 2026. Sizing math from public model cards and the vLLM docs; not my own benchmarks. In my last post https://dev.to/qisuancloud/your-7b-model-fits-on-a-4090-until-you-open-a-128k-context-window I showed that a 128K context window can push a 7B model from 17 GB to 33 GB of VRAM — and that you have three moves, in order of cost-effectiveness: quantize the cache, pick a GQA-heavy architecture, or rent the bigger card. This post is about move 1, the cheapest one. It halves the KV cache with a single flag. In vLLM: vllm serve meta-llama/Llama-3.1-8B-Instruct \ --kv-cache-dtype fp8 That's it. The weights stay in FP16/BF16 — untouched. Only the KV cache is stored as 1 byte per value instead of 2. Per the vLLM docs https://docs.vllm.ai/en/latest/features/quantization/quantized kvcache/ , fp8 maps to E4M3, and there are three calibration options: no calibration scales default to 1.0 , random-token warmup, or dataset calibration via llm-compressor recommended for max accuracy . Weight quantization gets all the attention, but KV values are arguably the easier target: within a single attention head, their values live in a narrow, well-behaved range. E4M3's tighter range and 3-bit mantissa match that distribution well; E5M2's wider range is mostly wasted on values that never appear, and its 2-bit mantissa visibly degrades attention scores that compound across layers. In other words, the format choice matters — fp8 E4M3 is the right default, not fp8 e5m2 . Llama 3.1 8B, FP16 weights ~16 GB , 2 GB overhead: | Context | FP16 cache → total | FP8 cache → total | |---|---|---| | 32K | 4 GB → ~22 GB ✅ | 2 GB → ~20 GB ✅ | | 64K | 8 GB → ~26 GB ❌ | 4 GB → ~22 GB ✅ fits 4090 | | 128K | 16 GB → ~34 GB ❌ | 8 GB → ~26 GB ❌ still over | Honest note: at 128K, FP8 cache alone doesn't save Llama 3.1 8B on a 24 GB card — 26 GB is still 26 GB. Your options there: pair it with FP8 weights 8 + 8 + 2 ≈ 18 GB ✅ , or pick the GQA-heavy architecture from last post — Qwen2.5 7B at 128K needs ~7 GB of FP16 cache, ~3.5 GB in FP8, for a total around 19 GB ✅. The flag is powerful, not magic; the arithmetic still rules. 1. Hardware matters. The biggest wins are on Hopper GPUs with native FP8 support. On older cards, check your vLLM version's kernel support — the memory savings are real, but the compute path may not be as optimized. 2. Calibrate if quality matters. Default scales of 1.0 are fine to start, but for production, dataset calibration llm-compressor exists for a reason. Per-head quantization — finer-grained than per-tensor — needs the FlashAttention backend plus that calibration path. 3. Measure retrieval, not just perplexity. Standard benchmarks show small degradation, but long-context tasks with exact-recall demands needle-in-a-haystack style document QA are where you'd notice it first. Run your own eval before and after. The whole point: quantization keeps you on the cheap card. A 64K-context 7B workload that needed an 80 GB card at ~$1.50+/hr in FP16 fits on a 24 GB 4090 at ~$0.35/hr with FP8 cache — same model, same context, roughly a quarter of the hourly cost, for one CLI flag. I built a free GPU Advisor https://qisuanai.com/advisor that walks through these tradeoffs model size, context, quantization tolerance and recommends the cheapest fitting setup. No signup. Have you tried FP8 KV cache in production — and did the quality hold up on your workload? Curious where it breaks in practice.