# Your 128K Context Fits on a 4090 — If You Quantize the KV Cache

> Source: <https://dev.to/qisuancloud/your-128k-context-fits-on-a-4090-if-you-quantize-the-kv-cache-51bp>
> Published: 2026-10-19 06:10:55+00:00

*Researched October 2026. Sizing math from public model cards and the vLLM docs; not my own benchmarks.*

In my [last post](https://dev.to/qisuancloud/your-7b-model-fits-on-a-4090-until-you-open-a-128k-context-window) I showed that a 128K context window can push a 7B model from 17 GB to 33 GB of VRAM — and that you have three moves, in order of cost-effectiveness: quantize the cache, pick a GQA-heavy architecture, or rent the bigger card. This post is about move #1, the cheapest one. It halves the KV cache with a single flag.

In vLLM:

```
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8
```

That's it. The weights stay in FP16/BF16 — untouched. Only the KV cache is stored as 1 byte per value instead of 2. Per the [vLLM docs](https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/), `fp8` maps to E4M3, and there are three calibration options: no calibration (scales default to 1.0), random-token warmup, or dataset calibration via llm-compressor (recommended for max accuracy).

Weight quantization gets all the attention, but KV values are arguably the easier target: within a single attention head, their values live in a narrow, well-behaved range. E4M3's tighter range and 3-bit mantissa match that distribution well; E5M2's wider range is mostly wasted on values that never appear, and its 2-bit mantissa visibly degrades attention scores that compound across layers. In other words, the format choice matters — `fp8` (E4M3) is the right default, not `fp8_e5m2`.

Llama 3.1 8B, FP16 weights (~16 GB), 2 GB overhead:

| Context | FP16 cache → total | FP8 cache → total | 
|---|---|---|
| 32K | 4 GB → ~22 GB ✅ | 2 GB → ~20 GB ✅ | 
| 64K | 8 GB → ~26 GB ❌ | 4 GB → ~22 GB ✅ fits 4090 | 
| 128K | 16 GB → ~34 GB ❌ | 8 GB → ~26 GB ❌ still over | 

Honest note: at 128K, FP8 cache alone doesn't save Llama 3.1 8B on a 24 GB card — 26 GB is still 26 GB. Your options there: pair it with FP8 weights (8 + 8 + 2 ≈ 18 GB ✅), or pick the GQA-heavy architecture from last post — Qwen2.5 7B at 128K needs ~7 GB of FP16 cache, ~3.5 GB in FP8, for a total around 19 GB ✅. The flag is powerful, not magic; the arithmetic still rules.

**1. Hardware matters.** The biggest wins are on Hopper GPUs with native FP8 support. On older cards, check your vLLM version's kernel support — the memory savings are real, but the compute path may not be as optimized.

**2. Calibrate if quality matters.** Default scales of 1.0 are fine to start, but for production, dataset calibration (llm-compressor) exists for a reason. Per-head quantization — finer-grained than per-tensor — needs the FlashAttention backend plus that calibration path.

**3. Measure retrieval, not just perplexity.** Standard benchmarks show small degradation, but long-context tasks with exact-recall demands (needle-in-a-haystack style document QA) are where you'd notice it first. Run your own eval before and after.

The whole point: quantization keeps you on the cheap card. A 64K-context 7B workload that needed an 80 GB card at ~$1.50+/hr in FP16 fits on a 24 GB 4090 at ~$0.35/hr with FP8 cache — same model, same context, roughly a quarter of the hourly cost, for one CLI flag.

I built a [free GPU Advisor](https://qisuanai.com/advisor) that walks through these tradeoffs (model size, context, quantization tolerance) and recommends the cheapest fitting setup. No signup.

Have you tried FP8 KV cache in production — and did the quality hold up on your workload? Curious where it breaks in practice.