{"slug": "kv-cache-vram-calculator-how-much-memory-does-context-length-need-2026", "title": "KV Cache VRAM Calculator: How Much Memory Does Context Length Need? (2026)", "summary": "A cloudgpu.app writeup lays out a formula for sizing the KV cache in transformer models, showing that cache memory grows linearly with context length: 2 × layers × KV heads × head dimension × bytes per value per token. For Qwen3 32B at FP16 the cache costs 256 KiB per token, roughly 8 GB at a 32k-token context on top of model weights, which the piece says explains why models that load fine can still run out of memory when context is raised. It tabulates per-token cache sizes for open models including Llama 3.1 8B, Qwen3 8B/14B/32B, Mistral Small 3 24B, Llama 3.3 70B and Qwen2.5 72B, and gives context ceilings for 24, 32, 48 and 80 GB GPUs.", "body_md": "**The KV cache is the memory a transformer uses to remember every token in the context, and its size grows linearly with context length:** bytes per token = 2 × layers × KV heads × head dimension × bytes per value. For Qwen3 32B at FP16 that is 256 KiB per token, or about 8 GB for a 32k-token context, on top of the model weights. This is why a model that loads fine can still run out of memory when you raise the context.\n\nThis post gives the formula, the per-token numbers for popular open models (read from their `config.json`), and how much context fits on 24, 32, 48 and 80 GB cards.\n\n```\nkv_bytes_per_token = 2 × num_hidden_layers × num_key_value_heads × head_dim × bytes_per_value\nkv_total_bytes     = kv_bytes_per_token × context_tokens × concurrent_sequences\n```\n\n`num_key_value_heads` is smaller than the number of attention heads in models with grouped-query attention (GQA), which is why modern models have much smaller caches than older ones.`bytes_per_value` is 2 for FP16/BF16, 1 for 8-bit KV cache, about 0.5 for 4-bit.\nArchitecture values are from each model’s `config.json` on Hugging Face (read 2026-10-23). GB here means GiB (1024³ bytes).\n\n| Model | Layers | KV heads | head_dim | KV per token | 8k ctx | 32k ctx | 128k ctx | \n|---|---|---|---|---|---|---|---|\n| Llama 3.1 8B | 32 | 8 | 128 | 128 KiB | 1.0 GB | 4.0 GB | 16 GB | \n| Qwen3 8B | 36 | 8 | 128 | 144 KiB | 1.1 GB | 4.5 GB | 18 GB* | \n| Qwen3 14B | 40 | 8 | 128 | 160 KiB | 1.25 GB | 5.0 GB | 20 GB* | \n| Mistral Small 3 24B | 40 | 8 | 128 | 160 KiB | 1.25 GB | 5.0 GB | — (32k max) | \n| Qwen3 32B | 64 | 8 | 128 | 256 KiB | 2.0 GB | 8.0 GB | 32 GB* | \n| Llama 3.3 70B | 80 | 8 | 128 | 320 KiB | 2.5 GB | 10 GB | 40 GB | \n| Qwen2.5 72B | 80 | 8 | 128 | 320 KiB | 2.5 GB | 10 GB | 40 GB* | \n\n* Beyond the model’s native window. Qwen3 dense models are documented with a native 32,768-token context, extendable with YaRN rope scaling; check the model card before going past the native length.\n\nThe formula does not apply as-is to models with sliding-window layers (for example Gemma 3 and gpt-oss) or latent attention (DeepSeek V3-style MLA); their caches are smaller than this formula suggests.\n\nTotal memory ≈ weights + KV cache + overhead. Weights are the Q4_K_M sizes on our [/run](https://cloudgpu.app/run) pages; overhead is an **assumed 1.5 GB** for CUDA context, buffers and the runtime. The result is a ceiling for a single sequence; runtimes reserve extra buffers, so plan below it.\n\n**Qwen3 32B at Q4_K_M (≈ 19.5 GB weights, 256 KiB/token):**\n\n| Card class | Left for KV after weights + 1.5 GB | Context ceiling (1 sequence) | Practical guidance | \n|---|---|---|---|\n| 24 GB (RTX 4090 / 3090) | ≈ 3 GB | ≈ 12k tokens | about 8k; tight | \n| 32 GB (RTX 5090) | ≈ 11 GB | ≈ 45k (native max 32k) | 16k–32k comfortable | \n| 48 GB (RTX 4090 48G) | ≈ 27 GB | ≈ 110k | full native 32k, or several parallel slots | \n\n**Llama 3.3 70B at Q4_K_M (≈ 41.5 GB weights, 320 KiB/token):**\n\n| Card class | Left for KV | Context ceiling (1 sequence) | Practical guidance | \n|---|---|---|---|\n| 48 GB (RTX 4090 48G / A6000) | ≈ 5 GB | ≈ 16k | about 8k; tight | \n| 80 GB (A100 / A800) | ≈ 37 GB | ≈ 121k | 32k comfortable, with room for parallel requests | \n\nThe “practical guidance” column matches the per-model notes on our [Qwen3 32B page](https://cloudgpu.app/run/qwen3-32b) and [Llama 3.3 70B page](https://cloudgpu.app/run/llama-3-3-70b).\n\n``` python\ndef kv_gb(layers, kv_heads, head_dim, ctx, bytes_per_value=2, seqs=1):\n    return 2 * layers * kv_heads * head_dim * bytes_per_value * ctx * seqs / 1024**3\n\ndef max_ctx(vram_gb, weights_gb, layers, kv_heads, head_dim,\n            bytes_per_value=2, overhead_gb=1.5, seqs=1):\n    free = vram_gb - weights_gb - overhead_gb\n    per_tok = 2 * layers * kv_heads * head_dim * bytes_per_value / 1024**3\n    return max(0, int(free / (per_tok * seqs)))\n\n# Qwen3 32B (64 layers, 8 KV heads, head_dim 128), Q4 weights ≈ 19.5 GB\nprint(kv_gb(64, 8, 128, 32768))          # 8.0\nprint(max_ctx(32, 19.5, 64, 8, 128))     # ≈ 45k (native limit is 32k)\nprint(max_ctx(24, 19.5, 64, 8, 128, bytes_per_value=1))  # 8-bit KV: ≈ 24k\n```\n\nRead `num_hidden_layers`, `num_key_value_heads` and `head_dim` (or `hidden_size / num_attention_heads`) from the model’s `config.json`.\n\n**1. Quantise the KV cache.** Halving bytes per value halves the cache.\n\n```\n# Ollama: 8-bit KV cache (requires flash attention)\nexport OLLAMA_FLASH_ATTENTION=1\nexport OLLAMA_KV_CACHE_TYPE=q8_0\nollama serve\n# vLLM: FP8 KV cache\nvllm serve Qwen/Qwen3-32B-AWQ --kv-cache-dtype fp8 --max-model-len 32768\n```\n\nQuality impact is usually small at 8-bit and more noticeable at 4-bit; test on your own prompts.\n\n**2. Set the context you need, not the maximum.** In Ollama, pass `\"options\": {\"num_ctx\": 16384}` per request (or set `OLLAMA_CONTEXT_LENGTH`). In vLLM, `--max-model-len` caps the context and `--gpu-memory-utilization` sets how much VRAM the cache may take.\n\n**3. Watch parallelism.** Ollama’s `OLLAMA_NUM_PARALLEL` and vLLM’s batching both allocate cache per sequence. If memory runs out under load but not in single tests, this is usually why.\n\nSay you run Qwen3 32B (Q4_K_M) with Ollama on a 24 GB RTX 4090, set `num_ctx` to 32768, and get out-of-memory errors or a sudden slowdown (layers spilling to CPU). The formula explains it: about 19.5 GB of weights plus about 8 GB of FP16 KV cache is more than 24 GB. Try these, least effort first:\n\n`OLLAMA_FLASH_ATTENTION=1` and `OLLAMA_KV_CACHE_TYPE=q8_0`, the same memory holds roughly twice the context — about 24k by the formula (a ceiling; leave headroom).`OLLAMA_NUM_PARALLEL=1` to rule out several slots sharing the memory.\nAfter each change, check real usage with `nvidia-smi` and note the number; next time you size a card, you will have your own data.\n\nOnce you have weights + KV + overhead, round up to the next VRAM class and check [Deploy](https://cloudgpu.app/deploy) for live stock. List prices on 2026-10-23 23:58 Beijing time: RTX 4090 24G $0.33/h, RTX 5090 32G $0.59/h, RTX 4090 48G $0.65/h, A800 80G $1.09/h, A100 80G $1.49/h, billed per minute. Instances start in about 2 minutes. For a side-by-side of those cards, see [RTX 4090 vs RTX 5090 vs A100](https://cloudgpu.app/compare/rtx-4090-vs-5090-vs-a100).\n\n**How do I calculate the KV cache size for a model?**\n\nMultiply 2 × layers × KV heads × head dimension × bytes per value (2 for FP16) to get bytes per token, then multiply by context length and the number of parallel sequences. The values are in the model’s config.json.\n\n**How much VRAM does a 32k context need for Qwen3 32B?**\n\nAbout 8 GB of FP16 KV cache (256 KiB per token × 32,768 tokens), on top of about 19.5 GB of Q4_K_M weights, so a 32 GB card is the comfortable choice.\n\n**Why does my model run out of memory only at long context?**\n\nThe weights are fixed, but the KV cache grows with every token of context and with each parallel request. Lower num_ctx / max-model-len, reduce parallel slots, or quantise the KV cache.\n\n**Does an 8-bit KV cache halve memory?**\n\nYes, the cache itself is halved compared with FP16. In Ollama set OLLAMA_KV_CACHE_TYPE=q8_0 with flash attention on; in vLLM use --kv-cache-dtype fp8.\n\n**Does the formula work for every model?**\n\nIt works for standard and grouped-query attention. Models with sliding-window layers or latent attention use less memory than it predicts.\n\nOnce you have your own numbers, the [RTX 4090 vs RTX 5090 vs A100 comparison](https://cloudgpu.app/compare/rtx-4090-vs-5090-vs-a100) lines up the cards by VRAM class. *Originally published at [cloudgpu.app](https://cloudgpu.app/blog/kv-cache-vram-context-length-calculator-2026).*\n\nCloudGPU (cloudgpu.app) is an independent service. It has no partnership with, authorization from, or endorsement by the model vendor. Model names are trademarks of their owners. Context ceilings are estimates under the stated assumptions, not measurements. Prices are list prices read on 2026-10-23 23:58 Beijing time (UTC+8); live prices and stock on [/deploy](https://cloudgpu.app/deploy) take precedence.", "url": "https://wpnews.pro/news/kv-cache-vram-calculator-how-much-memory-does-context-length-need-2026", "canonical_source": "https://dev.to/cloudgpu_app/kv-cache-vram-calculator-how-much-memory-does-context-length-need-2026-gc3", "published_at": "2026-10-24 13:07:30+00:00", "updated_at": "2026-10-24 13:12:33.865119+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "mlops", "ai-tools"], "entities": ["Qwen3 32B", "Llama 3.3 70B", "Llama 3.1 8B", "Qwen2.5 72B", "Mistral Small 3 24B", "Hugging Face", "cloudgpu.app", "NVIDIA RTX 5090"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/kv-cache-vram-calculator-how-much-memory-does-context-length-need-2026", "markdown": "https://wpnews.pro/news/kv-cache-vram-calculator-how-much-memory-does-context-length-need-2026.md", "text": "https://wpnews.pro/news/kv-cache-vram-calculator-how-much-memory-does-context-length-need-2026.txt", "jsonld": "https://wpnews.pro/news/kv-cache-vram-calculator-how-much-memory-does-context-length-need-2026.jsonld"}}