# Stop Guessing Your Context Budget: Practical KV Cache Sizing for Local LLMs > Source: > Published: 2026-10-29 05:03:30+00:00 You set a 32K context, the weights fit in VRAM, and the model still refuses to load. Or it loads, and generation crawls because the card is one allocation away from paging. The culprit is almost never the model file. It is the KV cache: the per-token key and value tensors that every attention layer keeps so it does not have to recompute the past. It is reserved up front, it scales linearly with context, and on a 7B model at 32K it is measured in gigabytes, not megabytes. This post is the arithmetic I wish I had the first time a "small" context change OOMed a 12 GB card, plus the flags that actually control the allocation in llama.cpp and Ollama. I will also show why the popular "quantize it and it is half the memory" advice is close but not correct, and how to check whether the cheaper cache wrecked your model's recall. For a standard transformer with full attention, the cache holds two tensors per layer, per token, per KV head. So: ``` bytes_per_token = 2 * n_layers * n_kv_heads * head_dim * bytes_per_element ``` The factor of 2 is K and V. The term that surprises people is **n_kv_heads**, not the dtype. Take Qwen2.5-7B-Instruct, whose config declares 28 hidden layers, 28 attention heads, 4 key/value heads, and a hidden size of 3584 (so head_dim is 3584 / 28 = 128). Its cache holds: ``` 2 * 28 * 4 * 128 = 28,672 values per token ``` At f16 (2 bytes per value) that is 57,344 bytes, or **56 KiB per token**. At 32,768 tokens that is 1.75 GiB of VRAM before a single token is generated. Mistral-7B-Instruct-v0.3 has the same head_dim but 32 layers and 8 KV heads: ``` 2 * 32 * 8 * 128 = 65,536 values per token = 128 KiB per token ``` At 32K that is **4 GiB**. Same parameter count as Qwen, more than twice the cache, because it keeps twice as many KV heads per layer. That is the whole GQA story. A 7B model with plain multi-head attention (32 KV heads) would need 512 KiB per token, or 16 GiB at 32K, which is why nobody ships 7B MHA models for consumer GPUs any more. When you compare two models, compare their KV head counts and layer counts first. Parameter count tells you almost nothing about context cost. Two footnotes before the tables: llama.cpp does not store one float per value in a quantized cache. It stores blocks of 32 values with a scale. The ggml block for q8_0 is a 2-byte half plus 32 int8 values, so 34 bytes for 32 values, not 32. q4_0 is a 2-byte scale plus 16 packed bytes, so 18 bytes for 32 values. That overhead is small but it is exactly why "half" and "quarter" are approximations: | Cache type | Bytes per value | Share of f16 | |---|---|---| | f16 / bf16 | 2.0 | 100% | | q8_0 | 34 / 32 = 1.0625 | 53.1% | | q5_1 | 24 / 32 = 0.75 | 37.5% | | q4_1 | 20 / 32 = 0.625 | 31.3% | | q4_0 | 18 / 32 = 0.5625 | 28.1% | So q8_0 saves 47% of the cache, not 50%. q4_0 saves 72%, not 75%. On a 4 GiB cache the difference between the claim and reality is around 120 MiB, which is enough to matter on an 8 GB card running a 14B model. Real numbers for the two models above at 32K, computed with the formula and the table: | Model | f16 | q8_0 | q4_0 | |---|---|---|---| | Qwen2.5-7B (28L / 4 KV) | 1.75 GiB | 952 MiB | 504 MiB | | Mistral-7B v0.3 (32L / 8 KV) | 4.00 GiB | 2.13 GiB | 1.13 GiB | Those figures are pure cache. Add weights, the compute buffer, and the CUDA context on top. On a 12 GB card, a 7B Q4_K_M model plus a 32K f16 cache plus buffers is a genuinely tight fit, which is why the same setup with q8_0 suddenly has headroom. Start with an explicit, measurable baseline. Explicit matters, because the defaults are doing more than you think. ``` llama-server \ -m qwen2.5-7b-instruct-q4_k_m.gguf \ --host 127.0.0.1 --port 8080 \ -c 32768 \ -ngl 99 \ -fa on \ -ctk f16 -ctv f16 ``` `-c 32768` is the total context pool. Left at 0, llama.cpp takes the model's trained context, which on a 128K model is a spectacular way to run out of memory.`-fa on` forces Flash Attention. The default is `auto`, which enables it when the backend supports it.`-ctk` and `-ctv` set the K and V cache types independently. Allowed values include f32, f16, bf16, q8_0, q5_1, q5_0, q4_1, q4_0 and iq4_nl. The cache is allocated at load time, so measure before sending traffic: ``` nvidia-smi --query-gpu=memory.used --format=csv,noheader ``` Then restart with the quantized cache and the same everything else: ``` llama-server \ -m qwen2.5-7b-instruct-q4_k_m.gguf \ --host 127.0.0.1 --port 8080 \ -c 32768 \ -ngl 99 \ -fa on \ -ctk q8_0 -ctv q8_0 ``` On the Qwen example the delta should be roughly 800 MiB. If the delta is zero, the cache did not change, and you should read the startup log rather than assume success. Confirm the effective context from the running server: ``` curl -s http://127.0.0.1:8080/props | jq '.default_generation_settings.n_ctx' # if that path is absent in your build, inspect the whole object: # curl -s http://127.0.0.1:8080/props | jq . ``` Quantizing V without Flash Attention does not silently fall back. It fails at context creation: ``` llama_new_context_with_model: V cache quantization requires flash_attn common_init_from_params: failed to create context with model '...' ``` This is reported in llama.cpp issue #10378, and it is still the first thing to check when a quantized cache will not start. Keep `-fa on` when you quantize V. If you deliberately disabled Flash Attention for a backend reason, you cannot use a quantized V cache on that backend. **`--fit` is on by default.** It adjusts arguments you did not set so the model fits in device memory, including context size, with a floor set by `--fit-ctx` (default 4096). That is helpful for a quick run and terrible for reproducibility: you can ask for a model default of 128K, get 4K, and never notice. If a specific context is a requirement, pass `-c` explicitly and watch the load log. **`--kv-unified` changes what parallel slots cost.** When slots are auto-sized, llama.cpp uses one unified KV buffer shared by all sequences, so the pool is sized by the total `-c`, not multiplied by the slot count. If you set `--kv-unified-per-slot N` without `-c`, the pool becomes `n_parallel * N`, which is a very different number. Read the flag you actually passed. **`--cache-ram` is host RAM, not VRAM.** It defaults to 8192 MiB and holds the prompt cache so idle slots can be parked. On a 16 GB mini-PC that is a meaningful chunk of system memory. Set it to 0 to disable if RAM, not VRAM, is your constraint. If VRAM is the hard limit and you can afford the latency, `--no-kv-offload` keeps the cache in system RAM instead of offloading it to the GPU. The model still runs, just slower, and the freed VRAM can go to more layers or a longer context. Ollama has the same physics and two extra traps. First, the default context is **4096 tokens**, not the model's trained context. That is why a model can appear to ignore the second half of a long document. Raise it with `OLLAMA_CONTEXT_LENGTH`, or per request with `num_ctx`. Second, parallel requests multiply the cache. The official FAQ is explicit: required RAM scales with `OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH`. Each slot gets the configured context, so 8K with 4 parallel requests reserves 32K worth of cache, not 8K. Quantized cache in Ollama is an environment variable, and it applies to every model the server loads: ``` sudo mkdir -p /etc/systemd/system/ollama.service.d sudo tee /etc/systemd/system/ollama.service.d/kv-cache.conf >/dev/null <<'EOF' [Service] Environment="OLLAMA_FLASH_ATTENTION=1" Environment="OLLAMA_KV_CACHE_TYPE=q8_0" Environment="OLLAMA_CONTEXT_LENGTH=32768" Environment="OLLAMA_NUM_PARALLEL=1" EOF sudo systemctl daemon-reload sudo systemctl restart ollama ``` Then confirm what actually loaded: ``` ollama ps journalctl -u ollama -n 50 --no-pager ``` The SIZE column in `ollama ps` includes the cache, so compare it before and after the change with the same context and the same model. If SIZE does not move, the drop-in did not apply: check for a typo in the unit name, and remember that a model already loaded has to be unloaded or restarted before new settings take effect. Ollama documents the trade-off honestly: q8_0 uses about half the memory of f16 with a very small precision loss, q4_0 about a quarter with a loss that can become noticeable at long context. It also notes that models with a high ratio of query heads to KV heads, such as Qwen2, may be more sensitive. That matches the arithmetic: the fewer KV heads a model has, the more each one carries. Memory is easy to measure. Quality is not, and "it looked fine in one chat" is not a test. A cheap, repeatable check is a needle test: bury a random code deep in a long prompt and ask for it back. Run it at f16, q8_0 and q4_0, several times each, and compare. ``` python #!/usr/bin/env python3 import json, random, urllib.request BASE = "http://127.0.0.1:8080" NEEDLE = f"{random.randint(0, 999999):06d}" FILLER = ("The archive index lists routine maintenance notes for the eastern " "warehouse, including shelf counts, inspection dates, and handler " "initials. ") def build(target_sentences=450, depth=0.2): parts = [FILLER] * target_sentences parts.insert(int(len(parts) * depth), f" IMPORTANT: the vault code is {NEEDLE}. ") return "".join(parts) prompt = build() payload = { "model": "qwen2.5-7b-instruct", "messages": [{ "role": "user", "content": prompt + "\n\nAnswer with only the six-digit vault code. " "If it is not present, answer NONE.", }], "temperature": 0, "max_tokens": 16, } req = urllib.request.Request( BASE + "/v1/chat/completions", data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"}, ) with urllib.request.urlopen(req, timeout=600) as resp: out = json.load(resp) answer = out["choices"][0]["message"]["content"].strip() print("needle:", NEEDLE, "answer:", answer) print("prompt_tokens:", out.get("usage", {}).get("prompt_tokens")) print("PASS" if NEEDLE in answer else "FAIL") ``` At roughly 450 sentences of that filler you land near 12K to 14K tokens, which is enough to stress a 32K cache without waiting all afternoon. The `prompt_tokens` field tells you the real depth; do not guess it, and scale the sentence count if your tokenizer disagrees. A needle test measures retrieval, not reasoning. Add one task that resembles your actual workload and has a checkable answer: extract fields into JSON, answer a question about a specific clause, or produce a diff. Exact-match and structured-output tasks degrade before free-form prose does, so they are the sensitive canaries. If q4_0 passes retrieval but fails your JSON extraction, you have your answer, and it is not the one the memory table suggested. For throughput, compare generation speed at the same context length. The native `/completion` endpoint returns a `timings` object with `predicted_per_second`, which is more useful than a stopwatch: ``` curl -s http://127.0.0.1:8080/completion \ -H 'Content-Type: application/json' \ -d '{"prompt":"Summarize the plot of a heist film in three sentences.","n_predict":128}' \ | jq '.timings' ``` Quantized caches usually help at long context because attention reads less memory, but dequantization costs compute, and the balance depends on your backend. Measure it; do not take a percentage from a blog post, including this one. `2 * layers * kv_heads * head_dim * bytes`, and read kv_heads from config.json.`--fit` may shrink a context you did not pin.`OLLAMA_NUM_PARALLEL` multiplies the bill. The cache is the difference between "the model fits" and "the model works". Once you know its size, context length stops being a guess and becomes a number you choose. `-fa`, `--kv-unified`, `--kv-unified-per-slot`, `--cache-ram`, `--fit-ctx`: