My RTX 4070 was running Qwen 35B beautifully after the --cpu-moe
trick from a previous run. The tokens/sec were where I wanted them. VRAM sat at 11,714 MiB out of 12,281 — 95% full.
That leaves 600 MiB. Not enough for a serious agent.
The context window I was giving llama.cpp was -c 4096
. Fine for chat. Not fine when a Claude Code-style agent hands the model 12,000 tokens of tool definitions before it says hello.
I wanted -c 32768
. That's an 8× jump. And the memory that grows with context length is the KV cache. Multiply the cache by 8 with 600 MiB free, and llama.cpp dies during warm-up. I know because I tried it first.
After off the MoE experts to CPU (the previous chapter's trick), the GPU is holding two things:
The first is fixed. The second grows linearly with context length. Double the context, double the cache. -c 4096 → -c 32768
doesn't just want 8× more tokens processed, it wants 8× more cache resident in VRAM the whole time.
There is no room. So the cache itself has to shrink.
llama.cpp takes two flags for KV cache dtype:
llama-server -m qwen35.gguf -ngl 99 --cpu-moe -c 32768 \
-ctk q8_0 -ctv q8_0
-ctk
is the Key cache, -ctv
is the Value cache. Default is f16 (16-bit). q8_0
cuts each in half. Halving both means the KV cache footprint drops by roughly 50%.
That freed-up VRAM is exactly what I need to make the context 8× bigger without touching the model weights.
Same prompt, same seed, two runs — one at f16 KV, one at q8_0 KV:
| KV dtype | Max -c I could allocate |
Tokens/sec (decode) | Perplexity delta |
|---|---|---|---|
| f16 (default) | 4096 | ~34.6 | baseline |
| q8_0 | 32768 | ~34.1 | negligible in my tests |
The speed loss is inside noise. The context is 8× longer. The quality drop I could not tell apart from run-to-run variance.
Community measurements agree: symmetric q8_0 KV lands somewhere under 0.1% perplexity delta on most models. Going harder — q4_0 on both K and V — is where you start seeing real degradation, and asymmetric setups (-ctk q4_0 -ctv q8_0
) end up being the pragmatic Q4 configuration if you go there.
I have not gone there. q8_0 was already enough headroom for the agent workloads I run.
Keep it in your head as two rows:
That is the entire decision.
-c
Bigger -c
is not free even with quantized KV. llama.cpp reserves the full window up front — a 32k context eats 32k worth of cache the moment the server boots, whether you use it or not.
So I tier it:
-c 4096
. Do not bother quantizing KV.-c 8192
covered almost every case I hit. No quantization needed if you have any headroom.-c 32768
with -ctk q8_0 -ctv q8_0
. The first agent turn alone can hit 19,000 tokens once system prompt and tool schemas load. 8192 will crash mid-invocation.-c
to the actual document size. Do not pre-provision 32k for a 6k document.The generalization "agents need big context and quantized KV" is too coarse. The right number depends entirely on the CLI's tool definition weight. Two agent frameworks on the same model can want radically different -c
values.
If you have a 12GB card and you want Qwen 35B to run agents:
That is a total of three flags: --cpu-moe -ctk q8_0 -ctv q8_0
. For that, you get roughly the same tokens/sec you had, an 8× longer usable context, and no measurable quality regression on the workloads I have tested.
I spent a week assuming I needed a 4090 to get here. I did not.
If you're pointing a Claude Code-style agent at this setup — where tool schemas and system prompts eat 12k tokens before you type anything — the sizing decisions and CLAUDE.md hardening are covered here:
Claude Code Mastery — agent workflows, context budgeting, and hardening