{"slug": "vbr-k-v-cache-is-actually-usable-buun-llama", "title": "VBR k/v cache is actually usable (buun-llama)", "summary": "A user running Qwen3.8-Flash-Next at IQ3 quantization on a system with 64GB of RAM, an AMD 9950X3D, a 9070XT, and a ZFS stripe of three mid-range NVMe drives reported fitting 255K tokens of context using buun-llama's VBR k/v cache, achieving roughly 120 prompt-processing tokens per second and about 30 tokens per second of generation. The configuration used the flags -ctk vbr -ctv vbr --vbr-min-bits t4 --moe-cache soft with a draft-mtp speculative model, and the user said the setup completed several reasoning tasks quickly via qwen-code without tuning batch or MoE cache settings.", "body_md": "So i have been running qwen-3.8-flash-next iq3 with great results with just 64gb of ram, a 9950x3d, a 9070xt and a zfs znvme stripe of 3 mid range nvmes.\n\nthe vbr that buun uses fits 255k of context on this setup and i am getting ~ 120 pp and ~ 30 of token gen.\n\nthis is the actual command\n\nllama-server \\\n\n-m Qwen3.8-Flash-*00001*.gguf \\\n\n-c $((255* 1024)) \\\n\n-fa on -ctk vbr -ctv vbr --vbr-min-bits t4 --moe-cache soft \\\n\n–host 0.0.0.0 --port 11434 --reasoning on --reasoning-effort medium --reasoning-preserve --metrics \\\n\n–spec-type draft-mtp --model-draft ../mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \\\n\n-b 512 -ub 512 --fit on --slot-prompt-similarity 0.2\n\ni have not really gone and optimized the b /ub and moe cache but yea with qwen-code it achieved several re tasks, quite fast too.", "url": "https://wpnews.pro/news/vbr-k-v-cache-is-actually-usable-buun-llama", "canonical_source": "https://forum.level1techs.com/t/vbr-k-v-cache-is-actually-usable-buun-llama/256271#post_1", "published_at": "2026-09-16 13:12:59+00:00", "updated_at": "2026-09-16 13:41:45.745511+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["Qwen3.8-Flash-Next", "buun-llama", "llama-server", "AMD 9950X3D", "AMD 9070XT", "qwen-code"], "alternates": {"html": "https://wpnews.pro/news/vbr-k-v-cache-is-actually-usable-buun-llama", "markdown": "https://wpnews.pro/news/vbr-k-v-cache-is-actually-usable-buun-llama.md", "text": "https://wpnews.pro/news/vbr-k-v-cache-is-actually-usable-buun-llama.txt", "jsonld": "https://wpnews.pro/news/vbr-k-v-cache-is-actually-usable-buun-llama.jsonld"}}