{"slug": "qwen3-8-27b-best-llama-cpp-config-on-rtx-4090-24gb-beellama-ud-q4-k-xl-kvarn6-kv", "title": "Qwen3.8-27B best llama.cpp config on RTX 4090 24GB (BeeLlama, UD-Q4_K_XL, kvarn6 + kv-tail 2048 @ 130K, MTP n-max 3, fit off)", "summary": "An engineer shared a quality-first llama.cpp configuration for running the Qwen3.8-27B model on a single RTX 4090 24GB GPU, achieving 60–70 tokens per second. The setup requires the BeeLlama fork because mainline llama.cpp silently falls back to CPU for non-q4 KV caches on Qwen3.x hybrid architecture. The configuration uses UD-Q4_K_XL mixed-precision quantization, kvarn6 KV cache quantization with a 2048-token full-precision tail, and multi-token prediction with n-max 3.", "body_md": "Quality-first setup, measured on a single 24GB GPU (finalized 2026-08-16). Use the BeeLlama fork image — mainline llama.cpp silently falls back to CPU for non-q4 KV caches on Qwen3.x hybrid architecture (no error is reported).\n\n```\ndocker run -d --name llama-server \\\n  --gpus all --shm-size=1gb -p 8001:8001 \\\n  -v \"$PWD/models:/models\" \\\n  ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.4.3 \\\n  --model /models/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf \\\n  --mmproj /models/unsloth/Qwen3.8-27B-GGUF/mmproj-F16.gguf \\\n  --alias Qwen3.8-27B \\\n  --ctx-size 130000 \\\n  --cache-type-k kvarn6 --cache-type-v kvarn6 \\\n  --kv-tail-tokens 2048 \\\n  --fit off \\\n  --flash-attn on \\\n  --n-gpu-layers auto \\\n  --load-mode mmap \\\n  --host 0.0.0.0 --port 8001 \\\n  --spec-type draft-mtp --spec-draft-n-max 3 \\\n  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 \\\n  --image-max-tokens 4000 --image-min-tokens 1024 \\\n  --reasoning-budget 4000 \\\n  --chat-template-kwargs '{\"reasoning_effort\": \"low\"}' \\\n  --reasoning-preserve \\\n  --reasoning-budget-message \"... reasoning budget exceeded, need to answer.\\n\"\n```\n\nModel: [unsloth/Qwen3.8-27B-GGUF](https://modelscope.cn/models/unsloth/Qwen3.8-27B-GGUF)\n(UD-Q4_K_XL mixed precision + mmproj-F16).\n\n| Flag | Reason |\n|---|---|\n| UD-Q4_K_XL | mixed-precision quant, ~780MB heavier than Q4_K_M, better quality |\n| kvarn6/kvarn6 | KVarN supersedes stock quants: better quality at the same VRAM |\n`--kv-tail-tokens 2048` |\nlast 2K tokens kept full-precision F16 (~55 MiB); fixes \"attention drift / missing details\" in long contexts |\n`--ctx-size 130000` |\n150K OOMs in real multi-turn (restorable-prefix) usage; 130K leaves headroom |\n`--fit off` |\nCUDA graphs conflict with MTP draft; fit on randomly drops speed to 9–20 t/s |\nMTP `n-max 3` |\nmeasured best: n=1 → 60 t/s, n=2 → 37 t/s, n=3 → best |\n`--flash-attn on` |\nrequired for quantized V, loader refuses otherwise |\n`mmap` / `n-gpu-layers auto` |\nBeeLlama has no `auto` load mode |\n| temp/top_p/top_k/min_p | thinking-mode sampling recommended by unsloth |\n| reasoning budget/preserve | effort `low` keeps thinking short; preserve keeps multi-turn thinking context |\n\n~60–70 t/s decode in typical use; 50–62 t/s at 126.6K input context.\n\n**BeeLlama required**: non-q4 KV on mainline = silent CPU fallback (GPU 0–30%).- f16/bf16 KV + 27B on 24GB OOMs outright.\n- OOM depends on request batch shape, not just ctx depth: a restorable-prefix multi-turn continuation (KV only 66.8K) can OOM even when a 150K prefill passes. Validate ctx ceilings with restorable-continuation traffic, not single-shot prefill.\n- Throughput: read response\n`timings`\n\n(`predicted_n/predicted_ms`\n\n), not`completion_tokens / total time`\n\n(dragged down by prefill).\n\nKV quant selection: [Anbeeld benchmarks](https://anbeeld.com/articles/kv-cache-quantization-benchmarks-for-long-context) ·\n[llama.cpp #23470](https://github.com/ggml-org/llama.cpp/discussions/23470) ·\n[BeeLlama](https://github.com/Anbeeld/BeeLlama.cpp)", "url": "https://wpnews.pro/news/qwen3-8-27b-best-llama-cpp-config-on-rtx-4090-24gb-beellama-ud-q4-k-xl-kvarn6-kv", "canonical_source": "https://gist.github.com/ryan4yin/19db9fa44972c5735c1d181e8888d4fe", "published_at": "2026-08-17 10:03:12+00:00", "updated_at": "2026-08-18 06:40:59.706908+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "mlops", "ai-infrastructure"], "entities": ["Qwen3.8-27B", "llama.cpp", "BeeLlama", "RTX 4090", "unsloth", "Anbeeld"], "alternates": {"html": "https://wpnews.pro/news/qwen3-8-27b-best-llama-cpp-config-on-rtx-4090-24gb-beellama-ud-q4-k-xl-kvarn6-kv", "markdown": "https://wpnews.pro/news/qwen3-8-27b-best-llama-cpp-config-on-rtx-4090-24gb-beellama-ud-q4-k-xl-kvarn6-kv.md", "text": "https://wpnews.pro/news/qwen3-8-27b-best-llama-cpp-config-on-rtx-4090-24gb-beellama-ud-q4-k-xl-kvarn6-kv.txt", "jsonld": "https://wpnews.pro/news/qwen3-8-27b-best-llama-cpp-config-on-rtx-4090-24gb-beellama-ud-q4-k-xl-kvarn6-kv.jsonld"}}