Qwen3.8-27B best llama.cpp config on RTX 4090 24GB (BeeLlama, UD-Q4_K_XL, kvarn6 + kv-tail 2048 @ 130K, MTP n-max 3, fit off)

An engineer shared a quality-first llama.cpp configuration for running the Qwen3.8-27B model on a single RTX 4090 24GB GPU, achieving 60–70 tokens per second. The setup requires the BeeLlama fork because mainline llama.cpp silently falls back to CPU for non-q4 KV caches on Qwen3.x hybrid architecture. The configuration uses UD-Q4_K_XL mixed-precision quantization, kvarn6 KV cache quantization with a 2048-token full-precision tail, and multi-token prediction with n-max 3.

Quality-first setup, measured on a single 24GB GPU finalized 2026-08-16 . Use the BeeLlama fork image — mainline llama.cpp silently falls back to CPU for non-q4 KV caches on Qwen3.x hybrid architecture no error is reported . docker run -d --name llama-server \ --gpus all --shm-size=1gb -p 8001:8001 \ -v "$PWD/models:/models" \ ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.4.3 \ --model /models/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4 K XL.gguf \ --mmproj /models/unsloth/Qwen3.8-27B-GGUF/mmproj-F16.gguf \ --alias Qwen3.8-27B \ --ctx-size 130000 \ --cache-type-k kvarn6 --cache-type-v kvarn6 \ --kv-tail-tokens 2048 \ --fit off \ --flash-attn on \ --n-gpu-layers auto \ --load-mode mmap \ --host 0.0.0.0 --port 8001 \ --spec-type draft-mtp --spec-draft-n-max 3 \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 \ --image-max-tokens 4000 --image-min-tokens 1024 \ --reasoning-budget 4000 \ --chat-template-kwargs '{"reasoning effort": "low"}' \ --reasoning-preserve \ --reasoning-budget-message "... reasoning budget exceeded, need to answer.\n" Model: unsloth/Qwen3.8-27B-GGUF https://modelscope.cn/models/unsloth/Qwen3.8-27B-GGUF UD-Q4 K XL mixed precision + mmproj-F16 . | Flag | Reason | |---|---| | UD-Q4 K XL | mixed-precision quant, ~780MB heavier than Q4 K M, better quality | | kvarn6/kvarn6 | KVarN supersedes stock quants: better quality at the same VRAM | --kv-tail-tokens 2048 | last 2K tokens kept full-precision F16 ~55 MiB ; fixes "attention drift / missing details" in long contexts | --ctx-size 130000 | 150K OOMs in real multi-turn restorable-prefix usage; 130K leaves headroom | --fit off | CUDA graphs conflict with MTP draft; fit on randomly drops speed to 9–20 t/s | MTP n-max 3 | measured best: n=1 → 60 t/s, n=2 → 37 t/s, n=3 → best | --flash-attn on | required for quantized V, loader refuses otherwise | mmap / n-gpu-layers auto | BeeLlama has no auto load mode | | temp/top p/top k/min p | thinking-mode sampling recommended by unsloth | | reasoning budget/preserve | effort low keeps thinking short; preserve keeps multi-turn thinking context | ~60–70 t/s decode in typical use; 50–62 t/s at 126.6K input context. BeeLlama required : non-q4 KV on mainline = silent CPU fallback GPU 0–30% .- f16/bf16 KV + 27B on 24GB OOMs outright. - OOM depends on request batch shape, not just ctx depth: a restorable-prefix multi-turn continuation KV only 66.8K can OOM even when a 150K prefill passes. Validate ctx ceilings with restorable-continuation traffic, not single-shot prefill. - Throughput: read response timings predicted n/predicted ms , not completion tokens / total time dragged down by prefill . KV quant selection: Anbeeld benchmarks https://anbeeld.com/articles/kv-cache-quantization-benchmarks-for-long-context · llama.cpp 23470 https://github.com/ggml-org/llama.cpp/discussions/23470 · BeeLlama https://github.com/Anbeeld/BeeLlama.cpp