# Qwen3.8-27B best llama.cpp config on RTX 4090 24GB (BeeLlama, UD-Q4_K_XL, kvarn6 + kv-tail 2048 @ 130K, MTP n-max 3, fit off)

> Source: <https://gist.github.com/ryan4yin/19db9fa44972c5735c1d181e8888d4fe>
> Published: 2026-08-17 10:03:12+00:00

Quality-first setup, measured on a single 24GB GPU (finalized 2026-08-16). Use the BeeLlama fork image — mainline llama.cpp silently falls back to CPU for non-q4 KV caches on Qwen3.x hybrid architecture (no error is reported).

```
docker run -d --name llama-server \
  --gpus all --shm-size=1gb -p 8001:8001 \
  -v "$PWD/models:/models" \
  ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.4.3 \
  --model /models/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf \
  --mmproj /models/unsloth/Qwen3.8-27B-GGUF/mmproj-F16.gguf \
  --alias Qwen3.8-27B \
  --ctx-size 130000 \
  --cache-type-k kvarn6 --cache-type-v kvarn6 \
  --kv-tail-tokens 2048 \
  --fit off \
  --flash-attn on \
  --n-gpu-layers auto \
  --load-mode mmap \
  --host 0.0.0.0 --port 8001 \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 \
  --image-max-tokens 4000 --image-min-tokens 1024 \
  --reasoning-budget 4000 \
  --chat-template-kwargs '{"reasoning_effort": "low"}' \
  --reasoning-preserve \
  --reasoning-budget-message "... reasoning budget exceeded, need to answer.\n"
```

Model: [unsloth/Qwen3.8-27B-GGUF](https://modelscope.cn/models/unsloth/Qwen3.8-27B-GGUF)
(UD-Q4_K_XL mixed precision + mmproj-F16).

| Flag | Reason |
|---|---|
| UD-Q4_K_XL | mixed-precision quant, ~780MB heavier than Q4_K_M, better quality |
| kvarn6/kvarn6 | KVarN supersedes stock quants: better quality at the same VRAM |
`--kv-tail-tokens 2048` |
last 2K tokens kept full-precision F16 (~55 MiB); fixes "attention drift / missing details" in long contexts |
`--ctx-size 130000` |
150K OOMs in real multi-turn (restorable-prefix) usage; 130K leaves headroom |
`--fit off` |
CUDA graphs conflict with MTP draft; fit on randomly drops speed to 9–20 t/s |
MTP `n-max 3` |
measured best: n=1 → 60 t/s, n=2 → 37 t/s, n=3 → best |
`--flash-attn on` |
required for quantized V, loader refuses otherwise |
`mmap` / `n-gpu-layers auto` |
BeeLlama has no `auto` load mode |
| temp/top_p/top_k/min_p | thinking-mode sampling recommended by unsloth |
| reasoning budget/preserve | effort `low` keeps thinking short; preserve keeps multi-turn thinking context |

~60–70 t/s decode in typical use; 50–62 t/s at 126.6K input context.

**BeeLlama required**: non-q4 KV on mainline = silent CPU fallback (GPU 0–30%).- f16/bf16 KV + 27B on 24GB OOMs outright.
- OOM depends on request batch shape, not just ctx depth: a restorable-prefix multi-turn continuation (KV only 66.8K) can OOM even when a 150K prefill passes. Validate ctx ceilings with restorable-continuation traffic, not single-shot prefill.
- Throughput: read response
`timings`

(`predicted_n/predicted_ms`

), not`completion_tokens / total time`

(dragged down by prefill).

KV quant selection: [Anbeeld benchmarks](https://anbeeld.com/articles/kv-cache-quantization-benchmarks-for-long-context) ·
[llama.cpp #23470](https://github.com/ggml-org/llama.cpp/discussions/23470) ·
[BeeLlama](https://github.com/Anbeeld/BeeLlama.cpp)