Quality-first setup, measured on a single 24GB GPU (finalized 2026-08-16). Use the BeeLlama fork image — mainline llama.cpp silently falls back to CPU for non-q4 KV caches on Qwen3.x hybrid architecture (no error is reported).
docker run -d --name llama-server \
--gpus all --shm-size=1gb -p 8001:8001 \
-v "$PWD/models:/models" \
ghcr.io/anbeeld/beellama.cpp:server-cuda13-v0.4.3 \
--model /models/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf \
--mmproj /models/unsloth/Qwen3.8-27B-GGUF/mmproj-F16.gguf \
--alias Qwen3.8-27B \
--ctx-size 130000 \
--cache-type-k kvarn6 --cache-type-v kvarn6 \
--kv-tail-tokens 2048 \
--fit off \
--flash-attn on \
--n-gpu-layers auto \
--load-mode mmap \
--host 0.0.0.0 --port 8001 \
--spec-type draft-mtp --spec-draft-n-max 3 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 \
--image-max-tokens 4000 --image-min-tokens 1024 \
--reasoning-budget 4000 \
--chat-template-kwargs '{"reasoning_effort": "low"}' \
--reasoning-preserve \
--reasoning-budget-message "... reasoning budget exceeded, need to answer.\n"
Model: unsloth/Qwen3.8-27B-GGUF (UD-Q4_K_XL mixed precision + mmproj-F16).
| Flag | Reason |
|---|---|
| UD-Q4_K_XL | mixed-precision quant, ~780MB heavier than Q4_K_M, better quality |
| kvarn6/kvarn6 | KVarN supersedes stock quants: better quality at the same VRAM |
--kv-tail-tokens 2048 |
|
| last 2K tokens kept full-precision F16 (~55 MiB); fixes "attention drift / missing details" in long contexts | |
--ctx-size 130000 |
|
| 150K OOMs in real multi-turn (restorable-prefix) usage; 130K leaves headroom | |
--fit off |
|
| CUDA graphs conflict with MTP draft; fit on randomly drops speed to 9–20 t/s | |
MTP n-max 3 |
|
| measured best: n=1 → 60 t/s, n=2 → 37 t/s, n=3 → best | |
--flash-attn on |
|
| required for quantized V, refuses otherwise | |
mmap / n-gpu-layers auto |
|
BeeLlama has no auto load mode |
|
| temp/top_p/top_k/min_p | thinking-mode sampling recommended by unsloth |
| reasoning budget/preserve | effort low keeps thinking short; preserve keeps multi-turn thinking context |
~60–70 t/s decode in typical use; 50–62 t/s at 126.6K input context.
BeeLlama required: non-q4 KV on mainline = silent CPU fallback (GPU 0–30%).- f16/bf16 KV + 27B on 24GB OOMs outright.
- OOM depends on request batch shape, not just ctx depth: a restorable-prefix multi-turn continuation (KV only 66.8K) can OOM even when a 150K prefill passes. Validate ctx ceilings with restorable-continuation traffic, not single-shot prefill.
- Throughput: read response
timings
(predicted_n/predicted_ms
), notcompletion_tokens / total time
(dragged down by prefill).
KV quant selection: Anbeeld benchmarks · llama.cpp #23470 · BeeLlama
source & further reading
gist.github.com — original article
This is my OpenCode setup for local models, mainly using a customized llama.cpp configuration.
2026-08-17 Daily work report — human and AI activity attribution
An LLM skill for understanding sarcasm without asking for clarification