Best llama.cpp config for Qwen3.8-Flash-Next (RTX 4090 24GB)

A developer has published a configuration guide for running the Qwen3.8-Flash-Next 125B MoE model with llama.cpp on an RTX 4090 24GB system, achieving up to 29 tokens per second decode speed. The setup uses IQ3_XXS quantization and auto-offload to fit the 78GB model into 24GB VRAM plus system RAM, with detailed benchmarks and troubleshooting notes.

Speed/quality balanced setup for a 125B MoE 78GB, IQ3 XXS , measured on RTX 4090 24GB + Intel Core Ultra 7 270K Plus 24C/24T, no HT + 96GB DDR5 finalized 2026-08-28 . Uses mainline llama.cpp full-cuda13 — no fork needed. Measured with a 32K/96K-token technical corpus 200 generated tokens per run, fresh KV each . docker run -d --name llama-flash-next \ --gpus all --shm-size=1gb -p 63082:8001 \ -v "$PWD/models:/models" \ ghcr.io/ggml-org/llama.cpp:full-cuda13 \ --server \ --model /models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-IQ3 XXS/Qwen3.8-Flash-Next-UD-IQ3 XXS-00001-of-00003.gguf \ --alias Qwen3.8-Flash-Next \ --ctx-size 102400 \ --fit on --fit-target 512 \ -t 20 -tb 24 \ -b 6144 -ub 6144 \ --flash-attn on \ --cache-type-k q8 0 --cache-type-v q8 0 \ --cont-batching \ --reasoning-budget 4000 \ --chat-template-kwargs '{"reasoning effort": "medium"}' \ --reasoning-preserve \ --reasoning-budget-message "... reasoning budget exceeded, need to answer.\n" \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 \ -np 1 \ --metrics Model: unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ3 XXS, 3 shards, ~78GB . No mmproj. Download links at the bottom. | Flag | Reason | |---|---| mainline full-cuda13 | Qwen4 qwen4exp arch supported in mainline; no BeeLlama fork needed | | UD-IQ3 XXS | 78GB fits 24G VRAM + 93G RAM; IQ3 better decode than IQ1, not much slower | --fit on --fit-target 512 | auto offload beats manual -ncmoe ; 512MiB margin measured best 1536/1024/512 sweep | --ctx-size 102400 | satisfies 100K; lower ctx frees KV VRAM for fit to offload more layers | -t 20 -tb 24 | decode on 20 threads 24 all-cores slightly slower ; batch/prefill on all 24. CPU-dependent: tuned on an Intel Core Ultra 7 270K Plus 24C/24T, no HT — adjust to your core count | -b/-ub 6144 | sweet spot: near-8192 prefill 982/870 t/s with near-4096 decode 29.5/21.9 | --cache-type-k/v q8 0 | q4 0 ≠ faster dequant overhead cancels smaller KV ; q8 0 keeps precision for free | --flash-attn on | required for quantized KV | reasoning effort medium | Qwen3.8 defaults xhigh long thinking ; medium balances quality/tokens | Decode is read from the server timings predicted n/predicted ms , not completion tokens / total time . | Input ctx | Prefill | Decode | VRAM | |---|---|---|---| | 32K | 966 t/s | 29.0 t/s | 23.2 GB | | 96K | 857 t/s | 22.0 t/s | 23.2 GB | Context-decode curve is monotonic flash-attn O KV : 29.0 → 25.9 → 22.0 t/s at 32K/60K/96K. ~10-20 t/s faster than the PR 27742 reference 21 t/s on the same hardware class. : -b and -ub must match -b 8192 -ub 6144 splits each batch into 6144+2048 ; the 2048 tail batch drags prefill down to ~863 instead of ~983 t/s. MTP not usable yet : the GGUF does ship an MTP head output hc up/down/norm , no mtp prefix . --spec-type draft-mtp OOM-crash-loops trying to build a 131K draft context; wait for llama.cpp to wire it up. ngram speculation doesn't engage on this build speculative.types = none,ngram-simple , spec counters stay 0 . Bottleneck is RAM bandwidth , not GPU/CPU util: decode shows GPU ~40%, CPU ~35% — every token streams ~3.5GB of CPU-resident IQ3 XXS expert weights from DDR5. Don't chase GPU util %. Q4 K XL is slower, not faster : 87GB 78GB means more RAM reads per token. Stay on IQ3. mainline has no kvarn6/q6 0 KV ; q5 0/q5 1 KV silently falls back to CPU on this arch. Use q8 0 or q4 0. -b 8192 with --fit off OOMs 15.7GB CUDA compute buffer crash-loop ; with --fit on it loads but decode drops ~9% fit offloads fewer layers to make room . - Model: unsloth/Qwen3.8-Flash-Next-GGUF https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF · community speed reports https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/discussions/3 - Offload reference: llama.cpp PR 27742 https://github.com/ggml-org/llama.cpp/pull/27742 ModelScope https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF best for China, much faster than HF mirror : modelscope download --model unsloth/Qwen3.8-Flash-Next-GGUF --local-dir models/unsloth/Qwen3.8-Flash-Next-GGUF \ --include " UD-IQ3 XXS " direct URLs 3 shards, ~78GB : https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF/resolve/master/UD-IQ3 XXS/Qwen3.8-Flash-Next-UD-IQ3 XXS-00001-of-00003.gguf https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF/resolve/master/UD-IQ3 XXS/Qwen3.8-Flash-Next-UD-IQ3 XXS-00002-of-00003.gguf https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF/resolve/master/UD-IQ3 XXS/Qwen3.8-Flash-Next-UD-IQ3 XXS-00003-of-00003.gguf hf download unsloth/Qwen3.8-Flash-Next-GGUF --local-dir models/unsloth/Qwen3.8-Flash-Next-GGUF \ --include " UD-IQ3 XXS " direct URLs: replace modelscope.cn/models with huggingface.co and resolve/master with resolve/main