Speed/quality balanced setup for a 125B MoE (78GB, IQ3_XXS), measured on
RTX 4090 24GB + Intel Core Ultra 7 270K Plus (24C/24T, no HT) + 96GB DDR5
(finalized 2026-08-28). Uses mainline llama.cpp full-cuda13
— no fork needed. Measured with a 32K/96K-token technical corpus (200 generated tokens per run, fresh KV each).
docker run -d --name llama-flash-next \
--gpus all --shm-size=1gb -p 63082:8001 \
-v "$PWD/models:/models" \
ghcr.io/ggml-org/llama.cpp:full-cuda13 \
--server \
--model /models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
--alias Qwen3.8-Flash-Next \
--ctx-size 102400 \
--fit on --fit-target 512 \
-t 20 -tb 24 \
-b 6144 -ub 6144 \
--flash-attn on \
--cache-type-k q8_0 --cache-type-v q8_0 \
--cont-batching \
--reasoning-budget 4000 \
--chat-template-kwargs '{"reasoning_effort": "medium"}' \
--reasoning-preserve \
--reasoning-budget-message "... reasoning budget exceeded, need to answer.\n" \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 \
-np 1 \
--metrics
Model: unsloth/Qwen3.8-Flash-Next-GGUF (UD-IQ3_XXS, 3 shards, ~78GB). No mmproj. Download links at the bottom.
| Flag | Reason |
|---|---|
mainline full-cuda13 |
|
| Qwen4 (qwen4exp) arch supported in mainline; no BeeLlama fork needed | |
| UD-IQ3_XXS | 78GB fits 24G VRAM + 93G RAM; IQ3 better decode than IQ1, not much slower |
--fit on --fit-target 512 |
|
auto offload beats manual -ncmoe ; 512MiB margin measured best (1536/1024/512 sweep) |
|
--ctx-size 102400 |
|
| satisfies >100K; lower ctx frees KV VRAM for fit to offload more layers | |
-t 20 -tb 24 |
|
| decode on 20 threads (24 all-cores slightly slower); batch/prefill on all 24. CPU-dependent: tuned on an Intel Core Ultra 7 270K Plus (24C/24T, no HT) — adjust to your core count | |
-b/-ub 6144 |
|
| sweet spot: near-8192 prefill (982/870 t/s) with near-4096 decode (29.5/21.9) | |
--cache-type-k/v q8_0 |
|
| q4_0 ≠ faster (dequant overhead cancels smaller KV); q8_0 keeps precision for free | |
--flash-attn on |
|
| required for quantized KV | |
reasoning_effort medium |
|
| Qwen3.8 defaults xhigh (long thinking); medium balances quality/tokens |
Decode is read from the server timings
(predicted_n/predicted_ms
), not
completion_tokens / total time
.
| Input ctx | Prefill | Decode | VRAM |
|---|---|---|---|
| 32K | 966 t/s | ||
| 29.0 t/s | |||
| 23.2 GB | |||
| 96K | 857 t/s | ||
| 22.0 t/s | |||
| 23.2 GB |
Context-decode curve is monotonic (flash-attn O(KV)): 29.0 → 25.9 → 22.0 t/s at 32K/60K/96K. ~10-20 t/s faster than the PR #27742 reference (21 t/s) on the same hardware class.
:-b
and-ub
must match-b 8192 -ub 6144
splits each batch into6144+2048
; the 2048 tail batch drags prefill down to ~863 instead of ~983 t/s.MTP not usable yet: the GGUFdoesship an MTP head (output_hc_up/down/norm
, nomtp
prefix).--spec-type draft-mtp
OOM-crash-loops trying to build a 131K draft context; wait for llama.cpp to wire it up.ngram speculation doesn't engage on this build (speculative.types = none,ngram-simple
, spec counters stay 0).Bottleneck is RAM bandwidth, not GPU/CPU util: decode shows GPU ~40%, CPU ~35% — every token streams ~3.5GB of CPU-resident IQ3_XXS expert weights from DDR5. Don't chase GPU util %.Q4_K_XL is slower, not faster: 87GB > 78GB meansmoreRAM reads per token. Stay on IQ3.mainline has no kvarn6/q6_0 KV; q5_0/q5_1 KV silently falls back to CPU on this arch. Use q8_0 or q4_0.-b 8192 with --fit off OOMs(15.7GB CUDA compute buffer crash-loop); with--fit on
it loads but decode drops ~9% (fit offloads fewer layers to make room).
- Model: unsloth/Qwen3.8-Flash-Next-GGUF·community speed reports - Offload reference: llama.cpp PR #27742
ModelScope (best for China, much faster than HF mirror):
modelscope download --model unsloth/Qwen3.8-Flash-Next-GGUF --local-dir models/unsloth/Qwen3.8-Flash-Next-GGUF \
--include "*UD-IQ3_XXS*"
hf download unsloth/Qwen3.8-Flash-Next-GGUF --local-dir models/unsloth/Qwen3.8-Flash-Next-GGUF \
--include "*UD-IQ3_XXS*"