# Best llama.cpp config for Qwen3.8-Flash-Next (RTX 4090 24GB)

> Source: <https://gist.github.com/ryan4yin/48617bbddacc7067f10799770b7cc33f>
> Published: 2026-08-28 15:21:47+00:00

Speed/quality balanced setup for a 125B MoE (78GB, IQ3_XXS), measured on
RTX 4090 24GB + Intel Core Ultra 7 270K Plus (24C/24T, no HT) + 96GB DDR5
(finalized 2026-08-28). Uses mainline llama.cpp `full-cuda13`

— no fork needed.
Measured with a 32K/96K-token technical corpus (200 generated tokens per run, fresh KV each).

```
docker run -d --name llama-flash-next \
  --gpus all --shm-size=1gb -p 63082:8001 \
  -v "$PWD/models:/models" \
  ghcr.io/ggml-org/llama.cpp:full-cuda13 \
  --server \
  --model /models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
  --alias Qwen3.8-Flash-Next \
  --ctx-size 102400 \
  --fit on --fit-target 512 \
  -t 20 -tb 24 \
  -b 6144 -ub 6144 \
  --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --cont-batching \
  --reasoning-budget 4000 \
  --chat-template-kwargs '{"reasoning_effort": "medium"}' \
  --reasoning-preserve \
  --reasoning-budget-message "... reasoning budget exceeded, need to answer.\n" \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 \
  -np 1 \
  --metrics
```

Model: unsloth/Qwen3.8-Flash-Next-GGUF (UD-IQ3_XXS, 3 shards, ~78GB). No mmproj. Download links at the bottom.

| Flag | Reason |
|---|---|
mainline `full-cuda13` |
Qwen4 (qwen4exp) arch supported in mainline; no BeeLlama fork needed |
| UD-IQ3_XXS | 78GB fits 24G VRAM + 93G RAM; IQ3 better decode than IQ1, not much slower |
`--fit on --fit-target 512` |
auto offload beats manual `-ncmoe` ; 512MiB margin measured best (1536/1024/512 sweep) |
`--ctx-size 102400` |
satisfies >100K; lower ctx frees KV VRAM for fit to offload more layers |
`-t 20 -tb 24` |
decode on 20 threads (24 all-cores slightly slower); batch/prefill on all 24. CPU-dependent: tuned on an Intel Core Ultra 7 270K Plus (24C/24T, no HT) — adjust to your core count |
`-b/-ub 6144` |
sweet spot: near-8192 prefill (982/870 t/s) with near-4096 decode (29.5/21.9) |
`--cache-type-k/v q8_0` |
q4_0 ≠ faster (dequant overhead cancels smaller KV); q8_0 keeps precision for free |
`--flash-attn on` |
required for quantized KV |
`reasoning_effort medium` |
Qwen3.8 defaults xhigh (long thinking); medium balances quality/tokens |

Decode is read from the server `timings`

(`predicted_n/predicted_ms`

), not
`completion_tokens / total time`

.

| Input ctx | Prefill | Decode | VRAM |
|---|---|---|---|
| 32K | 966 t/s |
29.0 t/s |
23.2 GB |
| 96K | 857 t/s |
22.0 t/s |
23.2 GB |

Context-decode curve is monotonic (flash-attn O(KV)): 29.0 → 25.9 → 22.0 t/s at 32K/60K/96K. ~10-20 t/s faster than the PR #27742 reference (21 t/s) on the same hardware class.

:`-b`

and`-ub`

must match`-b 8192 -ub 6144`

splits each batch into`6144+2048`

; the 2048 tail batch drags prefill down to ~863 instead of ~983 t/s.**MTP not usable yet**: the GGUF*does*ship an MTP head (`output_hc_up/down/norm`

, no`mtp`

prefix).`--spec-type draft-mtp`

OOM-crash-loops trying to build a 131K draft context; wait for llama.cpp to wire it up.**ngram speculation doesn't engage** on this build (`speculative.types = none,ngram-simple`

, spec counters stay 0).**Bottleneck is RAM bandwidth**, not GPU/CPU util: decode shows GPU ~40%, CPU ~35% — every token streams ~3.5GB of CPU-resident IQ3_XXS expert weights from DDR5. Don't chase GPU util %.**Q4_K_XL is slower, not faster**: 87GB > 78GB means*more*RAM reads per token. Stay on IQ3.**mainline has no kvarn6/q6_0 KV**; q5_0/q5_1 KV silently falls back to CPU on this arch. Use q8_0 or q4_0.**-b 8192 with --fit off OOMs**(15.7GB CUDA compute buffer crash-loop); with`--fit on`

it loads but decode drops ~9% (fit offloads fewer layers to make room).

- Model:
[unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF)·[community speed reports](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/discussions/3) - Offload reference:
[llama.cpp PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742)

[ModelScope](https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF) (**best for China,
much faster than HF mirror**):

```
modelscope download --model unsloth/Qwen3.8-Flash-Next-GGUF --local-dir models/unsloth/Qwen3.8-Flash-Next-GGUF \
  --include "*UD-IQ3_XXS*"

# direct URLs (3 shards, ~78GB):
# https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF/resolve/master/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf
# https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF/resolve/master/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00002-of-00003.gguf
# https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF/resolve/master/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00003-of-00003.gguf
hf download unsloth/Qwen3.8-Flash-Next-GGUF --local-dir models/unsloth/Qwen3.8-Flash-Next-GGUF \
  --include "*UD-IQ3_XXS*"

# direct URLs: replace `modelscope.cn/models` with `huggingface.co` and `resolve/master` with `resolve/main`
```