{"slug": "best-llama-cpp-config-for-qwen3-8-flash-next-rtx-4090-24gb", "title": "Best llama.cpp config for Qwen3.8-Flash-Next (RTX 4090 24GB)", "summary": "A developer has published a configuration guide for running the Qwen3.8-Flash-Next 125B MoE model with llama.cpp on an RTX 4090 24GB system, achieving up to 29 tokens per second decode speed. The setup uses IQ3_XXS quantization and auto-offload to fit the 78GB model into 24GB VRAM plus system RAM, with detailed benchmarks and troubleshooting notes.", "body_md": "Speed/quality balanced setup for a 125B MoE (78GB, IQ3_XXS), measured on\nRTX 4090 24GB + Intel Core Ultra 7 270K Plus (24C/24T, no HT) + 96GB DDR5\n(finalized 2026-08-28). Uses mainline llama.cpp `full-cuda13`\n\n— no fork needed.\nMeasured with a 32K/96K-token technical corpus (200 generated tokens per run, fresh KV each).\n\n```\ndocker run -d --name llama-flash-next \\\n  --gpus all --shm-size=1gb -p 63082:8001 \\\n  -v \"$PWD/models:/models\" \\\n  ghcr.io/ggml-org/llama.cpp:full-cuda13 \\\n  --server \\\n  --model /models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \\\n  --alias Qwen3.8-Flash-Next \\\n  --ctx-size 102400 \\\n  --fit on --fit-target 512 \\\n  -t 20 -tb 24 \\\n  -b 6144 -ub 6144 \\\n  --flash-attn on \\\n  --cache-type-k q8_0 --cache-type-v q8_0 \\\n  --cont-batching \\\n  --reasoning-budget 4000 \\\n  --chat-template-kwargs '{\"reasoning_effort\": \"medium\"}' \\\n  --reasoning-preserve \\\n  --reasoning-budget-message \"... reasoning budget exceeded, need to answer.\\n\" \\\n  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 \\\n  -np 1 \\\n  --metrics\n```\n\nModel: unsloth/Qwen3.8-Flash-Next-GGUF (UD-IQ3_XXS, 3 shards, ~78GB). No mmproj. Download links at the bottom.\n\n| Flag | Reason |\n|---|---|\nmainline `full-cuda13` |\nQwen4 (qwen4exp) arch supported in mainline; no BeeLlama fork needed |\n| UD-IQ3_XXS | 78GB fits 24G VRAM + 93G RAM; IQ3 better decode than IQ1, not much slower |\n`--fit on --fit-target 512` |\nauto offload beats manual `-ncmoe` ; 512MiB margin measured best (1536/1024/512 sweep) |\n`--ctx-size 102400` |\nsatisfies >100K; lower ctx frees KV VRAM for fit to offload more layers |\n`-t 20 -tb 24` |\ndecode on 20 threads (24 all-cores slightly slower); batch/prefill on all 24. CPU-dependent: tuned on an Intel Core Ultra 7 270K Plus (24C/24T, no HT) — adjust to your core count |\n`-b/-ub 6144` |\nsweet spot: near-8192 prefill (982/870 t/s) with near-4096 decode (29.5/21.9) |\n`--cache-type-k/v q8_0` |\nq4_0 ≠ faster (dequant overhead cancels smaller KV); q8_0 keeps precision for free |\n`--flash-attn on` |\nrequired for quantized KV |\n`reasoning_effort medium` |\nQwen3.8 defaults xhigh (long thinking); medium balances quality/tokens |\n\nDecode is read from the server `timings`\n\n(`predicted_n/predicted_ms`\n\n), not\n`completion_tokens / total time`\n\n.\n\n| Input ctx | Prefill | Decode | VRAM |\n|---|---|---|---|\n| 32K | 966 t/s |\n29.0 t/s |\n23.2 GB |\n| 96K | 857 t/s |\n22.0 t/s |\n23.2 GB |\n\nContext-decode curve is monotonic (flash-attn O(KV)): 29.0 → 25.9 → 22.0 t/s at 32K/60K/96K. ~10-20 t/s faster than the PR #27742 reference (21 t/s) on the same hardware class.\n\n:`-b`\n\nand`-ub`\n\nmust match`-b 8192 -ub 6144`\n\nsplits each batch into`6144+2048`\n\n; the 2048 tail batch drags prefill down to ~863 instead of ~983 t/s.**MTP not usable yet**: the GGUF*does*ship an MTP head (`output_hc_up/down/norm`\n\n, no`mtp`\n\nprefix).`--spec-type draft-mtp`\n\nOOM-crash-loops trying to build a 131K draft context; wait for llama.cpp to wire it up.**ngram speculation doesn't engage** on this build (`speculative.types = none,ngram-simple`\n\n, spec counters stay 0).**Bottleneck is RAM bandwidth**, not GPU/CPU util: decode shows GPU ~40%, CPU ~35% — every token streams ~3.5GB of CPU-resident IQ3_XXS expert weights from DDR5. Don't chase GPU util %.**Q4_K_XL is slower, not faster**: 87GB > 78GB means*more*RAM reads per token. Stay on IQ3.**mainline has no kvarn6/q6_0 KV**; q5_0/q5_1 KV silently falls back to CPU on this arch. Use q8_0 or q4_0.**-b 8192 with --fit off OOMs**(15.7GB CUDA compute buffer crash-loop); with`--fit on`\n\nit loads but decode drops ~9% (fit offloads fewer layers to make room).\n\n- Model:\n[unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF)·[community speed reports](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/discussions/3) - Offload reference:\n[llama.cpp PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742)\n\n[ModelScope](https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF) (**best for China,\nmuch faster than HF mirror**):\n\n```\nmodelscope download --model unsloth/Qwen3.8-Flash-Next-GGUF --local-dir models/unsloth/Qwen3.8-Flash-Next-GGUF \\\n  --include \"*UD-IQ3_XXS*\"\n\n# direct URLs (3 shards, ~78GB):\n# https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF/resolve/master/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf\n# https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF/resolve/master/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00002-of-00003.gguf\n# https://modelscope.cn/models/unsloth/Qwen3.8-Flash-Next-GGUF/resolve/master/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00003-of-00003.gguf\nhf download unsloth/Qwen3.8-Flash-Next-GGUF --local-dir models/unsloth/Qwen3.8-Flash-Next-GGUF \\\n  --include \"*UD-IQ3_XXS*\"\n\n# direct URLs: replace `modelscope.cn/models` with `huggingface.co` and `resolve/master` with `resolve/main`\n```\n\n", "url": "https://wpnews.pro/news/best-llama-cpp-config-for-qwen3-8-flash-next-rtx-4090-24gb", "canonical_source": "https://gist.github.com/ryan4yin/48617bbddacc7067f10799770b7cc33f", "published_at": "2026-08-28 15:21:47+00:00", "updated_at": "2026-08-28 16:18:42.380193+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["llama.cpp", "Qwen3.8-Flash-Next", "RTX 4090", "Intel Core Ultra 7 270K Plus", "unsloth", "Hugging Face", "ModelScope"], "alternates": {"html": "https://wpnews.pro/news/best-llama-cpp-config-for-qwen3-8-flash-next-rtx-4090-24gb", "markdown": "https://wpnews.pro/news/best-llama-cpp-config-for-qwen3-8-flash-next-rtx-4090-24gb.md", "text": "https://wpnews.pro/news/best-llama-cpp-config-for-qwen3-8-flash-next-rtx-4090-24gb.txt", "jsonld": "https://wpnews.pro/news/best-llama-cpp-config-for-qwen3-8-flash-next-rtx-4090-24gb.jsonld"}}