{"slug": "qwen3-8-flash-next-ud-q4-k-xl-on-one-dgx-spark-128k-ctx-q8-kv-fix-for-qwen4exp-k", "title": "Qwen3.8-Flash-Next UD-Q4_K_XL on one DGX Spark: 128K ctx + q8 KV (fix for qwen4exp self_k_rot assert)", "summary": "A developer tested the Qwen3.8-Flash-Next model on an NVIDIA DGX Spark (GB10, 128 GB unified memory) using llama.cpp with quantized KV cache, achieving 128K context and 414 tok/s prefill. They identified a fix for a llama.cpp assertion failure by disabling Hadamard attention rotation via the LLAMA_ATTN_ROT_DISABLE=1 environment variable. The setup used a UD-Q4_K_XL quantized GGUF from unsloth and an optional vision mmproj, with performance metrics recorded.", "body_md": "Tested 2026-08-26 on a DGX Spark (GB10, 128 GB unified). Measured, not estimated.\n\nWith `-ctk q8_0 -ctv q8_0`\n\n, llama.cpp PR #27742 (`qwen4exp`\n\n) dies at load:\n\n``` php\nsrc/models/qwen4exp.cpp:544: GGML_ASSERT(inp->self_k_rot == nullptr && inp->self_v_rot == nullptr) failed\n```\n\nCause: a quantized KV cache auto-enables llama.cpp's Hadamard \"attention rotation\" on K/V, and the qwen4exp QSA sparse-attention path doesn't support rotated caches yet.\n\n**Fix: disable the rotation with an env var.**\n\n```\nLLAMA_ATTN_ROT_DISABLE=1\n```\n\n- llama.cpp: PR #27742 (\n`qwen4exp`\n\n), commit`035e22731`\n\n, CUDA build - Weights:\n`unsloth/Qwen3.8-Flash-Next-GGUF`\n\n→`UD-Q4_K_XL/`\n\n(4 shards, 103.7 GiB) - Optional vision:\n`DevQuasar/Qwen.Qwen3.8-Flash-Next-GGUF`\n\n→`mmproj-…-f16.gguf`\n\n```\nLLAMA_ATTN_ROT_DISABLE=1 ./build/bin/llama-server \\\n  -m UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \\\n  --mmproj mmproj-Qwen.Qwen3.8-Flash-Next.f16.gguf \\\n  --host 0.0.0.0 --port 8000 \\\n  -ngl 999 -c 131072 -np 1 -fa on --jinja -t 20 \\\n  -ctk q8_0 -ctv q8_0 \\\n  --load-mode mmap --override-tensor 'per_layer_token_embd=CPU' \\\n  --reasoning-format auto\n```\n\nNotes:\n\n`--override-tensor 'per_layer_token_embd=CPU'`\n\n+`--load-mode mmap`\n\nkeeps the 51B n-gram / PLE table mmap'd from SSD instead of resident. On unified memory that's the only \"offload\" that actually frees anything — host-offload flags are meaningless when host RAM == GPU RAM.- Load takes ~5 minutes. You'll see one\n`NVRM: ... Out of memory [NV_ERR_NO_MEMORY]`\n\nline in dmesg during the mmap page-in. It's harmless; the server comes up anyway. `-np 1`\n\n: one 128K slot. Only the full-attention layers have a KV cache to quantize — the GDN/linear layers keep recurrent state regardless, so KV is cheap either way.- Free memory first: everything else on the box has to be down (other models, TTS, embed servers). Baseline here was ~11 GB used before launch.\n\n| Idle after load | 89 GB used / 32 GB available |\n| 30,183-token prompt | prefill 414 tok/s (73 s), gen 15.7 tok/s |\n| Peak during that prompt | 91 GB used / 30 GB spare |\n| Short prompt gen | ~18.5 tok/s |\n\nOutput coherent; thinking on/off via `chat_template_kwargs.enable_thinking`\n\nworks; `reasoning_content`\n\npopulated.", "url": "https://wpnews.pro/news/qwen3-8-flash-next-ud-q4-k-xl-on-one-dgx-spark-128k-ctx-q8-kv-fix-for-qwen4exp-k", "canonical_source": "https://gist.github.com/CocaKova/3cbfbf4991e93eee39f4c7612712e487", "published_at": "2026-08-26 21:45:46+00:00", "updated_at": "2026-08-27 06:48:30.136607+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Qwen3.8-Flash-Next", "DGX Spark", "llama.cpp", "unsloth", "NVIDIA", "DevQuasar"], "alternates": {"html": "https://wpnews.pro/news/qwen3-8-flash-next-ud-q4-k-xl-on-one-dgx-spark-128k-ctx-q8-kv-fix-for-qwen4exp-k", "markdown": "https://wpnews.pro/news/qwen3-8-flash-next-ud-q4-k-xl-on-one-dgx-spark-128k-ctx-q8-kv-fix-for-qwen4exp-k.md", "text": "https://wpnews.pro/news/qwen3-8-flash-next-ud-q4-k-xl-on-one-dgx-spark-128k-ctx-q8-kv-fix-for-qwen4exp-k.txt", "jsonld": "https://wpnews.pro/news/qwen3-8-flash-next-ud-q4-k-xl-on-one-dgx-spark-128k-ctx-q8-kv-fix-for-qwen4exp-k.jsonld"}}