Qwen3.8-Flash-Next UD-Q4_K_XL on one DGX Spark: 128K ctx + q8 KV (fix for qwen4exp self_k_rot assert)

A developer tested the Qwen3.8-Flash-Next model on an NVIDIA DGX Spark (GB10, 128 GB unified memory) using llama.cpp with quantized KV cache, achieving 128K context and 414 tok/s prefill. They identified a fix for a llama.cpp assertion failure by disabling Hadamard attention rotation via the LLAMA_ATTN_ROT_DISABLE=1 environment variable. The setup used a UD-Q4_K_XL quantized GGUF from unsloth and an optional vision mmproj, with performance metrics recorded.

Tested 2026-08-26 on a DGX Spark GB10, 128 GB unified . Measured, not estimated. With -ctk q8 0 -ctv q8 0 , llama.cpp PR 27742 qwen4exp dies at load: php src/models/qwen4exp.cpp:544: GGML ASSERT inp- self k rot == nullptr && inp- self v rot == nullptr failed Cause: a quantized KV cache auto-enables llama.cpp's Hadamard "attention rotation" on K/V, and the qwen4exp QSA sparse-attention path doesn't support rotated caches yet. Fix: disable the rotation with an env var. LLAMA ATTN ROT DISABLE=1 - llama.cpp: PR 27742 qwen4exp , commit 035e22731 , CUDA build - Weights: unsloth/Qwen3.8-Flash-Next-GGUF → UD-Q4 K XL/ 4 shards, 103.7 GiB - Optional vision: DevQuasar/Qwen.Qwen3.8-Flash-Next-GGUF → mmproj-…-f16.gguf LLAMA ATTN ROT DISABLE=1 ./build/bin/llama-server \ -m UD-Q4 K XL/Qwen3.8-Flash-Next-UD-Q4 K XL-00001-of-00004.gguf \ --mmproj mmproj-Qwen.Qwen3.8-Flash-Next.f16.gguf \ --host 0.0.0.0 --port 8000 \ -ngl 999 -c 131072 -np 1 -fa on --jinja -t 20 \ -ctk q8 0 -ctv q8 0 \ --load-mode mmap --override-tensor 'per layer token embd=CPU' \ --reasoning-format auto Notes: --override-tensor 'per layer token embd=CPU' + --load-mode mmap keeps the 51B n-gram / PLE table mmap'd from SSD instead of resident. On unified memory that's the only "offload" that actually frees anything — host-offload flags are meaningless when host RAM == GPU RAM.- Load takes ~5 minutes. You'll see one NVRM: ... Out of memory NV ERR NO MEMORY line in dmesg during the mmap page-in. It's harmless; the server comes up anyway. -np 1 : one 128K slot. Only the full-attention layers have a KV cache to quantize — the GDN/linear layers keep recurrent state regardless, so KV is cheap either way.- Free memory first: everything else on the box has to be down other models, TTS, embed servers . Baseline here was ~11 GB used before launch. | Idle after load | 89 GB used / 32 GB available | | 30,183-token prompt | prefill 414 tok/s 73 s , gen 15.7 tok/s | | Peak during that prompt | 91 GB used / 30 GB spare | | Short prompt gen | ~18.5 tok/s | Output coherent; thinking on/off via chat template kwargs.enable thinking works; reasoning content populated.