Tested 2026-08-26 on a DGX Spark (GB10, 128 GB unified). Measured, not estimated.
With -ctk q8_0 -ctv q8_0
, llama.cpp PR #27742 (qwen4exp
) dies at load:
src/models/qwen4exp.cpp:544: GGML_ASSERT(inp->self_k_rot == nullptr && inp->self_v_rot == nullptr) failed
Cause: a quantized KV cache auto-enables llama.cpp's Hadamard "attention rotation" on K/V, and the qwen4exp QSA sparse-attention path doesn't support rotated caches yet.
Fix: disable the rotation with an env var.
LLAMA_ATTN_ROT_DISABLE=1
- llama.cpp: PR #27742 (
qwen4exp
), commit035e22731
, CUDA build - Weights:
unsloth/Qwen3.8-Flash-Next-GGUF
→UD-Q4_K_XL/
(4 shards, 103.7 GiB) - Optional vision:
DevQuasar/Qwen.Qwen3.8-Flash-Next-GGUF
→mmproj-…-f16.gguf
LLAMA_ATTN_ROT_DISABLE=1 ./build/bin/llama-server \
-m UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
--mmproj mmproj-Qwen.Qwen3.8-Flash-Next.f16.gguf \
--host 0.0.0.0 --port 8000 \
-ngl 999 -c 131072 -np 1 -fa on --jinja -t 20 \
-ctk q8_0 -ctv q8_0 \
--load-mode mmap --override-tensor 'per_layer_token_embd=CPU' \
--reasoning-format auto
Notes:
--override-tensor 'per_layer_token_embd=CPU'
+--load-mode mmap
keeps the 51B n-gram / PLE table mmap'd from SSD instead of resident. On unified memory that's the only "offload" that actually frees anything — host-offload flags are meaningless when host RAM == GPU RAM.- Load takes ~5 minutes. You'll see one
NVRM: ... Out of memory [NV_ERR_NO_MEMORY]
line in dmesg during the mmap page-in. It's harmless; the server comes up anyway. -np 1
: one 128K slot. Only the full-attention layers have a KV cache to quantize — the GDN/linear layers keep recurrent state regardless, so KV is cheap either way.- Free memory first: everything else on the box has to be down (other models, TTS, embed servers). Baseline here was ~11 GB used before launch.
| Idle after load | 89 GB used / 32 GB available | | 30,183-token prompt | prefill 414 tok/s (73 s), gen 15.7 tok/s | | Peak during that prompt | 91 GB used / 30 GB spare | | Short prompt gen | ~18.5 tok/s |
Output coherent; thinking on/off via chat_template_kwargs.enable_thinking
works; reasoning_content
populated.