A 284B-parameter mixture-of-experts model, quantized to 2-bit, is 81 GB — bigger than the
62 GiB of RAM on an AMD Strix Halo (Ryzen AI Max+ 395). It still runs at ~1.9 tok/s, coherent,
because only 6 of 256 experts fire per token: you mmap
the file, keep the ~13 GB dense/attention core hot in page cache, and stream the cold 2-bit experts off the NVMe SSD on demand.
CPU-only, ~1.9 tok/s decode / ~6 tok/s prefill. Slow, but it runs — a usable offline reasoning tool.
- AMD Strix Halo — Ryzen AI Max+ 395 (16C/32T Zen 5), Radeon 8060S iGPU
- 62 GiB unified LPDDR5X (~256 GB/s), NVMe SSD ~3.8 GB/s read, Fedora 43
Stock llama.cpp cannot load V4 (novel arch: lightning indexer / DeepSeek Sparse Attention, MLA compressor, hyper-connections). Use the antirez fork:
git clone https://github.com/antirez/llama.cpp-deepseek-v4-flash llama-v4-src
cd llama-v4-src
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_OPENMP=ON -DLLAMA_CURL=OFF
cmake --build build -j
CPU-only on purpose: unified memory means the iGPU adds no bandwidth for decode (it's memory/IO-bound), and it sidesteps whether the novel ops have GPU kernels.
antirez/deepseek-v4-gguf
— the IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8
chat imatrix build (80.76 GiB): routed experts 2-bit, attention/shared-experts/output at Q8. On btrfs, disable CoW first to avoid fragmenting the big mmap'd file:
mkdir -p ~/models/dsv4 && chattr +C ~/models/dsv4 # btrfs nodatacow, BEFORE down into it
./build/bin/llama-completion \
-m ~/models/dsv4/DeepSeek-V4-Flash-IQ2XXS-...-imatrix.gguf \
-t 6 \ # 4–6 threads is OPTIMAL. More cores are SLOWER (see below).
-c 4096 \ # explicit context; too-small context => compressor-cache assert
--mmap \ # REQUIRED and inverted from usual: mlock OFF. Streams experts from SSD.
-fa on \ # REQUIRED with quantized KV, else llama_new_context() aborts
-ctk q8_0 -ctv q8_0 \ # shrink KV to leave RAM for expert pages (MLA KV is tiny anyway)
-n 200 -no-cnv \
-p "Explain how mixture-of-experts models work:"
1. mmap ON, mlock OFF — the opposite of a normal local model. You can't lock 81 GB into 62 GB. mmap lets the OS cache the hot core and evict cold expert pages. mmap'd model pages are file-backed = always reclaimable, so they can never trigger a system OOM.
2. Fewer threads are faster — dramatically. It's memory-bandwidth-bound; extra threads just fight over the one LPDDR5X bus and thrash the page-fault path. Measured decode (tok/s):
| threads | 2 | 4 | 6 | 8 | 16 | 24 | 32 | |---|---|---|---|---|---|---|---| | tok/s | 1.29 | 1.98 | 1.87 | 1.81 | 1.05 | 0.70 | 0.22 |
-t 4
beats -t 32
by 9×. Use -t 6
for a balance (best prefill, near-best decode).
3. -fa on is mandatory with
-ctk/-ctv q8_0
(MLA attention), or context creation asserts.4. Compressor-cache assert GGML_ASSERT(n_comp_visible <= n_comp_cache)
(deepseek4.cpp):
the DeepSeek Sparse Attention cache is sized from n_ctx
.
- Long
prefill batch: chunk with
-ub 128 -b 128
. - Long
generation: just pass a real-c
(e.g. 4096). Underllama-bench
it looks like a ~128-token cap — that's a bench artifact (tight context), not a real limit.
5. ☠️ NEVER inspect metadata with llama-gguf <model> r. Read mode loads
every tensor as anonymous heap— 53 GB, no mmap — which exhausted RAM+swap and triggered the global OOM killer (took down my terminal and several containers). Use
gguf_dump.py --no-tensors
instead.6. Blast-radius guard. Run big loads in a swap-disabled cgroup so a surprise can't take the
desktop down: systemd-run --user --scope -p MemorySwapMax=0 -- <your llama command>
.
Do NOT add a MemoryMax
cap — it charges the mmap page cache to the cgroup and forces evict+re-read thrashing (measured 211 GB of re-reads at a 50 GB cap).
MTP speculative decode: the MTP head loads as archdeepseek4_mtp_support
andllama-speculative
/-md
reject it as a standalone draft. And in an IO-bound regime it wouldn't help much anyway — verifying K draft tokens still streams K×(6×43) experts.GPU: deferred. Unified memory means no decode bandwidth win; possible future use is prefill only.
81 GB model, 62 GiB RAM, no GPU, ~1.9 tok/s, coherent. MoE sparsity + mmap + a fast SSD makes "bigger than RAM" a solvable problem. Tune threads DOWN, not up.