Running DeepSeek-V4-Flash (284B MoE) on a 64GB Strix Halo via SSD expert-streaming A developer successfully ran DeepSeek-V4-Flash, a 284B-parameter mixture-of-experts model, on a 64GB AMD Strix Halo laptop by streaming cold experts from SSD via mmap. The model achieves ~1.9 tok/s decode with only 6 of 256 experts active per token, using a custom llama.cpp fork by antirez. Key optimizations include disabling mlock, using fewer threads (4-6), and enabling flash attention with quantized KV cache. A 284B-parameter mixture-of-experts model, quantized to 2-bit, is 81 GB — bigger than the 62 GiB of RAM on an AMD Strix Halo Ryzen AI Max+ 395 . It still runs at ~1.9 tok/s, coherent , because only 6 of 256 experts fire per token : you mmap the file, keep the ~13 GB dense/attention core hot in page cache, and stream the cold 2-bit experts off the NVMe SSD on demand. CPU-only, ~1.9 tok/s decode / ~6 tok/s prefill. Slow, but it runs — a usable offline reasoning tool. - AMD Strix Halo — Ryzen AI Max+ 395 16C/32T Zen 5 , Radeon 8060S iGPU - 62 GiB unified LPDDR5X ~256 GB/s , NVMe SSD ~3.8 GB/s read, Fedora 43 Stock llama.cpp cannot load V4 novel arch: lightning indexer / DeepSeek Sparse Attention, MLA compressor, hyper-connections . Use the antirez fork: git clone https://github.com/antirez/llama.cpp-deepseek-v4-flash llama-v4-src cd llama-v4-src cmake -B build -DCMAKE BUILD TYPE=Release -DGGML NATIVE=ON -DGGML OPENMP=ON -DLLAMA CURL=OFF cmake --build build -j CPU-only on purpose: unified memory means the iGPU adds no bandwidth for decode it's memory/IO-bound , and it sidesteps whether the novel ops have GPU kernels. antirez/deepseek-v4-gguf — the IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 chat imatrix build 80.76 GiB : routed experts 2-bit, attention/shared-experts/output at Q8. On btrfs, disable CoW first to avoid fragmenting the big mmap'd file: mkdir -p ~/models/dsv4 && chattr +C ~/models/dsv4 btrfs nodatacow, BEFORE downloading into it download the -IQ2XXS- imatrix.gguf into ~/models/dsv4/ ./build/bin/llama-completion \ -m ~/models/dsv4/DeepSeek-V4-Flash-IQ2XXS-...-imatrix.gguf \ -t 6 \ 4–6 threads is OPTIMAL. More cores are SLOWER see below . -c 4096 \ explicit context; too-small context = compressor-cache assert --mmap \ REQUIRED and inverted from usual: mlock OFF. Streams experts from SSD. -fa on \ REQUIRED with quantized KV, else llama new context aborts -ctk q8 0 -ctv q8 0 \ shrink KV to leave RAM for expert pages MLA KV is tiny anyway -n 200 -no-cnv \ -p "Explain how mixture-of-experts models work:" 1. mmap ON, mlock OFF — the opposite of a normal local model. You can't lock 81 GB into 62 GB. mmap lets the OS cache the hot core and evict cold expert pages. mmap'd model pages are file-backed = always reclaimable , so they can never trigger a system OOM. 2. Fewer threads are faster — dramatically. It's memory-bandwidth-bound; extra threads just fight over the one LPDDR5X bus and thrash the page-fault path. Measured decode tok/s : | threads | 2 | 4 | 6 | 8 | 16 | 24 | 32 | |---|---|---|---|---|---|---|---| | tok/s | 1.29 | 1.98 | 1.87 | 1.81 | 1.05 | 0.70 | 0.22 | -t 4 beats -t 32 by 9× . Use -t 6 for a balance best prefill, near-best decode . 3. -fa on is mandatory with -ctk/-ctv q8 0 MLA attention , or context creation asserts. 4. Compressor-cache assert GGML ASSERT n comp visible <= n comp cache deepseek4.cpp : the DeepSeek Sparse Attention cache is sized from n ctx . - Long prefill batch : chunk with -ub 128 -b 128 . - Long generation : just pass a real -c e.g. 4096 . Under llama-bench it looks like a ~128-token cap — that's a bench artifact tight context , not a real limit. 5. ☠️ NEVER inspect metadata with llama-gguf