{"slug": "running-deepseek-v4-flash-284b-moe-on-a-64gb-strix-halo-via-ssd-expert-streaming", "title": "Running DeepSeek-V4-Flash (284B MoE) on a 64GB Strix Halo via SSD expert-streaming", "summary": "A developer successfully ran DeepSeek-V4-Flash, a 284B-parameter mixture-of-experts model, on a 64GB AMD Strix Halo laptop by streaming cold experts from SSD via mmap. The model achieves ~1.9 tok/s decode with only 6 of 256 experts active per token, using a custom llama.cpp fork by antirez. Key optimizations include disabling mlock, using fewer threads (4-6), and enabling flash attention with quantized KV cache.", "body_md": "A 284B-parameter mixture-of-experts model, quantized to 2-bit, is **81 GB** — bigger than the\n62 GiB of RAM on an AMD Strix Halo (Ryzen AI Max+ 395). It still runs at **~1.9 tok/s, coherent**,\nbecause only **6 of 256 experts fire per token**: you `mmap`\n\nthe file, keep the ~13 GB dense/attention\ncore hot in page cache, and stream the cold 2-bit experts off the NVMe SSD on demand.\n\nCPU-only, ~1.9 tok/s decode / ~6 tok/s prefill. Slow, but it *runs* — a usable offline reasoning tool.\n\n- AMD Strix Halo — Ryzen AI Max+ 395 (16C/32T Zen 5), Radeon 8060S iGPU\n- 62 GiB unified LPDDR5X (~256 GB/s), NVMe SSD ~3.8 GB/s read, Fedora 43\n\nStock llama.cpp **cannot** load V4 (novel arch: lightning indexer / DeepSeek Sparse Attention,\nMLA compressor, hyper-connections). Use the antirez fork:\n\n```\ngit clone https://github.com/antirez/llama.cpp-deepseek-v4-flash llama-v4-src\ncd llama-v4-src\ncmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_OPENMP=ON -DLLAMA_CURL=OFF\ncmake --build build -j\n```\n\nCPU-only on purpose: unified memory means the iGPU adds no bandwidth for decode (it's memory/IO-bound), and it sidesteps whether the novel ops have GPU kernels.\n\n`antirez/deepseek-v4-gguf`\n\n— the `IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8`\n\nchat imatrix build (80.76 GiB):\nrouted experts 2-bit, attention/shared-experts/output at Q8. On btrfs, disable CoW first to avoid\nfragmenting the big mmap'd file:\n\n```\nmkdir -p ~/models/dsv4 && chattr +C ~/models/dsv4   # btrfs nodatacow, BEFORE downloading into it\n# download the *-IQ2XXS-*imatrix.gguf into ~/models/dsv4/\n./build/bin/llama-completion \\\n  -m ~/models/dsv4/DeepSeek-V4-Flash-IQ2XXS-...-imatrix.gguf \\\n  -t 6 \\                 # 4–6 threads is OPTIMAL. More cores are SLOWER (see below).\n  -c 4096 \\              # explicit context; too-small context => compressor-cache assert\n  --mmap \\               # REQUIRED and inverted from usual: mlock OFF. Streams experts from SSD.\n  -fa on \\               # REQUIRED with quantized KV, else llama_new_context() aborts\n  -ctk q8_0 -ctv q8_0 \\  # shrink KV to leave RAM for expert pages (MLA KV is tiny anyway)\n  -n 200 -no-cnv \\\n  -p \"Explain how mixture-of-experts models work:\"\n```\n\n**1. mmap ON, mlock OFF — the opposite of a normal local model.** You *can't* lock 81 GB into\n62 GB. mmap lets the OS cache the hot core and evict cold expert pages. mmap'd model pages are\n**file-backed = always reclaimable**, so they can never trigger a system OOM.\n\n**2. Fewer threads are faster — dramatically.** It's memory-bandwidth-bound; extra threads just\nfight over the one LPDDR5X bus and thrash the page-fault path. Measured decode (tok/s):\n\n| threads | 2 | 4 |\n6 | 8 | 16 | 24 | 32 |\n|---|---|---|---|---|---|---|---|\n| tok/s | 1.29 | 1.98 |\n1.87 | 1.81 | 1.05 | 0.70 | 0.22 |\n\n`-t 4`\n\nbeats `-t 32`\n\nby **9×**. Use `-t 6`\n\nfor a balance (best prefill, near-best decode).\n\n**3. -fa on is mandatory** with\n\n`-ctk/-ctv q8_0`\n\n(MLA attention), or context creation asserts.**4. Compressor-cache assert** `GGML_ASSERT(n_comp_visible <= n_comp_cache)`\n\n(deepseek4.cpp):\nthe DeepSeek Sparse Attention cache is sized from `n_ctx`\n\n.\n\n- Long\n*prefill batch*: chunk with`-ub 128 -b 128`\n\n. - Long\n*generation*: just pass a real`-c`\n\n(e.g. 4096). Under`llama-bench`\n\nit looks like a ~128-token cap — that's a bench artifact (tight context), not a real limit.\n\n**5. ☠️ NEVER inspect metadata with llama-gguf <model> r.** Read mode loads\n\n*every tensor as anonymous heap*— 53 GB, no mmap — which exhausted RAM+swap and triggered the global OOM killer (took down my terminal and several containers). Use\n\n`gguf_dump.py --no-tensors`\n\ninstead.**6. Blast-radius guard.** Run big loads in a swap-disabled cgroup so a surprise can't take the\ndesktop down: `systemd-run --user --scope -p MemorySwapMax=0 -- <your llama command>`\n\n.\nDo NOT add a `MemoryMax`\n\ncap — it charges the mmap page cache to the cgroup and forces\nevict+re-read thrashing (measured 211 GB of re-reads at a 50 GB cap).\n\n**MTP speculative decode**: the MTP head loads as arch`deepseek4_mtp_support`\n\nand`llama-speculative`\n\n/`-md`\n\nreject it as a standalone draft. And in an IO-bound regime it wouldn't help much anyway — verifying K draft tokens still streams K×(6×43) experts.**GPU**: deferred. Unified memory means no decode bandwidth win; possible future use is prefill only.\n\n81 GB model, 62 GiB RAM, no GPU, ~1.9 tok/s, coherent. MoE sparsity + mmap + a fast SSD makes \"bigger than RAM\" a solvable problem. Tune threads DOWN, not up.", "url": "https://wpnews.pro/news/running-deepseek-v4-flash-284b-moe-on-a-64gb-strix-halo-via-ssd-expert-streaming", "canonical_source": "https://gist.github.com/AlexsJones/9b43e7b8f3682679d17f255a3ca0d9d3", "published_at": "2026-07-05 18:49:01+00:00", "updated_at": "2026-08-04 08:02:11.238488+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "developer-tools"], "entities": ["DeepSeek-V4-Flash", "AMD Strix Halo", "Ryzen AI Max+ 395", "llama.cpp", "antirez", "btrfs", "Fedora 43"], "alternates": {"html": "https://wpnews.pro/news/running-deepseek-v4-flash-284b-moe-on-a-64gb-strix-halo-via-ssd-expert-streaming", "markdown": "https://wpnews.pro/news/running-deepseek-v4-flash-284b-moe-on-a-64gb-strix-halo-via-ssd-expert-streaming.md", "text": "https://wpnews.pro/news/running-deepseek-v4-flash-284b-moe-on-a-64gb-strix-halo-via-ssd-expert-streaming.txt", "jsonld": "https://wpnews.pro/news/running-deepseek-v4-flash-284b-moe-on-a-64gb-strix-halo-via-ssd-expert-streaming.jsonld"}}