# Running DeepSeek-V4-Flash (284B MoE) on a 64GB Strix Halo via SSD expert-streaming

> Source: <https://gist.github.com/AlexsJones/9b43e7b8f3682679d17f255a3ca0d9d3>
> Published: 2026-07-05 18:49:01+00:00

A 284B-parameter mixture-of-experts model, quantized to 2-bit, is **81 GB** — bigger than the
62 GiB of RAM on an AMD Strix Halo (Ryzen AI Max+ 395). It still runs at **~1.9 tok/s, coherent**,
because only **6 of 256 experts fire per token**: you `mmap`

the file, keep the ~13 GB dense/attention
core hot in page cache, and stream the cold 2-bit experts off the NVMe SSD on demand.

CPU-only, ~1.9 tok/s decode / ~6 tok/s prefill. Slow, but it *runs* — a usable offline reasoning tool.

- AMD Strix Halo — Ryzen AI Max+ 395 (16C/32T Zen 5), Radeon 8060S iGPU
- 62 GiB unified LPDDR5X (~256 GB/s), NVMe SSD ~3.8 GB/s read, Fedora 43

Stock llama.cpp **cannot** load V4 (novel arch: lightning indexer / DeepSeek Sparse Attention,
MLA compressor, hyper-connections). Use the antirez fork:

```
git clone https://github.com/antirez/llama.cpp-deepseek-v4-flash llama-v4-src
cd llama-v4-src
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_OPENMP=ON -DLLAMA_CURL=OFF
cmake --build build -j
```

CPU-only on purpose: unified memory means the iGPU adds no bandwidth for decode (it's memory/IO-bound), and it sidesteps whether the novel ops have GPU kernels.

`antirez/deepseek-v4-gguf`

— the `IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8`

chat imatrix build (80.76 GiB):
routed experts 2-bit, attention/shared-experts/output at Q8. On btrfs, disable CoW first to avoid
fragmenting the big mmap'd file:

```
mkdir -p ~/models/dsv4 && chattr +C ~/models/dsv4   # btrfs nodatacow, BEFORE downloading into it
# download the *-IQ2XXS-*imatrix.gguf into ~/models/dsv4/
./build/bin/llama-completion \
  -m ~/models/dsv4/DeepSeek-V4-Flash-IQ2XXS-...-imatrix.gguf \
  -t 6 \                 # 4–6 threads is OPTIMAL. More cores are SLOWER (see below).
  -c 4096 \              # explicit context; too-small context => compressor-cache assert
  --mmap \               # REQUIRED and inverted from usual: mlock OFF. Streams experts from SSD.
  -fa on \               # REQUIRED with quantized KV, else llama_new_context() aborts
  -ctk q8_0 -ctv q8_0 \  # shrink KV to leave RAM for expert pages (MLA KV is tiny anyway)
  -n 200 -no-cnv \
  -p "Explain how mixture-of-experts models work:"
```

**1. mmap ON, mlock OFF — the opposite of a normal local model.** You *can't* lock 81 GB into
62 GB. mmap lets the OS cache the hot core and evict cold expert pages. mmap'd model pages are
**file-backed = always reclaimable**, so they can never trigger a system OOM.

**2. Fewer threads are faster — dramatically.** It's memory-bandwidth-bound; extra threads just
fight over the one LPDDR5X bus and thrash the page-fault path. Measured decode (tok/s):

| threads | 2 | 4 |
6 | 8 | 16 | 24 | 32 |
|---|---|---|---|---|---|---|---|
| tok/s | 1.29 | 1.98 |
1.87 | 1.81 | 1.05 | 0.70 | 0.22 |

`-t 4`

beats `-t 32`

by **9×**. Use `-t 6`

for a balance (best prefill, near-best decode).

**3. -fa on is mandatory** with

`-ctk/-ctv q8_0`

(MLA attention), or context creation asserts.**4. Compressor-cache assert** `GGML_ASSERT(n_comp_visible <= n_comp_cache)`

(deepseek4.cpp):
the DeepSeek Sparse Attention cache is sized from `n_ctx`

.

- Long
*prefill batch*: chunk with`-ub 128 -b 128`

. - Long
*generation*: just pass a real`-c`

(e.g. 4096). Under`llama-bench`

it looks like a ~128-token cap — that's a bench artifact (tight context), not a real limit.

**5. ☠️ NEVER inspect metadata with llama-gguf <model> r.** Read mode loads

*every tensor as anonymous heap*— 53 GB, no mmap — which exhausted RAM+swap and triggered the global OOM killer (took down my terminal and several containers). Use

`gguf_dump.py --no-tensors`

instead.**6. Blast-radius guard.** Run big loads in a swap-disabled cgroup so a surprise can't take the
desktop down: `systemd-run --user --scope -p MemorySwapMax=0 -- <your llama command>`

.
Do NOT add a `MemoryMax`

cap — it charges the mmap page cache to the cgroup and forces
evict+re-read thrashing (measured 211 GB of re-reads at a 50 GB cap).

**MTP speculative decode**: the MTP head loads as arch`deepseek4_mtp_support`

and`llama-speculative`

/`-md`

reject it as a standalone draft. And in an IO-bound regime it wouldn't help much anyway — verifying K draft tokens still streams K×(6×43) experts.**GPU**: deferred. Unified memory means no decode bandwidth win; possible future use is prefill only.

81 GB model, 62 GiB RAM, no GPU, ~1.9 tok/s, coherent. MoE sparsity + mmap + a fast SSD makes "bigger than RAM" a solvable problem. Tune threads DOWN, not up.