Or: But sweetie we dont need gorgon halo, have 192gb strix halo at home
The 3-Device MoE Experiment
TL;DR
I took a Strix Halo APU (a Ryzen AI MAX+ 395 with 128 GB of unified memory and a Radeon 8060S integrated GPU) and bolted two Radeon R9700 32 GB discrete GPUs onto it through an external PCIe switch. The result is a 3-device platform: 128 GB unified + 32 GB + 32 GB, where the iGPU acts as a ~120 GB / ~256 GB/s-class device on top of two fast dedicated GPUs.
After fixing three kernel-level problems and one driver-level Oops, I benchmarked three MoE models across every device configuration and learned that:
2x R9700 is the fastest config – gpt-oss-120b crossed 100 tok/s.
- The 3-device split beats the iGPU alone but loses to 2x R9700 (the iGPU adds capacity, not throughput, for these models). Prompt processing scales with device count that’s the real reason to spread a model across GPUs.
- A community fork of llama.cpp made the biggest model runnable on the iGPU alone, a thing mainline can’t do on HIP. The 3-device split runs the model at its full native 262k context with 100% needle retrieval long-document workloads without a RAG pipeline.
- Results show that the best use of this is not as much practical as research, or possibly, this gives you a research platform mixing “different” RDNA and multiple models at once.
[!CAUTION]
DISCLAIMER read first. This is an enthusiast/experimental build on a rolling-release distro (CachyOS) with bleeding-edge AMD drivers and a brand-new GPU architecture. This of this more like “blog” than “how-to.” Expect rough edges: GPU resets, etc.
- What Was The Config
The machine (what hardware works)
Strix Halo APU: Ryzen AI MAX+ 395, 128 GB unified memory, Radeon 8060S iGPU (gfx1151) with ~112 GB exposed to the GPU as GTT. 2x AMD Radeon R9700 (gfx1201), 32 GB each, attached via an external PCIe switch.
- CachyOS, kernel 7.1.x, ROCm 7.2.4, llama.cpp built for both GPU architectures.
Note: It’s basically become impossible to find Pcie 5.0 PEX/PLX bridges, but you can still pick up PCIe 4.0 bridge cards on ebay. NOTE These are not the same as a Pcie 5.0 redriver/retimer. With the latter the platform is responsible for lane bifrucation.. i.e. you need a 16 lane slot configured for x8 x8 to get two x8 x8 slots. Here it’s a true bridge and that’s something different.
Here is a pcie 4.0 version of the PEX. Note these are different than bifrucation adapters. This is a true pcie bridge. 88XXX chipsets are gen 4, 89XXX chipsets are gen five. Also pay attention to the connectors. These are slim-sas style connectors and MCIO are modern connectors. You can sometimes find Pcie gen4 cards that have mcio connectors – those would be preferable to the connectors in this ebay listing.
The bet: people already run 35B-class MoE models on Strix Halo at speed straight from unified memory. What happens when you add two real discreete GPUs to that picture? Can you get MoE prefill on the fast discreetes and expert-weight streaming from the big unified pool?
First, I had to make it work (software edition)
This was not plug-and-play. Three separate kernel-level problems stood between me and a working 3-GPU box, each can be overcome:
Only one R9700 was visible. The second GPU’s memory BAR failed to allocate the external switch’s memory window had been consumed by the first GPU, so amdgpu
never bound the second card. The machine was effectively 1x R9700 + iGPU, which defeats the whole premise.
Fix: pci=realloc
on the kernel command line.
A load storm after that reboot (load avg ~39). Both R9700s and the switch ports had entered runtime power management and then failed to wake D3 to D0 transitions failing, devices stuck in resuming
, wedging the display path, hanging KDE, triggering a KDE restart loop plus kworker
/khugepaged
CPU storms. Not a fork bomb, but it sure felt like one.
Fix: amdgpu.runpm=0 pcie_port_pm=off
.
A kernel Oops on every GPU memory alloc/free. Signature: general protection fault
at amdgpu_vm_cpu_update
, reached through svm_range_validate_and_map
. It crashed torch and HIP, and HSA_XNACK=1
didn’t help. Root cause (from source analysis): an amdgpu.vm_size=128
override on the command line, combined with the RDNA4 GPU’s 57-bit virtual memory and the CPU’s 4-level paging, produced a pathological page-table geometry that exposed a use-after-free in the page-table allocator.
Fix: remove amdgpu.vm_size=128
(back to the default). Zero faults across the entire benchmark session afterward.
The full working command line ended up as:
quiet nowatchdog splash rw rootflags=subvol=/@ root=UUID=3b6b507e-3c7c-4010-82c7-7ac72beeb0d8 amdgpu.gttsize=114688 ttm.pages_limit=29360128 iommu=off pci=realloc amdgpu.runpm=0 pcie_port_pm=off
Phase 1: peer-to-peer baseline
Before any model benchmarks, I measured raw copy bandwidth between every device pair:
| Path | Measured | | R9700 to R9700 | ~13.4 GB/s (symmetric) | | R9700 to iGPU | ~7 GB/s | | host to R9700 | ~7 GB/s | | host to iGPU | ~80 GB/s the unified-memory win |
Two takeaways. First, the host→iGPU path at ~80 GB/s validates the whole premise: the iGPU’s unified-memory access is an order of magnitude faster than anything through the external switch. Second, the R9700 to R9700 peer-to-peer copy is bottlenecked by the external switch’s peer-to-peer fabric, not by the connection itself even a twice-as-wide connection would still be fabric-bound. If you’re planning multi-GPU inference through an external switch, that’s the number to know. This made me want to order the cheaper-anyway PCIe 4.0 PEX88xxx bridges from ebay since this is within Gen4 bandwidth. Or Maybe I have been had on my bridge and its not really gen 5 (a real possibility…)
I also confirmed RCCL all_reduce
on the two R9700s, but RCCL ships no kernels for the iGPU (gfx1151), so 3-GPU RCCL collectives aren’t possible on this box. As far as I’m aware. I spent a long time on this one. That doesn’t matter for llama.cpp, which does multi-GPU with its own backend (and no RCCL).
The benchmark matrix
All numbers are llama.cpp text-generation (tg) tok/s, full offload, on this box:
| Model | Quant | iGPU alone | 2x R9700 | 3-device | iGPU-only (EngramHalo fork) | | Qwen3.6-35B-A3B | Q8_0 | 48.8 | 72.3 | 55.5 | | | gpt-oss-120b | Q4_K_M | 56.5 | 100.4 | 67.8 | | | Qwen3.8-Flash-Next | Q4_K_M | not viable (GTT OOM) | | 26.1 / 27.8 | 24.0 / 24.3 |
Three models, three device configs, and a clear story:
2x R9700 is the fastest config. gpt-oss-120b crossed 100 tok/s; That’s a 120B-parameter MoE (5.1B active) decoding at 100 tokens/sec off two 32 GB cards. The 3-device split beats the iGPU alone (+14–26% on tg) but is slower than 2x R9700 (-23% to -33%). The iGPU holds the largest share of layers, and its ~215 GB/s effective bandwidth becomes the decode bottleneck. Verdict: the iGPU adds capacity, not throughput, for these models. Qwen3.8-Flash-Next (176.9B params, 512 experts, ~111 GiB in Q4_K_M) is the big one it only runs across devices. On the 3-device config it decodes at ~26–28 tok/s.
The prompt-processing aspect (why you spread across GPUs)
Here’s the part that surprised me most, and the strongest argument for multi-GPU. Even when text-generation is comparable, prompt processing scales with device count:
| Model | iGPU alone (pp512) | 2x R9700 (pp512) | 3-device (pp512) | | Qwen3.6-35B-A3B Q8_0 | 1182 | 2606 | 1420 | | Qwen3.8-Flash-Next Q4_K_M | 416 (fork) | | 520 |
For Qwen3.6-35B, two R9700s process prompts at 2.2× the iGPU-alone rate (2606 vs 1182 tok/s at pp512). For Flash-Next, the 3-device config prefills at 520 tok/s vs 416 on the iGPU alone. Prefill is a parallelizable, compute-heavy phase more devices genuinely helps here, even when decode is bandwidth-bound. If you care about time-to-first-token on long prompts, spreading across GPUs is the move.
This doesn’t seem like a huge uplift, but this is also way, way off the beaten path. I suspect that I could throw a lot of time at this and get similar PP speedups like we saw with gpt-oss 120b.
The iGPU-only saga
Mainline llama.cpp cannot run Qwen3.8-Flash-Next on the iGPU alone on HIP. The reason is because Flash-Next has a really nifty architectural feature (and this is why you see this model reported as 120b when it’s really ~170b): the model carries a ~33 GiB PLE n-gram embedding table designed to live in host RAM, but on HIP there’s no zero-copy mmap path so the full 111.38 GiB (including that table) gets allocated into the iGPU’s ~112 GiB GTT. Big oof for Our Halos.
On a 125 GB-RAM box that leaves almost nothing for the OS, and the box will thrash until it dropps off the network.
The fix came from a community fork: EngramHalo.cpp, branch strix-halo-qwen4exp
(commit 1423f6899). Its per-buffer-mmap keeps the PLE table on SSD as a CPU-mapped buffer instead of pushing it into GTT:
create_tensor: tensor per_layer_token_embd.weight (size = 33569 MiB) lazy read enabled
load_tensors: CPU_Mapped model buffer size = 33910.51 MiB <- PLE, SSD-backed
load_tensors: ROCm2 model buffer size = 80137.57 MiB <- main transformer, GTT
Result: iGPU-only is now viable for the biggest model on this box at 24.0–24.3 tok/s, i.e. 86–92% of the 3-device config, with the PLE served from SSD. The fork also fixes the long-context decode collapse with a sparse-attention gather and a wide top-k kernel: a 58.5k-token prompt prefilled at 287.6 tok/s and decoded at 13.1 tok/s at depth the first iGPU-only long-context run that completes on this model.
The lesson: dGPU offload isn’t just faster it’s what makes the model runnable at all. And when a mainline path is blocked by a driver limitation, a good fork with a surgical patch can unblock it.
But that wasn’t enough for me. I wanted Deep Context (I like deep context and I cannot lie.)
- Deep context: the 3-GPU config unlocks the full window
Everything above is about throughput. But there’s a second axis that matters just as much: context the model holding a lot of what you feed it and still answering questions about it.
Qwen3.8-Flash-Next advertises a 262k-token native context window, and the 3-device split was the best way I found to get there. On the iGPU-only path (EngramHalo fork) I’ve tested it at 64k context with 8-bit KV that works, but the KV cache competes with the weights for the same ~112 GB GTT, so the window is capped in practice. The 3-device split changes the math: the two R9700s hold a big share of the layers, freeing GTT headroom for the KV cache. That’s the structural reason the full native window is reachable here and not on the iGPU alone.
128k and 262k: I didn’t expect this to Actually Work ™
I ran two deep-context passes on the 3-device config with 8-bit KV (-ctk q8_0 -ctv q8_0
), each with a real comprehension question embedded deep in the prompt:
| Test | Prompt length | Prefill | Decode @ depth | Coherence | | deep 128k | 126,749 tokens | 222.4 t/s | 9.8 t/s | coherent correctly answered a question about the capital of Eldoria embedded at depth | | deep 262k (full native context) | 253,479 tokens | 213.0 t/s | 5.5 t/s | coherent answered an espresso question at the very end of the 253k prompt |
The 262k run is the headline: the model ran at its full native context, the KV cache fit comfortably across the dGPU VRAM plus iGPU GTT (MemFree held steady at ~61–64 GB), and it stayed coherent all the way out correctly answering a question whose answer sat at the very end of a 253k-token prompt. No context shrinking, no crash, no collapse.
Needle in a haystack: 4/4 retrieval
To test retrieval rather than just coherence, I used the classic needle-in-a-haystack setup a unique secret code buried in filler prose, with the model asked to quote it. Four runs, four different context lengths and depths:
| Run |
Prompt length |
Depth |
Prefill |
Decode |
Code found |
| needle 110k |
~110k tokens |
25% |
224.8 t/s |
11.2 t/s |
8472-QUX-ALPHA |
| needle 110k |
~110k tokens |
50% |
225.9 t/s |
11.1 t/s |
1937-ZETA-BRAVO |
| needle 110k |
~110k tokens |
75% |
224.4 t/s |
11.2 t/s |
5501-OMEGA-DELTA |
| needle 220k |
~218.6k tokens |
50% |
211.5 t/s |
6.1 t/s |
9264-RHO-SIGMA |
4/4 retrieval at every tested depth, including at 218.6k tokens. The model explicitly quoted the exact secret code from the haystack in all four runs.
Prefill does not collapse with context it stays at ~213–226 t/s across all context sizes, from 110k to 253k tokens, thanks to the EngramHalo fork’s ROCm kernels (sparse-attention gather + wide top-k).
Decode scales down gracefully with context length: 9.8 t/s at 128k, 5.5 t/s at the full 262k window a smooth curve, not a cliff.
-Decode Performance Optimizations Given I could reach ~30t/s with this model, but without using disk-based n-grams, there is likely a lot of room for performance optimization
Retrieval is 100% at every depth tested, up to 218.6k tokens.
This is the practical payoff of the 3-GPU config: not just throughput, but running the model at its native context with real retrieval capability something the iGPU-only path can’t deliver.
A 120B-class MoE holding 250k+ tokens of context with working retrieval is genuinely useful on a desktop-class box. Long-document analysis, “RAG without RAG” (paste the whole source into context and ask) no external retrieval pipeline needed. Whole-book-in-context workloads (a 250k-token window is roughly a full novel).
Still, there are a lot of rough edges. MoE models that are a bit less bleeding edge will run faster AND have deeper context in this setup. Remember that RCCL is not a great option here for 3 GPUs, either.
- But Why!??!!!?
The platform question. Strix Halo is genuinely interesting because it’s a huge unified-memory pool with a real GPU attached. The community already runs 35B MoE models on it at speed. The open question was: what does the picture look like when it’s not a unified platform anymore 128 + 32 + 32, three devices with wildly different memory bandwidths? Nobody had benchmarked that mix on this APU with these discreetes.
On paper this looks awesome because we know we’ve got 192gb devices coming, and we wish that Strix Halo had even faster prompt processing. This device is (the best of?|the worst of?) both worlds.
In other words mixture-of-experts is bandwidth-bound at decode and compute-bound at prefill. That asymmetry means a heterogeneous setup might be a win: fast discreetes for prefill, the big unified pool for capacity. The data says the discreetes win for throughput, and the iGPU earns its place by making models fit that otherwise wouldn’t plus the prefill scaling is real.
The engineering story. Getting three AMD GPUs of two different architectures working together on a rolling distro was not fun: a BAR allocation failure, sabotaged by runtime PM, and a kernel use-after-free in the SVM path. That’s the kind of thing these forums are for Oh, and I resisted so much so much clickbait nonsense opportunity here.
- Practical applications
A 120B MoE at 100 tok/s on two 32 GB cards is a genuinely usable local-inference box for coding, agent loops, or anything token-hungry. That’s the standout result. More to come there. The iGPU is a capacity multiplier, not a speedup. Use it to fit models that don’t fit in dGPU VRAM (the 176.9B Flash-Next runs at ~26 tok/s across all three devices, and at ~24 tok/s on the iGPU alone with the fork). Don’t expect it to make small models faster. If time-to-first-token on long prompts matters, spread the model across GPUs. Prefill scales with device count even when decode doesn’t. The 3-device split unlocks the model’s full native context. With 8-bit KV it runs Qwen3.8-Flash-Next at its full 262k window with 100% needle retrieval long-document and whole-book-in-context workloads are a single prompt, no RAG pipeline needed (see Section 2). External PCIe switches are a real bottleneck for GPU↔GPU traffic. If you need tight multi-GPU collectives, use a platform with native GPU-to-GPU paths. For llama.cpp-style layer-split inference the switch is fine. On HIP, the iGPU can’t read dGPU memory (the runtime advertises peer access that faults in practice). Device order matters: iGPU first in the pipeline, or prefill crashes.
- How to reproduce
I’m not going to walk through every test configuration (P2P microbenchmarks, RCCL sweeps, fork archaeology, etc) because that’s not important.
The source of truth for the kernel command line is /etc/default/limine
**never hand-edit **/boot/limine.conf
, it’s auto-generated:
sudo cp /etc/default/limine /etc/default/limine.bak-$(date +%Y%m%d-%H%M%S)
sudo nano /etc/default/limine # edit the KERNEL_CMDLINE[default]+="..." line
sudo limine-update
sudo reboot
Three fixes, in order of discovery:
Second GPU invisible Just add pci=realloc
. Symptom: lspci
/rocm-smi
show only one R9700; dmesg
shows BAR allocation failures (Problem resizing BAR0 (-22)
/ can't assign ... no space
). The external switch’s memory window was consumed by the first GPU.
Load storm after reboot (load ~30–40) Just add amdgpu.runpm=0 pcie_port_pm=off
. Symptom: kwin_wayland
spinning, KDE restart loop, dmesg
showing Unable to change power state from D3hot to D0
and devices stuck in resuming
. The GPUs/switch ports entered runtime PM and failed to wake.
Kernel Oops on any GPU alloc/free Just remove amdgpu.vm_size=128
if you have it. Symptom: general protection fault
at amdgpu_vm_cpu_update
via svm_range_validate_and_map
, crashing torch and HIP. On RDNA4 with 4-level paging this override produces pathological page tables and a use-after-free. This may have been an artifact on my Framework Desktop system from earlier experiments, or something from one of my earlier guides.
Verify:
cat /proc/cmdline # all three params present, no vm_size override
/opt/rocm/bin/rocm-smi --showproductname # 3 devices: 2x R9700 + 8060S
cat /sys/module/amdgpu/parameters/runpm # 0
dmesg | grep -c "general protection fault" # 0
A benign boot-time sdma0 ring timeout
that self-recovers is maybe normal if you see it in dmesg
5.2 Build llama.cpp for both GPU architectures
The R9700s are gfx1201 and the iGPU is gfx1151 you need both kernels in one build:
cd ~
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
export ROCM_PATH=/opt/rocm # clang HIP autodetect needs this
cmake -U GPU_TARGETS
cmake -B build -DCMAKE_BUILD_TYPE=Release \
-DGGML_HIP=ON -DGGML_HIP_NO_VMM=ON -DGGML_HIP_RCCL=OFF \
-DAMDGPU_TARGETS="gfx1201;gfx1151"
cmake --build build --target llama-cli llama-bench llama-server -j32
Those two gotchas are the ones that cost me real time: ROCM_PATH
must be exported or the build can’t find the HIP toolchain, and a stale GPU_TARGETS
in the CMakeCache silently overrides AMDGPU_TARGETS
the build looks like it targets both but only compiles for one arch. Purge it with cmake -U GPU_TARGETS
.
5.3 Run on the 3-device config (iGPU first!)
The device order is mandatory: iGPU first. ROCm2/ROCm0/ROCm1
. dGPU-first crashes on prefill with a gfxhub page fault on HIP the iGPU cannot read memory allocated on a dGPU, even though the runtime reports peer access. This is a ROCm/driver-level limitation, not a llama.cpp bug. The low-hanging fruit optimization here is to then patch to make sure the prefill task happens asymmetrically – send to dGPUs – but I didn’t explicitly do that for these results.
HIP_VISIBLE_DEVICES=0,1,2 \
build/bin/llama-bench -m /path/to/model.gguf \
--device ROCm2/ROCm0/ROCm1 -sm layer \
-p 512 -n 256 -r 3 -ngl 999
LLAMA_ATTN_ROT_DISABLE=1 \
HIP_VISIBLE_DEVICES=0,1,2 \
build/bin/llama-cli -m /path/to/model.gguf \
--device ROCm2,ROCm0,ROCm1 -sm layer -ngl 999 -p "Hello"
Notes:
-sm layer
(pipeline split) is required -sm row
(tensor split) fails to load MoE models in llama.cpp.
LLAMA_ATTN_ROT_DISABLE=1
is required for Qwen3.8-Flash-Next (QSA/quantized-KV Hadamard fix).
HIP_VISIBLE_DEVICES=0,1,2
without --device
silently drops the iGPU always pass the device list explicitly.
- For Qwen3.8-Flash-Next, the first load could take a long time (25K tensors, single-threaded ). 300 s timeout is likely not enough.
-c 262144
works on the 3-device config with 8-bit KV (-ctk q8_0 -ctv q8_0
) the full native window fits across dGPU VRAM + iGPU GTT (MemFree stays ~61–64 GB). See 5.6 for the deep-context recipe.
5.4 Run on 2x R9700 (the fastest config)
HIP_VISIBLE_DEVICES=0,1 \
build/bin/llama-bench -m /path/to/model.gguf \
--device ROCm0/ROCm1 -sm layer \
-p 128 -n 256 -r 3 -ngl 999
gpt-oss-120b Q4_K_M is a tight fit here (~63.4 of 64 GiB) fine at 4K context, don’t push the context window.
5.5 The EngramHalo fork + iGPU-only (for the big MoE)
git clone -b strix-halo-qwen4exp https://github.com/Aristo94/EngramHalo.cpp ~/llama-engramhalo
cd ~/llama-engramhalo
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_HIP=ON \
-DAMDGPU_TARGETS="gfx1201;gfx1151" -DGGML_HIP_NO_VMM=ON \
-DCMAKE_C_COMPILER=/opt/rocm/bin/hipcc -DCMAKE_CXX_COMPILER=/opt/rocm/bin/hipcc
cmake --build build --target llama-cli llama-bench -j32
Run Qwen3.8-Flash-Next on the iGPU alone at 64K context with 8-bit KV the PLE n-gram table stays on SSD:
HSA_ENABLE_SDMA=0 HSA_XNACK=1 ROCBLAS_USE_HIPBLASLT=1 \
build/bin/llama-cli -m /path/to/Qwen3.8-Flash-Next-Q4_K_M.gguf \
--device ROCm2 -ngl 999 -fa on -ctk q8_0 -ctv q8_0 \
-c 65536 -n 8 -p "Hello" --no-display-prompt -lm mmap
Expected: ~24 tok/s, MemFree stable at ~40 GB (the 33.9 GiB PLE lives on SSD, only ~1–2 GiB resident), and long-context decode that holds up (~13 tok/s at 58.5k depth).
5.6 Deep context on the 3-device config (the full 262k window)
The same 3-device command from 5.3, with the context window opened up and 8-bit KV so the cache fits:
LLAMA_ATTN_ROT_DISABLE=1 \
HSA_ENABLE_SDMA=0 HSA_XNACK=1 ROCBLAS_USE_HIPBLASLT=1 \
HIP_VISIBLE_DEVICES=0,1,2 \
build/bin/llama-cli -m /path/to/Qwen3.8-Flash-Next-Q4_K_M.gguf \
--device ROCm2,ROCm0,ROCm1 -sm layer -ngl 999 -fa on \
-ctk q8_0 -ctv q8_0 -c 262144 -st -p "..." --no-display-prompt -lm mmap
-c 262144
with 8-bit KV: full native context, MemFree holds ~61–64 GB. Prefill stays ~213–226 t/s at any context size; decode is ~9.8 t/s at 128k, ~5.5 t/s at 262k.
-st
(single-turn) avoids the interactive REPL runaway I hit on the first 128k attempt use it for clean exits.
Closing
This box is the most fun I’ve had with a GPU in a while, and also the most debugging. The headline is simple: two R9700s make a Strix Halo genuinely fast at MoE inference (100 tok/s on gpt-oss-120b), the iGPU makes the big models fit, prefill scales with device count and the 3-device config runs the model at its full native 262k context with 100% retrieval. The asterisks are the external-switch P2P bottleneck, the iGPU-can’t-read-dGPU-memory limitation, and the fact that a bleeding-edge AMD setup on a rolling distro will occasionally need a power button and a prayer.
If you build something like this, I’d love to hear how it goes. And if you’re the maintainer of a llama.cpp fork that makes 176B-parameter models run on an iGPU thank you.
I think there is a lot of performance left on the table, but it’s stable at least.