Franken Strix Halo: 2x R9700s + 128gb Strix Halo unified memory

An enthusiast experiment combined a Strix Halo APU (Ryzen AI MAX+ 395 with 128 GB unified memory and Radeon 8060S iGPU) with two Radeon R9700 32 GB discrete GPUs via an external PCIe switch, achieving over 100 tok/s on gpt-oss-120b in the fastest configuration. The 3-device split outperformed the iGPU alone but lost to the dual discrete GPUs, and the setup enabled full 262k context with 100% needle retrieval without RAG. The build required kernel fixes and used CachyOS with ROCm 7.2.4.

Or: But sweetie we dont need gorgon halo, have 192gb strix halo at home The 3-Device MoE Experiment TL;DR I took a Strix Halo APU a Ryzen AI MAX+ 395 with 128 GB of unified memory and a Radeon 8060S integrated GPU and bolted two Radeon R9700 32 GB discrete GPUs onto it through an external PCIe switch. The result is a 3-device platform: 128 GB unified + 32 GB + 32 GB, where the iGPU acts as a ~120 GB / ~256 GB/s-class device on top of two fast dedicated GPUs. After fixing three kernel-level problems and one driver-level Oops, I benchmarked three MoE models across every device configuration and learned that: 2x R9700 is the fastest config – gpt-oss-120b crossed 100 tok/s . - The 3-device split beats the iGPU alone but loses to 2x R9700 the iGPU adds capacity , not throughput, for these models . Prompt processing scales with device count that’s the real reason to spread a model across GPUs. - A community fork of llama.cpp made the biggest model runnable on the iGPU alone , a thing mainline can’t do on HIP. The 3-device split runs the model at its full native 262k context with 100% needle retrieval long-document workloads without a RAG pipeline. - Results show that the best use of this is not as much practical as research, or possibly, this gives you a research platform mixing “different” RDNA and multiple models at once. CAUTION DISCLAIMER read first. This is an enthusiast/experimental build on a rolling-release distro CachyOS with bleeding-edge AMD drivers and a brand-new GPU architecture. This of this more like “blog” than “how-to.” Expect rough edges: GPU resets, etc. 1. What Was The Config The machine what hardware works Strix Halo APU : Ryzen AI MAX+ 395, 128 GB unified memory, Radeon 8060S iGPU gfx1151 with ~112 GB exposed to the GPU as GTT. 2x AMD Radeon R9700 gfx1201 , 32 GB each, attached via an external PCIe switch . - CachyOS, kernel 7.1.x, ROCm 7.2.4, llama.cpp built for both GPU architectures. Note: It’s basically become impossible to find Pcie 5.0 PEX/PLX bridges, but you can still pick up PCIe 4.0 bridge cards on ebay. NOTE These are not the same as a Pcie 5.0 redriver/retimer. With the latter the platform is responsible for lane bifrucation.. i.e. you need a 16 lane slot configured for x8 x8 to get two x8 x8 slots. Here it’s a true bridge and that’s something different. Here is a pcie 4.0 version of the PEX. Note these are different than bifrucation adapters. This is a true pcie bridge . 88XXX chipsets are gen 4, 89XXX chipsets are gen five. Also pay attention to the connectors. These are slim-sas style connectors and MCIO are modern connectors. You can sometimes find Pcie gen4 cards that have mcio connectors – those would be preferable to the connectors in this ebay listing. The bet: people already run 35B-class MoE models on Strix Halo at speed straight from unified memory. What happens when you add two real discreete GPUs to that picture? Can you get MoE prefill on the fast discreetes and expert-weight streaming from the big unified pool? First, I had to make it work software edition This was not plug-and-play. Three separate kernel-level problems stood between me and a working 3-GPU box, each can be overcome: - Only one R9700 was visible. The second GPU’s memory BAR failed to allocate the external switch’s memory window had been consumed by the first GPU, so amdgpu never bound the second card. The machine was effectively 1x R9700 + iGPU, which defeats the whole premise. Fix: pci=realloc on the kernel command line. - A load storm after that reboot load avg ~39 . Both R9700s and the switch ports had entered runtime power management and then failed to wake D3 to D0 transitions failing, devices stuck in resuming , wedging the display path, hanging KDE, triggering a KDE restart loop plus kworker / khugepaged CPU storms. Not a fork bomb, but it sure felt like one. Fix: amdgpu.runpm=0 pcie port pm=off . - A kernel Oops on every GPU memory alloc/free. Signature: general protection fault at amdgpu vm cpu update , reached through svm range validate and map . It crashed torch and HIP, and HSA XNACK=1 didn’t help. Root cause from source analysis : an amdgpu.vm size=128 override on the command line, combined with the RDNA4 GPU’s 57-bit virtual memory and the CPU’s 4-level paging, produced a pathological page-table geometry that exposed a use-after-free in the page-table allocator. Fix: remove amdgpu.vm size=128 back to the default . Zero faults across the entire benchmark session afterward. The full working command line ended up as: quiet nowatchdog splash rw rootflags=subvol=/@ root=UUID=3b6b507e-3c7c-4010-82c7-7ac72beeb0d8 amdgpu.gttsize=114688 ttm.pages limit=29360128 iommu=off pci=realloc amdgpu.runpm=0 pcie port pm=off Phase 1: peer-to-peer baseline Before any model benchmarks, I measured raw copy bandwidth between every device pair: | Path | Measured | | R9700 to R9700 | ~13.4 GB/s symmetric | | R9700 to iGPU | ~7 GB/s | | host to R9700 | ~7 GB/s | | host to iGPU | ~80 GB/s the unified-memory win | Two takeaways. First, the host→iGPU path at ~80 GB/s validates the whole premise: the iGPU’s unified-memory access is an order of magnitude faster than anything through the external switch. Second, the R9700 to R9700 peer-to-peer copy is bottlenecked by the external switch’s peer-to-peer fabric , not by the connection itself even a twice-as-wide connection would still be fabric-bound. If you’re planning multi-GPU inference through an external switch, that’s the number to know. This made me want to order the cheaper-anyway PCIe 4.0 PEX88xxx bridges from ebay since this is within Gen4 bandwidth. Or Maybe I have been had on my bridge and its not really gen 5 a real possibility… I also confirmed RCCL all reduce on the two R9700s, but RCCL ships no kernels for the iGPU gfx1151 , so 3-GPU RCCL collectives aren’t possible on this box. As far as I’m aware. I spent a long time on this one. That doesn’t matter for llama.cpp, which does multi-GPU with its own backend and no RCCL . The benchmark matrix All numbers are llama.cpp text-generation tg tok/s , full offload, on this box: | Model | Quant | iGPU alone | 2x R9700 | 3-device | iGPU-only EngramHalo fork | | Qwen3.6-35B-A3B | Q8 0 | 48.8 | 72.3 | 55.5 | | | gpt-oss-120b | Q4 K M | 56.5 | 100.4 | 67.8 | | | Qwen3.8-Flash-Next | Q4 K M | not viable GTT OOM | | 26.1 / 27.8 | 24.0 / 24.3 | Three models, three device configs, and a clear story: 2x R9700 is the fastest config. gpt-oss-120b crossed 100 tok/s; That’s a 120B-parameter MoE 5.1B active decoding at 100 tokens/sec off two 32 GB cards. The 3-device split beats the iGPU alone +14–26% on tg but is slower than 2x R9700 -23% to -33% . The iGPU holds the largest share of layers, and its ~215 GB/s effective bandwidth becomes the decode bottleneck. Verdict: the iGPU adds capacity, not throughput , for these models. Qwen3.8-Flash-Next 176.9B params, 512 experts, ~111 GiB in Q4 K M is the big one it only runs across devices. On the 3-device config it decodes at ~26–28 tok/s. The prompt-processing aspect why you spread across GPUs Here’s the part that surprised me most, and the strongest argument for multi-GPU. Even when text-generation is comparable, prompt processing scales with device count : | Model | iGPU alone pp512 | 2x R9700 pp512 | 3-device pp512 | | Qwen3.6-35B-A3B Q8 0 | 1182 | 2606 | 1420 | | Qwen3.8-Flash-Next Q4 K M | 416 fork | | 520 | For Qwen3.6-35B, two R9700s process prompts at 2.2× the iGPU-alone rate 2606 vs 1182 tok/s at pp512 . For Flash-Next, the 3-device config prefills at 520 tok/s vs 416 on the iGPU alone. Prefill is a parallelizable, compute-heavy phase more devices genuinely helps here, even when decode is bandwidth-bound. If you care about time-to-first-token on long prompts, spreading across GPUs is the move. This doesn’t seem like a huge uplift, but this is also way, way off the beaten path. I suspect that I could throw a lot of time at this and get similar PP speedups like we saw with gpt-oss 120b. The iGPU-only saga Mainline llama.cpp cannot run Qwen3.8-Flash-Next on the iGPU alone on HIP. The reason is because Flash-Next has a really nifty architectural feature and this is why you see this model reported as 120b when it’s really ~170b : the model carries a ~33 GiB PLE n-gram embedding table designed to live in host RAM, but on HIP there’s no zero-copy mmap path so the full 111.38 GiB including that table gets allocated into the iGPU’s ~112 GiB GTT. Big oof for Our Halos. On a 125 GB-RAM box that leaves almost nothing for the OS, and the box will thrash until it dropps off the network. The fix came from a community fork: EngramHalo.cpp https://github.com/Aristo94/EngramHalo.cpp , branch strix-halo-qwen4exp commit 1423f6899 . Its per-buffer-mmap loader keeps the PLE table on SSD as a CPU-mapped buffer instead of pushing it into GTT: create tensor: tensor per layer token embd.weight size = 33569 MiB lazy read enabled load tensors: CPU Mapped model buffer size = 33910.51 MiB <- PLE, SSD-backed load tensors: ROCm2 model buffer size = 80137.57 MiB <- main transformer, GTT Result: iGPU-only is now viable for the biggest model on this box at 24.0–24.3 tok/s, i.e. 86–92% of the 3-device config, with the PLE served from SSD. The fork also fixes the long-context decode collapse with a sparse-attention gather and a wide top-k kernel: a 58.5k-token prompt prefilled at 287.6 tok/s and decoded at 13.1 tok/s at depth the first iGPU-only long-context run that completes on this model. The lesson: dGPU offload isn’t just faster it’s what makes the model runnable at all. And when a mainline path is blocked by a driver limitation, a good fork with a surgical loader patch can unblock it. But that wasn’t enough for me. I wanted Deep Context I like deep context and I cannot lie. 2. Deep context: the 3-GPU config unlocks the full window Everything above is about throughput. But there’s a second axis that matters just as much: context the model holding a lot of what you feed it and still answering questions about it. Qwen3.8-Flash-Next advertises a 262k-token native context window, and the 3-device split was the best way I found to get there. On the iGPU-only path EngramHalo fork I’ve tested it at 64k context with 8-bit KV that works, but the KV cache competes with the weights for the same ~112 GB GTT, so the window is capped in practice. The 3-device split changes the math: the two R9700s hold a big share of the layers, freeing GTT headroom for the KV cache. That’s the structural reason the full native window is reachable here and not on the iGPU alone. 128k and 262k: I didn’t expect this to Actually Work ™ I ran two deep-context passes on the 3-device config with 8-bit KV -ctk q8 0 -ctv q8 0 , each with a real comprehension question embedded deep in the prompt: | Test | Prompt length | Prefill | Decode @ depth | Coherence | | deep 128k | 126,749 tokens | 222.4 t/s | 9.8 t/s | coherent correctly answered a question about the capital of Eldoria embedded at depth | | deep 262k full native context | 253,479 tokens | 213.0 t/s | 5.5 t/s | coherent answered an espresso question at the very end of the 253k prompt | The 262k run is the headline: the model ran at its full native context , the KV cache fit comfortably across the dGPU VRAM plus iGPU GTT MemFree held steady at ~61–64 GB , and it stayed coherent all the way out correctly answering a question whose answer sat at the very end of a 253k-token prompt. No context shrinking, no crash, no collapse. Needle in a haystack: 4/4 retrieval To test retrieval rather than just coherence, I used the classic needle-in-a-haystack setup a unique secret code buried in filler prose, with the model asked to quote it. Four runs, four different context lengths and depths: | Run | Prompt length | Depth | Prefill | Decode | Code found | | needle 110k | ~110k tokens | 25% | 224.8 t/s | 11.2 t/s | 8472-QUX-ALPHA | | needle 110k | ~110k tokens | 50% | 225.9 t/s | 11.1 t/s | 1937-ZETA-BRAVO | | needle 110k | ~110k tokens | 75% | 224.4 t/s | 11.2 t/s | 5501-OMEGA-DELTA | | needle 220k | ~218.6k tokens | 50% | 211.5 t/s | 6.1 t/s | 9264-RHO-SIGMA | 4/4 retrieval at every tested depth, including at 218.6k tokens. The model explicitly quoted the exact secret code from the haystack in all four runs. - Prefill does not collapse with context it stays at ~213–226 t/s across all context sizes, from 110k to 253k tokens, thanks to the EngramHalo fork’s ROCm kernels sparse-attention gather + wide top-k . - Decode scales down gracefully with context length: 9.8 t/s at 128k, 5.5 t/s at the full 262k window a smooth curve, not a cliff. - Decode Performance Optimizations Given I could reach ~30t/s with this model, but without using disk-based n-grams, there is likely a lot of room for performance optimization - Retrieval is 100% at every depth tested, up to 218.6k tokens. This is the practical payoff of the 3-GPU config: not just throughput, but running the model at its native context with real retrieval capability something the iGPU-only path can’t deliver. A 120B-class MoE holding 250k+ tokens of context with working retrieval is genuinely useful on a desktop-class box. Long-document analysis, “RAG without RAG” paste the whole source into context and ask no external retrieval pipeline needed. Whole-book-in-context workloads a 250k-token window is roughly a full novel . Still, there are a lot of rough edges. MoE models that are a bit less bleeding edge will run faster AND have deeper context in this setup. Remember that RCCL is not a great option here for 3 GPUs, either. 3. But Why ?? ? The platform question. Strix Halo is genuinely interesting because it’s a huge unified-memory pool with a real GPU attached. The community already runs 35B MoE models on it at speed. The open question was: what does the picture look like when it’s not a unified platform anymore 128 + 32 + 32, three devices with wildly different memory bandwidths? Nobody had benchmarked that mix on this APU with these discreetes. On paper this looks awesome because we know we’ve got 192gb devices coming, and we wish that Strix Halo had even faster prompt processing. This device is the best of?|the worst of? both worlds. In other words mixture-of-experts is bandwidth-bound at decode and compute-bound at prefill. That asymmetry means a heterogeneous setup might be a win: fast discreetes for prefill, the big unified pool for capacity. The data says the discreetes win for throughput, and the iGPU earns its place by making models fit that otherwise wouldn’t plus the prefill scaling is real. The engineering story. Getting three AMD GPUs of two different architectures working together on a rolling distro was not fun: a BAR allocation failure, sabotaged by runtime PM, and a kernel use-after-free in the SVM path. That’s the kind of thing these forums are for Oh, and I resisted so much so much clickbait nonsense opportunity here. 4. Practical applications A 120B MoE at 100 tok/s on two 32 GB cards is a genuinely usable local-inference box for coding, agent loops, or anything token-hungry. That’s the standout result. More to come there. The iGPU is a capacity multiplier, not a speedup. Use it to fit models that don’t fit in dGPU VRAM the 176.9B Flash-Next runs at ~26 tok/s across all three devices, and at ~24 tok/s on the iGPU alone with the fork . Don’t expect it to make small models faster. If time-to-first-token on long prompts matters, spread the model across GPUs. Prefill scales with device count even when decode doesn’t. The 3-device split unlocks the model’s full native context. With 8-bit KV it runs Qwen3.8-Flash-Next at its full 262k window with 100% needle retrieval long-document and whole-book-in-context workloads are a single prompt, no RAG pipeline needed see Section 2 . External PCIe switches are a real bottleneck for GPU↔GPU traffic. If you need tight multi-GPU collectives, use a platform with native GPU-to-GPU paths. For llama.cpp-style layer-split inference the switch is fine. On HIP, the iGPU can’t read dGPU memory the runtime advertises peer access that faults in practice . Device order matters: iGPU first in the pipeline, or prefill crashes. 5. How to reproduce I’m not going to walk through every test configuration P2P microbenchmarks, RCCL sweeps, fork archaeology, etc because that’s not important. The source of truth for the kernel command line is /etc/default/limine never hand-edit /boot/limine.conf , it’s auto-generated: sudo cp /etc/default/limine /etc/default/limine.bak-$ date +%Y%m%d-%H%M%S sudo nano /etc/default/limine edit the KERNEL CMDLINE default +="..." line sudo limine-update sudo reboot Three fixes, in order of discovery: Second GPU invisible Just add pci=realloc . Symptom: lspci / rocm-smi show only one R9700; dmesg shows BAR allocation failures Problem resizing BAR0 -22 / can't assign ... no space . The external switch’s memory window was consumed by the first GPU. Load storm after reboot load ~30–40 Just add amdgpu.runpm=0 pcie port pm=off . Symptom: kwin wayland spinning, KDE restart loop, dmesg showing Unable to change power state from D3hot to D0 and devices stuck in resuming . The GPUs/switch ports entered runtime PM and failed to wake. Kernel Oops on any GPU alloc/free Just remove amdgpu.vm size=128 if you have it. Symptom: general protection fault at amdgpu vm cpu update via svm range validate and map , crashing torch and HIP. On RDNA4 with 4-level paging this override produces pathological page tables and a use-after-free. This may have been an artifact on my Framework Desktop system from earlier experiments, or something from one of my earlier guides. Verify: cat /proc/cmdline all three params present, no vm size override /opt/rocm/bin/rocm-smi --showproductname 3 devices: 2x R9700 + 8060S cat /sys/module/amdgpu/parameters/runpm 0 dmesg | grep -c "general protection fault" 0 A benign boot-time sdma0 ring timeout that self-recovers is maybe normal if you see it in dmesg 5.2 Build llama.cpp for both GPU architectures The R9700s are gfx1201 and the iGPU is gfx1151 you need both kernels in one build: cd ~ git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp export ROCM PATH=/opt/rocm clang HIP autodetect needs this If you previously configured a single-arch build, purge the stale cache var first: cmake -U GPU TARGETS cmake -B build -DCMAKE BUILD TYPE=Release \ -DGGML HIP=ON -DGGML HIP NO VMM=ON -DGGML HIP RCCL=OFF \ -DAMDGPU TARGETS="gfx1201;gfx1151" cmake --build build --target llama-cli llama-bench llama-server -j32 Those two gotchas are the ones that cost me real time: ROCM PATH must be exported or the build can’t find the HIP toolchain, and a stale GPU TARGETS in the CMakeCache silently overrides AMDGPU TARGETS the build looks like it targets both but only compiles for one arch. Purge it with cmake -U GPU TARGETS . 5.3 Run on the 3-device config iGPU first The device order is mandatory: iGPU first. ROCm2/ROCm0/ROCm1 . dGPU-first crashes on prefill with a gfxhub page fault on HIP the iGPU cannot read memory allocated on a dGPU, even though the runtime reports peer access. This is a ROCm/driver-level limitation, not a llama.cpp bug. The low-hanging fruit optimization here is to then patch to make sure the prefill task happens asymmetrically – send to dGPUs – but I didn’t explicitly do that for these results. llama-bench slashes between devices HIP VISIBLE DEVICES=0,1,2 \ build/bin/llama-bench -m /path/to/model.gguf \ --device ROCm2/ROCm0/ROCm1 -sm layer \ -p 512 -n 256 -r 3 -ngl 999 llama-cli commas between devices yes, the separator differs LLAMA ATTN ROT DISABLE=1 \ HIP VISIBLE DEVICES=0,1,2 \ build/bin/llama-cli -m /path/to/model.gguf \ --device ROCm2,ROCm0,ROCm1 -sm layer -ngl 999 -p "Hello" Notes: -sm layer pipeline split is required -sm row tensor split fails to load MoE models in llama.cpp. LLAMA ATTN ROT DISABLE=1 is required for Qwen3.8-Flash-Next QSA/quantized-KV Hadamard fix . HIP VISIBLE DEVICES=0,1,2 without --device silently drops the iGPU always pass the device list explicitly. - For Qwen3.8-Flash-Next, the first load could take a long time 25K tensors, single-threaded loader . 300 s timeout is likely not enough. -c 262144 works on the 3-device config with 8-bit KV -ctk q8 0 -ctv q8 0 the full native window fits across dGPU VRAM + iGPU GTT MemFree stays ~61–64 GB . See 5.6 for the deep-context recipe. 5.4 Run on 2x R9700 the fastest config HIP VISIBLE DEVICES=0,1 \ build/bin/llama-bench -m /path/to/model.gguf \ --device ROCm0/ROCm1 -sm layer \ -p 128 -n 256 -r 3 -ngl 999 gpt-oss-120b Q4 K M is a tight fit here ~63.4 of 64 GiB fine at 4K context, don’t push the context window. 5.5 The EngramHalo fork + iGPU-only for the big MoE git clone -b strix-halo-qwen4exp https://github.com/Aristo94/EngramHalo.cpp ~/llama-engramhalo cd ~/llama-engramhalo cmake -B build -DCMAKE BUILD TYPE=Release -DGGML HIP=ON \ -DAMDGPU TARGETS="gfx1201;gfx1151" -DGGML HIP NO VMM=ON \ -DCMAKE C COMPILER=/opt/rocm/bin/hipcc -DCMAKE CXX COMPILER=/opt/rocm/bin/hipcc cmake --build build --target llama-cli llama-bench -j32 Run Qwen3.8-Flash-Next on the iGPU alone at 64K context with 8-bit KV the PLE n-gram table stays on SSD: HSA ENABLE SDMA=0 HSA XNACK=1 ROCBLAS USE HIPBLASLT=1 \ build/bin/llama-cli -m /path/to/Qwen3.8-Flash-Next-Q4 K M.gguf \ --device ROCm2 -ngl 999 -fa on -ctk q8 0 -ctv q8 0 \ -c 65536 -n 8 -p "Hello" --no-display-prompt -lm mmap Expected: ~24 tok/s, MemFree stable at ~40 GB the 33.9 GiB PLE lives on SSD, only ~1–2 GiB resident , and long-context decode that holds up ~13 tok/s at 58.5k depth . 5.6 Deep context on the 3-device config the full 262k window The same 3-device command from 5.3, with the context window opened up and 8-bit KV so the cache fits: LLAMA ATTN ROT DISABLE=1 \ HSA ENABLE SDMA=0 HSA XNACK=1 ROCBLAS USE HIPBLASLT=1 \ HIP VISIBLE DEVICES=0,1,2 \ build/bin/llama-cli -m /path/to/Qwen3.8-Flash-Next-Q4 K M.gguf \ --device ROCm2,ROCm0,ROCm1 -sm layer -ngl 999 -fa on \ -ctk q8 0 -ctv q8 0 -c 262144 -st -p "..." --no-display-prompt -lm mmap -c 262144 with 8-bit KV: full native context, MemFree holds ~61–64 GB. Prefill stays ~213–226 t/s at any context size; decode is ~9.8 t/s at 128k, ~5.5 t/s at 262k. -st single-turn avoids the interactive REPL runaway I hit on the first 128k attempt use it for clean exits. Closing This box is the most fun I’ve had with a GPU in a while, and also the most debugging. The headline is simple: two R9700s make a Strix Halo genuinely fast at MoE inference 100 tok/s on gpt-oss-120b , the iGPU makes the big models fit, prefill scales with device count and the 3-device config runs the model at its full native 262k context with 100% retrieval. The asterisks are the external-switch P2P bottleneck, the iGPU-can’t-read-dGPU-memory limitation, and the fact that a bleeding-edge AMD setup on a rolling distro will occasionally need a power button and a prayer. If you build something like this, I’d love to hear how it goes. And if you’re the maintainer of a llama.cpp fork that makes 176B-parameter models run on an iGPU thank you. I think there is a lot of performance left on the table, but it’s stable at least.