Ryzen AI Halo: Halogen Server Testing Notes

wpnews.pro

#

I dunno that this will become a video, theres so many videos but I wanted to share this with the community as its my notes from experimenting with the “Halogen” engine on the Strix Halo platform.

Maybe I’ll tweet about it.

What’s Halogen? It’s this sort of interesting optimized server for running qwen 3.8 flash next on Strix halo. You should read about the project, it’s interesting.

This was tested on several Strix Halo platforms including the ASUS ROG Flow Z13.

Original project: github.com/peonist-ai/halogen-flash-server — the fastest way to run Qwen3.8-Flash-Next on AMD Strix Halo (gfx1151).

Model: peonist-ai/halogen-qwen3.8-flash-next on Hugging Face.

Platform: AMD Ryzen AI MAX+ 395 (Radeon 8060S, 128 GB unified memory).

This guide covers a real-world deployment on a Strix Halo mini-PC. Normally I use Ubuntu LTS with Docker, but I decided to try Podman on Debian 13 this time around.

We walk through the stock launch, power tuning, YaRN 1M context extension, and the amd_iommu=off kernel parameter that’s good for about +10% perf uplift. Each step includes the commands, the reasoning, and the performance impact.

#

Prerequisites & Hardware
Stock Launch (Reference)
Power Tuning: Platform Profile & CPU Governor
YaRN 1M Context Extension
amd_iommu=off – The Secret Sauce
Container Management
Performance Results
Troubleshooting
Appendix: Rollback & Recovery

#

AMD Ryzen AI Halo

GMKtec

ROG Flow Z13

Framework 13

These systems are all bananas. The software has made the story. This should all work on AMD’s Gorgon Halo as well – 400 series with 192gb ram.

These all had 128gb LPDDR5 unified memory.

OS : Debian 13 “Trixie” (or any recent Linux with ROCm 7.14 support)
Container runtime : Podman (Docker works too; adjust--group-add accordingly)
ROCm : 7.14 (shipped with the halogen-flash-server image)
Kernel : 6.18.44+rex+5-amd64 (vendor kernel from AMD on the AI Halo, but CachyOS kernel tested and works too.)

Download the model weights and n-gram table from Hugging Face (~111 GB total):

pip install huggingface-hub

huggingface-cli download peonist-ai/halogen-qwen3.8-flash-next \
  --local-dir ~/halogen-models

^ strictly speaking this isn’t needed as the halogen github link, at first run, downloads the model. I like to do this to ensure that “extra” downloads across test runs do not happen. Save my precious bandwidth.

Expected files:

If you aren’t read in on the whole N-gram thing, this is a “new” thing where part of the model is designed to run from dram at dram speeds. This is a unified memory platform so it manifests as a speed benefit, but the N-gram model work should be exciting to everyone since it promises decent performance on “normal” systems that have fast vram (faster than LPDDR5) and slower system memory.

#

This is the baseline from the GitHub README. Run the container as-is, no tuning:

podman run -d --name halogen-flash-server \
  -p 8731:8731 \
  --device /dev/kfd --device /dev/dri \
  --group-add keep-groups \
  --ipc=host \
  --ulimit memlock=-1:-1 \
  -v ~/halogen-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.15.0

The server listens on port 8731. Check health:

curl -s http://localhost:8731/health | python3 -m json.tool

Key observations at stock:

Platform profile: balanced
CPU governor: powersave
GPU clocks: ~2000 MHz under load
GPU power: ~60W
Context: 262,144 tokens (native, no YaRN)
Output cap: 65,536 tokens

#

The stock balanced profile and powersave governor leave a lot of performance on the table. I suspect some of the benchmarks posted on github on strix halo were run with a conservative CPU governor. The GPU can hit 2900 MHz and 130W, but the power envelope needs to be opened up. The GMKtek has been tuned for > 150W, too, but I have omitted the results from this writeup to keep it from being confusing (it was only worth about 2% more, at best, performance).

cat /sys/firmware/acpi/platform_profile

cat /sys/devices/system/cpu/cpufreq/policy*/scaling_governor | sort -u

cat /sys/class/drm/card0/device/pp_dpm_sclk

sudo -S sh -c 'echo performance > /sys/firmware/acpi/platform_profile'

for f in /sys/devices/system/cpu/cpufreq/policy*/scaling_governor; do
  sudo -S sh -c "echo performance > \"$f\""
done

cat /sys/firmware/acpi/platform_profile    # → performance
cat /sys/devices/system/cpu/cpufreq/policy0/scaling_governor  # → performance

   sudo -S sh -c 'echo balanced > /sys/firmware/acpi/platform_profile'
for f in /sys/devices/system/cpu/cpufreq/policy*/scaling_governor; do
   sudo -S sh -c "echo powersave > \"$f\""
done

Note: These settings are ephemeral — they reset on reboot. See the Post-Reboot Automation section to persist them.

#

The model’s native context is 262,144 tokens. With YaRN (Yet another RoPE extensioN) at factor 4, we extend this to 1,048,576 tokens (~1M). This requires a larger KV cache pool — about 28.8 GiB of the 128 GiB unified memory.

podman run -d --name halogen-flash-server \
  -p 8731:8731 \
  --device /dev/kfd --device /dev/dri \
  --group-add keep-groups \
  --ipc=host \
  --ulimit memlock=-1:-1 \
  -e HALOGEN_ROPE_YARN=4 \
  -e HALOGEN_CTX=1048576 \
  -e HALOGEN_MAX_THINKING_TOKENS=262144 \
  -e HALOGEN_MAX_TOKENS_DEFAULT=393216 \
  -e HALOGEN_MAX_TOKENS_CAP=393216 \
  -v ~/halogen-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.15.0

curl -s http://localhost:8731/health | python3 -c "
import sys, json
d = json.load(sys.stdin)
print('Context:', d['context'])
print('YaRN:', d.get('rope_scaling'))
print('Max tokens default:', d.get('max_tokens_default'))
print('Max thinking tokens:', d.get('max_thinking_tokens_default'))
"

Expected output:

Context: 1048576
YaRN: {'type': 'yarn', 'factor': 4.0, 'original_context': 262144}
Max tokens default: 393216
Max thinking tokens: 262144

rope: static YaRN, factor 4 over the native 262144 (attention scale 1.138629)
startup [   3.6 s] KV pool reserved: 1048576 positions (about 28.8 GiB)
startup [   3.7 s] memory: 62.1 GiB of weights locked in RAM, 28.8 GiB of KV pool, 8.1 GiB of working memory, 99.0 GiB in all
startup [   3.7 s] host memory left for everything else: ~20 GiB

With YaRN 1M enabled, the server uses ~99 GiB of the 128 GiB pool:

62 GiB — model weights (pinned in RAM)
29 GiB — KV cache (1,048,576 positions)
8 GiB — working memory
~20 GiB — remaining for the OS and other services

If you’re tight on memory, you can halve the KV pool:

-e HALOGEN_KV_POOL_POSITIONS=524288
-e HALOGEN_CTX=524288

This reduces context to 524K but frees ~14 GiB.

#

The single biggest performance unlock on Strix Halo is disabling the IOMMU. On this platform, the IOMMU adds overhead to GPU memory access that visibly impacts prefill throughput. The project’s reference machine runs with amd_iommu=off as part of its kernel command line, and the README notes it’s worth 13–16% of prefill performance.

Note: The reference machine also uses additional GPU-tuning kernel parameters: amdgpu.vm_update_mode=0 amdgpu.noretry=0 amdgpu.gttsize=126976 ttm.pages_limit=32505856 amdgpu.sg_display=0. We did NOT apply these in our testing — our results come from amd_iommu=off alone, combined with the userspace power tuning from Section 3.

The system uses systemd-boot (not GRUB). Boot entries live on the EFI System Partition at /efi//entries/.

Step 1: Identify the current boot entry

bootctl status | grep "Current Entry"

Step 2: Back up the entry

sudo cp /efi//entries/amd-ryzen-ai-developer-platform-6.18.44+rex+5-amd64.conf \
       /efi//entries/amd-ryzen-ai-developer-platform-6.18.44+rex+5-amd64.conf.bak-iommu

Step 3: Add amd_iommu=off to the kernel command line

sudo sed -i 's/^options /options amd_iommu=off /' \
  /efi//entries/amd-ryzen-ai-developer-platform-6.18.44+rex+5-amd64.conf

Step 4: Verify the edit

cat /efi//entries/amd-ryzen-ai-developer-platform-6.18.44+rex+5-amd64.conf

Should show:

options amd_iommu=off     root=UUID=... splash quiet loglevel=3

Step 5: Reboot

sudo systemctl reboot

Step 6: Verify after reboot

cat /proc/cmdline | tr ' ' '\n' | grep iommu

sudo dmesg | grep -c AMD-Vi

sudo dmesg | grep -i kfd

Yes. On this kernel (6.18.44+rex+5-amd64), the AMD KFD (Kernel Fusion Driver) falls back to GART-based initialization when the IOMMU is disabled. You’ll see this in dmesg:

kfd kfd: amdgpu: Allocated 3969056 bytes on gart
kfd kfd: amdgpu: Total number of KFD nodes to be created: 1
kfd kfd: amdgpu: added device 1002:1586

ROCm 7.14 works normally. The container needs --group-add with the render group’s GID (typically 992) because rootless Podman remaps device ownership, but that’s a container-runtime detail, not an IOMMU issue.

If the system fails to boot after adding amd_iommu=off, use the boot menu to select the backup entry or use a recovery USB to restore the backup:

sudo cp /efi//entries/...conf.bak-iommu /efi//entries/...conf

#

On this Debian 13 system, rootless Podman remaps device node ownership inside the container. The /dev/kfd and /dev/dri/renderD128 devices show up as owned by nobody:nogroup, which breaks ROCm’s permission check.

Fix: Run the container under sudo podman with the render group’s GID:

grep ^render /etc/group | cut -d: -f3

sudo podman run -d --name halogen-flash-server \
  -p 8731:8731 \
  --device /dev/kfd --device /dev/dri \
  --group-add 992 \
  --ipc=host \
  --ulimit memlock=-1:-1 \
  -e HALOGEN_ROPE_YARN=4 \
  -e HALOGEN_CTX=1048576 \
  -e HALOGEN_MAX_THINKING_TOKENS=262144 \
  -e HALOGEN_MAX_TOKENS_DEFAULT=393216 \
  -e HALOGEN_MAX_TOKENS_CAP=393216 \
  -v /home/w/halogen-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.15.0

Save this as ~/halogen-podman and chmod +x:

#!/bin/bash
PASS=your_sudo_password_here
ACTION="$1"
shift
case "$ACTION" in
  start|stop|restart|logs|rm)
    echo "$PASS" | sudo -S podman "$ACTION" halogen-flash-server "$@"
    ;;
  ps)
    echo "$PASS" | sudo -S podman ps -a --filter name=halogen-flash-server "$@"
    ;;
  health)
    echo "$PASS" | sudo -S curl -s http://localhost:8731/health "$@"
    ;;
  run)
    echo "$PASS" | sudo -S podman run -d --name halogen-flash-server \
      -p 8731:8731 \
      --device /dev/kfd --device /dev/dri \
      --group-add 992 \
      --ipc=host \
      --ulimit memlock=-1:-1 \
      -e HALOGEN_ROPE_YARN=4 \
      -e HALOGEN_CTX=1048576 \
      -e HALOGEN_MAX_THINKING_TOKENS=262144 \
      -e HALOGEN_MAX_TOKENS_DEFAULT=393216 \
      -e HALOGEN_MAX_TOKENS_CAP=393216 \
      -v /home/w/halogen-models:/models \
      ghcr.io/peonist-ai/halogen-flash-server:0.15.0
    ;;
  *)
    echo "Usage: halogen-podman {start|stop|restart|logs|rm|ps|health|run}"
    ;;
esac

Usage:

./halogen-podman start    # Start the container
./halogen-podman stop     # Stop it
./halogen-podman restart  # Restart it
./halogen-podman logs     # View logs
./halogen-podman health   # Check health endpoint
./halogen-podman ps       # Container status
./halogen-podman run      # Create a fresh container

There is probably a better way to do this. This kind of thing isn’t necessary with Docker, or I might need more XP with podman to understand what I’ve missed. I suspect that because when you’re in the docker group, you’ve basically got root, and this is a different architectural choice in podman to prevent that. But I need the hardware. So this is just making some notes for myself about that.

After every reboot, you need to:

Re-apply the performance profile + governor
Start the halogen container

A simple systemd oneshot service can handle this kind of thing, though. Create /etc/systemd/system/halogen-tune.service:

[Unit]
Description=Halogen power tuning
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/halogen-tune.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

And /usr/local/bin/halogen-tune.sh:

#!/bin/bash
echo performance > /sys/firmware/acpi/platform_profile
for f in /sys/devices/system/cpu/cpufreq/policy*/scaling_governor; do
  echo performance > "$f"
done
sudo systemctl enable halogen-tune.service

Then add the container start to your crontab (@reboot /home/w/halogen-podman start) or another systemd unit.

Different distros handle this in different ways. I’m getting a bit rusty on Debian and there may be a less brute-force way to do this. But this is good practice for you sysadmin-in-training folks out there.

#

We used the built-in sweep command, which measures end-to-end HTTP request latency including prefill and decode:

sudo podman run --rm ... halogen-flash-server:0.15.0 sweep -p 8192,32768 -n 128 -r 3
sudo podman run --rm ... halogen-flash-server:0.15.0 sweep -p 131072 -n 128 -r 1

sudo podman run --rm ... halogen-flash-server:0.15.0 bench mtp 256 low 1

All benchmarks use the MTP (Multi-Token Prediction) drafter, which is the default and fastest mode.

| Test | Ref (0.14.1) | Baseline (0.14.0) | Pass 1 (stock) | Pass 2 (tuned) | Pass 3 (tuned+IOMMU) | | pp8192 | 1,584 t/s | 1,268 t/s | 1,350 t/s | 1,679 t/s | 1,768 t/s | | pp32768 | 1,567 t/s | 1,442 t/s | 1,216 t/s | 1,469 t/s | 1,695 t/s | | pp131072 | 1,517 t/s | 1,390 t/s | 1,171 t/s | 1,426 t/s | 1,621 t/s | | tg128 MTP | 46.0 t/s | 39.9 t/s | 48.1 t/s | 52.0 t/s | 52.2 t/s | | mtp 256 low 1 | — | — | — | — | 53.4 t/s |

pp = prefill (prompt processing), tg = token generation. All values in tokens/second. Higher is better. Reference and Baseline columns from the project README.

Most real-world usage will be at 256K context or less – the model’s native length. The YaRN 1M config is available for deep-context tasks, but 256K covers the vast majority of chat, coding, and analysis workloads. Here’s how performance compares:

| Metric | 256K Context (native) | 1M Context (YaRN 4x) | | KV pool size | ~7 GiB | ~29 GiB | | Total memory used | ~77 GiB | ~99 GiB | | Host memory free | ~42 GiB | ~20 GiB | | Prefill @ 8K | ~1,770 t/s | ~1,770 t/s | | Prefill @ 32K | ~1,700 t/s | ~1,700 t/s | | Prefill @ 131K | ~1,620 t/s | ~1,620 t/s | | Decode (MTP, short ctx) | ~53 t/s | ~53 t/s | | Decode (MTP, deep ctx ~260K) | — | ~45 t/s |

The prefill and short-context decode numbers are nearly identical between configs – the YaRN scaling doesn’t add meaningful overhead for prompt processing or short generations. The difference appears at deep context: the README reports 45.0 tok/s decode at 258K context with the 1M config, vs 46.0 tok/s at 32K context. The 1M config also enables prompt caching across very long sessions (the follow-up turn at 100K context is ~2s).

Recommendation: Run with the 1M config by default. The memory cost (~22 GiB extra for the larger KV pool) is worth the flexibility, and performance at 256K seems basically identical. Only drop to 524K or 256K if you’re running other memory-hungry services alongside the server.

| Metric | Stock (Pass 1) | Tuned (Pass 2) | Tuned+IOMMU (Pass 3) | Ref Machine | | GPU clock under load | ~2000 MHz | ~2850 MHz | ~2900 MHz | — | | GPU power under load | ~60W | ~130W | ~140W | ~85W | | Idle clock | 600 MHz | 600 MHz | 600 MHz | — | | Idle power | ~5W | ~5W | ~5W | — |

The GPU hits its 2900 MHz ceiling and 140W power limit (160w on the GMKtek) after tuning. The reference machine runs at ~85W sustained – our higher power draw is expected given the performance profile (the reference likely uses a tuned balanced profile with IOMMU=off and the additional GPU kernel parameters). The remaining small gap vs reference at 131K prefill is likely memory-bandwidth bound rather than clock-limited.

pp8192:     ████████████████████░░░░░░░░░░  1,768 t/s  (+12% vs ref 1,584)
pp32768:    █████████████████████░░░░░░░░░░  1,695 t/s  (+8% vs ref 1,567)
pp131072:   ████████████████████████░░░░░░░░  1,621 t/s  (+7% vs ref 1,517)
Decode:     ██████████████████████████░░░░░░  53.4 t/s  (+16% vs ref 46.0)

Pass 3 (tuned + IOMMU=off) exceeds the reference at every prefill size, despite not using the additional GPU kernel parameters (amdgpu.vm_update_mode=0 , etc.) that the reference machine employs. This suggests the userspace power tuning (performance profile + governor) is doing significant work beyond just IOMMU=off.
Pass 1 was below the 0.14.0 baseline — our stock configuration (IOMMU on, balanced profile) was leaving ~20% on the table.
Pass 2 (tuning alone) recovered most of the gap — the power envelope change from ~60W to ~130W and GPU clocks from ~2000 MHz to ~2850 MHz was the dominant factor.
Pass 3 (IOMMU=off) added the remaining ~7-15% — consistent with the project’s stated 13-16% IOMMU overhead.

#

Symptoms: Container starts but immediately exits. Logs show HIP /src/halogen/src/flash_ops.h:7060: no ROCm-capable device is detected.

Causes:

Rootless Podman — Device nodes are remapped tonobody:nogroup . Usesudo podman with--group-add <render_GID> .
User not in render group — Add your user:sudo usermod -aG render,video $USER then log out and back in.
KFD not initialized — Checksudo dmesg | grep kfd . If empty, the kernel may needamd_iommu=off removed or KFD may needmodprobe amdkfd (though on the AMD AI halo kernel it’s built-in).

Symptoms: Startup log shows memory budget close to limit, then engine refuses to start.

Fix: Reduce KV pool size:

-e HALOGEN_KV_POOL_POSITIONS=524288
-e HALOGEN_CTX=524288

This halves the KV cache from 1M to 524K positions, freeing ~14 GiB.

Symptoms: OOM during weight or KV pool reservation.

Fix: Reduce HALOGEN_MAX_TOK (working memory):

-e HALOGEN_MAX_TOK=16384

The engine auto-caps this at 16384 for 1M context anyway, but setting it explicitly avoids the warning.

If a future kernel update changes the KFD behavior with amd_iommu=off, rollback:

sudo cp /efi//entries/...conf.bak-iommu /efi//entries/...conf
sudo reboot

On kernel 6.18.44+rex+5-amd64 with ROCm 7.14, KFD works correctly via GART fallback. The vendor’s reference machine runs the same configuration.

systemd-boot sorts entries by sort-key, then by version. If you edited the 6.18.35 entry but the system booted 6.18.44, apply the edit to the actually-booted entry. Check with bootctl status | grep "Current Entry". I’m just noting this here because I swear I remember this behaving differently in the past, but I wouldn’t expect the Debian maintainers to have changed this. Maybe Limine on CachyOS is overwriting my memories lol

#

sudo cp /efi//entries/amd-ryzen-ai-developer-platform-6.18.44+rex+5-amd64.conf.bak-iommu \
       /efi//entries/amd-ryzen-ai-developer-platform-6.18.44+rex+5-amd64.conf
sudo reboot

   sudo -S sh -c 'echo balanced > /sys/firmware/acpi/platform_profile'
for f in /sys/devices/system/cpu/cpufreq/policy*/scaling_governor; do
    sudo -S sh -c "echo powersave > \"$f\""
done

sudo podman stop halogen-flash-server
sudo podman rm halogen-flash-server

curl -s http://localhost:8731/health | python3 -c "import sys,json; d=json.load(sys.stdin); print('OK' if d['status']=='ok' else 'FAIL')"

curl -s -X POST http://localhost:8731/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"halogen-qwen3.8-flash-next","messages":[{"role":"user","content":"Say hello in one word."}],"max_completion_tokens":10,"temperature":0,"enable_thinking":false}'

#

Peonist AI for thehalogen-flash-server project and theQwen3.8-Flash-Next model weights
AMD for the Strix Halo platform and ROCm 7.14
Qwen team (Alibaba) for the underlying Qwen3.8 architecture

I am genuinely very impressed with how fast the software ecosystem is improving. Halogen apparently cutting away all the cruft has led to more coherence and performance than I would have thought possible on the Strix Halo platform. 45t/s at 1M!

source & further reading

forum.level1techs.com — original article When Is It Worth Personally Preparing for AI Disasters? What should the EA community be doing now that the (AIS) Overton window has opened up again? Money Pumps for Agents Who Are Ambiguity Averse or Ambiguity Seeking

Ryzen AI Halo: Halogen Server Testing Notes

Run your AI side-project on zahid.host