# N5 Max: Proxmox Strix Halo with Docker, ROCm FP4 and MTP - Ultimate Setup Guide

> Source: <https://forum.level1techs.com/t/n5-max-proxmox-strix-halo-with-docker-rocm-fp4-and-mtp-ultimate-setup-guide/251182#post_14>
> Published: 2026-08-24 14:09:28+00:00

Strix Halo ROCmFP4 + MTP LLM Inference Guide

**Hardware:** AMD Ryzen AI MAX+ 395 (16C/32T) + Radeon 8060S (gfx1151) + 64 GB unified memory

[MINISFORUM N5 MAX AI NAS | AMD Ryzen AI Max+ 395 | 126 TOPS NPU | Radeon™ 8060S graphics](https://s.minisforum.com/4fzJPa5)

**Last tested:** 2026-06-06 (Proxmox VE 9.1.1, kernel 6.17.2-1-pve, ROCm 7.2.1)

Background

TODO: video goes here, explainer

My vision for this is to have an “all-in-one” home appliance for AI, Virtual Machines, internet ingress/egress (experimenting with Pangolin on this currently, maybe for a future video).. Home Assistant.. you name it. In a nutshell we’re using Proxmox as our NAS OS then setting up an LXC with full access to the underlying hardware for AI workloads *via* Docker! The best of both worlds. Or something.

[This thing is a beast for any use case.](https://store.minisforum.com/products/minisforum-n5-max-ai-nas)

**Models tested:**

- Qwopus3.6-27B-v2 (ROCmFP4, 14 GB),
- Qwen3.6-27B (Q8_0, ~27 GB)
- TODO?

Table of Contents

[Overview & What’s New](#1-overview--whats-new)
[Architecture Overview](#2-architecture-overview)
[Prerequisites & System Inventory](#3-prerequisites--system-inventory)
[Phase 1: Proxmox Host Preparation](#4-phase-1-proxmox-host-preparation)
[Phase 2: Kernel Boot Parameters (Critical)](#5-phase-2-kernel-boot-parameters-critical)
[Phase 3: Create the LXC Container](#6-phase-3-create-the-lxc-container)
[Phase 4: GPU Passthrough to LXC](#7-phase-4-gpu-passthrough-to-lxc)
[Phase 5: ROCm Installation (Inside LXC)](#8-phase-5-rocm-installation-inside-lxc)
[Phase 6: Build the ROCmFP4 llama.cpp Fork](#9-phase-6-build-the-rocmfp4-llamacpp-fork)
[Phase 7: Build Stock llama.cpp (for Q8 baseline)](#10-phase-7-build-stock-llamacpp-for-q8-baseline)
[Phase 8: Download Models](#11-phase-8-download-models)
[Phase 9: Run Inference](#12-phase-9-run-inference)
[Phase 10: Benchmarks](#13-phase-10-benchmarks)
[vLLM Docker Setup (Alternative Backend)](#14-vllm-docker-setup-alternative-backend)
[Troubleshooting](#15-troubleshooting)
[Reference: Key Community Resources](#16-reference-key-community-resources)

1. Overview & What’s New

This guide covers running **ROCmFP4-quantized models** (a custom 4-bit format with native FP4 tensor layouts optimized for AMD GPUs) on Strix Halo hardware using a custom llama.cpp fork with **Multi-Token Prediction (MTP)** support.

Most of the work was done on the Minisforum N5 Max NAS, but I also validated on the GMK Tek Strix Halo, The Minisforum S1 Max and The Framework Desktop Strix Halo machine they offer.

This was quite an adventure to get setup. Many thinks to kyz0 and other resources linked throughout this guide.

What’s different from standard llama.cpp ROCm guides:

| Feature |
Standard ROCm guide |
This guide |
**Model format** |
Standard GGUF (Q4_K_M, Q8_0, etc.) |
ROCmFP4 custom 4-bit GGUF |
**llama.cpp fork** |
`ggml-org/llama.cpp` |
`charlie12345/rocmfp4-llama` (branch `mtp-rocmfp4-strix` ) |
**MTP support** |
No |
Yes — speculative decoding via draft-MTP |
`HSA_OVERRIDE_GFX_VERSION` |
`11.0.0` (maps gfx1151→gfx1100) |
`11.5.1` (native gfx1151 FP4 support) |
`GGML_HIP_ENABLE_UNIFIED_MEMORY` |
Not needed |
**Required** for models > 2 GB |
`-dio` flag |
Required |
**Required** for models > ~6 GB |
`ulimit -l unlimited` |
Not mentioned |
**Required** — ROCm locks GPU memory |
`setsid` wrapper |
Not needed |
Needed inside `lxc-attach` shells |

2. Architecture Overview

```
Proxmox Host (kernel 6.17.2-1-pve)
  ├── amdgpu.ko (in-tree, gfx1151 support)
  ├── /dev/dri/card1       ───┐
  ├── /dev/dri/renderD128  ───┤── LXC bind-mount
  ├── /dev/kfd             ───┤
  └── /dev/accel/accel0    ───┘  (NPU — not used for LLM today)
        │
  LXC Container (Ubuntu 24.04, privileged, 56 GB RAM)
    ├── ROCm 7.2.1 userspace (no DKMS — host kernel has drivers)
    ├── rocmfp4-llama fork (built for gfx1151 + ROCmFP4)
    ├── stock llama.cpp (for Q8 baseline comparisons)
    ├── Docker (for vLLM ROCm container)
    └── Models:
        ├── Qwopus3.6-27B-v2-MTP-ROCmFP4 (14 GB)
        └── Qwen3.6-27B-Q8_0 (~27 GB)
```

3. Prerequisites & System Inventory

Hardware Requirements

| Component |
Minimum |
Recommended |
**CPU** |
Ryzen AI 300 series |
Ryzen AI MAX+ 395 |
**RAM** |
32 GB |
64 GB (128 GB for 70B models) |
**Proxmox** |
8.x |
9.1+ (Debian 13 base) |
**Kernel** |
6.14+ |
7+ (Proxmox v9 Kernel 7 series) |
**Storage** |
100 GB free |
300 GB+ (for multiple models + builds) |

Verify your hardware

```
# CPU
cat /proc/cpuinfo | grep "model name" | head -1

# GPU (should show Radeon 8060S)
lspci | grep -i display

# RAM
grep MemTotal /proc/meminfo

# GPU device nodes
ls -la /dev/dri/* /dev/kfd /dev/accel/accel0

# Kernel version
uname -r

# Proxmox version
pveversion
```

Expected output

```
model name : AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
90:00.0 Display controller: AMD/ATI Strix Halo [Radeon 8060S Graphics]
MemTotal:       63407216 kB  (~60.5 GB)
crw-rw---- 1 root render 226, 128 /dev/dri/renderD128
crw-rw---- 1 root render 234,   0 /dev/kfd
6.17.2-1-pve
pve-manager/9.1.1
```

4. Phase 1: Proxmox Host Preparation

4.1 Enable the no-subscription repository

```
# Disable the enterprise repo (which will 401 without a subscription key)
rm /etc/apt/sources.list.d/pve-enterprise.sources

# Add the no-subscription repo
cat > /etc/apt/sources.list.d/pve-no-subscription.sources << 'EOF'
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

# Also fix the Ceph repo to use no-subscription
cat > /etc/apt/sources.list.d/ceph.sources << 'EOF'
Types: deb
URIs: http://download.proxmox.com/debian/ceph-squid
Suites: trixie
Components: no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

apt update
```

4.2 Install necessary tools

```
apt install -y curl wget pve-headers build-essential
```

5. Phase 2: Kernel Boot Parameters (Critical)

5.1 Why these parameters

The Strix Halo GPU (`gfx1151`

) needs explicit boot parameters to access all 64 GB of unified memory. Without these, ROCm memory allocation is severely limited.

| Parameter |
Purpose |
`amd_iommu=off` |
Disables IOMMU for the GPU. Required for the GPU to access all system RAM as unified memory. Without this, ROCm memory allocation is capped at ~2 GB. |
`amdgpu.gttsize=131072` |
Sets the GTT (Graphics Translation Table) size to 128 GB. This is a **ceiling** — it tells the GPU it can map up to 128 GB of system RAM. On a 64 GB system, this effectively enables full unified memory access. The value is in megabytes: 131072 MB = 128 GB. |
`ttm.pages_limit=31457280` |
Increases the TTM (Translation Table Manager) page limit. Prevents allocation failures when loading large models. Value = number of 4 KB pages. |

**Note on GTT sizing:** The system has ~60.5 GB of usable RAM. Setting `gttsize=131072`

(128 GB) is safe — it’s just an upper bound. The GPU will only use what’s physically available. For a 64 GB system, a value of ~57344 (56 GB) would also work, but 131072 is harmless and future-proofs the config for systems with more RAM.

5.2 Apply the parameters

```
sed -i 's/GRUB_CMDLINE_Linux_DEFAULT="quiet"/GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=off amdgpu.gttsize=131072 ttm.pages_limit=31457280"/' /etc/default/grub
update-grub
reboot
```

5.3 Verify after reboot

```
# Check GTT size (should be ~128 GB)
cat /sys/class/drm/card1/device/mem_info_gtt_total
# Expected: 137438953472 (128 GB)

# Check boot params applied
cat /proc/cmdline | grep -o 'amdgpu.gttsize=[^ ]*\|amd_iommu=[^ ]*\|ttm.pages_limit=[^ ]*'
# Expected: amd_iommu=off amdgpu.gttsize=131072 ttm.pages_limit=31457280

# Verify /dev/kfd exists
ls -la /dev/kfd
```

6. Phase 3: Create the LXC Container

6.1 Choose a template

Ubuntu 24.04 is recommended over Debian 13 for ROCm compatibility:

```
pveam available | grep ubuntu-24
pveam download local ubuntu-24.04-standard_24.04-2_amd64.tar.zst
```

6.2 Create the container

**Must be privileged** for GPU device access:

```
CT_ID=100
CT_HOSTNAME=strix-ai
CT_STORAGE=local-lvm

pct create $CT_ID local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst \
  --storage $CT_STORAGE \
  --memory 57344 \
  --swap 0 \
  --cores 16 \
  --unprivileged 0 \
  --hostname $CT_HOSTNAME \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --rootfs $CT_STORAGE:100 \
  --features nesting=1
```

**Resource notes:**

`--memory 57344`

— 56 GB RAM (leaves ~4 GB for Proxmox host)
`--swap 0`

— No swap. With unified memory, swapping would thrash the GPU.
`--cores 16`

— All 16 physical cores.
`--unprivileged 0`

— **Privileged** container. Required for GPU device nodes to have correct ownership.
`--rootfs 100`

— 100 GB minimum. 300 GB recommended if storing models locally.

6.3 Start and set up

```
pct start $CT_ID
pct enter $CT_ID

# Inside the container
apt update && apt upgrade -y
apt install -y curl wget git build-essential cmake python3 python3-pip sudo
```

7. Phase 4: GPU Passthrough to LXC

7.1 Identify device nodes on the host

From the **Proxmox host**:

```
ls -la /dev/dri/* /dev/kfd
```

Expected:

```
crw-rw---- 1 root video  226,   1 /dev/dri/card1
crw-rw---- 1 root render 226, 128 /dev/dri/renderD128
crw-rw---- 1 root render 234,   0 /dev/kfd
```

Note the render group GID:

```
getent group render
# Output: render:x:993:
```

7.2 Add device passthrough to container config

Edit `/etc/pve/lxc/$CT_ID.conf`

on the **Proxmox host**:

```
cat >> /etc/pve/lxc/$CT_ID.conf << 'EOF'

# GPU device nodes (use wildcard for major 226 to cover all minors)
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 234:* rwm

# Mount GPU devices into container
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file
EOF
```

7.3 Restart and verify

```
# On the host
pct stop $CT_ID
pct start $CT_ID

# Inside the container
pct enter $CT_ID
ls -la /dev/dri/* /dev/kfd
# Should show root:render and root:video ownership
```

8. Phase 5: ROCm Installation (Inside LXC)

8.1 Add the AMD ROCm repository

```
wget -q -O - https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor > /etc/apt/trusted.gpg.d/rocm.gpg

# Ubuntu 24.04 (noble) — use ROCm 7.2.1 which has noble support
echo "deb [arch=amd64] https://repo.radeon.com/rocm/apt/7.2.1 noble main" > /etc/apt/sources.list.d/rocm.list

# Pin ROCm packages to prefer the AMD repo
cat > /etc/apt/preferences.d/rocm-pin << 'EOF'
Package: *
Pin: release o=repo.radeon.com
Pin-Priority: 600
EOF

apt update
```

8.2 Install ROCm userspace (no DKMS)

**Critical:** Do NOT install `rocm-dkms`

. The amdgpu and amdkfd drivers are already in the Proxmox host kernel.

```
apt install -y rocm-hip-sdk rocwmma-dev
```

Also install Vulkan SDK (needed by the ROCmFP4 fork build):

```
apt install -y libvulkan-dev vulkan-tools
```

8.3 Set up memlock limits (Critical!)

ROCm needs to lock GPU memory pages. The default ulimit is 8 MB — far too low for LLM models:

```
# Add to /etc/security/limits.conf
echo 'root soft memlock unlimited' >> /etc/security/limits.conf
echo 'root hard memlock unlimited' >> /etc/security/limits.conf

# Apply for current session
ulimit -l unlimited
```

8.4 Verify ROCm

```
/opt/rocm/bin/rocminfo
```

Expected output should show:

```
Name:                    gfx1151
Marketing Name:          Radeon 8060S Graphics
Compute Unit:            40
Max Clock Frequency:     2900 MHz

Pool Info:
  Segment:               GLOBAL; FLAGS: COARSE GRAINED
  Size:                  125829120 KB  (120 GB GTT — full unified memory)
```

Also check:

```
/opt/rocm/bin/rocm-smi
```

9. Phase 6: Build the ROCmFP4 llama.cpp Fork

9.1 Clone the fork

```
cd /root
git clone https://github.com/charlie12345/rocmfp4-llama.git
cd rocmfp4-llama
git checkout mtp-rocmfp4-strix
```

9.2 Build

```
export PATH=/opt/rocm/bin:/opt/rocm/llvm/bin:$PATH
export HIP_PATH=/opt/rocm
export ROCM_PATH=/opt/rocm
export ROCM_WMMA_INCLUDE=/opt/rocm/include
export GGML_HIP_ROCWMMA_FATTN=ON
export CMAKE_HIP_ARCHITECTURES=gfx1151
export JOBS=$(nproc)

# The build script references a custom rocWMMA path; we set ROCM_WMMA_INCLUDE
# to point to the standard ROCm install location instead.
bash scripts/build-strix-rocmfp4-mtp.sh
```

**Note:** The build script enables Vulkan by default, which requires `libvulkan-dev`

. If you get Vulkan compilation errors (e.g., `spv`

namespace issues), reconfigure with `-DGGML_VULKAN=OFF`

:

```
cmake -S . -B build-strix-rocmfp4 \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON \
  -DGGML_HIP_ROCWMMA_FATTN=ON \
  -DGGML_HIP_FORCE_MMQ=ON \
  -DGGML_VULKAN=OFF \
  -DGGML_CUDA=OFF \
  -DCMAKE_HIP_ARCHITECTURES=gfx1151 \
  -DLLAMA_BUILD_SERVER=ON
cmake --build build-strix-rocmfp4 -j$(nproc) --target llama-cli llama-server llama-bench
```

9.3 Verify

```
HSA_OVERRIDE_GFX_VERSION=11.5.1 \
GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
./build-strix-rocmfp4/bin/llama-cli --list-devices
```

Expected:

```
Available devices:
  ROCm0: Radeon 8060S Graphics (122880 MiB, 4096 MiB free)
```

10. Phase 7: Build Stock llama.cpp (for Q8 baseline)

For comparing ROCmFP4 against standard quantization formats:

```
cd /root
git clone https://github.com/ggml-org/llama.cpp.git stock-llama
cd stock-llama

HIPCXX="$(hipconfig -l)/clang" \
HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build \
  -DGGML_HIP=ON \
  -DGPU_TARGETS=gfx1151 \
  -DGGML_HIP_ROCWMMA_FATTN=ON \
  -DGGML_HIP_NO_VMM=ON \
  -DGGML_HIP_MMQ_MFMA=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build --config Release -j$(nproc) --target llama-cli llama-server llama-bench llama-perplexity
```

11. Phase 8: Download Models

11.1 ROCmFP4 Model (14 GB)

```
mkdir -p /models/rocmfp4
cd /models/rocmfp4

wget 'https://huggingface.co/jcbtc/qwopus3.6-27b-v2-chadrock-rocmfp4-mtp/resolve/main/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX_LEAN.gguf'

# Optional: vision projection model
wget 'https://huggingface.co/jcbtc/qwopus3.6-27b-v2-chadrock-rocmfp4-mtp/resolve/main/mmproj-F32.mmproj'
```

Expected SHA256: `1f1c0a9d63b9b38b06fe...`

(verify after download)

11.2 Qwen3.6-27B Q8 Baseline (~27 GB)

```
mkdir -p /models/qwen36-27b
cd /models/qwen36-27b

wget 'https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF/resolve/main/Qwen3.6-27B-Q8_0.gguf'
```

11.3 Qwen3.6-27B Q4_K_M (~16 GB, optional additional baseline)

```
wget 'https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF/resolve/main/Qwen3.6-27B-Q4_K_M.gguf'
```

12. Phase 9: Run Inference

12.1 Critical: The `setsid`

Workaround

When running llama-cli inside an `lxc-attach`

shell, the process receives `SIGTTOU`

when trying to set terminal attributes because it’s not the foreground process group. **Always use **`setsid -w`

to create a new session:

```
# Inside the container via lxc-attach
setsid -w bash -c '
ulimit -l unlimited
HSA_OVERRIDE_GFX_VERSION=11.5.1 \
GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
/root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-cli \
  -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX_LEAN.gguf \
  -ngl 999 \
  -fa on \
  -dio \
  -p "Your prompt here." \
  -n 200
'
```

**Alternatively**, if you `pct enter`

the container and run interactively (not via `lxc-attach`

), the `setsid`

wrapper is not needed.

12.2 ROCmFP4 Model with MTP (Full Config)

```
HSA_OVERRIDE_GFX_VERSION=11.5.1 \
GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
/root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-server \
  -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX_LEAN.gguf \
  --mmproj /models/rocmfp4/mmproj-F32.mmproj \
  --alias qwopus3.6-27b-v2-chadrock \
  --host 127.0.0.1 \
  --port 8080 \
  --jinja \
  -c 262144 \
  -ngl 999 \
  -fa on \
  -dev ROCm0 \
  -b 512 \
  -ub 512 \
  -t 16 \
  -tb 32 \
  -ctk q4_0 \
  -ctv q4_0 \
  --spec-type draft-mtp \
  --spec-draft-device ROCm0 \
  --spec-draft-ngl all \
  --spec-draft-type-k q4_0 \
  --spec-draft-type-v q4_0 \
  --spec-draft-n-max 4 \
  --spec-draft-n-min 0 \
  --spec-draft-p-min 0.0 \
  --spec-draft-p-split 0.10 \
  --parallel 1 \
  --metrics \
  --no-mmap
```

12.3 Q8 Baseline (Stock llama.cpp)

```
HSA_OVERRIDE_GFX_VERSION=11.5.1 \
GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
/root/stock-llama/build/bin/llama-cli \
  -m /models/qwen36-27b/Qwen3.6-27B-Q8_0.gguf \
  -ngl 999 \
  -fa on \
  -dio \
  -p "Your prompt here." \
  -n 200
```

12.4 Runtime Flag Reference

| Flag |
Purpose |
`-ngl 999` |
Offload all layers to GPU |
`-fa on` |
Enable flash attention (rocWMMA path) |
`-dio` |
**Required** for models > ~6 GB on gfx1151. Without this, loading hangs. |
`-dev ROCm0` |
Explicitly select the ROCm device |
`--no-mmap` |
Disable memory-mapped model loading (needed for some setups) |
`--spec-type draft-mtp` |
Enable Multi-Token Prediction speculative decoding |
`--spec-draft-*` |
MTP draft model configuration |
`HSA_OVERRIDE_GFX_VERSION=11.5.1` |
Required for ROCmFP4 kernel paths on gfx1151 |
`GGML_HIP_ENABLE_UNIFIED_MEMORY=1` |
**Required** — enables GPU access to system RAM beyond dedicated VRAM |
`ulimit -l unlimited` |
**Required** — ROCm needs to lock GPU memory pages |

13. Phase 10: Benchmarks

13.1 ROCmFP4 Model Benchmark

```
setsid -w bash -c '
ulimit -l unlimited
HSA_OVERRIDE_GFX_VERSION=11.5.1 \
GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
/root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-bench \
  -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX_LEAN.gguf \
  -ngl 999 \
  -fa on \
  -dio \
  -n 512 \
  -p 512
'
```

13.2 Q8 Baseline Benchmark

```
setsid -w bash -c '
ulimit -l unlimited
HSA_OVERRIDE_GFX_VERSION=11.5.1 \
GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
/root/stock-llama/build/bin/llama-bench \
  -m /models/qwen36-27b/Qwen3.6-27B-Q8_0.gguf \
  -ngl 999 \
  -fa on \
  -dio \
  -n 512 \
  -p 512
'
```

13.3 Large Context Test

```
# Test prompt processing at 32K context
setsid -w bash -c '
ulimit -l unlimited
HSA_OVERRIDE_GFX_VERSION=11.5.1 \
GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
/root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-bench \
  -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX_LEAN.gguf \
  -ngl 999 \
  -fa on \
  -dio \
  -p 32768 \
  -n 128
'
```

13.4 MTP Speedup Test

Compare with and without `--spec-type draft-mtp`

to measure speculative decoding benefit.

14. vLLM Docker Setup (Alternative Backend)

vLLM provides a production-grade serving backend. On Strix Halo, it requires the ROCm Docker image and the `HSA_OVERRIDE_GFX_VERSION=11.0.0`

workaround.

We don’t want to run docker on the Proxmox host; we can run it in the LXC Container. I’ve skipped how to install docker in the LXC – you can follow the normal docker installation guide and it’ll work fine, just make sure you’re installing docker *inside* the lxc (not the Proxmox host).

i.e. run the `wget get.docker.com `

script from inside the LXC then run `docker run hello-world`

to make sure docker is working fine inside the LXC.

14.1 Pull the ROCm vLLM image

```
docker pull rocm/vllm:latest
```

14.2 Run vLLM with the Qwen3.6-27B model (HF format)

The model is already available in safetensors format at `/nfs/models/Jackrong/Qwopus3.6-27B-v2/`

(52 GB).

```
docker run -it --rm \
  --device=/dev/dri \
  --device=/dev/kfd \
  --group-add=render \
  --ipc=host \
  --network=host \
  -v /nfs/models:/models \
  -e HSA_OVERRIDE_GFX_VERSION=11.0.0 \
  -e HIP_VISIBLE_DEVICES=0 \
  rocm/vllm:latest \
  --model /models/Jackrong/Qwopus3.6-27B-v2 \
  --dtype float32 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90
```

**Note:** vLLM on Strix Halo requires `float32`

— bfloat16 causes GPU hangs on current kernel/driver combos. The `HSA_OVERRIDE_GFX_VERSION=11.0.0`

maps gfx1151 to gfx1100 code paths.

14.3 Benchmark via vLLM

```
docker exec <container-id> python3 -m vllm.benchmarks.benchmark_throughput \
  --model /models/Jackrong/Qwopus3.6-27B-v2 \
  --dtype float32 \
  --num-prompts 100 \
  --input-len 512 \
  --output-len 256
```

15. Troubleshooting

If you have a problem not listed here **please** do comment below so I can improve the guide!

15.1 Model loading hangs with no output

**Symptoms:** `llama-cli`

starts but produces zero output and never completes.

**Causes and fixes:**

| Cause |
Fix |
**SIGTTOU** (inside `lxc-attach` ) |
Use `setsid -w` wrapper, or run interactively via `pct enter` |
**Missing **`-dio` flag |
Add `-dio` to the command line |
**Missing **`ulimit -l unlimited` |
Set `ulimit -l unlimited` before running |
**Missing **`GGML_HIP_ENABLE_UNIFIED_MEMORY=1` |
Set this environment variable |

15.2 “ROCk module is not loaded” in rocminfo

The `amdgpu`

module is loaded but the ROCm compute path (`amdkfd`

) isn’t initialized.

```
# On the Proxmox host
cat /boot/config-$(uname -r) | grep CONFIG_HSA_AMD
# Should show: CONFIG_HSA_AMD=y

# If it's a module, load it
modprobe amdkfd

# Check /dev/kfd exists
ls -la /dev/kfd
```

15.3 “Not enough memory” errors

Check GTT size:

```
cat /sys/class/drm/card1/device/mem_info_gtt_total
```

If this shows less than ~60 GB, the `amdgpu.gttsize`

kernel parameter isn’t being applied:

```
cat /proc/cmdline | grep gttsize
```

If missing, re-run `update-grub`

and reboot.

15.4 Container can’t see GPU devices

```
# On the host
ls -la /dev/dri/* /dev/kfd

# Check container config
cat /etc/pve/lxc/$CT_ID.conf | grep -E 'lxc\.(cgroup|mount)'

# Restart the container
pct stop $CT_ID
pct start $CT_ID
```

15.5 Permission denied on /dev/kfd inside container

Ensure the container is **privileged** (`unprivileged: 0`

in config). If using an unprivileged container, add:

```
lxc.idmap: g 993 993 1
```

15.6 Vulkan compilation errors during build

The ROCmFP4 fork’s build script enables Vulkan by default. If you get `spv`

namespace errors, rebuild with Vulkan disabled:

```
cmake -S . -B build-strix-rocmfp4 \
  -DGGML_VULKAN=OFF \
  # ... other flags ...
```

15.7 Slow performance

- Verify you’re using the ROCm backend, not CPU fallback: the log should show
`ggml_rocm_init`

- Check that
`GPU_TARGETS=gfx1151`

was used during build
- Verify flash attention is enabled:
`-fa on`

- Check GPU clock speed:
`rocm-smi`

- For first runs, kernel compilation may slow things down — subsequent runs will be faster

16. Reference: Key Community Resources

Appendix A: Quick-Start Script

**DANGER**: I am providing this for reference, but again, if it works (or doesn’t) please comment below. Thanks!

``` bash
#!/bin/bash
# strix-halo-rocmfp4-setup.sh — Run on Proxmox host as root
# WARNING: Review each section before running. This will reboot the host.

set -euo pipefail

CT_ID=${1:-100}
CT_NAME=${2:-strix-ai}

echo "=== Step 1: Add no-subscription repos ==="
rm -f /etc/apt/sources.list.d/pve-enterprise.sources
cat > /etc/apt/sources.list.d/pve-no-subscription.sources << 'EOF'
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
apt update

echo "=== Step 2: Add boot parameters ==="
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=off amdgpu.gttsize=131072 ttm.pages_limit=31457280"/' /etc/default/grub
update-grub

echo "=== Step 3: Download LXC template ==="
pveam download local ubuntu-24.04-standard_24.04-2_amd64.tar.zst

echo "=== Step 4: Create container ==="
pct create $CT_ID local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst \
  --storage local-lvm \
  --memory 57344 \
  --swap 0 \
  --cores 16 \
  --unprivileged 0 \
  --hostname $CT_NAME \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --rootfs local-lvm:100 \
  --features nesting=1

echo "=== Step 5: Add GPU passthrough ==="
cat >> /etc/pve/lxc/$CT_ID.conf << 'EOF'

lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 234:* rwm
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file
EOF

echo "=== Done! Reboot to apply the new kernel ==="
echo "After reboot:"
echo "  1. pct start $CT_ID"
echo "  2. pct enter $CT_ID"
echo "  3. Run the inside-container setup (ROCm + llama.cpp builds)"
echo ""
echo "Rebooting in 10 seconds... Ctrl+C to cancel."
sleep 10
reboot
```

Appendix B: Okay, that’s setup? Where’s my home lab?

So from here, you can setup containers or VMs or run Docker for whatever you like. Check out [Proxmox VE Helper-Scripts](https://community-scripts.org/) to get some ideas.

[Jellyfin Media Server | Proxmox VE Helper Scripts](https://community-scripts.org/scripts/jellyfin)

[Frigate | Proxmox VE Helper Scripts](https://community-scripts.org/scripts/frigate)

[Home Assistant OS | Proxmox VE Helper Scripts](https://community-scripts.org/scripts/haos-vm)

…and so many more!