N5 Max: Proxmox Strix Halo with Docker, ROCm FP4 and MTP - Ultimate Setup Guide A new setup guide from Minisforum details running ROCmFP4-quantized models with Multi-Token Prediction (MTP) on the N5 Max AI NAS, powered by the AMD Ryzen AI MAX+ 395 (16C/32T) with Radeon 8060S (gfx1151) and 64 GB unified memory. The guide, last tested on 2026-06-06 with Proxmox VE 9.1.1, kernel 6.17.2-1-pve, and ROCm 7.2.1, covers a full Proxmox LXC and Docker setup, including kernel boot parameters, GPU passthrough, and building a custom llama.cpp fork (charlie12345/rocmfp4-llama, branch mtp-rocmfp4-strix) to achieve native FP4 support and speculative decoding via MTP. Strix Halo ROCmFP4 + MTP LLM Inference Guide Hardware: AMD Ryzen AI MAX+ 395 16C/32T + Radeon 8060S gfx1151 + 64 GB unified memory MINISFORUM N5 MAX AI NAS | AMD Ryzen AI Max+ 395 | 126 TOPS NPU | Radeon™ 8060S graphics https://s.minisforum.com/4fzJPa5 Last tested: 2026-06-06 Proxmox VE 9.1.1, kernel 6.17.2-1-pve, ROCm 7.2.1 Background TODO: video goes here, explainer My vision for this is to have an “all-in-one” home appliance for AI, Virtual Machines, internet ingress/egress experimenting with Pangolin on this currently, maybe for a future video .. Home Assistant.. you name it. In a nutshell we’re using Proxmox as our NAS OS then setting up an LXC with full access to the underlying hardware for AI workloads via Docker The best of both worlds. Or something. This thing is a beast for any use case. https://store.minisforum.com/products/minisforum-n5-max-ai-nas Models tested: - Qwopus3.6-27B-v2 ROCmFP4, 14 GB , - Qwen3.6-27B Q8 0, ~27 GB - TODO? Table of Contents Overview & What’s New 1-overview--whats-new Architecture Overview 2-architecture-overview Prerequisites & System Inventory 3-prerequisites--system-inventory Phase 1: Proxmox Host Preparation 4-phase-1-proxmox-host-preparation Phase 2: Kernel Boot Parameters Critical 5-phase-2-kernel-boot-parameters-critical Phase 3: Create the LXC Container 6-phase-3-create-the-lxc-container Phase 4: GPU Passthrough to LXC 7-phase-4-gpu-passthrough-to-lxc Phase 5: ROCm Installation Inside LXC 8-phase-5-rocm-installation-inside-lxc Phase 6: Build the ROCmFP4 llama.cpp Fork 9-phase-6-build-the-rocmfp4-llamacpp-fork Phase 7: Build Stock llama.cpp for Q8 baseline 10-phase-7-build-stock-llamacpp-for-q8-baseline Phase 8: Download Models 11-phase-8-download-models Phase 9: Run Inference 12-phase-9-run-inference Phase 10: Benchmarks 13-phase-10-benchmarks vLLM Docker Setup Alternative Backend 14-vllm-docker-setup-alternative-backend Troubleshooting 15-troubleshooting Reference: Key Community Resources 16-reference-key-community-resources 1. Overview & What’s New This guide covers running ROCmFP4-quantized models a custom 4-bit format with native FP4 tensor layouts optimized for AMD GPUs on Strix Halo hardware using a custom llama.cpp fork with Multi-Token Prediction MTP support. Most of the work was done on the Minisforum N5 Max NAS, but I also validated on the GMK Tek Strix Halo, The Minisforum S1 Max and The Framework Desktop Strix Halo machine they offer. This was quite an adventure to get setup. Many thinks to kyz0 and other resources linked throughout this guide. What’s different from standard llama.cpp ROCm guides: | Feature | Standard ROCm guide | This guide | Model format | Standard GGUF Q4 K M, Q8 0, etc. | ROCmFP4 custom 4-bit GGUF | llama.cpp fork | ggml-org/llama.cpp | charlie12345/rocmfp4-llama branch mtp-rocmfp4-strix | MTP support | No | Yes — speculative decoding via draft-MTP | HSA OVERRIDE GFX VERSION | 11.0.0 maps gfx1151→gfx1100 | 11.5.1 native gfx1151 FP4 support | GGML HIP ENABLE UNIFIED MEMORY | Not needed | Required for models 2 GB | -dio flag | Required | Required for models ~6 GB | ulimit -l unlimited | Not mentioned | Required — ROCm locks GPU memory | setsid wrapper | Not needed | Needed inside lxc-attach shells | 2. Architecture Overview Proxmox Host kernel 6.17.2-1-pve ├── amdgpu.ko in-tree, gfx1151 support ├── /dev/dri/card1 ───┐ ├── /dev/dri/renderD128 ───┤── LXC bind-mount ├── /dev/kfd ───┤ └── /dev/accel/accel0 ───┘ NPU — not used for LLM today │ LXC Container Ubuntu 24.04, privileged, 56 GB RAM ├── ROCm 7.2.1 userspace no DKMS — host kernel has drivers ├── rocmfp4-llama fork built for gfx1151 + ROCmFP4 ├── stock llama.cpp for Q8 baseline comparisons ├── Docker for vLLM ROCm container └── Models: ├── Qwopus3.6-27B-v2-MTP-ROCmFP4 14 GB └── Qwen3.6-27B-Q8 0 ~27 GB 3. Prerequisites & System Inventory Hardware Requirements | Component | Minimum | Recommended | CPU | Ryzen AI 300 series | Ryzen AI MAX+ 395 | RAM | 32 GB | 64 GB 128 GB for 70B models | Proxmox | 8.x | 9.1+ Debian 13 base | Kernel | 6.14+ | 7+ Proxmox v9 Kernel 7 series | Storage | 100 GB free | 300 GB+ for multiple models + builds | Verify your hardware CPU cat /proc/cpuinfo | grep "model name" | head -1 GPU should show Radeon 8060S lspci | grep -i display RAM grep MemTotal /proc/meminfo GPU device nodes ls -la /dev/dri/ /dev/kfd /dev/accel/accel0 Kernel version uname -r Proxmox version pveversion Expected output model name : AMD RYZEN AI MAX+ 395 w/ Radeon 8060S 90:00.0 Display controller: AMD/ATI Strix Halo Radeon 8060S Graphics MemTotal: 63407216 kB ~60.5 GB crw-rw---- 1 root render 226, 128 /dev/dri/renderD128 crw-rw---- 1 root render 234, 0 /dev/kfd 6.17.2-1-pve pve-manager/9.1.1 4. Phase 1: Proxmox Host Preparation 4.1 Enable the no-subscription repository Disable the enterprise repo which will 401 without a subscription key rm /etc/apt/sources.list.d/pve-enterprise.sources Add the no-subscription repo cat /etc/apt/sources.list.d/pve-no-subscription.sources << 'EOF' Types: deb URIs: http://download.proxmox.com/debian/pve Suites: trixie Components: pve-no-subscription Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg EOF Also fix the Ceph repo to use no-subscription cat /etc/apt/sources.list.d/ceph.sources << 'EOF' Types: deb URIs: http://download.proxmox.com/debian/ceph-squid Suites: trixie Components: no-subscription Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg EOF apt update 4.2 Install necessary tools apt install -y curl wget pve-headers build-essential 5. Phase 2: Kernel Boot Parameters Critical 5.1 Why these parameters The Strix Halo GPU gfx1151 needs explicit boot parameters to access all 64 GB of unified memory. Without these, ROCm memory allocation is severely limited. | Parameter | Purpose | amd iommu=off | Disables IOMMU for the GPU. Required for the GPU to access all system RAM as unified memory. Without this, ROCm memory allocation is capped at ~2 GB. | amdgpu.gttsize=131072 | Sets the GTT Graphics Translation Table size to 128 GB. This is a ceiling — it tells the GPU it can map up to 128 GB of system RAM. On a 64 GB system, this effectively enables full unified memory access. The value is in megabytes: 131072 MB = 128 GB. | ttm.pages limit=31457280 | Increases the TTM Translation Table Manager page limit. Prevents allocation failures when loading large models. Value = number of 4 KB pages. | Note on GTT sizing: The system has ~60.5 GB of usable RAM. Setting gttsize=131072 128 GB is safe — it’s just an upper bound. The GPU will only use what’s physically available. For a 64 GB system, a value of ~57344 56 GB would also work, but 131072 is harmless and future-proofs the config for systems with more RAM. 5.2 Apply the parameters sed -i 's/GRUB CMDLINE Linux DEFAULT="quiet"/GRUB CMDLINE LINUX DEFAULT="quiet amd iommu=off amdgpu.gttsize=131072 ttm.pages limit=31457280"/' /etc/default/grub update-grub reboot 5.3 Verify after reboot Check GTT size should be ~128 GB cat /sys/class/drm/card1/device/mem info gtt total Expected: 137438953472 128 GB Check boot params applied cat /proc/cmdline | grep -o 'amdgpu.gttsize= ^ \|amd iommu= ^ \|ttm.pages limit= ^ ' Expected: amd iommu=off amdgpu.gttsize=131072 ttm.pages limit=31457280 Verify /dev/kfd exists ls -la /dev/kfd 6. Phase 3: Create the LXC Container 6.1 Choose a template Ubuntu 24.04 is recommended over Debian 13 for ROCm compatibility: pveam available | grep ubuntu-24 pveam download local ubuntu-24.04-standard 24.04-2 amd64.tar.zst 6.2 Create the container Must be privileged for GPU device access: CT ID=100 CT HOSTNAME=strix-ai CT STORAGE=local-lvm pct create $CT ID local:vztmpl/ubuntu-24.04-standard 24.04-2 amd64.tar.zst \ --storage $CT STORAGE \ --memory 57344 \ --swap 0 \ --cores 16 \ --unprivileged 0 \ --hostname $CT HOSTNAME \ --net0 name=eth0,bridge=vmbr0,ip=dhcp \ --rootfs $CT STORAGE:100 \ --features nesting=1 Resource notes: --memory 57344 — 56 GB RAM leaves ~4 GB for Proxmox host --swap 0 — No swap. With unified memory, swapping would thrash the GPU. --cores 16 — All 16 physical cores. --unprivileged 0 — Privileged container. Required for GPU device nodes to have correct ownership. --rootfs 100 — 100 GB minimum. 300 GB recommended if storing models locally. 6.3 Start and set up pct start $CT ID pct enter $CT ID Inside the container apt update && apt upgrade -y apt install -y curl wget git build-essential cmake python3 python3-pip sudo 7. Phase 4: GPU Passthrough to LXC 7.1 Identify device nodes on the host From the Proxmox host : ls -la /dev/dri/ /dev/kfd Expected: crw-rw---- 1 root video 226, 1 /dev/dri/card1 crw-rw---- 1 root render 226, 128 /dev/dri/renderD128 crw-rw---- 1 root render 234, 0 /dev/kfd Note the render group GID: getent group render Output: render:x:993: 7.2 Add device passthrough to container config Edit /etc/pve/lxc/$CT ID.conf on the Proxmox host : cat /etc/pve/lxc/$CT ID.conf << 'EOF' GPU device nodes use wildcard for major 226 to cover all minors lxc.cgroup2.devices.allow: c 226: rwm lxc.cgroup2.devices.allow: c 234: rwm Mount GPU devices into container lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file EOF 7.3 Restart and verify On the host pct stop $CT ID pct start $CT ID Inside the container pct enter $CT ID ls -la /dev/dri/ /dev/kfd Should show root:render and root:video ownership 8. Phase 5: ROCm Installation Inside LXC 8.1 Add the AMD ROCm repository wget -q -O - https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor /etc/apt/trusted.gpg.d/rocm.gpg Ubuntu 24.04 noble — use ROCm 7.2.1 which has noble support echo "deb arch=amd64 https://repo.radeon.com/rocm/apt/7.2.1 noble main" /etc/apt/sources.list.d/rocm.list Pin ROCm packages to prefer the AMD repo cat /etc/apt/preferences.d/rocm-pin << 'EOF' Package: Pin: release o=repo.radeon.com Pin-Priority: 600 EOF apt update 8.2 Install ROCm userspace no DKMS Critical: Do NOT install rocm-dkms . The amdgpu and amdkfd drivers are already in the Proxmox host kernel. apt install -y rocm-hip-sdk rocwmma-dev Also install Vulkan SDK needed by the ROCmFP4 fork build : apt install -y libvulkan-dev vulkan-tools 8.3 Set up memlock limits Critical ROCm needs to lock GPU memory pages. The default ulimit is 8 MB — far too low for LLM models: Add to /etc/security/limits.conf echo 'root soft memlock unlimited' /etc/security/limits.conf echo 'root hard memlock unlimited' /etc/security/limits.conf Apply for current session ulimit -l unlimited 8.4 Verify ROCm /opt/rocm/bin/rocminfo Expected output should show: Name: gfx1151 Marketing Name: Radeon 8060S Graphics Compute Unit: 40 Max Clock Frequency: 2900 MHz Pool Info: Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 125829120 KB 120 GB GTT — full unified memory Also check: /opt/rocm/bin/rocm-smi 9. Phase 6: Build the ROCmFP4 llama.cpp Fork 9.1 Clone the fork cd /root git clone https://github.com/charlie12345/rocmfp4-llama.git cd rocmfp4-llama git checkout mtp-rocmfp4-strix 9.2 Build export PATH=/opt/rocm/bin:/opt/rocm/llvm/bin:$PATH export HIP PATH=/opt/rocm export ROCM PATH=/opt/rocm export ROCM WMMA INCLUDE=/opt/rocm/include export GGML HIP ROCWMMA FATTN=ON export CMAKE HIP ARCHITECTURES=gfx1151 export JOBS=$ nproc The build script references a custom rocWMMA path; we set ROCM WMMA INCLUDE to point to the standard ROCm install location instead. bash scripts/build-strix-rocmfp4-mtp.sh Note: The build script enables Vulkan by default, which requires libvulkan-dev . If you get Vulkan compilation errors e.g., spv namespace issues , reconfigure with -DGGML VULKAN=OFF : cmake -S . -B build-strix-rocmfp4 \ -DCMAKE BUILD TYPE=Release \ -DGGML HIP=ON \ -DGGML HIP ROCWMMA FATTN=ON \ -DGGML HIP FORCE MMQ=ON \ -DGGML VULKAN=OFF \ -DGGML CUDA=OFF \ -DCMAKE HIP ARCHITECTURES=gfx1151 \ -DLLAMA BUILD SERVER=ON cmake --build build-strix-rocmfp4 -j$ nproc --target llama-cli llama-server llama-bench 9.3 Verify HSA OVERRIDE GFX VERSION=11.5.1 \ GGML HIP ENABLE UNIFIED MEMORY=1 \ ./build-strix-rocmfp4/bin/llama-cli --list-devices Expected: Available devices: ROCm0: Radeon 8060S Graphics 122880 MiB, 4096 MiB free 10. Phase 7: Build Stock llama.cpp for Q8 baseline For comparing ROCmFP4 against standard quantization formats: cd /root git clone https://github.com/ggml-org/llama.cpp.git stock-llama cd stock-llama HIPCXX="$ hipconfig -l /clang" \ HIP PATH="$ hipconfig -R " \ cmake -S . -B build \ -DGGML HIP=ON \ -DGPU TARGETS=gfx1151 \ -DGGML HIP ROCWMMA FATTN=ON \ -DGGML HIP NO VMM=ON \ -DGGML HIP MMQ MFMA=ON \ -DCMAKE BUILD TYPE=Release cmake --build build --config Release -j$ nproc --target llama-cli llama-server llama-bench llama-perplexity 11. Phase 8: Download Models 11.1 ROCmFP4 Model 14 GB mkdir -p /models/rocmfp4 cd /models/rocmfp4 wget 'https://huggingface.co/jcbtc/qwopus3.6-27b-v2-chadrock-rocmfp4-mtp/resolve/main/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX LEAN.gguf' Optional: vision projection model wget 'https://huggingface.co/jcbtc/qwopus3.6-27b-v2-chadrock-rocmfp4-mtp/resolve/main/mmproj-F32.mmproj' Expected SHA256: 1f1c0a9d63b9b38b06fe... verify after download 11.2 Qwen3.6-27B Q8 Baseline ~27 GB mkdir -p /models/qwen36-27b cd /models/qwen36-27b wget 'https://huggingface.co/bartowski/Qwen Qwen3.6-27B-GGUF/resolve/main/Qwen3.6-27B-Q8 0.gguf' 11.3 Qwen3.6-27B Q4 K M ~16 GB, optional additional baseline wget 'https://huggingface.co/bartowski/Qwen Qwen3.6-27B-GGUF/resolve/main/Qwen3.6-27B-Q4 K M.gguf' 12. Phase 9: Run Inference 12.1 Critical: The setsid Workaround When running llama-cli inside an lxc-attach shell, the process receives SIGTTOU when trying to set terminal attributes because it’s not the foreground process group. Always use setsid -w to create a new session: Inside the container via lxc-attach setsid -w bash -c ' ulimit -l unlimited HSA OVERRIDE GFX VERSION=11.5.1 \ GGML HIP ENABLE UNIFIED MEMORY=1 \ /root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-cli \ -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX LEAN.gguf \ -ngl 999 \ -fa on \ -dio \ -p "Your prompt here." \ -n 200 ' Alternatively , if you pct enter the container and run interactively not via lxc-attach , the setsid wrapper is not needed. 12.2 ROCmFP4 Model with MTP Full Config HSA OVERRIDE GFX VERSION=11.5.1 \ GGML HIP ENABLE UNIFIED MEMORY=1 \ /root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-server \ -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX LEAN.gguf \ --mmproj /models/rocmfp4/mmproj-F32.mmproj \ --alias qwopus3.6-27b-v2-chadrock \ --host 127.0.0.1 \ --port 8080 \ --jinja \ -c 262144 \ -ngl 999 \ -fa on \ -dev ROCm0 \ -b 512 \ -ub 512 \ -t 16 \ -tb 32 \ -ctk q4 0 \ -ctv q4 0 \ --spec-type draft-mtp \ --spec-draft-device ROCm0 \ --spec-draft-ngl all \ --spec-draft-type-k q4 0 \ --spec-draft-type-v q4 0 \ --spec-draft-n-max 4 \ --spec-draft-n-min 0 \ --spec-draft-p-min 0.0 \ --spec-draft-p-split 0.10 \ --parallel 1 \ --metrics \ --no-mmap 12.3 Q8 Baseline Stock llama.cpp HSA OVERRIDE GFX VERSION=11.5.1 \ GGML HIP ENABLE UNIFIED MEMORY=1 \ /root/stock-llama/build/bin/llama-cli \ -m /models/qwen36-27b/Qwen3.6-27B-Q8 0.gguf \ -ngl 999 \ -fa on \ -dio \ -p "Your prompt here." \ -n 200 12.4 Runtime Flag Reference | Flag | Purpose | -ngl 999 | Offload all layers to GPU | -fa on | Enable flash attention rocWMMA path | -dio | Required for models ~6 GB on gfx1151. Without this, loading hangs. | -dev ROCm0 | Explicitly select the ROCm device | --no-mmap | Disable memory-mapped model loading needed for some setups | --spec-type draft-mtp | Enable Multi-Token Prediction speculative decoding | --spec-draft- | MTP draft model configuration | HSA OVERRIDE GFX VERSION=11.5.1 | Required for ROCmFP4 kernel paths on gfx1151 | GGML HIP ENABLE UNIFIED MEMORY=1 | Required — enables GPU access to system RAM beyond dedicated VRAM | ulimit -l unlimited | Required — ROCm needs to lock GPU memory pages | 13. Phase 10: Benchmarks 13.1 ROCmFP4 Model Benchmark setsid -w bash -c ' ulimit -l unlimited HSA OVERRIDE GFX VERSION=11.5.1 \ GGML HIP ENABLE UNIFIED MEMORY=1 \ /root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-bench \ -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX LEAN.gguf \ -ngl 999 \ -fa on \ -dio \ -n 512 \ -p 512 ' 13.2 Q8 Baseline Benchmark setsid -w bash -c ' ulimit -l unlimited HSA OVERRIDE GFX VERSION=11.5.1 \ GGML HIP ENABLE UNIFIED MEMORY=1 \ /root/stock-llama/build/bin/llama-bench \ -m /models/qwen36-27b/Qwen3.6-27B-Q8 0.gguf \ -ngl 999 \ -fa on \ -dio \ -n 512 \ -p 512 ' 13.3 Large Context Test Test prompt processing at 32K context setsid -w bash -c ' ulimit -l unlimited HSA OVERRIDE GFX VERSION=11.5.1 \ GGML HIP ENABLE UNIFIED MEMORY=1 \ /root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-bench \ -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX LEAN.gguf \ -ngl 999 \ -fa on \ -dio \ -p 32768 \ -n 128 ' 13.4 MTP Speedup Test Compare with and without --spec-type draft-mtp to measure speculative decoding benefit. 14. vLLM Docker Setup Alternative Backend vLLM provides a production-grade serving backend. On Strix Halo, it requires the ROCm Docker image and the HSA OVERRIDE GFX VERSION=11.0.0 workaround. We don’t want to run docker on the Proxmox host; we can run it in the LXC Container. I’ve skipped how to install docker in the LXC – you can follow the normal docker installation guide and it’ll work fine, just make sure you’re installing docker inside the lxc not the Proxmox host . i.e. run the wget get.docker.com script from inside the LXC then run docker run hello-world to make sure docker is working fine inside the LXC. 14.1 Pull the ROCm vLLM image docker pull rocm/vllm:latest 14.2 Run vLLM with the Qwen3.6-27B model HF format The model is already available in safetensors format at /nfs/models/Jackrong/Qwopus3.6-27B-v2/ 52 GB . docker run -it --rm \ --device=/dev/dri \ --device=/dev/kfd \ --group-add=render \ --ipc=host \ --network=host \ -v /nfs/models:/models \ -e HSA OVERRIDE GFX VERSION=11.0.0 \ -e HIP VISIBLE DEVICES=0 \ rocm/vllm:latest \ --model /models/Jackrong/Qwopus3.6-27B-v2 \ --dtype float32 \ --max-model-len 4096 \ --gpu-memory-utilization 0.90 Note: vLLM on Strix Halo requires float32 — bfloat16 causes GPU hangs on current kernel/driver combos. The HSA OVERRIDE GFX VERSION=11.0.0 maps gfx1151 to gfx1100 code paths. 14.3 Benchmark via vLLM docker exec