{"slug": "n5-max-proxmox-strix-halo-with-docker-rocm-fp4-and-mtp-ultimate-setup-guide", "title": "N5 Max: Proxmox Strix Halo with Docker, ROCm FP4 and MTP - Ultimate Setup Guide", "summary": "A new setup guide from Minisforum details running ROCmFP4-quantized models with Multi-Token Prediction (MTP) on the N5 Max AI NAS, powered by the AMD Ryzen AI MAX+ 395 (16C/32T) with Radeon 8060S (gfx1151) and 64 GB unified memory. The guide, last tested on 2026-06-06 with Proxmox VE 9.1.1, kernel 6.17.2-1-pve, and ROCm 7.2.1, covers a full Proxmox LXC and Docker setup, including kernel boot parameters, GPU passthrough, and building a custom llama.cpp fork (charlie12345/rocmfp4-llama, branch mtp-rocmfp4-strix) to achieve native FP4 support and speculative decoding via MTP.", "body_md": "Strix Halo ROCmFP4 + MTP LLM Inference Guide\n\n**Hardware:** AMD Ryzen AI MAX+ 395 (16C/32T) + Radeon 8060S (gfx1151) + 64 GB unified memory\n\n[MINISFORUM N5 MAX AI NAS | AMD Ryzen AI Max+ 395 | 126 TOPS NPU | Radeon™ 8060S graphics](https://s.minisforum.com/4fzJPa5)\n\n**Last tested:** 2026-06-06 (Proxmox VE 9.1.1, kernel 6.17.2-1-pve, ROCm 7.2.1)\n\nBackground\n\nTODO: video goes here, explainer\n\nMy vision for this is to have an “all-in-one” home appliance for AI, Virtual Machines, internet ingress/egress (experimenting with Pangolin on this currently, maybe for a future video).. Home Assistant.. you name it. In a nutshell we’re using Proxmox as our NAS OS then setting up an LXC with full access to the underlying hardware for AI workloads *via* Docker! The best of both worlds. Or something.\n\n[This thing is a beast for any use case.](https://store.minisforum.com/products/minisforum-n5-max-ai-nas)\n\n**Models tested:**\n\n- Qwopus3.6-27B-v2 (ROCmFP4, 14 GB),\n- Qwen3.6-27B (Q8_0, ~27 GB)\n- TODO?\n\nTable of Contents\n\n[Overview & What’s New](#1-overview--whats-new)\n[Architecture Overview](#2-architecture-overview)\n[Prerequisites & System Inventory](#3-prerequisites--system-inventory)\n[Phase 1: Proxmox Host Preparation](#4-phase-1-proxmox-host-preparation)\n[Phase 2: Kernel Boot Parameters (Critical)](#5-phase-2-kernel-boot-parameters-critical)\n[Phase 3: Create the LXC Container](#6-phase-3-create-the-lxc-container)\n[Phase 4: GPU Passthrough to LXC](#7-phase-4-gpu-passthrough-to-lxc)\n[Phase 5: ROCm Installation (Inside LXC)](#8-phase-5-rocm-installation-inside-lxc)\n[Phase 6: Build the ROCmFP4 llama.cpp Fork](#9-phase-6-build-the-rocmfp4-llamacpp-fork)\n[Phase 7: Build Stock llama.cpp (for Q8 baseline)](#10-phase-7-build-stock-llamacpp-for-q8-baseline)\n[Phase 8: Download Models](#11-phase-8-download-models)\n[Phase 9: Run Inference](#12-phase-9-run-inference)\n[Phase 10: Benchmarks](#13-phase-10-benchmarks)\n[vLLM Docker Setup (Alternative Backend)](#14-vllm-docker-setup-alternative-backend)\n[Troubleshooting](#15-troubleshooting)\n[Reference: Key Community Resources](#16-reference-key-community-resources)\n\n1. Overview & What’s New\n\nThis guide covers running **ROCmFP4-quantized models** (a custom 4-bit format with native FP4 tensor layouts optimized for AMD GPUs) on Strix Halo hardware using a custom llama.cpp fork with **Multi-Token Prediction (MTP)** support.\n\nMost of the work was done on the Minisforum N5 Max NAS, but I also validated on the GMK Tek Strix Halo, The Minisforum S1 Max and The Framework Desktop Strix Halo machine they offer.\n\nThis was quite an adventure to get setup. Many thinks to kyz0 and other resources linked throughout this guide.\n\nWhat’s different from standard llama.cpp ROCm guides:\n\n| Feature |\nStandard ROCm guide |\nThis guide |\n**Model format** |\nStandard GGUF (Q4_K_M, Q8_0, etc.) |\nROCmFP4 custom 4-bit GGUF |\n**llama.cpp fork** |\n`ggml-org/llama.cpp` |\n`charlie12345/rocmfp4-llama` (branch `mtp-rocmfp4-strix` ) |\n**MTP support** |\nNo |\nYes — speculative decoding via draft-MTP |\n`HSA_OVERRIDE_GFX_VERSION` |\n`11.0.0` (maps gfx1151→gfx1100) |\n`11.5.1` (native gfx1151 FP4 support) |\n`GGML_HIP_ENABLE_UNIFIED_MEMORY` |\nNot needed |\n**Required** for models > 2 GB |\n`-dio` flag |\nRequired |\n**Required** for models > ~6 GB |\n`ulimit -l unlimited` |\nNot mentioned |\n**Required** — ROCm locks GPU memory |\n`setsid` wrapper |\nNot needed |\nNeeded inside `lxc-attach` shells |\n\n2. Architecture Overview\n\n```\nProxmox Host (kernel 6.17.2-1-pve)\n  ├── amdgpu.ko (in-tree, gfx1151 support)\n  ├── /dev/dri/card1       ───┐\n  ├── /dev/dri/renderD128  ───┤── LXC bind-mount\n  ├── /dev/kfd             ───┤\n  └── /dev/accel/accel0    ───┘  (NPU — not used for LLM today)\n        │\n  LXC Container (Ubuntu 24.04, privileged, 56 GB RAM)\n    ├── ROCm 7.2.1 userspace (no DKMS — host kernel has drivers)\n    ├── rocmfp4-llama fork (built for gfx1151 + ROCmFP4)\n    ├── stock llama.cpp (for Q8 baseline comparisons)\n    ├── Docker (for vLLM ROCm container)\n    └── Models:\n        ├── Qwopus3.6-27B-v2-MTP-ROCmFP4 (14 GB)\n        └── Qwen3.6-27B-Q8_0 (~27 GB)\n```\n\n3. Prerequisites & System Inventory\n\nHardware Requirements\n\n| Component |\nMinimum |\nRecommended |\n**CPU** |\nRyzen AI 300 series |\nRyzen AI MAX+ 395 |\n**RAM** |\n32 GB |\n64 GB (128 GB for 70B models) |\n**Proxmox** |\n8.x |\n9.1+ (Debian 13 base) |\n**Kernel** |\n6.14+ |\n7+ (Proxmox v9 Kernel 7 series) |\n**Storage** |\n100 GB free |\n300 GB+ (for multiple models + builds) |\n\nVerify your hardware\n\n```\n# CPU\ncat /proc/cpuinfo | grep \"model name\" | head -1\n\n# GPU (should show Radeon 8060S)\nlspci | grep -i display\n\n# RAM\ngrep MemTotal /proc/meminfo\n\n# GPU device nodes\nls -la /dev/dri/* /dev/kfd /dev/accel/accel0\n\n# Kernel version\nuname -r\n\n# Proxmox version\npveversion\n```\n\nExpected output\n\n```\nmodel name : AMD RYZEN AI MAX+ 395 w/ Radeon 8060S\n90:00.0 Display controller: AMD/ATI Strix Halo [Radeon 8060S Graphics]\nMemTotal:       63407216 kB  (~60.5 GB)\ncrw-rw---- 1 root render 226, 128 /dev/dri/renderD128\ncrw-rw---- 1 root render 234,   0 /dev/kfd\n6.17.2-1-pve\npve-manager/9.1.1\n```\n\n4. Phase 1: Proxmox Host Preparation\n\n4.1 Enable the no-subscription repository\n\n```\n# Disable the enterprise repo (which will 401 without a subscription key)\nrm /etc/apt/sources.list.d/pve-enterprise.sources\n\n# Add the no-subscription repo\ncat > /etc/apt/sources.list.d/pve-no-subscription.sources << 'EOF'\nTypes: deb\nURIs: http://download.proxmox.com/debian/pve\nSuites: trixie\nComponents: pve-no-subscription\nSigned-By: /usr/share/keyrings/proxmox-archive-keyring.gpg\nEOF\n\n# Also fix the Ceph repo to use no-subscription\ncat > /etc/apt/sources.list.d/ceph.sources << 'EOF'\nTypes: deb\nURIs: http://download.proxmox.com/debian/ceph-squid\nSuites: trixie\nComponents: no-subscription\nSigned-By: /usr/share/keyrings/proxmox-archive-keyring.gpg\nEOF\n\napt update\n```\n\n4.2 Install necessary tools\n\n```\napt install -y curl wget pve-headers build-essential\n```\n\n5. Phase 2: Kernel Boot Parameters (Critical)\n\n5.1 Why these parameters\n\nThe Strix Halo GPU (`gfx1151`\n\n) needs explicit boot parameters to access all 64 GB of unified memory. Without these, ROCm memory allocation is severely limited.\n\n| Parameter |\nPurpose |\n`amd_iommu=off` |\nDisables IOMMU for the GPU. Required for the GPU to access all system RAM as unified memory. Without this, ROCm memory allocation is capped at ~2 GB. |\n`amdgpu.gttsize=131072` |\nSets the GTT (Graphics Translation Table) size to 128 GB. This is a **ceiling** — it tells the GPU it can map up to 128 GB of system RAM. On a 64 GB system, this effectively enables full unified memory access. The value is in megabytes: 131072 MB = 128 GB. |\n`ttm.pages_limit=31457280` |\nIncreases the TTM (Translation Table Manager) page limit. Prevents allocation failures when loading large models. Value = number of 4 KB pages. |\n\n**Note on GTT sizing:** The system has ~60.5 GB of usable RAM. Setting `gttsize=131072`\n\n(128 GB) is safe — it’s just an upper bound. The GPU will only use what’s physically available. For a 64 GB system, a value of ~57344 (56 GB) would also work, but 131072 is harmless and future-proofs the config for systems with more RAM.\n\n5.2 Apply the parameters\n\n```\nsed -i 's/GRUB_CMDLINE_Linux_DEFAULT=\"quiet\"/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet amd_iommu=off amdgpu.gttsize=131072 ttm.pages_limit=31457280\"/' /etc/default/grub\nupdate-grub\nreboot\n```\n\n5.3 Verify after reboot\n\n```\n# Check GTT size (should be ~128 GB)\ncat /sys/class/drm/card1/device/mem_info_gtt_total\n# Expected: 137438953472 (128 GB)\n\n# Check boot params applied\ncat /proc/cmdline | grep -o 'amdgpu.gttsize=[^ ]*\\|amd_iommu=[^ ]*\\|ttm.pages_limit=[^ ]*'\n# Expected: amd_iommu=off amdgpu.gttsize=131072 ttm.pages_limit=31457280\n\n# Verify /dev/kfd exists\nls -la /dev/kfd\n```\n\n6. Phase 3: Create the LXC Container\n\n6.1 Choose a template\n\nUbuntu 24.04 is recommended over Debian 13 for ROCm compatibility:\n\n```\npveam available | grep ubuntu-24\npveam download local ubuntu-24.04-standard_24.04-2_amd64.tar.zst\n```\n\n6.2 Create the container\n\n**Must be privileged** for GPU device access:\n\n```\nCT_ID=100\nCT_HOSTNAME=strix-ai\nCT_STORAGE=local-lvm\n\npct create $CT_ID local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst \\\n  --storage $CT_STORAGE \\\n  --memory 57344 \\\n  --swap 0 \\\n  --cores 16 \\\n  --unprivileged 0 \\\n  --hostname $CT_HOSTNAME \\\n  --net0 name=eth0,bridge=vmbr0,ip=dhcp \\\n  --rootfs $CT_STORAGE:100 \\\n  --features nesting=1\n```\n\n**Resource notes:**\n\n`--memory 57344`\n\n— 56 GB RAM (leaves ~4 GB for Proxmox host)\n`--swap 0`\n\n— No swap. With unified memory, swapping would thrash the GPU.\n`--cores 16`\n\n— All 16 physical cores.\n`--unprivileged 0`\n\n— **Privileged** container. Required for GPU device nodes to have correct ownership.\n`--rootfs 100`\n\n— 100 GB minimum. 300 GB recommended if storing models locally.\n\n6.3 Start and set up\n\n```\npct start $CT_ID\npct enter $CT_ID\n\n# Inside the container\napt update && apt upgrade -y\napt install -y curl wget git build-essential cmake python3 python3-pip sudo\n```\n\n7. Phase 4: GPU Passthrough to LXC\n\n7.1 Identify device nodes on the host\n\nFrom the **Proxmox host**:\n\n```\nls -la /dev/dri/* /dev/kfd\n```\n\nExpected:\n\n```\ncrw-rw---- 1 root video  226,   1 /dev/dri/card1\ncrw-rw---- 1 root render 226, 128 /dev/dri/renderD128\ncrw-rw---- 1 root render 234,   0 /dev/kfd\n```\n\nNote the render group GID:\n\n```\ngetent group render\n# Output: render:x:993:\n```\n\n7.2 Add device passthrough to container config\n\nEdit `/etc/pve/lxc/$CT_ID.conf`\n\non the **Proxmox host**:\n\n```\ncat >> /etc/pve/lxc/$CT_ID.conf << 'EOF'\n\n# GPU device nodes (use wildcard for major 226 to cover all minors)\nlxc.cgroup2.devices.allow: c 226:* rwm\nlxc.cgroup2.devices.allow: c 234:* rwm\n\n# Mount GPU devices into container\nlxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir\nlxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file\nEOF\n```\n\n7.3 Restart and verify\n\n```\n# On the host\npct stop $CT_ID\npct start $CT_ID\n\n# Inside the container\npct enter $CT_ID\nls -la /dev/dri/* /dev/kfd\n# Should show root:render and root:video ownership\n```\n\n8. Phase 5: ROCm Installation (Inside LXC)\n\n8.1 Add the AMD ROCm repository\n\n```\nwget -q -O - https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor > /etc/apt/trusted.gpg.d/rocm.gpg\n\n# Ubuntu 24.04 (noble) — use ROCm 7.2.1 which has noble support\necho \"deb [arch=amd64] https://repo.radeon.com/rocm/apt/7.2.1 noble main\" > /etc/apt/sources.list.d/rocm.list\n\n# Pin ROCm packages to prefer the AMD repo\ncat > /etc/apt/preferences.d/rocm-pin << 'EOF'\nPackage: *\nPin: release o=repo.radeon.com\nPin-Priority: 600\nEOF\n\napt update\n```\n\n8.2 Install ROCm userspace (no DKMS)\n\n**Critical:** Do NOT install `rocm-dkms`\n\n. The amdgpu and amdkfd drivers are already in the Proxmox host kernel.\n\n```\napt install -y rocm-hip-sdk rocwmma-dev\n```\n\nAlso install Vulkan SDK (needed by the ROCmFP4 fork build):\n\n```\napt install -y libvulkan-dev vulkan-tools\n```\n\n8.3 Set up memlock limits (Critical!)\n\nROCm needs to lock GPU memory pages. The default ulimit is 8 MB — far too low for LLM models:\n\n```\n# Add to /etc/security/limits.conf\necho 'root soft memlock unlimited' >> /etc/security/limits.conf\necho 'root hard memlock unlimited' >> /etc/security/limits.conf\n\n# Apply for current session\nulimit -l unlimited\n```\n\n8.4 Verify ROCm\n\n```\n/opt/rocm/bin/rocminfo\n```\n\nExpected output should show:\n\n```\nName:                    gfx1151\nMarketing Name:          Radeon 8060S Graphics\nCompute Unit:            40\nMax Clock Frequency:     2900 MHz\n\nPool Info:\n  Segment:               GLOBAL; FLAGS: COARSE GRAINED\n  Size:                  125829120 KB  (120 GB GTT — full unified memory)\n```\n\nAlso check:\n\n```\n/opt/rocm/bin/rocm-smi\n```\n\n9. Phase 6: Build the ROCmFP4 llama.cpp Fork\n\n9.1 Clone the fork\n\n```\ncd /root\ngit clone https://github.com/charlie12345/rocmfp4-llama.git\ncd rocmfp4-llama\ngit checkout mtp-rocmfp4-strix\n```\n\n9.2 Build\n\n```\nexport PATH=/opt/rocm/bin:/opt/rocm/llvm/bin:$PATH\nexport HIP_PATH=/opt/rocm\nexport ROCM_PATH=/opt/rocm\nexport ROCM_WMMA_INCLUDE=/opt/rocm/include\nexport GGML_HIP_ROCWMMA_FATTN=ON\nexport CMAKE_HIP_ARCHITECTURES=gfx1151\nexport JOBS=$(nproc)\n\n# The build script references a custom rocWMMA path; we set ROCM_WMMA_INCLUDE\n# to point to the standard ROCm install location instead.\nbash scripts/build-strix-rocmfp4-mtp.sh\n```\n\n**Note:** The build script enables Vulkan by default, which requires `libvulkan-dev`\n\n. If you get Vulkan compilation errors (e.g., `spv`\n\nnamespace issues), reconfigure with `-DGGML_VULKAN=OFF`\n\n:\n\n```\ncmake -S . -B build-strix-rocmfp4 \\\n  -DCMAKE_BUILD_TYPE=Release \\\n  -DGGML_HIP=ON \\\n  -DGGML_HIP_ROCWMMA_FATTN=ON \\\n  -DGGML_HIP_FORCE_MMQ=ON \\\n  -DGGML_VULKAN=OFF \\\n  -DGGML_CUDA=OFF \\\n  -DCMAKE_HIP_ARCHITECTURES=gfx1151 \\\n  -DLLAMA_BUILD_SERVER=ON\ncmake --build build-strix-rocmfp4 -j$(nproc) --target llama-cli llama-server llama-bench\n```\n\n9.3 Verify\n\n```\nHSA_OVERRIDE_GFX_VERSION=11.5.1 \\\nGGML_HIP_ENABLE_UNIFIED_MEMORY=1 \\\n./build-strix-rocmfp4/bin/llama-cli --list-devices\n```\n\nExpected:\n\n```\nAvailable devices:\n  ROCm0: Radeon 8060S Graphics (122880 MiB, 4096 MiB free)\n```\n\n10. Phase 7: Build Stock llama.cpp (for Q8 baseline)\n\nFor comparing ROCmFP4 against standard quantization formats:\n\n```\ncd /root\ngit clone https://github.com/ggml-org/llama.cpp.git stock-llama\ncd stock-llama\n\nHIPCXX=\"$(hipconfig -l)/clang\" \\\nHIP_PATH=\"$(hipconfig -R)\" \\\ncmake -S . -B build \\\n  -DGGML_HIP=ON \\\n  -DGPU_TARGETS=gfx1151 \\\n  -DGGML_HIP_ROCWMMA_FATTN=ON \\\n  -DGGML_HIP_NO_VMM=ON \\\n  -DGGML_HIP_MMQ_MFMA=ON \\\n  -DCMAKE_BUILD_TYPE=Release\n\ncmake --build build --config Release -j$(nproc) --target llama-cli llama-server llama-bench llama-perplexity\n```\n\n11. Phase 8: Download Models\n\n11.1 ROCmFP4 Model (14 GB)\n\n```\nmkdir -p /models/rocmfp4\ncd /models/rocmfp4\n\nwget 'https://huggingface.co/jcbtc/qwopus3.6-27b-v2-chadrock-rocmfp4-mtp/resolve/main/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX_LEAN.gguf'\n\n# Optional: vision projection model\nwget 'https://huggingface.co/jcbtc/qwopus3.6-27b-v2-chadrock-rocmfp4-mtp/resolve/main/mmproj-F32.mmproj'\n```\n\nExpected SHA256: `1f1c0a9d63b9b38b06fe...`\n\n(verify after download)\n\n11.2 Qwen3.6-27B Q8 Baseline (~27 GB)\n\n```\nmkdir -p /models/qwen36-27b\ncd /models/qwen36-27b\n\nwget 'https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF/resolve/main/Qwen3.6-27B-Q8_0.gguf'\n```\n\n11.3 Qwen3.6-27B Q4_K_M (~16 GB, optional additional baseline)\n\n```\nwget 'https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF/resolve/main/Qwen3.6-27B-Q4_K_M.gguf'\n```\n\n12. Phase 9: Run Inference\n\n12.1 Critical: The `setsid`\n\nWorkaround\n\nWhen running llama-cli inside an `lxc-attach`\n\nshell, the process receives `SIGTTOU`\n\nwhen trying to set terminal attributes because it’s not the foreground process group. **Always use **`setsid -w`\n\nto create a new session:\n\n```\n# Inside the container via lxc-attach\nsetsid -w bash -c '\nulimit -l unlimited\nHSA_OVERRIDE_GFX_VERSION=11.5.1 \\\nGGML_HIP_ENABLE_UNIFIED_MEMORY=1 \\\n/root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-cli \\\n  -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX_LEAN.gguf \\\n  -ngl 999 \\\n  -fa on \\\n  -dio \\\n  -p \"Your prompt here.\" \\\n  -n 200\n'\n```\n\n**Alternatively**, if you `pct enter`\n\nthe container and run interactively (not via `lxc-attach`\n\n), the `setsid`\n\nwrapper is not needed.\n\n12.2 ROCmFP4 Model with MTP (Full Config)\n\n```\nHSA_OVERRIDE_GFX_VERSION=11.5.1 \\\nGGML_HIP_ENABLE_UNIFIED_MEMORY=1 \\\n/root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-server \\\n  -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX_LEAN.gguf \\\n  --mmproj /models/rocmfp4/mmproj-F32.mmproj \\\n  --alias qwopus3.6-27b-v2-chadrock \\\n  --host 127.0.0.1 \\\n  --port 8080 \\\n  --jinja \\\n  -c 262144 \\\n  -ngl 999 \\\n  -fa on \\\n  -dev ROCm0 \\\n  -b 512 \\\n  -ub 512 \\\n  -t 16 \\\n  -tb 32 \\\n  -ctk q4_0 \\\n  -ctv q4_0 \\\n  --spec-type draft-mtp \\\n  --spec-draft-device ROCm0 \\\n  --spec-draft-ngl all \\\n  --spec-draft-type-k q4_0 \\\n  --spec-draft-type-v q4_0 \\\n  --spec-draft-n-max 4 \\\n  --spec-draft-n-min 0 \\\n  --spec-draft-p-min 0.0 \\\n  --spec-draft-p-split 0.10 \\\n  --parallel 1 \\\n  --metrics \\\n  --no-mmap\n```\n\n12.3 Q8 Baseline (Stock llama.cpp)\n\n```\nHSA_OVERRIDE_GFX_VERSION=11.5.1 \\\nGGML_HIP_ENABLE_UNIFIED_MEMORY=1 \\\n/root/stock-llama/build/bin/llama-cli \\\n  -m /models/qwen36-27b/Qwen3.6-27B-Q8_0.gguf \\\n  -ngl 999 \\\n  -fa on \\\n  -dio \\\n  -p \"Your prompt here.\" \\\n  -n 200\n```\n\n12.4 Runtime Flag Reference\n\n| Flag |\nPurpose |\n`-ngl 999` |\nOffload all layers to GPU |\n`-fa on` |\nEnable flash attention (rocWMMA path) |\n`-dio` |\n**Required** for models > ~6 GB on gfx1151. Without this, loading hangs. |\n`-dev ROCm0` |\nExplicitly select the ROCm device |\n`--no-mmap` |\nDisable memory-mapped model loading (needed for some setups) |\n`--spec-type draft-mtp` |\nEnable Multi-Token Prediction speculative decoding |\n`--spec-draft-*` |\nMTP draft model configuration |\n`HSA_OVERRIDE_GFX_VERSION=11.5.1` |\nRequired for ROCmFP4 kernel paths on gfx1151 |\n`GGML_HIP_ENABLE_UNIFIED_MEMORY=1` |\n**Required** — enables GPU access to system RAM beyond dedicated VRAM |\n`ulimit -l unlimited` |\n**Required** — ROCm needs to lock GPU memory pages |\n\n13. Phase 10: Benchmarks\n\n13.1 ROCmFP4 Model Benchmark\n\n```\nsetsid -w bash -c '\nulimit -l unlimited\nHSA_OVERRIDE_GFX_VERSION=11.5.1 \\\nGGML_HIP_ENABLE_UNIFIED_MEMORY=1 \\\n/root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-bench \\\n  -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX_LEAN.gguf \\\n  -ngl 999 \\\n  -fa on \\\n  -dio \\\n  -n 512 \\\n  -p 512\n'\n```\n\n13.2 Q8 Baseline Benchmark\n\n```\nsetsid -w bash -c '\nulimit -l unlimited\nHSA_OVERRIDE_GFX_VERSION=11.5.1 \\\nGGML_HIP_ENABLE_UNIFIED_MEMORY=1 \\\n/root/stock-llama/build/bin/llama-bench \\\n  -m /models/qwen36-27b/Qwen3.6-27B-Q8_0.gguf \\\n  -ngl 999 \\\n  -fa on \\\n  -dio \\\n  -n 512 \\\n  -p 512\n'\n```\n\n13.3 Large Context Test\n\n```\n# Test prompt processing at 32K context\nsetsid -w bash -c '\nulimit -l unlimited\nHSA_OVERRIDE_GFX_VERSION=11.5.1 \\\nGGML_HIP_ENABLE_UNIFIED_MEMORY=1 \\\n/root/rocmfp4-llama/build-strix-rocmfp4/bin/llama-bench \\\n  -m /models/rocmfp4/Qwopus3.6-27B-v2-MTP-BF16-to-ROCmFP4-STRIX_LEAN.gguf \\\n  -ngl 999 \\\n  -fa on \\\n  -dio \\\n  -p 32768 \\\n  -n 128\n'\n```\n\n13.4 MTP Speedup Test\n\nCompare with and without `--spec-type draft-mtp`\n\nto measure speculative decoding benefit.\n\n14. vLLM Docker Setup (Alternative Backend)\n\nvLLM provides a production-grade serving backend. On Strix Halo, it requires the ROCm Docker image and the `HSA_OVERRIDE_GFX_VERSION=11.0.0`\n\nworkaround.\n\nWe don’t want to run docker on the Proxmox host; we can run it in the LXC Container. I’ve skipped how to install docker in the LXC – you can follow the normal docker installation guide and it’ll work fine, just make sure you’re installing docker *inside* the lxc (not the Proxmox host).\n\ni.e. run the `wget get.docker.com `\n\nscript from inside the LXC then run `docker run hello-world`\n\nto make sure docker is working fine inside the LXC.\n\n14.1 Pull the ROCm vLLM image\n\n```\ndocker pull rocm/vllm:latest\n```\n\n14.2 Run vLLM with the Qwen3.6-27B model (HF format)\n\nThe model is already available in safetensors format at `/nfs/models/Jackrong/Qwopus3.6-27B-v2/`\n\n(52 GB).\n\n```\ndocker run -it --rm \\\n  --device=/dev/dri \\\n  --device=/dev/kfd \\\n  --group-add=render \\\n  --ipc=host \\\n  --network=host \\\n  -v /nfs/models:/models \\\n  -e HSA_OVERRIDE_GFX_VERSION=11.0.0 \\\n  -e HIP_VISIBLE_DEVICES=0 \\\n  rocm/vllm:latest \\\n  --model /models/Jackrong/Qwopus3.6-27B-v2 \\\n  --dtype float32 \\\n  --max-model-len 4096 \\\n  --gpu-memory-utilization 0.90\n```\n\n**Note:** vLLM on Strix Halo requires `float32`\n\n— bfloat16 causes GPU hangs on current kernel/driver combos. The `HSA_OVERRIDE_GFX_VERSION=11.0.0`\n\nmaps gfx1151 to gfx1100 code paths.\n\n14.3 Benchmark via vLLM\n\n```\ndocker exec <container-id> python3 -m vllm.benchmarks.benchmark_throughput \\\n  --model /models/Jackrong/Qwopus3.6-27B-v2 \\\n  --dtype float32 \\\n  --num-prompts 100 \\\n  --input-len 512 \\\n  --output-len 256\n```\n\n15. Troubleshooting\n\nIf you have a problem not listed here **please** do comment below so I can improve the guide!\n\n15.1 Model loading hangs with no output\n\n**Symptoms:** `llama-cli`\n\nstarts but produces zero output and never completes.\n\n**Causes and fixes:**\n\n| Cause |\nFix |\n**SIGTTOU** (inside `lxc-attach` ) |\nUse `setsid -w` wrapper, or run interactively via `pct enter` |\n**Missing **`-dio` flag |\nAdd `-dio` to the command line |\n**Missing **`ulimit -l unlimited` |\nSet `ulimit -l unlimited` before running |\n**Missing **`GGML_HIP_ENABLE_UNIFIED_MEMORY=1` |\nSet this environment variable |\n\n15.2 “ROCk module is not loaded” in rocminfo\n\nThe `amdgpu`\n\nmodule is loaded but the ROCm compute path (`amdkfd`\n\n) isn’t initialized.\n\n```\n# On the Proxmox host\ncat /boot/config-$(uname -r) | grep CONFIG_HSA_AMD\n# Should show: CONFIG_HSA_AMD=y\n\n# If it's a module, load it\nmodprobe amdkfd\n\n# Check /dev/kfd exists\nls -la /dev/kfd\n```\n\n15.3 “Not enough memory” errors\n\nCheck GTT size:\n\n```\ncat /sys/class/drm/card1/device/mem_info_gtt_total\n```\n\nIf this shows less than ~60 GB, the `amdgpu.gttsize`\n\nkernel parameter isn’t being applied:\n\n```\ncat /proc/cmdline | grep gttsize\n```\n\nIf missing, re-run `update-grub`\n\nand reboot.\n\n15.4 Container can’t see GPU devices\n\n```\n# On the host\nls -la /dev/dri/* /dev/kfd\n\n# Check container config\ncat /etc/pve/lxc/$CT_ID.conf | grep -E 'lxc\\.(cgroup|mount)'\n\n# Restart the container\npct stop $CT_ID\npct start $CT_ID\n```\n\n15.5 Permission denied on /dev/kfd inside container\n\nEnsure the container is **privileged** (`unprivileged: 0`\n\nin config). If using an unprivileged container, add:\n\n```\nlxc.idmap: g 993 993 1\n```\n\n15.6 Vulkan compilation errors during build\n\nThe ROCmFP4 fork’s build script enables Vulkan by default. If you get `spv`\n\nnamespace errors, rebuild with Vulkan disabled:\n\n```\ncmake -S . -B build-strix-rocmfp4 \\\n  -DGGML_VULKAN=OFF \\\n  # ... other flags ...\n```\n\n15.7 Slow performance\n\n- Verify you’re using the ROCm backend, not CPU fallback: the log should show\n`ggml_rocm_init`\n\n- Check that\n`GPU_TARGETS=gfx1151`\n\nwas used during build\n- Verify flash attention is enabled:\n`-fa on`\n\n- Check GPU clock speed:\n`rocm-smi`\n\n- For first runs, kernel compilation may slow things down — subsequent runs will be faster\n\n16. Reference: Key Community Resources\n\nAppendix A: Quick-Start Script\n\n**DANGER**: I am providing this for reference, but again, if it works (or doesn’t) please comment below. Thanks!\n\n``` bash\n#!/bin/bash\n# strix-halo-rocmfp4-setup.sh — Run on Proxmox host as root\n# WARNING: Review each section before running. This will reboot the host.\n\nset -euo pipefail\n\nCT_ID=${1:-100}\nCT_NAME=${2:-strix-ai}\n\necho \"=== Step 1: Add no-subscription repos ===\"\nrm -f /etc/apt/sources.list.d/pve-enterprise.sources\ncat > /etc/apt/sources.list.d/pve-no-subscription.sources << 'EOF'\nTypes: deb\nURIs: http://download.proxmox.com/debian/pve\nSuites: trixie\nComponents: pve-no-subscription\nSigned-By: /usr/share/keyrings/proxmox-archive-keyring.gpg\nEOF\napt update\n\necho \"=== Step 2: Add boot parameters ===\"\nsed -i 's/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet\"/GRUB_CMDLINE_LINUX_DEFAULT=\"quiet amd_iommu=off amdgpu.gttsize=131072 ttm.pages_limit=31457280\"/' /etc/default/grub\nupdate-grub\n\necho \"=== Step 3: Download LXC template ===\"\npveam download local ubuntu-24.04-standard_24.04-2_amd64.tar.zst\n\necho \"=== Step 4: Create container ===\"\npct create $CT_ID local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst \\\n  --storage local-lvm \\\n  --memory 57344 \\\n  --swap 0 \\\n  --cores 16 \\\n  --unprivileged 0 \\\n  --hostname $CT_NAME \\\n  --net0 name=eth0,bridge=vmbr0,ip=dhcp \\\n  --rootfs local-lvm:100 \\\n  --features nesting=1\n\necho \"=== Step 5: Add GPU passthrough ===\"\ncat >> /etc/pve/lxc/$CT_ID.conf << 'EOF'\n\nlxc.cgroup2.devices.allow: c 226:* rwm\nlxc.cgroup2.devices.allow: c 234:* rwm\nlxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir\nlxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file\nEOF\n\necho \"=== Done! Reboot to apply the new kernel ===\"\necho \"After reboot:\"\necho \"  1. pct start $CT_ID\"\necho \"  2. pct enter $CT_ID\"\necho \"  3. Run the inside-container setup (ROCm + llama.cpp builds)\"\necho \"\"\necho \"Rebooting in 10 seconds... Ctrl+C to cancel.\"\nsleep 10\nreboot\n```\n\nAppendix B: Okay, that’s setup? Where’s my home lab?\n\nSo from here, you can setup containers or VMs or run Docker for whatever you like. Check out [Proxmox VE Helper-Scripts](https://community-scripts.org/) to get some ideas.\n\n[Jellyfin Media Server | Proxmox VE Helper Scripts](https://community-scripts.org/scripts/jellyfin)\n\n[Frigate | Proxmox VE Helper Scripts](https://community-scripts.org/scripts/frigate)\n\n[Home Assistant OS | Proxmox VE Helper Scripts](https://community-scripts.org/scripts/haos-vm)\n\n…and so many more!", "url": "https://wpnews.pro/news/n5-max-proxmox-strix-halo-with-docker-rocm-fp4-and-mtp-ultimate-setup-guide", "canonical_source": "https://forum.level1techs.com/t/n5-max-proxmox-strix-halo-with-docker-rocm-fp4-and-mtp-ultimate-setup-guide/251182#post_14", "published_at": "2026-08-24 14:09:28+00:00", "updated_at": "2026-08-24 14:14:29.169123+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["Minisforum", "AMD", "Ryzen AI MAX+ 395", "Radeon 8060S", "Proxmox VE", "ROCm", "llama.cpp", "charlie12345"], "alternates": {"html": "https://wpnews.pro/news/n5-max-proxmox-strix-halo-with-docker-rocm-fp4-and-mtp-ultimate-setup-guide", "markdown": "https://wpnews.pro/news/n5-max-proxmox-strix-halo-with-docker-rocm-fp4-and-mtp-ultimate-setup-guide.md", "text": "https://wpnews.pro/news/n5-max-proxmox-strix-halo-with-docker-rocm-fp4-and-mtp-ultimate-setup-guide.txt", "jsonld": "https://wpnews.pro/news/n5-max-proxmox-strix-halo-with-docker-rocm-fp4-and-mtp-ultimate-setup-guide.jsonld"}}