{"slug": "running-whisper-llms-on-an-amd-npu-under-linux", "title": "Running Whisper + LLMs on an AMD NPU under Linux", "summary": "A developer successfully ran OpenAI's Whisper large-v3-turbo transcription model and an LLM on an AMD NPU under Linux, achieving a real-time factor of approximately 0.18 on an MSI Stealth A16 AI+ laptop with a Ryzen AI 9 365 processor. The setup used the mainline amdxdna kernel driver, XRT tools, and the FastFlowLM runtime, bypassing the typical Windows-centric Ryzen AI SDK and ONNX Runtime VitisAI path. The developer highlighted the need to set an unlimited memlock rlimit and noted the process took about 20 minutes once the correct stack was identified.", "body_md": "TL;DR— On a MSI Stealth A16 AI+ (Ryzen AI 9 365, XDNA2 NPU) running Arch,\n\nI got OpenAI's`whisper-large-v3-turbo`\n\ntranscribing on theNPU— not the\n\nCPU, not the GPU — atRTF ≈ 0.18(a 30 s clip in ~5.2 s) for roughly a\n\ntenth of the energythe same job costs on the CPU, plus an LLM answering\n\non the same NPU through an OpenAI-compatible API. The\n\nwhole path is local and offline. This is the write-up of the driver stack,\n\nthe one real gotcha (memlock), and the runtime that made it a 20-minute job\n\ninstead of a weekend.\n\nAMD's \"Ryzen AI\" NPU (the XDNA / XDNA2 block in Phoenix / Hawk Point / Strix\n\nPoint laptops) is marketed almost entirely around Windows: the Ryzen AI SDK,\n\nthe ONNX Runtime **VitisAI** execution provider, Lemonade, and the demos all\n\nassume you're on Windows with the official stack. On Linux the picture in early\n\n2026 is better than most people think — the NPU driver has been **in the\nmainline kernel** as\n\n`amdxdna`\n\nsince 6.14 — but the \"load a real model and runHere's what actually worked, end to end.\n\n| Part | Detail |\n|---|---|\n| Laptop | MSI Stealth A16 AI+ A3HVGG |\n| APU | AMD Ryzen AI 9 365 (Strix Point) |\n| NPU | XDNA2, 8 columns, exposed as `/dev/accel/accel0`\n|\n| NPU firmware | `1.1.2.64` |\n| Kernel | 7.1.9-arch1 (`amdxdna` in-tree) |\n| OS | Omarchy (Arch Linux) |\n\nAMD quotes the Strix Point NPU at [up to 50 TOPS, INT8](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-9-365.html).\n\nThree pieces have to be in place before any runtime can touch the NPU:\n\n`amdxdna`\n\n`/dev/accel/accel0`\n\n. Check it's bound:\n\n``` bash\n   $ ls /dev/accel/\n   accel0\n   $ dmesg | grep -i amdxdna\n```\n\n`xrt-plugin-amdxdna`\n\n`extra`\n\n:\n\n``` bash\n   $ sudo pacman -S xrt xrt-plugin-amdxdna\n   $ xrt-smi examine\n   ...\n   XRT\n     Version              : 2.21.75\n     NPU Firmware Version  : 1.1.2.64\n\n   Device(s) Present\n   |BDF             |Name          |\n   |----------------|--------------|\n   |[0000:66:00.1]  |RyzenAI-npu4  |\n```\n\nYou want a `Device(s) Present`\n\nline with a `RyzenAI-npu*`\n\nname. If XRT is\n\ninstalled but the plugin isn't, `xrt-smi`\n\nruns but that table is empty.\n\n`xrt-smi examine --report platform`\n\nthen shows `Total Columns : 8`\n\n— the\n\nXDNA2 array this SoC exposes.\n\n`xrt 2.21.75`\n\n, `xrt-plugin-amdxdna`\n\n(same\nrelease), NPU firmware `1.1.2.64`\n\n, and `flm validate`\n\nreporting the `amdxdna`\n\ndriver interface as The NPU runtime pins model weights into physical RAM, so the calling user needs\n\nan **unlimited memlock rlimit**. The default (usually 8 MiB or 64 MiB) is nowhere\n\nnear enough and the failure mode is an unhelpful allocation error deep in the\n\nruntime.\n\n``` bash\n$ sudo tee -a /etc/security/limits.conf <<< \"$USER soft memlock unlimited\"\n$ sudo tee -a /etc/security/limits.conf <<< \"$USER hard memlock unlimited\"\n# log out and back in\n```\n\nYou want to see this afterwards:\n\n``` bash\n$ ulimit -l\nunlimited\n```\n\nThe \"official\" Linux route is: build ONNX Runtime with the VitisAI EP, install\n\nthe Ryzen AI SDK bits, quantize your model to the NPU's format, wrangle a Python\n\nvenv full of `onnxruntime-vitisai`\n\nand Vitis tooling. It's a lot, and much of it\n\nis Windows-first.\n\n** FastFlowLM** (\n\n`flm`\n\n) skips all of that. It's a`libwhisper_npu.so`\n\nin the package itself:\n\n``` bash\n$ ls /usr/share/flm/xclbins/\nencoder_attn  encoder_dequant  encoder_mm  whisper_head  ...\n$ flm --version\nFLM v1.0.2\n```\n\nBecause the kernels are bundled, **there is no onnxruntime-vitisai / Ryzen AI SDK\nvenv to build**. (Arch's stock\n\n`python-onnxruntime-cpu`\n\nonly has`CPUExecutionProvider`\n\nanyway — irrelevant here.) On Arch: `sudo pacman -S`\n\nfastflowlm\n\n.Validate the whole stack in one shot:\n\n``` bash\n$ flm validate\n[Linux]  Kernel: 7.1.9-arch1-2\n[Linux]  NPU: /dev/accel/accel0 with 8 columns\n[Linux]  NPU FW Version: 1.1.2.64\n[Linux]  amdxdna version: 0.8\n[Linux]  Memlock Limit: infinity\n```\n\nAll green = ready. If `Memlock Limit`\n\nsays anything other than `infinity`\n\n, go\n\nback to the limits.conf step.\n\nPull the model — `whisper-v3:turbo`\n\nis `large-v3-turbo`\n\nquantized for XDNA2:\n\n``` php\n$ flm pull whisper-v3:turbo\n# ~650 MB: model.q4nx + tokenizers -> ~/.config/flm/models/Whisper-V3-Turbo-NPU2/\n```\n\nServe it. On FLM 1.0.2+ Whisper loads **standalone** — older docs claimed you had\n\nto co-load an LLM, but you don't:\n\n``` bash\n$ flm serve --asr 1          # OpenAI-compatible server on :52625\n```\n\nTranscribe over the HTTP API (anything `ffmpeg`\n\ncan decode — wav/mp3/ogg/m4a/flac):\n\n``` bash\n$ curl http://127.0.0.1:52625/v1/audio/transcriptions \\\n    -F \"file=@audio.ogg\" \\\n    -F \"model=whisper-v3\"\n```\n\nThere's also a CLI path: `flm run <model> --asr 1`\n\n, then `/input \"clip.mp3\"`\n\nin\n\nthe chat prompt.\n\nBenchmarked with the bundled `bench.py`\n\n— 10 runs, first 2 discarded as warm-up,\n\naudio length read from the file via `ffprobe`\n\nso the RTF is honest and\n\nreproducible:\n\n| Metric | Value |\n|---|---|\n| Audio length | 30.0 s (JFK, Rice University speech excerpt) |\n| Transcription wall time (warm) |\n5.2 s (σ 0.04 s within a run; 5.17–5.6 s across sessions) |\nReal-time factor (RTF) |\n≈ 0.17–0.19 |\n| Transcript accuracy | correct, verbatim |\n\nRoughly **5–6× faster than real time**. Within a single benchmark the spread is\n\nunder 1%; between sessions the mean drifts a few hundred ms with machine\n\ntemperature and background load.\n\nTwo checks. First, the FLM log prints `[NPU Locked!]`\n\nwhen a job starts and\n\n`[NPU Lock Released!]`\n\nwhen it finishes. Second — and more convincing — sample\n\nsystem load while the benchmark runs and see that nothing else is doing the\n\nwork:\n\n| Device | Idle baseline | During 10 transcriptions |\n|---|---|---|\n| CPU (20 threads, system-wide) | 2.2 % |\n4.3 % mean, 14.4 % peak |\n| iGPU (Radeon 890M) | 7 % |\n10 % mean, 15 % peak |\n| dGPU (RTX 4070) | 0 % | 0 % |\n\nThe CPU rises about two points over idle — that's the `curl`\n\n/harness overhead and\n\nthe server's I/O thread, not inference. The iGPU delta is desktop compositing\n\n(Hyprland renders on the 890M), and the discrete GPU is never touched at all.\n\nThe 30 seconds of audio is being processed somewhere that doesn't show up in any\n\nof these three counters, which is exactly the point: the CPU and both GPUs stay\n\nfree while the NPU works.\n\nSpeed alone isn't the story — `whisper-large-v3-turbo`\n\nhas a tiny decoder and\n\nruns fine on CPU. So I built `whisper.cpp`\n\nfrom source (the Arch package's ggml\n\nbackend is currently broken) and ran the **same 30 s clip through the same\nmodel** on the CPU, tuned to 16 threads, reading the RAPL energy counters\n\n`/sys/class/powercap/intel-rapl:0`\n\n) around every run.\nNPU (FastFlowLM) |\nCPU (whisper.cpp, `-t 16` ) |\n|\n|---|---|---|\n| Wall time (30 s clip) | ~5.3 s | ~6.5 s |\n| RTF | 0.18 | 0.22 |\n| CPU-package power while running | ~20 W | ~73 W |\nCPU-core power while running |\n~0.8 W | ~10 W |\nEnergy per transcription, over idle |\n~45 J |\n~410 J |\n| Energy per transcription, total package | ~105 J | ~478 J |\n\nThe wall-clock win is modest — about 25%. The **energy** difference is the\n\npoint: transcribing that clip on the NPU costs roughly **an order of magnitude\nless energy** than doing it on the CPU (~45 J vs ~410 J above idle). Package\n\n*(Measured on a live desktop, so absolute wattages drift a few watts between\nruns with background activity — \"energy over idle\" is the stable figure and what\nthe comparison rests on. *\n\n`whisper-cli`\n\nalso reloads the 1.6 GB model each run,\nwhich pads its wall time slightly but not its energy. Both harnesses are in the\n`bench.py --power`\n\nfor the NPU column, `bench_cpu.py`\n\nfor the CPU column.)FLM serves LLMs on the NPU through the same OpenAI-compatible surface. Its model\n\ncatalogue covers the usual small-to-mid open weights:\n\n``` bash\n$ flm list\n  gemma3:1b        ✅\n  qwen3:1.7b       ⏬\n  llama3.2:3b      ⏬\n  phi4-mini-it:4b  ⏬\n  deepseek-r1:8b   ⏬\n  gpt-oss:20b      ⏬\n  whisper-v3:turbo ✅\n  ...\n```\n\nOne server can expose **both** ASR and chat:\n\n``` bash\n$ flm serve gemma3:1b --asr 1\n$ curl http://127.0.0.1:52625/v1/chat/completions \\\n    -H 'content-type: application/json' \\\n    -d '{\"model\":\"gemma3:1b\",\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]}'\n```\n\nSo `/v1/audio/transcriptions`\n\nand `/v1/chat/completions`\n\nare both live on\n\n`:52625`\n\nfrom a single process on the NPU.\n\nWith the endpoint working, the rest is glue:\n\n— a zero-dependency\n\n`npu-whisper`\n\n`flm serve --asr 1`\n\n`curl`\n\ns the transcript, and leaves the`--json`\n\n, `--status`\n\n, `--stop`\n\n. — a\n\n`local-ai-assistant`\n\n`pw-record → Whisper (NPU) → LLM (NPU) → piper TTS → speaker`\n\n. One FLM server`chat`\n\nkeeps conversation history across runs.Both are deliberately small — the interesting work was getting the NPU to do the\n\ninference, not the plumbing on top.\n\n`flm validate`\n\ncatches it.`.msi`\n\n— 1.0.2 is fine to stay on for Linux.`.q4nx`\n\nweights + bundled xclbins\nare FastFlowLM's; you can't point llama.cpp or vanilla ONNX Runtime at them.`whisper.cpp`\n\non CPU is far slower for\n`large-v3-turbo`\n\n.`amdxdna`\n\n+ XRT + FLM combination works well but you have to assemble it\nyourself. That's the gap this post is trying to close.\n\n```\n# 1. driver stack\nsudo pacman -S xrt xrt-plugin-amdxdna fastflowlm\nsudo tee -a /etc/security/limits.conf <<< \"$USER soft memlock unlimited\"\nsudo tee -a /etc/security/limits.conf <<< \"$USER hard memlock unlimited\"\n# log out / back in\nflm validate            # want: all green, Memlock Limit: infinity\n\n# 2. models\nflm pull whisper-v3:turbo\nflm pull gemma3:1b\n\n# 3. run\nflm serve gemma3:1b --asr 1\ncurl http://127.0.0.1:52625/v1/audio/transcriptions -F file=@clip.wav -F model=whisper-v3\n```\n\nRequirements: a Ryzen AI (XDNA / XDNA2) laptop, kernel ≥ 6.14 with `amdxdna`\n\n,\n\nand the memlock bump.", "url": "https://wpnews.pro/news/running-whisper-llms-on-an-amd-npu-under-linux", "canonical_source": "https://dev.to/jac-76/running-whisper-llms-on-an-amd-npu-under-linux-2o1h", "published_at": "2026-09-03 12:40:54+00:00", "updated_at": "2026-09-03 12:55:14.915528+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools", "ai-infrastructure"], "entities": ["AMD", "OpenAI", "MSI", "Ryzen AI 9 365", "XDNA2", "FastFlowLM", "amdxdna", "XRT"], "alternates": {"html": "https://wpnews.pro/news/running-whisper-llms-on-an-amd-npu-under-linux", "markdown": "https://wpnews.pro/news/running-whisper-llms-on-an-amd-npu-under-linux.md", "text": "https://wpnews.pro/news/running-whisper-llms-on-an-amd-npu-under-linux.txt", "jsonld": "https://wpnews.pro/news/running-whisper-llms-on-an-amd-npu-under-linux.jsonld"}}