Running Whisper + LLMs on an AMD NPU under Linux

A developer successfully ran OpenAI's Whisper large-v3-turbo transcription model and an LLM on an AMD NPU under Linux, achieving a real-time factor of approximately 0.18 on an MSI Stealth A16 AI+ laptop with a Ryzen AI 9 365 processor. The setup used the mainline amdxdna kernel driver, XRT tools, and the FastFlowLM runtime, bypassing the typical Windows-centric Ryzen AI SDK and ONNX Runtime VitisAI path. The developer highlighted the need to set an unlimited memlock rlimit and noted the process took about 20 minutes once the correct stack was identified.

TL;DR— On a MSI Stealth A16 AI+ Ryzen AI 9 365, XDNA2 NPU running Arch, I got OpenAI's whisper-large-v3-turbo transcribing on theNPU— not the CPU, not the GPU — atRTF ≈ 0.18 a 30 s clip in ~5.2 s for roughly a tenth of the energythe same job costs on the CPU, plus an LLM answering on the same NPU through an OpenAI-compatible API. The whole path is local and offline. This is the write-up of the driver stack, the one real gotcha memlock , and the runtime that made it a 20-minute job instead of a weekend. AMD's "Ryzen AI" NPU the XDNA / XDNA2 block in Phoenix / Hawk Point / Strix Point laptops is marketed almost entirely around Windows: the Ryzen AI SDK, the ONNX Runtime VitisAI execution provider, Lemonade, and the demos all assume you're on Windows with the official stack. On Linux the picture in early 2026 is better than most people think — the NPU driver has been in the mainline kernel as amdxdna since 6.14 — but the "load a real model and runHere's what actually worked, end to end. | Part | Detail | |---|---| | Laptop | MSI Stealth A16 AI+ A3HVGG | | APU | AMD Ryzen AI 9 365 Strix Point | | NPU | XDNA2, 8 columns, exposed as /dev/accel/accel0 | | NPU firmware | 1.1.2.64 | | Kernel | 7.1.9-arch1 amdxdna in-tree | | OS | Omarchy Arch Linux | AMD quotes the Strix Point NPU at up to 50 TOPS, INT8 https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-9-365.html . Three pieces have to be in place before any runtime can touch the NPU: amdxdna /dev/accel/accel0 . Check it's bound: bash $ ls /dev/accel/ accel0 $ dmesg | grep -i amdxdna xrt-plugin-amdxdna extra : bash $ sudo pacman -S xrt xrt-plugin-amdxdna $ xrt-smi examine ... XRT Version : 2.21.75 NPU Firmware Version : 1.1.2.64 Device s Present |BDF |Name | |----------------|--------------| | 0000:66:00.1 |RyzenAI-npu4 | You want a Device s Present line with a RyzenAI-npu name. If XRT is installed but the plugin isn't, xrt-smi runs but that table is empty. xrt-smi examine --report platform then shows Total Columns : 8 — the XDNA2 array this SoC exposes. xrt 2.21.75 , xrt-plugin-amdxdna same release , NPU firmware 1.1.2.64 , and flm validate reporting the amdxdna driver interface as The NPU runtime pins model weights into physical RAM, so the calling user needs an unlimited memlock rlimit . The default usually 8 MiB or 64 MiB is nowhere near enough and the failure mode is an unhelpful allocation error deep in the runtime. bash $ sudo tee -a /etc/security/limits.conf <<< "$USER soft memlock unlimited" $ sudo tee -a /etc/security/limits.conf <<< "$USER hard memlock unlimited" log out and back in You want to see this afterwards: bash $ ulimit -l unlimited The "official" Linux route is: build ONNX Runtime with the VitisAI EP, install the Ryzen AI SDK bits, quantize your model to the NPU's format, wrangle a Python venv full of onnxruntime-vitisai and Vitis tooling. It's a lot, and much of it is Windows-first. FastFlowLM flm skips all of that. It's a libwhisper npu.so in the package itself: bash $ ls /usr/share/flm/xclbins/ encoder attn encoder dequant encoder mm whisper head ... $ flm --version FLM v1.0.2 Because the kernels are bundled, there is no onnxruntime-vitisai / Ryzen AI SDK venv to build . Arch's stock python-onnxruntime-cpu only has CPUExecutionProvider anyway — irrelevant here. On Arch: sudo pacman -S fastflowlm .Validate the whole stack in one shot: bash $ flm validate Linux Kernel: 7.1.9-arch1-2 Linux NPU: /dev/accel/accel0 with 8 columns Linux NPU FW Version: 1.1.2.64 Linux amdxdna version: 0.8 Linux Memlock Limit: infinity All green = ready. If Memlock Limit says anything other than infinity , go back to the limits.conf step. Pull the model — whisper-v3:turbo is large-v3-turbo quantized for XDNA2: php $ flm pull whisper-v3:turbo ~650 MB: model.q4nx + tokenizers - ~/.config/flm/models/Whisper-V3-Turbo-NPU2/ Serve it. On FLM 1.0.2+ Whisper loads standalone — older docs claimed you had to co-load an LLM, but you don't: bash $ flm serve --asr 1 OpenAI-compatible server on :52625 Transcribe over the HTTP API anything ffmpeg can decode — wav/mp3/ogg/m4a/flac : bash $ curl http://127.0.0.1:52625/v1/audio/transcriptions \ -F "file=@audio.ogg" \ -F "model=whisper-v3" There's also a CLI path: flm run <model --asr 1 , then /input "clip.mp3" in the chat prompt. Benchmarked with the bundled bench.py — 10 runs, first 2 discarded as warm-up, audio length read from the file via ffprobe so the RTF is honest and reproducible: | Metric | Value | |---|---| | Audio length | 30.0 s JFK, Rice University speech excerpt | | Transcription wall time warm | 5.2 s σ 0.04 s within a run; 5.17–5.6 s across sessions | Real-time factor RTF | ≈ 0.17–0.19 | | Transcript accuracy | correct, verbatim | Roughly 5–6× faster than real time . Within a single benchmark the spread is under 1%; between sessions the mean drifts a few hundred ms with machine temperature and background load. Two checks. First, the FLM log prints NPU Locked when a job starts and NPU Lock Released when it finishes. Second — and more convincing — sample system load while the benchmark runs and see that nothing else is doing the work: | Device | Idle baseline | During 10 transcriptions | |---|---|---| | CPU 20 threads, system-wide | 2.2 % | 4.3 % mean, 14.4 % peak | | iGPU Radeon 890M | 7 % | 10 % mean, 15 % peak | | dGPU RTX 4070 | 0 % | 0 % | The CPU rises about two points over idle — that's the curl /harness overhead and the server's I/O thread, not inference. The iGPU delta is desktop compositing Hyprland renders on the 890M , and the discrete GPU is never touched at all. The 30 seconds of audio is being processed somewhere that doesn't show up in any of these three counters, which is exactly the point: the CPU and both GPUs stay free while the NPU works. Speed alone isn't the story — whisper-large-v3-turbo has a tiny decoder and runs fine on CPU. So I built whisper.cpp from source the Arch package's ggml backend is currently broken and ran the same 30 s clip through the same model on the CPU, tuned to 16 threads, reading the RAPL energy counters /sys/class/powercap/intel-rapl:0 around every run. NPU FastFlowLM | CPU whisper.cpp, -t 16 | | |---|---|---| | Wall time 30 s clip | ~5.3 s | ~6.5 s | | RTF | 0.18 | 0.22 | | CPU-package power while running | ~20 W | ~73 W | CPU-core power while running | ~0.8 W | ~10 W | Energy per transcription, over idle | ~45 J | ~410 J | | Energy per transcription, total package | ~105 J | ~478 J | The wall-clock win is modest — about 25%. The energy difference is the point: transcribing that clip on the NPU costs roughly an order of magnitude less energy than doing it on the CPU ~45 J vs ~410 J above idle . Package Measured on a live desktop, so absolute wattages drift a few watts between runs with background activity — "energy over idle" is the stable figure and what the comparison rests on. whisper-cli also reloads the 1.6 GB model each run, which pads its wall time slightly but not its energy. Both harnesses are in the bench.py --power for the NPU column, bench cpu.py for the CPU column. FLM serves LLMs on the NPU through the same OpenAI-compatible surface. Its model catalogue covers the usual small-to-mid open weights: bash $ flm list gemma3:1b ✅ qwen3:1.7b ⏬ llama3.2:3b ⏬ phi4-mini-it:4b ⏬ deepseek-r1:8b ⏬ gpt-oss:20b ⏬ whisper-v3:turbo ✅ ... One server can expose both ASR and chat: bash $ flm serve gemma3:1b --asr 1 $ curl http://127.0.0.1:52625/v1/chat/completions \ -H 'content-type: application/json' \ -d '{"model":"gemma3:1b","messages": {"role":"user","content":"hello"} }' So /v1/audio/transcriptions and /v1/chat/completions are both live on :52625 from a single process on the NPU. With the endpoint working, the rest is glue: — a zero-dependency npu-whisper flm serve --asr 1 curl s the transcript, and leaves the --json , --status , --stop . — a local-ai-assistant pw-record → Whisper NPU → LLM NPU → piper TTS → speaker . One FLM server chat keeps conversation history across runs.Both are deliberately small — the interesting work was getting the NPU to do the inference, not the plumbing on top. flm validate catches it. .msi — 1.0.2 is fine to stay on for Linux. .q4nx weights + bundled xclbins are FastFlowLM's; you can't point llama.cpp or vanilla ONNX Runtime at them. whisper.cpp on CPU is far slower for large-v3-turbo . amdxdna + XRT + FLM combination works well but you have to assemble it yourself. That's the gap this post is trying to close. 1. driver stack sudo pacman -S xrt xrt-plugin-amdxdna fastflowlm sudo tee -a /etc/security/limits.conf <<< "$USER soft memlock unlimited" sudo tee -a /etc/security/limits.conf <<< "$USER hard memlock unlimited" log out / back in flm validate want: all green, Memlock Limit: infinity 2. models flm pull whisper-v3:turbo flm pull gemma3:1b 3. run flm serve gemma3:1b --asr 1 curl http://127.0.0.1:52625/v1/audio/transcriptions -F file=@clip.wav -F model=whisper-v3 Requirements: a Ryzen AI XDNA / XDNA2 laptop, kernel ≥ 6.14 with amdxdna , and the memlock bump.