Running Whisper + LLMs on an AMD NPU under Linux

TL;DR— On a MSI Stealth A16 AI+ (Ryzen AI 9 365, XDNA2 NPU) running Arch,

I got OpenAI'swhisper-large-v3-turbo

transcribing on theNPU— not the

CPU, not the GPU — atRTF ≈ 0.18(a 30 s clip in ~5.2 s) for roughly a

tenth of the energythe same job costs on the CPU, plus an LLM answering

on the same NPU through an OpenAI-compatible API. The

whole path is local and offline. This is the write-up of the driver stack,

the one real gotcha (memlock), and the runtime that made it a 20-minute job

instead of a weekend.

AMD's "Ryzen AI" NPU (the XDNA / XDNA2 block in Phoenix / Hawk Point / Strix

Point laptops) is marketed almost entirely around Windows: the Ryzen AI SDK,

the ONNX Runtime VitisAI execution provider, Lemonade, and the demos all

assume you're on Windows with the official stack. On Linux the picture in early

2026 is better than most people think — the NPU driver has been in the mainline kernel as

amdxdna

since 6.14 — but the "load a real model and runHere's what actually worked, end to end.

Part	Detail
Laptop	MSI Stealth A16 AI+ A3HVGG
APU	AMD Ryzen AI 9 365 (Strix Point)
NPU	XDNA2, 8 columns, exposed as /dev/accel/accel0
NPU firmware	1.1.2.64
Kernel	7.1.9-arch1 (amdxdna in-tree)
OS	Omarchy (Arch Linux)

AMD quotes the Strix Point NPU at up to 50 TOPS, INT8.

Three pieces have to be in place before any runtime can touch the NPU:

amdxdna

/dev/accel/accel0

. Check it's bound:

   $ ls /dev/accel/
   accel0
   $ dmesg | grep -i amdxdna

xrt-plugin-amdxdna

extra

:

   $ sudo pacman -S xrt xrt-plugin-amdxdna
   $ xrt-smi examine
   ...
   XRT
     Version              : 2.21.75
     NPU Firmware Version  : 1.1.2.64

   Device(s) Present
   |BDF             |Name          |
   |----------------|--------------|
   |[0000:66:00.1]  |RyzenAI-npu4  |

You want a Device(s) Present

line with a RyzenAI-npu*

name. If XRT is

installed but the plugin isn't, xrt-smi

runs but that table is empty.

xrt-smi examine --report platform

then shows Total Columns : 8

— the

XDNA2 array this SoC exposes.

xrt 2.21.75

, xrt-plugin-amdxdna

(same release), NPU firmware 1.1.2.64

, and flm validate

reporting the amdxdna

driver interface as The NPU runtime pins model weights into physical RAM, so the calling user needs

an unlimited memlock rlimit. The default (usually 8 MiB or 64 MiB) is nowhere

near enough and the failure mode is an unhelpful allocation error deep in the

runtime.

$ sudo tee -a /etc/security/limits.conf <<< "$USER soft memlock unlimited"
$ sudo tee -a /etc/security/limits.conf <<< "$USER hard memlock unlimited"

You want to see this afterwards:

$ ulimit -l
unlimited

The "official" Linux route is: build ONNX Runtime with the VitisAI EP, install

the Ryzen AI SDK bits, quantize your model to the NPU's format, wrangle a Python

venv full of onnxruntime-vitisai

and Vitis tooling. It's a lot, and much of it

is Windows-first.

** FastFlowLM** (

flm

) skips all of that. It's alibwhisper_npu.so

in the package itself:

$ ls /usr/share/flm/xclbins/
encoder_attn  encoder_dequant  encoder_mm  whisper_head  ...
$ flm --version
FLM v1.0.2

Because the kernels are bundled, there is no onnxruntime-vitisai / Ryzen AI SDK venv to build. (Arch's stock

python-onnxruntime-cpu

only hasCPUExecutionProvider

anyway — irrelevant here.) On Arch: sudo pacman -S

fastflowlm

.Validate the whole stack in one shot:

$ flm validate
[Linux]  Kernel: 7.1.9-arch1-2
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 0.8
[Linux]  Memlock Limit: infinity

All green = ready. If Memlock Limit

says anything other than infinity

, go

back to the limits.conf step.

Pull the model — whisper-v3:turbo

is large-v3-turbo

quantized for XDNA2:

$ flm pull whisper-v3:turbo

Serve it. On FLM 1.0.2+ Whisper loads standalone — older docs claimed you had

to co-load an LLM, but you don't:

$ flm serve --asr 1          # OpenAI-compatible server on :52625

Transcribe over the HTTP API (anything ffmpeg

can decode — wav/mp3/ogg/m4a/flac):

$ curl http://127.0.0.1:52625/v1/audio/transcriptions \
    -F "file=@audio.ogg" \
    -F "model=whisper-v3"

There's also a CLI path: flm run <model> --asr 1

, then /input "clip.mp3"

in

the chat prompt.

Benchmarked with the bundled bench.py

— 10 runs, first 2 discarded as warm-up,

audio length read from the file via ffprobe

so the RTF is honest and

reproducible:

Metric	Value
Audio length	30.0 s (JFK, Rice University speech excerpt)
Transcription wall time (warm)
5.2 s (σ 0.04 s within a run; 5.17–5.6 s across sessions)
Real-time factor (RTF)
≈ 0.17–0.19
Transcript accuracy	correct, verbatim

Roughly 5–6× faster than real time. Within a single benchmark the spread is

under 1%; between sessions the mean drifts a few hundred ms with machine

temperature and background load.

Two checks. First, the FLM log prints [NPU Locked!]

when a job starts and

[NPU Lock Released!]

when it finishes. Second — and more convincing — sample

system load while the benchmark runs and see that nothing else is doing the

work:

Device	Idle baseline	During 10 transcriptions
CPU (20 threads, system-wide)	2.2 %
4.3 % mean, 14.4 % peak
iGPU (Radeon 890M)	7 %
10 % mean, 15 % peak
dGPU (RTX 4070)	0 %	0 %

The CPU rises about two points over idle — that's the curl

/harness overhead and

the server's I/O thread, not inference. The iGPU delta is desktop compositing

(Hyprland renders on the 890M), and the discrete GPU is never touched at all.

The 30 seconds of audio is being processed somewhere that doesn't show up in any

of these three counters, which is exactly the point: the CPU and both GPUs stay

free while the NPU works.

Speed alone isn't the story — whisper-large-v3-turbo

has a tiny decoder and

runs fine on CPU. So I built whisper.cpp

from source (the Arch package's ggml

backend is currently broken) and ran the same 30 s clip through the same model on the CPU, tuned to 16 threads, reading the RAPL energy counters

/sys/class/powercap/intel-rapl:0

) around every run. NPU (FastFlowLM) | CPU (whisper.cpp, -t 16 ) | | |---|---|---| | Wall time (30 s clip) | ~5.3 s | ~6.5 s | | RTF | 0.18 | 0.22 | | CPU-package power while running | ~20 W | ~73 W | CPU-core power while running | ~0.8 W | ~10 W | Energy per transcription, over idle | ~45 J | ~410 J | | Energy per transcription, total package | ~105 J | ~478 J |

The wall-clock win is modest — about 25%. The energy difference is the

point: transcribing that clip on the NPU costs roughly an order of magnitude less energy than doing it on the CPU (~45 J vs ~410 J above idle). Package

*(Measured on a live desktop, so absolute wattages drift a few watts between runs with background activity — "energy over idle" is the stable figure and what the comparison rests on. *

whisper-cli

also reloads the 1.6 GB model each run, which pads its wall time slightly but not its energy. Both harnesses are in the bench.py --power

for the NPU column, bench_cpu.py

for the CPU column.)FLM serves LLMs on the NPU through the same OpenAI-compatible surface. Its model

catalogue covers the usual small-to-mid open weights:

$ flm list
  gemma3:1b        ✅
  qwen3:1.7b       ⏬
  llama3.2:3b      ⏬
  phi4-mini-it:4b  ⏬
  deepseek-r1:8b   ⏬
  gpt-oss:20b      ⏬
  whisper-v3:turbo ✅
  ...

One server can expose both ASR and chat:

$ flm serve gemma3:1b --asr 1
$ curl http://127.0.0.1:52625/v1/chat/completions \
    -H 'content-type: application/json' \
    -d '{"model":"gemma3:1b","messages":[{"role":"user","content":"hello"}]}'

So /v1/audio/transcriptions

and /v1/chat/completions

are both live on

:52625

from a single process on the NPU.

With the endpoint working, the rest is glue:

— a zero-dependency

npu-whisper

flm serve --asr 1

curl

s the transcript, and leaves the--json

, --status

, --stop

. — a

local-ai-assistant

pw-record → Whisper (NPU) → LLM (NPU) → piper TTS → speaker

. One FLM serverchat

keeps conversation history across runs.Both are deliberately small — the interesting work was getting the NPU to do the

inference, not the plumbing on top.

flm validate

catches it..msi

— 1.0.2 is fine to stay on for Linux..q4nx

weights + bundled xclbins are FastFlowLM's; you can't point llama.cpp or vanilla ONNX Runtime at them.whisper.cpp

on CPU is far slower for large-v3-turbo

.amdxdna

• XRT + FLM combination works well but you have to assemble it yourself. That's the gap this post is trying to close.

sudo pacman -S xrt xrt-plugin-amdxdna fastflowlm
sudo tee -a /etc/security/limits.conf <<< "$USER soft memlock unlimited"
sudo tee -a /etc/security/limits.conf <<< "$USER hard memlock unlimited"
flm validate            # want: all green, Memlock Limit: infinity

flm pull whisper-v3:turbo
flm pull gemma3:1b

flm serve gemma3:1b --asr 1
curl http://127.0.0.1:52625/v1/audio/transcriptions -F file=@clip.wav -F model=whisper-v3

Requirements: a Ryzen AI (XDNA / XDNA2) laptop, kernel ≥ 6.14 with amdxdna

,

and the memlock bump.

source & further reading

dev.to — original article Why System Prompts Won't Save Your AI Agent: Building Architectural Injection Immunity Four Ways Your Background Job Disappears (And How to Stop Each One) Why `anthropics/skills` Earned 277 Stars Today: Reusable Agent Workflows, Not More Prompts