# Running Whisper + LLMs on an AMD NPU under Linux

> Source: <https://dev.to/jac-76/running-whisper-llms-on-an-amd-npu-under-linux-2o1h>
> Published: 2026-09-03 12:40:54+00:00

TL;DR— On a MSI Stealth A16 AI+ (Ryzen AI 9 365, XDNA2 NPU) running Arch,

I got OpenAI's`whisper-large-v3-turbo`

transcribing on theNPU— not the

CPU, not the GPU — atRTF ≈ 0.18(a 30 s clip in ~5.2 s) for roughly a

tenth of the energythe same job costs on the CPU, plus an LLM answering

on the same NPU through an OpenAI-compatible API. The

whole path is local and offline. This is the write-up of the driver stack,

the one real gotcha (memlock), and the runtime that made it a 20-minute job

instead of a weekend.

AMD's "Ryzen AI" NPU (the XDNA / XDNA2 block in Phoenix / Hawk Point / Strix

Point laptops) is marketed almost entirely around Windows: the Ryzen AI SDK,

the ONNX Runtime **VitisAI** execution provider, Lemonade, and the demos all

assume you're on Windows with the official stack. On Linux the picture in early

2026 is better than most people think — the NPU driver has been **in the
mainline kernel** as

`amdxdna`

since 6.14 — but the "load a real model and runHere's what actually worked, end to end.

| Part | Detail |
|---|---|
| Laptop | MSI Stealth A16 AI+ A3HVGG |
| APU | AMD Ryzen AI 9 365 (Strix Point) |
| NPU | XDNA2, 8 columns, exposed as `/dev/accel/accel0`
|
| NPU firmware | `1.1.2.64` |
| Kernel | 7.1.9-arch1 (`amdxdna` in-tree) |
| OS | Omarchy (Arch Linux) |

AMD quotes the Strix Point NPU at [up to 50 TOPS, INT8](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-9-365.html).

Three pieces have to be in place before any runtime can touch the NPU:

`amdxdna`

`/dev/accel/accel0`

. Check it's bound:

``` bash
   $ ls /dev/accel/
   accel0
   $ dmesg | grep -i amdxdna
```

`xrt-plugin-amdxdna`

`extra`

:

``` bash
   $ sudo pacman -S xrt xrt-plugin-amdxdna
   $ xrt-smi examine
   ...
   XRT
     Version              : 2.21.75
     NPU Firmware Version  : 1.1.2.64

   Device(s) Present
   |BDF             |Name          |
   |----------------|--------------|
   |[0000:66:00.1]  |RyzenAI-npu4  |
```

You want a `Device(s) Present`

line with a `RyzenAI-npu*`

name. If XRT is

installed but the plugin isn't, `xrt-smi`

runs but that table is empty.

`xrt-smi examine --report platform`

then shows `Total Columns : 8`

— the

XDNA2 array this SoC exposes.

`xrt 2.21.75`

, `xrt-plugin-amdxdna`

(same
release), NPU firmware `1.1.2.64`

, and `flm validate`

reporting the `amdxdna`

driver interface as The NPU runtime pins model weights into physical RAM, so the calling user needs

an **unlimited memlock rlimit**. The default (usually 8 MiB or 64 MiB) is nowhere

near enough and the failure mode is an unhelpful allocation error deep in the

runtime.

``` bash
$ sudo tee -a /etc/security/limits.conf <<< "$USER soft memlock unlimited"
$ sudo tee -a /etc/security/limits.conf <<< "$USER hard memlock unlimited"
# log out and back in
```

You want to see this afterwards:

``` bash
$ ulimit -l
unlimited
```

The "official" Linux route is: build ONNX Runtime with the VitisAI EP, install

the Ryzen AI SDK bits, quantize your model to the NPU's format, wrangle a Python

venv full of `onnxruntime-vitisai`

and Vitis tooling. It's a lot, and much of it

is Windows-first.

** FastFlowLM** (

`flm`

) skips all of that. It's a`libwhisper_npu.so`

in the package itself:

``` bash
$ ls /usr/share/flm/xclbins/
encoder_attn  encoder_dequant  encoder_mm  whisper_head  ...
$ flm --version
FLM v1.0.2
```

Because the kernels are bundled, **there is no onnxruntime-vitisai / Ryzen AI SDK
venv to build**. (Arch's stock

`python-onnxruntime-cpu`

only has`CPUExecutionProvider`

anyway — irrelevant here.) On Arch: `sudo pacman -S`

fastflowlm

.Validate the whole stack in one shot:

``` bash
$ flm validate
[Linux]  Kernel: 7.1.9-arch1-2
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 0.8
[Linux]  Memlock Limit: infinity
```

All green = ready. If `Memlock Limit`

says anything other than `infinity`

, go

back to the limits.conf step.

Pull the model — `whisper-v3:turbo`

is `large-v3-turbo`

quantized for XDNA2:

``` php
$ flm pull whisper-v3:turbo
# ~650 MB: model.q4nx + tokenizers -> ~/.config/flm/models/Whisper-V3-Turbo-NPU2/
```

Serve it. On FLM 1.0.2+ Whisper loads **standalone** — older docs claimed you had

to co-load an LLM, but you don't:

``` bash
$ flm serve --asr 1          # OpenAI-compatible server on :52625
```

Transcribe over the HTTP API (anything `ffmpeg`

can decode — wav/mp3/ogg/m4a/flac):

``` bash
$ curl http://127.0.0.1:52625/v1/audio/transcriptions \
    -F "file=@audio.ogg" \
    -F "model=whisper-v3"
```

There's also a CLI path: `flm run <model> --asr 1`

, then `/input "clip.mp3"`

in

the chat prompt.

Benchmarked with the bundled `bench.py`

— 10 runs, first 2 discarded as warm-up,

audio length read from the file via `ffprobe`

so the RTF is honest and

reproducible:

| Metric | Value |
|---|---|
| Audio length | 30.0 s (JFK, Rice University speech excerpt) |
| Transcription wall time (warm) |
5.2 s (σ 0.04 s within a run; 5.17–5.6 s across sessions) |
Real-time factor (RTF) |
≈ 0.17–0.19 |
| Transcript accuracy | correct, verbatim |

Roughly **5–6× faster than real time**. Within a single benchmark the spread is

under 1%; between sessions the mean drifts a few hundred ms with machine

temperature and background load.

Two checks. First, the FLM log prints `[NPU Locked!]`

when a job starts and

`[NPU Lock Released!]`

when it finishes. Second — and more convincing — sample

system load while the benchmark runs and see that nothing else is doing the

work:

| Device | Idle baseline | During 10 transcriptions |
|---|---|---|
| CPU (20 threads, system-wide) | 2.2 % |
4.3 % mean, 14.4 % peak |
| iGPU (Radeon 890M) | 7 % |
10 % mean, 15 % peak |
| dGPU (RTX 4070) | 0 % | 0 % |

The CPU rises about two points over idle — that's the `curl`

/harness overhead and

the server's I/O thread, not inference. The iGPU delta is desktop compositing

(Hyprland renders on the 890M), and the discrete GPU is never touched at all.

The 30 seconds of audio is being processed somewhere that doesn't show up in any

of these three counters, which is exactly the point: the CPU and both GPUs stay

free while the NPU works.

Speed alone isn't the story — `whisper-large-v3-turbo`

has a tiny decoder and

runs fine on CPU. So I built `whisper.cpp`

from source (the Arch package's ggml

backend is currently broken) and ran the **same 30 s clip through the same
model** on the CPU, tuned to 16 threads, reading the RAPL energy counters

`/sys/class/powercap/intel-rapl:0`

) around every run.
NPU (FastFlowLM) |
CPU (whisper.cpp, `-t 16` ) |
|
|---|---|---|
| Wall time (30 s clip) | ~5.3 s | ~6.5 s |
| RTF | 0.18 | 0.22 |
| CPU-package power while running | ~20 W | ~73 W |
CPU-core power while running |
~0.8 W | ~10 W |
Energy per transcription, over idle |
~45 J |
~410 J |
| Energy per transcription, total package | ~105 J | ~478 J |

The wall-clock win is modest — about 25%. The **energy** difference is the

point: transcribing that clip on the NPU costs roughly **an order of magnitude
less energy** than doing it on the CPU (~45 J vs ~410 J above idle). Package

*(Measured on a live desktop, so absolute wattages drift a few watts between
runs with background activity — "energy over idle" is the stable figure and what
the comparison rests on. *

`whisper-cli`

also reloads the 1.6 GB model each run,
which pads its wall time slightly but not its energy. Both harnesses are in the
`bench.py --power`

for the NPU column, `bench_cpu.py`

for the CPU column.)FLM serves LLMs on the NPU through the same OpenAI-compatible surface. Its model

catalogue covers the usual small-to-mid open weights:

``` bash
$ flm list
  gemma3:1b        ✅
  qwen3:1.7b       ⏬
  llama3.2:3b      ⏬
  phi4-mini-it:4b  ⏬
  deepseek-r1:8b   ⏬
  gpt-oss:20b      ⏬
  whisper-v3:turbo ✅
  ...
```

One server can expose **both** ASR and chat:

``` bash
$ flm serve gemma3:1b --asr 1
$ curl http://127.0.0.1:52625/v1/chat/completions \
    -H 'content-type: application/json' \
    -d '{"model":"gemma3:1b","messages":[{"role":"user","content":"hello"}]}'
```

So `/v1/audio/transcriptions`

and `/v1/chat/completions`

are both live on

`:52625`

from a single process on the NPU.

With the endpoint working, the rest is glue:

— a zero-dependency

`npu-whisper`

`flm serve --asr 1`

`curl`

s the transcript, and leaves the`--json`

, `--status`

, `--stop`

. — a

`local-ai-assistant`

`pw-record → Whisper (NPU) → LLM (NPU) → piper TTS → speaker`

. One FLM server`chat`

keeps conversation history across runs.Both are deliberately small — the interesting work was getting the NPU to do the

inference, not the plumbing on top.

`flm validate`

catches it.`.msi`

— 1.0.2 is fine to stay on for Linux.`.q4nx`

weights + bundled xclbins
are FastFlowLM's; you can't point llama.cpp or vanilla ONNX Runtime at them.`whisper.cpp`

on CPU is far slower for
`large-v3-turbo`

.`amdxdna`

+ XRT + FLM combination works well but you have to assemble it
yourself. That's the gap this post is trying to close.

```
# 1. driver stack
sudo pacman -S xrt xrt-plugin-amdxdna fastflowlm
sudo tee -a /etc/security/limits.conf <<< "$USER soft memlock unlimited"
sudo tee -a /etc/security/limits.conf <<< "$USER hard memlock unlimited"
# log out / back in
flm validate            # want: all green, Memlock Limit: infinity

# 2. models
flm pull whisper-v3:turbo
flm pull gemma3:1b

# 3. run
flm serve gemma3:1b --asr 1
curl http://127.0.0.1:52625/v1/audio/transcriptions -F file=@clip.wav -F model=whisper-v3
```

Requirements: a Ryzen AI (XDNA / XDNA2) laptop, kernel ≥ 6.14 with `amdxdna`

,

and the memlock bump.