whisper.cpp v1.9.5, one recording, four builds (x86 CPU, Intel iGPU, Arm CPU, RTX Spark CUDA): medium was faster on both CPUs, turbo was faster or tied on both GPUs, and the per-phase timings had the same shape on all four.
This is an English adaptation of an article I originally published in Japanese on Zenn: whisper の turbo と medium、Arm の CPU と CUDA でも速い順は入れ替わった. The measurements and conclusions are mine; I used an AI tool to help translate and restructure the text, and I checked the numbers against the original.
Short version: on an x86 CPU and an Arm CPU, medium was faster than large-v3-turbo; on two different GPUs, turbo was faster (or tied in total), and the per-phase timings explained the flip the same way on all four.
A 67-second video summary (narration is synthetic, English captions available):
system_info shows it.large-v3-turbo (fp16), medium, and turbo's q5_0 quantization.-t 8 vs. -t 20 on the Arm CPU.
This continues an earlier article (in Japanese): whisper の turbo と medium、速いほうが CPU と内蔵 GPU で入れ替わった. That one covers the method (same recording, same arguments, each condition run 5 times in a row with the first run discarded) and how to read the phase timings. There, on one PC, medium was faster on the CPU and turbo was faster on the integrated GPU. This time I added a second PC with a different CPU architecture and a different GPU family.
whisper.cpp at v1.9.5 on both machines, same recording, same arguments. Average of runs 2–5, in ms.
| Model | x86 CPU | x86 iGPU | Arm CPU | Arm CUDA |
|---|---|---|---|---|
| large-v3-turbo (fp16) | 13,192 | 2,393 | 13,873 | 1,537 |
| medium | 10,370 | 2,408 | 9,698 | 1,794 |
| large-v3-turbo (q5_0) | 12,021 | 1,371 | 7,108 | 795 |
| Faster of turbo and medium | medium (1.27×) | tie (15ms apart) | medium (1.43×) | turbo (by 257ms) |
The previous comparison was within one PC, so I couldn't tell whether the explanation "where the encoder is heavy, the model with the smaller encoder wins" was specific to that PC. The same direction showed up on a CPU with a different instruction set and on a different family of GPU, so the explanation now covers a little more ground.
Both machines used the same recording (Japanese speech recorded with the watch's microphone, 6.635 s, 16kHz, mono, 16-bit) and the same arguments:
whisper-cli -m <model> -l ja -t <threads> -bs 5 -bo 5 -f sample.wav -otxt -nt
| x86 PC | Arm PC | |
|---|---|---|
| Machine | Laptop with Intel Core Ultra 7 258V | Surface Laptop Ultra (Windows 11 Arm64) |
| CPU | 8 cores, 8 threads (AVX2) | Arm cores of the NVIDIA RTX Spark N1X (20 threads) |
| GPU | Integrated Intel Arc 140V (Vulkan build) | RTX Spark N1X GPU (6144 cores, Blackwell, CUDA 13.4 build) |
| CPU build | MSVC, AVX2. No BLAS | clang 19.1.5 ( arm64-windows-llvm toolchain), NEON. No BLAS |
| Threads | -t 8 |
-t 8 (20-thread results later) |
libomp140.aarch64.dll) at run time. I used the debug, non-redistributable copy bundled with Visual Studio, so At first I built the Arm CPU version with MSVC. These are its numbers:
| Model | MSVC build | clang build | Ratio |
|---|---|---|---|
| large-v3-turbo (fp16) | 84,531ms | 13,873ms | 6.1 |
| medium | 53,894ms | 9,698ms | 5.6 |
| large-v3-turbo (q5_0) | 72,145ms | 7,108ms | 10.2 |
6.6 seconds of audio took 84 seconds. One line of system_info shows why:
system_info: n_threads = 8 / 20 | ... | CPU : ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | OPENMP = 1 | REPACK = 1 |
system_info: n_threads = 8 / 20 | ... | CPU : NEON = 1 | ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | OPENMP = 1 | REPACK = 1 |
With the MSVC I used, the list of predefined macros (cl /PD) had no __ARM_NEON. ggml's CPU backend switches code paths on whether __ARM_NEON is defined, so an MSVC build runs matrix multiplication and the like in generic C code. When I tried to work around it by passing /D__ARM_NEON=1 from outside, compilation failed in the NEON helper functions. (At least in v1.9.5, that means ggml's NEON path can't be used with MSVC.)
The whisper.cpp repository has a clang toolchain file for Windows on Arm, and specifying it gave NEON = 1. Other things I tried on the MSVC build, such as turning off flash attention or enabling llamafile's sgemm, made no difference.
Look at system_info before measuring speed. That's all it is, but if I had only looked at the totals, I would have stopped at "the Arm CPU is slow." As far as I found, when building this kind of library on Windows on Arm, the choice of compiler seems to make an order-of-magnitude difference.
From the phase timings whisper.cpp prints on exit, I took the encoder (encode) and the decoder side ( batchd) and lined up turbo minus medium. Positive means turbo loses; negative means turbo gains.
| Environment | encode diff | batchd diff | Total diff |
|---|---|---|---|
| x86 CPU | +4,604ms | −1,653ms | +2,822ms (turbo slower) |
| Arm CPU | +5,267ms | −1,067ms | +4,176ms (turbo slower) |
| x86 iGPU | +312ms | −455ms | −15ms (tie) |
| Arm CUDA | +24ms | −391ms | −257ms (turbo faster) |
turbo is large-v3 with the decoder cut from 32 layers to 4. The encoder stays the same as large-v3: 32 layers, 1280 dimensions (medium has 24 layers, 1024 dimensions). So turbo gains on the decoder side and loses in the encoder. Which of the two is larger flips with the environment.
On the CPUs the encoder loss is large. On the GPUs the encoder gap shrinks, so the decoder gain wins. On CUDA the encoder gap was down to 24ms, and the cost of turbo's larger encoder was barely visible.
The raw breakdown (-t 8, average of runs 2–5, ms):
| Environment | Model | load | encode | batchd | Total |
|---|---|---|---|---|---|
| x86 CPU | turbo fp16 | 1,268 | 11,235 | 519 | 13,192 |
| x86 CPU | medium | 1,223 | 6,631 | 2,171 | 10,370 |
| Arm CPU | turbo fp16 | 1,186 | 12,005 | 564 | 13,873 |
| Arm CPU | medium | 1,146 | 6,738 | 1,631 | 9,698 |
| x86 iGPU | turbo fp16 | 1,230 | 720 | 309 | 2,393 |
| x86 iGPU | medium | 1,057 | 408 | 763 | 2,408 |
| Arm CUDA | turbo fp16 | 1,118 | 130 | 147 | 1,537 |
| Arm CUDA | medium | 974 | 106 | 538 | 1,794 |
On CUDA, 1,118ms of turbo's 1,537ms total is ; only 419ms is computation. Re-ranked with excluded:
| Environment | turbo fp16 | medium | Difference |
|---|---|---|---|
| x86 CPU | 11,924ms | 9,147ms | medium faster by 2,778ms |
| Arm CPU | 12,687ms | 8,551ms | medium faster by 4,136ms |
| x86 iGPU | 1,163ms | 1,352ms | turbo faster by 188ms |
| Arm CUDA | 419ms | 820ms | turbo faster by 401ms |
Both the integrated GPU, which looked like a tie in total, and CUDA, where is most of the time, show a clear order once is taken out. It also means the same models can rank differently depending on how you run them: starting a process every time (CLI) versus keeping one running and sending requests to it (resident server).
I also measured the round trip of sending the same recording over HTTP to a resident server. then happens only once, at startup.
| Server setting | Model | x86 iGPU | Arm CUDA |
|---|---|---|---|
| Default (greedy) | large-v3-turbo | 867ms | 202ms |
| Default (greedy) | medium | 921ms | 473ms |
-bs 5 (same width as the CLI) |
large-v3-turbo | 1,269ms | 340ms |
-bs 5 (same width as the CLI) |
medium | 1,501ms | 724ms |
As a resident server, turbo was faster on both GPUs. Compared with the integrated GPU, CUDA returned in roughly a quarter of the time for turbo and roughly half for medium. Note that whisper-server defaults to greedy decoding when nothing is specified, while whisper-cli defaults to beam search with width 5: the defaults are opposite. (Last time, this left my conditions mismatched.)
As a third model, I measured turbo's q5_0 (quantized) version under the same conditions.
| Environment | q5_0 total | Excluding | vs. fp16 (excluding ) |
|---|---|---|---|
| x86 CPU | 12,021ms | 11,525ms | q5_0 faster by 400ms |
| Arm CPU | 7,108ms | 6,690ms | q5_0 faster by 5,997ms |
| x86 iGPU | 1,371ms | 944ms | q5_0 faster by 219ms |
| Arm CUDA | 795ms | 404ms | q5_0 faster by 15ms |
HAVE_FP16_VECTOR_ARITHMETIC isn't set), while the integer instructions are available. My guess is that this is why the two fp16 models were slow, but whisper.cpp sees 20 threads on the Arm PC (n_threads = 8 / 20 in the output), so I measured both -t 8 and -t 20.
| Model | -t 8 total |
-t 20 total |
encode | batchd |
|---|---|---|---|---|
| large-v3-turbo (fp16) | 13,873ms | 11,015ms | 12,005 → 9,004ms | 564 → 579ms |
| medium | 9,698ms | 7,478ms | 6,738 → 4,162ms | 1,631 → 2,069ms |
| large-v3-turbo (q5_0) | 7,108ms | 4,448ms | 6,326 → 3,714ms | 276 → 226ms |
Every total went down, but not evenly inside. The encoder is one call that passes the whole audio through at once, so it shrinks cleanly. The decoder side is more than 160 small calls, so it shrinks less, and for medium it actually grew. More threads don't make everything uniformly faster; that's what this table supports.
On variance: turbo fp16 at -t 20 ranged from 10,294 to 11,674ms (13%) even across runs 2–5, wider than the other conditions (a few %). If you measure only the total once and compare, any difference smaller than that spread disappears.
-march=armv8.7-a and others).
whisper.cpp is licensed under MIT (ggml-org/whisper.cpp). The phase names come from the comments on the counters in the source.
This is part of my series on a DIY ESP32-S3 smartwatch. The watch sends recorded speech to a PC, which transcribes it; this article and the previous one compare models for that step. The Arm PC here is the same Surface Laptop Ultra I've written about in English before. Related articles:
The original of this article: whisper の turbo と medium、Arm の CPU と CUDA でも速い順は入れ替わった.