cd /news/machine-learning/whisper-turbo-vs-medium-on-arm-and-c… · home › topics › machine-learning › article
[ARTICLE · art-171440] src=dev.to ↗ pub= topic=machine-learning verified=true sentiment=· neutral

Whisper turbo vs. medium on Arm and CUDA: The Faster Model Still Flipped Between CPU and GPU

A developer benchmarked whisper.cpp v1.9.5 across four builds (x86 CPU, Intel iGPU, Arm CPU, and an RTX Spark CUDA GPU) using the same 6.6-second Japanese recording, finding that medium beat large-v3-turbo on both CPUs while turbo was faster or tied on both GPUs. Per-phase timings explained the flip consistently across all four setups, supporting the explanation that the model with the smaller encoder wins where the encoder dominates. The developer also found that building the Arm CPU version with MSVC, which lacked NEON, made inference roughly 6x slower than a clang/NEON build.

recommendation

Build whisper.cpp for Arm with clang/NEON rather than MSVC, which lacked __ARM_NEON and ran about 6x slower; pick medium over large-v3-turbo on CPU-only hosts and turbo on GPUs.

more in playbook: Inference and self-hosting
by read9 min views3 publishedNov 18, 2026

whisper.cpp v1.9.5, one recording, four builds (x86 CPU, Intel iGPU, Arm CPU, RTX Spark CUDA): medium was faster on both CPUs, turbo was faster or tied on both GPUs, and the per-phase timings had the same shape on all four.

This is an English adaptation of an article I originally published in Japanese on Zenn: whisper の turbo と medium、Arm の CPU と CUDA でも速い順は入れ替わった. The measurements and conclusions are mine; I used an AI tool to help translate and restructure the text, and I checked the numbers against the original.

Short version: on an x86 CPU and an Arm CPU, medium was faster than large-v3-turbo; on two different GPUs, turbo was faster (or tied in total), and the per-phase timings explained the flip the same way on all four.

A 67-second video summary (narration is synthetic, English captions available):

system_info shows it.large-v3-turbo (fp16), medium, and turbo's q5_0 quantization.-t 8 vs. -t 20 on the Arm CPU. This continues an earlier article (in Japanese): whisper の turbo と medium、速いほうが CPU と内蔵 GPU で入れ替わった. That one covers the method (same recording, same arguments, each condition run 5 times in a row with the first run discarded) and how to read the phase timings. There, on one PC, medium was faster on the CPU and turbo was faster on the integrated GPU. This time I added a second PC with a different CPU architecture and a different GPU family.

whisper.cpp at v1.9.5 on both machines, same recording, same arguments. Average of runs 2–5, in ms.

Model x86 CPU x86 iGPU Arm CPU Arm CUDA
large-v3-turbo (fp16) 13,192 2,393 13,873 1,537
medium 10,370 2,408 9,698 1,794
large-v3-turbo (q5_0) 12,021 1,371 7,108 795
Faster of turbo and medium medium (1.27×) tie (15ms apart) medium (1.43×) turbo (by 257ms)

The previous comparison was within one PC, so I couldn't tell whether the explanation "where the encoder is heavy, the model with the smaller encoder wins" was specific to that PC. The same direction showed up on a CPU with a different instruction set and on a different family of GPU, so the explanation now covers a little more ground.

Both machines used the same recording (Japanese speech recorded with the watch's microphone, 6.635 s, 16kHz, mono, 16-bit) and the same arguments:

whisper-cli -m <model> -l ja -t <threads> -bs 5 -bo 5 -f sample.wav -otxt -nt
x86 PC Arm PC
Machine Laptop with Intel Core Ultra 7 258V Surface Laptop Ultra (Windows 11 Arm64)
CPU 8 cores, 8 threads (AVX2) Arm cores of the NVIDIA RTX Spark N1X (20 threads)
GPU Integrated Intel Arc 140V (Vulkan build) RTX Spark N1X GPU (6144 cores, Blackwell, CUDA 13.4 build)
CPU build MSVC, AVX2. No BLAS clang 19.1.5 ( arm64-windows-llvm toolchain), NEON. No BLAS
Threads -t 8 -t 8 (20-thread results later)

libomp140.aarch64.dll) at run time. I used the debug, non-redistributable copy bundled with Visual Studio, so At first I built the Arm CPU version with MSVC. These are its numbers:

Model MSVC build clang build Ratio
large-v3-turbo (fp16) 84,531ms 13,873ms 6.1
medium 53,894ms 9,698ms 5.6
large-v3-turbo (q5_0) 72,145ms 7,108ms 10.2

6.6 seconds of audio took 84 seconds. One line of system_info shows why:

system_info: n_threads = 8 / 20 | ... | CPU : ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | OPENMP = 1 | REPACK = 1 |

system_info: n_threads = 8 / 20 | ... | CPU : NEON = 1 | ARM_FMA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | OPENMP = 1 | REPACK = 1 |

With the MSVC I used, the list of predefined macros (cl /PD) had no __ARM_NEON. ggml's CPU backend switches code paths on whether __ARM_NEON is defined, so an MSVC build runs matrix multiplication and the like in generic C code. When I tried to work around it by passing /D__ARM_NEON=1 from outside, compilation failed in the NEON helper functions. (At least in v1.9.5, that means ggml's NEON path can't be used with MSVC.)

The whisper.cpp repository has a clang toolchain file for Windows on Arm, and specifying it gave NEON = 1. Other things I tried on the MSVC build, such as turning off flash attention or enabling llamafile's sgemm, made no difference.

Look at system_info before measuring speed. That's all it is, but if I had only looked at the totals, I would have stopped at "the Arm CPU is slow." As far as I found, when building this kind of library on Windows on Arm, the choice of compiler seems to make an order-of-magnitude difference.

From the phase timings whisper.cpp prints on exit, I took the encoder (encode) and the decoder side ( batchd) and lined up turbo minus medium. Positive means turbo loses; negative means turbo gains.

Environment encode diff batchd diff Total diff
x86 CPU +4,604ms −1,653ms +2,822ms (turbo slower)
Arm CPU +5,267ms −1,067ms +4,176ms (turbo slower)
x86 iGPU +312ms −455ms −15ms (tie)
Arm CUDA +24ms −391ms −257ms (turbo faster)

turbo is large-v3 with the decoder cut from 32 layers to 4. The encoder stays the same as large-v3: 32 layers, 1280 dimensions (medium has 24 layers, 1024 dimensions). So turbo gains on the decoder side and loses in the encoder. Which of the two is larger flips with the environment.

On the CPUs the encoder loss is large. On the GPUs the encoder gap shrinks, so the decoder gain wins. On CUDA the encoder gap was down to 24ms, and the cost of turbo's larger encoder was barely visible.

The raw breakdown (-t 8, average of runs 2–5, ms):

Environment Model load encode batchd Total
x86 CPU turbo fp16 1,268 11,235 519 13,192
x86 CPU medium 1,223 6,631 2,171 10,370
Arm CPU turbo fp16 1,186 12,005 564 13,873
Arm CPU medium 1,146 6,738 1,631 9,698
x86 iGPU turbo fp16 1,230 720 309 2,393
x86 iGPU medium 1,057 408 763 2,408
Arm CUDA turbo fp16 1,118 130 147 1,537
Arm CUDA medium 974 106 538 1,794

On CUDA, 1,118ms of turbo's 1,537ms total is ; only 419ms is computation. Re-ranked with excluded:

Environment turbo fp16 medium Difference
x86 CPU 11,924ms 9,147ms medium faster by 2,778ms
Arm CPU 12,687ms 8,551ms medium faster by 4,136ms
x86 iGPU 1,163ms 1,352ms turbo faster by 188ms
Arm CUDA 419ms 820ms turbo faster by 401ms

Both the integrated GPU, which looked like a tie in total, and CUDA, where is most of the time, show a clear order once is taken out. It also means the same models can rank differently depending on how you run them: starting a process every time (CLI) versus keeping one running and sending requests to it (resident server).

I also measured the round trip of sending the same recording over HTTP to a resident server. then happens only once, at startup.

Server setting Model x86 iGPU Arm CUDA
Default (greedy) large-v3-turbo 867ms 202ms
Default (greedy) medium 921ms 473ms
-bs 5 (same width as the CLI) large-v3-turbo 1,269ms 340ms
-bs 5 (same width as the CLI) medium 1,501ms 724ms

As a resident server, turbo was faster on both GPUs. Compared with the integrated GPU, CUDA returned in roughly a quarter of the time for turbo and roughly half for medium. Note that whisper-server defaults to greedy decoding when nothing is specified, while whisper-cli defaults to beam search with width 5: the defaults are opposite. (Last time, this left my conditions mismatched.)

As a third model, I measured turbo's q5_0 (quantized) version under the same conditions.

Environment q5_0 total Excluding vs. fp16 (excluding )
x86 CPU 12,021ms 11,525ms q5_0 faster by 400ms
Arm CPU 7,108ms 6,690ms q5_0 faster by 5,997ms
x86 iGPU 1,371ms 944ms q5_0 faster by 219ms
Arm CUDA 795ms 404ms q5_0 faster by 15ms

HAVE_FP16_VECTOR_ARITHMETIC isn't set), while the integer instructions are available. My guess is that this is why the two fp16 models were slow, but whisper.cpp sees 20 threads on the Arm PC (n_threads = 8 / 20 in the output), so I measured both -t 8 and -t 20.

Model -t 8 total -t 20 total encode batchd
large-v3-turbo (fp16) 13,873ms 11,015ms 12,005 → 9,004ms 564 → 579ms
medium 9,698ms 7,478ms 6,738 → 4,162ms 1,631 → 2,069ms
large-v3-turbo (q5_0) 7,108ms 4,448ms 6,326 → 3,714ms 276 → 226ms

Every total went down, but not evenly inside. The encoder is one call that passes the whole audio through at once, so it shrinks cleanly. The decoder side is more than 160 small calls, so it shrinks less, and for medium it actually grew. More threads don't make everything uniformly faster; that's what this table supports.

On variance: turbo fp16 at -t 20 ranged from 10,294 to 11,674ms (13%) even across runs 2–5, wider than the other conditions (a few %). If you measure only the total once and compare, any difference smaller than that spread disappears.

-march=armv8.7-a and others). whisper.cpp is licensed under MIT (ggml-org/whisper.cpp). The phase names come from the comments on the counters in the source.

This is part of my series on a DIY ESP32-S3 smartwatch. The watch sends recorded speech to a PC, which transcribes it; this article and the previous one compare models for that step. The Arm PC here is the same Surface Laptop Ultra I've written about in English before. Related articles:

The original of this article: whisper の turbo と medium、Arm の CPU と CUDA でも速い順は入れ替わった.

── more in #machine-learning 4 stories · sorted by recency
── more on @whisper.cpp 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/whisper-turbo-vs-med…] indexed:0 read:9min 2026-11-18 · —