Shipping local speech-to-text inside a Tauri app

Set, a fully offline Markdown notes app built on the Tauri framework, ships local dictation using OpenAI's Whisper small model in its 5-bit quantized form via whisper.cpp, a 190 MB download that adds 2.8 MB to the app's binary. On 1,000 LibriSpeech clips (almost 20,000 words), the q5_1 model produced a 4.2% word error rate on clean speech and 10.8% on harder recordings, and on a 2020 M1 MacBook Pro it transcribed a one-minute take in 3.5 seconds with the GPU and 4.9 seconds on CPU alone, peaking at 346 MB of memory. The developer chose whisper.cpp because platform speech APIs differ across macOS, Windows and Linux, and linked it into the Rust app through whisper-rs 0.16 bindings with Metal enabled.

Shipping local speech-to-text inside a Tauri app Set is a fully offline notes app that keeps every page as a Markdown file on your own disk. It also has local dictation as an opt-in feature. The speech recognition runs on device, using OpenAI's Whisper model through whisper.cpp https://github.com/ggml-org/whisper.cpp . Set ships the 5-bit quantized version of Whisper small, a 190 MB download. On 1,000 LibriSpeech https://www.openslr.org/12 clips almost 20,000 words , it got 3.9% of words wrong on clean speech and 10.2% on harder recordings. On a 2020 M1 MacBook Pro, once warmed up, it transcribes a minute of speech in 3.5 seconds with the GPU, or 4.9 seconds on the CPU alone, using under 350 MB of memory with the GPU. The code adds 2.8 MB to the app's binary. This is a rundown of why I built it, how it fits inside a Tauri app, and the benchmarks behind the model. Why I built dictation I think out loud. When I'm working through a problem or an idea, I talk it through, usually, and the structure I end up writing often starts from this random stream of thinking. I could also be reading an article and talking through the interesting points in it. So I wanted speaking to be a way to start a page, or an idea inside one, and then shape the writing from there. This has to be on device and work offline, both for privacy and latency reasons. It has to be accurate enough, but it’s a draft, not a transcript, it doesn't need to be perfect. Why whisper.cpp Platforms do offer their own speech recognition, but the APIs differ across macOS, Windows and Linux Linux has no standard one , and Set runs on all three. whisper.cpp is a C/C++ port of Whisper with no runtime dependencies, and it uses Apple's Metal GPU API on Macs. Set is a Rust app Tauri https://tauri.app , so it links whisper.cpp through the whisper-rs bindings, which compile whisper.cpp from source as part of the Cargo build. It’s the same model and code running on each platform. Choosing the model Whisper comes in sizes from tiny 39M parameters to large 1.55B . whisper.cpp publishes them as single ggml files, including quantized variants. I benchmarked warm six on the machine I develop on: - Machine: MacBook Pro 13-inch, M1, 2020 , 16 GB, macOS 14.2.1 - Library: whisper-rs 0.16 with Metal on - Settings: exactly as Set runs them: greedy decoding, language auto-detected, no timestamps, 7 threads - Accuracy: word error rate WER on 100 random utterances each from LibriSpeech https://www.openslr.org/12 test-clean 13.4 minutes, 2,200 words and test-other 10.7 minutes, 1,706 words; harder speakers and recording conditions - Latency: time to transcribe a 30, 60 and 120 second recording, the length of a real dictation take, median of 3 - Memory: peak memory footprint as macOS reports it /usr/bin/time -l | Model | Download | WER clean | WER other | 30 s take | 60 s take | 120 s take | Peak memory | |---|---|---|---|---|---|---|---| | tiny | 78 MB | 7.6% | 20.8% | 0.8 s | 1.8 s | 3.5 s | 165 MB | | base | 148 MB | 5.8% | 16.1% | 1.1 s | 1.3 s | 3.9 s | 237 MB | | small | 488 MB | 4.0% | 11.0% | 2.1 s | 3.5 s | 5.4 s | 627 MB | | small, q5 1 | 190 MB | 4.2% | 10.8% | 2.1 s | 3.5 s | 4.4 s | 346 MB | | medium | 1.5 GB | 3.4% | 8.7% | 5.4 s | 6.9 s | 10.8 s | 1,795 MB | | large-v3-turbo | 1.6 GB | 2.5% | 7.6% | 5.4 s | 8.7 s | 14.0 s | 1,793 MB | WER uses a simple normalizer lowercase, no punctuation, numbers spelled out , which is stricter than the one in the Whisper paper, so absolute numbers run slightly higher than published ones. Why small: - tiny and base miss too much: On clean speech they're fine, but on harder audio they get one word in five or six wrong. - small is the best trade: It cuts base's errors on hard audio by a third 16.1% → 11.0% , and a one-minute take comes back in 3.5 seconds. - medium and turbo cost more than they provide: They take two to three times as long and nearly three times the memory, for 0.6–1.5 points of WER on clean speech and 2.3–3.4 on hard audio. - Multilingual, not .en : Whisper also comes in English-only variants, but Set detects the language by default and lets you pick one of 15 languages, so it needs the multilingual model. These numbers are from an M1 with Metal. On Windows and Linux, Set's builds run whisper.cpp on the CPU only more below . The quantized model small-q5 1 is the same model with its weights quantized to 5 bits. On the first 200 clips it looked as accurate as the full model, so I ran both on 1,000 clips almost 20,000 words and compared them clip by clip: | | small | small, q5 1 | |---|---|---| | clean, 10,468 words | 3.93% | 3.90% | | other, 9,194 words | 10.23% | 10.18% | There's no measurable difference. On 468 of the 500 clean clips they make exactly the same number of errors, and the rest split evenly: 16 better, 16 worse. It's as fast or faster, a 190 MB download instead of 488 MB, and uses about 280 MB less memory, with Metal and on the CPU. Language detection On 6 of the 1,000 clips, each model decided that English speech was another language and transcribed it as Polish, Portuguese or Welsh, mostly on short clips of 5 to 8 words. Choosing English in settings fixed 6 of the 7 clips involved. Audio Audio comes in through cpal from the default input device at whatever format and rate the device prefers, which in practice is 44.1 or 48 kHz, often stereo, as floats or integers depending on the platform and device. Whisper wants 16 kHz mono floats. So Set: - Accepts every common sample format f32, i16, u16, i32, i8, u8 and converts to f32. - Downmixes to mono by averaging the channels. - Buffers the whole take in memory. Five minutes of mono f32 at 48 kHz is about 58 MB. - Reports the input level every 66 ms, for the level meter while you talk. - Caps a take at five minutes , then stops recording and transcribes what it has. When you stop, it resamples to 16 kHz by averaging each window of input samples rather than picking every third one, so high frequencies don't fold back into the speech band as noise aliasing . A proper low-pass filter would be better, but averaging is enough for speech going into Whisper. The engine Transcription only happens once for now when you stop talking: js let mut params = FullParams::new SamplingStrategy::Greedy { best of: 1 } ; params.set n threads threads ; // cores - 1, at most 8 params.set language Some language.unwrap or "auto" ; params.set no timestamps true ; params.set suppress blank true ; params.set suppress nst true ; // no " Music " or " coughs " - Greedy decoding , one candidate at a time, the fastest way to decode. - The model is loaded for each take and dropped afterwards. Loading it takes about 90 ms once the file is in the OS cache. The first dictation after launching the app is much slower, because the model is read from disk and Metal compiles its GPU kernels: with the full model, Set's log shows 8 seconds of speech taking 13.6 seconds, and 11 seconds taking 1.8 seconds two minutes later. - The text is inserted where you're writing. It replaces the selection, or goes in at the cursor. - whisper.cpp and ggml log constantly to stderr , which a bundled app sends to the system log. Set installs log callbacks that drop everything below a warning and route the rest into its own log. It never logs what you said. Costs Build: whisper-rs compiles whisper.cpp from C/C++ source. From scratch, that takes about 24 seconds on this 8-core M1, and longer on smaller CI runners. Binary size: The arm64 set binary is 32.7 MB with dictation and 29.9 MB without, so whisper.cpp, the audio library and the HTTP client add 2.8 MB 2.4 MB after stripping symbols . The model is the real cost, and that's downloaded only by people who turn dictation on. Model download: The model isn't bundled with the app. It would turn a 74 MB app into a 264 MB one for everyone, including people who never dictate. Instead, the first time you turn dictation on, Set downloads it from the whisper.cpp model repository on Hugging Face into the app's data folder. GPU support: Metal on macOS only. whisper.cpp also supports CUDA, Vulkan and HIP, but those need the vendor's toolkit at build time, which I can't assume on every build machine. So on Windows and Linux, dictation runs on the CPU for now. Measured on the same M1 with the GPU turned off: | small, M1 | WER clean | WER other | 30 s take | 60 s take | 120 s take | Peak memory | |---|---|---|---|---|---|---| | Metal | 4.0% | 11.0% | 2.1 s | 3.5 s | 5.4 s | 627 MB | | CPU only | 4.0% | 11.0% | 3.1 s | 5.0 s | 8.3 s | 968 MB | | CPU only, q5 1 | 4.2% | 10.8% | 2.8 s | 4.9 s | 7.3 s | 655 MB | With the full model, 199 of the 200 transcripts are identical to the Metal run 194 with q5 1 . It's about 50% slower and still usable: a minute of speech in five seconds. That's an M1's CPU, though, which is fast for a laptop. I haven't measured on the older x86 laptops that many Windows and Linux users will have. Next - Benchmark other languages. The benchmarks are only run against English speech. - Streaming. Right now you see nothing until you stop talking. whisper.cpp can transcribe in overlapping windows as audio arrives, which would show text as you speak. - GPU backends beyond Metal , once they can be built reliably on each platform's release runner.