A language appearing in a model’s supported list says little about whether the model will work for your speakers. Why Whisper struggled with the speech I needed, and what I now check before building on any speech model.*
When I built the first version of Scribe AI in July 2026, speech recognition was the part I expected to be easy.
OpenAI’s Whisper was open, widely used, and described as multilingual. I ran it locally through Faster-Whisper, wired it into the live pipeline, and moved on to the parts I thought would be hard.
That expectation did not survive the speech I actually needed to handle. On the African languages and accents I cared about, the results were disappointing—not reliable enough to build on.
The problem was not a bug in my pipeline. When I went back to the Whisper paper, I found that the results were consistent with what the model’s own training data would lead us to expect.
What “Multilingual” Meant in Practice
Whisper was trained on 680,000 hours of labelled audio. The paper’s dataset statistics show how that total is divided:
438,218 hours of English speech recognition
125,739 hours of speech translation into English
117,113 hours of speech recognition across 96 other languages
Within that last group, the distribution is extremely uneven.
Swahili, one of Africa’s most widely spoken languages, is listed with approximately 5.4 hours of transcribed speech. Dholuo does not appear in Whisper’s language list.
“The amount of pre-training speech recognition data for a given language is very predictive of zero-shot performance on that language.” — Radford et al., the Whisper paper
Whisper was behaving in a way its training distribution helps explain.
The mistake was mine: I had treated multilingual as a uniform property of the model, when it actually described a highly uneven distribution of language coverage and capability.
The speech I needed sat in its thin tail.
Wider Coverage, the Same Question
Whisper is not the only option, and the field has moved forward.
In 2023, Meta released Massively Multilingual Speech (MMS), extending speech recognition to more than 1,100 languages. Much of its training data came from recorded readings of the New Testament, with roughly 32 hours per language.
In late 2025, Meta released Omnilingual ASR, which it describes as supporting more than 1,600 languages under the Apache 2.0 licence.
These are significant achievements. They expand what is possible for languages that previously had little or no speech recognition support.
But they sharpen the lesson rather than eliminate it.
Meta reports a character error rate below 10 for approximately 78% of the languages evaluated with Omnilingual ASR. Among low-resource languages, that figure falls to approximately 36%.
Coverage is a list. Capability is a distribution of error rates—usually measured on someone else’s test set.
When evaluating a speech model now, three questions matter more to me than whether a language appears on its supported-language list.
1. Does It Work on My Speakers?
Training and benchmark audio is often relatively clean and may consist largely of read speech.
Real-world audio can be very different.
A sermon, a phone call, or a customer-support conversation may contain spontaneous speech, unfamiliar accents, background noise, overlapping speakers, and frequent language switching.
A model that performs well on a benchmark may struggle under those conditions.
The most useful next step is to evaluate it on a small set of recordings representative of the intended environment, transcribed by hand to establish reference transcripts.
Then measure word error rate (WER) or character error rate (CER), as appropriate for the language and task.
The goal is not to find a model with an impressive headline metric. It is to find out how it performs on the speech your application actually needs to handle.
2. Can I Use It for What I Am Building?
Model quality is only one part of the decision. Licensing matters, too.
The original MMS models were released for non-commercial use only. That may be acceptable for some research projects, but it can prevent their use in a commercial product under those terms.
A model can perform well technically and still be unsuitable for a particular deployment.
Before building around one, I now consider the licence alongside performance, intended use, redistribution conditions, and deployment requirements.
An evaluation that ignores licensing can lead to an expensive architectural dead end.
3. Can It Run Where My Users Are?
Scribe AI is designed to run on the kind of laptop available at a church production desk, without depending on an internet connection.
That rules out models that are too large or too slow for the target hardware, regardless of how accurate they might be in a data centre.
Inference speed, memory requirements, quantisation support, and CPU performance all matter.
A model that achieves excellent accuracy on powerful hardware but cannot keep up with live speech on the intended machine is not a practical solution for that setting.
This is why model evaluation cannot end with a benchmark table. The deployment environment is part of the problem being solved.
Where That Left Scribe—and Me
For Scribe, the practical response was architectural. Speech recognition sits in its own module, separate from detection and display, so a better model can be introduced later. A formal interface for swapping models is still on the roadmap.
Every detection also passes through a human operator before reaching the screen.
Neither decision improves the underlying recognition quality. Together, however, they help keep the system usable while recognition remains imperfect.
For me, the experience raised a larger question.
If a developer in Kenya wants to build a voice product for people who speak Dholuo, Kalenjin, or even Kenyan-accented English, what should they build it on?
Searching for an answer led me deeper into African speech and language AI as a field—and eventually to starting Sauti Labs.
I wrote separately about that journey: Why I Started Sauti Labs.
The broader motivation is straightforward: building useful speech technology requires more than making a language appear on a list. It requires understanding the data, the speakers, the deployment environment, and the limitations that remain.
Sources
Radford, A., et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv.
OpenAI. Whisper language list (tokenizer.py). GitHub.
Pratap, V., et al. (2023). Scaling Speech Technology to 1,000+ Languages. arXiv.
The Decoder. Meta’s new open-source models speak more than 1,100 languages.
Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages. arXiv.
The Decoder. Meta’s Omnilingual ASR brings speech recognition to 1,600 languages.
Project: Scribe AI — an offline-first system for detecting scripture references during church services, with human approval before display.