{"slug": "coverage-is-not-capability-what-whisper-taught-me-about-african-speech", "title": "Coverage Is Not Capability: What Whisper Taught Me About African Speech", "summary": "A developer building Scribe AI found that OpenAI's Whisper, despite being marketed as multilingual, performed poorly on the African languages and accents the project needed, tracing the failure to the model's training distribution: Whisper's 680,000 hours of audio include only about 5.4 hours of transcribed Swahili, and Dholuo is absent from its language list. The developer now argues that a language's presence on a supported-language list says little about capability, citing Meta's Omnilingual ASR, which reports character error rates below 10 for roughly 78% of evaluated languages but only about 36% of low-resource ones, and recommends hand-transcribed, in-domain test sets measured by WER or CER before committing to any speech model.", "body_md": "A language appearing in a model’s supported list says little about whether the model will work for your speakers. Why Whisper struggled with the speech I needed, and what I now check before building on any speech model.*\n\nWhen I built the first version of Scribe AI in July 2026, speech recognition was the part I expected to be easy.\n\nOpenAI’s Whisper was open, widely used, and described as multilingual. I ran it locally through Faster-Whisper, wired it into the live pipeline, and moved on to the parts I thought would be hard.\n\nThat expectation did not survive the speech I actually needed to handle. On the African languages and accents I cared about, the results were disappointing—not reliable enough to build on.\n\nThe problem was not a bug in my pipeline. When I went back to the Whisper paper, I found that the results were consistent with what the model’s own training data would lead us to expect.\n\n**What “Multilingual” Meant in Practice**\n\nWhisper was trained on 680,000 hours of labelled audio. The paper’s dataset statistics show how that total is divided:\n\n438,218 hours of English speech recognition\n\n125,739 hours of speech translation into English\n\n117,113 hours of speech recognition across 96 other languages\n\nWithin that last group, the distribution is extremely uneven.\n\nSwahili, one of Africa’s most widely spoken languages, is listed with approximately 5.4 hours of transcribed speech. Dholuo does not appear in Whisper’s language list.\n\n“The amount of pre-training speech recognition data for a given language is very predictive of zero-shot performance on that language.” — Radford et al., the Whisper paper\n\nWhisper was behaving in a way its training distribution helps explain.\n\nThe mistake was mine: I had treated multilingual as a uniform property of the model, when it actually described a highly uneven distribution of language coverage and capability.\n\nThe speech I needed sat in its thin tail.\n\n**Wider Coverage, the Same Question**\n\nWhisper is not the only option, and the field has moved forward.\n\nIn 2023, Meta released Massively Multilingual Speech (MMS), extending speech recognition to more than 1,100 languages. Much of its training data came from recorded readings of the New Testament, with roughly 32 hours per language.\n\nIn late 2025, Meta released Omnilingual ASR, which it describes as supporting more than 1,600 languages under the Apache 2.0 licence.\n\nThese are significant achievements. They expand what is possible for languages that previously had little or no speech recognition support.\n\nBut they sharpen the lesson rather than eliminate it.\n\nMeta reports a character error rate below 10 for approximately 78% of the languages evaluated with Omnilingual ASR. Among low-resource languages, that figure falls to approximately 36%.\n\nCoverage is a list. Capability is a distribution of error rates—usually measured on someone else’s test set.\n\nWhen evaluating a speech model now, three questions matter more to me than whether a language appears on its supported-language list.\n\n**1. Does It Work on My Speakers?**\n\nTraining and benchmark audio is often relatively clean and may consist largely of read speech.\n\nReal-world audio can be very different.\n\nA sermon, a phone call, or a customer-support conversation may contain spontaneous speech, unfamiliar accents, background noise, overlapping speakers, and frequent language switching.\n\nA model that performs well on a benchmark may struggle under those conditions.\n\nThe most useful next step is to evaluate it on a small set of recordings representative of the intended environment, transcribed by hand to establish reference transcripts.\n\nThen measure word error rate (WER) or character error rate (CER), as appropriate for the language and task.\n\nThe goal is not to find a model with an impressive headline metric. It is to find out how it performs on the speech your application actually needs to handle.\n\n**2. Can I Use It for What I Am Building?**\n\nModel quality is only one part of the decision. Licensing matters, too.\n\nThe original MMS models were released for non-commercial use only. That may be acceptable for some research projects, but it can prevent their use in a commercial product under those terms.\n\nA model can perform well technically and still be unsuitable for a particular deployment.\n\nBefore building around one, I now consider the licence alongside performance, intended use, redistribution conditions, and deployment requirements.\n\nAn evaluation that ignores licensing can lead to an expensive architectural dead end.\n\n**3. Can It Run Where My Users Are?**\n\nScribe AI is designed to run on the kind of laptop available at a church production desk, without depending on an internet connection.\n\nThat rules out models that are too large or too slow for the target hardware, regardless of how accurate they might be in a data centre.\n\nInference speed, memory requirements, quantisation support, and CPU performance all matter.\n\nA model that achieves excellent accuracy on powerful hardware but cannot keep up with live speech on the intended machine is not a practical solution for that setting.\n\nThis is why model evaluation cannot end with a benchmark table. The deployment environment is part of the problem being solved.\n\n**Where That Left Scribe—and Me**\n\nFor Scribe, the practical response was architectural.\n\nSpeech recognition sits in its own module, separate from detection and display, so a better model can be introduced later. A formal interface for swapping models is still on the roadmap.\n\nEvery detection also passes through a human operator before reaching the screen.\n\nNeither decision improves the underlying recognition quality. Together, however, they help keep the system usable while recognition remains imperfect.\n\nFor me, the experience raised a larger question.\n\nIf a developer in Kenya wants to build a voice product for people who speak Dholuo, Kalenjin, or even Kenyan-accented English, what should they build it on?\n\nSearching for an answer led me deeper into African speech and language AI as a field—and eventually to starting Sauti Labs.\n\nI wrote separately about that journey: Why I Started Sauti Labs.\n\nThe broader motivation is straightforward: building useful speech technology requires more than making a language appear on a list. It requires understanding the data, the speakers, the deployment environment, and the limitations that remain.\n\n**Sources**\n\nRadford, A., et al. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv.\n\nOpenAI. Whisper language list (tokenizer.py). GitHub.\n\nPratap, V., et al. (2023). Scaling Speech Technology to 1,000+ Languages. arXiv.\n\nThe Decoder. Meta’s new open-source models speak more than 1,100 languages.\n\nOmnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages. arXiv.\n\nThe Decoder. Meta’s Omnilingual ASR brings speech recognition to 1,600 languages.\n\nProject: Scribe AI — an offline-first system for detecting scripture references during church services, with human approval before display.", "url": "https://wpnews.pro/news/coverage-is-not-capability-what-whisper-taught-me-about-african-speech", "canonical_source": "https://dev.to/bernard_abuto_9970170b9c4/coverage-is-not-capability-what-whisper-taught-me-about-african-speech-39l2", "published_at": "2026-10-27 21:01:38+00:00", "updated_at": "2026-10-27 21:12:14.123672+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-tools"], "entities": ["OpenAI", "Whisper", "Scribe AI", "Meta", "Massively Multilingual Speech", "Omnilingual ASR", "Faster-Whisper"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/coverage-is-not-capability-what-whisper-taught-me-about-african-speech", "markdown": "https://wpnews.pro/news/coverage-is-not-capability-what-whisper-taught-me-about-african-speech.md", "text": "https://wpnews.pro/news/coverage-is-not-capability-what-whisper-taught-me-about-african-speech.txt", "jsonld": "https://wpnews.pro/news/coverage-is-not-capability-what-whisper-taught-me-about-african-speech.jsonld"}}