Originally published at ai.bedvibe.studio.
I built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it.
Worse, the metric everyone reaches for — Word Error Rate — was lying to me in both directions. It flagged perfectly good audio as broken because the script said 3:30 PM
and the transcript said "three thirty pee em." And it missed real failures on short tokens, where the speech recogniser is as unreliable as the TTS.
So I wrote the QA framework I wished I had, packaged it as ttsproof, and then ran it as a blind study against a production TTS service so the results would be more than an opinion.
A TTS pipeline breaks in two different ways, and a single WER number blurs both.
Structural defects. The clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, straight from the waveform.
Pronunciation and content errors on the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names.
ttsproof splits them apart and handles each one honestly:
3:30 PM
against "three thirty" stops counting as an error.I evaluated the method against a production neural TTS service — 130 edge cases × 3 voices = 390 samples — and published it as a citable technical report (DOI 10.5281/zenodo.20757553, CC-BY-4.0).
That 45/55 split is the entire argument for having a quarantine verdict at all. Auto-passing that zone ships 19 real mispronunciations. Naive ASR-WER auto-failing it wrongly kills 23 correct clips. Neither is acceptable, so ttsproof refuses to guess there.
All 19 genuine failures were short isolated letters and acronyms, and the pattern is oddly specific:
| Failure mode | Examples |
|---|---|
| A-vowel substitution | |
NATO → "NITO", USA → "USI", CIA → "CII" |
|
| Trailing appended phoneme | |
GPU → "GPUB", EU → "EUU" |
|
| Early truncation | |
R chopped short |
|
| Doubling | |
X said twice |
|
| Other substitution | |
CEO → "CEE", Z → "SZ" |
The structural detectors did not fire on any of these. "GPUB" is intelligible speech, not a click. Structural checks and ASR quarantine are complementary; neither alone catches everything.
Beyond the study, ttsproof ships a corpus of 817 curated edge cases across 39 categories — numbers, currencies, dates, ISO timestamps, phone numbers, URLs, file paths, pronunciation-torture words (Worcestershire, synecdoche), proper names (Reykjavík, Nguyễn), Greek, Norwegian and more. The corpus is versioned independently of the software, so published scores stay comparable across tool updates.
It is engine-agnostic — point it at any TTS via a command template, or at a folder of audio you already generated:
ttsproof benchmark --cmd "mytts --text {text} --wav {out}"
You get a category scoreboard, a self-contained report.html
with waveforms, an audio player and what the ASR actually heard, and a CI regression gate. Closed-source engines work too through a SpeechSDK wrapper — an integration a user suggested after the first release.
pip install ttsproof # structural checks + metrics + corpus
pip install "ttsproof[asr]" # + faster-whisper for pronunciation gating
It has already had its first outside contribution, a community fix for a real number-formatting bug, which is exactly what I hoped for. If your TTS breaks on something, open an issue with the case — the corpus grows from real failures.