{"slug": "your-tts-model-sounds-great-until-it-says-gpub", "title": "Your TTS Model Sounds Great — Until It Says \"GPUB\"", "summary": "A developer built ttsproof, an open-source QA framework for text-to-speech systems that separates structural defects from pronunciation errors, addressing the shortcomings of Word Error Rate metrics. In a blind study against a production neural TTS service with 390 samples, the tool identified 19 genuine mispronunciations, all involving short letters and acronyms, and demonstrated that naive ASR-based evaluation can wrongly fail correct clips. The framework includes a corpus of 817 edge cases and is available via pip.", "body_md": "*Originally published at ai.bedvibe.studio.*\n\nI built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it.\n\nWorse, the metric everyone reaches for — Word Error Rate — was lying to me in **both** directions. It flagged perfectly good audio as broken because the script said `3:30 PM`\n\nand the transcript said \"three thirty pee em.\" And it *missed* real failures on short tokens, where the speech recogniser is as unreliable as the TTS.\n\nSo I wrote the QA framework I wished I had, packaged it as **ttsproof**, and then ran it as a blind study against a production TTS service so the results would be more than an opinion.\n\nA TTS pipeline breaks in two different ways, and a single WER number blurs both.\n\n**Structural defects.** The clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, straight from the waveform.\n\n**Pronunciation and content errors** on the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names.\n\nttsproof splits them apart and handles each one honestly:\n\n`3:30 PM`\n\nagainst \"three thirty\" stops counting as an error.I evaluated the method against a production neural TTS service — **130 edge cases × 3 voices = 390 samples** — and published it as a citable technical report ([DOI 10.5281/zenodo.20757553](https://doi.org/10.5281/zenodo.20757553), CC-BY-4.0).\n\nThat 45/55 split is the entire argument for having a quarantine verdict at all. Auto-passing that zone ships 19 real mispronunciations. Naive ASR-WER auto-failing it wrongly kills 23 correct clips. Neither is acceptable, so ttsproof refuses to guess there.\n\nAll 19 genuine failures were short isolated letters and acronyms, and the pattern is oddly specific:\n\n| Failure mode | Examples |\n|---|---|\n| A-vowel substitution |\n`NATO` → \"NITO\", `USA` → \"USI\", `CIA` → \"CII\" |\n| Trailing appended phoneme |\n`GPU` → \"GPUB\", `EU` → \"EUU\" |\n| Early truncation |\n`R` chopped short |\n| Doubling |\n`X` said twice |\n| Other substitution |\n`CEO` → \"CEE\", `Z` → \"SZ\" |\n\nThe structural detectors did **not** fire on any of these. \"GPUB\" is intelligible speech, not a click. Structural checks and ASR quarantine are complementary; neither alone catches everything.\n\nBeyond the study, ttsproof ships a corpus of **817 curated edge cases across 39 categories** — numbers, currencies, dates, ISO timestamps, phone numbers, URLs, file paths, pronunciation-torture words (Worcestershire, synecdoche), proper names (Reykjavík, Nguyễn), Greek, Norwegian and more. The corpus is versioned independently of the software, so published scores stay comparable across tool updates.\n\nIt is engine-agnostic — point it at any TTS via a command template, or at a folder of audio you already generated:\n\n```\nttsproof benchmark --cmd \"mytts --text {text} --wav {out}\"\n```\n\nYou get a category scoreboard, a self-contained `report.html`\n\nwith waveforms, an audio player and what the ASR actually heard, and a CI regression gate. Closed-source engines work too through a SpeechSDK wrapper — an integration a user suggested after the first release.\n\n```\npip install ttsproof            # structural checks + metrics + corpus\npip install \"ttsproof[asr]\"     # + faster-whisper for pronunciation gating\n```\n\nIt has already had its first outside contribution, a community fix for a real number-formatting bug, which is exactly what I hoped for. If your TTS breaks on something, open an issue with the case — the corpus grows from real failures.", "url": "https://wpnews.pro/news/your-tts-model-sounds-great-until-it-says-gpub", "canonical_source": "https://dev.to/bedvibe_studios/your-tts-model-sounds-great-until-it-says-gpub-1d77", "published_at": "2026-08-25 03:10:05+00:00", "updated_at": "2026-08-25 03:43:20.488995+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "machine-learning"], "entities": ["ttsproof", "faster-whisper", "SpeechSDK", "Zenodo"], "alternates": {"html": "https://wpnews.pro/news/your-tts-model-sounds-great-until-it-says-gpub", "markdown": "https://wpnews.pro/news/your-tts-model-sounds-great-until-it-says-gpub.md", "text": "https://wpnews.pro/news/your-tts-model-sounds-great-until-it-says-gpub.txt", "jsonld": "https://wpnews.pro/news/your-tts-model-sounds-great-until-it-says-gpub.jsonld"}}