{"slug": "whisper-hallucinations-thanks-for-watching-in-9-of-my-transcripts", "title": "Whisper Hallucinations: 'Thanks for Watching' in 9% of My Transcripts", "summary": "A developer behind the voice interview practice app Preterview found that faster-whisper's large-v3 model hallucinated text in 9.2% of 2,000 answer clips, inserting phrases like \"Thanks for watching\" and \"Thank you\" into silent stretches and causing an LLM scorer to grade users on words they never said. The built-in no_speech_threshold failed to catch confident hallucinations, but enabling Silero VAD with speech_pad_ms=400, disabling condition_on_previous_text, and adding a guard on known phrases cut flagged clips from 183 to 4.", "body_md": "A user emailed me a screenshot of their feedback report. Under \"Communication,\" my scoring model had written: *\"Closed the answer with 'Thanks for watching,' which feels scripted and out of place in a live conversation.\"*\n\nThey never said it. They paused for about four seconds, decided they were done, and stopped talking. Whisper filled the silence for them, and then an LLM graded them on the words Whisper made up.\n\nThis post covers Whisper hallucinations on silence: how often they happened in my pipeline (9.2% of clips), why the built-in `no_speech_threshold` didn't catch them, and the config change that got the count from 183 to 4.\n\n`no_speech_threshold` alone won't save you.`no_speech_prob` is high `avg_logprob` is low. Confident hallucinations pass.`vad_filter=True` (Silero VAD) with `speech_pad_ms=400`, `condition_on_previous_text=False`, plus a small guard on known phrases. 183 dropped to 4.\nI run [Preterview](https://preterview.com/en), a voice interview practice app (full disclosure: I built it). You talk to an AI interviewer, and afterward you get a written report scored from your transcript. Live turn-taking runs on a streaming ASR, but the report uses a second, slower pass: every answer clip goes through faster-whisper with `large-v3` on a single GPU, and the cleaned transcript goes to Claude for scoring.\n\nThat second pass is where accuracy matters, so I thought it was the safe part. It was the part lying to me.\n\nIn my data, about 1 clip in 11. I pulled 2,000 answer clips from one week of sessions and flagged anything matching a list of known hallucination phrases, plus any segment with `compression_ratio > 2.4` (a decent signal for repetition loops). That flagged 196 clips. I listened to all 196. It took an afternoon and a lot of coffee.\n\n13 were real: people genuinely saying \"thank you\" at the end. The other 183 were invented.\n\n| Hallucination type | Clips | \n|---|---|\n| \"Thanks for watching!\" / \"Thank you for watching.\" | 71 | \n| Standalone \"Thank you.\" | 58 | \n| Repetition loops (same phrase 3+ times) | 27 | \n| Other (\"Please subscribe\", random short phrases) | 18 | \n| \"Subtitles by the Amara.org community\" | 9 | \n| **Total** | **183 / 2,000 (9.2%)** | \n\nThe location pattern was the giveaway. 164 of the 183 sat in the first or last segment of the clip, right next to at least two seconds of silence. When I split by trailing silence:\n\nInterview answers are basically designed to trigger this. People think before they speak, and they trail off when they finish. My recorder kept capturing a few seconds after the last word so it wouldn't clip anyone. That safety margin was the hallucination fuel.\n\nWhisper's decoder is a language model, and it was trained on a huge amount of weakly labeled web audio paired with captions. When the audio gives it almost nothing, the decoder falls back on its prior: what text usually appears over quiet, music, or a fade-out in its training data? Video outros. \"Thanks for watching.\" Caption credits.\n\nIt also always works on 30-second windows. A short clip gets padded, so the model is constantly looking at stretches of nothing and being asked to produce text for them. Most of the time it emits nothing or a no-speech token. Sometimes it confidently writes a sign-off.\n\nThe repetition loops have a separate cause. With `condition_on_previous_text=True` (the default), the previous segment's text gets fed in as a prompt. Once one segment goes wrong, the next one is conditioned on the wrong text, and you get \"I think that I think that I think that\" until the window ends.\n\nThis one cost me the most time. I assumed `no_speech_threshold=0.6` meant \"drop anything Whisper thinks is 60%+ likely to be silence.\" It doesn't. Here's the logic from openai-whisper's `transcribe.py` (faster-whisper does the same thing):\n\n```\nif no_speech_threshold is not None:\n    should_skip = result.no_speech_prob > no_speech_threshold\n    if logprob_threshold is not None and result.avg_logprob > logprob_threshold:\n        # don't skip if the logprob is high enough, despite the no_speech_prob\n        should_skip = False\n```\n\nA segment only gets skipped if no-speech probability is high **and** the average log-probability of the decoded text is low. If the model is confident about its made-up text, the segment survives.\n\nAnd \"Thanks for watching\" is something Whisper is *very* confident about. Across my 183 hallucinations, the median `no_speech_prob` was 0.71, comfortably above the 0.6 threshold. But 129 of them had `avg_logprob` above -1.0, so the skip got overruled.\n\nWhat I tried before the real fix:\n\n`logprob_threshold=None`\nThresholds were the wrong layer. The problem was that silence reached the decoder at all.\n\nStop sending silence to the model. faster-whisper ships a built-in Silero VAD filter that strips non-speech regions before decoding. Here's the config I landed on:\n\n``` python\nfrom faster_whisper import WhisperModel\n\nmodel = WhisperModel(\"large-v3\", device=\"cuda\", compute_type=\"float16\")\n\nKNOWN_GHOSTS = {\n    \"thanks for watching\", \"thank you for watching\", \"thank you\",\n    \"please subscribe\", \"subtitles by the amara.org community\",\n}\n\ndef transcribe_answer(path: str) -> str:\n    segments, _ = model.transcribe(\n        path,\n        beam_size=5,\n        vad_filter=True,\n        vad_parameters=dict(min_silence_duration_ms=500, speech_pad_ms=400),\n        condition_on_previous_text=False,\n    )\n    kept = []\n    for seg in segments:\n        norm = seg.text.strip().lower().rstrip(\".!\")\n        if norm in KNOWN_GHOSTS and seg.no_speech_prob > 0.5:\n            continue  # confident ghost in a quiet region\n        kept.append(seg.text.strip())\n    return \" \".join(kept)\n```\n\nI added changes one at a time and re-ran the same 2,000 clips after each:\n\n| Change | Hallucinations | Real speech lost | \n|---|---|---|\n| Baseline | 183 | 0 | \n| `condition_on_previous_text=False` | 159 | 0 | \n| + `vad_filter=True` (default padding) | 22 | 11 clips | \n| + `speech_pad_ms=400` | 24 | 3 clips | \n| + known-phrase guard | **4** | 3 clips + 1 real \"thank you\" | \n\nTurning off conditioning killed the loops (27 down to 3). VAD did the heavy lifting on everything else.\n\nThe VAD had its own failure mode: with default padding, 11 clips lost their first word. Soft openers like \"So,\" or \"Um, yeah\" got classified as noise. Bumping `speech_pad_ms` to 400 brought that to 3 and let 2 hallucinations back in. I took that trade.\n\nThe phrase guard is deliberately narrow. It only drops a known phrase when the segment's own `no_speech_prob` is above 0.5. Of the 13 real \"thank you\"s, 12 had `no_speech_prob` under 0.2 and survived. One person said it so quietly that it got dropped. I can live with that.\n\nTwo were \"Thank you.\" in clips with loud keyboard noise, where VAD called the typing speech. One was a lone \"you.\" One was a three-time \"um\" loop that survived because conditioning wasn't the cause. I stopped there. Going from 4 to 0 would mean more aggressive VAD, and that eats real words.\n\nTwo things, both cheap.\n\n**The scorer prompt now knows transcripts are machine-made.** I added a line telling Claude the transcript came from automatic speech recognition, may contain artifacts like sign-offs or repeated phrases, and must never penalize those. Even with perfect ASR config, a judge that trusts its input blindly is one bad clip away from writing nonsense about a real person.\n\n**Hallucination rate is now a metric.** A nightly job runs the phrase list and the compression-ratio check over the day's transcripts and alerts if the flagged rate goes above 1%. It sits around 0.3%. If a model upgrade or a recorder change pushes it back up, I'll know before a user emails me a screenshot.\n\nA bonus I didn't expect: VAD made transcription faster. Skipping silent regions cut average GPU time per clip from 0.84s to 0.61s. The fix paid for itself.\n\nWhisper says \"Thanks for watching\" on silence because its decoder is a language model trained on web video captions, and when the audio carries no speech it falls back to the text that most often appears over quiet endings in that data: video outros and caption credits. The default `no_speech_threshold` doesn't stop it, since a segment is only skipped when no-speech probability is high *and* decoding confidence is low, and these phrases decode with high confidence. In my pipeline that meant 183 of 2,000 clips (9.2%) contained invented text. The reliable fix is to remove silence before decoding: enable `vad_filter=True` in faster-whisper with some speech padding, set `condition_on_previous_text=False` to stop repetition loops, and add a narrow guard for known phrases in high no-speech segments. That took my count from 183 to 4.\n\n*Written by the developer behind [Preterview](https://preterview.com/en), an interview prep platform.*", "url": "https://wpnews.pro/news/whisper-hallucinations-thanks-for-watching-in-9-of-my-transcripts", "canonical_source": "https://dev.to/ji_ai/whisper-hallucinations-thanks-for-watching-in-9-of-my-transcripts-go5", "published_at": "2026-10-17 23:23:14+00:00", "updated_at": "2026-10-17 23:37:41.528281+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-tools", "natural-language-processing"], "entities": ["Preterview", "Whisper", "faster-whisper", "Silero VAD", "Claude", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/whisper-hallucinations-thanks-for-watching-in-9-of-my-transcripts", "markdown": "https://wpnews.pro/news/whisper-hallucinations-thanks-for-watching-in-9-of-my-transcripts.md", "text": "https://wpnews.pro/news/whisper-hallucinations-thanks-for-watching-in-9-of-my-transcripts.txt", "jsonld": "https://wpnews.pro/news/whisper-hallucinations-thanks-for-watching-in-9-of-my-transcripts.jsonld"}}