{"slug": "your-auto-captioner-is-measuring-the-wrong-thing-a-ratio-that-fired-on-a-good", "title": "Your auto-captioner is measuring the wrong thing: a ratio that fired on a perfectly good clip", "summary": "A developer building an automated subtitle pipeline for AI-generated ad videos discovered that their caption timing validation was flawed. The developer found that a ratio-based guard, which rejected clips where a single subtitle segment exceeded 40% of the video duration, incorrectly flagged a perfectly good clip featuring a long, natural speech segment. The fix involved recognizing that a long sentence taking a long time is not a defect, and instead validating timing based on the amount of speech content.", "body_md": "I burn subtitles onto short AI-generated ad videos. The pipeline is small: generate the clip, transcribe it, write an `.ass` file, let ffmpeg burn it in.\n\nIt broke three times, and each break taught me something narrower and more useful than the last. The third one is the interesting one, because my fix was wrong in a way that looked completely right.\n\nThe first version used the timings from my own script. I wrote the lines, I knew roughly how long each one took to say, so I typed them in:\n\n```\n[\"A1_partner_snore\", [\n  [0.0, 3.2, \"He snored through our entire honeymoon.\"],\n  [3.2, 7.0, \"I stopped hearing it on night one.\"]\n]]\n```\n\nEvery video was out of sync. Not slightly — by a second or more, drifting worse toward the end.\n\nThe reason is obvious in hindsight: **I was timing my own reading of the line, and a generative video model has its own pacing.** It pauses where it wants. It rushes short clauses. Nothing about my estimate was connected to the audio that actually exists in the file.\n\nThe only reliable source of timing is the audio itself:\n\n``` python\nimport whisper\nmodel = whisper.load_model(\"small\")\nres = model.transcribe(str(wav), language=\"en\", fp16=False)\nsegs = [(s[\"start\"], s[\"end\"], s[\"text\"].strip())\n        for s in res[\"segments\"] if s[\"text\"].strip()]\n```\n\nNow the timing was right and the words were wrong.\n\nWhisper mishears. In one clip \"earplugs\" came out as \"Urplugs.\" Separately, the video model itself sometimes drops a word from the line it was handed.\n\nSo there are three versions of every sentence in play:\n\n| Source | Timing | Wording | \n|---|---|---|\n| My script | wrong | correct | \n| Whisper transcript | correct | wrong, twice over | \n| What I want on screen | correct | correct | \n\nNeither source is usable alone, and the fix is to stop treating them as competing answers.\n\n**Take timing from the audio, take text from the script.**\n\n``` python\ndef align(heard, script_lines):\n    if len(heard) == len(script_lines):\n        return [(heard[i][0], heard[i][1], script_lines[i])\n                for i in range(len(script_lines))]\n    # segment counts disagree: spread the script across the spoken span,\n    # weighted by word count\n    t0, t1 = heard[0][0], heard[-1][1]\n    weights = [max(1, len(s.split())) for s in script_lines]\n    total, out, t = sum(weights), [], t0\n    for line, wt in zip(script_lines, weights):\n        d = (t1 - t0) * wt / total\n        out.append((t, t + d, line))\n        t += d\n    return out\n```\n\nThis generalizes past captions. When two sources each have a known-good field and a known-bad field, joining on the good fields beats picking a winner.\n\nTwo clips came out with a single subtitle sitting on screen for about eight seconds, unmoving, while nothing was being said.\n\nThose clips barely had dialogue. When speech is sparse, whisper happily extends a segment's `end` through the trailing silence. The segment is still correct *as a transcript*. It just isn't a caption cue.\n\nI needed a guard that says *this timing is untrustworthy, fall back to hand-written cues*. Here was my first attempt:\n\n```\n# if any single segment covers more than 40% of the clip, distrust the timing\nlongest = max(e - s for s, e, _ in segs)\nif longest / duration > 0.40:\n    return None\n```\n\nIt caught both broken clips. It also rejected a clip that was completely fine.\n\nThe clip it wrongly rejected: 6.4 seconds inside a 15-second video, so 43% — over my threshold. But that segment was one long line of dialogue, delivered at **1.72 words per second**, which is just... a person talking.\n\nI spent a while wondering whether 40% should have been 50%, or 60%. That was the wrong question, and it's the part worth generalizing:\n\n**A long sentence taking a long time is not a defect.** My ratio couldn't tell \"someone spoke\n\nfor six seconds\" apart from \"someone spoke for one second and then there was five seconds of\n\nnothing,\" because duration was in the numerator and nothing in the formula represented how\n\nmuch was actually *said*.\n\nNo threshold fixes that. Under that measurement the two cases are genuinely identical, so any cutoff that catches one catches the other. Moving the number only chooses which failure you get.\n\nWhat separates them is **speech rate**. Normal delivery runs about 2–3 words per second. Padding a segment with silence adds seconds without adding words, so the rate collapses:\n\n```\nMIN_WORDS_PER_SEC = 1.5\n\nslowest = min((len(text.split()) / (e - s), s, e, text)\n              for s, e, text in segs if e > s)\nif slowest[0] < MIN_WORDS_PER_SEC:\n    print(f\"slowest segment is {slowest[0]:.2f} words/sec \"\n          f\"({slowest[2] - slowest[1]:.1f}s of audio) \"\n          f\"- silence merged into speech, falling back to manual cues\")\n    return None\n```\n\nSame shape of check, still one threshold, completely different behaviour — because the denominator now normalizes by the thing that actually causes the variance.\n\n| Clip | Longest segment | % of clip | Words/sec | Ratio guard | Rate guard | \n|---|---|---|---|---|---|\n| Long single line (good) | 6.4s | 43% | 1.72 | ❌ rejected | ✅ accepted | \n| Sparse dialogue (broken) | 8.1s | 54% | 0.49 | ✅ rejected | ✅ rejected | \n\nThe heuristic that survives is the one whose units mean something. \"Fraction of clip\" is a number about the clip. \"Words per second\" is a number about speech — and speech was what I was trying to judge all along.\n\nI only found the false positive because of a rule I now apply to every guard I write:\n\nRun the regression in **both** directions. The bad input must be rejected **and** a\n\nknown-good input must still be accepted.\n\nTesting only the bug you just saw is how you ship a fix that breaks something adjacent. My ratio guard passed the test I wrote for it — it rejected the broken clip perfectly. The clip it silently damaged was one I wasn't looking at, because it had never been broken.\n\nThat's the same failure mode as writing your own test fixtures and being pleased when they all pass. If the input and the expected output come out of the same head at the same moment, agreement between them proves nothing. Run the real corpus. Look at what changed for the items you didn't touch.\n\nNumber 3 is the one I keep re-learning. A badly tuned threshold is loud and easy to fix. A well-tuned threshold on the wrong quantity looks like a working system, right up until it quietly throws away something good.\n\n*I write about running unattended automation and AI pipelines in production, including the\nparts where the fix turns out to be the bug. The full system is the [Claude Code Automation Playbook](https://alphatech4.gumroad.com/l/claude-code-automation-playbook).*", "url": "https://wpnews.pro/news/your-auto-captioner-is-measuring-the-wrong-thing-a-ratio-that-fired-on-a-good", "canonical_source": "https://dev.to/youfuhsu/your-auto-captioner-is-measuring-the-wrong-thing-a-ratio-that-fired-on-a-perfectly-good-clip-1dh2", "published_at": "2026-09-07 04:54:19+00:00", "updated_at": "2026-09-07 05:28:22.947928+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "natural-language-processing"], "entities": ["Whisper", "ffmpeg"], "alternates": {"html": "https://wpnews.pro/news/your-auto-captioner-is-measuring-the-wrong-thing-a-ratio-that-fired-on-a-good", "markdown": "https://wpnews.pro/news/your-auto-captioner-is-measuring-the-wrong-thing-a-ratio-that-fired-on-a-good.md", "text": "https://wpnews.pro/news/your-auto-captioner-is-measuring-the-wrong-thing-a-ratio-that-fired-on-a-good.txt", "jsonld": "https://wpnews.pro/news/your-auto-captioner-is-measuring-the-wrong-thing-a-ratio-that-fired-on-a-good.jsonld"}}