Your auto-captioner is measuring the wrong thing: a ratio that fired on a perfectly good clip

A developer building an automated subtitle pipeline for AI-generated ad videos discovered that their caption timing validation was flawed. The developer found that a ratio-based guard, which rejected clips where a single subtitle segment exceeded 40% of the video duration, incorrectly flagged a perfectly good clip featuring a long, natural speech segment. The fix involved recognizing that a long sentence taking a long time is not a defect, and instead validating timing based on the amount of speech content.

I burn subtitles onto short AI-generated ad videos. The pipeline is small: generate the clip, transcribe it, write an .ass file, let ffmpeg burn it in. It broke three times, and each break taught me something narrower and more useful than the last. The third one is the interesting one, because my fix was wrong in a way that looked completely right. The first version used the timings from my own script. I wrote the lines, I knew roughly how long each one took to say, so I typed them in: "A1 partner snore", 0.0, 3.2, "He snored through our entire honeymoon." , 3.2, 7.0, "I stopped hearing it on night one." Every video was out of sync. Not slightly — by a second or more, drifting worse toward the end. The reason is obvious in hindsight: I was timing my own reading of the line, and a generative video model has its own pacing. It pauses where it wants. It rushes short clauses. Nothing about my estimate was connected to the audio that actually exists in the file. The only reliable source of timing is the audio itself: python import whisper model = whisper.load model "small" res = model.transcribe str wav , language="en", fp16=False segs = s "start" , s "end" , s "text" .strip for s in res "segments" if s "text" .strip Now the timing was right and the words were wrong. Whisper mishears. In one clip "earplugs" came out as "Urplugs." Separately, the video model itself sometimes drops a word from the line it was handed. So there are three versions of every sentence in play: | Source | Timing | Wording | |---|---|---| | My script | wrong | correct | | Whisper transcript | correct | wrong, twice over | | What I want on screen | correct | correct | Neither source is usable alone, and the fix is to stop treating them as competing answers. Take timing from the audio, take text from the script. python def align heard, script lines : if len heard == len script lines : return heard i 0 , heard i 1 , script lines i for i in range len script lines segment counts disagree: spread the script across the spoken span, weighted by word count t0, t1 = heard 0 0 , heard -1 1 weights = max 1, len s.split for s in script lines total, out, t = sum weights , , t0 for line, wt in zip script lines, weights : d = t1 - t0 wt / total out.append t, t + d, line t += d return out This generalizes past captions. When two sources each have a known-good field and a known-bad field, joining on the good fields beats picking a winner. Two clips came out with a single subtitle sitting on screen for about eight seconds, unmoving, while nothing was being said. Those clips barely had dialogue. When speech is sparse, whisper happily extends a segment's end through the trailing silence. The segment is still correct as a transcript . It just isn't a caption cue. I needed a guard that says this timing is untrustworthy, fall back to hand-written cues . Here was my first attempt: if any single segment covers more than 40% of the clip, distrust the timing longest = max e - s for s, e, in segs if longest / duration 0.40: return None It caught both broken clips. It also rejected a clip that was completely fine. The clip it wrongly rejected: 6.4 seconds inside a 15-second video, so 43% — over my threshold. But that segment was one long line of dialogue, delivered at 1.72 words per second , which is just... a person talking. I spent a while wondering whether 40% should have been 50%, or 60%. That was the wrong question, and it's the part worth generalizing: A long sentence taking a long time is not a defect. My ratio couldn't tell "someone spoke for six seconds" apart from "someone spoke for one second and then there was five seconds of nothing," because duration was in the numerator and nothing in the formula represented how much was actually said . No threshold fixes that. Under that measurement the two cases are genuinely identical, so any cutoff that catches one catches the other. Moving the number only chooses which failure you get. What separates them is speech rate . Normal delivery runs about 2–3 words per second. Padding a segment with silence adds seconds without adding words, so the rate collapses: MIN WORDS PER SEC = 1.5 slowest = min len text.split / e - s , s, e, text for s, e, text in segs if e s if slowest 0 < MIN WORDS PER SEC: print f"slowest segment is {slowest 0 :.2f} words/sec " f" {slowest 2 - slowest 1 :.1f}s of audio " f"- silence merged into speech, falling back to manual cues" return None Same shape of check, still one threshold, completely different behaviour — because the denominator now normalizes by the thing that actually causes the variance. | Clip | Longest segment | % of clip | Words/sec | Ratio guard | Rate guard | |---|---|---|---|---|---| | Long single line good | 6.4s | 43% | 1.72 | ❌ rejected | ✅ accepted | | Sparse dialogue broken | 8.1s | 54% | 0.49 | ✅ rejected | ✅ rejected | The heuristic that survives is the one whose units mean something. "Fraction of clip" is a number about the clip. "Words per second" is a number about speech — and speech was what I was trying to judge all along. I only found the false positive because of a rule I now apply to every guard I write: Run the regression in both directions. The bad input must be rejected and a known-good input must still be accepted. Testing only the bug you just saw is how you ship a fix that breaks something adjacent. My ratio guard passed the test I wrote for it — it rejected the broken clip perfectly. The clip it silently damaged was one I wasn't looking at, because it had never been broken. That's the same failure mode as writing your own test fixtures and being pleased when they all pass. If the input and the expected output come out of the same head at the same moment, agreement between them proves nothing. Run the real corpus. Look at what changed for the items you didn't touch. Number 3 is the one I keep re-learning. A badly tuned threshold is loud and easy to fix. A well-tuned threshold on the wrong quantity looks like a working system, right up until it quietly throws away something good. I write about running unattended automation and AI pipelines in production, including the parts where the fix turns out to be the bug. The full system is the Claude Code Automation Playbook https://alphatech4.gumroad.com/l/claude-code-automation-playbook .