Your auto-captioner is measuring the wrong thing: a ratio that fired on a perfectly good clip

I burn subtitles onto short AI-generated ad videos. The pipeline is small: generate the clip, transcribe it, write an .ass file, let ffmpeg burn it in.

It broke three times, and each break taught me something narrower and more useful than the last. The third one is the interesting one, because my fix was wrong in a way that looked completely right.

The first version used the timings from my own script. I wrote the lines, I knew roughly how long each one took to say, so I typed them in:

["A1_partner_snore", [
  [0.0, 3.2, "He snored through our entire honeymoon."],
  [3.2, 7.0, "I stopped hearing it on night one."]
]]

Every video was out of sync. Not slightly — by a second or more, drifting worse toward the end.

The reason is obvious in hindsight: I was timing my own reading of the line, and a generative video model has its own pacing. It s where it wants. It rushes short clauses. Nothing about my estimate was connected to the audio that actually exists in the file.

The only reliable source of timing is the audio itself:

import whisper
model = whisper.load_model("small")
res = model.transcribe(str(wav), language="en", fp16=False)
segs = [(s["start"], s["end"], s["text"].strip())
        for s in res["segments"] if s["text"].strip()]

Now the timing was right and the words were wrong.

Whisper mishears. In one clip "earplugs" came out as "Urplugs." Separately, the video model itself sometimes drops a word from the line it was handed.

So there are three versions of every sentence in play:

Neither source is usable alone, and the fix is to stop treating them as competing answers.

Take timing from the audio, take text from the script.

def align(heard, script_lines):
    if len(heard) == len(script_lines):
        return [(heard[i][0], heard[i][1], script_lines[i])
                for i in range(len(script_lines))]
    t0, t1 = heard[0][0], heard[-1][1]
    weights = [max(1, len(s.split())) for s in script_lines]
    total, out, t = sum(weights), [], t0
    for line, wt in zip(script_lines, weights):
        d = (t1 - t0) * wt / total
        out.append((t, t + d, line))
        t += d
    return out

This generalizes past captions. When two sources each have a known-good field and a known-bad field, joining on the good fields beats picking a winner.

Two clips came out with a single subtitle sitting on screen for about eight seconds, unmoving, while nothing was being said.

Those clips barely had dialogue. When speech is sparse, whisper happily extends a segment's end through the trailing silence. The segment is still correct as a transcript. It just isn't a caption cue.

I needed a guard that says this timing is untrustworthy, fall back to hand-written cues. Here was my first attempt:

longest = max(e - s for s, e, _ in segs)
if longest / duration > 0.40:
    return None

It caught both broken clips. It also rejected a clip that was completely fine.

The clip it wrongly rejected: 6.4 seconds inside a 15-second video, so 43% — over my threshold. But that segment was one long line of dialogue, delivered at 1.72 words per second, which is just... a person talking.

I spent a while wondering whether 40% should have been 50%, or 60%. That was the wrong question, and it's the part worth generalizing:

A long sentence taking a long time is not a defect. My ratio couldn't tell "someone spoke

for six seconds" apart from "someone spoke for one second and then there was five seconds of

nothing," because duration was in the numerator and nothing in the formula represented how

much was actually said.

No threshold fixes that. Under that measurement the two cases are genuinely identical, so any cutoff that catches one catches the other. Moving the number only chooses which failure you get.

What separates them is speech rate. Normal delivery runs about 2–3 words per second. Padding a segment with silence adds seconds without adding words, so the rate collapses:

MIN_WORDS_PER_SEC = 1.5

slowest = min((len(text.split()) / (e - s), s, e, text)
              for s, e, text in segs if e > s)
if slowest[0] < MIN_WORDS_PER_SEC:
    print(f"slowest segment is {slowest[0]:.2f} words/sec "
          f"({slowest[2] - slowest[1]:.1f}s of audio) "
          f"- silence merged into speech, falling back to manual cues")
    return None

Same shape of check, still one threshold, completely different behaviour — because the denominator now normalizes by the thing that actually causes the variance.

The heuristic that survives is the one whose units mean something. "Fraction of clip" is a number about the clip. "Words per second" is a number about speech — and speech was what I was trying to judge all along.

I only found the false positive because of a rule I now apply to every guard I write:

Run the regression in both directions. The bad input must be rejected and a

known-good input must still be accepted.

Testing only the bug you just saw is how you ship a fix that breaks something adjacent. My ratio guard passed the test I wrote for it — it rejected the broken clip perfectly. The clip it silently damaged was one I wasn't looking at, because it had never been broken.

That's the same failure mode as writing your own test fixtures and being pleased when they all pass. If the input and the expected output come out of the same head at the same moment, agreement between them proves nothing. Run the real corpus. Look at what changed for the items you didn't touch.

Number 3 is the one I keep re-learning. A badly tuned threshold is loud and easy to fix. A well-tuned threshold on the wrong quantity looks like a working system, right up until it quietly throws away something good.

I write about running unattended automation and AI pipelines in production, including the parts where the fix turns out to be the bug. The full system is the Claude Code Automation Playbook.