New here (hey 👋) and wanted to share something. We recently made public an ad intel app we'd been using internally in our agency. The app/system runs video ads through Gemini to pull out a transcript, the hook, and a few timed moments. It works fairly well - especially for ideation. The transcription is genuinely accurate.
A user opened a 2:22 video and saw a moment marked at 3:37.
Not a rounding error. Fifty percent past the end of a video the user was looking at.
The obvious guess is that the model didn't know how long the video was. That would be a reasonable bug and an easy fix.
But in the same JSON response, Gemini reported duration_seconds
correctly.
We measured it properly on one ad, four identical runs, same file, same prompt:
duration_seconds
reported by the model: t=214
The words were right. The clock was 50% long. The model was holding the correct duration and writing impossible timestamps anyway.
That is not a knowledge problem. It's a consistency problem, and those don't respond to being asked nicely.
We tried the things you'd try. State the duration in the prompt. Tell it no timestamp may exceed that. Restate the constraint after the schema. Put it in caps.
Failure rate went down. It didn't go to zero. A hardened run still drifted to t=218
on a 142-second video.
This is where I'd argue the general lesson lives:
If a model can violate a constraint, prompting reduces how often it does. Only code makes it impossible.
For most output that trade is fine, because most output is hard to falsify. Nobody checks whether a "psychological driver" is really the third most important one.
Timestamps are different. A timestamp is falsifiable in one click. The user scrubs to 3:37 on a 2:22 video and the whole analysis loses credibility, including all the parts that were correct. One checkable wrong number poisons everything unverifiable around it.
So we needed a ground truth the model couldn't wander away from.
The video is already in memory as a Buffer
before it goes anywhere near the model. MP4 files carry their own duration in the mvhd
atom (movie header). No API call, no dependency, and ffprobe
isn't installed on our host anyway.
The layout after the 4-byte mvhd
type is: version (1 byte), flags (3), then for version 0 — created (4), modified (4), timescale (4), duration (4). For version 1, the timestamps are 8 bytes and duration is 8.
Duration in seconds is duration / timescale
.
function mp4DurationSeconds(buf: Buffer): number | null {
const limit = buf.length - 24;
for (let i = 0; i < limit; i++) {
// scan for the ASCII bytes 'm','v','h','d'
if (buf[i] !== 0x6d || buf[i+1] !== 0x76 || buf[i+2] !== 0x68 || buf[i+3] !== 0x64) continue;
const version = buf[i + 4];
try {
const base = i + 8; // past 'mvhd' + version(1) + flags(3)
if (version === 1) {
const timescale = buf.readUInt32BE(base + 16);
const duration = Number(buf.readBigUInt64BE(base + 20));
if (timescale > 0 && duration > 0) return duration / timescale;
} else {
const timescale = buf.readUInt32BE(base + 8);
const duration = buf.readUInt32BE(base + 12);
if (timescale > 0 && duration > 0) return duration / timescale;
}
} catch {
// malformed atom, keep scanning for a valid one
}
}
return null;
}
Two deliberate choices worth explaining.
We scan for the atom rather than walking the box tree. Walking is more correct in principle. Scanning handles both faststart files (moov
at the front) and files with moov
at the end, without implementing a parser for a container format we only need one number out of.
A malformed atom keeps the loop going instead of throwing. The four bytes mvhd
can appear inside compressed video data by coincidence. If you bail on the first match that doesn't parse, a random byte sequence takes down your duration check.
With a real ceiling, every timestamp gets validated server-side before it reaches the client. Same doctrine as validating any untrusted input, because that's what model output is.
The interesting decisions were about what to do with a bad value, not how to spot one.
Keep the words, drop the clock. The transcription is accurate. Only the timing is untrustworthy. So an impossible timestamp is deleted and the text survives. Our transcript component already renders a blank gutter for a missing t
, so a stripped timestamp degrades into a clean script instead of a wrong number.
Null both ends of a pair, not just the bad one. We track "curiosity loops" with an open and a close time. If the close is impossible, nulling only that one leaves a surviving open time next to a blank close, which renders as precision we don't have. Both go.
This one cost me an afternoon and it's pure JavaScript, nothing to do with AI.
The first version coerced timestamps with Number(x)
. Looks harmless:
// WRONG
const t = Number(chunk.t);
Number(null)
is 0
.
A model that honestly returns null
for "I don't know when this happened" gets that turned into 0
, which renders as 0:00, which then reads as a loop closing before it opens, which trips the validation that was supposed to be protecting us. An honest null became a confident lie in one implicit coercion.
const asSeconds = (x: any): number | null => {
if (x === null || x === undefined || (typeof x === 'string' && !x.trim())) return null;
const n = Number(x);
return Number.isFinite(n) ? n : null;
};
Caught by a unit test, thankfully, and not by a user. If you're sanitising LLM output, null
and 0
mean completely different things and JavaScript will happily merge them for you.
The last piece is the one I'd have got wrong without looking at the actual failures.
The observed mode is a drifting tail. The model transcribes accurately the whole way through, but its clock stretches as it goes, so the late chunks are impossible while every earlier timestamp is fine. Those early ones are good data and worth keeping.
Scattered bad timestamps through the body are a different signal entirely. That means the timing pass itself is unreliable, and presenting a mix of right and wrong as though it were precise is worse than presenting nothing.
So: a contiguous run of bad timestamps ending at the final chunk is treated as tail drift and trimmed. Anything else strips all timings.
My first instinct was a percentage threshold, something like "if more than 25% are impossible, drop everything". That's wrong, and the reason is worth internalising:
The same failure would be judged differently depending on how many chunks the model happened to emit. A five-chunk drift is 19% of a 26-chunk transcript and 28% of an 18-chunk one. Identical failure, opposite verdicts, decided by something you don't control. Threshold on the shape of the failure, not its proportion.
A model can hold a fact and violate it in the same response. Reporting the duration correctly told us nothing about whether it would respect it.
Prompting sets a tendency. Code sets a guarantee. For anything a user can check in one click, you need the guarantee.
Find the ground truth in the artefact you already have. The buffer was in memory the whole time. We were asking a language model for a number that was sitting in a binary header 20 lines of code away.
Partial output usually beats no output, if you're honest about which part you dropped. The words were always right. Only the clock was wrong. Shipping an accurate transcript with no timestamps is better than shipping either a wrong timestamp or nothing at all.
I build SOCIALFUEL, which pulls the live ads any brand is running and decodes what makes them work. The video pipeline behind it is where this bug lived. If you're doing timed analysis on video with an LLM, I'd check your timestamps against the container before you trust them.