{"slug": "prompt-engineering-couldn-t-fix-this-llm-bug-20-lines-of-binary-parsing-did", "title": "Prompt engineering couldn't fix this LLM bug. 20 lines of binary parsing did.", "summary": "An agency's ad intelligence app, which uses Gemini to extract transcripts and timed moments from video ads, was producing impossible timestamps—one video marked at 3:37 for a 2:22 clip—despite the model correctly reporting duration_seconds in the same JSON response. Prompt engineering reduced but did not eliminate the errors, so the developer implemented a binary parser to read the MP4's mvhd atom directly, guaranteeing a correct duration ceiling. The fix ensures timestamps are always within the video's actual length, preventing a single checkable wrong number from undermining the entire analysis.", "body_md": "New here (hey 👋) and wanted to share something. We recently made public an ad intel app we'd been using internally in our agency. The app/system runs video ads through Gemini to pull out a transcript, the hook, and a few timed moments. It works fairly well - especially for ideation. The transcription is genuinely accurate.\n\nA user opened a 2:22 video and saw a moment marked at **3:37**.\n\nNot a rounding error. Fifty percent past the end of a video the user was looking at.\n\nThe obvious guess is that the model didn't know how long the video was. That would be a reasonable bug and an easy fix.\n\nBut in the *same JSON response*, Gemini reported `duration_seconds`\n\ncorrectly.\n\nWe measured it properly on one ad, four identical runs, same file, same prompt:\n\n`duration_seconds`\n\nreported by the model: `t=214`\n\nThe words were right. The clock was 50% long. The model was holding the correct duration and writing impossible timestamps anyway.\n\nThat is not a knowledge problem. It's a consistency problem, and those don't respond to being asked nicely.\n\nWe tried the things you'd try. State the duration in the prompt. Tell it no timestamp may exceed that. Restate the constraint after the schema. Put it in caps.\n\nFailure rate went down. It didn't go to zero. A hardened run still drifted to `t=218`\n\non a 142-second video.\n\nThis is where I'd argue the general lesson lives:\n\nIf a model can violate a constraint, prompting reduces how often it does. Only code makes it impossible.\n\nFor most output that trade is fine, because most output is hard to falsify. Nobody checks whether a \"psychological driver\" is really the third most important one.\n\nTimestamps are different. **A timestamp is falsifiable in one click.** The user scrubs to 3:37 on a 2:22 video and the whole analysis loses credibility, including all the parts that were correct. One checkable wrong number poisons everything unverifiable around it.\n\nSo we needed a ground truth the model couldn't wander away from.\n\nThe video is already in memory as a `Buffer`\n\nbefore it goes anywhere near the model. MP4 files carry their own duration in the `mvhd`\n\natom (movie header). No API call, no dependency, and `ffprobe`\n\nisn't installed on our host anyway.\n\nThe layout after the 4-byte `mvhd`\n\ntype is: version (1 byte), flags (3), then for version 0 — created (4), modified (4), timescale (4), duration (4). For version 1, the timestamps are 8 bytes and duration is 8.\n\nDuration in seconds is `duration / timescale`\n\n.\n\n``` js\nfunction mp4DurationSeconds(buf: Buffer): number | null {\n  const limit = buf.length - 24;\n  for (let i = 0; i < limit; i++) {\n    // scan for the ASCII bytes 'm','v','h','d'\n    if (buf[i] !== 0x6d || buf[i+1] !== 0x76 || buf[i+2] !== 0x68 || buf[i+3] !== 0x64) continue;\n\n    const version = buf[i + 4];\n    try {\n      const base = i + 8; // past 'mvhd' + version(1) + flags(3)\n      if (version === 1) {\n        const timescale = buf.readUInt32BE(base + 16);\n        const duration  = Number(buf.readBigUInt64BE(base + 20));\n        if (timescale > 0 && duration > 0) return duration / timescale;\n      } else {\n        const timescale = buf.readUInt32BE(base + 8);\n        const duration  = buf.readUInt32BE(base + 12);\n        if (timescale > 0 && duration > 0) return duration / timescale;\n      }\n    } catch {\n      // malformed atom, keep scanning for a valid one\n    }\n  }\n  return null;\n}\n```\n\nTwo deliberate choices worth explaining.\n\n**We scan for the atom rather than walking the box tree.** Walking is more correct in principle. Scanning handles both faststart files (`moov`\n\nat the front) and files with `moov`\n\nat the end, without implementing a parser for a container format we only need one number out of.\n\n**A malformed atom keeps the loop going instead of throwing.** The four bytes `mvhd`\n\ncan appear inside compressed video data by coincidence. If you bail on the first match that doesn't parse, a random byte sequence takes down your duration check.\n\nWith a real ceiling, every timestamp gets validated server-side before it reaches the client. Same doctrine as validating any untrusted input, because that's what model output is.\n\nThe interesting decisions were about what to do with a bad value, not how to spot one.\n\n**Keep the words, drop the clock.** The transcription is accurate. Only the timing is untrustworthy. So an impossible timestamp is deleted and the text survives. Our transcript component already renders a blank gutter for a missing `t`\n\n, so a stripped timestamp degrades into a clean script instead of a wrong number.\n\n**Null both ends of a pair, not just the bad one.** We track \"curiosity loops\" with an open and a close time. If the close is impossible, nulling only that one leaves a surviving open time next to a blank close, which renders as precision we don't have. Both go.\n\nThis one cost me an afternoon and it's pure JavaScript, nothing to do with AI.\n\nThe first version coerced timestamps with `Number(x)`\n\n. Looks harmless:\n\n``` js\n// WRONG\nconst t = Number(chunk.t);\n```\n\n`Number(null)`\n\nis `0`\n\n.\n\nA model that *honestly* returns `null`\n\nfor \"I don't know when this happened\" gets that turned into `0`\n\n, which renders as **0:00**, which then reads as a loop closing before it opens, which trips the validation that was supposed to be protecting us. An honest null became a confident lie in one implicit coercion.\n\n``` js\nconst asSeconds = (x: any): number | null => {\n  if (x === null || x === undefined || (typeof x === 'string' && !x.trim())) return null;\n  const n = Number(x);\n  return Number.isFinite(n) ? n : null;\n};\n```\n\nCaught by a unit test, thankfully, and not by a user. If you're sanitising LLM output, `null`\n\nand `0`\n\nmean completely different things and JavaScript will happily merge them for you.\n\nThe last piece is the one I'd have got wrong without looking at the actual failures.\n\nThe observed mode is a **drifting tail**. The model transcribes accurately the whole way through, but its clock stretches as it goes, so the late chunks are impossible while every earlier timestamp is fine. Those early ones are good data and worth keeping.\n\nScattered bad timestamps through the body are a different signal entirely. That means the timing pass itself is unreliable, and presenting a mix of right and wrong as though it were precise is worse than presenting nothing.\n\nSo: a contiguous run of bad timestamps ending at the final chunk is treated as tail drift and trimmed. Anything else strips all timings.\n\nMy first instinct was a percentage threshold, something like \"if more than 25% are impossible, drop everything\". That's wrong, and the reason is worth internalising:\n\n**The same failure would be judged differently depending on how many chunks the model happened to emit.** A five-chunk drift is 19% of a 26-chunk transcript and 28% of an 18-chunk one. Identical failure, opposite verdicts, decided by something you don't control. Threshold on the *shape* of the failure, not its proportion.\n\n**A model can hold a fact and violate it in the same response.** Reporting the duration correctly told us nothing about whether it would respect it.\n\n**Prompting sets a tendency. Code sets a guarantee.** For anything a user can check in one click, you need the guarantee.\n\n**Find the ground truth in the artefact you already have.** The buffer was in memory the whole time. We were asking a language model for a number that was sitting in a binary header 20 lines of code away.\n\n**Partial output usually beats no output, if you're honest about which part you dropped.** The words were always right. Only the clock was wrong. Shipping an accurate transcript with no timestamps is better than shipping either a wrong timestamp or nothing at all.\n\n*I build SOCIALFUEL, which pulls the live ads any brand is running and decodes what makes them work. The video pipeline behind it is where this bug lived. If you're doing timed analysis on video with an LLM, I'd check your timestamps against the container before you trust them.*", "url": "https://wpnews.pro/news/prompt-engineering-couldn-t-fix-this-llm-bug-20-lines-of-binary-parsing-did", "canonical_source": "https://dev.to/socialfuel/prompt-engineering-couldnt-fix-this-llm-bug-20-lines-of-binary-parsing-did-3n37", "published_at": "2026-08-05 02:23:37+00:00", "updated_at": "2026-08-05 02:41:36.580247+00:00", "lang": "en", "topics": ["large-language-models", "generative-ai", "developer-tools"], "entities": ["Gemini", "MP4", "mvhd"], "alternates": {"html": "https://wpnews.pro/news/prompt-engineering-couldn-t-fix-this-llm-bug-20-lines-of-binary-parsing-did", "markdown": "https://wpnews.pro/news/prompt-engineering-couldn-t-fix-this-llm-bug-20-lines-of-binary-parsing-did.md", "text": "https://wpnews.pro/news/prompt-engineering-couldn-t-fix-this-llm-bug-20-lines-of-binary-parsing-did.txt", "jsonld": "https://wpnews.pro/news/prompt-engineering-couldn-t-fix-this-llm-bug-20-lines-of-binary-parsing-did.jsonld"}}