How to test a video your AI agent rendered: From seek bugs to the final MP4 A developer published a three-part method for validating video rendered by an AI agent: making every frame a pure function of time so seek and scrub operations are deterministic, gating each export against an explicit delivery spec with ffprobe, and wiring both checks into the agent's instructions and CI so a bad render cannot be reported as done. In a demonstration, a scene that eased a marker toward its target using state carried between calls rendered the marker at 11.2% on a fresh page at second 4 but at roughly 62.5% after playing the full 10 seconds and seeking back, while a pure-function rewrite passed a Node seek test that sweeps backwards and forwards across fixed timestamps and compares each result against a fresh scene. An AI agent can edit a video project from one prompt and still hand back a bad result. A scene can look right when it plays from zero and wrong when someone scrubs back. An MP4 can export at the wrong size, with no audio track, or a second short. Both are cheap to catch before a person watches anything. The fix has three parts. Make every frame a pure function of time and test the render path, check each export against an explicit delivery spec with ffprobe, and run both checks through the agent's instructions and CI so a bad render cannot be reported as done. Below is the code for each. The Node seek test, the Python gate and the CI script were run on generated clips and stub scenes. The render step uses CoAnimator's coa CLI docs version 0.7.2 . The seek rules and the gate work with any renderer that writes an MP4. Seeking is the first thing I test, because everything else builds on it. A preview plays from zero. An exporter that starts at second 3, or a reviewer who drags the playhead backwards, lands on a state the animation never visited if the scene keeps history between calls. Smoothing is one common way that happens. This scene eases the marker toward its target instead of jumping there. js export function createScene { let x = 6; // lives between calls return function render t, els { const target = 6 + t 6.5; x += target - x 0.2; // eases toward the target, so it depends on the last call els.marker.style.left = ${x}% ; }; } On a fresh page, rendering second 4 puts the marker at 11.2%. After playing the full 10 seconds at 30 fps and then seeking to second 4, it sits at about 62.5%. Playback looks smooth, while the exporter or a scrubbing reviewer gets a different frame. The fix is to compute everything from t . js const clamp = n, lo, hi = Math.max lo, Math.min hi, n ; export function frameAt t { t = clamp t, 0, 10 ; const p = clamp t / 2, 0, 1 ; return { x: 6 + t 6.5, opacity: 1 - Math.pow 1 - p, 3 , caption: t < 3 ? 'Import' : t < 7 ? 'Map columns' : 'Review', }; } export const createScene = = t, els = { const f = frameAt t ; els.marker.style.left = ${f.x}% ; els.marker.style.opacity = String f.opacity ; els.caption.textContent = f.caption; }; The scene-state seek test exercises the render path, not just the math. It plays a scene to the end on one stage, then sweeps backwards and forwards through fixed timestamps on that same stage, and compares each result with a fresh scene rendered at the same time. The stage is a plain object, so the test runs in Node without a browser. It checks the values the scene writes, which are marker position, opacity and caption. js const { createScene } = await import process.argv 2 ; const stage = = { marker: { style: {} }, caption: {} } ; const snap = s = JSON.stringify s.marker.style.left, s.marker.style.opacity, s.caption.textContent ; const freshAt = t = { const s = stage ; createScene t, s ; return snap s ; }; // Play to the end on one stage, then scrub backwards and forwards on that same stage const played = stage ; const render = createScene ; for let i = 0; i <= 300; i++ render i / 30, played ; const times = ; for let t = 10; t = 0; t -= 0.25 times.push t ; // backwards sweep for let t = 0; t <= 10; t += 0.7 times.push t ; // forwards sweep, off the grid for const t of times { render t, played ; if snap played == freshAt t { console.error mismatch at ${t.toFixed 2 }s\n scrubbed ${snap played }\n fresh ${freshAt t } ; process.exit 1 ; } } console.log seek ok, ${times.length} timestamps ; Saved as scene-smoothed.js and scene-pure.js , the two scenes give these results. The sweep is fixed, so the failure repeats on every run. The smoothed scene only sets left , so the other two values print as null. bash $ node seek.test.mjs ./scene-smoothed.js mismatch at 10.00s scrubbed "70.30666666666669%",null,null fresh "19%",null,null $ node seek.test.mjs ./scene-pure.js seek ok, 56 timestamps A headless browser that compares screenshots at the same timestamps would also cover pixels and CSS. This test is the quick layer that runs without one. It assumes the scene logic sits in a module that the page and the test both import. If your renderer needs a single HTML file, keep the module as the source and inline it at build time. Three things live outside the scene code, so check for them in review. t . The same rule applies on page load, where a skipped render shows up as a flash of the wrong frame. An export worker can open the animation at a later time through a ?t= parameter, and the page has to paint that frame before anything is visible. Here the stage starts hidden with stage { visibility: hidden; } in the page's CSS, the requested frame is drawn immediately, and the stage is revealed once fonts and images are ready. js import { createScene } from './scene.js'; const duration = 10; const stage = document.querySelector ' stage' ; const els = { marker: document.querySelector ' marker' , caption: document.querySelector ' caption' , }; const scene = createScene ; const startAt = Math.min Math.max Number new URLSearchParams location.search .get 't' || 0, 0 , duration ; scene startAt, els ; await Promise.all document.fonts.load '600 32px "Inter"' , // one entry per font the scenes use ... ...document.images .map img = img.decode .catch = {} , ; scene startAt, els ; stage.style.visibility = 'visible'; I use document.fonts.load here instead of document.fonts.ready . Per MDN, ready resolves once the document has finished loading the fonts it needs. Fonts load when text that uses them is rendered, so a font used only by a hidden scene may not have been requested yet. Replace the example font with each font your scenes use. CoAnimator animations also expose window. kcControl with seek , seekAndPause , play , pause , getTime and getDuration . I route seek and seekAndPause into the same render function, so preview and export share one code path. The full player contract is in the timeline reference that ships with the app. With seeking covered, I render only the scene that changed after each edit. coa render ./release-demo --timeline main \ --start 3 --end 7 --fps 30 --out ./import-scene-review.mp4 The end is exclusive, so 3 to 7 at 30 fps is 120 frames. coa exits with 0 on success and 1 on failure. It also checks dimensions, frame count, fps, duration and expected audio before it replaces the destination, and a failed run leaves any existing file in place. An old good MP4 can therefore remain at the path after a failed render, so I write every render to a unique name and let a nonzero exit stop the script. That check uses the tool's own rules. A second one should use mine. The gate takes the delivery spec as arguments, so it fails when a file is valid for the renderer and wrong for the brief. It checks size, frame rate, decoded frame count, duration and, on request, an audio stream. bash /usr/bin/env python3 """Fail the build when a rendered MP4 does not match the spec in the arguments.""" import argparse, json, subprocess, sys from fractions import Fraction p = argparse.ArgumentParser p.add argument "file" p.add argument "--width", type=int, required=True p.add argument "--height", type=int, required=True p.add argument "--fps", type=float, required=True p.add argument "--duration", type=float, required=True p.add argument "--tolerance", type=float, default=0.1, help="seconds" p.add argument "--frame-tolerance", type=int, default=1, help="frames" p.add argument "--no-count", action="store true", help="skip decoding, skip the frame check" p.add argument "--require-audio", action="store true" a = p.parse args def number value : """Parse '30000/1001' or '12.5'. Return None for 0/0, N/A or missing.""" try: return float Fraction value except TypeError, ValueError, ZeroDivisionError : return None cmd = "ffprobe", "-v", "error" + if a.no count else "-count frames" + "-show streams", "-show format", "-of", "json", a.file r = subprocess.run cmd, capture output=True, text=True if r.returncode = 0: sys.exit f"FAIL ffprobe cannot read {a.file}: {r.stderr.strip }" info = json.loads r.stdout streams = info.get "streams", video = next s for s in streams if s.get "codec type" == "video" and not s.get "disposition", {} .get "attached pic" , None audio = next s for s in streams if s.get "codec type" == "audio" , None problems = if video is None: problems.append "no video stream" else: size = video.get "width" , video.get "height" if size = a.width, a.height : problems.append f"size {size 0 }x{size 1 }, expected {a.width}x{a.height}" fps = number video.get "avg frame rate" if fps is None: problems.append "frame rate unavailable" elif abs fps - a.fps 0.01: problems.append f"fps {fps:g}, expected {a.fps:g}" if not a.no count: frames = video.get "nb read frames", "" expected = round a.duration a.fps if not frames.isdigit : problems.append "frame count unavailable" elif abs int frames - expected a.frame tolerance: problems.append f"{frames} frames, expected {expected}" duration = number info.get "format", {} .get "duration" if duration is None: problems.append "duration unavailable" elif abs duration - a.duration a.tolerance: problems.append f"duration {duration:.2f}s, expected {a.duration:g}s" if a.require audio and audio is None: problems.append "no audio stream" if problems: print "FAIL " + a.file for line in problems: print " - " + line sys.exit 1 print f"OK {a.file}" bash $ python3 check export.py short.mp4 --width 1280 --height 720 --fps 30 --duration 4 --require-audio FAIL short.mp4 - 90 frames, expected 120 - duration 3.00s, expected 4s $ python3 check export.py good.mp4 --width 1280 --height 720 --fps 30 --duration 4 --require-audio OK good.mp4 Real files are rarely exact, so two settings soften the checks. Frame counts get a one frame tolerance by default. A 29.97 fps file with 600 frames lasts about 20.02 seconds. Against a 20 second spec the expected count rounds to 599, so the default tolerance passes it and --frame-tolerance 0 fails it. The --no-count flag skips the full decode that -count frames needs, which suits long films, and it drops the frame check with it. The frame rate check reads ffprobe's average, so it assumes a constant frame rate. A file with no audio stream fails with no audio stream . A file with a silent audio track still passes, because the gate checks that a stream exists, not what it contains. A file ffprobe cannot read, such as a text file renamed to .mp4 or a truncated export, exits with 1 and ffprobe's error. The gate does not check the container, so a valid WebM passes. Add a check on format name if the brief requires MP4. A check only helps if the agent runs it. The instruction file the agent reads, such as AGENTS.md or CLAUDE.md , is the place for that. Rendering - Write each render to renders/ with a timestamp in the name. Never overwrite an earlier file. - After every render, run python3 check export.py