Turning a 5-minute talk into 7 images for an LLM: picking one sharp frame per scene

A developer built a scene-aware keyframe extractor that reduces a 4:44 conference talk from 28 fixed-interval frames to 7 sharp frames, one per slide with all bullets visible, before sending video to a vision LLM. The pipeline decodes once at 320 px grayscale, detects cuts using a hard-jump threshold plus a median-baseline test that tolerates camera motion, scores frames by sharpness divided by local motion, and dedupes by comparing 64×36 thumbnails rather than 64-bit difference hashes, which had merged distinct slides. Batching frame selection into a single ffmpeg pass with a select filter was 5.7× faster than one process per frame, and timestamp-based selection handled an Ogg file that decoded 1,113 frames where 1,254 were expected.

If you send a video to a vision LLM, the usual first step is to sample a frame every few seconds. On a 4:44 conference talk, one frame every 10 seconds gives you 28 images. The talk has 7 slides. You pay for 21 repeats, and some of the 28 catch a slide halfway through a transition or before its last bullet appears. What you actually want is one sharp frame per scene: per shot in a film, per slide in a talk. Here is how I built that, with numbers from real videos. Decode the video once, at 320 px wide in grayscale, and keep 0.5 to 4 samples per second depending on length. For each sample, compute three numbers: This pass is the only full decode of the video, so it sets the cost. Two ffmpeg flags help a lot on H.264: -skip frame noref -skip loop filter all made the scan 1.85× faster, and since we only need a few samples per second at 320 px, nothing is lost. VP9 and Theora ignore them. A fixed threshold fails in both directions. Hard cuts in a film produce big jumps. Slide changes in a filmed talk jump by as little as 0.017, because the slide only fills part of the frame next to the speaker's camera. Camera motion, meanwhile, produces a steady stream of moderate differences for seconds at a time, so a low fixed threshold would cut on every pan. So a cut is either a hard jump, or a jump that is large relative to the recent baseline: python def find cuts diff, fps, hard=0.15, floor=0.025, min scene=1.0 : """Indices where a new scene starts: hard cuts, or jumps far above the recent baseline.""" win = max 3, int round fps 2 min len = max 1, int round min scene fps cuts = 0 for i in range 1, len diff : if i - cuts -1 < min len: continue local = float np.median diff max 0, i - win :i if i 1 else 0.0 if diff i = hard or diff i = floor and diff i = 4.0 local + 0.004 : cuts.append i return cuts On a static slide deck the baseline is about 0, so a 0.017 jump stands out. During a pan the baseline is high, so motion doesn't trigger cuts. Inside each scene, skip the first and last 15% transitions live there , skip flat frames, and score the rest by sharpness divided by local motion: score = sharpness / 1 + 20 × motion For slides this naturally picks the fully built slide, because more text means more edges. A pick is blurry if its sharpness is under 25% of the video's 75th-percentile sharpness. That catches motion blur and focus pulls without penalizing soft-looking films. Duplicates were the interesting bug. I started with a 64-bit difference hash a 9×8 thumbnail , the usual choice. On the talk it merged different slides: at 9×8, every slide is "white box on the left, dark webcam on the right." What works is comparing the 64×36 thumbnails directly and calling two frames duplicates only when under 1% of pixels changed noticeably. Different slides pass, and a shot the video cuts back to is kept once. Starting one ffmpeg process per frame cost about a second per frame, mostly process startup plus decoding forward from the previous keyframe. One decoding pass with a select filter was 5.7× faster on a 70-second clip 21 frames in 3.7 s instead of 21 s . My first version selected by frame number. It broke on an Ogg file that decoded 1,113 frames where 1,254 were expected, so the last targets never existed. Selecting by timestamp, the first frame at or after each target time, works on variable frame rates and dropped frames: select='gte t\,9.8 lt prev pts TB\,9.8 +gte t\,16.8 lt prev pts TB\,16.8 +...' For very long videos, seeking to each frame is still cheaper than decoding everything again, so the code picks whichever is faster using the measured scan time. Fixed sampling uses ffmpeg's fps=1/10 : | Video | Every 10 s | Smart keyframes | |---|---|---| | 4:44 talk with slides high sensitivity | 28 frames, 7 distinct slides | 7 frames, one per slide, all bullets visible | | 12:14 film, 1080p Tears of Steel | 73 frames | 159 scenes, 24 blurry picks dropped, best 100 kept | | 32 s trailer Big Buck Bunny | 3 frames | 16 frames, one per shot | I packaged this as Smart Video Keyframes https://apify.com/nookforgestudio/video-keyframe-extractor on Apify. Pass video file URLs and get keyframes with timestamps, a contact sheet and, optionally, the on-screen text of each frame. Two ready-made examples: Videos via Wikimedia Commons: "Research and Privacy on Wikipedia" lightning talk by Eli Asikin-Garmager CC0 , used for the cover; Tears of Steel and Big Buck Bunny by the Blender Foundation CC BY 3.0 .