{"slug": "turning-a-5-minute-talk-into-7-images-for-an-llm-picking-one-sharp-frame-per", "title": "Turning a 5-minute talk into 7 images for an LLM: picking one sharp frame per scene", "summary": "A developer built a scene-aware keyframe extractor that reduces a 4:44 conference talk from 28 fixed-interval frames to 7 sharp frames, one per slide with all bullets visible, before sending video to a vision LLM. The pipeline decodes once at 320 px grayscale, detects cuts using a hard-jump threshold plus a median-baseline test that tolerates camera motion, scores frames by sharpness divided by local motion, and dedupes by comparing 64×36 thumbnails rather than 64-bit difference hashes, which had merged distinct slides. Batching frame selection into a single ffmpeg pass with a select filter was 5.7× faster than one process per frame, and timestamp-based selection handled an Ogg file that decoded 1,113 frames where 1,254 were expected.", "body_md": "If you send a video to a vision LLM, the usual first step is to sample a frame every few seconds. On a 4:44 conference talk, one frame every 10 seconds gives you 28 images. The talk has 7 slides. You pay for 21 repeats, and some of the 28 catch a slide halfway through a transition or before its last bullet appears.\n\nWhat you actually want is one sharp frame per scene: per shot in a film, per slide in a talk. Here is how I built that, with numbers from real videos.\n\nDecode the video once, at 320 px wide in grayscale, and keep 0.5 to 4 samples per second depending on length. For each sample, compute three numbers:\n\nThis pass is the only full decode of the video, so it sets the cost. Two ffmpeg flags help a lot on H.264: `-skip_frame noref -skip_loop_filter all` made the scan 1.85× faster, and since we only need a few samples per second at 320 px, nothing is lost. (VP9 and Theora ignore them.)\n\nA fixed threshold fails in both directions. Hard cuts in a film produce big jumps. Slide changes in a filmed talk jump by as little as 0.017, because the slide only fills part of the frame next to the speaker's camera. Camera motion, meanwhile, produces a steady stream of moderate differences for seconds at a time, so a low fixed threshold would cut on every pan.\n\nSo a cut is either a hard jump, or a jump that is large relative to the recent baseline:\n\n``` python\ndef find_cuts(diff, fps, hard=0.15, floor=0.025, min_scene=1.0):\n    \"\"\"Indices where a new scene starts: hard cuts, or jumps far above the recent baseline.\"\"\"\n    win = max(3, int(round(fps * 2)))\n    min_len = max(1, int(round(min_scene * fps)))\n    cuts = [0]\n    for i in range(1, len(diff)):\n        if i - cuts[-1] < min_len:\n            continue\n        local = float(np.median(diff[max(0, i - win):i])) if i > 1 else 0.0\n        if diff[i] >= hard or (diff[i] >= floor and diff[i] >= 4.0 * local + 0.004):\n            cuts.append(i)\n    return cuts\n```\n\nOn a static slide deck the baseline is about 0, so a 0.017 jump stands out. During a pan the baseline is high, so motion doesn't trigger cuts.\n\nInside each scene, skip the first and last 15% (transitions live there), skip flat frames, and score the rest by sharpness divided by local motion:\n\n`score = sharpness / (1 + 20 × motion)`\n\nFor slides this naturally picks the fully built slide, because more text means more edges.\n\nA pick is blurry if its sharpness is under 25% of the video's 75th-percentile sharpness. That catches motion blur and focus pulls without penalizing soft-looking films.\n\nDuplicates were the interesting bug. I started with a 64-bit difference hash (a 9×8 thumbnail), the usual choice. On the talk it merged different slides: at 9×8, every slide is \"white box on the left, dark webcam on the right.\" What works is comparing the 64×36 thumbnails directly and calling two frames duplicates only when under 1% of pixels changed noticeably. Different slides pass, and a shot the video cuts back to is kept once.\n\nStarting one ffmpeg process per frame cost about a second per frame, mostly process startup plus decoding forward from the previous keyframe. One decoding pass with a select filter was 5.7× faster on a 70-second clip (21 frames in 3.7 s instead of 21 s).\n\nMy first version selected by frame number. It broke on an Ogg file that decoded 1,113 frames where 1,254 were expected, so the last targets never existed. Selecting by timestamp, the first frame at or after each target time, works on variable frame rates and dropped frames:\n\n```\nselect='gte(t\\,9.8)*lt(prev_pts*TB\\,9.8)+gte(t\\,16.8)*lt(prev_pts*TB\\,16.8)+...'\n```\n\nFor very long videos, seeking to each frame is still cheaper than decoding everything again, so the code picks whichever is faster using the measured scan time.\n\nFixed sampling uses ffmpeg's `fps=1/10`:\n\n| Video | Every 10 s | Smart keyframes | \n|---|---|---|\n| 4:44 talk with slides (high sensitivity) | 28 frames, 7 distinct slides | **7 frames, one per slide, all bullets visible** | \n| 12:14 film, 1080p (Tears of Steel) | 73 frames | **159 scenes, 24 blurry picks dropped, best 100 kept** | \n| 32 s trailer (Big Buck Bunny) | 3 frames | **16 frames, one per shot** | \n\nI packaged this as [Smart Video Keyframes](https://apify.com/nookforgestudio/video-keyframe-extractor) on Apify. Pass video file URLs and get keyframes with timestamps, a contact sheet and, optionally, the on-screen text of each frame. Two ready-made examples:\n\n*Videos via Wikimedia Commons: \"Research and Privacy on Wikipedia\" lightning talk by Eli Asikin-Garmager (CC0), used for the cover; Tears of Steel and Big Buck Bunny by the Blender Foundation (CC BY 3.0).*", "url": "https://wpnews.pro/news/turning-a-5-minute-talk-into-7-images-for-an-llm-picking-one-sharp-frame-per", "canonical_source": "https://dev.to/nookforgestudio/turning-a-5-minute-talk-into-7-images-for-an-llm-picking-one-sharp-frame-per-scene-3k43", "published_at": "2026-11-01 22:03:14+00:00", "updated_at": "2026-11-01 22:21:54.816471+00:00", "lang": "en", "topics": ["computer-vision", "ai-tools", "ai-products", "generative-ai"], "entities": ["ffmpeg"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/turning-a-5-minute-talk-into-7-images-for-an-llm-picking-one-sharp-frame-per", "markdown": "https://wpnews.pro/news/turning-a-5-minute-talk-into-7-images-for-an-llm-picking-one-sharp-frame-per.md", "text": "https://wpnews.pro/news/turning-a-5-minute-talk-into-7-images-for-an-llm-picking-one-sharp-frame-per.txt", "jsonld": "https://wpnews.pro/news/turning-a-5-minute-talk-into-7-images-for-an-llm-picking-one-sharp-frame-per.jsonld"}}