{"slug": "give-claude-eyes-watch-any-video-frame-by-frame-100-local-buildwith-conrad", "title": "Give Claude Eyes — watch any video frame-by-frame, 100% local (buildwith.conrad)", "summary": "A developer created a Claude Code skill called 'claude-watch' that enables Claude to analyze videos frame-by-frame locally. The skill splits videos into images and text with aligned timestamps, using free tools yt-dlp, FFmpeg, and whisper.cpp. It samples the first 15 seconds densely at 15 fps and the rest sparsely at 1 frame per 3.5 seconds to capture visual details missed by transcript-only analysis.", "body_md": "Freebie for the **EYES** keyword (reel-47, \"Give Claude eyes\"). Deliver as a public GitHub Gist —\nnumbered steps the user can run today, not a raw link. Value-first: by the end you have a Claude Code\nskill that *sees* a video (every cut, every on-screen detail), not just reads its transcript.\n\nClaude has no native video model. So every \"analyze this video\" tool just pulls the **transcript** — and\nmisses everything that's only on screen: the cuts, the text overlays, the visual hook, the b-roll. Half the\nmeaning of a good video lives in the pixels, not the words.\n\nThis skill fixes that. It splits any video into the two things Claude *does* understand — **images + text** —\nand hands both over with the timestamps lined up. Claude flips through the frames like a flip-book while it\nreads the script, so it knows exactly what's on screen the moment something is said. All local. Zero API cost.\n\nThree free command-line tools:\n\n**yt-dlp**— pulls the video from almost anywhere (YouTube, Instagram, Loom, TikTok, a direct URL).** FFmpeg**— rips frames + a clean audio track out of the video.** whisper.cpp**— transcribes the audio locally (free), if the source has no captions.\n\n```\n# macOS (Homebrew)\nbrew install yt-dlp ffmpeg whisper-cpp\n\n# grab a small, fast local transcription model (~150 MB, one time)\nmkdir -p ~/.claude/models\ncurl -L -o ~/.claude/models/ggml-base.en.bin \\\n  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin\n```\n\n(Windows/Linux: install `yt-dlp`\n\n, `ffmpeg`\n\n, and `whisper.cpp`\n\nfrom their repos — same three tools.)\n\nSave the file below as `~/.claude/skills/claude-watch/SKILL.md`\n\n, then restart Claude Code. That's it —\nClaude now has a `claude-watch`\n\nskill it will use whenever you hand it a video URL.\n\nIn Claude Code, just say:\n\n```\n/claude-watch https://www.instagram.com/reel/XXXXXXXXX/\n```\n\nor *\"watch this video and break down why it works: \"*. Claude downloads it, samples the frames,\ntranscribes the audio, and gives you a scene-by-scene breakdown — hook, cuts, on-screen text, the payoff.\n\nThe trick that makes it accurate:sample thefirst ~15 seconds at 15 fps(the hook carries the most motion and decides retention), and the rest at 1 frame / 3–4s. One frame every few seconds only catches end-states and misses the word-by-word reveals, the fast cuts, and the count-up animations. Dense on the hook, sparse on the body.\n\n```\n---\nname: claude-watch\ndescription: Use when the user gives a video URL (YouTube, Instagram, Loom, TikTok, direct link) and wants Claude to actually SEE the video — a scene-by-scene / frame-by-frame breakdown, not just the transcript. Triggers on \"/claude-watch <url>\", \"watch this video\", \"break this reel down\".\n---\n\n# claude-watch — see the video, not just the transcript\n\nClaude has no native video model, so this skill splits a video into images + text and reads both with\ntimestamps aligned. Run every step; never skip the dense-hook sampling.\n\n## Step 1 — Download the video\n``` bash\nmkdir -p /tmp/claude-watch && cd /tmp/claude-watch\nyt-dlp -o video.mp4 -f \"mp4\" \"<URL>\"\nffprobe -v error -show_entries format=duration -of csv=p=0 video.mp4   # duration (seconds)\n```\n\n## Step 2 — Two-stage frame sampling (the important part)\nThe hook holds the most motion and decides retention — sample it densely; sample the body sparsely.\n``` bash\n# HOOK: first 15s at 15 fps → tiled contact sheets (5 cols x 9 rows = 45 frames = 3s per sheet)\nmkdir -p hook && ffmpeg -y -t 15 -i video.mp4 \\\n  -vf \"fps=15,scale=200:-1,tile=5x9:margin=6:padding=4:color=white\" -an hook/sheet_%02d.jpg\n# BODY: 1 frame / 3.5s across the whole video\nmkdir -p body && ffmpeg -y -i video.mp4 -vf \"fps=1/3.5,scale=360:-1\" body/f_%02d.jpg\n```\nRead the sheets in order; frame N in the hook is at t = N / 15 seconds (sheet k, cell i → frame (k-1)*45 + i).\n\n## Step 3 — Transcribe the audio (free, local)\n``` bash\nffmpeg -y -i video.mp4 -ar 16000 -ac 1 audio.wav\nwhisper-cli -m ~/.claude/models/ggml-base.en.bin -f audio.wav -otxt -of transcript\n```\nIf YouTube has captions, use those (free, exact). Always sanity-check the transcript against the frames —\nlocal models mishear names (they'll hear \"Claude\" as \"plot\").\n\n## Step 4 — Read frames + transcript together, then output\nLook at the hook sheets and body frames, line them up with the transcript timestamps, and produce:\n1. **Transcript** (corrected against what's on screen).\n2. **Scene-by-scene breakdown** — for each chapter: time, what's on screen, the on-screen text, the cut.\n3. **Why it works** — 2–3 concrete mechanisms (hook structure, cut cadence, the one visual payoff, the CTA).\n4. **Steal-the-structure notes** — what to lift into your own video (change the content, keep the structure).\n\nClean up `/tmp/claude-watch` when done.\n```\n\n**Want Claude to run workflows like this across your whole business** — remembering your projects, your\nbrand, and your taste every session instead of starting cold? That's what my\n\n**get. kits** set up (the CLAUDE.md + skills + memory system). But this skill alone will already change how you study every video you save. 🚀\n\n*— Conrad · @buildwith.conrad*", "url": "https://wpnews.pro/news/give-claude-eyes-watch-any-video-frame-by-frame-100-local-buildwith-conrad", "canonical_source": "https://gist.github.com/conradcaffier03/ef914685fc9d7da91b6591947fa1ddb2", "published_at": "2026-07-09 19:59:59+00:00", "updated_at": "2026-07-10 08:11:47.472113+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "developer-tools", "computer-vision"], "entities": ["Claude Code", "yt-dlp", "FFmpeg", "whisper.cpp", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/give-claude-eyes-watch-any-video-frame-by-frame-100-local-buildwith-conrad", "markdown": "https://wpnews.pro/news/give-claude-eyes-watch-any-video-frame-by-frame-100-local-buildwith-conrad.md", "text": "https://wpnews.pro/news/give-claude-eyes-watch-any-video-frame-by-frame-100-local-buildwith-conrad.txt", "jsonld": "https://wpnews.pro/news/give-claude-eyes-watch-any-video-frame-by-frame-100-local-buildwith-conrad.jsonld"}}