Show HN: Turn narrated screen recordings into data for AI agents (local, MIT)

A new open-source MCP server called talkthrough-mcp, released under the MIT license, lets developers turn narrated screen recordings into structured data for AI agents without any cloud dependency. The tool runs locally using ffmpeg, faster-whisper, and RapidOCR, providing timestamped transcripts, speaker labels, keyframes, and OCR text through lazy retrieval tools that prevent context overflow. Its wall-clock timestamp anchoring distinguishes it from existing screen-recorder SaaS and video-analyzer MCPs, enabling agents to directly correlate moments like 'the checkout hung' with server logs.

Quickstart quickstart · Tools tools · FAQ faq · Troubleshooting /korovin-aa97/talkthrough-mcp/blob/main/docs/TROUBLESHOOTING.md · Changelog /korovin-aa97/talkthrough-mcp/blob/main/CHANGELOG.md · Contributing /korovin-aa97/talkthrough-mcp/blob/main/CONTRIBUTING.md Feedback ingestion for AI agents. Record your screen and talk; your agent does the rest — files the bugs, writes the spec, builds the backlog. talkthrough-mcp is a local-first MCP server that turns a narrated screen recording or any video/audio file into agent-ready structured data: timestamped transcript segments, optional who-said-what speaker labels, scene-change keyframes, OCR'd on-screen text, and wall-clock anchoring. Everything is served through lazy retrieval tools, so a 30-minute recording never floods the model context — the agent pulls exactly the transcript slice, moment bundle, or frame it needs. There is no LLM inside the server and no cloud anywhere in the path: ffmpeg, faster-whisper, and RapidOCR run on your machine, and the calling agent brings the intelligence. What makes it different from screen-recorder SaaS and video-analyzer MCPs: it works on arbitrary local files, it ships the agent workflows server prompts + example agents , and it anchors every timestamp to wall-clock time — so "the moment I said the checkout hung" maps straight to the right window of your server logs. One command, no system dependencies: ffmpeg falls back to a bundled build, OCR is pip-only, and whisper models download themselves on first use. The only prerequisite is uv https://docs.astral.sh/uv/ brew install uv or curl -LsSf https://astral.sh/uv/install.sh | sh . Two install paths — pick one , not both the plugin already includes the server; installing both would register it twice : Server only — the 7 tools + 5 prompts, and nothing else on your system. Choose this for a minimal setup, or when you manage MCP servers yourself across several clients: claude mcp add -s user talkthrough -- uvx "talkthrough-mcp diarization " Full plugin — the same server, plus native slash commands /talkthrough:triage-recording , … that handle the ceremony for you, a ready-made triage subagent, and an agent skill that teaches Claude the workflow. Choose this for the best out-of-the-box experience: /plugin marketplace add korovin-aa97/talkthrough-mcp /plugin install talkthrough@talkthrough OpenAI Codex CLI ~/.codex/config.toml or project-scoped .codex/config.toml in trusted projects : mcp servers.talkthrough command = "uvx" args = "talkthrough-mcp diarization " More: integrations/codex/ OpenCode opencode.json project or ~/.config/opencode/opencode.json : { "mcp": { "talkthrough": { "type": "local", "command": "uvx", "talkthrough-mcp diarization " , "enabled": true } } } More: integrations/opencode/ Any other MCP stdio client uses the same server command: uvx "talkthrough-mcp diarization " . Per-engine folders with exactly these snippets plus verification steps live in integrations/ /korovin-aa97/talkthrough-mcp/blob/main/integrations ; agents can self-install via . /korovin-aa97/talkthrough-mcp/blob/main/llms-install.md llms-install.md Multi-person recordings meetings, interviews, panels can carry S1 / S2 /… speaker labels. Every install button, snippet, and the plugin above already ship the diarization engine, so asking your agent "who said what" just works — diarization itself still runs only when requested per call process media path=..., diarize=true, num speakers=<count if known , and its models download once on first use. Prefer the minimal server without the diarization engine? Use uvx talkthrough-mcp as the command instead the MCP registry entry also resolves to this lean form — an explicit diarize=true will then answer with the one-line install fix. Details in Speakers speakers-optional-diarization . git clone https://github.com/korovin-aa97/talkthrough-mcp claude mcp add talkthrough -- uv run --directory /path/to/talkthrough-mcp talkthrough-mcp Then, in your agent: Process ~/Desktop/recording.mov and triage it — or just invoke the triage-recording server prompt. | Tool | What it does | |---|---| process media path, recorded at?, vocabulary?, language?, model?, diarize?, num speakers?, force? | Ingest a video/audio file: local STT, keyframes, OCR, wall-clock, opt-in speaker labels. Returns a compact summary. Idempotent by content hash — re-calls are instant; diarize=true on a processed job adds speakers without re-transcribing. | get transcript job id, start ms?, end ms?, format? | Paginated transcript as segments , text , or srt speaker-prefixed when diarized, plus a roster header ; truncation returns next start ms . | get frames job id, at ms? | start ms?+end ms?, max frames?, include duplicates? | Keyframe images nearest a timestamp or evenly thinned across a range unique frames by default, max 6/call ; each frame names its absolute path . | get moment job id, start ms, end ms | The "one remark" bundle: transcript slice + up to 3 frames + their OCR text + wall-clock range + speakers in range when diarized . | search job id, query | Substring search over the transcript AND on-screen OCR text; hits carry t ms / t wall , frame refs, and the speaker when diarized. | extract frame job id, at ms, crop? | Exact-timestamp full-resolution re-extract from the source video optional crop when keyframes miss the instant; returns the file's absolute path . | list jobs | Recent processed recordings with durations, wall-clock starts, counts, and speaker counts when diarized. | Every tool description ships 10+ usage examples, so agents pick the right tool without extra prompting. | Prompt | Workflow | |---|---| triage-recording | Narrated screencast → precise findings JSON bug/feature/question routing, frame evidence | spec-from-workshop | Recorded workshop → structured spec with quoted decisions and open questions | backlog-from-demo | Product demo → prioritized backlog with timestamped evidence | meeting-actions | Meeting audio → action items, decisions, open questions | correlate-with-logs | Recording remarks ↔ system logs via wall-clock windows | The same prompts live as plain files in examples/prompts/ /korovin-aa97/talkthrough-mcp/blob/main/examples/prompts if your client doesn't surface MCP prompts. The findings contract used by triage-recording is . /korovin-aa97/talkthrough-mcp/blob/main/examples/output-contract.schema.json examples/output-contract.schema.json The same workflow ships as a cross-engine Agent Skill https://agentskills.io at .agents/skills/talkthrough/ /korovin-aa97/talkthrough-mcp/blob/main/.agents/skills/talkthrough — Claude Code, Codex CLI $talkthrough , Cursor, Copilot, Gemini CLI, Goose and other SKILL.md-compatible tools read it. Agents without MCP wiring can drive the CLI cli directly: talkthrough-mcp process recording.mov --json prints the same summary the MCP tool returns, and the job store is shared either way.Every timestamped result carries both t ms video-relative and t wall ISO 8601 real time once the recording start is known. Resolution ladder: recorded at parameter agent/user override → confidence exact - QuickTime com.apple.quicktime.creationdate tag, carries the local timezone QuickTime Player recordings; ⌘⇧5 wrote it before macOS 26 → high - Container creation time tag UTC → medium — macOS 26+ ⌘⇧5/ReplayKit screen recordings land here no creationdate tag anymore ; pass recorded at= when local-tz t wall matters - File mtime minus duration recorders finalize files at recording END → low - Nothing → tools still work with relative t ms only Why it matters: "the upload spinner froze here " becomes a ±30 s grep window in your server logs. With the diarization extra installed included in every generated config — see Quickstart who-said-what-speaker-diarization--included-in-the-configs-above , process media diarize=true labels who said what — locally, like everything else here sherpa-onnx https://github.com/k2-fsa/sherpa-onnx runtime, no torch, no accounts, no GPU : - Speakers become S1 , S2 , … in order of first appearance ; every transcript segment gets a speaker , and the tools surface it everywhere — roster with talk time in get transcript , speakers in range in get moment , speaker on search hits, S1: prefixes in the text/SRT formats, a speaker count in list jobs . Know the headcount? Pass Clustering to an exact k removes the main failure mode of unknown-count mode similar voices merging or one voice splitting . Agents are instructed to do this via the tool guidance; do the same in your own calls. num speakers . Already processed a recording? Calling process media diarize=true on it re-runs only diarization — whisper is not re-run, and labels appear in the existing job in seconds. Same for changing num speakers .- Labels are anonymous by design; mapping S1 → "Alice" is the calling agent's job self-introductions, vocatives, the attendees argument of the meeting-actions prompt . The server never guesses names. Models download once ~47 MB total from pinned, checksum-verified URLs into ~/.talkthrough/models/ ; warm runs are zero-network like the rest of the pipeline. Speed on an M-series CPU 4 threads : a 26-minute meeting diarizes in about 2 minutes RTF ≈ 0.08 , on top of the transcription time. Memory: expect on the order of 1–1.5 GB peak RSS while an hour-plus meeting is being diarized measured on a real 73-minute recording ; it is released when the stage completes. | Role | Model | Download | Weights license | |---|---|---|---| | Segmentation | | NeMo https://github.com/NVIDIA/NeMo en titanet small WeSpeaker https://github.com/wenet-e2e/wespeaker en voxceleb resnet34 LM 3D-Speaker https://github.com/modelscope/3D-Speaker campplus sv en voxceleb 16k The default won a real-meeting accept-eval RU/EN/ES + a 3-speaker 26-minute meeting : it was the only candidate to isolate all three real voices at num speakers=3 , at 2× the speed of the runner-up. Pick an alternate embedding model or point at your own .onnx file for offline machines via TALKTHROUGH DIARIZATION EMB MODEL ; tune the unknown-count sensitivity via TALKTHROUGH DIARIZATION THRESHOLD see docs/TROUBLESHOOTING.md /korovin-aa97/talkthrough-mcp/blob/main/docs/TROUBLESHOOTING.md . Honest quality notes live in Limitations limitations . Everything runs locally: your recordings never leave your machine, speech is transcribed by a local whisper model, OCR and speaker diarization are local ONNX inference, and there is no telemetry. The only network access is one-time tool/model downloads ffmpeg build, whisper model, OCR models, diarization models — the latter pinned by URL + sha256 . Diarization keeps no voiceprint database: voice embeddings live only in process memory, and only anonymous turn labels S1 / S2 land on disk. Narration in any of Whisper's ~99 languages works: the language is auto-detected per recording, and the summary reports both language and language probability so agents can tell a confident detection from a shaky one silence or music at the start can fool the detector — pin it with language="ru" and force=true when that happens . Speaker diarization is acoustic — it fingerprints voices, not words — so it is language-independent and works across all of those languages unchanged. Pick the model for your languages — per call model= parameter, agents do this themselves when a transcript comes back garbled or as the server default TALKTHROUGH WHISPER MODEL : | Model | Size | Best for | |---|---|---| small default | 464 MB | English and major-language narration on CPU | large-v3-turbo | ~1.5 GB | recommended for non-English — near-large quality at near-small speed | medium | ~1.5 GB | conservative alternative to turbo | tiny / base | 75–145 MB | quick drafts, CI | .en variants | — | English-only, slightly faster/better for EN | Tips that work in every language: pass product names via vocabulary="Term1, Term2" biases the decoder so jargon survives , and note that the workflow prompts instruct agents to write digests in the narrator's language while keeping quotes verbatim — the server never translates exact quotes are evidence; translation is the agent's job . On-screen text OCR defaults to RapidOCR's Latin + Chinese models. For other scripts set TALKTHROUGH OCR LANG to your language — ru / uk → the eslav pack , ja , ko , ar , hi , el , th , or any RapidOCR pack name like cyrillic — and reprocess with force=true ; the matching recognition model downloads once. Spoken-language support is unaffected either way. | Env var | Default | Meaning | |---|---|---| TALKTHROUGH WHISPER MODEL | small | default whisper model tiny / base / small / medium / large-v3 / large-v3-turbo ; the model tool param overrides per call | TALKTHROUGH OCR | on | set off to skip OCR | TALKTHROUGH OCR LANG | Latin+Chinese | recognition script for on-screen text: a language code ru , ja , ko , ar , hi , … or a RapidOCR pack name eslav , cyrillic , latin , … ; the model downloads once | TALKTHROUGH OCR PARAMS | — | advanced: JSON object of raw RapidOCR params merged over the derived ones, e.g. {"Rec.lang type": "cyrillic"} | TALKTHROUGH DIARIZE | off | set on to diarize by default needs the diarization extra; degrades with a warning without it ; an explicit diarize tool param always wins | TALKTHROUGH DIARIZATION THRESHOLD | 0.5 | clustering sensitivity when num speakers is unknown: fewer speakers than expected → lower it; more → raise it | TALKTHROUGH DIARIZATION SEG MODEL | pyannote-segmentation-3-0 | segmentation model: allowlist name or a path to a local .onnx offline preseed | TALKTHROUGH DIARIZATION EMB MODEL | nemo en titanet small | embedding model: allowlist name see .onnx path | TALKTHROUGH DIARIZATION THREADS min 4, cpus TALKTHROUGH MAX SECONDS 7200 TALKTHROUGH MAX FRAMES 600 duration/budget on long recordings TALKTHROUGH HOME ~/.talkthrough The pipeline is also a CLI — useful for pre-processing long recordings outside an agent session the store is content-addressed, so the agent then queries the same job instantly : talkthrough-mcp process ~/Videos/long-session.mov prints the summary talkthrough-mcp process demo.mov --json machine-readable talkthrough-mcp process sync.m4a --diarize --num-speakers 3 who said what talkthrough-mcp gc --keep-days 30 clean the job store talkthrough-mcp serve stdio MCP server default First run notes: missing system ffmpeg triggers a one-time static-ffmpeg download; the first transcription downloads the whisper model ~460 MB for small ; both are cached. After that, expect roughly 3× faster than real time on an Apple-Silicon CPU with the default model, OCR included a 2-minute clip processes in ~40 s — and instant re-runs on the same file. Progress streams as MCP progress notifications, and the CLI prints stage lines. More: docs/TROUBLESHOOTING.md /korovin-aa97/talkthrough-mcp/blob/main/docs/TROUBLESHOOTING.md . CI runs lint, the unit suite, a full CLI smoke, and a diarize smoke on windows-latest static-ffmpeg Windows build, whisper tiny transcription, OCR, the instant idempotent re-run, and a speaker-roster assert through the native sherpa-onnx stack . Notes: the per-job lock is POSIX fcntl and degrades to a no-op on Windows — fine for a single-user machine; quote paths with spaces uv run talkthrough-mcp process "C:\Videos\Screen Recording.mp4" . If something breaks, please open an issue. Video: .mov .mp4 .webm .mkv — audio-only: .m4a .mp3 .wav .ogg .flac transcript tools only; frame tools explain why they're unavailable . Local files only. Honest edges, so you can decide fast: Speaker labels are segment-level and opt-in. Diarization assigns each whisper segment ONE speaker by dominant overlap — a segment spanning a fast exchange gets the majority voice, sub-second interjections "yeah", "mhm" can be absorbed, and heavy crosstalk degrades clustering the segmentation model tracks at most 2 simultaneous voices . Quality is pyannote-3.x-generation. The comfort zone without hints is roughly 2–8 speakers; pass — it removes the worst failure mode at any size, and it is the way to go for large meetings 10+ . num speakers whenever the headcount is known Local files only. No URL/YouTube ingestion 5 https://github.com/korovin-aa97/talkthrough-mcp/issues/5 — download first. Keyframes + transcript, not motion analysis. A glitch between scene changes can be invisible in the frame set; extract frame re-checks any instant, but frame-by-frame motion reasoning is your multimodal model's job. STT quality tracks the model you pick. The default small favors speed; non-English narration wants model="large-v3-turbo" see Languages languages . OCR reads crisp UI text well; tiny or low-contrast print is best-effort. Wall-clock confidence depends on recorder metadata — worst case pass recorded at= see the ladder above . Windows caveats — POSIX lock degrades to a no-op; see the Windows section above. | talkthrough | cloud recorder SaaS | meeting notetakers | typical video-analyzer MCPs | | |---|---|---|---|---| | Runs fully locally | ✅ | ❌ | ❌ | varies | | Any local video/audio file | ✅ | browser/app captures | meetings only | ✅ | | Wall-clock anchoring log correlation | ✅ | ❌ | ❌ | ❌ | | Who-said-what speaker labels | ✅ local, opt-in | some | ✅ cloud | ❌ | | Ships agent workflows prompts, skill, findings contract | ✅ | ❌ | ❌ | ❌ | | OCR of on-screen text, searchable | ✅ | some | ❌ | rare | Why not just upload the video to a multimodal model e.g. Gemini ? For a short, non-sensitive clip — do that. The trade-offs appear with length and sensitivity: an hour of screen recording costs on the order of a million tokens per question , the file leaves your machine, and you still can't map a remark to 14:32:07 UTC to grep your server logs. talkthrough indexes once, locally, then answers any number of follow-ups from the index. Why not screenpipe? Different job. screenpipe is an always-on recorder of your machine going forward commercial license . It can't open the .mov a teammate or customer just sent you. talkthrough analyzes any file it's handed — the two compose fine. There are agent skills that "watch" videos. Why a server with an index? Watch-style skills push a budgeted frame dump into the context window and go sparse on long videos , often call cloud STT for the audio, and keep nothing. talkthrough builds a persistent local index — transcript + OCR, full-text searchable — retrieves exact frames lazily, anchors everything to wall-clock time, and answers the next question without reprocessing. I use Jam for bug reports — do I need this? Keep Jam for browser bugs: console+network captured at record time is great evidence. talkthrough covers what a browser extension can't — desktop apps, mobile screencasts, ops incidents, meetings, any file — with no account, and correlates with server-side logs via wall-clock time. Which agent model do I need to drive this? We ran a 150-run battery against v0.2.0 — 6 model configs Claude haiku/sonnet/opus, Codex gpt-5.5 at two reasoning efforts + gpt-5.4-mini × 10 verbatim-identical task prompts × 5 real recordings 30 s–73 min, RU/EN, 1–5 speakers , scored by a strict LLM judge plus mechanical evidence checks. Short version: point lookups and search "who said X and when" worked on every tier tested; minutes-with-owners and evidence-disciplined name mapping want the top tiers; reasoning effort moved results more than model family. Full matrices — score × time × tokens on every intersection — in docs/MODEL-NOTES.md /korovin-aa97/talkthrough-mcp/blob/main/docs/MODEL-NOTES.md . Can't I just script ffmpeg + whisper myself? Yes — that's exactly this pipeline. What you'd be rebuilding: scene-change detection with perceptual dedup, OCR, transcript+OCR search, the wall-clock ladder, MCP tools with embedded usage examples, five workflow prompts, and a findings contract. One uvx command instead of an afternoon of glue. Is it really local? What leaves my machine? Nothing at runtime. The network is used only for one-time downloads ffmpeg build, whisper/OCR/diarization models . No telemetry. See Privacy privacy — and SECURITY.md /korovin-aa97/talkthrough-mcp/blob/main/SECURITY.md treats a violation of this promise as a vulnerability. Machine-readable entry points, so AI agents can install and use this server without a human reading docs: — step-by-step install instructions for agents llms-install.md — index of the documentation llms.txt — an .agents/skills/talkthrough/SKILL.md Agent Skill https://agentskills.io teaching the tool workflow; discovered automatically inside a checkout by Codex CLI $talkthrough and readable by Claude Code, Cursor, Copilot, Gemini CLI and other SKILL.md-compatible tools— instructions for coding agents contributing to this repo AGENTS.md — MCP registry manifest server.json — per-engine adapters, all generated from one source of truth and drift-tested incl. the Claude Code plugin under integrations/ integrations/claude-code/ URL/YouTube ingestion · persistent speaker-name labels label speakers · word-level speaker splitting · cloud STT · embeddings/semantic search · hosted/remote mode · .mcpb bundle · whisper.cpp backend MIT