Record and Replay, teach AI agents desktop workflows by showing them once VideoDB released an open-source MCP server that records desktop workflows via native accessibility events and optional screen video, then compiles them into reusable skill files for AI agents. The tool allows developers to demonstrate a task once and generate versioned skill artifacts that agents can execute later, supporting macOS, Windows, and Linux. Record desktop workflows once. Compile them into reusable agent skills. Record → Compile → Give the skill to your agent Install installation · Features features · How It Works how-it-works · Docs https://docs.videodb.io · Report Bug https://github.com/video-db/open-record-replay/issues An MCP server that records a human-operated desktop workflow and compiles it into reusable skill files. The generated SKILL.md can then be given to an agent so the agent can perform the workflow later. Record — Captures native accessibility events, typed values, target metadata, and optional screen video to VideoDB for visual reference. Compile — An LLM transforms the event log and scene descriptions into reusable SKILL.json and human-readable SKILL.md files. Use — The server installs the generated SKILL.md into the agent's global skills directory, where future agent runs can use the skill instructions, inputs, verification checks, and execution guidance. Demonstrate a task once on screen, and the server produces a self-contained, versioned skill artifact. This repo does not include a replay engine; replay is performed by the agent that consumes the generated skill. Record and Replay.mp4 git clone https://github.com/video-db/open-record-replay.git cd open-record-replay uv sync Create a .env file in the project root: VIDEODB API KEY=your VIDEODB API KEY Add to your MCP config claude desktop config.json , VS Code MCP settings, etc. : { "mcpServers": { "videodb-record-replay": { "command": "uv", "args": "run", "python", "server.py" , "cwd": "/path/to/open-record-replay" } } } Five tools and two resources will appear. You're ready to record. Platform-specific setup macOS — Requires Screen Recording, Microphone, Accessibility, and Input Monitoring permissions. Run the permission helper first: uv run python scripts/smoke macos hook.py --prompt-permissions If ready for event recording is false, enable the terminal process in System Settings Privacy & Security Accessibility and Input Monitoring , then rerun. Windows — Uses UI Automation. No additional setup required beyond the standard install. Linux — Uses AT-SPI. Ensure at-spi2-core is installed and your desktop environment has accessibility enabled. Recording is human-in-the-loop. The agent starts recording, announces that recording is active, then waits. The human operator performs the UI workflow being captured. record skill tool "my-workflow", lead in seconds=5 → Agent tells the operator recording is active → Operator switches to the target app and performs the workflow → Operator returns to the MCP client and says "stop" → Agent calls stop recording tool trim end seconds=10 → Agent calls compile skill tool video id, "my-workflow" → Agent verifies/reports global skill md path for future use lead in seconds The recorder starts capture immediately, then the compiler ignores events before the effective workflow start. With lead in seconds=5 , the operator can switch from the MCP client to the target app and should begin the demonstrated workflow after 5 seconds. trim end seconds Discards events at the tail of the recording. Use when the operator must switch back to the MCP client to say "stop". For example, trim end seconds=10 ignores the final 10 seconds so the generated skill does not include the operator returning to the terminal or chat window. Events-only mode If VideoDB screen capture is unavailable, the system falls back to recording native accessibility events only. Call compile skill tool with video id="" or video id="none" to compile from events alone. Human performs workflow | v +------------------------------------------------------------------+ | Native AX/UIA/AT-SPI hooks - events.jsonl | | VideoDB Capture SDK - video id, when capture succeeds | +------------------------------------------------------------------+ | v +------------------------------------------------------------------+ | Compiler | | events.jsonl + optional scene summaries/transcript - SKILL.json | | SKILL.json - SKILL.md | +------------------------------------------------------------------+ | v +------------------------------------------------------------------+ | Agent skill install | | ~/.mcp-videodb/skills/