# Record and Replay, teach AI agents desktop workflows by showing them once > Source: > Published: 2026-07-10 08:04:05+00:00 Record desktop workflows once. Compile them into reusable agent skills. **Record → Compile → Give the skill to your agent** [Install](#installation) · [Features](#features) · [How It Works](#how-it-works) · [ Docs](https://docs.videodb.io) · [Report Bug](https://github.com/video-db/open-record-replay/issues) An MCP server that records a human-operated desktop workflow and compiles it into reusable skill files. The generated `SKILL.md` can then be given to an agent so the agent can perform the workflow later. **Record**— Captures native accessibility events, typed values, target metadata, and optional screen video to VideoDB for visual reference.**Compile**— An LLM transforms the event log and scene descriptions into reusable`SKILL.json` and human-readable`SKILL.md` files.**Use**— The server installs the generated`SKILL.md` into the agent's global skills directory, where future agent runs can use the skill instructions, inputs, verification checks, and execution guidance. Demonstrate a task once on screen, and the server produces a self-contained, versioned skill artifact. This repo does not include a replay engine; replay is performed by the agent that consumes the generated skill. ## Record_and_Replay.mp4 ``` git clone https://github.com/video-db/open-record-replay.git cd open-record-replay uv sync ``` Create a `.env` file in the project root: ``` VIDEODB_API_KEY=your_VIDEODB_API_KEY ``` Add to your MCP config (`claude_desktop_config.json` , VS Code MCP settings, etc.): ``` { "mcpServers": { "videodb-record-replay": { "command": "uv", "args": ["run", "python", "server.py"], "cwd": "/path/to/open-record-replay" } } } ``` Five tools and two resources will appear. You're ready to record. **Platform-specific setup** **macOS** — Requires Screen Recording, Microphone, Accessibility, and Input Monitoring permissions. Run the permission helper first: ``` uv run python scripts/smoke_macos_hook.py --prompt-permissions ``` If `ready_for_event_recording` is false, enable the terminal process in **System Settings > Privacy & Security > Accessibility** and **Input Monitoring**, then rerun. **Windows** — Uses UI Automation. No additional setup required beyond the standard install. **Linux** — Uses AT-SPI. Ensure `at-spi2-core` is installed and your desktop environment has accessibility enabled. Recording is human-in-the-loop. The agent starts recording, announces that recording is active, then waits. The human operator performs the UI workflow being captured. ``` record_skill_tool("my-workflow", lead_in_seconds=5) → Agent tells the operator recording is active → Operator switches to the target app and performs the workflow → Operator returns to the MCP client and says "stop" → Agent calls stop_recording_tool(trim_end_seconds=10) → Agent calls compile_skill_tool(video_id, "my-workflow") → Agent verifies/reports global_skill_md_path for future use ``` **lead_in_seconds** The recorder starts capture immediately, then the compiler ignores events before the effective workflow start. With `lead_in_seconds=5` , the operator can switch from the MCP client to the target app and should begin the demonstrated workflow after 5 seconds. **trim_end_seconds** Discards events at the tail of the recording. Use when the operator must switch back to the MCP client to say "stop". For example, `trim_end_seconds=10` ignores the final 10 seconds so the generated skill does not include the operator returning to the terminal or chat window. **Events-only mode** If VideoDB screen capture is unavailable, the system falls back to recording native accessibility events only. Call `compile_skill_tool` with `video_id=""` or `video_id="none"` to compile from events alone. ``` Human performs workflow | v +------------------------------------------------------------------+ | Native AX/UIA/AT-SPI hooks -> events.jsonl | | VideoDB Capture SDK -> video_id, when capture succeeds | +------------------------------------------------------------------+ | v +------------------------------------------------------------------+ | Compiler | | events.jsonl + optional scene summaries/transcript -> SKILL.json | | SKILL.json -> SKILL.md | +------------------------------------------------------------------+ | v +------------------------------------------------------------------+ | Agent skill install | | ~/.mcp-videodb/skills//SKILL.md | | ~/.codex/skills//SKILL.md, unless overridden | +------------------------------------------------------------------+ ``` The accessibility hooks provide the action log. VideoDB scene indexing adds visual context when screen capture is available. The compiler combines those signals into skill files that a future agent can use to perform the workflow. | Feature | Description | |---|---| Dual recording | Captures native accessibility events and, when full capture is available, screen video for visual reference | LLM compilation | VideoDB scene indexing and `generate_text` compile event logs, scene descriptions, and transcript text into structured `SKILL.json` | Graceful degradation | Falls back to events-only recording when screen capture is unavailable | Cross-platform | Native accessibility hooks for Windows (UIA), macOS (AX), and Linux (AT-SPI) | Skill versioning | Auto-increments on recompile; archives old versions as `SKILL.vN.json` | Variable templating | Prompts the compiler to turn recorded literals such as search queries, dates, dropdown choices, and file paths into reusable inputs | Human-in-the-loop | Recording is operator-driven, not agent-driven — the human demonstrates, the AI learns | | Tool | Parameters | Description | |---|---|---| `request_capture_permissions_tool` | — | Request microphone and screen capture permissions before recording | `record_skill_tool` | `name: str` , `lead_in_seconds: float = 0.0` | Start a human-in-the-loop workflow recording | `stop_recording_tool` | `trim_end_seconds: float = 0.0` | Stop the active recording and export video to VideoDB when capture is available | `compile_skill_tool` | `video_id: str` , `name: str` | Compile a recording into `SKILL.json` and `SKILL.md` , then install `SKILL.md` globally for future agent use | `list_skills_tool` | — | List all skills generated through this MCP | | Resource | Description | |---|---| `skills://list` | List all available skills as JSON | `skills://{name}/content` | Load a skill's `SKILL.md` into the agent context | Compiled skills land in `~/.mcp-videodb/skills//` : | File | Purpose | |---|---| `SKILL.json` | Structured skill definition with steps, inputs, verification checks, recorded surface data, and execution strategy | `SKILL.md` | Human and agent-readable markdown following the agentskills.io standard | `SKILL.vN.json` | Archived previous versions on recompile | Every generated `SKILL.json` includes an `execution_strategy` — `web_browser` , `desktop_app` , `hybrid` , `terminal` , `file_system` , or `unknown` — so the agent consuming the skill knows which tool path to prefer. Every `SKILL.md` includes execution guidance, continuous improvement guidance, and agent tool-priority guidance. After `SKILL.md` is created, `compile_skill_tool` also installs it into the agent's global skills directory, `~/.codex/skills//SKILL.md` by default, and returns `global_skill_md_path` plus an `agent_instruction` reminding the agent to verify or report the global install. Agents should ensure this global install step has happened so the skill is available in future runs. Set `CODEX_HOME` to change the base Codex directory, or set `AGENT_GLOBAL_SKILLS_ROOT` to override the global skills directory directly. ``` open-record-replay/ ├── server.py # FastMCP entry point, tool and resource definitions ├── state.py # Shared server state singleton ├── config.py # Constants, .env loading ├── registry.py # Skill CRUD + versioning │ ├── capture/ │ ├── recorder.py # Records native accessibility events + optional VideoDB capture │ ├── ax_client.py # JSONL IPC wrapper for native accessibility companion │ ├── capture_client.py # VideoDB Capture SDK wrapper │ └── native/ │ ├── ax_hook_win32.py # Windows: UI Automation + keyboard polling + TCP IPC │ ├── ax_hook_darwin.py # macOS: Accessibility API + pynput + pipe IPC │ └── ax_hook_linux.py # Linux: AT-SPI + pynput + pipe IPC │ ├── compiler/ │ ├── compiler.py # LLM compilation: index scenes → match events → prompt → normalize │ ├── prompts.py # LLM system prompt for structured skill generation │ ├── md_generator.py # Converts SKILL.json to agent-readable SKILL.md │ ├── tool_manifest.py # Surface-to-tool mapping for replay guidance │ └── recommended_tools.json │ ├── schema/ │ └── skill.schema.json # JSON Schema (draft-07) for SKILL.json validation │ ├── scripts/ │ └── smoke_macos_hook.py # macOS AX hook permission helper ``` **Recording won't start** - Verify `VIDEODB_API_KEY` is set in`.env` and is valid - Run `request_capture_permissions_tool` and approve any permission prompts - On macOS, check Screen Recording and Accessibility permissions - Check that no other application is using the accessibility hook **Compilation fails or returns empty steps** - Ensure the recording has meaningful UI interactions (not just idle time) - Try events-only compilation ( `video_id=""` ) if video indexing is slow - The LLM may need another generation attempt — the compiler is configured for up to 2 attempts **Permission prompts not appearing on macOS** ``` # Reset permissions and try again uv run python scripts/smoke_macos_hook.py --prompt-permissions ``` If `ready_for_event_recording` is false, manually enable the terminal in **System Settings > Privacy & Security > Accessibility** and **Input Monitoring**. **Windows: no keyboard events recorded** - Ensure the app being recorded has UI Automation support (most modern apps do) **Docs**:[docs.videodb.io](https://docs.videodb.io)** Issues**:[GitHub Issues](https://github.com/video-db/open-record-replay/issues)** Discord**:[Join the VideoDB community](https://discord.gg/py9P639jGz)** API Key**:[console.videodb.io](https://console.videodb.io) Made with ❤️ by the [VideoDB](https://videodb.io) team