{"slug": "record-and-replay-teach-ai-agents-desktop-workflows-by-showing-them-once", "title": "Record and Replay, teach AI agents desktop workflows by showing them once", "summary": "VideoDB released an open-source MCP server that records desktop workflows via native accessibility events and optional screen video, then compiles them into reusable skill files for AI agents. The tool allows developers to demonstrate a task once and generate versioned skill artifacts that agents can execute later, supporting macOS, Windows, and Linux.", "body_md": "Record desktop workflows once. Compile them into reusable agent skills.\n\n**Record → Compile → Give the skill to your agent**\n\n[Install](#installation)\n·\n[Features](#features)\n·\n[How It Works](#how-it-works)\n·\n[ Docs](https://docs.videodb.io)\n·\n\n[Report Bug](https://github.com/video-db/open-record-replay/issues)\n\nAn MCP server that records a human-operated desktop workflow and compiles it into reusable skill files. The generated `SKILL.md`\n\ncan then be given to an agent so the agent can perform the workflow later.\n\n**Record**— Captures native accessibility events, typed values, target metadata, and optional screen video to VideoDB for visual reference.**Compile**— An LLM transforms the event log and scene descriptions into reusable`SKILL.json`\n\nand human-readable`SKILL.md`\n\nfiles.**Use**— The server installs the generated`SKILL.md`\n\ninto the agent's global skills directory, where future agent runs can use the skill instructions, inputs, verification checks, and execution guidance.\n\nDemonstrate a task once on screen, and the server produces a self-contained, versioned skill artifact. This repo does not include a replay engine; replay is performed by the agent that consumes the generated skill.\n\n## Record_and_Replay.mp4\n\n```\ngit clone https://github.com/video-db/open-record-replay.git\ncd open-record-replay\nuv sync\n```\n\nCreate a `.env`\n\nfile in the project root:\n\n```\nVIDEODB_API_KEY=your_VIDEODB_API_KEY\n```\n\nAdd to your MCP config (`claude_desktop_config.json`\n\n, VS Code MCP settings, etc.):\n\n```\n{\n  \"mcpServers\": {\n    \"videodb-record-replay\": {\n      \"command\": \"uv\",\n      \"args\": [\"run\", \"python\", \"server.py\"],\n      \"cwd\": \"/path/to/open-record-replay\"\n    }\n  }\n}\n```\n\nFive tools and two resources will appear. You're ready to record.\n\n**Platform-specific setup**\n\n**macOS** — Requires Screen Recording, Microphone, Accessibility, and Input Monitoring permissions. Run the permission helper first:\n\n```\nuv run python scripts/smoke_macos_hook.py --prompt-permissions\n```\n\nIf `ready_for_event_recording`\n\nis false, enable the terminal process in **System Settings > Privacy & Security > Accessibility** and **Input Monitoring**, then rerun.\n\n**Windows** — Uses UI Automation. No additional setup required beyond the standard install.\n\n**Linux** — Uses AT-SPI. Ensure `at-spi2-core`\n\nis installed and your desktop environment has accessibility enabled.\n\nRecording is human-in-the-loop. The agent starts recording, announces that recording is active, then waits. The human operator performs the UI workflow being captured.\n\n```\nrecord_skill_tool(\"my-workflow\", lead_in_seconds=5)\n    → Agent tells the operator recording is active\n    → Operator switches to the target app and performs the workflow\n    → Operator returns to the MCP client and says \"stop\"\n    → Agent calls stop_recording_tool(trim_end_seconds=10)\n    → Agent calls compile_skill_tool(video_id, \"my-workflow\")\n    → Agent verifies/reports global_skill_md_path for future use\n```\n\n**lead_in_seconds**\n\nThe recorder starts capture immediately, then the compiler ignores events before the effective workflow start. With `lead_in_seconds=5`\n\n, the operator can switch from the MCP client to the target app and should begin the demonstrated workflow after 5 seconds.\n\n**trim_end_seconds**\n\nDiscards events at the tail of the recording. Use when the operator must switch back to the MCP client to say \"stop\". For example, `trim_end_seconds=10`\n\nignores the final 10 seconds so the generated skill does not include the operator returning to the terminal or chat window.\n\n**Events-only mode**\n\nIf VideoDB screen capture is unavailable, the system falls back to recording native accessibility events only. Call `compile_skill_tool`\n\nwith `video_id=\"\"`\n\nor `video_id=\"none\"`\n\nto compile from events alone.\n\n```\nHuman performs workflow\n        |\n        v\n+------------------------------------------------------------------+\n| Native AX/UIA/AT-SPI hooks -> events.jsonl                       |\n| VideoDB Capture SDK -> video_id, when capture succeeds           |\n+------------------------------------------------------------------+\n        |\n        v\n+------------------------------------------------------------------+\n| Compiler                                                         |\n| events.jsonl + optional scene summaries/transcript -> SKILL.json |\n| SKILL.json -> SKILL.md                                           |\n+------------------------------------------------------------------+\n        |\n        v\n+------------------------------------------------------------------+\n| Agent skill install                                              |\n| ~/.mcp-videodb/skills/<name>/SKILL.md                            |\n| ~/.codex/skills/<name>/SKILL.md, unless overridden               |\n+------------------------------------------------------------------+\n```\n\nThe accessibility hooks provide the action log. VideoDB scene indexing adds visual context when screen capture is available. The compiler combines those signals into skill files that a future agent can use to perform the workflow.\n\n| Feature | Description |\n|---|---|\nDual recording |\nCaptures native accessibility events and, when full capture is available, screen video for visual reference |\nLLM compilation |\nVideoDB scene indexing and `generate_text` compile event logs, scene descriptions, and transcript text into structured `SKILL.json` |\nGraceful degradation |\nFalls back to events-only recording when screen capture is unavailable |\nCross-platform |\nNative accessibility hooks for Windows (UIA), macOS (AX), and Linux (AT-SPI) |\nSkill versioning |\nAuto-increments on recompile; archives old versions as `SKILL.vN.json` |\nVariable templating |\nPrompts the compiler to turn recorded literals such as search queries, dates, dropdown choices, and file paths into reusable inputs |\nHuman-in-the-loop |\nRecording is operator-driven, not agent-driven — the human demonstrates, the AI learns |\n\n| Tool | Parameters | Description |\n|---|---|---|\n`request_capture_permissions_tool` |\n— | Request microphone and screen capture permissions before recording |\n`record_skill_tool` |\n`name: str` , `lead_in_seconds: float = 0.0` |\nStart a human-in-the-loop workflow recording |\n`stop_recording_tool` |\n`trim_end_seconds: float = 0.0` |\nStop the active recording and export video to VideoDB when capture is available |\n`compile_skill_tool` |\n`video_id: str` , `name: str` |\nCompile a recording into `SKILL.json` and `SKILL.md` , then install `SKILL.md` globally for future agent use |\n`list_skills_tool` |\n— | List all skills generated through this MCP |\n\n| Resource | Description |\n|---|---|\n`skills://list` |\nList all available skills as JSON |\n`skills://{name}/content` |\nLoad a skill's `SKILL.md` into the agent context |\n\nCompiled skills land in `~/.mcp-videodb/skills/<name>/`\n\n:\n\n| File | Purpose |\n|---|---|\n`SKILL.json` |\nStructured skill definition with steps, inputs, verification checks, recorded surface data, and execution strategy |\n`SKILL.md` |\nHuman and agent-readable markdown following the agentskills.io standard |\n`SKILL.vN.json` |\nArchived previous versions on recompile |\n\nEvery generated `SKILL.json`\n\nincludes an `execution_strategy`\n\n— `web_browser`\n\n, `desktop_app`\n\n, `hybrid`\n\n, `terminal`\n\n, `file_system`\n\n, or `unknown`\n\n— so the agent consuming the skill knows which tool path to prefer. Every `SKILL.md`\n\nincludes execution guidance, continuous improvement guidance, and agent tool-priority guidance.\n\nAfter `SKILL.md`\n\nis created, `compile_skill_tool`\n\nalso installs it into the\nagent's global skills directory, `~/.codex/skills/<name>/SKILL.md`\n\nby default,\nand returns `global_skill_md_path`\n\nplus an `agent_instruction`\n\nreminding the\nagent to verify or report the global install. Agents should ensure this global\ninstall step has happened so the skill is available in future runs. Set\n`CODEX_HOME`\n\nto change the base Codex directory, or set\n`AGENT_GLOBAL_SKILLS_ROOT`\n\nto override the global skills directory directly.\n\n```\nopen-record-replay/\n├── server.py                 # FastMCP entry point, tool and resource definitions\n├── state.py                  # Shared server state singleton\n├── config.py                 # Constants, .env loading\n├── registry.py               # Skill CRUD + versioning\n│\n├── capture/\n│   ├── recorder.py           # Records native accessibility events + optional VideoDB capture\n│   ├── ax_client.py          # JSONL IPC wrapper for native accessibility companion\n│   ├── capture_client.py     # VideoDB Capture SDK wrapper\n│   └── native/\n│       ├── ax_hook_win32.py   # Windows: UI Automation + keyboard polling + TCP IPC\n│       ├── ax_hook_darwin.py  # macOS: Accessibility API + pynput + pipe IPC\n│       └── ax_hook_linux.py   # Linux: AT-SPI + pynput + pipe IPC\n│\n├── compiler/\n│   ├── compiler.py           # LLM compilation: index scenes → match events → prompt → normalize\n│   ├── prompts.py            # LLM system prompt for structured skill generation\n│   ├── md_generator.py       # Converts SKILL.json to agent-readable SKILL.md\n│   ├── tool_manifest.py      # Surface-to-tool mapping for replay guidance\n│   └── recommended_tools.json\n│\n├── schema/\n│   └── skill.schema.json     # JSON Schema (draft-07) for SKILL.json validation\n│\n├── scripts/\n│   └── smoke_macos_hook.py   # macOS AX hook permission helper\n```\n\n**Recording won't start**\n\n- Verify\n`VIDEODB_API_KEY`\n\nis set in`.env`\n\nand is valid - Run\n`request_capture_permissions_tool`\n\nand approve any permission prompts - On macOS, check Screen Recording and Accessibility permissions\n- Check that no other application is using the accessibility hook\n\n**Compilation fails or returns empty steps**\n\n- Ensure the recording has meaningful UI interactions (not just idle time)\n- Try events-only compilation (\n`video_id=\"\"`\n\n) if video indexing is slow - The LLM may need another generation attempt — the compiler is configured for up to 2 attempts\n\n**Permission prompts not appearing on macOS**\n\n```\n# Reset permissions and try again\nuv run python scripts/smoke_macos_hook.py --prompt-permissions\n```\n\nIf `ready_for_event_recording`\n\nis false, manually enable the terminal in **System Settings > Privacy & Security > Accessibility** and **Input Monitoring**.\n\n**Windows: no keyboard events recorded**\n\n- Ensure the app being recorded has UI Automation support (most modern apps do)\n\n**Docs**:[docs.videodb.io](https://docs.videodb.io)** Issues**:[GitHub Issues](https://github.com/video-db/open-record-replay/issues)** Discord**:[Join the VideoDB community](https://discord.gg/py9P639jGz)** API Key**:[console.videodb.io](https://console.videodb.io)\n\nMade with ❤️ by the [VideoDB](https://videodb.io) team", "url": "https://wpnews.pro/news/record-and-replay-teach-ai-agents-desktop-workflows-by-showing-them-once", "canonical_source": "https://github.com/video-db/open-record-replay", "published_at": "2026-07-10 08:04:05+00:00", "updated_at": "2026-07-10 08:35:17.118075+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools"], "entities": ["VideoDB", "MCP"], "alternates": {"html": "https://wpnews.pro/news/record-and-replay-teach-ai-agents-desktop-workflows-by-showing-them-once", "markdown": "https://wpnews.pro/news/record-and-replay-teach-ai-agents-desktop-workflows-by-showing-them-once.md", "text": "https://wpnews.pro/news/record-and-replay-teach-ai-agents-desktop-workflows-by-showing-them-once.txt", "jsonld": "https://wpnews.pro/news/record-and-replay-teach-ai-agents-desktop-workflows-by-showing-them-once.jsonld"}}