{"slug": "show-hn-opencomputeruse-background-computer-use-for-agents-as-an-mcp-server", "title": "Show HN: OpenComputerUse – Background computer use for agents, as an MCP server", "summary": "OpenComputerUse launched as an open-source MCP server that lets AI agents drive macOS applications in the background without stealing pointer, keyboard focus or the frontmost app. The tool exposes MCP functions including start_session, screenshot, get_ui_tree, click, type_text and run_recipe, and routes recipe steps through a decision model — either TypeSafe Jev (jev-latest) or Cloudflare Clef/Clef-flash on Workers AI — that picks among a window's interactive elements with no branching and no generated text. The companion OpenComputerUse app, built with the IAmJSD/gpui fork, owns sessions and holds the Accessibility and Screen Recording permissions, and the server installs into Claude Code, Claude Desktop, Codex or OpenCode.", "body_md": "Computer use for agents that runs in the background, as an MCP server. Start a session with an app and get back a session id. Drive the app with that id, then end the session. The app works behind your other windows, and your pointer, keyboard focus and frontmost app stay where they are. Sessions end when the MCP server exits, including when it is killed.\n\n| Tool | What it does | \n|---|---|\n| `start_session` | Start an app (a `.app` path, bundle id or name on macOS; an executable elsewhere) and return a session id and its windows | \n| `end_session` ,`list_sessions` ,`list_windows` | Manage sessions | \n| `screenshot` | Capture a session window, even a covered one; `ui_tree: true` adds the accessibility tree | \n| `get_ui_tree` | One line per element: `[e12] Button \"Save\" @(x,y wxh) actions=press` | \n| `click` ,`move_mouse` ,`drag` ,`scroll` | Pointer actions at window coordinates, or `click` with`element: \"e12\"` | \n| `type_text` ,`press_key` | Text, and chords such as `cmd+s` or`ctrl+shift+tab enter` | \n| `set_value` ,`element_action` | Set an element's value, or run press, focus, showmenu, increment and similar actions | \n| `wait` | Let the app catch up | \n| `run_recipe` | Run a fixed list of steps with a decision model (see below) | \n| `permissions` | What the OS needs granted, and whether it is | \n\nEvery action returns a fresh screenshot unless you pass `screenshot: false`.\nPass `ui_tree: true` to also get the tree. Coordinates are points from the\nwindow's top-left, the same grid as its screenshot.\n\n`run_recipe` runs straight-line chores without the calling model in the\nloop:\n\n```\n{ \"session_id\": \"…\", \"steps\": [\n  \"click the address bar\",\n  \"type \\\"example.com\\\" into the address bar\",\n  \"press enter\",\n  { \"click\": \"the More information link\" }\n] }\n```\n\nFor each step, the runner sends the window's interactive elements to a\ndecision model as the options of one `choice` question. It acts on the\nchosen element when the model is confident enough. There is no branching,\nand the model generates no text. A step it cannot place stops the recipe and\nreturns a screenshot and the tree so the caller can take over. Two models\nare supported, chosen in the app's settings:\n\n- **TypeSafe Jev** (`api.typesafe.ai` , model`jev-latest` )\n- **Cloudflare Clef / Clef-flash** on Workers AI (`@cf/cloudflare/clef-flash` )\n\nEach platform implements the `Platform` and `Session` traits in\n`crates/ocu-core`. Everything above them is shared: session ids, ownership,\ncleanup, the MCP tools and recipes.\n\n- **Launch:** apps open without activating. Your app's windows are put back\non top, so the session window sits behind them.\n- **Screenshots:** ScreenCaptureKit captures the window itself, so covered\nwindows capture as they look.\n- **Tree and element actions:** the Accessibility API, which works on\nbackground windows.\n- **Pointer and keys:** posted to the app's process through SkyLight. The\napp is first told its window is active, without being raised; this\n\"focus without raise\" approach comes from yabai and trycua/cua.\n\nThe MCP server is a thin client. The **OpenComputerUse app** owns the\nsessions, holds the Accessibility and Screen Recording permissions, and\ndraws a halo and a gliding cursor over the window being driven. It is built\nwith GPUI (the `IAmJSD/gpui` fork). The MCP server starts the app through\nLaunchServices when needed, so the app keeps its own permissions whichever\nclient started the server. Opening the app shows its window:\npermissions, one-click install into Claude Code, Claude Desktop, Codex or OpenCode, live sessions, and\nrecipe settings.\n\n```\nCODESIGN_IDENTITY=\"Developer ID Application: …\" scripts/bundle-macos.sh   # universal (arm64 + x86_64)\ncp -R dist/OpenComputerUse.app /Applications/ && open /Applications/OpenComputerUse.app\n```\n\nThe icon is `assets/icon.svg`; `packaging/macos/icon.sh` rebuilds the `.icns` from it.\n\nSign the bundle with a real identity. An ad hoc signature changes on every build, and macOS then asks for the permissions again.\n\n- **Display:** each session gets a private Xvfb display with its own cookie.\nThe app starts on it via`DISPLAY` and`XAUTHORITY` , with Wayland\ndisabled.\n- **Input and screenshots:** input goes through XTEST; screenshots read the\nframebuffer, so menus and popups are included.\n- **Cleanup:** Xvfb and the app get`PR_SET_PDEATHSIG` , so they die with\nthe server.\n- **Where Xvfb is found:**`$OCU_XVFB` , next to the binary, or on`PATH` .\n- **Not yet:** the accessibility tree (AT-SPI), so element actions and\nrecipes are unavailable on Linux for now.\n- **Watching a session:** connect a VNC server to its display, for example`x11vnc -display :N -auth <xauthority>` . The display and auth file are in\nthe session details.\n\n```\ndocker build -f scripts/linux/Dockerfile -t ocu-linux . && docker run --rm ocu-linux\n```\n\n- **Launch:** apps start suspended inside a kill-on-close Job Object, so\ntheir whole process tree dies with the server. They are shown without\nactivating and sent to the back.\n- **Screenshots:**`PrintWindow` with full-content rendering.\n- **Input:** window messages posted to the control under the point, or to\nthe focused control.\n- **Tree and element actions:** UI Automation.\n\nFrom the app's window, or:\n\n```\nopencomputeruse install claude          # claude mcp add --scope user opencomputeruse -- <path> mcp\nopencomputeruse install claude-desktop  # an \"mcpServers\" entry in claude_desktop_config.json\nopencomputeruse install codex           # codex mcp add opencomputeruse -- <path> mcp\nopencomputeruse install opencode        # an \"mcp\" entry in ~/.config/opencode/opencode.json(c)\nopencomputeruse clients                 # which clients run this copy\n```\n\nClaude Desktop reads its config when it starts, so quit and reopen it after installing. A client whose entry runs another copy (an old build, or the app before it moved) shows as \"points elsewhere\"; installing again points it here.\n\nFor other clients, use `{ \"command\": \"<path to opencomputeruse>\", \"args\": [\"mcp\"] }`.\n\nOff by default. Turn on **Serve over HTTP** in the app (or the local MCP\nserver's `http_server` tool) and other devices can drive this computer\nthrough an HTTP API on port 8642, usually over Tailscale. While it is on, it\nstarts again at login, so it survives reboots.\n\nEach device needs a key. **Generate Skill** asks for the device's name and the\nURL it reaches this computer at (this computer's Tailscale name by default),\nand gives back a `SKILL.md` to install on that device. The skill carries the\nURL, the key and how to call every tool with curl. Keys are stored only as\nhashes, so the skill is the one place a key appears. Devices are listed in\nthe app with **Regenerate Key** and **Remove**, and in the local MCP server\nas `list_devices`, `generate_skill`, `regenerate_key` and `remove_device`.\nRegenerating or removing a key ends that device's sessions. None of this\nmanagement is reachable over HTTP.\n\nThe API:\n\n- `POST /v1/tools/<tool>` with JSON arguments returns`{content, isError}` ,\nthe same as an MCP tool call.`GET /v1/tools` lists the tools.\n- `POST /mcp` is MCP over HTTP:`claude mcp add --transport http <name> <url>/mcp --header \"Authorization: Bearer <key>\"` .\n- Every request needs `Authorization: Bearer <key>` , except`GET /health` .\n\nThere is no TLS, so use it over Tailscale or another trusted network. On\nLinux and Windows, `opencomputeruse serve` runs the server, and\n`serve --install` starts it at login.\n\nThe settings file is at `opencomputeruse config-path`. On macOS it is\nedited from the app. Elsewhere, edit it by hand or set `TYPESAFE_API_KEY`,\n`CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_API_TOKEN` and `OCU_RECIPE_PROVIDER`.\nSet `OCU_LOG=debug` for logs on stderr. The macOS agent logs to\n`~/Library/Application Support/OpenComputerUse/agent.log`.\n\nBump `version` in `Cargo.toml`, commit, and push a matching tag (` v0.2.0`).\n`.github/workflows/release.yml` builds the signed universal app\n(`OpenComputerUse.zip` for the updater, `OpenComputerUse.dmg` for first\ninstalls) and plain Linux and Windows binaries of the MCP server, then\npublishes them as a GitHub release. The app checks for releases daily and from\nits menu, and installs them in place when they are signed by the same team.\n\nThe macOS job signs and notarizes with these repository secrets, and builds\nunsigned without them: `MACOS_CERT_P12_BASE64`, `MACOS_CERT_P12_PASSWORD`\n(a Developer ID Application certificate), `APPLE_ID`,\n`APPLE_APP_SPECIFIC_PASSWORD` and `APPLE_TEAM_ID`.\n\n```\ncargo test                                     # core, recipes and the updater\ncargo run -p ocu-macos --example smoke         # drive TextEdit directly\npython3 scripts/mcp_smoke.py                   # drive it through the MCP server\n```\n\nThe widget kit in `src/agent/ui` comes from Schist (MIT; see\n`LICENSE-SCHIST`).\n\nMIT, © 2026 Astrid Gealer. See [LICENSE](https://github.com/IAmJSD/OpenComputerUse/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-opencomputeruse-background-computer-use-for-agents-as-an-mcp-server", "canonical_source": "https://github.com/IAmJSD/OpenComputerUse", "published_at": "2026-10-07 17:07:14+00:00", "updated_at": "2026-10-07 17:20:31.122359+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "developer-tools", "ai-tools"], "entities": ["OpenComputerUse", "TypeSafe Jev", "Cloudflare", "Workers AI", "Clef-flash", "Claude Code", "Claude Desktop", "Codex"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-opencomputeruse-background-computer-use-for-agents-as-an-mcp-server", "markdown": "https://wpnews.pro/news/show-hn-opencomputeruse-background-computer-use-for-agents-as-an-mcp-server.md", "text": "https://wpnews.pro/news/show-hn-opencomputeruse-background-computer-use-for-agents-as-an-mcp-server.txt", "jsonld": "https://wpnews.pro/news/show-hn-opencomputeruse-background-computer-use-for-agents-as-an-mcp-server.jsonld"}}