Show HN: OpenComputerUse – Background computer use for agents, as an MCP server

OpenComputerUse launched as an open-source MCP server that lets AI agents drive macOS applications in the background without stealing pointer, keyboard focus or the frontmost app. The tool exposes MCP functions including start_session, screenshot, get_ui_tree, click, type_text and run_recipe, and routes recipe steps through a decision model — either TypeSafe Jev (jev-latest) or Cloudflare Clef/Clef-flash on Workers AI — that picks among a window's interactive elements with no branching and no generated text. The companion OpenComputerUse app, built with the IAmJSD/gpui fork, owns sessions and holds the Accessibility and Screen Recording permissions, and the server installs into Claude Code, Claude Desktop, Codex or OpenCode.

Computer use for agents that runs in the background, as an MCP server. Start a session with an app and get back a session id. Drive the app with that id, then end the session. The app works behind your other windows, and your pointer, keyboard focus and frontmost app stay where they are. Sessions end when the MCP server exits, including when it is killed. | Tool | What it does | |---|---| | start session | Start an app a .app path, bundle id or name on macOS; an executable elsewhere and return a session id and its windows | | end session , list sessions , list windows | Manage sessions | | screenshot | Capture a session window, even a covered one; ui tree: true adds the accessibility tree | | get ui tree | One line per element: e12 Button "Save" @ x,y wxh actions=press | | click , move mouse , drag , scroll | Pointer actions at window coordinates, or click with element: "e12" | | type text , press key | Text, and chords such as cmd+s or ctrl+shift+tab enter | | set value , element action | Set an element's value, or run press, focus, showmenu, increment and similar actions | | wait | Let the app catch up | | run recipe | Run a fixed list of steps with a decision model see below | | permissions | What the OS needs granted, and whether it is | Every action returns a fresh screenshot unless you pass screenshot: false . Pass ui tree: true to also get the tree. Coordinates are points from the window's top-left, the same grid as its screenshot. run recipe runs straight-line chores without the calling model in the loop: { "session id": "…", "steps": "click the address bar", "type \"example.com\" into the address bar", "press enter", { "click": "the More information link" } } For each step, the runner sends the window's interactive elements to a decision model as the options of one choice question. It acts on the chosen element when the model is confident enough. There is no branching, and the model generates no text. A step it cannot place stops the recipe and returns a screenshot and the tree so the caller can take over. Two models are supported, chosen in the app's settings: - TypeSafe Jev api.typesafe.ai , model jev-latest - Cloudflare Clef / Clef-flash on Workers AI @cf/cloudflare/clef-flash Each platform implements the Platform and Session traits in crates/ocu-core . Everything above them is shared: session ids, ownership, cleanup, the MCP tools and recipes. - Launch: apps open without activating. Your app's windows are put back on top, so the session window sits behind them. - Screenshots: ScreenCaptureKit captures the window itself, so covered windows capture as they look. - Tree and element actions: the Accessibility API, which works on background windows. - Pointer and keys: posted to the app's process through SkyLight. The app is first told its window is active, without being raised; this "focus without raise" approach comes from yabai and trycua/cua. The MCP server is a thin client. The OpenComputerUse app owns the sessions, holds the Accessibility and Screen Recording permissions, and draws a halo and a gliding cursor over the window being driven. It is built with GPUI the IAmJSD/gpui fork . The MCP server starts the app through LaunchServices when needed, so the app keeps its own permissions whichever client started the server. Opening the app shows its window: permissions, one-click install into Claude Code, Claude Desktop, Codex or OpenCode, live sessions, and recipe settings. CODESIGN IDENTITY="Developer ID Application: …" scripts/bundle-macos.sh universal arm64 + x86 64 cp -R dist/OpenComputerUse.app /Applications/ && open /Applications/OpenComputerUse.app The icon is assets/icon.svg ; packaging/macos/icon.sh rebuilds the .icns from it. Sign the bundle with a real identity. An ad hoc signature changes on every build, and macOS then asks for the permissions again. - Display: each session gets a private Xvfb display with its own cookie. The app starts on it via DISPLAY and XAUTHORITY , with Wayland disabled. - Input and screenshots: input goes through XTEST; screenshots read the framebuffer, so menus and popups are included. - Cleanup: Xvfb and the app get PR SET PDEATHSIG , so they die with the server. - Where Xvfb is found: $OCU XVFB , next to the binary, or on PATH . - Not yet: the accessibility tree AT-SPI , so element actions and recipes are unavailable on Linux for now. - Watching a session: connect a VNC server to its display, for example x11vnc -display :N -auth <xauthority . The display and auth file are in the session details. docker build -f scripts/linux/Dockerfile -t ocu-linux . && docker run --rm ocu-linux - Launch: apps start suspended inside a kill-on-close Job Object, so their whole process tree dies with the server. They are shown without activating and sent to the back. - Screenshots: PrintWindow with full-content rendering. - Input: window messages posted to the control under the point, or to the focused control. - Tree and element actions: UI Automation. From the app's window, or: opencomputeruse install claude claude mcp add --scope user opencomputeruse -- <path mcp opencomputeruse install claude-desktop an "mcpServers" entry in claude desktop config.json opencomputeruse install codex codex mcp add opencomputeruse -- <path mcp opencomputeruse install opencode an "mcp" entry in ~/.config/opencode/opencode.json c opencomputeruse clients which clients run this copy Claude Desktop reads its config when it starts, so quit and reopen it after installing. A client whose entry runs another copy an old build, or the app before it moved shows as "points elsewhere"; installing again points it here. For other clients, use { "command": "<path to opencomputeruse ", "args": "mcp" } . Off by default. Turn on Serve over HTTP in the app or the local MCP server's http server tool and other devices can drive this computer through an HTTP API on port 8642, usually over Tailscale. While it is on, it starts again at login, so it survives reboots. Each device needs a key. Generate Skill asks for the device's name and the URL it reaches this computer at this computer's Tailscale name by default , and gives back a SKILL.md to install on that device. The skill carries the URL, the key and how to call every tool with curl. Keys are stored only as hashes, so the skill is the one place a key appears. Devices are listed in the app with Regenerate Key and Remove , and in the local MCP server as list devices , generate skill , regenerate key and remove device . Regenerating or removing a key ends that device's sessions. None of this management is reachable over HTTP. The API: - POST /v1/tools/<tool with JSON arguments returns {content, isError} , the same as an MCP tool call. GET /v1/tools lists the tools. - POST /mcp is MCP over HTTP: claude mcp add --transport http <name <url /mcp --header "Authorization: Bearer <key " . - Every request needs Authorization: Bearer <key , except GET /health . There is no TLS, so use it over Tailscale or another trusted network. On Linux and Windows, opencomputeruse serve runs the server, and serve --install starts it at login. The settings file is at opencomputeruse config-path . On macOS it is edited from the app. Elsewhere, edit it by hand or set TYPESAFE API KEY , CLOUDFLARE ACCOUNT ID , CLOUDFLARE API TOKEN and OCU RECIPE PROVIDER . Set OCU LOG=debug for logs on stderr. The macOS agent logs to ~/Library/Application Support/OpenComputerUse/agent.log . Bump version in Cargo.toml , commit, and push a matching tag v0.2.0 . .github/workflows/release.yml builds the signed universal app OpenComputerUse.zip for the updater, OpenComputerUse.dmg for first installs and plain Linux and Windows binaries of the MCP server, then publishes them as a GitHub release. The app checks for releases daily and from its menu, and installs them in place when they are signed by the same team. The macOS job signs and notarizes with these repository secrets, and builds unsigned without them: MACOS CERT P12 BASE64 , MACOS CERT P12 PASSWORD a Developer ID Application certificate , APPLE ID , APPLE APP SPECIFIC PASSWORD and APPLE TEAM ID . cargo test core, recipes and the updater cargo run -p ocu-macos --example smoke drive TextEdit directly python3 scripts/mcp smoke.py drive it through the MCP server The widget kit in src/agent/ui comes from Schist MIT; see LICENSE-SCHIST . MIT, © 2026 Astrid Gealer. See LICENSE https://github.com/IAmJSD/OpenComputerUse/blob/main/LICENSE .