{"slug": "agent-desktop-vercel-labs-computer-use-cli-reads-the-accessibility-tree-not", "title": "agent-desktop: Vercel Labs' Computer Use CLI Reads the Accessibility Tree, Not Pixels", "summary": "Vercel Labs published agent-desktop, an Apache-2.0 Rust CLI and C library that gives AI agents computer use by reading the operating system accessibility tree — macOS Accessibility API, Windows UI Automation and Linux AT-SPI — instead of screenshots, returning JSON snapshots in which each actionable element carries a ref the agent clicks, types or scrolls by. Version 1.0.0 shipped October 27, 2026 with Linux support, and 1.0.2 followed on October 30; the repository had about 2.2k GitHub stars and installs via `npm install -g agent-desktop`. The project's CONCEPTS.md calls the tree \"the source of truth for observation and semantic interaction,\" and stale or ambiguous refs return STALE_REF or AMBIGUOUS_TARGET errors so agents re-snapshot rather than click the wrong row.", "body_md": "**agent-desktop** is a command-line tool that gives any AI agent computer use on the desktop without looking at pixels. Published under [vercel-labs/agent-desktop](https://github.com/vercel-labs/agent-desktop?ref=explainx) on GitHub (about 2.2k stars when we checked) and licensed Apache-2.0, it reads an app's real interface structure from the operating system accessibility tree, hands the agent a JSON snapshot, and lets it click, type and scroll by element reference. Version 1.0.0 landed on October 27, 2026 with Linux support, and 1.0.2 followed on October 30.\n\n*Header image: agent-desktop share card, courtesy of Vercel Labs, from the [vercel-labs/agent-desktop repository](https://github.com/vercel-labs/agent-desktop?ref=explainx).*\n\nThe pitch in the README is plain: \"Computer use for AI agents: read and operate desktop apps through the OS accessibility tree.\" Where most computer-use systems send a screenshot to a vision model and ask it to guess coordinates, agent-desktop asks the OS what is on screen and what can be done with it.\n\n## TL;DR: the questions people ask first\n\n| Question | Answer | \n|---|---|\n| What is it? | A Rust CLI plus C library for reading and operating desktop apps | \n| Who ships it? | Vercel Labs, under `vercel-labs/agent-desktop` | \n| License | Apache-2.0 | \n| Platforms | macOS, Windows, Linux (since 1.0.0 on Oct 27, 2026) | \n| Install | `npm install -g agent-desktop` | \n| How does the agent see the screen? | JSON snapshots of the accessibility tree, each actionable element gets a ref | \n| Does it need a vision model? | No for the native CLI; an optional JEV helper sends screen descriptions to a model provider | \n| Main risk | Snapshots and clipboard reads can contain private desktop content | \n\n## What is agent-desktop, and why does the accessibility tree matter?\n\nEvery major OS exposes a structured description of each window for screen readers: roles such as button, treeitem or window, names, values and the actions available. macOS calls it the Accessibility API, Windows has UI Automation, and Linux uses AT-SPI. agent-desktop treats that tree as \"the source of truth for observation and semantic interaction,\" according to the project's own [CONCEPTS.md glossary](https://github.com/vercel-labs/agent-desktop/blob/main/CONCEPTS.md?ref=explainx).\n\nComputer use agent dragging a tile into a tray, illustrating agent-desktop computer use through element refs\n\nA snapshot is the key primitive. Run `agent-desktop snapshot --app Finder -i` and you get one JSON envelope with a snapshot ID, a count of refs, and a tree of nodes. A node looks like `\"role\": \"treeitem\", \"name\": \"Documents\", \"ref_id\": \"@s8f3k2p9:e8\", \"available_actions\": [\"Click\"]`. The agent then runs `agent-desktop click @s8f3k2p9:e8` and snapshots again to verify the result. The README calls this the loop: observe, act on a ref, observe again, \"with no pixel matching; the loop stays in your agent.\"\n\nThat structure buys three things a screenshot loop struggles with:\n\n- **Determinism.** A button is the same ref across calls until the UI changes, so a click does not drift by a few pixels.\n- **Cheap observation.** A text tree is far smaller to feed back into a model than a full-resolution image, and the tool offers`--skeleton` ,`--compact` and`--root` flags to map a dense app like Slack first and then drill into one region.\n- **Honest failure.** Refs belong to their snapshot. A bare`@e8` is rejected with`INVALID_ARGS` , and when the UI changes an action returns`STALE_REF` or`AMBIGUOUS_TARGET` , so the agent re-snapshots instead of clicking the wrong row.\n\n## How does it work under the hood?\n\nThe project is more careful than a weekend wrapper. The README says that on macOS, CLI calls reuse a private command host per snapshot namespace and binary build, which retains native accessibility objects \"so a removed row cannot silently resolve to a surviving row with the same identifier.\" The host exits after 15 idle minutes, and `TMPDIR` must be owned by the current user with mode 0700.\n\nThe concepts glossary shows how much edge-case work sits behind that. It defines a **Partial Observation**, a snapshot that ran out of its time budget and returns what it saw, with completeness reported on the whole result and on each cut node. It defines an **Evidence Tri-State**, in which every property is Known, Absent or Unknown, so a failed read can never be mistaken for a negative answer. It also describes how Chromium and Electron apps wrap content in chains of anonymous containers, and how the walker skips those wrappers so dense web apps fit the default depth budget instead of understating their reachable content \"by an order of magnitude.\"\n\nActions are headless by default. The README says ref actions go through the accessibility API \"without moving the pointer or taking focus,\" and the global `--headed` flag allows physical input. Four commands, `key-down`, `key-up`, `mouse-down` and `mouse-up`, refuse to run until a daemon can release a held key or button, a small but telling safety choice.\n\n## What can you do with its 60 commands?\n\nThe README lists 60 command names, 56 of them operational. They fall into groups:\n\n| Group | Examples | \n|---|---|\n| Observe | `snapshot` ,`find` ,`get` ,`is` ,`screenshot` ,`list-surfaces` | \n| Act on a ref | `click` ,`double-click` ,`type` ,`set-value` ,`select` ,`toggle` ,`expand` ,`scroll` | \n| Keyboard and mouse | `press` ,`hover` ,`drag` ,`mouse-move` ,`mouse-wheel` | \n| Apps and windows | `launch` ,`close-app` ,`list-windows` ,`focus-window` ,`resize-window` ,`minimize` | \n| Notifications | `list-notifications` ,`dismiss-notification` ,`notification-action` | \n| Clipboard | `clipboard-get` ,`clipboard-set` ,`clipboard-clear` | \n| Control | `wait` ,`batch` ,`session` ,`trace` ,`permissions` ,`skills` | \n\nEvery command prints one JSON envelope with `ok` and either `data` or `error`. An error carries a code, a disposition and usually a suggestion, and the README says to retry \"only when `error.disposition.retry` is safe.\" That contract, documented in the repo's `docs/json-output.md`, is what lets an agent recover without a human reading logs.\n\nThere is also an FFI library. Each release ships `libagent_desktop_ffi` for macOS, Linux and Windows so a program can call the tool in process rather than spawning the CLI for every command.\n\n## How do I try it in Codex, Claude Code or Cursor?\n\nTerminal prompt with a block cursor, the starting point for installing the agent-desktop CLI\n\nInstall the CLI, then verify it:\n\n```\nnpm install -g agent-desktop\nagent-desktop version\nagent-desktop permissions --guide   # macOS: grant Accessibility\nagent-desktop list-apps\nagent-desktop snapshot --app Finder -i\n```\n\nBun, Yarn and pnpm install the same package, which includes native binaries for x64 and ARM64. The README says official Homebrew and Scoop packages are still being submitted, and that the repository no longer maintains its own tap or bucket. On Windows, no grant is needed for apps at your terminal's integrity level; on Linux you need your graphical session with `at-spi2-core`.\n\nTo teach an agent harness the tool, the project ships its own instructions inside the binary. Run `agent-desktop skills get desktop` for the loop, refs, verification and retry rules, and `agent-desktop skills get platform` for what differs on your OS. You can install it as a skill with `npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop`, or paste two lines into your `AGENTS.md` telling the agent to read those docs before first use. The README's own demo is a clip titled \"Codex.using.agent-desktop\", which shows the intended pairing with a coding agent. For how skills work in general, see our [agent skills guide](https://explainx.ai/blog/what-are-agent-skills-complete-guide).\n\n## How does it compare with Cursor, Cua and other computer-use tools?\n\nThe timing is not an accident. Computer use is becoming a standard coding-agent capability. [Cursor added a Computer Use tool on October 23](https://explainx.ai/blog/cursor-computer-use-desktop-apps-ios-simulator-local-agents-2026) so its agents can click through the iOS Simulator, [Cua Spaces](https://explainx.ai/blog/cua-spaces-cross-computer-ai-agents-2026) connects agents across several computers, and open-model work such as [Alibaba's OpenGCUA stack](https://explainx.ai/blog/alibaba-opengcua-open-stack-generalist-computer-use-agents-osworld-v2-2026) and [Microsoft's Mage-VL](https://explainx.ai/blog/microsoft-mage-vl-cowork-4b-computer-use-model-osworld-66) pushes vision-based agents forward on the OSWorld benchmarks.\n\n| Approach | How it sees the screen | Typical trade-off | \n|---|---|---|\n| agent-desktop | OS accessibility tree, JSON with refs | Precise and cheap, but only as good as the app's accessibility support | \n| Vision computer-use models | Screenshots, coordinates from a model | Works on anything visible, slower and less repeatable | \n| Built-in harness tools (Cursor and similar) | Vendor-specific | Tight integration, tied to one product | \n\nagent-desktop is complementary, not a replacement: it is a tool any harness can call, while the models in the other rows decide what to do. The honest limit of the accessibility approach is apps that expose a poor tree, such as canvas-heavy or game-like interfaces, where a vision fallback still helps. The README's mention of an optional JEV helper, which sends screen descriptions to a configured model provider, hints at that hybrid, though the README is explicit that it is separate from the native CLI.\n\n## What are the risks and limits?\n\nGiving an agent the whole desktop is a bigger grant than giving it a repo. The README is direct about it: \"Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content. Keep them out of public logs and uploads.\" Anything the agent snapshots, from a password manager window to a chat client, enters the model's context and possibly your transcripts.\n\nPractical guardrails:\n\n- Run agents under a separate OS user, VM or spare machine for anything beyond a demo.\n- Close sensitive apps before a session and avoid `clipboard-get` on shared sessions.\n- Prefer the default headless mode, and use `--headed` only when an app refuses accessibility actions.\n- Log commands, not snapshots, and review traces before sharing them.\n- Add approval gates for destructive actions. [AgentBeam](https://agentbeam.com/?ref=explainx) , the agent security platform from the explainx.ai team, is built to stop agents before they take dangerous actions.\n\nPlatform gaps matter too. The README's support table shows Linux mouse input only on X11 with `--headed`, screenshots on X11 windows or via the Wayland portal, and notifications only on GNOME Shell. On macOS, shell surfaces opened through `open-system-surface` are marked \"Planned,\" while Windows already has them. And macOS needs a one-time Accessibility grant; the 1.0.2 changelog notes that `permissions --guide` now offers the terminal that launched the tool rather than the binary itself, and that the command host restarts after a missing permission so commands succeed once you grant access.\n\n## What changed in the first week of releases?\n\nThe [CHANGELOG](https://github.com/vercel-labs/agent-desktop/blob/main/CHANGELOG.md?ref=explainx) shows a fast cadence. Version 1.0.0 on October 27 released the integrated macOS, Windows and Linux CLI and C library together, bundled native executables into npm installs \"including installations with lifecycle scripts disabled,\" and added release verification of archives, checksums and build provenance. Version 1.0.1 the same week fixed a stale version line in the skills and automated npm and GitHub releases. Version 1.0.2 on October 30 added a one-time stderr notice when npm blocks install scripts and the Node launcher slows each call, and pointed the package at the `vercel-labs` repository.\n\nEarlier listings of the project credit an individual developer's account, and the repo now sits in Vercel's labs organization, which is why you may see both names in older posts. We did not find a Vercel announcement post or a benchmark in the repository beyond a `benchmarks/locator-resolution` folder, so treat any speed or accuracy claims you see elsewhere as unverified until the project publishes numbers.\n\n## What this means for people who build with agents\n\nIf you run coding agents that need to test a real app, drive a native installer, or check a UI change, a structured computer-use CLI is a lower-cost, more repeatable path than screenshots. The loop is simple enough to script, the JSON contract is explicit about retries, and the FFI library means you can embed it in your own harness. The cost is that you now own a permission and privacy surface: accessibility access is among the most powerful grants on a desktop.\n\nA reasonable first test is read-only. Install it, run `list-apps` and one `snapshot` on a harmless app, read what comes back, and only then let an agent call `click`. Our [MCP guide](https://explainx.ai/blog/what-is-mcp-model-context-protocol-guide) explains how to wrap tools like this behind a protocol if your harness prefers MCP over shell calls. And for the broader debate about agents with OS-wide access, the [Vercel KVM sandbox escape write-up](https://explainx.ai/blog/vercel-kvm-zero-day-vm-escape-agent-sandboxes-2026) is a useful reminder that where an agent runs matters as much as what it can click.\n\n## Related reading\n\n*Version numbers, star counts and platform support reflect the repository as of October 31, 2026 and may change.*", "url": "https://wpnews.pro/news/agent-desktop-vercel-labs-computer-use-cli-reads-the-accessibility-tree-not", "canonical_source": "https://explainx.ai/blog/vercel-agent-desktop-accessibility-tree-computer-use-cli-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-10-31 00:00:00+00:00", "updated_at": "2026-10-31 04:46:46.732319+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["Vercel Labs", "agent-desktop", "GitHub", "macOS Accessibility API", "Windows UI Automation", "Linux AT-SPI", "Finder", "Slack"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/agent-desktop-vercel-labs-computer-use-cli-reads-the-accessibility-tree-not", "markdown": "https://wpnews.pro/news/agent-desktop-vercel-labs-computer-use-cli-reads-the-accessibility-tree-not.md", "text": "https://wpnews.pro/news/agent-desktop-vercel-labs-computer-use-cli-reads-the-accessibility-tree-not.txt", "jsonld": "https://wpnews.pro/news/agent-desktop-vercel-labs-computer-use-cli-reads-the-accessibility-tree-not.jsonld"}}