Vercel: Computer use CLI for AI agents

Vercel released agent-desktop, a CLI and C library that reads the accessibility tree of a desktop app and returns it as JSON so AI agents can act on UI elements by ref, shipping for macOS 13 or later, Windows 10 1809 or later, and Ubuntu 24.04 or later. The tool exposes the same 60 command names on every platform, with four commands (key-down, key-up, mouse-down, mouse-up) failing closed everywhere, and contains no model itself — the observe, act, verify loop lives in the calling agent. It targets desktop apps only, not web pages, and cannot see a UI that publishes no accessibility tree.

agent-desktop agent-desktop reads the accessibility tree of a desktop app and returns it as JSON. Every element an agent can act on gets a ref. The agent acts on a ref, reads the UI again to check the result, and repeats. It ships as a CLI and as a C library for macOS, Windows, and Linux. What agent-desktop is and is not what-agent-desktop-is-and-is-not - A tool that agents call. It is not an agent. The observe, act, and verify loop lives in the calling agent or script. - No model inside. It never embeds an LLM, has no GUI, and has no interactive prompt. Every command prints one JSON envelope and exits. - Desktop apps only. It does not automate web pages; use agent-browser https://agent-browser.dev/ for those. For the web contents of a Chromium or Electron app you launch, agent-desktop opens a DevTools port that agent-browser connects to. See Chromium and Electron apps https://agent-browser.dev/desktop/guides/chromium-and-electron . - Limited to what apps expose. It reads accessibility trees and does not match pixels, so it cannot see a UI that publishes no accessibility tree. The loop in four commands the-loop-in-four-commands An agent maps an app, drills into the region it needs, acts on a ref, and checks the result. Refs carry the id of the snapshot that produced them @s2txcl95uvubmn:e11 . Snapshots and refs https://agent-browser.dev/desktop/snapshots-and-refs defines them, and Skeleton and drill-down https://agent-browser.dev/desktop/skeleton-and-drill-down explains the first two commands. A successful action does not prove the UI changed, so the loop ends with a read. The quick start https://agent-browser.dev/desktop/quick-start runs these commands with their output. How it works how-it-works The same chain in text, from the caller to the app: 1. The calling agent: your code or an LLM loop. It runs a command and reads the JSON envelope it prints. 2. The agent-desktop CLI, or the C library https://agent-browser.dev/desktop/ffi when you call it in process. 3. agent-desktop-core : the commands, RefStore , and snapshot building. Core defines the PlatformAdapter trait and every shared type. 4. PlatformAdapter , the only way core reaches an operating system. 5. One platform crate: agent-desktop-macos , agent-desktop-windows , or agent-desktop-linux . 6. The OS accessibility API: the AX API on macOS, UI Automation on Windows, AT-SPI on Linux. 7. The app, which exposes its UI through that API. Core never imports a platform crate, and platform crates never import each other. Only the CLI and the C library connect a platform crate to core, and each picks it at compile time, so a binary drives the OS it was built for. Platform support platform-support | Platform | Requirement | Accessibility API | |---|---|---| | macOS https://agent-browser.dev/desktop/platforms/macos | macOS 13 or later, Intel or Apple Silicon. Accessibility permission for the app that runs agent-desktop. | AX API | | Windows https://agent-browser.dev/desktop/platforms/windows | Windows 10 1809 or later, or Windows Server 2019 or later, on x64 or ARM64, in the signed-in user's interactive desktop. | UI Automation | | Linux https://agent-browser.dev/desktop/platforms/linux | Ubuntu 24.04 or later on x86 64 or aarch64 glibc 2.35 floor , in the user's graphical session with a running AT-SPI bus. | AT-SPI | The CLI has the same 60 command names on every platform. Four of them key-down , key-up , mouse-down , mouse-up fail closed everywhere. Each platform page lists what works headless, what needs --headed , and what refuses. Next steps next-steps - Installation https://agent-browser.dev/desktop/installation : install the CLI, verify it, and grant permissions. - Quick start https://agent-browser.dev/desktop/quick-start : one complete run against a real app. - Skills https://agent-browser.dev/desktop/skills : the guides your agent reads before it acts. - Commands https://agent-browser.dev/desktop/commands : all 60 commands.