{"slug": "vercel-computer-use-cli-for-ai-agents", "title": "Vercel: Computer use CLI for AI agents", "summary": "Vercel released agent-desktop, a CLI and C library that reads the accessibility tree of a desktop app and returns it as JSON so AI agents can act on UI elements by ref, shipping for macOS 13 or later, Windows 10 1809 or later, and Ubuntu 24.04 or later. The tool exposes the same 60 command names on every platform, with four commands (key-down, key-up, mouse-down, mouse-up) failing closed everywhere, and contains no model itself — the observe, act, verify loop lives in the calling agent. It targets desktop apps only, not web pages, and cannot see a UI that publishes no accessibility tree.", "body_md": "# agent-desktop\n\nagent-desktop reads the accessibility tree of a desktop app and returns it as JSON. Every element an agent can act on gets a ref. The agent acts on a ref, reads the UI again to check the result, and repeats. It ships as a CLI and as a C library for macOS, Windows, and Linux.\n\n## [What agent-desktop is and is not](#what-agent-desktop-is-and-is-not)\n\n- **A tool that agents call.** It is not an agent. The observe, act, and verify loop lives in the calling agent or script.\n- **No model inside.** It never embeds an LLM, has no GUI, and has no interactive prompt. Every command prints one JSON envelope and exits.\n- **Desktop apps only.** It does not automate web pages; use[agent-browser](https://agent-browser.dev/) for those. For the web contents of a Chromium or Electron app you launch, agent-desktop opens a DevTools port that agent-browser connects to. See[Chromium and Electron apps](https://agent-browser.dev/desktop/guides/chromium-and-electron) .\n- **Limited to what apps expose.** It reads accessibility trees and does not match pixels, so it cannot see a UI that publishes no accessibility tree.\n\n## [The loop in four commands](#the-loop-in-four-commands)\n\nAn agent maps an app, drills into the region it needs, acts on a ref, and checks the result.\n\nRefs carry the id of the snapshot that produced them (`@s2txcl95uvubmn:e11`). [Snapshots and refs](https://agent-browser.dev/desktop/snapshots-and-refs) defines them, and [Skeleton and drill-down](https://agent-browser.dev/desktop/skeleton-and-drill-down) explains the first two commands. A successful action does not prove the UI changed, so the loop ends with a read. The [quick start](https://agent-browser.dev/desktop/quick-start) runs these commands with their output.\n\n## [How it works](#how-it-works)\n\nThe same chain in text, from the caller to the app:\n\n1. The calling agent: your code or an LLM loop. It runs a command and reads the JSON envelope it prints.\n2. The `agent-desktop` CLI, or the[C library](https://agent-browser.dev/desktop/ffi) when you call it in process.\n3. `agent-desktop-core` : the commands,`RefStore` , and snapshot building. Core defines the`PlatformAdapter` trait and every shared type.\n4. `PlatformAdapter` , the only way core reaches an operating system.\n5. One platform crate: `agent-desktop-macos` ,`agent-desktop-windows` , or`agent-desktop-linux` .\n6. The OS accessibility API: the AX API on macOS, UI Automation on Windows, AT-SPI on Linux.\n7. The app, which exposes its UI through that API.\n\nCore never imports a platform crate, and platform crates never import each other. Only the CLI and the C library connect a platform crate to core, and each picks it at compile time, so a binary drives the OS it was built for.\n\n## [Platform support](#platform-support)\n\n| Platform | Requirement | Accessibility API | \n|---|---|---|\n| [macOS](https://agent-browser.dev/desktop/platforms/macos) | macOS 13 or later, Intel or Apple Silicon. Accessibility permission for the app that runs agent-desktop. | AX API | \n| [Windows](https://agent-browser.dev/desktop/platforms/windows) | Windows 10 1809 or later, or Windows Server 2019 or later, on x64 or ARM64, in the signed-in user's interactive desktop. | UI Automation | \n| [Linux](https://agent-browser.dev/desktop/platforms/linux) | Ubuntu 24.04 or later on x86_64 or aarch64 (glibc 2.35 floor), in the user's graphical session with a running AT-SPI bus. | AT-SPI | \n\nThe CLI has the same 60 command names on every platform. Four of them (`key-down`, `key-up`, `mouse-down`, `mouse-up`) fail closed everywhere. Each platform page lists what works headless, what needs `--headed`, and what refuses.\n\n## [Next steps](#next-steps)\n\n- [Installation](https://agent-browser.dev/desktop/installation) : install the CLI, verify it, and grant permissions.\n- [Quick start](https://agent-browser.dev/desktop/quick-start) : one complete run against a real app.\n- [Skills](https://agent-browser.dev/desktop/skills) : the guides your agent reads before it acts.\n- [Commands](https://agent-browser.dev/desktop/commands) : all 60 commands.", "url": "https://wpnews.pro/news/vercel-computer-use-cli-for-ai-agents", "canonical_source": "https://agent-browser.dev/desktop", "published_at": "2026-11-03 05:26:34+00:00", "updated_at": "2026-11-03 05:51:24.185326+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["Vercel", "agent-desktop", "agent-desktop-core", "agent-browser", "macOS", "Windows", "Linux"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/vercel-computer-use-cli-for-ai-agents", "markdown": "https://wpnews.pro/news/vercel-computer-use-cli-for-ai-agents.md", "text": "https://wpnews.pro/news/vercel-computer-use-cli-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/vercel-computer-use-cli-for-ai-agents.jsonld"}}