agent-desktop reads the accessibility tree of a desktop app and returns it as JSON. Every element an agent can act on gets a ref. The agent acts on a ref, reads the UI again to check the result, and repeats. It ships as a CLI and as a C library for macOS, Windows, and Linux.
What agent-desktop is and is not #
- A tool that agents call. It is not an agent. The observe, act, and verify loop lives in the calling agent or script.
- No model inside. It never embeds an LLM, has no GUI, and has no interactive prompt. Every command prints one JSON envelope and exits.
- Desktop apps only. It does not automate web pages; useagent-browser for those. For the web contents of a Chromium or Electron app you launch, agent-desktop opens a DevTools port that agent-browser connects to. SeeChromium and Electron apps .
- Limited to what apps expose. It reads accessibility trees and does not match pixels, so it cannot see a UI that publishes no accessibility tree.
The loop in four commands #
An agent maps an app, drills into the region it needs, acts on a ref, and checks the result.
Refs carry the id of the snapshot that produced them (@s2txcl95uvubmn:e11). Snapshots and refs defines them, and Skeleton and drill-down explains the first two commands. A successful action does not prove the UI changed, so the loop ends with a read. The quick start runs these commands with their output.
How it works #
The same chain in text, from the caller to the app:
- The calling agent: your code or an LLM loop. It runs a command and reads the JSON envelope it prints.
- The
agent-desktopCLI, or theC library when you call it in process. agent-desktop-core: the commands,RefStore, and snapshot building. Core defines thePlatformAdaptertrait and every shared type.PlatformAdapter, the only way core reaches an operating system.- One platform crate:
agent-desktop-macos,agent-desktop-windows, oragent-desktop-linux. - The OS accessibility API: the AX API on macOS, UI Automation on Windows, AT-SPI on Linux.
- The app, which exposes its UI through that API.
Core never imports a platform crate, and platform crates never import each other. Only the CLI and the C library connect a platform crate to core, and each picks it at compile time, so a binary drives the OS it was built for.
Platform support #
| Platform | Requirement | Accessibility API |
|---|---|---|
| macOS | macOS 13 or later, Intel or Apple Silicon. Accessibility permission for the app that runs agent-desktop. | AX API |
| Windows | Windows 10 1809 or later, or Windows Server 2019 or later, on x64 or ARM64, in the signed-in user's interactive desktop. | UI Automation |
| Linux | Ubuntu 24.04 or later on x86_64 or aarch64 (glibc 2.35 floor), in the user's graphical session with a running AT-SPI bus. | AT-SPI |
The CLI has the same 60 command names on every platform. Four of them (key-down, key-up, mouse-down, mouse-up) fail closed everywhere. Each platform page lists what works headless, what needs --headed, and what refuses.
## [Next steps](#next-steps)
- [Installation](https://agent-browser.dev/desktop/installation) : install the CLI, verify it, and grant permissions.
- [Quick start](https://agent-browser.dev/desktop/quick-start) : one complete run against a real app.
- [Skills](https://agent-browser.dev/desktop/skills) : the guides your agent reads before it acts.
- [Commands](https://agent-browser.dev/desktop/commands) : all 60 commands.