Let an AI agent click any desktop app by naming a grid cell — not guessing pixels.
A cross-platform Rust CLI for GUI automation: screenshots, window management, mouse and keyboard control, strict JSON in / JSON out. Zero dependencies — no crates, one small binary, direct OS APIs.
Real gui-tool output — orient to a labeled grid → zoom into a cell → click the crosshair → verify.
Built for AI desktop agents (Claude Code, Codex, Gemini CLI, and friends), but it works fine as a general-purpose GUI automation tool.
Why it's different: agents are bad at guessing pixel coordinates from a screenshot, and every other automation tool makes them try. gui-tool
removes pixels entirely — it overlays a labeled grid with a crosshair at each cell's center, the agent names a cell, and the click lands exactly on that crosshair. Need more precision? Zoom into the cell for a sub-grid and name a sub-cell (C7.F3
). It's the whole workflow in one line: orient → zoom → click → verify.
It sees pixels, not an accessibility tree. Most desktop-automation tools for agents find elements by reading the OS accessibility tree — which only helps when that tree exists, is complete, and is correct. gui-tool
never touches it. It works from the actual rendered screen, so it drives what tree-based tools can't see: games, <canvas>
/ WebGL apps, Flutter and other custom-drawn UIs, remote desktops and VNC, video, and any app with missing, mislabeled, or wrong accessibility data. If a human can see it, gui-tool can click it.
Everything is hand-rolled against raw OS APIs — its own PNG encoder, D-Bus client, JSON output, and DEFLATE — so there are no crates to audit, no build surprises, and it runs natively on GNOME/Wayland where xdotool
and pyautogui
give up.
End-to-end in a real browser — navigate, click a field by cell, type, and submit, no pixel coordinates anywhere.
Concrete things it's been used for:
Automating apps with no CLI or API— drive GUI-only software (legacy tools, proprietary apps, installers, settings panels) the same way a person would, so an agent can script a workflow that otherwise has no headless path.Debugging GUI applications— let an agent open the app, reproduce a bug by clicking through it, screenshot each state, and read what's actually on screen — hands and eyes for the parts a log file doesn't capture.Reaching what the accessibility tree can't— pixel-precise grid zoom clicks targets in apps whose accessibility data is missing, incomplete, or wrong: games,<canvas>
/WebGL, Flutter and other custom-drawn UIs, remote desktops, and video.
Grid targeting: Overlay a labeled grid on screenshots with red crosshairs at each cell center. Click by cell label — no pixel coordinates. Supports recursive zoom (B2.C1
) and between-cell targeting (D3+E3
).No accessibility tree required: Targets purely from what's on screen — never AT-SPI, UI Automation, or the AX APIs. Works where the accessibility tree is missing, incomplete, or wrong: games,<canvas>
/WebGL, custom-drawn UIs (Flutter, etc.), and remote desktops.Contextual zoom: Zoomed views show the target cell with a coarser sub-grid, surrounded by dimmed context from adjacent cells with parent-level labels for spatial orientation.No dependencies: Pure std Rust, direct FFI to OS APIs (CoreGraphics, user32.dll, D-Bus). Compiles to a single small binary.Wayland support: Works natively on GNOME/Wayland via XDG Desktop Portals and thewindow-calls
extension, where tools likexdotool
andpyautogui
break.JSON output: Every command returns structured JSON, so agents don't have to parse text output.
The full workflow — orient → zoom → zoom → ... → click → verify — in commands:
gui-tool screenshot --window-id 123 --grid --output /tmp/grid.png
gui-tool screenshot --window-id 123 --grid --cell B2 --output /tmp/zoom.png
gui-tool screenshot --window-id 123 --grid --cell B2.C1 --output /tmp/zoom2.png
gui-tool mouse click --cell B2.C1 --window-id 123
gui-tool mouse click --cell D3+E3 --window-id 123
gui-tool screenshot --output /tmp/screen.png
gui-tool screenshot --window "Firefox" --output /tmp/firefox.png
gui-tool screenshot --window-id 2045481940 --output /tmp/app.png
gui-tool windows list
gui-tool windows raise 1234567890
gui-tool mouse click
gui-tool mouse click --button right
gui-tool mouse click --cell B2 --window-id 2045481940
gui-tool mouse click --cell D3+E3 --window-id 2045481940
All targeting uses --cell
with grid references. There are no pixel coordinate commands — zoom the grid until a crosshair is on the target, then click.
gui-tool key type "hello world"
gui-tool key press "ctrl+a"
gui-tool key press "alt+f4"
gui-tool key press "super"
gui-tool key press "ctrl+shift+t"
gui-tool key type "hello" --window "Terminal"
gui-tool key press "ctrl+a" --window-id 2045481940
A skill definition following the Agent Skills standard is included in skills/gui-tool/SKILL.md
.
1. Add gui-tool to your PATH (after building):
sudo ln -s $(pwd)/target/release/gui-tool /usr/local/bin/gui-tool
ln -s $(pwd)/target/release/gui-tool ~/.local/bin/gui-tool
2. Install the skill:
mkdir -p ~/.claude/skills/gui-tool
cp skills/gui-tool/SKILL.md ~/.claude/skills/gui-tool/SKILL.md
mkdir -p ~/.codex/skills/gui-tool
cp skills/gui-tool/SKILL.md ~/.codex/skills/gui-tool/SKILL.md
Pick the easiest that fits. On Linux and macOS there's a one-time platform-setup step after you get the binary (input permissions and the GNOME window-calls
extension on Linux; Accessibility + Screen Recording permissions on macOS) — see Platform Requirements. Windows needs nothing.
Prebuilt binary (no Rust toolchain) — grab your platform's archive from the latest release, then:
tar xzf gui-tool-*.tar.gz # or unzip on Windows
sudo install gui-tool-*/gui-tool /usr/local/bin/ # or: mv … ~/.local/bin/
From crates.io (needs the Rust toolchain):
cargo install gui-tool
From source:
git clone https://github.com/ZachRouan/agent-desktop-interface
cd agent-desktop-interface
./setup.sh # detects your OS, does platform setup, and builds
After a prebuilt or cargo install on Linux/macOS, run the platform setup without rebuilding — either follow the manual steps in
Platform Requirements, or from a clone:
./setup.sh --skip-build
| Platform | Version | Setup |
|---|---|---|
| Linux | ||
| GNOME/Wayland | input group + udev rule + |
|
setup.sh ) |
||
| macOS | ||
| 10.15+ | Grant Accessibility + Screen Recording permissions in System Settings | |
| Windows | ||
| 8+ | None (cargo build --release in MSYS2, Git Bash, or PowerShell) |
The macOS
Accessibilitypermission grants the right tosynthesizemouse/keyboard input (postCGEvent
s) — it isnotused to read the accessibility tree.gui-tool
never inspects the tree on any platform.
~9,100 lines of Rust, no external crates. Each platform uses direct OS APIs:
Linux:/dev/uinput
for input via ioctl syscalls. Full D-Bus wire protocol implementation (SASL auth, message framing, type marshalling) for XDG Desktop Portal screenshots and GNOMEwindow-calls
window management.macOS: CoreGraphics FFI (CGEventCreateMouseEvent
,CGEventCreateKeyboardEvent
) for input.CGWindowListCreateImage
for screenshots. Objective-C runtime bindings for window activation.Windows:user32.dll
(SendInput
,EnumWindows
,SetForegroundWindow
,VkKeyScanW
) andgdi32.dll
(BitBlt
,GetDIBits
) for input, window management, and screenshots.
macOS window raise activates the owning application, not necessarily the specific window — if an app has multiple windows, the one that ends up frontmost may not be the one you targeted.macOS screenshots useCGWindowListCreateImage
, a CoreGraphics API deprecated in macOS 14. It still works today; a ScreenCaptureKit-based backend is future work.Linux desktop-size detection prefers GNOME Mutter'sDisplayConfig
D-Bus interface and falls back to a DRM-sysfs heuristic on non-GNOME/non-Mutter sessions.
MIT