{"slug": "show-hn-gui-tool-zero-dependency-rust-cli-for-desktop-gui-automation", "title": "Show HN: GUI-tool – zero-dependency Rust CLI for desktop GUI automation", "summary": "A developer released gui-tool, a zero-dependency Rust CLI for desktop GUI automation that uses a labeled grid overlay to let AI agents click on any screen element by naming a cell, bypassing pixel coordinates and accessibility trees. The tool supports cross-platform window management, mouse and keyboard control, and works natively on GNOME/Wayland, enabling automation of apps with no CLI or API, including games, WebGL, and custom-drawn UIs.", "body_md": "**Let an AI agent click any desktop app by naming a grid cell — not guessing pixels.**\n\nA cross-platform Rust CLI for GUI automation: screenshots, window management, mouse and keyboard control, strict JSON in / JSON out. **Zero dependencies** — no crates, one small binary, direct OS APIs.\n\nReal gui-tool output — orient to a labeled grid → zoom into a cell → click the crosshair → verify.\n\nBuilt for AI desktop agents (Claude Code, Codex, Gemini CLI, and friends), but it works fine as a general-purpose GUI automation tool.\n\n**Why it's different:** agents are bad at guessing pixel coordinates from a screenshot, and every other automation tool makes them try. `gui-tool`\n\nremoves pixels entirely — it overlays a **labeled grid** with a crosshair at each cell's center, the agent names a cell, and the click lands exactly on that crosshair. Need more precision? Zoom into the cell for a sub-grid and name a sub-cell (`C7.F3`\n\n). It's the whole workflow in one line: **orient → zoom → click → verify.**\n\n**It sees pixels, not an accessibility tree.** Most desktop-automation tools for agents find elements by reading the OS accessibility tree — which only helps when that tree *exists*, is *complete*, and is *correct*. `gui-tool`\n\nnever touches it. It works from the actual rendered screen, so it drives what tree-based tools can't see: games, `<canvas>`\n\n/ WebGL apps, Flutter and other custom-drawn UIs, remote desktops and VNC, video, and any app with missing, mislabeled, or wrong accessibility data. **If a human can see it, gui-tool can click it.**\n\nEverything is hand-rolled against raw OS APIs — its own PNG encoder, D-Bus client, JSON output, and DEFLATE — so there are no crates to audit, no build surprises, and it runs natively on **GNOME/Wayland** where `xdotool`\n\nand `pyautogui`\n\ngive up.\n\nEnd-to-end in a real browser — navigate, click a field by cell, type, and submit, no pixel coordinates anywhere.\n\nConcrete things it's been used for:\n\n**Automating apps with no CLI or API**— drive GUI-only software (legacy tools, proprietary apps, installers, settings panels) the same way a person would, so an agent can script a workflow that otherwise has no headless path.**Debugging GUI applications**— let an agent open the app, reproduce a bug by clicking through it, screenshot each state, and read what's actually on screen — hands and eyes for the parts a log file doesn't capture.**Reaching what the accessibility tree can't**— pixel-precise grid zoom clicks targets in apps whose accessibility data is missing, incomplete, or wrong: games,`<canvas>`\n\n/WebGL, Flutter and other custom-drawn UIs, remote desktops, and video.\n\n**Grid targeting:** Overlay a labeled grid on screenshots with red crosshairs at each cell center. Click by cell label — no pixel coordinates. Supports recursive zoom (`B2.C1`\n\n) and between-cell targeting (`D3+E3`\n\n).**No accessibility tree required:** Targets purely from what's on screen — never AT-SPI, UI Automation, or the AX APIs. Works where the accessibility tree is missing, incomplete, or wrong: games,`<canvas>`\n\n/WebGL, custom-drawn UIs (Flutter, etc.), and remote desktops.**Contextual zoom:** Zoomed views show the target cell with a coarser sub-grid, surrounded by dimmed context from adjacent cells with parent-level labels for spatial orientation.**No dependencies:** Pure std Rust, direct FFI to OS APIs (CoreGraphics, user32.dll, D-Bus). Compiles to a single small binary.**Wayland support:** Works natively on GNOME/Wayland via XDG Desktop Portals and the`window-calls`\n\nextension, where tools like`xdotool`\n\nand`pyautogui`\n\nbreak.**JSON output:** Every command returns structured JSON, so agents don't have to parse text output.\n\nThe full workflow — **orient → zoom → zoom → ... → click → verify** — in commands:\n\n```\n# Screenshot with labeled grid overlay (auto-scales: 16x9 for full screen)\ngui-tool screenshot --window-id 123 --grid --output /tmp/grid.png\n\n# Zoom into a cell — shows sub-grid with dimmed context from neighbors\ngui-tool screenshot --window-id 123 --grid --cell B2 --output /tmp/zoom.png\n\n# Recursive zoom for precision\ngui-tool screenshot --window-id 123 --grid --cell B2.C1 --output /tmp/zoom2.png\n\n# Click at a cell center (moves + clicks in one step)\ngui-tool mouse click --cell B2.C1 --window-id 123\n\n# Target straddles two cells? Zoom/click centered on the boundary\ngui-tool mouse click --cell D3+E3 --window-id 123\n# Full screen\ngui-tool screenshot --output /tmp/screen.png\n\n# Cropped to a specific window\ngui-tool screenshot --window \"Firefox\" --output /tmp/firefox.png\n\n# Screenshot by window ID (cropped)\ngui-tool screenshot --window-id 2045481940 --output /tmp/app.png\n# List all windows (returns JSON array of IDs, titles, PIDs, and bounds)\ngui-tool windows list\n\n# Bring a window to front by ID\ngui-tool windows raise 1234567890\n# Click at current position\ngui-tool mouse click\ngui-tool mouse click --button right\n\n# Click at a grid cell center (moves + clicks in one step)\ngui-tool mouse click --cell B2 --window-id 2045481940\n\n# Between-cell click (centered on boundary)\ngui-tool mouse click --cell D3+E3 --window-id 2045481940\n```\n\nAll targeting uses `--cell`\n\nwith grid references. There are no pixel coordinate commands — zoom the grid until a crosshair is on the target, then click.\n\n```\n# Type text into focused window\ngui-tool key type \"hello world\"\n\n# Press key combinations\ngui-tool key press \"ctrl+a\"\ngui-tool key press \"alt+f4\"\ngui-tool key press \"super\"\ngui-tool key press \"ctrl+shift+t\"\n\n# Type into a specific window\ngui-tool key type \"hello\" --window \"Terminal\"\ngui-tool key press \"ctrl+a\" --window-id 2045481940\n```\n\nA skill definition following the [Agent Skills](https://agentskills.io) standard is included in `skills/gui-tool/SKILL.md`\n\n.\n\n**1. Add gui-tool to your PATH** (after building):\n\n```\n# Linux/macOS\nsudo ln -s $(pwd)/target/release/gui-tool /usr/local/bin/gui-tool\n\n# Or without sudo\nln -s $(pwd)/target/release/gui-tool ~/.local/bin/gui-tool\n```\n\n**2. Install the skill:**\n\n```\n# Claude Code\nmkdir -p ~/.claude/skills/gui-tool\ncp skills/gui-tool/SKILL.md ~/.claude/skills/gui-tool/SKILL.md\n\n# Codex\nmkdir -p ~/.codex/skills/gui-tool\ncp skills/gui-tool/SKILL.md ~/.codex/skills/gui-tool/SKILL.md\n```\n\nPick the easiest that fits. On **Linux and macOS** there's a one-time platform-setup step after you get the binary (input permissions and the GNOME `window-calls`\n\nextension on Linux; Accessibility + Screen Recording permissions on macOS) — see [Platform Requirements](#platform-requirements). Windows needs nothing.\n\n**Prebuilt binary** (no Rust toolchain) — grab your platform's archive from the [latest release](https://github.com/ZachRouan/agent-desktop-interface/releases/latest), then:\n\n```\ntar xzf gui-tool-*.tar.gz                      # or unzip on Windows\nsudo install gui-tool-*/gui-tool /usr/local/bin/   # or: mv … ~/.local/bin/\n```\n\n**From crates.io** (needs the [Rust toolchain](https://rustup.rs/)):\n\n```\ncargo install gui-tool\n```\n\n**From source:**\n\n```\ngit clone https://github.com/ZachRouan/agent-desktop-interface\ncd agent-desktop-interface\n./setup.sh          # detects your OS, does platform setup, and builds\n```\n\nAfter a **prebuilt or cargo install** on Linux/macOS, run the platform setup without rebuilding — either follow the manual steps in\n\n[Platform Requirements](#platform-requirements), or from a clone:\n\n```\n./setup.sh --skip-build\n```\n\n| Platform | Version | Setup |\n|---|---|---|\nLinux |\nGNOME/Wayland | `input` group + udev rule +\n`setup.sh` ) |\nmacOS |\n10.15+ | Grant Accessibility + Screen Recording permissions in System Settings |\nWindows |\n8+ | None (`cargo build --release` in MSYS2, Git Bash, or PowerShell) |\n\nThe macOS\n\nAccessibilitypermission grants the right tosynthesizemouse/keyboard input (post`CGEvent`\n\ns) — it isnotused to read the accessibility tree.`gui-tool`\n\nnever inspects the tree on any platform.\n\n~9,100 lines of Rust, no external crates. Each platform uses direct OS APIs:\n\n**Linux:**`/dev/uinput`\n\nfor input via ioctl syscalls. Full D-Bus wire protocol implementation (SASL auth, message framing, type marshalling) for XDG Desktop Portal screenshots and GNOME`window-calls`\n\nwindow management.**macOS:** CoreGraphics FFI (`CGEventCreateMouseEvent`\n\n,`CGEventCreateKeyboardEvent`\n\n) for input.`CGWindowListCreateImage`\n\nfor screenshots. Objective-C runtime bindings for window activation.**Windows:**`user32.dll`\n\n(`SendInput`\n\n,`EnumWindows`\n\n,`SetForegroundWindow`\n\n,`VkKeyScanW`\n\n) and`gdi32.dll`\n\n(`BitBlt`\n\n,`GetDIBits`\n\n) for input, window management, and screenshots.\n\n**macOS window raise** activates the owning application, not necessarily the specific window — if an app has multiple windows, the one that ends up frontmost may not be the one you targeted.**macOS screenshots** use`CGWindowListCreateImage`\n\n, a CoreGraphics API deprecated in macOS 14. It still works today; a ScreenCaptureKit-based backend is future work.**Linux desktop-size detection** prefers GNOME Mutter's`DisplayConfig`\n\nD-Bus interface and falls back to a DRM-sysfs heuristic on non-GNOME/non-Mutter sessions.\n\nMIT", "url": "https://wpnews.pro/news/show-hn-gui-tool-zero-dependency-rust-cli-for-desktop-gui-automation", "canonical_source": "https://github.com/ZachRouan/agent-desktop-interface", "published_at": "2026-07-08 12:50:14+00:00", "updated_at": "2026-07-08 13:00:51.775643+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "computer-vision"], "entities": ["gui-tool", "Claude Code", "Codex", "Gemini CLI", "GNOME", "Wayland", "Rust"], "alternates": {"html": "https://wpnews.pro/news/show-hn-gui-tool-zero-dependency-rust-cli-for-desktop-gui-automation", "markdown": "https://wpnews.pro/news/show-hn-gui-tool-zero-dependency-rust-cli-for-desktop-gui-automation.md", "text": "https://wpnews.pro/news/show-hn-gui-tool-zero-dependency-rust-cli-for-desktop-gui-automation.txt", "jsonld": "https://wpnews.pro/news/show-hn-gui-tool-zero-dependency-rust-cli-for-desktop-gui-automation.jsonld"}}