{"slug": "show-hn-glass-a-see-act-debug-loop-over-any-gui-app-for-agents-mcp", "title": "Show HN: Glass – a see/act/debug loop over any GUI app for agents (MCP)", "summary": "A new open-source tool called Glass gives AI coding agents a closed build-see-interact-debug loop over native GUI applications, enabling them to launch, drive, and verify apps without per-step screenshots. The Rust-based MCP server supports Linux (X11 and Wayland), Windows, macOS, Android (via AVD emulator), and iOS (Simulator) backends, allowing agents to reproduce bugs from the accessibility tree and fix code autonomously. Created by developer fixed-width, Glass is available on GitHub and aims to reduce the need for human feedback during GUI development.", "body_md": "**Give your coding agent hands for the app it's building** — launch it, drive it, and verify the\nresult, without burning a screenshot on every step.\n\nA Rust [MCP](https://modelcontextprotocol.io) server that gives an AI coding agent a closed **build →\nsee → interact → debug** loop over external native GUI applications.\n\nglass lets an agent launch a GUI app, capture what is on screen, inject mouse and keyboard input, read the app's logs, and detect visual changes — so a coding agent can build and debug UI applications independently instead of asking the user \"does this look right?\".\n\nglass drives apps as an external black box, so it works with any native GUI app regardless of toolkit\nor language. It has two Linux backends (**X11** and **Wayland**), a **Windows** backend, an\n**Android** backend (an AVD emulator, driven over `adb`\n\nfrom any host), an **iOS** backend (native\napps in the Simulator over `xcrun simctl`\n\n, with input and the accessibility tree via `idb_companion`\n\n;\nmulti-touch gestures excepted), and a **macOS** backend, behind a platform-agnostic core.\n\nAn agent building a GUI app runs it under glass, reproduces a bug **from the accessibility tree**\n(no screenshots), fixes the code, and re-verifies — the loop it otherwise can't close on its own.\n[Try it yourself](#try-it-in-60-seconds).\n\n-\nDownload glass for your platform from the\n\n[Releases page](https://github.com/fixed-width/glass/releases/latest)and[connect it to your agent](/fixed-width/glass/blob/master/docs/how-to/connect-an-agent.md). -\nGet the example app — clone this repo, or download\n\n(on Linux it needs`examples/tasks_demo.py`\n\n`sudo apt install python3-gi gir1.2-gtk-4.0`\n\n). -\nPaste this to your agent:\n\nUse glass to run\n\n`examples/tasks_demo.py`\n\nwith accessibility on. There's a bug: clicking**Add** doesn't add the typed task. Reproduce it by driving the UI and checking the accessibility tree (don't just screenshot), then find and fix the bug in the code and verify a task actually appears.\n\nYour agent launches the app, reproduces the bug from the accessibility tree, fixes the one-line\nwiring bug, and confirms the task appears — the whole build → see → interact → debug loop, start to\nfinish. (`glass-mcp doctor`\n\nchecks your environment if anything's off.)\n\nPoint an agent at a GUI app and it runs the whole cycle itself. When the app exposes an accessibility\ntree, the agent drives it semantically — addressing widgets by `#id`\n\nand confirming each step from\ntext, no per-step screenshot:\n\n```\nglass_start            { \"run\": [\"python3\", \"app.py\"], \"a11y\": true }   // launch (+ private a11y bus)\nglass_a11y_snapshot                        // the tree: role, name, #id, bounds — as text\nglass_click_element    { \"id\": 5 }         // click by #id, not pixels\nglass_wait_for_element { \"name\": \"Save\", \"condition\": \"enabled\" }       // wait on state — no polling\nglass_set_value        { \"id\": 4, \"value\": \"hello\" }   // set a field / toggle / dropdown\nglass_logs                                 // read the app's stderr\n```\n\nFor a canvas or custom-rendered app with no accessibility tree, drive it by pixels instead —\n`glass_screenshot`\n\n, `glass_click {x,y}`\n\n, and `glass_diff`\n\n, which returns `changed_pct`\n\n+ a `bbox`\n\nas\ntext, so routine checks between screenshots cost no vision tokens. Why the loop is shaped this way:\n[the build → see → interact → debug loop](/fixed-width/glass/blob/master/docs/explanation/the-loop.md).\n\nDownload the latest build for your platform from the\n[Releases page](https://github.com/fixed-width/glass/releases/latest), then set up your host:\n\n**Linux**—[docs/how-to/setup-linux.md](/fixed-width/glass/blob/master/docs/how-to/setup-linux.md)(X11 or Wayland;`Xvfb`\n\n/`sway`\n\n+ bubblewrap)**Windows**—[docs/how-to/setup-windows.md](/fixed-width/glass/blob/master/docs/how-to/setup-windows.md)(a prebuilt`.exe`\n\n+ Sandboxie)**macOS**—[docs/how-to/setup-macos.md](/fixed-width/glass/blob/master/docs/how-to/setup-macos.md)(install the notarized`.dmg`\n\n; no build needed)**Android**—[docs/how-to/setup-android.md](/fixed-width/glass/blob/master/docs/how-to/setup-android.md)(an AVD emulator, from any host)**iOS**—[docs/how-to/setup-ios.md](/fixed-width/glass/blob/master/docs/how-to/setup-ios.md)(the Simulator, macOS host only)\n\nEvery asset is listed in [docs/reference/platforms.md](/fixed-width/glass/blob/master/docs/reference/platforms.md#release-artifacts).\nPrefer to compile, or on an architecture with no published asset? See\n[docs/how-to/build-from-source.md](/fixed-width/glass/blob/master/docs/how-to/build-from-source.md) — it is a single `cargo build`\n\n.\n\nThen [connect glass to your agent](/fixed-width/glass/blob/master/docs/how-to/connect-an-agent.md) and run `glass-mcp doctor`\n\nto check\nthe environment. New here? Follow [the tutorial](/fixed-width/glass/blob/master/docs/tutorial/first-drive.md) for a guaranteed first\nsuccess.\n\nglass needs no app integration and no skill to run, but an agent drives it far more reliably with the\nopen [glass-drive](/fixed-width/glass/blob/master/docs/how-to/drive-glass-well.md) Agent Skill — it stops the agent spending its first\nturns rediscovering the verify-cheaply-then-look loop. **Installing it is the single highest-leverage\nthing you can add** when pointing an agent at glass.\n\n**✓** supported · **◑** partial · **–** not supported · **🚧** planned.\n\n| Capability | Linux (X11 + Wayland) | Windows | Android (AVD) | iOS (Simulator) | macOS |\n|---|---|---|---|---|---|\n| Capture · input · windows · clipboard · logs | ✓ | ✓ | ✓ | ✓ | ✓ |\n| Accessibility (semantic addressing) | ✓ AT-SPI | ✓ UI Automation | ✓ UIAutomator | ✓ idb | ✓ AX |\n| Containment / sandboxing | ✓ bubblewrap | ✓ Sandboxie | ✓ the emulator VM | ✓ the Simulator | ✓ Seatbelt |\n| Display isolation (app off your desktop) | ✓ headless Xvfb / sway | ◑ virtual display · VM tier | ✓ headless emulator | ✓ headless simctl boot | 🚧 |\n\nFull matrix, per-capability detail, and system requirements:\n[docs/reference/platforms.md](/fixed-width/glass/blob/master/docs/reference/platforms.md). Transport is MCP over stdio (default) or\nnetwork HTTP.\n\nThe full docs — tutorial, how-to guides, reference, and explanations — are under\n. See\n\n`docs/`\n\n[for release notes, and](/fixed-width/glass/blob/master/CHANGELOG.md)\n\n`CHANGELOG.md`\n\n[Stability and versioning](/fixed-width/glass/blob/master/docs/reference/stability.md)for what a 1.0 release guarantees.\n\nglass is **open core**, licensed **Apache-2.0** — see [ LICENSE-APACHE](/fixed-width/glass/blob/master/LICENSE-APACHE).", "url": "https://wpnews.pro/news/show-hn-glass-a-see-act-debug-loop-over-any-gui-app-for-agents-mcp", "canonical_source": "https://github.com/fixed-width/glass", "published_at": "2026-07-21 12:49:48+00:00", "updated_at": "2026-07-21 13:13:38.192004+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "artificial-intelligence"], "entities": ["Glass", "MCP", "fixed-width", "GitHub", "Linux", "Windows", "macOS", "Android"], "alternates": {"html": "https://wpnews.pro/news/show-hn-glass-a-see-act-debug-loop-over-any-gui-app-for-agents-mcp", "markdown": "https://wpnews.pro/news/show-hn-glass-a-see-act-debug-loop-over-any-gui-app-for-agents-mcp.md", "text": "https://wpnews.pro/news/show-hn-glass-a-see-act-debug-loop-over-any-gui-app-for-agents-mcp.txt", "jsonld": "https://wpnews.pro/news/show-hn-glass-a-see-act-debug-loop-over-any-gui-app-for-agents-mcp.jsonld"}}