{"slug": "show-hn-hands-rust-mcp-cli-that-sees-the-windows-desktop-and-clicks-real-chrome", "title": "Show HN: Hands-Rust MCP/CLI that sees the Windows desktop and clicks real Chrome", "summary": "Ryan-AI-Studios released Hands, an open-source Rust MCP/CLI that lets coding agents control a real Windows desktop and Chrome browser via OS-level input, avoiding automation flags. The tool, available on GitHub under MIT license, supports tools like observe, click, type, and scroll, and is designed for personal research tasks, but it is not a sandbox and has best-effort safeguards against unintended actions.", "body_md": "I built Hands because I wanted a coding agent to use this Windows PC and a real Chrome profile the way I do: look at the screen, move the real mouse, type, click , without turning Chrome into an automation browser.\n\nIt is a Rust MCP/CLI. A harness (Grok, Codex, Claude Code, OpenCode, etc.) calls tools like observe, click, type, scroll. Observe is a screenshot path plus a small element list (UIA + optional Chrome DOM ids). Click is OS SendInput on a Bézier path, not a Chrome DevTools click.\n\nThere is no Playwright, no Puppeteer, no remote debugging port. Daily Chrome is launched with no extra flags, or attached if it’s already open. Sites that key on CDP/automation flags mostly don’t see that. They can still see injected input (LLMHF_INJECTED).\n\nA tiny unpacked Chrome extension can fuse page structure (chr: ids, listing cards) so the model isn’t guessing from pixels. Sideload is manual. Fusion dies if the service worker goes inactive; reload the card.\n\nWhat it is good for: personal research on your own desk. “Find a Camry on cars.com,” read a page, fill a ZIP, dismiss a cookie banner.\n\nWhat it is not: • Not a sandbox. It can click whatever is on screen, including checkout and Easy Apply. • Confirm-before-money is best-effort classification in the binary, not a guarantee. Prompt injection from the screenshot/DOM is real; the binary treats that text as untrusted, the model might not. • Not a CAPTCHA solver on daily Chrome. Two visible tries, then it yields and waits for the puzzle to go away. • Windows only. • Install is: build the exe, register a native-messaging host, sideload the extension, point an MCP client at hands mcp. README is the runbook. Missing an API key does not fail the build; do_task is optional. • Logs live under %LOCALAPPDATA%\\hands\\logs\\. The extension asks for so it can map the tab you’re looking at.\n\nRepo: [https://github.com/Ryan-AI-Studios/hands](https://github.com/Ryan-AI-Studios/hands) (MIT)\n\nHappy to answer how observe/fusion/the fence work. If you try it, Pause/Break is the kill switch.\n\nComments URL: [https://news.ycombinator.com/item?id=49405405](https://news.ycombinator.com/item?id=49405405)\n\nPoints: 1\n\n# Comments: 0", "url": "https://wpnews.pro/news/show-hn-hands-rust-mcp-cli-that-sees-the-windows-desktop-and-clicks-real-chrome", "canonical_source": "https://news.ycombinator.com/item?id=49405405", "published_at": "2026-08-23 01:16:43+00:00", "updated_at": "2026-08-23 01:43:11.719556+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools"], "entities": ["Ryan-AI-Studios", "Hands", "GitHub", "Chrome", "Windows"], "alternates": {"html": "https://wpnews.pro/news/show-hn-hands-rust-mcp-cli-that-sees-the-windows-desktop-and-clicks-real-chrome", "markdown": "https://wpnews.pro/news/show-hn-hands-rust-mcp-cli-that-sees-the-windows-desktop-and-clicks-real-chrome.md", "text": "https://wpnews.pro/news/show-hn-hands-rust-mcp-cli-that-sees-the-windows-desktop-and-clicks-real-chrome.txt", "jsonld": "https://wpnews.pro/news/show-hn-hands-rust-mcp-cli-that-sees-the-windows-desktop-and-clicks-real-chrome.jsonld"}}