{"slug": "hands-lets-your-llm-agent-control-your-actual-windows-desktop", "title": "Hands lets your LLM agent control your actual Windows desktop", "summary": "Hands, an open-source tool from Ryan-AI-Studios, lets LLM agents control a real Windows desktop via the Model Context Protocol (MCP) and a CLI, using the Windows SendInput API to move the mouse and type into active windows. It combines a screenshot-based Observe Tool with a Fusion Layer from a Chrome extension to interact with known page elements, avoiding browser hijacking and anti-bot flags. The tool requires manual setup, including building from Rust source and sideloading a Chrome extension, and has limitations such as no hard safety guarantee for financial transactions and inability to solve CAPTCHAs.", "body_md": "# Hands lets your LLM agent control your actual Windows desktop\n\n[MCP](/en/tags/mcp/)(Model Context Protocol) and CLI to let agents interact with your OS exactly like a human would.\n\nInstead of trying to hijack a browser via remote debugging ports, Hands uses the Windows `SendInput`\n\nAPI to move the mouse along Bézier paths and type into active windows. It essentially turns your actual, daily Chrome profile—the one with all your logins and cookies—into a controllable environment without needing any special launch flags.\n\n## How the architecture actually works\n\nThe setup relies on a few moving parts to make sure the LLM isn't just \"guessing\" where pixels are. It uses a combination of visual observation and structural data:\n\n**The Observe Tool:** This provides the agent with a screenshot path and a lightweight list of elements. It pulls from UIA (User Interface Automation) and, if you use the extension, the Chrome DOM.**The Fusion Layer:** This is the clever part. A tiny, unpacked Chrome extension maps out the page structure (like specific IDs or listing cards). This way, when the model wants to \"click the third car in the list,\" it isn't just clicking coordinates; it's interacting with a known element.**The Click Mechanism:** Because it uses OS-level input, it avoids the`LLMHF_INJECTED`\n\nflags that many anti-bot systems look for.\n\n## Implementation and Deployment\n\nIf you want to try this as a hands-on guide for your own local AI workflow, you can't just `npm install`\n\nit. It requires a bit of manual setup because it's interacting with your hardware:\n\n1. Build the executable from the Rust source.\n\n2. Register the project as a native-messaging host on your Windows machine.\n\n3. Sideload the provided Chrome extension manually.\n\n4. Point your MCP client (like [Claude Code](/en/tags/claude%20code/) or any other LLM agent harness) to the Hands MCP server.\n\nIf you run into issues, the logs are tucked away in `%LOCALAPPDATA%\\hands\\logs\\`\n\n.\n\n## Real-world limitations to keep in mind\n\nThis isn't a \"set it and forget it\" sandbox. Since it's operating on your actual desktop, there are real risks. If you tell an agent to \"find a cheap flight\" and it accidentally clicks \"Confirm Purchase\" on a high-priced ticket, the tool won't stop it. The \"confirm-before-money\" check is just a best-effort classification within the binary; it’s not a hard safety guarantee.\n\nAlso, don't expect it to be a CAPTCHA-solving god. It can try a couple of visible interactions, but if it hits a puzzle, it will yield and wait for you to handle it manually.\n\nFor those interested in the technical specifics of the `observe`\n\nand `fusion`\n\nlogic, the source is available here:\n\n```\nhttps://github.com/Ryan-AI-Studios/hands\n```\n\nIt's a fascinating look at how we can bridge the gap between high-level LLM reasoning and low-level OS control without the overhead of traditional web automation frameworks.\n\n[Next Coding agents fail most often because they guess too much →](/en/threads/7427/)\n\n[a library of Claude prompt techniques](https://tanyan888.com/), with plenty of directly applicable cases.", "url": "https://wpnews.pro/news/hands-lets-your-llm-agent-control-your-actual-windows-desktop", "canonical_source": "https://promptcube3.com/en/threads/7428/", "published_at": "2026-08-23 16:45:58+00:00", "updated_at": "2026-08-23 17:13:00.043791+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "artificial-intelligence"], "entities": ["Ryan-AI-Studios", "Hands", "Microsoft Windows", "Model Context Protocol", "Chrome", "Claude Code", "SendInput API"], "alternates": {"html": "https://wpnews.pro/news/hands-lets-your-llm-agent-control-your-actual-windows-desktop", "markdown": "https://wpnews.pro/news/hands-lets-your-llm-agent-control-your-actual-windows-desktop.md", "text": "https://wpnews.pro/news/hands-lets-your-llm-agent-control-your-actual-windows-desktop.txt", "jsonld": "https://wpnews.pro/news/hands-lets-your-llm-agent-control-your-actual-windows-desktop.jsonld"}}