MCP(Model Context Protocol) and CLI to let agents interact with your OS exactly like a human would.
Instead of trying to hijack a browser via remote debugging ports, Hands uses the Windows SendInput
API to move the mouse along Bézier paths and type into active windows. It essentially turns your actual, daily Chrome profile—the one with all your logins and cookies—into a controllable environment without needing any special launch flags.
How the architecture actually works #
The setup relies on a few moving parts to make sure the LLM isn't just "guessing" where pixels are. It uses a combination of visual observation and structural data:
The Observe Tool: This provides the agent with a screenshot path and a lightweight list of elements. It pulls from UIA (User Interface Automation) and, if you use the extension, the Chrome DOM.The Fusion Layer: This is the clever part. A tiny, unpacked Chrome extension maps out the page structure (like specific IDs or listing cards). This way, when the model wants to "click the third car in the list," it isn't just clicking coordinates; it's interacting with a known element.The Click Mechanism: Because it uses OS-level input, it avoids theLLMHF_INJECTED
flags that many anti-bot systems look for.
Implementation and Deployment #
If you want to try this as a hands-on guide for your own local AI workflow, you can't just npm install
it. It requires a bit of manual setup because it's interacting with your hardware:
-
Build the executable from the Rust source.
-
Register the project as a native-messaging host on your Windows machine.
-
Sideload the provided Chrome extension manually.
-
Point your MCP client (like Claude Code or any other LLM agent harness) to the Hands MCP server.
If you run into issues, the logs are tucked away in %LOCALAPPDATA%\hands\logs\
.
Real-world limitations to keep in mind #
This isn't a "set it and forget it" sandbox. Since it's operating on your actual desktop, there are real risks. If you tell an agent to "find a cheap flight" and it accidentally clicks "Confirm Purchase" on a high-priced ticket, the tool won't stop it. The "confirm-before-money" check is just a best-effort classification within the binary; it’s not a hard safety guarantee.
Also, don't expect it to be a CAPTCHA-solving god. It can try a couple of visible interactions, but if it hits a puzzle, it will yield and wait for you to handle it manually.
For those interested in the technical specifics of the observe
and fusion
logic, the source is available here:
https://github.com/Ryan-AI-Studios/hands
It's a fascinating look at how we can bridge the gap between high-level LLM reasoning and low-level OS control without the overhead of traditional web automation frameworks.
Next Coding agents fail most often because they guess too much →
a library of Claude prompt techniques, with plenty of directly applicable cases.