{"slug": "i-built-an-mcp-server-that-lets-ai-agents-use-my-windows-apps-while-i-keep", "title": "I built an MCP server that lets AI agents use my Windows apps while I keep typing", "summary": "A developer built Sidehand, an MCP server for Windows 10 and 11 that lets AI agents read and operate background application windows while the user keeps typing in the foreground. The tool exposes a window's controls as a compact text tree and supports typing, clicking and list selection without moving the mouse or stealing focus, with the author reporting 4-5x fewer tokens than agent-desktop on typical apps and about 20x fewer on an Edge page. Sidehand is source-available under PolyForm Shield 1.0.0 and installs via npx or npm.", "body_md": "Most \"computer use\" tools take over the machine. The pointer moves, windows jump to the front, and you wait until the agent is done. I wanted an agent to work in one Windows app while I kept typing in another, so I built Sidehand.\n\nSidehand is an MCP server (plus a CLI and a pi extension) for Windows 10 and 11. Claude, Codex, Cursor or any MCP client can read and operate Windows apps that sit behind the window you are using. It types, clicks, picks from lists and reads the screen as text. Your mouse does not move and your keyboard stays with you.\n\n``` bash\n$ sidehand tree \"Contoso Orders\"\nwindow: Contoso Orders  (18 elements)\n  e4 Edit \"Customer\"\n  e12 Button \"Submit order\"\n$ sidehand act e4 type \"Blue-B Studio\"\n$ sidehand act e12 click\n```\n\nOne short line per control. Reading a window takes about 4 to 5 times fewer tokens than agent-desktop on typical apps, and about 20 times fewer on an Edge page.\n\nThe method is in docs/benchmark.md. In the benchmark task:\n\nSome apps bring their own window to the front while they handle a click. Windows gives a background process no way to forbid that, so Sidehand watches the foreground while an action runs and puts your window back as it happens. In my tests on the real desktop that took 11 to 13 ms. A key you press in that moment can still be lost, and the result tells the agent when it happened.\n\n```\nclaude mcp add sidehand -- npx -y sidehand mcp\n```\n\nCursor, Codex and Claude Desktop config are in the README. For the CLI: `npm install -g sidehand`.\n\n`--allow-foreground`, which is your permission to take the foreground for the keys. Enter on a button or link works without it.`<select>` drop-down cannot be operated in the background.\nSidehand is source-available under PolyForm Shield 1.0.0: free for any use, including at work, except building or offering a product that competes with Sidehand. It is not OSI open source.\n\nRepo: [https://github.com/Blue-B/sidehand](https://github.com/Blue-B/sidehand)\n\nBug reports for apps that do not behave in the background are very welcome.", "url": "https://wpnews.pro/news/i-built-an-mcp-server-that-lets-ai-agents-use-my-windows-apps-while-i-keep", "canonical_source": "https://dev.to/imper_7cde72b79d2529291ec/i-built-an-mcp-server-that-lets-ai-agents-use-my-windows-apps-while-i-keep-typing-320e", "published_at": "2026-11-18 13:11:06+00:00", "updated_at": "2026-11-18 13:14:03.517339+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools"], "entities": ["Sidehand", "Blue-B", "Claude", "Codex", "Cursor", "Windows 10", "Windows 11", "PolyForm Shield 1.0.0"], "also_reported_by": [], "takeaway": "Exposing a window's controls as a one-line-per-element text tree cuts token use roughly 4-5x versus agent-desktop on typical apps and about 20x on an Edge page, while a foreground watcher restores the user's window in 11-13 ms.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/i-built-an-mcp-server-that-lets-ai-agents-use-my-windows-apps-while-i-keep", "markdown": "https://wpnews.pro/news/i-built-an-mcp-server-that-lets-ai-agents-use-my-windows-apps-while-i-keep.md", "text": "https://wpnews.pro/news/i-built-an-mcp-server-that-lets-ai-agents-use-my-windows-apps-while-i-keep.txt", "jsonld": "https://wpnews.pro/news/i-built-an-mcp-server-that-lets-ai-agents-use-my-windows-apps-while-i-keep.jsonld"}}