{"slug": "i-wanted-to-game-while-my-ai-agent-ran-tests-so-i-built-it-a-cursor-of-its-own", "title": "I wanted to game while my AI agent ran tests — so I built it a cursor of its own", "summary": "A developer built a small, driverless, user-mode Windows tool that lets a coding agent act in desktop applications without moving the real system cursor or stealing foreground focus. Instead of synthetic pointer injection, the tool drives native controls via window messages (BM_CLICK, EM_*) and UI Automation patterns, refusing any control that accepts neither rather than falling back to the shared mouse. The developer says the design is deliberately \"boring on purpose\" — no driver, no virtual HID, no input injection — and that an earlier synthetic pointer-injection experiment was discarded because it reproduced the exact cursor-stealing bug it was meant to fix.", "body_md": "I run coding agents on my home PC in the evening. They are genuinely useful: while I play a game, an agent can run a test suite, click through a desktop app, and report back.\n\nExcept for one thing: **the agent uses my mouse.**\n\nScreenshot-and-click computer use works by moving the one real system cursor and taking the foreground. That is fine when nobody is sitting at the PC. It is miserable when somebody is — mid-game, the cursor teleports, a window jumps to the front, and my input goes somewhere I never intended.\n\nI didn't want a second PC, and I didn't want a VM. I wanted one machine where a human and an agent work at the same time, in different windows, without fighting over a single pointer.\n\n**1. SendInput / coordinate clicking (what most computer-use agents do)**\n\nThere is exactly one system cursor. Any design that injects \"real\" mouse input is, by construction, a design that steals it. No scheduling trick fixes that; it's the wrong layer.\n\n**2. A VM or a second session**\n\nIsolation works, but it's heavy, and it changes the problem: the agent is no longer testing *my* apps in *my* session, with my files and my logged-in state. For some people that's the right answer. For \"run tests on this PC while I use this PC,\" it wasn't mine.\n\n**3. MouseMux**\n\nMouseMux is a real, well-made product: multiple independent cursors on one Windows desktop, and it now even ships an MCP \"Input Mapper\" so an AI agent can get its own virtual user. If several *humans* share one PC — point of sale, trading desks, remote collaboration — stop reading and go buy it.\n\nIt didn't fit my case, for three reasons:\n\n**4. Virtual HID drivers (FakerInput and friends)**\n\nSame anti-cheat problem, worse: now there's a kernel-adjacent input driver on a machine that runs games with strict anti-cheat. I wasn't willing to install that on my gaming PC.\n\nHere's the thing I got wrong at first: I assumed an agent needs to *click*, the way a human clicks.\n\nMost Windows applications don't actually require that. Native controls accept window messages (`BM_CLICK`, `EM_*` and friends). Modern frameworks expose UI Automation patterns — Invoke, Value, Selection and so on. Both work **in the background, on a window that is not in the foreground, without moving the pointer at all.**\n\nAnd for the controls that accept neither? The honest answer is not \"fall back to the real mouse.\" The honest answer is **\"this one, I can't do.\"** A refusal is a feature: it keeps the guarantee intact, and it keeps the human's session untouched.\n\nA small, driverless, user-mode tool — one executable, no network calls — that gives an agent its own way to act:\n\nAn earlier experiment is worth naming because it failed in an instructive way: synthetic pointer injection. Technically it worked. Practically, it moved the real cursor and stole the foreground — the exact bug I was trying to fix — so I threw it out. The most dangerous component in a design like this is the one that \"works\" by breaking your core guarantee.\n\nTo be clear about what this is *not*: it is not a game automation tool, it never interacts with a game process, and my rule is that agent operation stays out of protected matches entirely — the tool is excluded from my gaming setup's resident startup, and I exit it before playing anything with strict anti-cheat.\n\nThe anti-cheat relevance runs the other direction: the design is *boring on purpose*. No driver, no virtual HID, no input injection, no process interference. Those absences are the feature.\n\nBuild your own only if your constraints look like mine: one PC, one human who refuses to give up the mouse, agents that must act in real desktop apps, and a hard requirement that \"no\" is an acceptable answer from the tool.\n\nIn a follow-up I'll go through the failure catalogue in detail — the approaches above, what each one actually did when measured, and the acceptance checks I use to prove the real pointer never moved.\n\nIf you're evaluating MouseMux (or Anode, NexusWindows, agent-desktop, Cua Driver — this space got busy fast) for an agent use case, I'd genuinely like to hear where my decision tree diverges from yours.\n\n*Draft v1 — 2026-11-19. Personal project, built and run on my own hardware. No employer systems, data, or know-how involved.*", "url": "https://wpnews.pro/news/i-wanted-to-game-while-my-ai-agent-ran-tests-so-i-built-it-a-cursor-of-its-own", "canonical_source": "https://dev.to/keyaburi/i-wanted-to-game-while-my-ai-agent-ran-tests-so-i-built-it-a-cursor-of-its-own-m22", "published_at": "2026-11-19 04:58:11+00:00", "updated_at": "2026-11-19 05:11:49.689783+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["MouseMux", "FakerInput", "Windows", "UI Automation", "Anode", "NexusWindows", "agent-desktop", "Cua"], "also_reported_by": [], "takeaway": "Driving native controls through window messages and UI Automation patterns lets an agent act in background windows without moving the system cursor, and refusing unsupported controls preserves that guarantee where synthetic pointer injection breaks it.", "recommendation": {"advice": "Give desktop agents window-message and UI Automation control paths instead of synthetic mouse input, and make the tool refuse controls it cannot drive so the human's cursor and foreground focus stay untouched.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/i-wanted-to-game-while-my-ai-agent-ran-tests-so-i-built-it-a-cursor-of-its-own", "markdown": "https://wpnews.pro/news/i-wanted-to-game-while-my-ai-agent-ran-tests-so-i-built-it-a-cursor-of-its-own.md", "text": "https://wpnews.pro/news/i-wanted-to-game-while-my-ai-agent-ran-tests-so-i-built-it-a-cursor-of-its-own.txt", "jsonld": "https://wpnews.pro/news/i-wanted-to-game-while-my-ai-agent-ran-tests-so-i-built-it-a-cursor-of-its-own.jsonld"}}