Notes on Cross-Platform Computer Use Agents for macOS and Windows — Selection with Hermes Agent and cua-driver as of September 2026

Tadashi Shigeoka· Thu, September 3, 2026 Someone on our team asked “we want to seriously evaluate a desktop automation agent, but unless it works on both macOS and Windows we can’t roll it out internally.” That prompted this comparison, with Nous Research’s Hermes Agent as the primary candidate, alongside trycua’s cua-driver, ByteDance’s UI-TARS Desktop, Simular’s Agent S, and Open Interpreter. I also included ChatGPT desktop app Computer Use and Claude desktop app Cowork in scope.

This article is based on official documentation, official GitHub repos, and public releases as of 2026-09-03. It is not a hands-on comparison where every candidate was run against every OS on real hardware. Even so, the things that don’t show up in a filled-in feature matrix (the caveats around background operation, OS-level walls like TCC / UIPI / Session 0, and multi-monitor and Japanese-input pitfalls) are worth pinning down before selection, so I’ve organized the notes into five topics.

Computer Use Actually Has Two Different Scopes #

Terminology first. “Computer Use” as a phrase collapses two very different scopes that should be kept separate.

The first is “browser-only” control: Claude in Chrome, Google Gemini 2.5 Computer Use, the browser environment of the OpenAI Computer Use API. Clicks and inputs are dispatched through the DOM or the Chromium process, so the surface is web apps only.

The second is “full-OS” control including native applications: the ChatGPT desktop app Computer Use plugin, Claude Cowork’s Computer use, Hermes Agent, cua-driver, UI-TARS Desktop, Agent S, and so on. Here the agent can touch Excel, Finder, Explorer, internal LOB clients directly.

If your internal systems don’t expose APIs, or if you want the agent to shuttle data between Microsoft Excel and a bespoke line-of-business app, only the second scope satisfies the requirement. This comparison narrows down to that second scope.

Topic 1: Only ChatGPT and Claude Cover Two OSes and Full-OS Control Officially #

As of 2026-09-03, the official apps that clear both bars (two-OS support and full-OS control) realistically come down to two.

ChatGPT desktop app Computer Use is available on macOS and Windows, and can operate the GUI broadly once you pre-approve target apps. That said, the Windows build explicitly takes over the pointer and input on the active desktop in foreground mode. If you want to keep working on the same machine in parallel, in practice the Windows path means a dedicated PC or a virtual machine.

Claude desktop app Computer use inside Cowork is in beta and available on both macOS and Windows. The toggle sits under “Settings > General > Enable computer use,” and on macOS you need to grant Accessibility and Screen Recording permissions up front. Current help pages scope it to Pro / Max; Team and Enterprise are outside that scope. Separately, Claude in Chrome went GA in 2026, but it’s browser-only and outside this comparison.

On the other hand, Google’s Gemini Computer Use landed inside Gemini 3.5 Flash in June 2026, with Gemini 3.8 Flash as the currently recommended model. It now covers browser, mobile, and desktop environments (OS-level cursor commands) through the API. Even so, Computer Use inside the Gemini desktop app itself was still a closed beta as of 2026-08, so it hasn’t yet cleared the “GA official app on both OSes with full-OS control” bar. Microsoft Copilot’s Copilot Actions is an experimental preview on Windows 11, and I couldn’t confirm an equivalent agent feature on macOS Copilot. Apple Intelligence’s next-generation Siri is not general-purpose “screen control” but structured app integration through the App Intents framework, a fundamentally different model.

Filtered to “two OSes + full-OS + generally available,” the official-app answer today is a two-horse race between ChatGPT and Claude.

Topic 2: On the OSS Side, Hermes Agent + cua-driver, UI-TARS, Agent S, and Open Interpreter Are the Real Candidates #

Official apps lock you into one model and one workflow. If you want to embed the agent in your own product, swap models freely, or run everything on local hardware, you need OSS candidates. The five OSS options I ended up shortlisting for native-app operation on both OSes are:

• Hermes Agent (fromNous Research ,MIT license ): an autonomous AI agent orchestration layer. Ships acomputer_use toolset that callscua-driver overMCP . Model-agnostic: Claude, GPT, Gemini, or local VLMs can be swapped in
• cua-driver (fromtrycua , MIT license): a shared operation layer providing screen capture, UI element inspection, and input dispatch. No planning layer of its own. Available as Python SDK, TypeScript SDK, MCP server, and CLI
• UI-TARS Desktop (fromByteDance ,Apache-2.0 ): an Electron-based GUI agent built around theUI-TARS model . “Pure Vision,” which infers coordinates from screenshots alone with no accessibility tree dependency
• Agent S (fromSimular ,Apache-2.0 ): a Python-SDK agent framework that separates hierarchical planning from grounding. Backed by a paper accepted atICLR 2025
• Open Interpreter : thecurrent version is a fork of OpenAI Codex with Rust-native model harnesses, and its QA skill usesagent-browser for the web and trycua for native apps. Assumptions have shifted since the older Python-era “OS mode” writeups

Of these, cua-driver is the shared operation layer and Hermes Agent is the orchestration layer, so they are complements rather than competitors. The Hermes official Computer Use page also documents the setup where Hermes connects to cua-driver over MCP over stdio.

Topic 3: The Real Operating Scope of the “No-Foreground Contract” #

The Hermes Agent and cua-driver pitch leans heavily on “background operation without stealing cursor or focus.” The user keeps coding or writing in the foreground while the agent operates another app behind the scenes. This is presented as the main differentiator against classic foreground-takeover agents.

Under the hood, cua-driver achieves this by using different OS-specific APIs. On macOS, it walks the window hierarchy via the Accessibility API (AX) and the undocumented SkyLight SPI, and dispatches events with SLPSPostEventRecordTo, which is scoped by PID and injects events into the target window without moving the system cursor. On Windows, it walks the control tree via UI Automation and dispatches input via a mix of SendInput and PostMessage, designed to not steal foreground focus.

That said, background operation is not unconditional. The cua docs page Best-effort background explicitly describes it as best-effort background, and calls out that when background operation isn’t possible, a foreground fallback is required. Cua’s Known Limits page lists custom-render canvas apps like Blender and Unity, plus most native games, as the concrete non-supported cases (windows launched with administrator privileges are covered separately by the Windows elevated-integrity constraint discussed in Topic 4).

So the accurate reading is not “install Hermes and every app operation stays out of your way” but “background operation is preferred, but foreground fallback remains for certain apps and OS states.” At selection time, you have to verify empirically that background operation actually holds up for the specific apps you plan to use.

Topic 4: OS-Native Walls That Show Up on macOS and Windows Separately #

Cross-platform doesn’t mean the OS-level constraints disappear. Each OS has its own walls, and they apply equally to official apps and OSS candidates.

On macOS, the first wall is permission management via TCC (Transparency, Consent, and Control). The agent process needs explicit manual grants for both “Screen Recording” and “Accessibility” under “System Settings > Privacy & Security,” and this dialog assumes a GUI session. That’s why provisioning to CI/CD or headless servers is awkward. Additionally, the AX and SkyLight surfaces that cua-driver relies on are SPIs, so behavior can shift across macOS updates. The Hermes platform support table also lists macOS Tier 1 as Apple Silicon only, with Intel Macs marked Unsupported. That’s a real constraint if your distribution target still includes Intel machines.

On Windows, the primary walls are UIPI (User Interface Privilege Isolation) and Session 0 isolation. UIPI means an agent running at Medium Integrity can’t send UI Automation calls or keystrokes to apps running at High Integrity or to UAC elevation dialogs. The cua platform support table also lists elevated-integrity boundaries as not-supported / not-verified. Session 0 isolation means an SSH session cannot see the regular interactive user desktop. cua publishes a separate Windows SSH how-to where the GUI-driver runs inside an interactive session and SSH connects into it, a two-tier setup.

Another common trap across both OSes is multi-monitor support. UI-TARS Desktop and Agent S both explicitly assume a single monitor in their docs, and PyAutoGUI handles the primary monitor only. If multiple displays are baked into the workflow, that filter alone thins the list.

Topic 5: Japanese IME, Full-Width Characters, and DPI Scaling #

Demo videos almost always show English UI receiving English input, so this comes up late. The moment the target is a Japanese business app, IME behavior starts to matter.

Japanese input goes through IME composition, so the raw virtual keycodes emitted by SendInput or SLPSPostEventRecordTo don’t always commit the intended string. Whether the agent falls back to clipboard paste for text input can flip the success rate for Japanese sentences by a wide margin. I didn’t find any of the candidates documenting Japanese IME behavior explicitly in scope for this survey.

Full-width spaces, full-width parentheses, and symbol variants get rejected by input validation in some LOB apps. Model-side tokenization and OS-side character encoding conversions can also misalign, so English-language success rates don’t extrapolate cleanly.

DPI scaling and coordinate systems are also easy to underweight. On a 4K display at 200% scaling, Pure Vision agents tend to lose the correspondence between the image resolution the model receives and the logical coordinates that need to be clicked. Setups that can pull element bounds from an accessibility tree (like cua-driver’s approach) are more robust here.

None of these show up in English demos or in OSWorld scores, so verify them yourself with short input / save / reopen tasks in Japanese on your own environment before adopting anything for LOB apps.

An Initial Configuration with Hermes Agent at the Center #

Given the topics above, the initial configuration I settled on for a Hermes-Agent-centered setup is below. This is a configuration derived from public docs, not one tuned against measurements.

- Orchestration layer: [Hermes Agent](https://hermes-agent.nousresearch.com/) (Hermes Desktop or CLI)
- Operation layer: connect to [cua-driver](https://github.com/trycua/cua) via MCP over stdio through Hermes’s built-in`computer_use` toolset
- Inference model: Claude, GPT, or Gemini depending on task and sensitivity, or a local VLM via vLLM, [LM Studio](https://lmstudio.ai/) , or[Ollama](https://ollama.com/)
- Web-first tasks: use Hermes’s browser toolset (headless [Chromium](https://www.chromium.org/) ) instead of`computer_use`

• Permission mode: start with bounded (pre-approved capability manifest) before any production use; keep YOLO / unrestricted confined to disposable VMs

Install per the official installation guide. There are official installers for macOS and Windows; the Windows path can be the native build and does not require WSL. The config directory is ~/.hermes/ on both OSes (or /root/.hermes/, or $HERMES_HOME, for a root install). For existing installs, hermes update pulls updates.

To enable Computer Use, follow the official Computer Use page: hermes computer-use doctor reports missing drivers or permissions. On macOS, grant Accessibility and Screen Recording to the apps listed by doctor before the first run.

For the first shakedown, pick a task that goes through a full input / save / re-verify cycle in the shape of real work. For example, “open a notes app, type a short Japanese sentence, save to a designated verification file, then re-open the file and confirm.” A one-off click demo won’t touch IME composition, file dialogs, or a save-and-reopen roundtrip, so the differences between the two OSes get lost.

Where Each SDK Sits for Custom Product Integration #

If the goal is to embed one of these into your own product, this is how I read the SDK landscape:

• Strongest shared operation layer: cua-driver . Three surfaces are available (Python’scua_driver , TypeScript’s@trycua/cua-driver , and the MCP server). You bring the planning and the model
• Custom Node.js / Electron GUI agent: UI-TARS SDK (@ui-tars/sdk ). Officially marked Experimental, but the NutJSOperator gives you cross-platform reach

- Python research and accuracy comparison: [Agent S](https://github.com/simular-ai/Agent-S) (`gui-agents` ). A hierarchical setup with a separate grounding model alongside the main model
- Minimal mouse / key control: [PyAutoGUI](https://pyautogui.readthedocs.io/en/latest/) or[nut.js](https://github.com/nut-tree/nut.js/) . Add AI-driven understanding and planning on top yourself

• Direct API use: OpenAI Computer Use API orAnthropic Computer Use tool . The API itself doesn’t touch the user’s PC, so you implement the screen-capture / input-dispatch / iterate / verify loop yourself

For an Electron-based product integration, the cleaner starting choice is between cua-driver’s TypeScript SDK and an out-of-process MCP connection. If you also want to own the vision model and the operation loop, UI-TARS SDK becomes the next candidate to evaluate.

Security Boundaries and Permission Modes Are Unavoidable #

An agent that can operate the whole OS exposes a broad attack surface as-is. Indirect prompt injection is especially unavoidable to design against: an agent visually reads an invisible malicious instruction embedded in a web page and executes a shell command on the local terminal. That scenario is being discussed for real.

The three-tier permission model that Hermes Agent documents handles this cleanly. standard mode requires interactive approval per operation (default), bounded mode is a fail-closed setup that only permits actions listed in a pre-defined capability manifest (allowed app identifiers, allowed web origins, specific tools), and unrestricted / YOLO mode drops all interactive confirmation for throughput and is recommended for use inside disposable VMs only.

Destructive actions (click, type, drag, scroll, key, focus_app) require approval by default, and inputs like sudo rm -rf /, curl | bash, or fork bombs are hard-blocked. Dangerous key combinations like “empty trash” or “log out” are blocked in the same way.

On the commercial side, ChatGPT Computer Use blocks direct terminal-app operation and mandates confirmation on financial sites. On Claude’s side, Cowork Computer use has an explicit permission-approval flow, and on macOS it requires the Accessibility and Screen Recording grants explicitly.

Regardless of the tool, don’t run an unrestricted agent directly on your host OS. Either pin the permission mode to bounded, or box the agent inside an isolation environment like Lume or Windows Sandbox. That design decision has to come first.

Local vs Cloud Are Two Separate Choices #

“Run Computer Use locally” actually collapses two independent choices: where operation runs, and where inference runs.

Operation can be local while the model is a cloud API, and in that case the screenshots and tool results the model needs are still sent to the cloud. For workflows that touch screens with sensitive information, inference also has to be local, or you are effectively exfiltrating the data. Under that constraint, running UI-TARS models on local GPU, or pairing Hermes Agent with a local vision model via vLLM / LM Studio / Ollama, are the realistic options.

Cost for API-based use isn’t just the model price either. Model inference, screenshot images, tool results, retries, and, if a dedicated VM is part of the pipeline, VM runtime all stack up. OpenAI’s docs say the execution environment is the user’s responsibility; Anthropic’s Computer Use tool documentation explains token pricing including images and tool results.

cua also has sandbox, fleet, and VM primitives separate from the local cua-driver. Using those to drive Windows or Linux in the cloud is a different architecture from operating apps on your own local macOS or Windows box, and needs to be evaluated separately on both requirements and cost.

Selection Guide #

Given the above, splitting the decision into three stages avoids the usual dithering.

For stage 1, “try it end-user-first” on both OSes as fast as possible, the shortest path is [ChatGPT desktop app Computer Use](https://learn.chatgpt.com/docs/computer-use) or [Claude desktop app Cowork > Computer use](https://support.claude.com/en/articles/14128542-let-claude-use-your-computer-in-cowork). If you already pay for ChatGPT or Claude, incremental cost is effectively zero. Verify Claude Cowork’s edition and virtualization requirements separately if the target is Windows.

For stage 2, “embed in your own product,” if you want to keep the model swappable, [cua-driver](https://github.com/trycua/cua) as the shared operation layer with [Hermes Agent](https://hermes-agent.nousresearch.com/) as the orchestration layer on top is the strongest starting point. If the scope is browser-only, evaluate a dedicated tool like [Browser Use](https://github.com/browser-use/browser-use) or [Skyvern](https://github.com/Skyvern-AI/skyvern) instead. You don’t need a full-OS agent for that.

For stage 3, “no sensitive data leaves the machine,” the first-line candidates are [UI-TARS Desktop](https://github.com/bytedance/UI-TARS-desktop) running with a local LLM, or Hermes Agent paired with a local vision model (vLLM / LM Studio / Ollama). The load-bearing decision inputs here are measured model size, GPU requirements, and inference latency.

Whatever configuration you pick, define three cross-OS tasks and measure them before rolling anything out:

1. Type a short Japanese sentence, save to file, reopen, and verify contents match
2. Change one setting in a native app and verify the changed state
3. Copy a short piece of information across two apps and verify the result matches

Run each task several times and record success rate, elapsed time, model cost, number of manual interventions, and how much cursor / focus interference the user experienced. That plain-measurement approach ended up being the most useful input for the decision. Vendor benchmark rankings (OSWorld scores and so on) are informative but don’t guarantee real-world viability on Japanese UI, so I wouldn’t put them at the center of the selection.

Caveats #

The content above is compiled from official docs, official GitHub repos, and public releases as of 2026-09-03, not from a comparison where every candidate was exercised on both OSes on real hardware. As a memo to seed daily-standup or team-meeting discussion, I want to leave these explicitly on the table:

• Main-branch documentation can be ahead of the released packages; when introducing any of these, pin the version and cross-check the corresponding release notes
• OSWorld and other benchmark scores vary with the benchmark version (original / Verified / 2.0) and with step count, most numbers are self-reported by vendors, and I’ve deliberately not used them for direct ranking
• Computer Use inside the Gemini desktop app (closed beta), Copilot agent features on macOS, and Apple’s next-generation Siri are all in preview / testing / not-yet-shipped stages, and I couldn’t confirm general-availability timing or details within this survey
• Japanese UI, IME, full-width characters, and display scaling weren’t in horizontal-comparison scope this time; for LOB app adoption, verify on real hardware in your own environment

That’s all from notes on cross-platform Computer Use agents for macOS and Windows, with Hermes Agent and cua-driver at the center, from the Gemba.