How do "computer use" agents work?

Computer use agents, which let large language models operate a full computer through screenshots, clicks, and keystrokes, remain challenging to build, as the OSWorld benchmark shows. Introduced by Anthropic in October 2024, the capability has since expanded to over a dozen options, including OpenAI's Codex, Claude Cowork, Google's Gemini Agent, and open-source agents like OpenClaw and Hermes. The main technical approaches include screenshot-based vision with click coordinates and accessibility tree/DOM parsing, each with trade-offs in cost, latency, and robustness.

How do "computer use" agents work? Why it's surprisingly challenging to let an LLM use a computer for you. You might’ve heard of these “autonomous” agents that run 24/7 on your computer. They use your computer to do things like order a smashburger, or book a flight to Cancun YOLO . You might even have one running, right now, on a new Mac Mini. The term ‘computer use agent’ gets thrown around for three different things. The first two are components of the real thing: A vision-language model VLM looks at your screen and decides what to do clicks + keystrokes .A browser agent operates websites.A computer use agent controls the entire operating system https://technically.dev/universe/operating-system . The OSWorld benchmark https://os-world.github.io/ confirms that computer use is the hard part, as anyone who’s tried to get a computer use agent to open a file in Excel can attest. Let’s get into why that is, and where we’ve made progress. What does “computer use agent” mean? A computer use agent is an LLM https://technically.dev/universe/llm -powered agent that operates a full computer. It does so through the same interfaces a human uses: the filesystem, terminal https://technically.dev/universe/terminal , browser, and applications like Minesweeper when it’s bored. Computer use agents click buttons, fill in forms, create files, and work across multiple Git repositories at once. Most importantly, computer use agents retain a memory of historical sessions + context, and all the software installed to pick up where they left off. The interface is the same, but the way agents work on the computer diverges significantly from humans. We think we can multi-task, but computer use agents actually can, by spinning up multiple sub-agents to work on smaller tasks in parallel. Most people would say that computer use agents were introduced by Anthropic https://www.anthropic.com/news/3-5-models-and-computer-use in October 2024 as really a model capability vs a full agent, and that it didn’t work very well at the outset. Just a couple short years later, there are now over a dozen options: OpenAI gave Codex the ability to operate your Mac or Windows desktop https://openai.com/index/codex-for-almost-everything/ , Claude Cowork https://support.claude.com/en/articles/13345190-get-started-with-claude-cowork is one of the most praised products from Anthropic, Google folded Project Mariner into Gemini Agent https://www.androidauthority.com/google-project-mariner-shutdown-3664323/ , Factory offers persistent Droid Computers https://factory.ai/news/droid-computers , Perplexity launched its Computer https://www.perplexity.ai/products/computer , and open-source agents like OpenClaw https://github.com/openclaw/openclaw and Hermes https://github.com/NousResearch/hermes-agent have gone viral. How an AI agent uses a computer Agents “seeing” the screen and moving your cursor carry a certain sense of magic, but it’s not all the same magic. There are a few technical approaches: 1. Screenshot-based vision + click coordinates This was the early approach of Anthropic’s Computer Use 2024 and OpenAI’s CUA powering Operator https://openai.com/index/introducing-operator/ 2025 . The agent takes a screenshot, feeds it to a vision-language model VLM , which then decides to “click at pixel 432, 187 ” or “type ‘hello’”. It executes the action, takes a new screenshot, and repeats the loop. The agent controls the mouse, keyboard, and applications. This approach is expensive, and the screenshot → model inference https://technically.dev/universe/inference → action workflow has high latency although companies like Moondream are working on that . It’s also fragile, and errors compound when interacting with UI elements, low contrast, or dense layouts. Moving or scaling UI elements, or changing from light to dark mode, can cause the agent to fall over. 2. Accessibility tree / DOM parsing The challenges above are why we rely on structured approaches too. Every operating system and every browser exposes a structured tree of UI elements buttons, text fields, labels with their positions, roles, and states. An agent can read this tree as text, which is orders of magnitude cheaper and faster than reading it as images. What does this actually mean? When you visit a website, your browser builds two internal data structures: First, the DOM Document Object Model , a tree of every HTML https://technically.dev/universe/html element on the page. Second, on top of the DOM, the browser builds an accessibility tree , a simplified version that strips away decorative elements and keeps only what you can interact with: buttons, links, form fields, headings. Each element gets a role ”button”, “link” , a name ”Submit”, “Search” , and a state ”disabled”, “checked” . One common accessibility-tree tool is Microsoft’s Playwright MCP https://github.com/microsoft/playwright-mcp , which is also useful for coding agents to review their work on an app’s UI. But even structured input has real limits. In the original WebVoyager benchmark https://arxiv.org/abs/2401.13919 , a text-only agent reading just the accessibility tree scored 40.1% versus 59.1% for the same agent with vision. Dynamic, JavaScript-heavy sites like Booking.com http://booking.com/ and Google Flights were among the hardest categories, because the DOM gets messy. 3. Hybrid the 2026 consensus Hybrid can mean two different things: It means defaulting to the cheap, structured signal DOM or the accessibility tree , and falling back to screenshots VLM only when the structure runs out. Routing each task to whichever method suits the computer use problem at hand. At Factory where I work our computer-use agents do the second. They route across four backends https://technically.dev/universe/backend depending on the task: Accessibility-tree snapshots drive native desktop apps, letting the agent operate Finder or Slack by element rather than by pixel.The Chrome DevTools Protocol drives web and Electron apps,reading the page’s structure directly and screenshotting only when it has to.A virtual pseudoterminal PTY reads terminals as pure text , consuming the character stream and typing keystrokes back. This is how the agent runs git or installs a package.A terminal emulator captures actual pixels, for what the text stream can’t express: for apps with a full-screen terminal UI TUI , that might contain elements like progress bars + colored outputs. Only one of the four the terminal emulator works primarily from pixels. The rest read text or structure, because structured signal if it exists is cheaper and steadier. Computer use agents stack Every product that gets called a “computer use agent” is some combination of three layers: The model the LLM or vision-language model that decides what to do next The the agent software wrapped around the model . This one contains loops, tools, or sub-agent orchestration. IMO harnesses and models are converging a little. harness https://technically.dev/posts/whats-harness-engineering The computer , the environment the agent actually works in. Before, some people used containers https://technically.dev/posts/whats-docker-and-what-are-containers , but the consensus nowadays is using microVMs https://technically.dev/posts/what-are-code-sandboxes . Most arguments about what “counts” as computer use come from mixing up these layers. Anthropic’s original computer use was a model capability. Some agents are harnesses you install on your own computer. Others sell the harness and the computer together, with automatic routing between models. What a computer use agent needs Computer environment local or cloud The agent needs its own work environment. It can have its own literal computer your Mac Mini , or a secure cloud https://technically.dev/universe/cloud environment like an E2B sandbox. Persistence and long-running sessions True computer use means that the agent has a persistent relationship with a machine - it doesn’t break up, fly to Cancun, and then get back together later. An agent working across a real engineering workflow opening a pull request https://technically.dev/universe/pull-request , waiting on CI, responding to a code review, pushing a follow-up commit needs to resume from where it stopped. Memory and state An LLM remembers nothing between sessions, so that state has to live somewhere outside the prompt. You might hear “state” mentioned a lot online, and it might mean a few things: Files . Memory and instructions stored as Markdown files within folders. In OpenClaw, HEARTBEAT.md is the task list, or Hermes saves finished workflows as reusable SKILL.md http://skill.md files. The environment . Git repos, package dependencies https://technically.dev/posts/whats-a-package-manager , and test results just stay put, so the agent resumes where it left off. External memory stores . Vector databases or services like Mem0, Letta, or Zep hold what the agent learned and can retrieve it later. Parallelism An agent can spin up sub-agents to run tasks in parallel. For example, in Factory, worker agents work in sequence, but each of them can spawn specialized sub-agents during the workflow. Action space A human reaches for the mouse, the keyboard, the terminal, or a browser tab depending on what the task needs. An agent needs the same flexibility expressed as a defined set of primitives: click, type, scroll, run a shell command, read or write a file, call an API https://technically.dev/universe/api . The action space is a design decision with real tradeoffs: too narrow and the agent can’t complete tasks that fall outside it, too broad and it becomes harder to audit what the agent actually did and why. Most production https://technically.dev/universe/production systems restrict irreversible actions file deletion, form submission, outbound messages behind explicit confirmation steps. Tool integration Whenever an external service has an API, the agent should call it rather than navigate a UI to get the same result. This is where MCP Model Context Protocol https://technically.dev/posts/what-is-mcp becomes important. It’s a standardized protocol that lets agents discover and call tools across calendars, codebases, databases, and communication apps without custom wiring for every combination. As more services expose APIs and more MCP servers https://technically.dev/universe/server get built, the portion of work that requires the agent to actually “see” a screen keeps shrinking. Verification and error recovery A human notices when something went wrong: a form didn’t submit, a build failed, a page didn’t load. An agent needs to check this explicitly at each step, and validate that the output actually works https://factory.ai/news/missions-architecture , not just looks good in the code. Why are computer use agents going mainstream now? Not long ago, AI models couldn’t reason well enough to drive a computer use agent. Sandbox environments made for agents were just being built. There was no standard way for agents to connect to tools. And context windows were so small your agent forgot what it was doing halfway through a three-step task. All of that changed, more or less simultaneously. On OSWorld https://osworld-v1.xlang.ai/ , agents went from roughly 12% at the benchmark’s launch in 2024 to the low-to-mid 80s by mid-2026. The estimated human baseline is ~72%, which seems high if the task is to use Microsoft Teams. Comparison of computer use products All computer use agent products look similar on the surface, but work very differently under the hood. Here’s a quick summary of the field, current as of publication but moving quickly: Manus Manus originally ran each task in a fresh cloud sandbox, but it now also offers Cloud Computer launched April 30, 2026 , a persistent Ubuntu VM for 24/7 bots, databases, and scheduled automations. The standard Sandbox still recycles after 7 days free or 21 days paid . OpenClaw OpenClaw is the project that made “an agent living on your machine” a mainstream idea. It’s an agent that lives on a machine you provide a VPS, a Mac Mini, your laptop rather than a full OS-automation engine, though it can reach desktop tools and services via MCP servers. Hermes Agent Architecturally, Hermes is in the same family as OpenClaw: a self-hosted, persistent agent that lives on your server or PC, reachable over Telegram, Discord, Slack, or WhatsApp. Its defining feature is the self-improvement loop: persistent cross-session memory plus a skills system where the agent writes down how it solved a task and reuses that skill next time. Factory Droid Computers Disclosure: I work at Factory. Factory runs persistent cloud machines that pause when idle and resume with full filesystem and memory snapshots. You can Bring Your Own Machine https://docs.factory.ai/cli/features/droid-computers or use virtual computers. Factory’s observation layer is hybrid. Its Droid Control https://docs.factory.ai/cli/features/droid-control tooling picks the right backend for the target: Playwright-backed automation with Chrome DevTools Protocol support for web apps and Electron apps, accessibility-tree snapshots with ref-based element selection for native desktop apps, virtual-terminal automation for driving real TUI apps, and video recording of what the agent did. OpenAI Codex computer use In 2026, Codex got its own virtual cursor and can see, click, and type across all the apps on your Mac. Windows support followed in May 2026. It’s a vision-driven approach — screenshots plus simulated input — with one important twist: if a plugin or integration https://technically.dev/universe/integration exists for an app, Codex prefers the structured path and falls back to pixels only when it has to. Claude Cowork Claude Cowork now runs remote https://technically.dev/universe/remote persistent sessions by default beta, rolling out in 2026 . The agent loop and code execution run on Anthropic’s servers, and sessions persist across desktop, web, and mobile. For local desktop control, Claude Cowork can also use Computer Use via the Claude Desktop app. Perplexity Computer Perplexity Computer is fundamentally different from all of the above: in its cloud form it does not control your screen at all, running entirely in Perplexity’s cloud, in isolated VMs, orchestrating 20+ AI models as sub-agents. Think of it as a task orchestration layer. In April 2026 Perplexity shipped Personal Computer, a Mac app that runs Perplexity Computer locally on a Mac or Mac mini. Personal Computer can control local files, native Mac apps, the Comet browser, and Perplexity’s cloud sandbox. It is available to all Mac users as of May 2026. Devin Each Devin session runs in its own isolated Linux https://technically.dev/universe/linux VM with a full desktop environment, browser, terminal, and filesystem. Since 2.2, Devin can launch and test desktop applications, take screenshots to visually verify its work, and send back screen recordings for human review. In 2026 Devin got its own Windows VMs and a desktop command center for managing fleets of agents. Zo Computer Zo has the most literal implementation of this article’s definition — the agent and you share one persistent machine with a real filesystem — aimed at people who want personal software running on infrastructure https://technically.dev/universe/infrastructure they control rather than an enterprise SDLC. Every Zo user gets a persistent Linux server with real file storage, hosting, scheduled automations, and an always-on agent you can text via iMessage, SMS, Telegram, or email. Infrastructure, not agents Worth separating out: providers like E2B Firecracker microVMs , OpenComputer persistent full VMs with hibernate/wake and checkpoints , and Orgo https://www.orgo.ai/ headless cloud VMs for agents provide the machines that agents run on, but not the agent itself. What is still a challenge for computer use agents The PC was redefined once before, from terminal to GUI. It’s being redefined again, from a computer you use to a computer that uses itself sometimes with a human . I still see the three approaches to computer use screenshot-based, accessibility tree, and hybrid as an interim solution stitched onto human software, not as an agent-first solution. On the agent side, there are other specific challenges. CAPTCHAs were literally designed to block non-human users, but now we might need to distinguish it from CAPTCHAs for agents. Authentication https://technically.dev/universe/authentication is still a mess: agents need scoped tokens with limited permissions, not your username and password. More challenges like this could be summarized as “the web wasn’t built for agents, and it shows.” On the human side, we need to decide what role agents play in organizations. Are they autonomous AI employees, or just pieces of software? Should they be given high autonomy, or more frequent humans in the loop? How specialized vs general should they be? Either way, we are heading to an exciting future, where agents and humans collaborate together.