You can't solve computer use by ignoring the interface

A new analysis argues that current AI computer-use agents fail to generalize to real-world interfaces because they bypass the UI entirely, relying on API calls and scripting instead of human-level interaction. On OSWorld-V2, the best model achieves only 20.6% task completion, and on Agents' Last Exam just 26.2%, while humans exceed 95% on the WebGames benchmark. The authors contend that scaling models and tokens is a dead end for solving computer use.

Computer use is far from solved Right now, agentic computer use is one of the biggest levers for real-world AI impact. LLM-based agents are transforming software development, but most intellectual work is gated behind using software. When coding agents are so good, it is natural to ask: can they file my taxes in a government portal, fix a text document, test a website? We are not there yet. On OSWorld-V2, a leading benchmark of long-horizon computer tasks, the best model achieves only 20.6% completion rate. On Agents' Last Exam the best result is 26.2% . Users accustomed to the impressive performance of LLMs in chat interactions expect similar results from computer use. But they are met with frustration: agents are unreliable, slow and expensive. At the moment, it's easier to just do the work yourself. In 2024–2025 every major AI lab shipped a computer-use prototype. At the time, these showed excellent performance on benchmarks such as OSWorld, WebArena, AndroidWorld and WebVoyager, boasting task completion rates from 60.76% up to 97.4%. Those benchmarks were a good fit for the agents. With few exceptions, they presented each task as mostly static environments with clean text representations and small pre-defined action spaces — which is nothing like the real world of messy interfaces. Hence the inflated leaderboards and the gap between user expectations and real performance. So far, the answer to improving computer use has been more of the same: larger models and more tokens. In our opinion, this direction is a dead end. The established approaches look good in cushioned environments, but don't generalize to real GUI work. The core flaw of computer use Computer use has well-known hard parts such as perception, planning and preserving context across a long task. Those issues are being addressed by the community and some of this genuinely requires smarter models. However, a simpler issue is getting less attention, and it might be the bottleneck: interface-using agents mostly avoid the interface. On OSWorld-V2, the best models often aren't operating the UI at all. In task 052, to book a hotel suite, GPT-5.5 injected JavaScript to read the site's source, found the internal API endpoint, and POSTed the reservation. In task 065, to buy a train ticket, both GPT-5.5 and Claude Opus skipped the website and POSTed straight to the API. To compose two images in GIMP in task 003, GPT-5.5 scripted in Python. Overall, most tasks were not solved as intended, but bypassed instead. This can be overlooked or even seen as advantageous. Why touch the GUI if you can make an API call? However, some work can only be done through the UI. Bypassing the UI can have unexpected side-effects, because the software was not designed to be used this way. More importantly, a lot of work is much easier to get done through the UI — which is important to get economically feasible computer use. Clicking buttons, filling forms and navigating websites are not hard tasks. A person does it without thinking. Reverse-engineering a site's API, disassembling its JavaScript or scripting a compositor from scratch is more complex than the click it replaces. Doing it directly would be cheaper, faster and much more reliable. However, for frontier models it's not an option. The WebGames benchmark challenges models to complete simple tasks that require the reaction time and motor control of an average website user. The human success rate is over 95% , but models lag far behind despite their intelligence. That gap does not narrow with more tokens or parameters. This is why, in long-horizon tasks, models reach for hard solutions: they do not have the ability to use interfaces on a human level, so they have to work around that limitation. This leads to a suboptimal compute allocation. Agents spend most of their actions on perception and action. Figure 10 from OSWorld 2.0 breaks down the action budget: visual grounding, low-level manipulation and tool-use overhead dominate the rest. Planning how to solve a task is supposed to be harder than clicking a button, but most computation goes to seeing and clicking. Reasoning, reflection and error recovery get what's left. In our opinion, this is the core issue with agentic computer use: we use trillion-scale reasoners to work around clicks. Proper manipulation would allow agents to spend effort on solving the task instead of hacking around interface limitations. This would enable faster execution, fewer tokens burned in vain, improved reliability and better long-horizon performance. Towards a steelman of agentic computer use All computer-use solutions so far have been built on the screenshot–toolcall loop: - Obtain a screenshot and a text representation of the environment. - Make an LLM reason over it. - Output a toolcall such as click elem submit . - Execute. - Repeat. This is a natural extension of the coding-agent ReAct loop to UI interaction. But trivial UI actions remain hard for such agents, because they use the same large model to do everything. The result is an agent that overly relies on text, misses all the temporal information and can't react in time. This approach works in opposition to all the assumptions the UIs make about their users. Adding more parameters and tokens will not solve the problem. Even executing this loop 10 times faster will only fill the LLM's context faster. To build usable computer-use agents, we need to challenge the core assumptions. At Steelman Labs, we are working around a core principle: a computer-use agent must be able to use any interface a person can. To satisfy this condition, we separate planning from execution and introduce a System 1 for agents: a manipulator that takes care of motor control. We take a vision-first approach, treat the screen as a dynamic environment and pursue human-level reaction times. hello@steelmanlabs.com mailto:hello@steelmanlabs.com