{"slug": "you-can-t-solve-computer-use-by-ignoring-the-interface", "title": "You can't solve computer use by ignoring the interface", "summary": "A new analysis argues that current AI computer-use agents fail to generalize to real-world interfaces because they bypass the UI entirely, relying on API calls and scripting instead of human-level interaction. On OSWorld-V2, the best model achieves only 20.6% task completion, and on Agents' Last Exam just 26.2%, while humans exceed 95% on the WebGames benchmark. The authors contend that scaling models and tokens is a dead end for solving computer use.", "body_md": "# Computer use is far from solved\n\nRight now, agentic computer use is one of the biggest levers for real-world AI impact. LLM-based agents are transforming software development, but most intellectual work is gated behind using software. When coding agents are so good, it is natural to ask: can they file my taxes in a government portal, fix a text document, test a website?\n\nWe are not there yet. On OSWorld-V2, a leading benchmark of long-horizon computer tasks, the best model achieves only *20.6%* completion rate. On Agents' Last Exam the best result is *26.2%*. Users accustomed to the impressive performance of LLMs in chat interactions expect similar results from computer use. But they are met with frustration: agents are unreliable, slow and expensive. At the moment, it's easier to just do the work yourself.\n\nIn 2024–2025 every major AI lab shipped a computer-use prototype. At the time, these showed excellent performance on benchmarks such as OSWorld, WebArena, AndroidWorld and WebVoyager, boasting task completion rates from 60.76% up to 97.4%. Those benchmarks were a good fit for the agents. With few exceptions, they presented each task as mostly static environments with clean text representations and small pre-defined action spaces — which is nothing like the real world of messy interfaces. Hence the inflated leaderboards and the gap between user expectations and real performance.\n\nSo far, the answer to improving computer use has been more of the same: larger models and more tokens. *In our opinion, this direction is a dead end.* The established approaches look good in cushioned environments, but don't generalize to real GUI work.\n\n## The core flaw of computer use\n\nComputer use has well-known hard parts such as perception, planning and preserving context across a long task. Those issues are being addressed by the community and some of this genuinely requires smarter models. However, a simpler issue is getting less attention, and it might be the bottleneck: *interface-using agents mostly avoid the interface.*\n\nOn OSWorld-V2, the best models often aren't operating the UI at all. In task 052, to book a hotel suite, GPT-5.5 injected JavaScript to read the site's source, found the internal API endpoint, and POSTed the reservation. In task 065, to buy a train ticket, both GPT-5.5 and Claude Opus skipped the website and POSTed straight to the API. To compose two images in GIMP in task 003, GPT-5.5 scripted in Python. Overall, most tasks were not solved as intended, but bypassed instead.\n\nThis can be overlooked or even seen as advantageous. Why touch the GUI if you can make an API call? However, some work can only be done through the UI. Bypassing the UI can have unexpected side-effects, because the software was not designed to be used this way. More importantly, a lot of work is much easier to get done through the UI — which is important to get economically feasible computer use.\n\nClicking buttons, filling forms and navigating websites are not hard tasks. A person does it without thinking. Reverse-engineering a site's API, disassembling its JavaScript or scripting a compositor from scratch is more complex than the click it replaces. Doing it directly would be cheaper, faster and much more reliable.\n\nHowever, for frontier models it's not an option. The WebGames benchmark challenges models to complete simple tasks that require the reaction time and motor control of an average website user. The human success rate is over *95%*, but models lag far behind despite their intelligence. That gap does not narrow with more tokens or parameters. This is why, in long-horizon tasks, models reach for hard solutions: they do not have the ability to use interfaces on a human level, so they have to work around that limitation.\n\nThis leads to a suboptimal compute allocation. Agents spend most of their actions on perception and action. Figure 10 from OSWorld 2.0 breaks down the action budget: visual grounding, low-level manipulation and tool-use overhead dominate the rest. Planning how to solve a task is supposed to be harder than clicking a button, but most computation goes to seeing and clicking. Reasoning, reflection and error recovery get what's left.\n\nIn our opinion, this is the core issue with agentic computer use: *we use trillion-scale reasoners to work around clicks.* Proper manipulation would allow agents to spend effort on solving the task instead of hacking around interface limitations. This would enable faster execution, fewer tokens burned in vain, improved reliability and better long-horizon performance.\n\n## Towards a steelman of agentic computer use\n\nAll computer-use solutions so far have been built on the screenshot–toolcall loop:\n\n- Obtain a screenshot and a text representation of the environment.\n- Make an LLM reason over it.\n- Output a toolcall such as\n`click(elem[#submit])`\n\n. - Execute.\n- Repeat.\n\nThis is a natural extension of the coding-agent ReAct loop to UI interaction. But trivial UI actions remain hard for such agents, because they use the same large model to do everything. The result is an agent that overly relies on text, misses all the temporal information and can't react in time. This approach works in opposition to all the assumptions the UIs make about their users.\n\nAdding more parameters and tokens will not solve the problem. Even executing this loop 10 times faster will only fill the LLM's context faster. To build usable computer-use agents, we need to challenge the core assumptions.\n\nAt Steelman Labs, we are working around a core principle: *a computer-use agent must be able to use any interface a person can.* To satisfy this condition, we separate planning from execution and introduce a System 1 for agents: a manipulator that takes care of motor control. We take a vision-first approach, treat the screen as a dynamic environment and pursue human-level reaction times.\n\n[hello@steelmanlabs.com](mailto:hello@steelmanlabs.com)", "url": "https://wpnews.pro/news/you-can-t-solve-computer-use-by-ignoring-the-interface", "canonical_source": "https://steelmanlabs.com/blog/computer-use-is-far-from-solved", "published_at": "2026-07-30 11:28:51+00:00", "updated_at": "2026-07-30 11:52:26.678080+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research"], "entities": ["OSWorld-V2", "Agents' Last Exam", "WebGames", "GPT-5.5", "Claude Opus"], "alternates": {"html": "https://wpnews.pro/news/you-can-t-solve-computer-use-by-ignoring-the-interface", "markdown": "https://wpnews.pro/news/you-can-t-solve-computer-use-by-ignoring-the-interface.md", "text": "https://wpnews.pro/news/you-can-t-solve-computer-use-by-ignoring-the-interface.txt", "jsonld": "https://wpnews.pro/news/you-can-t-solve-computer-use-by-ignoring-the-interface.jsonld"}}