How OpenAI's Codex Turned Computer Use From Party Trick to Tool

OpenAI engineers say Codex's computer-use feature crossed a reliability tipping point in the last year or so, moving it from what one engineer called an "impressive party trick" to a tool fast enough for casual, constant use. The capability, which lets an AI agent move a cursor, click buttons, scroll pages and fill forms inside real applications and websites, began as a developer-facing feature that displayed the scripts and selectors it ran and now presents a simple on-screen pointer in Codex's desktop app. OpenAI is investing in both computer use and direct integrations such as Slack and email because connectors are often faster and more token-efficient than clicking through a UI.

How OpenAI's Codex Turned Computer Use From Party Trick to Tool OpenAI engineers explain how Codex's computer-use feature clicks, browses and fills forms across your desktop, and why it took years to get reliable. What is computer use in OpenAI’s Codex? Computer use is the capability that lets an AI agent operate a computer the way a person does: moving a cursor, clicking buttons, scrolling pages, and filling in forms inside real applications and websites, rather than calling an API behind the scenes. In OpenAI’s Codex, this started as a developer-facing feature built on coding agents that ran visible scripts. As Codex expanded into a full desktop app for a broader audience, the same underlying capability got a friendlier face: a pointer that visibly clicks around the screen instead of exposing the raw automation script. TL;DR - Computer use began as a niche developer tool where Codex showed the actual scripts and selectors it used to click through an app, which worked but wasn’t approachable for non-technical users. - The interface changed as the audience grew , replacing visible code with a simple on-screen pointer so people could watch the agent work without needing to understand what’s happening underneath. - Reliability crossed a tipping point in the last year or so , moving computer use from what one OpenAI engineer called an “impressive party trick” that was slow to something fast enough to use casually and constantly. - Computer use now acts as a universal connector for tasks that have no API or MCP integration, like government paperwork, PDFs, or legacy web forms that will likely never get a dedicated connector. - Direct integrations Slack, email, and similar connectors still matter because they’re often faster and more token-efficient than clicking through a UI, so OpenAI is investing in both approaches rather than picking one. - Internal adoption at OpenAI happened unevenly by function , with coding teams switching first because their work lived in local command-line tools, and other teams like legal following once the model got good enough at reasoning over large amounts of material. - The bigger shift is behavioral , according to OpenAI staff: the habit that matters most is catching yourself before doing a task manually and routing it to the agent instead. Remy doesn't write the code. It manages the agents who do. Remy runs the project. The specialists do the work. You work with the PM, not the implementers. How did computer use evolve from a party trick to a reliable tool? According to OpenAI engineers who build the Codex desktop app, computer use has existed in some form for a while, but it stayed a developer tool for a long time because it needed technical users who wanted to see the scripts, selectors, and clicks the model was running under the hood. That transparency worked for engineers debugging automation, but it doesn’t hold up when you’re trying to reach a general audience who just wants a task done. The shift to Codex as a full desktop product forced a design change. Instead of showing code, the interface now shows a simple pointer that clicks around the screen, sometimes with a picture-in-picture view of the action. The model doing the work underneath is largely the same coding agent architecture. What changed is the presentation: less visibility into the mechanics, more trust that the output is correct. The bigger change was speed. Early computer use was visually impressive but slow enough that using it for real work felt impractical. OpenAI engineers describe a tipping point where the capability became fast enough that people stopped noticing when an agent quietly switched from a direct integration to clicking through a UI instead. The task just finished, and the method became invisible. Why does computer use matter when APIs and MCP integrations already exist? The pitch for MCP Model Context Protocol and direct API integrations is efficiency: they’re typically faster and more token-efficient than watching a cursor click through a web page. But that only helps if the integration exists. In practice, most of the systems people deal with daily, government portals, insurance paperwork, pediatrician forms still handled by fax, legacy enterprise software, will never get a clean API or MCP connector built for them. That’s the gap computer use fills. One OpenAI engineer described initially underestimating “the scaling law potential of computer use,” expecting that machine-readable interfaces would always win out. Instead, once computer use became reliable and fast, it turned into what the team calls a universal connector: a fallback that works on virtually anything with a screen, regardless of whether a developer ever built a proper integration for it. This matters for anyone building with AI agents today. You don’t need to wait for a vendor to ship an API. If a task can be done through a browser or desktop app, an agent can plausibly be pointed at it directly. Are integrations still worth building if computer use works everywhere? Yes, according to OpenAI’s own engineering approach. Direct connectors and plugins for tools like Slack and email still get investment because, when they exist and work well, they’re faster and cheaper in terms of token usage than clicking through a UI step by step. The team describes focusing on latency, reliability, and making sure connectors expose as much of a third-party service’s functionality as possible, rather than treating computer use as a replacement for integrations. Other agents start typing. Remy starts asking. Scoping, trade-offs, edge cases — the real work. Before a line of code. The practical framing: build a direct connector when a service is common enough and important enough to justify it, and rely on computer use for the long tail of tools that will never get that treatment. Both approaches expanded around the same period, and the team says model quality improvements made existing connectors dramatically more useful even before computer use matured, since earlier models weren’t always good at using the plugins that already existed. How is this changing how people actually work? The examples described by OpenAI staff go beyond writing code. One engineer mentioned Codex catching a significant error in a personal tax filing, in the user’s favor, worth thousands of dollars, simply by being asked to review the paperwork. Others described using the agent to fill out forms, dig through PDFs, or handle bureaucratic tasks that would otherwise require finding, reading, and manually transcribing information across disconnected systems. Inside OpenAI, adoption reportedly didn’t happen all at once across the company. Coding teams shifted first, largely because their work already lived in command-line tools that agents could operate directly. Other functions, including legal, picked it up once the volume of material to review and the model’s reasoning ability made the agent genuinely useful for that kind of work. The transition happened function by function, with each team hitting its own tipping point depending on what kind of work the model was good at, and what data or file access it had at the time. The behavioral shift described by engineers there is simple to state and harder to practice: when a task comes up, the instinct is to just do it yourself. The habit that changes outcomes is catching that instinct and routing the task to the agent first, then checking the result rather than doing the work manually by default. Frequently Asked Questions What does “computer use” mean for an AI agent? It means the agent interacts with software the way a human would, by clicking, typing, scrolling, and navigating visual interfaces, rather than only calling structured APIs or protocols like MCP. Is computer use replacing MCP and API integrations? No. OpenAI engineers describe them as complementary. Direct integrations remain faster and more efficient when they exist, but computer use covers everything that lacks a built integration, which is most of the software people deal with day to day. Why did computer use take so long to become useful? Early versions were reliable enough to demonstrate but too slow for practical daily use, which is why one OpenAI engineer called it an “impressive party trick” for a long stretch before speed and reliability improved. Can computer use handle tasks with no digital integration at all, like paper forms or faxes? Based on examples described by OpenAI staff, yes: tasks involving scanned PDFs, government paperwork, and similarly disconnected systems are exactly the kind of work computer use is meant to handle, since those systems will likely never get dedicated API access. Did every team at OpenAI adopt agent-driven workflows at the same time? No. Engineers describe an uneven rollout internally, with coding teams switching first because their work already ran through command-line tools, and other functions like legal adopting agent workflows later as model capability and data access improved.