{"slug": "openai-computer-using-agent-sample-app", "title": "OpenAI Computer Using Agent Sample App", "summary": "OpenAI released an open-source sample app on GitHub demonstrating computer-use agents that operate software through code, with two implementations: a JavaScript agent using Playwright to control a browser and a Python agent using PyAutoGUI to control a desktop via screenshots and keystrokes. The repository, openai-cua-sample-app, requires Node.js 22.20.0, an OpenAI API key, and pnpm 10.26.0, and includes shared labs for scenarios like kanban boards and hotel booking.", "body_md": "Computer-use agents are AI systems that operate software to complete a task. They inspect an interface, choose an action, execute it, and check the result. At OpenAI, we build this loop around models that write code to interact with software. Code lets the model combine actions, process observations, and choose when to look again.\n\nA persistent runtime keeps useful state and helper functions available between calls. The model can write a loop to fill several fields, check that each change took effect, and return only the text or screenshots it needs. This can reduce model round trips and repeated input context while giving the model feedback to correct mistakes.\n\nThis repository contains two computer-use agents: one runs JavaScript with **Playwright** to control a browser; the other runs Python with **PyAutoGUI** to control a desktop through screenshots, mouse input, and keystrokes. Both use the Responses API and the same console and labs. They show how to build this pattern with standard libraries; OpenAI products use their own runtimes and additional controls.\n\nYou need **Node.js 22.20.0** and an **[OpenAI API key](https://platform.openai.com/api-keys)** with access to the configured model. Corepack uses the repository's pinned **pnpm 10.26.0**. Run these commands in your terminal, replacing the API key placeholder:\n\n```\ngit clone https://github.com/openai/openai-cua-sample-app.git\ncd openai-cua-sample-app\ncorepack enable\npnpm install --frozen-lockfile\ncp .env.example .env\nexport OPENAI_API_KEY=\"your_api_key_here\"\n```\n\nBoth apps read the root `.env`; shell variables take precedence. You can save the key in `.env` to use it in later sessions.\n\nRun either quickstart below from the repository root. Once it starts, open the [console](http://127.0.0.1:3000), choose a scenario, edit its prompt, and select **Start Run**. Runs make real API calls. Use **Stop** to interrupt a run; use **Ctrl+C** and wait for shutdown before switching apps.\n\nThe JavaScript agent uses Playwright locators, screenshots, and browser controls in a persistent session, with a TypeScript server and agent loop. See the [JavaScript README](/openai/openai-cua-sample-app/blob/main/javascript-app/README.md) for the code walkthrough and recovery guidance.\n\n```\npnpm playwright:install\npnpm dev:js\n```\n\nThe Python agent runs its server, agent loop, and persistent PyAutoGUI worker in Python, controlling a visible browser on your desktop. Install **uv**, use **Python 3.10+** (the repo pins 3.12), and follow the [Python README](/openai/openai-cua-sample-app/blob/main/python-app/README.md#quickstart) for desktop permissions, platform setup, and a code walkthrough.\n\n```\npnpm python:install\npnpm python:playwright:install\nuv run --project python-app python -m app.desktop.worker --check\npnpm dev:python\n.\n|-- console/                           Shared web console\n|-- contracts/                         Shared request, event, and replay types\n|-- javascript-app/\n|   |-- src/\n|   |   |-- responses-loop.ts          Core agent loop: model, code, feedback\n|   |   |-- javascript-worker.ts       Persistent JavaScript execution worker\n|   |   |-- browser/                   Browser session and worker communication\n|   |   |-- runner-manager.ts          Run lifecycle and saved artifacts\n|   |   `-- lab-catalog.ts             Shared lab catalog access\n|   |-- docs/                          Architecture and contributing guides\n|   `-- tests/                         JavaScript tests\n|-- labs/\n|   |-- catalog.json                   Scenario defaults and task prompts\n|   |-- kanban-lab-template/           Project board: cards, columns, and tasks\n|   |-- paint-lab-template/            Drawing canvas: shapes, layers, and tools\n|   |-- booking-lab-template/          Hotel search and mock reservations\n|   |-- docs/                          Task examples and lab guides\n|   `-- tests/                         Lab and integration tests\n|-- python-app/\n|   |-- app/\n|   |   |-- responses_loop.py          Core agent loop: model, code, feedback\n|   |   |-- desktop/\n|   |   |   |-- worker.py              Persistent Python execution worker\n|   |   |   `-- runtime.py             Worker lifecycle and interruption\n|   |   |-- runner.py                  Run lifecycle and saved artifacts\n|   |   `-- lab_catalog.py             Shared lab catalog access\n|   |-- docs/                          Architecture and contributing guides\n|   `-- tests/                         Python tests\n`-- scripts/\n    `-- launch.mjs                     Starts one app and the shared console\n```\n\nEach run gets a fresh copy of a lab template. Workspaces, screenshots, and replays stay in the selected app's ignored `data/` directory. Use **Replay JSON** to inspect the recorded trace and the screenshot timeline to review earlier environment states. Generated dependencies, builds, and caches are omitted above. See the [lab guide](/openai/openai-cua-sample-app/blob/main/labs/docs/README.md) for task examples.\n\n**Run finished** means the agent loop and cleanup ended normally. Inspect the screenshots, recorded trace, and model response to judge whether the requested task was accomplished.\n\n- Generated code runs with your user permissions. These samples do not provide an operating-system sandbox or OpenAI's production action-review controls. Use the labs or other environments you control.\n- Python controls your real mouse and keyboard, and screenshots may include other windows. Use a dedicated desktop session and keep the lab window in front on your primary monitor.\n- A final answer does not prove the task succeeded. Inspect the result before relying on it.\n- Keep API keys and run artifacts private. Environment files and generated run data are excluded from Git; keep services on their default loopback addresses.\n- A crash can leave desktop input held down. Read the Python app's [interruption and recovery notes](/openai/openai-cua-sample-app/blob/main/python-app/README.md#interruption-and-recovery) before using it.\n\nLicensed under the [MIT License](/openai/openai-cua-sample-app/blob/main/LICENSE).", "url": "https://wpnews.pro/news/openai-computer-using-agent-sample-app", "canonical_source": "https://github.com/openai/openai-cua-sample-app", "published_at": "2026-09-07 05:20:54+00:00", "updated_at": "2026-09-07 05:56:14.438719+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "ai-research"], "entities": ["OpenAI", "Playwright", "PyAutoGUI", "Node.js", "pnpm", "openai-cua-sample-app"], "alternates": {"html": "https://wpnews.pro/news/openai-computer-using-agent-sample-app", "markdown": "https://wpnews.pro/news/openai-computer-using-agent-sample-app.md", "text": "https://wpnews.pro/news/openai-computer-using-agent-sample-app.txt", "jsonld": "https://wpnews.pro/news/openai-computer-using-agent-sample-app.jsonld"}}