{"slug": "teach-your-ai-agent-to-browse-the-web-with-claude-computer-use", "title": "Teach Your AI Agent to Browse the Web with Claude Computer Use", "summary": "Anthropic's Claude computer-use tool can now drive a real Chromium browser via Playwright, enabling autonomous web browsing through screenshots, clicks, and typing, as demonstrated in a tutorial by Priya Nair. The Python agent uses the claude-opus-5 model with the computer-use-2025-11-24 beta header and a custom navigate tool to perform multi-step tasks like searching Wikipedia. The setup requires Python 3.10+, anthropic 0.120.2, and playwright 1.62.0, and runs headless on macOS, Linux, or Windows without Docker.", "body_md": "# Teach Your AI Agent to Browse the Web with Claude Computer Use\n\nWire Claude's computer-use tool to a Playwright browser so it can screenshot, click, and type autonomously.\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)\n\n## What you'll build\n\nA Python agent that gives [Claude](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) eyes and hands inside a real Chromium browser: it takes screenshots, clicks, types, and scrolls through [Playwright](https://playwright.dev/python/), looping autonomously until a multi-step web task — searching Wikipedia and reading the answer off the page — is done.\n\n## Prerequisites\n\nVersions verified for this tutorial:\n\n- Python 3.10+ (tested on 3.13)\n`anthropic`\n\n0.120.2 and`playwright`\n\n1.62.0 (Chromium 151 headless shell, installed by Playwright)- An Anthropic API key with credit, from the\n[Claude Console](https://platform.claude.com/)— exported as`ANTHROPIC_API_KEY`\n\n- Model\n`claude-opus-5`\n\nwith the`computer-use-2025-11-24`\n\nbeta header\n\nComputer use is a beta feature. Everything here runs headless on macOS, Linux, or Windows — no X11 or Docker required.\n\n## 1. Understand what the API actually does\n\nComputer use is a client-side tool: Claude never touches your browser. You declare a schema-less `computer_20251124`\n\ntool, and Claude responds with `tool_use`\n\nblocks containing abstract actions — `screenshot`\n\n, `left_click`\n\nwith a `[x, y]`\n\ncoordinate, `type`\n\nwith text, `key`\n\nfor shortcuts, `scroll`\n\n, and so on. Your code executes each action, returns the result (screenshots go back as base64 PNG blocks), and calls the API again. That request → execute → respond cycle repeats while `stop_reason`\n\nis `\"tool_use\"`\n\n— the agent loop.\n\nAnthropic's reference implementation wraps a whole Linux desktop in Docker — Xvfb, a window manager, Firefox. That's the right call for general desktop automation, but for web-only tasks a Playwright-controlled headless browser is a much smaller surface: no container, no display server, and the screenshot dimensions are exactly what you configure, which matters because Claude's click coordinates are pixel positions on the image you send.\n\nOne wrinkle for browsers: headless Chromium has no address bar, so Claude can't navigate by typing a URL into a screenshot. The API explicitly supports mixing custom tools into the same `tools`\n\narray, so we'll add a tiny `navigate`\n\ntool alongside the computer tool.\n\n## 2. Install the dependencies\n\n```\npython3 -m venv .venv && source .venv/bin/activate\npip install anthropic playwright\nplaywright install chromium\nexport ANTHROPIC_API_KEY=\"sk-ant-...\"\n```\n\n`playwright install chromium`\n\ndownloads a ~95 MB pinned browser build; your system Chrome is not used.\n\n## 3. Write the agent\n\nSave this as `browser_agent.py`\n\n. It's complete — no elisions.\n\n```\n\"\"\"Minimal computer-use agent that drives a real Chromium browser via Playwright.\"\"\"\n\nimport base64\nimport sys\n\nimport anthropic\nfrom playwright.sync_api import sync_playwright\n\nWIDTH, HEIGHT = 1280, 800\nMODEL = \"claude-opus-5\"\nCOMPUTER_USE_BETA = \"computer-use-2025-11-24\"\nMAX_ITERATIONS = 30\n\nTOOLS = [\n    {\n        \"type\": \"computer_20251124\",\n        \"name\": \"computer\",\n        \"display_width_px\": WIDTH,\n        \"display_height_px\": HEIGHT,\n    },\n    {\n        \"name\": \"navigate\",\n        \"description\": \"Load a URL in the browser. Use this instead of an address bar.\",\n        \"input_schema\": {\n            \"type\": \"object\",\n            \"properties\": {\"url\": {\"type\": \"string\", \"description\": \"Absolute URL to open\"}},\n            \"required\": [\"url\"],\n        },\n    },\n]\n\n# Claude emits xdotool-style key names; Playwright uses its own. Map the common ones.\nKEY_MAP = {\n    \"return\": \"Enter\", \"enter\": \"Enter\", \"tab\": \"Tab\", \"escape\": \"Escape\", \"esc\": \"Escape\",\n    \"backspace\": \"Backspace\", \"back_space\": \"Backspace\", \"delete\": \"Delete\", \"space\": \"Space\",\n    \"up\": \"ArrowUp\", \"down\": \"ArrowDown\", \"left\": \"ArrowLeft\", \"right\": \"ArrowRight\",\n    \"page_down\": \"PageDown\", \"page_up\": \"PageUp\", \"home\": \"Home\", \"end\": \"End\",\n    \"ctrl\": \"Control\", \"control\": \"Control\", \"alt\": \"Alt\", \"shift\": \"Shift\",\n    \"super\": \"Meta\", \"cmd\": \"Meta\",\n}\n\ndef to_playwright_keys(combo: str) -> str:\n    return \"+\".join(KEY_MAP.get(k.lower(), k) for k in combo.split(\"+\"))\n\nclass BrowserComputer:\n    def __init__(self, page):\n        self.page = page\n\n    def screenshot(self):\n        png = self.page.screenshot()\n        return [{\n            \"type\": \"image\",\n            \"source\": {\n                \"type\": \"base64\",\n                \"media_type\": \"image/png\",\n                \"data\": base64.b64encode(png).decode(),\n            },\n        }]\n\n    def handle(self, name: str, action: dict):\n        \"\"\"Execute one tool call and return tool_result content.\"\"\"\n        if name == \"navigate\":\n            self.page.goto(action[\"url\"], wait_until=\"domcontentloaded\")\n            return f\"Loaded {self.page.url}\"\n\n        kind = action[\"action\"]\n        if kind == \"screenshot\":\n            return self.screenshot()\n        if kind in (\"left_click\", \"right_click\", \"middle_click\", \"double_click\", \"triple_click\"):\n            x, y = action[\"coordinate\"]\n            button = {\"right_click\": \"right\", \"middle_click\": \"middle\"}.get(kind, \"left\")\n            clicks = {\"double_click\": 2, \"triple_click\": 3}.get(kind, 1)\n            self.page.mouse.click(x, y, button=button, click_count=clicks)\n            return f\"{kind} at ({x}, {y})\"\n        if kind == \"type\":\n            self.page.keyboard.type(action[\"text\"])\n            return f\"Typed {action['text']!r}\"\n        if kind == \"key\":\n            keys = to_playwright_keys(action[\"text\"])\n            self.page.keyboard.press(keys)\n            return f\"Pressed {keys}\"\n        if kind == \"scroll\":\n            x, y = action[\"coordinate\"]\n            amount = action.get(\"scroll_amount\", 3) * 100\n            dx, dy = {\"down\": (0, amount), \"up\": (0, -amount),\n                      \"right\": (amount, 0), \"left\": (-amount, 0)}[action[\"scroll_direction\"]]\n            self.page.mouse.move(x, y)\n            self.page.mouse.wheel(dx, dy)\n            return f\"Scrolled {action['scroll_direction']} at ({x}, {y})\"\n        if kind == \"mouse_move\":\n            x, y = action[\"coordinate\"]\n            self.page.mouse.move(x, y)\n            return f\"Moved mouse to ({x}, {y})\"\n        if kind == \"wait\":\n            self.page.wait_for_timeout(action.get(\"duration\", 1) * 1000)\n            return \"Waited\"\n        return f\"Unsupported action: {kind}\"\n\ndef run(task: str):\n    client = anthropic.Anthropic()\n    with sync_playwright() as p:\n        browser = p.chromium.launch(headless=True)\n        page = browser.new_page(viewport={\"width\": WIDTH, \"height\": HEIGHT})\n        computer = BrowserComputer(page)\n        messages = [{\"role\": \"user\", \"content\": task}]\n\n        for _ in range(MAX_ITERATIONS):\n            response = client.beta.messages.create(\n                model=MODEL,\n                max_tokens=4096,\n                tools=TOOLS,\n                messages=messages,\n                betas=[COMPUTER_USE_BETA],\n            )\n            messages.append({\"role\": \"assistant\", \"content\": response.content})\n\n            tool_results = []\n            for block in response.content:\n                if block.type == \"text\":\n                    print(f\"[claude] {block.text}\")\n                elif block.type == \"tool_use\":\n                    print(f\"[action] {block.name}: {block.input}\")\n                    try:\n                        result = computer.handle(block.name, block.input)\n                        tool_results.append({\n                            \"type\": \"tool_result\",\n                            \"tool_use_id\": block.id,\n                            \"content\": result,\n                        })\n                    except Exception as exc:\n                        tool_results.append({\n                            \"type\": \"tool_result\",\n                            \"tool_use_id\": block.id,\n                            \"content\": f\"Error: {exc}\",\n                            \"is_error\": True,\n                        })\n\n            if response.stop_reason != \"tool_use\":\n                break\n            messages.append({\"role\": \"user\", \"content\": tool_results})\n\n        browser.close()\n\nif __name__ == \"__main__\":\n    run(sys.argv[1] if len(sys.argv) > 1 else\n        \"Navigate to https://en.wikipedia.org, use the search box to search for \"\n        \"'Alan Turing', and tell me his year of birth. Take a screenshot after \"\n        \"each step to verify it worked.\")\n```\n\nFour details are load-bearing:\n\n**The viewport must exactly match** Claude's click coordinates are pixel positions on the screenshot you sent; any mismatch and every click lands off-target.`display_width_px`\n\n/`display_height_px`\n\n.**Key names need translation.** Claude was trained on xdotool keysyms (`Return`\n\n,`ctrl+l`\n\n,`Page_Down`\n\n); Playwright wants`Enter`\n\n,`Control+l`\n\n,`PageDown`\n\n. The`KEY_MAP`\n\nbridges them.**Failures go back as**, not a crash — Claude reads the error and tries another approach.`tool_result`\n\nwith`is_error: true`\n\n**Append the full** On`response.content`\n\nback into`messages`\n\n.`claude-opus-5`\n\nthinking is on by default, and thinking blocks must be returned unchanged for the next turn to be accepted.\n\nThe screenshot-after-each-step instruction in the default task is lifted from Anthropic's prompting guidance — without it, Claude sometimes assumes an action worked instead of checking.\n\n## 4. Run it\n\n```\npython browser_agent.py\n```\n\nEach loop iteration costs one API call carrying the conversation so far, and every screenshot is roughly a thousand input tokens, so a typical run of this task is 6–10 iterations. `MAX_ITERATIONS`\n\nis your cost circuit breaker. If you later move to an older model to cut cost, Anthropic's docs note that for computer use specifically, `medium`\n\neffort is the accuracy-to-cost sweet spot on Sonnet 4.6 and Opus 4.6 (set via `output_config={\"effort\": \"medium\"}`\n\n), while `max`\n\nadds tokens without improving UI accuracy.\n\n## Verify it works\n\nYou should see an action trace like this (Claude's exact wording and coordinates will vary):\n\n```\n[claude] I'll navigate to Wikipedia first.\n[action] navigate: {'url': 'https://en.wikipedia.org'}\n[action] computer: {'action': 'screenshot'}\n[action] computer: {'action': 'left_click', 'coordinate': [583, 384]}\n[action] computer: {'action': 'type', 'text': 'Alan Turing'}\n[action] computer: {'action': 'key', 'text': 'Return'}\n[action] computer: {'action': 'screenshot'}\n[claude] Alan Turing was born in 1912 (23 June 1912, in Maida Vale, London).\n```\n\nSuccess looks like: a `navigate`\n\ncall, at least one screenshot, click/type/key actions on the search box, and a final `[claude]`\n\ntext line containing **1912** with no further `[action]`\n\nlines after it (meaning `stop_reason`\n\nflipped from `tool_use`\n\nto `end_turn`\n\n).\n\n## Troubleshooting\n\n** BrowserType.launch: Executable doesn't exist at .../chrome-headless-shell followed by \"Looks like Playwright was just installed or updated.\"** You installed the\n\n`playwright`\n\npackage but not the browser, or installed it under a different virtualenv. Run `playwright install chromium`\n\nfrom the same venv that runs the script.** anthropic.BadRequestError: Error code: 400 saying tools.0.type: Input tag 'computer_20251124' ... does not match any of the expected tags.** The tool version and beta header are mismatched.\n\n`computer_20251124`\n\nrequires `betas=[\"computer-use-2025-11-24\"]`\n\nand a supporting model (Claude Opus 5, Sonnet 5, Opus 4.8/4.7/4.6, Sonnet 4.6, or Opus 4.5). Older models like Sonnet 4.5 and Haiku 4.5 need the `computer_20250124`\n\ntool with `computer-use-2025-01-24`\n\ninstead.`anthropic.AuthenticationError: Error code: 401 - {'type': 'error', 'error': {'type': 'authentication_error', 'message': 'invalid x-api-key'}}`\n\n.`ANTHROPIC_API_KEY`\n\nis unset, stale, or not exported into the shell running the script. `echo $ANTHROPIC_API_KEY`\n\nto confirm, and re-export after opening a new terminal.\n\n**Clicks consistently land in the wrong place.** Your screenshots aren't the size you declared. This happens when adapting the code to a headed browser on a Retina/HiDPI display, where screenshots come back at 2x scale. Keep `device_scale_factor`\n\nat 1 (the headless default) or downscale screenshots to exactly `WIDTH×HEIGHT`\n\nbefore sending.\n\n## Next steps\n\nSwap the default task for your own — form filling, price checks, multi-page research — but keep two of Anthropic's warnings in mind: content on web pages can prompt-inject the agent (the API runs injection classifiers on screenshots and will steer Claude to ask for confirmation), and anything consequential — purchases, logins, accepting terms — should require human sign-off. From here, try [Anthropic's computer-use reference implementation](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo), a Dockerized full Linux desktop with Firefox and a web UI; add `\"enable_zoom\": true`\n\nto the tool definition so Claude can inspect small text at full resolution; or bolt the schema-less `bash_20250124`\n\nand `text_editor_20250728`\n\ntools into the same `tools`\n\narray so the agent can save what it finds to disk.\n\n## Sources & further reading\n\n-\n[Computer use tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool)— platform.claude.com -\n[Computer use reference implementation](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo)— github.com -\n[Playwright for Python - Installation](https://playwright.dev/python/docs/intro)— playwright.dev -\n[computer_20250124 does not match any of expected tags](https://github.com/anthropics/anthropic-quickstarts/issues/251)— github.com\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer\n\nPriya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/teach-your-ai-agent-to-browse-the-web-with-claude-computer-use", "canonical_source": "https://sourcefeed.dev/a/teach-your-ai-agent-to-browse-the-web-with-claude-computer-use", "published_at": "2026-08-01 17:42:20+00:00", "updated_at": "2026-08-01 17:55:28.408611+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": ["Anthropic", "Claude", "Playwright", "Priya Nair", "Chromium", "Python", "Wikipedia"], "alternates": {"html": "https://wpnews.pro/news/teach-your-ai-agent-to-browse-the-web-with-claude-computer-use", "markdown": "https://wpnews.pro/news/teach-your-ai-agent-to-browse-the-web-with-claude-computer-use.md", "text": "https://wpnews.pro/news/teach-your-ai-agent-to-browse-the-web-with-claude-computer-use.txt", "jsonld": "https://wpnews.pro/news/teach-your-ai-agent-to-browse-the-web-with-claude-computer-use.jsonld"}}