# Teach Your AI Agent to Browse the Web with Claude Computer Use > Source: > Published: 2026-08-01 17:42:20+00:00 # Teach Your AI Agent to Browse the Web with Claude Computer Use Wire Claude's computer-use tool to a Playwright browser so it can screenshot, click, and type autonomously. [Priya Nair](https://sourcefeed.dev/u/priya_nair) ## What you'll build A Python agent that gives [Claude](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) eyes and hands inside a real Chromium browser: it takes screenshots, clicks, types, and scrolls through [Playwright](https://playwright.dev/python/), looping autonomously until a multi-step web task — searching Wikipedia and reading the answer off the page — is done. ## Prerequisites Versions verified for this tutorial: - Python 3.10+ (tested on 3.13) `anthropic` 0.120.2 and`playwright` 1.62.0 (Chromium 151 headless shell, installed by Playwright)- An Anthropic API key with credit, from the [Claude Console](https://platform.claude.com/)— exported as`ANTHROPIC_API_KEY` - Model `claude-opus-5` with the`computer-use-2025-11-24` beta header Computer use is a beta feature. Everything here runs headless on macOS, Linux, or Windows — no X11 or Docker required. ## 1. Understand what the API actually does Computer use is a client-side tool: Claude never touches your browser. You declare a schema-less `computer_20251124` tool, and Claude responds with `tool_use` blocks containing abstract actions — `screenshot` , `left_click` with a `[x, y]` coordinate, `type` with text, `key` for shortcuts, `scroll` , and so on. Your code executes each action, returns the result (screenshots go back as base64 PNG blocks), and calls the API again. That request → execute → respond cycle repeats while `stop_reason` is `"tool_use"` — the agent loop. Anthropic's reference implementation wraps a whole Linux desktop in Docker — Xvfb, a window manager, Firefox. That's the right call for general desktop automation, but for web-only tasks a Playwright-controlled headless browser is a much smaller surface: no container, no display server, and the screenshot dimensions are exactly what you configure, which matters because Claude's click coordinates are pixel positions on the image you send. One wrinkle for browsers: headless Chromium has no address bar, so Claude can't navigate by typing a URL into a screenshot. The API explicitly supports mixing custom tools into the same `tools` array, so we'll add a tiny `navigate` tool alongside the computer tool. ## 2. Install the dependencies ``` python3 -m venv .venv && source .venv/bin/activate pip install anthropic playwright playwright install chromium export ANTHROPIC_API_KEY="sk-ant-..." ``` `playwright install chromium` downloads a ~95 MB pinned browser build; your system Chrome is not used. ## 3. Write the agent Save this as `browser_agent.py` . It's complete — no elisions. ``` """Minimal computer-use agent that drives a real Chromium browser via Playwright.""" import base64 import sys import anthropic from playwright.sync_api import sync_playwright WIDTH, HEIGHT = 1280, 800 MODEL = "claude-opus-5" COMPUTER_USE_BETA = "computer-use-2025-11-24" MAX_ITERATIONS = 30 TOOLS = [ { "type": "computer_20251124", "name": "computer", "display_width_px": WIDTH, "display_height_px": HEIGHT, }, { "name": "navigate", "description": "Load a URL in the browser. Use this instead of an address bar.", "input_schema": { "type": "object", "properties": {"url": {"type": "string", "description": "Absolute URL to open"}}, "required": ["url"], }, }, ] # Claude emits xdotool-style key names; Playwright uses its own. Map the common ones. KEY_MAP = { "return": "Enter", "enter": "Enter", "tab": "Tab", "escape": "Escape", "esc": "Escape", "backspace": "Backspace", "back_space": "Backspace", "delete": "Delete", "space": "Space", "up": "ArrowUp", "down": "ArrowDown", "left": "ArrowLeft", "right": "ArrowRight", "page_down": "PageDown", "page_up": "PageUp", "home": "Home", "end": "End", "ctrl": "Control", "control": "Control", "alt": "Alt", "shift": "Shift", "super": "Meta", "cmd": "Meta", } def to_playwright_keys(combo: str) -> str: return "+".join(KEY_MAP.get(k.lower(), k) for k in combo.split("+")) class BrowserComputer: def __init__(self, page): self.page = page def screenshot(self): png = self.page.screenshot() return [{ "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": base64.b64encode(png).decode(), }, }] def handle(self, name: str, action: dict): """Execute one tool call and return tool_result content.""" if name == "navigate": self.page.goto(action["url"], wait_until="domcontentloaded") return f"Loaded {self.page.url}" kind = action["action"] if kind == "screenshot": return self.screenshot() if kind in ("left_click", "right_click", "middle_click", "double_click", "triple_click"): x, y = action["coordinate"] button = {"right_click": "right", "middle_click": "middle"}.get(kind, "left") clicks = {"double_click": 2, "triple_click": 3}.get(kind, 1) self.page.mouse.click(x, y, button=button, click_count=clicks) return f"{kind} at ({x}, {y})" if kind == "type": self.page.keyboard.type(action["text"]) return f"Typed {action['text']!r}" if kind == "key": keys = to_playwright_keys(action["text"]) self.page.keyboard.press(keys) return f"Pressed {keys}" if kind == "scroll": x, y = action["coordinate"] amount = action.get("scroll_amount", 3) * 100 dx, dy = {"down": (0, amount), "up": (0, -amount), "right": (amount, 0), "left": (-amount, 0)}[action["scroll_direction"]] self.page.mouse.move(x, y) self.page.mouse.wheel(dx, dy) return f"Scrolled {action['scroll_direction']} at ({x}, {y})" if kind == "mouse_move": x, y = action["coordinate"] self.page.mouse.move(x, y) return f"Moved mouse to ({x}, {y})" if kind == "wait": self.page.wait_for_timeout(action.get("duration", 1) * 1000) return "Waited" return f"Unsupported action: {kind}" def run(task: str): client = anthropic.Anthropic() with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page(viewport={"width": WIDTH, "height": HEIGHT}) computer = BrowserComputer(page) messages = [{"role": "user", "content": task}] for _ in range(MAX_ITERATIONS): response = client.beta.messages.create( model=MODEL, max_tokens=4096, tools=TOOLS, messages=messages, betas=[COMPUTER_USE_BETA], ) messages.append({"role": "assistant", "content": response.content}) tool_results = [] for block in response.content: if block.type == "text": print(f"[claude] {block.text}") elif block.type == "tool_use": print(f"[action] {block.name}: {block.input}") try: result = computer.handle(block.name, block.input) tool_results.append({ "type": "tool_result", "tool_use_id": block.id, "content": result, }) except Exception as exc: tool_results.append({ "type": "tool_result", "tool_use_id": block.id, "content": f"Error: {exc}", "is_error": True, }) if response.stop_reason != "tool_use": break messages.append({"role": "user", "content": tool_results}) browser.close() if __name__ == "__main__": run(sys.argv[1] if len(sys.argv) > 1 else "Navigate to https://en.wikipedia.org, use the search box to search for " "'Alan Turing', and tell me his year of birth. Take a screenshot after " "each step to verify it worked.") ``` Four details are load-bearing: **The viewport must exactly match** Claude's click coordinates are pixel positions on the screenshot you sent; any mismatch and every click lands off-target.`display_width_px` /`display_height_px` .**Key names need translation.** Claude was trained on xdotool keysyms (`Return` ,`ctrl+l` ,`Page_Down` ); Playwright wants`Enter` ,`Control+l` ,`PageDown` . The`KEY_MAP` bridges them.**Failures go back as**, not a crash — Claude reads the error and tries another approach.`tool_result` with`is_error: true` **Append the full** On`response.content` back into`messages` .`claude-opus-5` thinking is on by default, and thinking blocks must be returned unchanged for the next turn to be accepted. The screenshot-after-each-step instruction in the default task is lifted from Anthropic's prompting guidance — without it, Claude sometimes assumes an action worked instead of checking. ## 4. Run it ``` python browser_agent.py ``` Each loop iteration costs one API call carrying the conversation so far, and every screenshot is roughly a thousand input tokens, so a typical run of this task is 6–10 iterations. `MAX_ITERATIONS` is your cost circuit breaker. If you later move to an older model to cut cost, Anthropic's docs note that for computer use specifically, `medium` effort is the accuracy-to-cost sweet spot on Sonnet 4.6 and Opus 4.6 (set via `output_config={"effort": "medium"}` ), while `max` adds tokens without improving UI accuracy. ## Verify it works You should see an action trace like this (Claude's exact wording and coordinates will vary): ``` [claude] I'll navigate to Wikipedia first. [action] navigate: {'url': 'https://en.wikipedia.org'} [action] computer: {'action': 'screenshot'} [action] computer: {'action': 'left_click', 'coordinate': [583, 384]} [action] computer: {'action': 'type', 'text': 'Alan Turing'} [action] computer: {'action': 'key', 'text': 'Return'} [action] computer: {'action': 'screenshot'} [claude] Alan Turing was born in 1912 (23 June 1912, in Maida Vale, London). ``` Success looks like: a `navigate` call, at least one screenshot, click/type/key actions on the search box, and a final `[claude]` text line containing **1912** with no further `[action]` lines after it (meaning `stop_reason` flipped from `tool_use` to `end_turn` ). ## Troubleshooting ** BrowserType.launch: Executable doesn't exist at .../chrome-headless-shell followed by "Looks like Playwright was just installed or updated."** You installed the `playwright` package but not the browser, or installed it under a different virtualenv. Run `playwright install chromium` from the same venv that runs the script.** anthropic.BadRequestError: Error code: 400 saying tools.0.type: Input tag 'computer_20251124' ... does not match any of the expected tags.** The tool version and beta header are mismatched. `computer_20251124` requires `betas=["computer-use-2025-11-24"]` and a supporting model (Claude Opus 5, Sonnet 5, Opus 4.8/4.7/4.6, Sonnet 4.6, or Opus 4.5). Older models like Sonnet 4.5 and Haiku 4.5 need the `computer_20250124` tool with `computer-use-2025-01-24` instead.`anthropic.AuthenticationError: Error code: 401 - {'type': 'error', 'error': {'type': 'authentication_error', 'message': 'invalid x-api-key'}}` .`ANTHROPIC_API_KEY` is unset, stale, or not exported into the shell running the script. `echo $ANTHROPIC_API_KEY` to confirm, and re-export after opening a new terminal. **Clicks consistently land in the wrong place.** Your screenshots aren't the size you declared. This happens when adapting the code to a headed browser on a Retina/HiDPI display, where screenshots come back at 2x scale. Keep `device_scale_factor` at 1 (the headless default) or downscale screenshots to exactly `WIDTH×HEIGHT` before sending. ## Next steps Swap the default task for your own — form filling, price checks, multi-page research — but keep two of Anthropic's warnings in mind: content on web pages can prompt-inject the agent (the API runs injection classifiers on screenshots and will steer Claude to ask for confirmation), and anything consequential — purchases, logins, accepting terms — should require human sign-off. From here, try [Anthropic's computer-use reference implementation](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo), a Dockerized full Linux desktop with Firefox and a web UI; add `"enable_zoom": true` to the tool definition so Claude can inspect small text at full resolution; or bolt the schema-less `bash_20250124` and `text_editor_20250728` tools into the same `tools` array so the agent can save what it finds to disk. ## Sources & further reading - [Computer use tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool)— platform.claude.com - [Computer use reference implementation](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo)— github.com - [Playwright for Python - Installation](https://playwright.dev/python/docs/intro)— playwright.dev - [computer_20250124 does not match any of expected tags](https://github.com/anthropics/anthropic-quickstarts/issues/251)— github.com [Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. ## Discussion 0 No comments yet Be the first to weigh in.