Teach Your AI Agent to Browse the Web with Claude Computer Use

Anthropic's Claude computer-use tool can now drive a real Chromium browser via Playwright, enabling autonomous web browsing through screenshots, clicks, and typing, as demonstrated in a tutorial by Priya Nair. The Python agent uses the claude-opus-5 model with the computer-use-2025-11-24 beta header and a custom navigate tool to perform multi-step tasks like searching Wikipedia. The setup requires Python 3.10+, anthropic 0.120.2, and playwright 1.62.0, and runs headless on macOS, Linux, or Windows without Docker.

Teach Your AI Agent to Browse the Web with Claude Computer Use Wire Claude's computer-use tool to a Playwright browser so it can screenshot, click, and type autonomously. Priya Nair https://sourcefeed.dev/u/priya nair What you'll build A Python agent that gives Claude https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool eyes and hands inside a real Chromium browser: it takes screenshots, clicks, types, and scrolls through Playwright https://playwright.dev/python/ , looping autonomously until a multi-step web task — searching Wikipedia and reading the answer off the page — is done. Prerequisites Versions verified for this tutorial: - Python 3.10+ tested on 3.13 anthropic 0.120.2 and playwright 1.62.0 Chromium 151 headless shell, installed by Playwright - An Anthropic API key with credit, from the Claude Console https://platform.claude.com/ — exported as ANTHROPIC API KEY - Model claude-opus-5 with the computer-use-2025-11-24 beta header Computer use is a beta feature. Everything here runs headless on macOS, Linux, or Windows — no X11 or Docker required. 1. Understand what the API actually does Computer use is a client-side tool: Claude never touches your browser. You declare a schema-less computer 20251124 tool, and Claude responds with tool use blocks containing abstract actions — screenshot , left click with a x, y coordinate, type with text, key for shortcuts, scroll , and so on. Your code executes each action, returns the result screenshots go back as base64 PNG blocks , and calls the API again. That request → execute → respond cycle repeats while stop reason is "tool use" — the agent loop. Anthropic's reference implementation wraps a whole Linux desktop in Docker — Xvfb, a window manager, Firefox. That's the right call for general desktop automation, but for web-only tasks a Playwright-controlled headless browser is a much smaller surface: no container, no display server, and the screenshot dimensions are exactly what you configure, which matters because Claude's click coordinates are pixel positions on the image you send. One wrinkle for browsers: headless Chromium has no address bar, so Claude can't navigate by typing a URL into a screenshot. The API explicitly supports mixing custom tools into the same tools array, so we'll add a tiny navigate tool alongside the computer tool. 2. Install the dependencies python3 -m venv .venv && source .venv/bin/activate pip install anthropic playwright playwright install chromium export ANTHROPIC API KEY="sk-ant-..." playwright install chromium downloads a ~95 MB pinned browser build; your system Chrome is not used. 3. Write the agent Save this as browser agent.py . It's complete — no elisions. """Minimal computer-use agent that drives a real Chromium browser via Playwright.""" import base64 import sys import anthropic from playwright.sync api import sync playwright WIDTH, HEIGHT = 1280, 800 MODEL = "claude-opus-5" COMPUTER USE BETA = "computer-use-2025-11-24" MAX ITERATIONS = 30 TOOLS = { "type": "computer 20251124", "name": "computer", "display width px": WIDTH, "display height px": HEIGHT, }, { "name": "navigate", "description": "Load a URL in the browser. Use this instead of an address bar.", "input schema": { "type": "object", "properties": {"url": {"type": "string", "description": "Absolute URL to open"}}, "required": "url" , }, }, Claude emits xdotool-style key names; Playwright uses its own. Map the common ones. KEY MAP = { "return": "Enter", "enter": "Enter", "tab": "Tab", "escape": "Escape", "esc": "Escape", "backspace": "Backspace", "back space": "Backspace", "delete": "Delete", "space": "Space", "up": "ArrowUp", "down": "ArrowDown", "left": "ArrowLeft", "right": "ArrowRight", "page down": "PageDown", "page up": "PageUp", "home": "Home", "end": "End", "ctrl": "Control", "control": "Control", "alt": "Alt", "shift": "Shift", "super": "Meta", "cmd": "Meta", } def to playwright keys combo: str - str: return "+".join KEY MAP.get k.lower , k for k in combo.split "+" class BrowserComputer: def init self, page : self.page = page def screenshot self : png = self.page.screenshot return { "type": "image", "source": { "type": "base64", "media type": "image/png", "data": base64.b64encode png .decode , }, } def handle self, name: str, action: dict : """Execute one tool call and return tool result content.""" if name == "navigate": self.page.goto action "url" , wait until="domcontentloaded" return f"Loaded {self.page.url}" kind = action "action" if kind == "screenshot": return self.screenshot if kind in "left click", "right click", "middle click", "double click", "triple click" : x, y = action "coordinate" button = {"right click": "right", "middle click": "middle"}.get kind, "left" clicks = {"double click": 2, "triple click": 3}.get kind, 1 self.page.mouse.click x, y, button=button, click count=clicks return f"{kind} at {x}, {y} " if kind == "type": self.page.keyboard.type action "text" return f"Typed {action 'text' r}" if kind == "key": keys = to playwright keys action "text" self.page.keyboard.press keys return f"Pressed {keys}" if kind == "scroll": x, y = action "coordinate" amount = action.get "scroll amount", 3 100 dx, dy = {"down": 0, amount , "up": 0, -amount , "right": amount, 0 , "left": -amount, 0 } action "scroll direction" self.page.mouse.move x, y self.page.mouse.wheel dx, dy return f"Scrolled {action 'scroll direction' } at {x}, {y} " if kind == "mouse move": x, y = action "coordinate" self.page.mouse.move x, y return f"Moved mouse to {x}, {y} " if kind == "wait": self.page.wait for timeout action.get "duration", 1 1000 return "Waited" return f"Unsupported action: {kind}" def run task: str : client = anthropic.Anthropic with sync playwright as p: browser = p.chromium.launch headless=True page = browser.new page viewport={"width": WIDTH, "height": HEIGHT} computer = BrowserComputer page messages = {"role": "user", "content": task} for in range MAX ITERATIONS : response = client.beta.messages.create model=MODEL, max tokens=4096, tools=TOOLS, messages=messages, betas= COMPUTER USE BETA , messages.append {"role": "assistant", "content": response.content} tool results = for block in response.content: if block.type == "text": print f" claude {block.text}" elif block.type == "tool use": print f" action {block.name}: {block.input}" try: result = computer.handle block.name, block.input tool results.append { "type": "tool result", "tool use id": block.id, "content": result, } except Exception as exc: tool results.append { "type": "tool result", "tool use id": block.id, "content": f"Error: {exc}", "is error": True, } if response.stop reason = "tool use": break messages.append {"role": "user", "content": tool results} browser.close if name == " main ": run sys.argv 1 if len sys.argv 1 else "Navigate to https://en.wikipedia.org, use the search box to search for " "'Alan Turing', and tell me his year of birth. Take a screenshot after " "each step to verify it worked." Four details are load-bearing: The viewport must exactly match Claude's click coordinates are pixel positions on the screenshot you sent; any mismatch and every click lands off-target. display width px / display height px . Key names need translation. Claude was trained on xdotool keysyms Return , ctrl+l , Page Down ; Playwright wants Enter , Control+l , PageDown . The KEY MAP bridges them. Failures go back as , not a crash — Claude reads the error and tries another approach. tool result with is error: true Append the full On response.content back into messages . claude-opus-5 thinking is on by default, and thinking blocks must be returned unchanged for the next turn to be accepted. The screenshot-after-each-step instruction in the default task is lifted from Anthropic's prompting guidance — without it, Claude sometimes assumes an action worked instead of checking. 4. Run it python browser agent.py Each loop iteration costs one API call carrying the conversation so far, and every screenshot is roughly a thousand input tokens, so a typical run of this task is 6–10 iterations. MAX ITERATIONS is your cost circuit breaker. If you later move to an older model to cut cost, Anthropic's docs note that for computer use specifically, medium effort is the accuracy-to-cost sweet spot on Sonnet 4.6 and Opus 4.6 set via output config={"effort": "medium"} , while max adds tokens without improving UI accuracy. Verify it works You should see an action trace like this Claude's exact wording and coordinates will vary : claude I'll navigate to Wikipedia first. action navigate: {'url': 'https://en.wikipedia.org'} action computer: {'action': 'screenshot'} action computer: {'action': 'left click', 'coordinate': 583, 384 } action computer: {'action': 'type', 'text': 'Alan Turing'} action computer: {'action': 'key', 'text': 'Return'} action computer: {'action': 'screenshot'} claude Alan Turing was born in 1912 23 June 1912, in Maida Vale, London . Success looks like: a navigate call, at least one screenshot, click/type/key actions on the search box, and a final claude text line containing 1912 with no further action lines after it meaning stop reason flipped from tool use to end turn . Troubleshooting BrowserType.launch: Executable doesn't exist at .../chrome-headless-shell followed by "Looks like Playwright was just installed or updated." You installed the playwright package but not the browser, or installed it under a different virtualenv. Run playwright install chromium from the same venv that runs the script. anthropic.BadRequestError: Error code: 400 saying tools.0.type: Input tag 'computer 20251124' ... does not match any of the expected tags. The tool version and beta header are mismatched. computer 20251124 requires betas= "computer-use-2025-11-24" and a supporting model Claude Opus 5, Sonnet 5, Opus 4.8/4.7/4.6, Sonnet 4.6, or Opus 4.5 . Older models like Sonnet 4.5 and Haiku 4.5 need the computer 20250124 tool with computer-use-2025-01-24 instead. anthropic.AuthenticationError: Error code: 401 - {'type': 'error', 'error': {'type': 'authentication error', 'message': 'invalid x-api-key'}} . ANTHROPIC API KEY is unset, stale, or not exported into the shell running the script. echo $ANTHROPIC API KEY to confirm, and re-export after opening a new terminal. Clicks consistently land in the wrong place. Your screenshots aren't the size you declared. This happens when adapting the code to a headed browser on a Retina/HiDPI display, where screenshots come back at 2x scale. Keep device scale factor at 1 the headless default or downscale screenshots to exactly WIDTH×HEIGHT before sending. Next steps Swap the default task for your own — form filling, price checks, multi-page research — but keep two of Anthropic's warnings in mind: content on web pages can prompt-inject the agent the API runs injection classifiers on screenshots and will steer Claude to ask for confirmation , and anything consequential — purchases, logins, accepting terms — should require human sign-off. From here, try Anthropic's computer-use reference implementation https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo , a Dockerized full Linux desktop with Firefox and a web UI; add "enable zoom": true to the tool definition so Claude can inspect small text at full resolution; or bolt the schema-less bash 20250124 and text editor 20250728 tools into the same tools array so the agent can save what it finds to disk. Sources & further reading - Computer use tool https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool — platform.claude.com - Computer use reference implementation https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo — github.com - Playwright for Python - Installation https://playwright.dev/python/docs/intro — playwright.dev - computer 20250124 does not match any of expected tags https://github.com/anthropics/anthropic-quickstarts/issues/251 — github.com Priya Nair https://sourcefeed.dev/u/priya nair · AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. Discussion 0 No comments yet Be the first to weigh in.