Wire Claude's computer-use tool to a Playwright browser so it can screenshot, click, and type autonomously.
What you'll build #
A Python agent that gives Claude eyes and hands inside a real Chromium browser: it takes screenshots, clicks, types, and scrolls through Playwright, looping autonomously until a multi-step web task — searching Wikipedia and reading the answer off the page — is done.
Prerequisites #
Versions verified for this tutorial:
- Python 3.10+ (tested on 3.13)
anthropic
0.120.2 andplaywright
1.62.0 (Chromium 151 headless shell, installed by Playwright)- An Anthropic API key with credit, from the
Claude Console— exported asANTHROPIC_API_KEY
- Model
claude-opus-5
with thecomputer-use-2025-11-24
beta header
Computer use is a beta feature. Everything here runs headless on macOS, Linux, or Windows — no X11 or Docker required.
1. Understand what the API actually does #
Computer use is a client-side tool: Claude never touches your browser. You declare a schema-less computer_20251124
tool, and Claude responds with tool_use
blocks containing abstract actions — screenshot
, left_click
with a [x, y]
coordinate, type
with text, key
for shortcuts, scroll
, and so on. Your code executes each action, returns the result (screenshots go back as base64 PNG blocks), and calls the API again. That request → execute → respond cycle repeats while stop_reason
is "tool_use"
— the agent loop.
Anthropic's reference implementation wraps a whole Linux desktop in Docker — Xvfb, a window manager, Firefox. That's the right call for general desktop automation, but for web-only tasks a Playwright-controlled headless browser is a much smaller surface: no container, no display server, and the screenshot dimensions are exactly what you configure, which matters because Claude's click coordinates are pixel positions on the image you send.
One wrinkle for browsers: headless Chromium has no address bar, so Claude can't navigate by typing a URL into a screenshot. The API explicitly supports mixing custom tools into the same tools
array, so we'll add a tiny navigate
tool alongside the computer tool.
2. Install the dependencies #
python3 -m venv .venv && source .venv/bin/activate
pip install anthropic playwright
playwright install chromium
export ANTHROPIC_API_KEY="sk-ant-..."
playwright install chromium
downloads a ~95 MB pinned browser build; your system Chrome is not used.
3. Write the agent #
Save this as browser_agent.py
. It's complete — no elisions.
"""Minimal computer-use agent that drives a real Chromium browser via Playwright."""
import base64
import sys
import anthropic
from playwright.sync_api import sync_playwright
WIDTH, HEIGHT = 1280, 800
MODEL = "claude-opus-5"
COMPUTER_USE_BETA = "computer-use-2025-11-24"
MAX_ITERATIONS = 30
TOOLS = [
{
"type": "computer_20251124",
"name": "computer",
"display_width_px": WIDTH,
"display_height_px": HEIGHT,
},
{
"name": "navigate",
"description": "Load a URL in the browser. Use this instead of an address bar.",
"input_schema": {
"type": "object",
"properties": {"url": {"type": "string", "description": "Absolute URL to open"}},
"required": ["url"],
},
},
]
KEY_MAP = {
"return": "Enter", "enter": "Enter", "tab": "Tab", "escape": "Escape", "esc": "Escape",
"backspace": "Backspace", "back_space": "Backspace", "delete": "Delete", "space": "Space",
"up": "ArrowUp", "down": "ArrowDown", "left": "ArrowLeft", "right": "ArrowRight",
"page_down": "PageDown", "page_up": "PageUp", "home": "Home", "end": "End",
"ctrl": "Control", "control": "Control", "alt": "Alt", "shift": "Shift",
"super": "Meta", "cmd": "Meta",
}
def to_playwright_keys(combo: str) -> str:
return "+".join(KEY_MAP.get(k.lower(), k) for k in combo.split("+"))
class BrowserComputer:
def __init__(self, page):
self.page = page
def screenshot(self):
png = self.page.screenshot()
return [{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": base64.b64encode(png).decode(),
},
}]
def handle(self, name: str, action: dict):
"""Execute one tool call and return tool_result content."""
if name == "navigate":
self.page.goto(action["url"], wait_until="domcontentloaded")
return f"Loaded {self.page.url}"
kind = action["action"]
if kind == "screenshot":
return self.screenshot()
if kind in ("left_click", "right_click", "middle_click", "double_click", "triple_click"):
x, y = action["coordinate"]
button = {"right_click": "right", "middle_click": "middle"}.get(kind, "left")
clicks = {"double_click": 2, "triple_click": 3}.get(kind, 1)
self.page.mouse.click(x, y, button=button, click_count=clicks)
return f"{kind} at ({x}, {y})"
if kind == "type":
self.page.keyboard.type(action["text"])
return f"Typed {action['text']!r}"
if kind == "key":
keys = to_playwright_keys(action["text"])
self.page.keyboard.press(keys)
return f"Pressed {keys}"
if kind == "scroll":
x, y = action["coordinate"]
amount = action.get("scroll_amount", 3) * 100
dx, dy = {"down": (0, amount), "up": (0, -amount),
"right": (amount, 0), "left": (-amount, 0)}[action["scroll_direction"]]
self.page.mouse.move(x, y)
self.page.mouse.wheel(dx, dy)
return f"Scrolled {action['scroll_direction']} at ({x}, {y})"
if kind == "mouse_move":
x, y = action["coordinate"]
self.page.mouse.move(x, y)
return f"Moved mouse to ({x}, {y})"
if kind == "wait":
self.page.wait_for_timeout(action.get("duration", 1) * 1000)
return "Waited"
return f"Unsupported action: {kind}"
def run(task: str):
client = anthropic.Anthropic()
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page(viewport={"width": WIDTH, "height": HEIGHT})
computer = BrowserComputer(page)
messages = [{"role": "user", "content": task}]
for _ in range(MAX_ITERATIONS):
response = client.beta.messages.create(
model=MODEL,
max_tokens=4096,
tools=TOOLS,
messages=messages,
betas=[COMPUTER_USE_BETA],
)
messages.append({"role": "assistant", "content": response.content})
tool_results = []
for block in response.content:
if block.type == "text":
print(f"[claude] {block.text}")
elif block.type == "tool_use":
print(f"[action] {block.name}: {block.input}")
try:
result = computer.handle(block.name, block.input)
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": result,
})
except Exception as exc:
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Error: {exc}",
"is_error": True,
})
if response.stop_reason != "tool_use":
break
messages.append({"role": "user", "content": tool_results})
browser.close()
if __name__ == "__main__":
run(sys.argv[1] if len(sys.argv) > 1 else
"Navigate to https://en.wikipedia.org, use the search box to search for "
"'Alan Turing', and tell me his year of birth. Take a screenshot after "
"each step to verify it worked.")
Four details are load-bearing:
The viewport must exactly match Claude's click coordinates are pixel positions on the screenshot you sent; any mismatch and every click lands off-target.display_width_px
/display_height_px
.Key names need translation. Claude was trained on xdotool keysyms (Return
,ctrl+l
,Page_Down
); Playwright wantsEnter
,Control+l
,PageDown
. TheKEY_MAP
bridges them.Failures go back as, not a crash — Claude reads the error and tries another approach.tool_result
withis_error: true
Append the full Onresponse.content
back intomessages
.claude-opus-5
thinking is on by default, and thinking blocks must be returned unchanged for the next turn to be accepted.
The screenshot-after-each-step instruction in the default task is lifted from Anthropic's prompting guidance — without it, Claude sometimes assumes an action worked instead of checking.
4. Run it #
python browser_agent.py
Each loop iteration costs one API call carrying the conversation so far, and every screenshot is roughly a thousand input tokens, so a typical run of this task is 6–10 iterations. MAX_ITERATIONS
is your cost circuit breaker. If you later move to an older model to cut cost, Anthropic's docs note that for computer use specifically, medium
effort is the accuracy-to-cost sweet spot on Sonnet 4.6 and Opus 4.6 (set via output_config={"effort": "medium"}
), while max
adds tokens without improving UI accuracy.
Verify it works #
You should see an action trace like this (Claude's exact wording and coordinates will vary):
[claude] I'll navigate to Wikipedia first.
[action] navigate: {'url': 'https://en.wikipedia.org'}
[action] computer: {'action': 'screenshot'}
[action] computer: {'action': 'left_click', 'coordinate': [583, 384]}
[action] computer: {'action': 'type', 'text': 'Alan Turing'}
[action] computer: {'action': 'key', 'text': 'Return'}
[action] computer: {'action': 'screenshot'}
[claude] Alan Turing was born in 1912 (23 June 1912, in Maida Vale, London).
Success looks like: a navigate
call, at least one screenshot, click/type/key actions on the search box, and a final [claude]
text line containing 1912 with no further [action]
lines after it (meaning stop_reason
flipped from tool_use
to end_turn
).
Troubleshooting #
** BrowserType.launch: Executable doesn't exist at .../chrome-headless-shell followed by "Looks like Playwright was just installed or updated."** You installed the
playwright
package but not the browser, or installed it under a different virtualenv. Run playwright install chromium
from the same venv that runs the script.** anthropic.BadRequestError: Error code: 400 saying tools.0.type: Input tag 'computer_20251124' ... does not match any of the expected tags.** The tool version and beta header are mismatched.
computer_20251124
requires betas=["computer-use-2025-11-24"]
and a supporting model (Claude Opus 5, Sonnet 5, Opus 4.8/4.7/4.6, Sonnet 4.6, or Opus 4.5). Older models like Sonnet 4.5 and Haiku 4.5 need the computer_20250124
tool with computer-use-2025-01-24
instead.anthropic.AuthenticationError: Error code: 401 - {'type': 'error', 'error': {'type': 'authentication_error', 'message': 'invalid x-api-key'}}
.ANTHROPIC_API_KEY
is unset, stale, or not exported into the shell running the script. echo $ANTHROPIC_API_KEY
to confirm, and re-export after opening a new terminal.
Clicks consistently land in the wrong place. Your screenshots aren't the size you declared. This happens when adapting the code to a headed browser on a Retina/HiDPI display, where screenshots come back at 2x scale. Keep device_scale_factor
at 1 (the headless default) or downscale screenshots to exactly WIDTH×HEIGHT
before sending.
Next steps #
Swap the default task for your own — form filling, price checks, multi-page research — but keep two of Anthropic's warnings in mind: content on web pages can prompt-inject the agent (the API runs injection classifiers on screenshots and will steer Claude to ask for confirmation), and anything consequential — purchases, logins, accepting terms — should require human sign-off. From here, try Anthropic's computer-use reference implementation, a Dockerized full Linux desktop with Firefox and a web UI; add "enable_zoom": true
to the tool definition so Claude can inspect small text at full resolution; or bolt the schema-less bash_20250124
and text_editor_20250728
tools into the same tools
array so the agent can save what it finds to disk.
Sources & further reading #
Computer use tool— platform.claude.com - Computer use reference implementation— github.com - Playwright for Python - Installation— playwright.dev - computer_20250124 does not match any of expected tags— github.com
Priya Nair· AI & Developer Experience Writer
Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.
Discussion 0 #
No comments yet
Be the first to weigh in.