Build a Browser-Automation Agent with OpenAI Function Calling and Playwright

A new tutorial shows developers how to build a browser-automation agent by combining OpenAI's function calling with Playwright, enabling an LLM to navigate, click, fill forms, and extract data from websites autonomously. The agent uses GPT-4o-mini to decide which browser actions to take, executing them via Playwright's synchronous API in a headless Chromium browser. This approach gives AI models direct access to web interactions, expanding their capabilities beyond text-based tasks.

Build a Browser-Automation Agent with OpenAI Function Calling and Playwright Give an LLM real hands and eyes on the web: wire up OpenAI function calling to a headless Playwright browser so it can navigate, click, fill forms, and pull data on its own. Priya Nair https://sourcefeed.dev/u/priya nair What you'll build A Python script that lets an OpenAI model drive a headless Chromium browser through Playwright. You give it a task in plain English "go to this site and tell me X" , and it decides which browser actions to call, executes them, reads the results, and loops until the task is done. Prerequisites - Python 3.9 or later - An OpenAI API key with access to a function-calling-capable model gpt-4o or gpt-4o-mini - macOS, Linux, or Windows Playwright supports all three, but Linux needs extra system libraries, noted below - Comfort with basic async-free Python; we're using Playwright's synchronous API to keep the control flow simple 1. Set up your project mkdir browser-agent && cd browser-agent python3 -m venv venv source venv/bin/activate Windows: venv\Scripts\activate pip install --upgrade pip pip install openai playwright playwright install chromium On Linux including CI containers , Chromium also needs system libraries: playwright install-deps chromium Set your API key as an environment variable rather than hardcoding it: export OPENAI API KEY="sk-..." macOS/Linux PowerShell: $env:OPENAI API KEY="sk-..." 2. Wrap Playwright in tool functions The model never touches the browser directly. It calls named functions with JSON arguments, and your code executes the actual Playwright API. Keep each function narrow and single-purpose, that's what makes the model's job of choosing between them tractable. Create agent.py : python import json from openai import OpenAI from playwright.sync api import sync playwright client = OpenAI MODEL = "gpt-4o-mini" class Browser: def init self : self.playwright = sync playwright .start self.browser = self.playwright.chromium.launch headless=True self.page = self.browser.new page def navigate self, url: str - str: self.page.goto url, wait until="domcontentloaded", timeout=15000 return f"Navigated to {self.page.url}" def click self, selector: str - str: self.page.click selector, timeout=5000 return f"Clicked {selector}" def fill self, selector: str, text: str - str: self.page.fill selector, text, timeout=5000 return f"Filled {selector} with '{text}'" def get text self, selector: str - str: el = self.page.query selector selector if not el: return f"No element matched selector: {selector}" return el.inner text :2000 def close self : self.browser.close self.playwright.stop Notice the 5-15 second timeouts. Without them, a bad selector hangs the whole agent instead of returning an error the model can react to. 3. Define the function-calling schema Each tool needs a JSON schema describing its name, purpose, and parameters. The model uses the description fields to decide when to call what, so write them like you're briefing a new engineer, not just labeling arguments. tools = {"type": "function", "function": { "name": "navigate", "description": "Go to a URL in the browser", "parameters": {"type": "object", "properties": { "url": {"type": "string", "description": "Full URL including https://"}}, "required": "url" }}}, {"type": "function", "function": { "name": "click", "description": "Click the first element matching a CSS selector", "parameters": {"type": "object", "properties": { "selector": {"type": "string"}}, "required": "selector" }}}, {"type": "function", "function": { "name": "fill", "description": "Type text into an input or textarea matching a CSS selector", "parameters": {"type": "object", "properties": { "selector": {"type": "string"}, "text": {"type": "string"}}, "required": "selector", "text" }}}, {"type": "function", "function": { "name": "get text", "description": "Read the visible text of an element matching a CSS selector", "parameters": {"type": "object", "properties": { "selector": {"type": "string"}}, "required": "selector" }}}, 4. Write the agent loop This is the core pattern for any function-calling agent: send the conversation, check if the model wants to call a tool, run it, feed the result back as a tool message, and repeat until the model responds with plain text instead of a tool call. php dispatch = {} def run agent task: str, max steps: int = 10 - str: browser = Browser dispatch.update { "navigate": lambda a: browser.navigate a "url" , "click": lambda a: browser.click a "selector" , "fill": lambda a: browser.fill a "selector" , a "text" , "get text": lambda a: browser.get text a "selector" , } messages = {"role": "system", "content": "You control a headless Chromium browser through function calls. " "Use navigate, click, fill, and get text to complete the user's task step by step. " "Call get text to check what's on the page before deciding your next move. " "When the task is finished, reply with plain text summarizing the result and " "do not call any more functions." }, {"role": "user", "content": task}, try: for step in range max steps : response = client.chat.completions.create model=MODEL, messages=messages, tools=tools, tool choice="auto" message = response.choices 0 .message if not message.tool calls: return message.content messages.append { "role": "assistant", "content": message.content, "tool calls": {"id": tc.id, "type": "function", "function": { "name": tc.function.name, "arguments": tc.function.arguments}} for tc in message.tool calls , } for tc in message.tool calls: name = tc.function.name args = json.loads tc.function.arguments print f"- {name} {args} " try: result = dispatch name args except Exception as e: result = f"Error: {e}" messages.append {"role": "tool", "tool call id": tc.id, "content": str result } return "Max steps reached without a final answer." finally: browser.close if name == " main ": print run agent "Go to https://quotes.toscrape.com, read the first quote on the page, " "and tell me the quote text and its author." A few details matter here. The assistant message with tool calls has to be appended to history exactly as shown, as a dict with type: "function" , or the next API call will reject it. And every single tool call id needs a matching tool message before you send the next request; the API errors out otherwise. Wrapping errors as strings instead of raising matters too, since it lets the model see the failure and try a different selector instead of crashing your process. Verify it works Run it: python agent.py You'll see the tool calls print as they happen, something like: php - navigate {'url': 'https://quotes.toscrape.com'} - get text {'selector': '.quote'} followed by a final plain-text answer naming the quote and author the first quote on that site is from Albert Einstein . If the model calls get text on a selector that doesn't exist yet, watch it recover: the error string comes back as the tool result, and the next completion usually tries body or a broader selector instead. Troubleshooting : you skipped playwright. impl. errors.Error: Executable doesn't exist playwright install chromium . Run it inside the same virtualenv you're executing the script from. 400 error about tool calls with no matching tool message : you either didn't respond to every tool call id from the previous assistant turn, or you sent them out of order. Each tool call needs exactly one corresponding role: tool message before the next create call. Agent loops without making progress : tighten the system prompt tell it explicitly to call get text before guessing selectors , and lower max steps while debugging so you're not burning tokens on a stuck loop. Timeouts on : the selector is probably right but the element isn't in the viewport or is inside an iframe. Playwright's click / fill page.frame locator handles iframes; that's a good next tool to add. Next steps Add more tools as the agent needs them: scroll , wait for selector , and screenshot returning a description, not raw pixels, unless you're using a vision-capable model . For structured extraction instead of free text, pair this with OpenAI's structured outputs response format with a JSON schema on the final answer. For production use, sandbox the browser to an allowlist of domains, run it inside Docker using Microsoft's official mcr.microsoft.com/playwright/python image, and cap wall-clock time per task, not just step count, since a single slow page load can otherwise stall the whole agent. Priya Nair https://sourcefeed.dev/u/priya nair · AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. Discussion 0 No comments yet Be the first to weigh in.