{"slug": "build-a-browser-automation-agent-with-openai-function-calling-and-playwright", "title": "Build a Browser-Automation Agent with OpenAI Function Calling and Playwright", "summary": "A new tutorial shows developers how to build a browser-automation agent by combining OpenAI's function calling with Playwright, enabling an LLM to navigate, click, fill forms, and extract data from websites autonomously. The agent uses GPT-4o-mini to decide which browser actions to take, executing them via Playwright's synchronous API in a headless Chromium browser. This approach gives AI models direct access to web interactions, expanding their capabilities beyond text-based tasks.", "body_md": "# Build a Browser-Automation Agent with OpenAI Function Calling and Playwright\n\nGive an LLM real hands and eyes on the web: wire up OpenAI function calling to a headless Playwright browser so it can navigate, click, fill forms, and pull data on its own.\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)\n\n## What you'll build\n\nA Python script that lets an OpenAI model drive a headless Chromium browser through Playwright. You give it a task in plain English (\"go to this site and tell me X\"), and it decides which browser actions to call, executes them, reads the results, and loops until the task is done.\n\n## Prerequisites\n\n- Python 3.9 or later\n- An OpenAI API key with access to a function-calling-capable model (\n`gpt-4o`\n\nor`gpt-4o-mini`\n\n) - macOS, Linux, or Windows (Playwright supports all three, but Linux needs extra system libraries, noted below)\n- Comfort with basic async-free Python; we're using Playwright's synchronous API to keep the control flow simple\n\n## 1. Set up your project\n\n```\nmkdir browser-agent && cd browser-agent\npython3 -m venv venv\nsource venv/bin/activate   # Windows: venv\\Scripts\\activate\npip install --upgrade pip\npip install openai playwright\nplaywright install chromium\n```\n\nOn Linux (including CI containers), Chromium also needs system libraries:\n\n```\nplaywright install-deps chromium\n```\n\nSet your API key as an environment variable rather than hardcoding it:\n\n```\nexport OPENAI_API_KEY=\"sk-...\"          # macOS/Linux\n# PowerShell: $env:OPENAI_API_KEY=\"sk-...\"\n```\n\n## 2. Wrap Playwright in tool functions\n\nThe model never touches the browser directly. It calls named functions with JSON arguments, and your code executes the actual Playwright API. Keep each function narrow and single-purpose, that's what makes the model's job of choosing between them tractable.\n\nCreate `agent.py`\n\n:\n\n``` python\nimport json\nfrom openai import OpenAI\nfrom playwright.sync_api import sync_playwright\n\nclient = OpenAI()\nMODEL = \"gpt-4o-mini\"\n\nclass Browser:\n    def __init__(self):\n        self.playwright = sync_playwright().start()\n        self.browser = self.playwright.chromium.launch(headless=True)\n        self.page = self.browser.new_page()\n\n    def navigate(self, url: str) -> str:\n        self.page.goto(url, wait_until=\"domcontentloaded\", timeout=15000)\n        return f\"Navigated to {self.page.url}\"\n\n    def click(self, selector: str) -> str:\n        self.page.click(selector, timeout=5000)\n        return f\"Clicked {selector}\"\n\n    def fill(self, selector: str, text: str) -> str:\n        self.page.fill(selector, text, timeout=5000)\n        return f\"Filled {selector} with '{text}'\"\n\n    def get_text(self, selector: str) -> str:\n        el = self.page.query_selector(selector)\n        if not el:\n            return f\"No element matched selector: {selector}\"\n        return el.inner_text()[:2000]\n\n    def close(self):\n        self.browser.close()\n        self.playwright.stop()\n```\n\nNotice the 5-15 second timeouts. Without them, a bad selector hangs the whole agent instead of returning an error the model can react to.\n\n## 3. Define the function-calling schema\n\nEach tool needs a JSON schema describing its name, purpose, and parameters. The model uses the `description`\n\nfields to decide when to call what, so write them like you're briefing a new engineer, not just labeling arguments.\n\n```\ntools = [\n    {\"type\": \"function\", \"function\": {\n        \"name\": \"navigate\",\n        \"description\": \"Go to a URL in the browser\",\n        \"parameters\": {\"type\": \"object\", \"properties\": {\n            \"url\": {\"type\": \"string\", \"description\": \"Full URL including https://\"}},\n            \"required\": [\"url\"]}}},\n    {\"type\": \"function\", \"function\": {\n        \"name\": \"click\",\n        \"description\": \"Click the first element matching a CSS selector\",\n        \"parameters\": {\"type\": \"object\", \"properties\": {\n            \"selector\": {\"type\": \"string\"}}, \"required\": [\"selector\"]}}},\n    {\"type\": \"function\", \"function\": {\n        \"name\": \"fill\",\n        \"description\": \"Type text into an input or textarea matching a CSS selector\",\n        \"parameters\": {\"type\": \"object\", \"properties\": {\n            \"selector\": {\"type\": \"string\"}, \"text\": {\"type\": \"string\"}},\n            \"required\": [\"selector\", \"text\"]}}},\n    {\"type\": \"function\", \"function\": {\n        \"name\": \"get_text\",\n        \"description\": \"Read the visible text of an element matching a CSS selector\",\n        \"parameters\": {\"type\": \"object\", \"properties\": {\n            \"selector\": {\"type\": \"string\"}}, \"required\": [\"selector\"]}}},\n]\n```\n\n## 4. Write the agent loop\n\nThis is the core pattern for any function-calling agent: send the conversation, check if the model wants to call a tool, run it, feed the result back as a `tool`\n\nmessage, and repeat until the model responds with plain text instead of a tool call.\n\n``` php\ndispatch = {}\n\ndef run_agent(task: str, max_steps: int = 10) -> str:\n    browser = Browser()\n    dispatch.update({\n        \"navigate\": lambda a: browser.navigate(a[\"url\"]),\n        \"click\": lambda a: browser.click(a[\"selector\"]),\n        \"fill\": lambda a: browser.fill(a[\"selector\"], a[\"text\"]),\n        \"get_text\": lambda a: browser.get_text(a[\"selector\"]),\n    })\n\n    messages = [\n        {\"role\": \"system\", \"content\": (\n            \"You control a headless Chromium browser through function calls. \"\n            \"Use navigate, click, fill, and get_text to complete the user's task step by step. \"\n            \"Call get_text to check what's on the page before deciding your next move. \"\n            \"When the task is finished, reply with plain text summarizing the result and \"\n            \"do not call any more functions.\")},\n        {\"role\": \"user\", \"content\": task},\n    ]\n\n    try:\n        for step in range(max_steps):\n            response = client.chat.completions.create(\n                model=MODEL, messages=messages, tools=tools, tool_choice=\"auto\")\n            message = response.choices[0].message\n\n            if not message.tool_calls:\n                return message.content\n\n            messages.append({\n                \"role\": \"assistant\",\n                \"content\": message.content,\n                \"tool_calls\": [\n                    {\"id\": tc.id, \"type\": \"function\", \"function\": {\n                        \"name\": tc.function.name, \"arguments\": tc.function.arguments}}\n                    for tc in message.tool_calls],\n            })\n\n            for tc in message.tool_calls:\n                name = tc.function.name\n                args = json.loads(tc.function.arguments)\n                print(f\"-> {name}({args})\")\n                try:\n                    result = dispatch[name](args)\n                except Exception as e:\n                    result = f\"Error: {e}\"\n                messages.append({\"role\": \"tool\", \"tool_call_id\": tc.id, \"content\": str(result)})\n\n        return \"Max steps reached without a final answer.\"\n    finally:\n        browser.close()\n\nif __name__ == \"__main__\":\n    print(run_agent(\n        \"Go to https://quotes.toscrape.com, read the first quote on the page, \"\n        \"and tell me the quote text and its author.\"))\n```\n\nA few details matter here. The assistant message with `tool_calls`\n\nhas to be appended to history exactly as shown, as a dict with `type: \"function\"`\n\n, or the next API call will reject it. And every single `tool_call_id`\n\nneeds a matching `tool`\n\nmessage before you send the next request; the API errors out otherwise. Wrapping errors as strings (instead of raising) matters too, since it lets the model see the failure and try a different selector instead of crashing your process.\n\n## Verify it works\n\nRun it:\n\n```\npython agent.py\n```\n\nYou'll see the tool calls print as they happen, something like:\n\n``` php\n-> navigate({'url': 'https://quotes.toscrape.com'})\n-> get_text({'selector': '.quote'})\n```\n\nfollowed by a final plain-text answer naming the quote and author (the first quote on that site is from Albert Einstein). If the model calls `get_text`\n\non a selector that doesn't exist yet, watch it recover: the error string comes back as the tool result, and the next completion usually tries `body`\n\nor a broader selector instead.\n\n## Troubleshooting\n\n: you skipped`playwright._impl._errors.Error: Executable doesn't exist`\n\n`playwright install chromium`\n\n. Run it inside the same virtualenv you're executing the script from.**400 error about tool_calls with no matching tool message**: you either didn't respond to every`tool_call_id`\n\nfrom the previous assistant turn, or you sent them out of order. Each tool call needs exactly one corresponding`role: tool`\n\nmessage before the next`create()`\n\ncall.**Agent loops without making progress**: tighten the system prompt (tell it explicitly to call`get_text`\n\nbefore guessing selectors), and lower`max_steps`\n\nwhile debugging so you're not burning tokens on a stuck loop.**Timeouts on**: the selector is probably right but the element isn't in the viewport or is inside an iframe. Playwright's`click`\n\n/`fill`\n\n`page.frame_locator()`\n\nhandles iframes; that's a good next tool to add.\n\n## Next steps\n\nAdd more tools as the agent needs them: `scroll`\n\n, `wait_for_selector`\n\n, and `screenshot`\n\n(returning a description, not raw pixels, unless you're using a vision-capable model). For structured extraction instead of free text, pair this with OpenAI's structured outputs (`response_format`\n\nwith a JSON schema) on the final answer. For production use, sandbox the browser to an allowlist of domains, run it inside Docker using Microsoft's official `mcr.microsoft.com/playwright/python`\n\nimage, and cap wall-clock time per task, not just step count, since a single slow page load can otherwise stall the whole agent.\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer\n\nPriya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/build-a-browser-automation-agent-with-openai-function-calling-and-playwright", "canonical_source": "https://sourcefeed.dev/a/build-a-browser-automation-agent-with-openai-function-calling-and-playwright", "published_at": "2026-07-16 07:32:30+00:00", "updated_at": "2026-07-16 07:41:41.826947+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["OpenAI", "Playwright", "GPT-4o-mini", "Chromium", "Python"], "alternates": {"html": "https://wpnews.pro/news/build-a-browser-automation-agent-with-openai-function-calling-and-playwright", "markdown": "https://wpnews.pro/news/build-a-browser-automation-agent-with-openai-function-calling-and-playwright.md", "text": "https://wpnews.pro/news/build-a-browser-automation-agent-with-openai-function-calling-and-playwright.txt", "jsonld": "https://wpnews.pro/news/build-a-browser-automation-agent-with-openai-function-calling-and-playwright.jsonld"}}