{"slug": "build-a-web-aware-agent-with-openai-agents-sdk-and-zenrows", "title": "Build a web-aware agent with OpenAI Agents SDK and Zenrows", "summary": "A developer guide demonstrates how to build a web-aware agent by combining OpenAI's Agents SDK with Zenrows' scraping API. The approach registers Zenrows Fetch as a function tool alongside OpenAI's WebSearchTool, enabling the agent to retrieve live, JavaScript-rendered page content when search results are stale or incomplete. The tutorial includes code examples and addresses limitations of built-in search for dynamic content.", "body_md": "*This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/web-aware-agent-openai-agents-sdk-zenrows*\n\nThis guide shows you how to register Zenrows Fetch as a `@function_tool`\n\nnext to `WebSearchTool`\n\nin a single OpenAI Agents SDK agent, so the model routes between discovery and live-page retrieval on its own each turn. You need Python 3.9+, an OpenAI API key, and a Zenrows API key.\n\nAll the code is on [GitHub](https://github.com/ZenRows/web-aware-agent-openai-agents-sdk-zenrows).\n\nInstall dependencies:\n\n```\npython3 -m pip install openai-agents python-dotenv\n```\n\nAdd both keys to a `.env`\n\nfile and add it to `.gitignore`\n\n:\n\n```\nOPENAI_API_KEY=your_openai_api_key_here\nZENROWS_API_KEY=your_zenrows_api_key_here\n```\n\nLoad them in your script:\n\n``` python\nfrom dotenv import load_dotenv\n\nload_dotenv()\n```\n\n`WebSearchTool`\n\nis a hosted tool that runs on OpenAI's infrastructure. Enabling it takes one line.\n\n``` python\nimport asyncio\nfrom dotenv import load_dotenv\nfrom agents import Agent, Runner, WebSearchTool\n\nload_dotenv()\n\nagent = Agent(\n    name=\"Product Researcher\",\n    instructions=(\n        \"You research products and web pages. \"\n        \"Report exactly what you find, and state plainly when \"\n        \"specific information is missing from your results.\"\n    ),\n    tools=[WebSearchTool()],\n)\n\nasync def main():\n    result = await Runner.run(\n        agent,\n        \"Which laptops did Apple release most recently, and where can I buy them?\",\n    )\n    print(result.final_output)\n\nasyncio.run(main())\n```\n\nThis works for broad discovery queries. The agent returns relevant pages with source links. The gap appears on live or protected targets.\n\nBuilt-in search returns OpenAI's cached copy of a page, not the current version. For documentation, that's usually fine. For prices, stock levels, or anything that changes frequently, the cached copy lags.\n\nMany pages ship an HTML shell and fill in prices, inventory, and reviews via JavaScript at runtime. The search index stores the shell. The data you want was never captured.\n\n``` python\nimport asyncio\nfrom dotenv import load_dotenv\nfrom agents import Agent, Runner, WebSearchTool\n\nload_dotenv()\n\nagent = Agent(\n    name=\"Product Researcher\",\n    instructions=(\n        \"You retrieve product details from web pages. \"\n        \"Report exactly what you find, and say so plainly if content is missing.\"\n    ),\n    tools=[WebSearchTool()],\n)\n\nasync def main():\n    result = await Runner.run(\n        agent,\n        \"What is the current price and stock status of \"\n        \"https://www.amazon.com/dp/B0GR1JKMBV/ref=fs_a_mbt2_us1?th=1\",\n    )\n    print(result.final_output)\n\nasyncio.run(main())\n```\n\nThe agent finds the page. It returns the product title and a clear statement that the price wasn't accessible:\n\n```\nI visited the Amazon product page for ASIN B0GR1JKMBV...\nThe page displays: \"To see product details, add this item to your cart.\"\nTherefore I could not retrieve the current price or availability.\npython\nimport os\nimport requests\nfrom agents import function_tool\n\n@function_tool\ndef fetch_page_content(url: str) -> str:\n    \"\"\"Fetch the full, current content of a specific web page as Markdown.\n\n    Use this when you need the complete content of a known URL rather than\n    a search result summary, such as live prices, stock levels, or data\n    that loads via JavaScript after the page opens.\n\n    Args:\n        url: The full URL of the page to retrieve.\n    \"\"\"\n    response = requests.get(\n        \"https://api.zenrows.com/v1/\",\n        params={\n            \"url\": url,\n            \"apikey\": os.getenv(\"ZENROWS_API_KEY\"),\n            # mode=auto lets Zenrows pick the right settings per site\n            \"mode\": \"auto\",\n            \"response_type\": \"markdown\",\n        },\n        timeout=90,\n    )\n\n    # return the error as a string so the agent can react instead of crashing\n    if response.status_code != 200:\n        return f\"Zenrows returned {response.status_code} for {url}\"\n\n    return response.text\n```\n\nFour things to note:\n\n`@function_tool`\n\n`mode=auto`\n\n`response_type=markdown`\n\n``` python\nimport asyncio\nimport os\nimport requests\nfrom dotenv import load_dotenv\nfrom agents import Agent, Runner, WebSearchTool, function_tool\n\nload_dotenv()\n\n@function_tool\ndef fetch_page_content(url: str) -> str:\n    \"\"\"Fetch the full, current content of a specific web page as Markdown.\n\n    Use this when you need the complete content of a known URL rather than\n    a search result summary, such as live prices, stock levels, or data\n    that loads via JavaScript after the page opens.\n\n    Args:\n        url: The full URL of the page to retrieve.\n    \"\"\"\n    response = requests.get(\n        \"https://api.zenrows.com/v1/\",\n        params={\n            \"url\": url,\n            \"apikey\": os.getenv(\"ZENROWS_API_KEY\"),\n            \"mode\": \"auto\",\n            \"response_type\": \"markdown\",\n        },\n        timeout=90,\n    )\n\n    if response.status_code != 200:\n        return f\"Zenrows returned {response.status_code} for {url}\"\n\n    return response.text\n\n# the model routes between both tools from their descriptions alone\nagent = Agent(\n    name=\"Web-Aware Researcher\",\n    instructions=(\n        \"You research products on the web. \"\n        \"Use web search to find the product page URL. \"\n        \"Then call fetch_page_content on that URL and report the price \"\n        \"and stock status from the fetched page content. \"\n        \"State which tool each figure came from.\"\n    ),\n    tools=[WebSearchTool(), fetch_page_content],\n)\n\ndef print_trace(items):\n    for item in items:\n        raw = getattr(item, \"raw_item\", None)\n        name = getattr(raw, \"name\", None) or getattr(raw, \"type\", \"\")\n        print(f\"[{item.type}] {name}\")\n\n        if item.type == \"tool_call_item\" and name == \"fetch_page_content\":\n            print(\"  args:\", getattr(raw, \"arguments\", \"\"))\n\n        if item.type == \"tool_call_output_item\":\n            out = str(getattr(item, \"output\", \"\"))\n            print(f\"  output: {len(out)} chars\")\n            print(f\"  body: {out[:500]}\")\n\nasync def main():\n    result = await Runner.run(\n        agent,\n        \"Find the PriceOye page for the iPhone 17 Pro \"\n        \"and report its current price and stock status.\",\n    )\n\n    print_trace(result.new_items)\n    print(\"\\n---\\n\")\n    print(result.final_output)\n\nasyncio.run(main())\n```\n\nThe trace shows the two-step routing. Turn one: `web_search_call`\n\nfor discovery. Turn two: `fetch_page_content`\n\nwith the URL it found.\n\n```\n[tool_call_item] web_search_call\n[message_output_item] message\n[tool_call_item] fetch_page_content\n  args: {\"url\":\"https://priceoye.pk/mobiles/apple/apple-iphone-17-pro\"}\n[tool_call_output_item]\n\n---\n\nHere are the details for the iPhone 17 Pro from its PriceOye product page:\n\nCurrent Price: Rs 471,999\nStock Status: Only 1 left in stock\n\nPrice and stock status are directly extracted from the fetched page content\n(functions.fetch_page_content tool).\n```\n\nNo conditionals. No orchestration code. The routing lives entirely in the tool descriptions.\n\nUse `WebSearchTool`\n\nfor discovery: finding pages, open-ended research, answering questions from indexed public content.\n\nUse Zenrows as the retrieval layer when you have a specific URL and need the live contents — protected targets, JavaScript-rendered pages, anything where freshness matters.\n\nFor structured field extraction instead of full-page Markdown, use [Zenrows Extract](https://docs.zenrows.com/extract/introduction). For high-volume or scheduled retrieval across many URLs, use [Zenrows Batch](https://www.zenrows.com/products/batch).\n\nThe same pattern carries over to multi-agent setups: [building a web research multi-agent system with AG2 and Zenrows](https://www.zenrows.com/blog/web-research-multi-agent-ag2-zenrows). And to smolagents, under a `@tool`\n\ndecorator: [Zenrows smolagents guide](https://www.zenrows.com/blog/zenrows-smolagents).\n\nYou now have an agent with two retrieval paths. `WebSearchTool`\n\nhandles discovery. Zenrows reads the full live page once the URL is known, including JavaScript-rendered and anti-bot-defended targets. The model routes between them from the tool descriptions alone.", "url": "https://wpnews.pro/news/build-a-web-aware-agent-with-openai-agents-sdk-and-zenrows", "canonical_source": "https://dev.to/zenrows/build-a-web-aware-agent-with-openai-agents-sdk-and-zenrows-1a34", "published_at": "2026-09-04 07:27:21+00:00", "updated_at": "2026-09-04 07:53:52.711578+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "artificial-intelligence"], "entities": ["OpenAI", "Zenrows", "OpenAI Agents SDK", "WebSearchTool", "Python"], "alternates": {"html": "https://wpnews.pro/news/build-a-web-aware-agent-with-openai-agents-sdk-and-zenrows", "markdown": "https://wpnews.pro/news/build-a-web-aware-agent-with-openai-agents-sdk-and-zenrows.md", "text": "https://wpnews.pro/news/build-a-web-aware-agent-with-openai-agents-sdk-and-zenrows.txt", "jsonld": "https://wpnews.pro/news/build-a-web-aware-agent-with-openai-agents-sdk-and-zenrows.jsonld"}}