This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/web-aware-agent-openai-agents-sdk-zenrows
This guide shows you how to register Zenrows Fetch as a @function_tool
next to WebSearchTool
in a single OpenAI Agents SDK agent, so the model routes between discovery and live-page retrieval on its own each turn. You need Python 3.9+, an OpenAI API key, and a Zenrows API key.
All the code is on GitHub.
Install dependencies:
python3 -m pip install openai-agents python-dotenv
Add both keys to a .env
file and add it to .gitignore
:
OPENAI_API_KEY=your_openai_api_key_here
ZENROWS_API_KEY=your_zenrows_api_key_here
Load them in your script:
from dotenv import load_dotenv
load_dotenv()
WebSearchTool
is a hosted tool that runs on OpenAI's infrastructure. Enabling it takes one line.
import asyncio
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool
load_dotenv()
agent = Agent(
name="Product Researcher",
instructions=(
"You research products and web pages. "
"Report exactly what you find, and state plainly when "
"specific information is missing from your results."
),
tools=[WebSearchTool()],
)
async def main():
result = await Runner.run(
agent,
"Which laptops did Apple release most recently, and where can I buy them?",
)
print(result.final_output)
asyncio.run(main())
This works for broad discovery queries. The agent returns relevant pages with source links. The gap appears on live or protected targets.
Built-in search returns OpenAI's cached copy of a page, not the current version. For documentation, that's usually fine. For prices, stock levels, or anything that changes frequently, the cached copy lags.
Many pages ship an HTML shell and fill in prices, inventory, and reviews via JavaScript at runtime. The search index stores the shell. The data you want was never captured.
import asyncio
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool
load_dotenv()
agent = Agent(
name="Product Researcher",
instructions=(
"You retrieve product details from web pages. "
"Report exactly what you find, and say so plainly if content is missing."
),
tools=[WebSearchTool()],
)
async def main():
result = await Runner.run(
agent,
"What is the current price and stock status of "
"https://www.amazon.com/dp/B0GR1JKMBV/ref=fs_a_mbt2_us1?th=1",
)
print(result.final_output)
asyncio.run(main())
The agent finds the page. It returns the product title and a clear statement that the price wasn't accessible:
I visited the Amazon product page for ASIN B0GR1JKMBV...
The page displays: "To see product details, add this item to your cart."
Therefore I could not retrieve the current price or availability.
python
import os
import requests
from agents import function_tool
@function_tool
def fetch_page_content(url: str) -> str:
"""Fetch the full, current content of a specific web page as Markdown.
Use this when you need the complete content of a known URL rather than
a search result summary, such as live prices, stock levels, or data
that loads via JavaScript after the page opens.
Args:
url: The full URL of the page to retrieve.
"""
response = requests.get(
"https://api.zenrows.com/v1/",
params={
"url": url,
"apikey": os.getenv("ZENROWS_API_KEY"),
"mode": "auto",
"response_type": "markdown",
},
timeout=90,
)
if response.status_code != 200:
return f"Zenrows returned {response.status_code} for {url}"
return response.text
Four things to note:
@function_tool
mode=auto
response_type=markdown
import asyncio
import os
import requests
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool, function_tool
load_dotenv()
@function_tool
def fetch_page_content(url: str) -> str:
"""Fetch the full, current content of a specific web page as Markdown.
Use this when you need the complete content of a known URL rather than
a search result summary, such as live prices, stock levels, or data
that loads via JavaScript after the page opens.
Args:
url: The full URL of the page to retrieve.
"""
response = requests.get(
"https://api.zenrows.com/v1/",
params={
"url": url,
"apikey": os.getenv("ZENROWS_API_KEY"),
"mode": "auto",
"response_type": "markdown",
},
timeout=90,
)
if response.status_code != 200:
return f"Zenrows returned {response.status_code} for {url}"
return response.text
agent = Agent(
name="Web-Aware Researcher",
instructions=(
"You research products on the web. "
"Use web search to find the product page URL. "
"Then call fetch_page_content on that URL and report the price "
"and stock status from the fetched page content. "
"State which tool each figure came from."
),
tools=[WebSearchTool(), fetch_page_content],
)
def print_trace(items):
for item in items:
raw = getattr(item, "raw_item", None)
name = getattr(raw, "name", None) or getattr(raw, "type", "")
print(f"[{item.type}] {name}")
if item.type == "tool_call_item" and name == "fetch_page_content":
print(" args:", getattr(raw, "arguments", ""))
if item.type == "tool_call_output_item":
out = str(getattr(item, "output", ""))
print(f" output: {len(out)} chars")
print(f" body: {out[:500]}")
async def main():
result = await Runner.run(
agent,
"Find the PriceOye page for the iPhone 17 Pro "
"and report its current price and stock status.",
)
print_trace(result.new_items)
print("\n---\n")
print(result.final_output)
asyncio.run(main())
The trace shows the two-step routing. Turn one: web_search_call
for discovery. Turn two: fetch_page_content
with the URL it found.
[tool_call_item] web_search_call
[message_output_item] message
[tool_call_item] fetch_page_content
args: {"url":"https://priceoye.pk/mobiles/apple/apple-iphone-17-pro"}
[tool_call_output_item]
---
Here are the details for the iPhone 17 Pro from its PriceOye product page:
Current Price: Rs 471,999
Stock Status: Only 1 left in stock
Price and stock status are directly extracted from the fetched page content
(functions.fetch_page_content tool).
No conditionals. No orchestration code. The routing lives entirely in the tool descriptions.
Use WebSearchTool
for discovery: finding pages, open-ended research, answering questions from indexed public content.
Use Zenrows as the retrieval layer when you have a specific URL and need the live contents — protected targets, JavaScript-rendered pages, anything where freshness matters.
For structured field extraction instead of full-page Markdown, use Zenrows Extract. For high-volume or scheduled retrieval across many URLs, use Zenrows Batch.
The same pattern carries over to multi-agent setups: building a web research multi-agent system with AG2 and Zenrows. And to smolagents, under a @tool
decorator: Zenrows smolagents guide.
You now have an agent with two retrieval paths. WebSearchTool
handles discovery. Zenrows reads the full live page once the URL is known, including JavaScript-rendered and anti-bot-defended targets. The model routes between them from the tool descriptions alone.