Every one of us who has tried to build an AI agent that fetches live data from the web has seen the limitations of the standard fetch tool included in the harness. You get blocked by anti-bot, the page is so big that your context is getting bloated by useless HTML, and so on.
Today, for this issue of “Tool of the Week”, we’re testing the Oxylabs Web API, a new product from Oxylabs that helps in building reliable agents, and we’ll see it live by building together a live price comparison agent in Claude.
What the Oxylabs Web API is #
The Web API is composed of two endpoints, each with a different task, as you can see from the quick start guide:
POST /v1/searchtakes a query and returns ranked results (title, URL, a few truncated snippets calledoutline, plus related searches).POST /v1/scrapetakes a URL and returns the page as HTML, Markdown, JSON or a screenshot.
On top of the generic scraper there’s a long list of dedicated scrapers, under /v1/scrape/{target}/{collection}, for sites like Amazon, Google and Best Buy, where Oxylabs maintains the parser for you.
We also have an MCP integration and the relative SKILL for the agent so that it knows what to call and how when you need live web data.
Additionally, there’s a page in the docs written for coding agents, For Agents, that contains all the instructions in one Markdown file: both endpoint schemas, a working Python client, the error rules, and a checklist.
In my opinion, this is the most interesting part of the documentation because it shows who Oxylabs believes the customer is. Every docs page also has a .md version, and you can append ?ask=<question>&goal=<goal> to get an answer generated from the docs.
But let’s see the main features together.
Markdown instead of HTML
As every API thought for agents, also Oxylabs Web API can return both Markdown and plain HTML files. Markdowns are, of course, the preferred option if you’re working with LLMs. On the Best Buy product page of the Sony WH-1000XM5 (the headphones I used for all the tests), the HTML returned by /v1/scrape is 1,234,158 characters. The Markdown of the same page is 10,318 characters, about 120 times smaller. Handling such a large HTML page will cost tokens and bloat all the context you’re working on.
JavaScript rendering with run_js
Setting run_js: true renders the page in a real browser before the results. On the same Best Buy page, the Markdown grew from 10,318 to 91,368 characters and contained the $228 price, which was not available in the plain HTML page. The cost is time: 2.3 seconds without rendering, 40.7 seconds with it.
Geolocation
Both endpoints accept a location parameter with a two-letter country code. On /search it changes the results in the way you’d expect for a shopping query. Searching “sony wh-1000xm5 price” with US returned Amazon, camelcamelcamel and sony.com, while with DE it returned idealo.de, billiger.de and geizhals.de. UsingIT , we got idealo.it, sony.it and trovaprezzi.it.
AI parsing
AI parsing extracts JSON from any page in the same request that fetches it. You set output: ["json"] and pass either a natural language prompt or an OpenAPI schema. The docs say the schema mode “guarantees the response matches your defined types, nested structures, and required fields”, and recommend it for production.
I wanted to see how stable it is, so I ran both modes three times on the Best Buy page, with and without JavaScript. This is the schema I used:
{
"type": "object",
"properties": {
"product_name": {"type": "string"},
"price": {"type": "number", "description": "Current price the buyer pays, as a number"},
"currency": {"type": "string", "description": "ISO 4217 code"},
"availability": {"type": "string", "enum": ["in_stock", "out_of_stock", "preorder", "unknown"]},
"seller": {"type": "string", "nullable": true}
},
"required": ["product_name", "price", "currency", "availability"]
}
With run_js, the schema mode returned the complete object in 3 runs out of 3, including seller: "Best Buy" and availability: "in_stock".
Without JavaScript, it returned price: null in 2 runs out of 3, even if price is marked as required. So the schema guarantees the shape of the output, not that every field is filled, and your code still needs to check for nulls.
AI parsing is also the most expensive feature: $3 per 1,000 parsed pages, on top of the scrape.
Dedicated scrapers and endpoint discovery
One of the most interesting features for agents is having the list of managed APIs with a simple command. GET /v1/scrapers returns the list of every scrape endpoint, and OPTIONS on one of them returns its parameters, with type, constraints and a description. When I called it, the list had 351 endpoints, 160 of them synchronous, from Amazon and Google to grocery chains like Kroger and Publix, plus YouTube, ChatGPT, Perplexity and Gemini.
from client import describe_scraper, list_scrapers
scrapers = list_scrapers()["scrapers"]
print(len(scrapers)) # 351
print(describe_scraper("scrape/amazon/product")["query"])
An agent can read this at runtime instead of using a fixed list of websites and options. For amazon/product, for example, OPTIONS tells you that location is a postal code and not a country, and that you can force the currency.
The output of a dedicated scraper is already parsed. amazon/product with an ASIN returned 50 fields in about 3 seconds (price, buybox, stock, rating, reviews count and so on). bestbuy/product with the SKU returned title, price, is_sold_out and rating in 2.2 seconds, and walmart/product returned the price together with the marketplace seller that offers it.
MCP server and agent skills
If you don’t want to write code, Oxylabs gives you two ready integrations. The MCP server (uv tool install git+https://github.com/oxylabs/web-api-mcp) exposes seven tools: search, scrape, extract, check_scrape, read_scraped (for large pages that are saved to disk and read in chunks), list_scrapers and scrape_target. The agent skills (npx skills add oxylabs/web-api-skills) give a coding agent the instructions on how to use the API, without a running server. For this article, I preferred to write my own tools because I wanted to control which endpoint the agent can call and see every request it makes.
Building the price comparison agent #
The documentation says that “Installing beats building”: if the web access is for your coding agent, install the skills or the MCP server and you’re done. The same page says to build a client “when the project’s code has to make the calls itself”.
Our agent is a program that runs on its own, so we’re in the second case. But I could still have connected it to the official MCP server instead of writing five tools, but I preferred writing some light wrappers.
With the Claude API, the model can’t make HTTP calls on its own. It can only ask to use a tool that you declared, and your code runs it. So my tools are just thin wrappers, a few lines each, around a single Web API call, created for mainly these four reasons.
The first is visibility. Every call of my tools goes through client.py, which logs status, latency, request id and size. All the numbers in this article come from that log, and with the MCP server the calls happen inside another process. The second is control. As you’ll see in the UK run, two details changed the result: a single retry on a failing page and where to cut a long page before passing it to the model. With my own code, I was able to change both in one line. The third is the tool descriptions. They’re what the model reads to choose a tool, and in mine I wrote the rules we found in the tests above, like “if the price is missing, call it again with run_js=true”. The fourth is cost. The agent can call only five endpoints, so it can’t choose the more expensive ones (Google with JavaScript, ChatGPT, or Perplexity scrapers) for a price check.
Now the tutorial. The goal is an agent that receives a product name and a country, finds the product in three online shops of that country and returns a JSON array with price, currency, availability and the tool it used for each shop. It stops at three results, because the point here is to see how the agent uses the tools, not to build the biggest price list.
I used Claude Opus 5.5 through the Anthropic Python SDK, with a plain tool use loop. There are higher-level options, like the SDK’s tool runner or connecting Claude to the Oxylabs MCP server, but with a manual loop you can print every tool call and see why the agent does what it does. The only dependencies are anthropic, requests and python-dotenv, which were already installed in our environment. You need two keys in a .env file: OXYLABS_WEB_API_KEY and ANTHROPIC_API_KEY.
The tools
The agent gets five tools, and each one maps to a single Web API call:
web_searchcalls/v1/searchscrape_pagecalls the generic/v1/scrapeon any URL and returns Markdown, with optionalrun_jsextract_productcalls the generic/v1/scrapewith AI parsing and the schema aboveamazon_productandbestbuy_productcall the dedicated scrapers by ASIN and SKU
The descriptions are where you put the knowledge from the tests above, because they’re what the model reads when it chooses a tool. This is the definition of scrape_page:
{
"name": "scrape_page",
"description": "Read any URL as Markdown (Oxylabs generic /v1/scrape). Fast without JavaScript. "
"If the price or the main content is missing, call it again with run_js=true "
"(real browser render, 20-40 seconds).",
"input_schema": {
"type": "object",
"properties": {
"url": {"type": "string"},
"run_js": {"type": "boolean", "default": False},
},
"required": ["url"],
},
}
And this is the code that runs the tools. Pages longer than 60,000 characters are cut, starting from the first H1 heading, and the agent is told about the cut. I’ll explain in the UK run why the cut starts from the H1 and not from the top of the page.
def run_tool(name: str, args: dict) -> dict:
if name == "web_search":
out = search(args["query"], max_results=10, location=args.get("location"))
return {"results": [{"title": r["title"], "url": r["url"], "outline": r.get("outline", [])[:1]}
for r in out["results"]]}
if name == "scrape_page":
out = scrape(args["url"], output=["markdown"], run_js=bool(args.get("run_js")), attempts=1)
md = out["results"][0].get("markdown") or ""
result = {"chars": len(md), "markdown": md[:MAX_PAGE_CHARS]}
if len(md) > MAX_PAGE_CHARS:
h1 = md.find("\n# ")
start = h1 if h1 > 0 else 0
result["markdown"] = md[start:start + MAX_PAGE_CHARS]
result["note"] = f"{len(md)} chars, showing {start}-{start + MAX_PAGE_CHARS} (from the first H1)"
return result
if name == "extract_product":
out = scrape(args["url"], output=["json"], json={"schema": PRICE_SCHEMA}, run_js=bool(args.get("run_js")))
return out["results"][0].get("json") or {}
if name == "amazon_product":
out = scrape_target("amazon/product", {"query": args["asin"], "domain": args.get("domain", "com"),
"output": ["json"]})
j = out["results"][0].get("json") or {}
keep = ("title", "price", "price_buybox", "currency", "stock", "rating", "reviews_count", "url")
return {k: j.get(k) for k in keep}
if name == "bestbuy_product":
out = scrape_target("bestbuy/product", {"query": args["sku"], "output": ["json"]})
return out["results"][0].get("json") or {}
raise ValueError(f"unknown tool {name}")
For scrape_page I turned off the client retries (attempts=1). In my first version the client retried a failing page three times, and since Oxylabs had already retried on its side, the agent waited more than two minutes before seeing the error. Now it gets the error at the first failure and can switch to run_js immediately.
The system prompt
The system prompt contains the rules from the For Agents page, adapted to prices:
You are a price research agent. The user gives you a product and a country.
Find the current price of that exact product in 3 different online shops of that country, then stop.
Rules:
- Start with one web_search. Choose shop product pages, not reviews, forums or price trackers.
- Read prices only from pages you scraped. Never use search snippets or your memory for a price.
- For Amazon use amazon_product, for Best Buy use bestbuy_product. For other shops use scrape_page first.
- If scrape_page fails or returns a page without the product price, call it again with run_js=true.
- Use extract_product only when the Markdown is ambiguous.
- If a page fails twice, report it as failed and pick another shop.
- Stop as soon as you have 3 prices.
- If the user lists shop URLs, skip the search and read exactly those pages.
Final answer: a JSON array with one object per shop:
{"shop", "url", "price", "currency", "availability", "tool_used", "run_js"}
followed by one short line on anything that failed.
The rule about search snippets isn’t there by chance, and we’ll see why in a moment.
The loop
The loop sends the conversation to Claude, runs every tool call in the response, sends back the results and repeats until the model stops asking for tools. A tool error goes back as a tool_result with is_error: true, so the model can react to it like any other result.
for turn in range(20):
response = client.messages.create(
model=MODEL, max_tokens=16000, system=SYSTEM, tools=TOOLS, messages=messages,
)
messages.append({"role": "assistant", "content": response.content})
if response.stop_reason != "tool_use":
final = "".join(b.text for b in response.content if b.type == "text")
break
results = []
for block in response.content:
if block.type != "tool_use":
continue
try:
output, is_error = run_tool(block.name, block.input), False
except (WebApiError, KeyError, IndexError, ValueError) as exc:
output, is_error = {"error": str(exc)}, True
results.append({"type": "tool_result", "tool_use_id": block.id,
"content": json.dumps(output, ensure_ascii=False), "is_error": is_error})
messages.append({"role": "user", "content": results})
The full script, with the logging I removed here, is agent.py in the oxylabs_web_api_agent folder of our GitHub repository. You run it with python agent.py "Sony WH-1000XM5" US, or you add up to three shop URLs after the country to skip the search.
Running the agent #
United States
The first run, on the US market, took 22 seconds and four tool calls. The agent searched once, then called the three shops in parallel in the same turn:
[turn 0] web_search({"query": "Sony WH-1000XM5 headphones buy", "location": "US"}) -> 1.7s
[turn 1] amazon_product({"asin": "B09XS7JWHH"}) -> 3.1s, 199.99 USD, In Stock
[turn 1] bestbuy_product({"sku": "6505727"}) -> 2.4s, 228 USD, is_sold_out: false
[turn 1] scrape_page({"url": "https://electronics.sony.com/audio/headphones/headband/p/wh1000xm5-b"}) -> 5.3s, 23,023 chars
Amazon at $199.99 and Best Buy at $228 came from the dedicated scrapers. The Sony store came from the generic scraper, with $299.99 (marked down from $399.99). The agent then added a note on its own. The search snippet for the Sony page said “Sale Price $228.00 Save $171.99”, so maybe the Markdown without JavaScript was stale and the page should be checked fully loaded.
I checked it manually and the Sony page shows $299.99 with and without run_js (4.1 seconds against 23.6), so the price in the Markdown was right, and the snippet was wrong. That’s exactly why the docs tell you to never answer fromoutline, and why that rule is in the system prompt.
Italy
On the Italian market (location: IT) the agent searched “Sony WH-1000XM5 prezzo”, used the Amazon dedicated scraper with domain: it and read Unieuro and the Sony Store Italia with the generic scraper. Everything was in the static Markdown: Amazon.it at €216.49, Unieuro at €199.00 and the Sony Store at €259.00. Total time was 25 seconds with four tool calls, and the agent also noted that Unieuro adds an extra 5% discount in the cart, which isn’t in the €199 price.
United Kingdom, where JavaScript is needed
In the first two markets, every page the agent read with the generic scraper had the price in the static Markdown. To see the JavaScript fallback in action, I needed a shop without a dedicated scraper that doesn’t work without rendering. To find one, I ran the same search in seven countries, scraped every shop page without JavaScript and checked which ones had no price.
Argos UK was the clearest case. The scrape without JavaScript failed every time I tried it (four times, between 22 and 43 seconds each), always with the 500 REQUEST_FAILED_AFTER_TOO_MANY_RETRIES that isn’t charged. With run_js the page came back in about 22 seconds, with the price. So for this run I passed the agent three UK URLs: Argos, Currys and Amazon UK.
[turn 0] scrape_page({"url": "https://www.argos.co.uk/product/2847319"}) -> 21.9s
error: 500 REQUEST_FAILED_AFTER_TOO_MANY_RETRIES
[turn 0] scrape_page({"url": "https://www.currys.co.uk/products/sony-wh1000xm5-...-10238756.html"}) -> 4.4s, 41,021 chars
[turn 0] amazon_product({"asin": "B09Y2MYL5C", "domain": "co.uk"}) -> 2.8s, 199 GBP, In stock
[turn 1] scrape_page({"url": "https://www.argos.co.uk/product/2847319", "run_js": true}) -> 22.6s
172,012 chars, showing 137143-197143 (from the first H1)
The agent read the error, followed the rule in the system prompt and asked for the same page with run_js: true. The final answer had Currys at £199.00 from the static Markdown, Argos at £229.99 from the rendered Markdown and Amazon UK at £199.00 from the dedicated scraper. Total time was 62 seconds with four tool calls. For Argos the agent also wrote that it couldn’t confirm availability, because the page shows an “Add to trolley” button but the stock depends on the postcode.
This run didn’t work at the first attempt, and the reason is worth telling. The rendered Argos page is 172,012 characters of Markdown, and the product title starts at character 137,143. Everything before it is menus, mega menus and navigation links. In my first version the tool cut the page at the first 60,000 characters, so the agent received only the menu. It then called extract_product with run_js (another 42 seconds) to get the price, and it couldn’t read the availability. After I changed the cut to start from the first H1, the same run went from 153 to 62 seconds and from 130,000 to 62,000 input tokens. With rendered pages, the Markdown conversion doesn’t remove the navigation, so it’s up to you to decide which part of the page the model reads.
The same first run also failed on Amazon UK with a 500 PARSE_NOT_SUPPORTED. That one was my error: I had passed the amazon.com ASIN of the headphones, and on amazon.co.uk the product has a different one (B09Y2MYL5C, the same as on amazon.it).
What it costs #
Oxylabs prices start with a Starter plan at $30 per month. At that tier, /search costs $1 per 1,000 searches, the generic /scrape starts from $0.50 per 1,000 results, custom domains with JavaScript cost $1.50, Amazon without JavaScript costs $0.50 and AI parsing $3 per 1,000 parsed pages. Volume discounts bring the generic scrape down to $0.25 per 1,000 above 10 million results.
So one run of our agent, with a search and three scrapes, costs a fraction of a cent on the Oxylabs side, even when one of the pages needs JavaScript. The bigger cost is the LLM. The US run used 20,472 input tokens and 848 output tokens, which at the Claude Opus 5.5 price ($4 per million input tokens and $20 per million output tokens) is about 10 cents. The UK run, with the rendered Argos page, used 61,759 input tokens, about 26 cents. Most of those tokens are page Markdown, so the 120 times reduction from HTML to Markdown is what makes this agent affordable at all, and what you cut from the Markdown before passing it to the model is the next place to save money.
Final considerations #
The Oxylabs Web API does what the quick start promises: a search endpoint and a scrape endpoint that an agent can use without knowing anything about proxies or rendering. In my tests, the searches answered in under 2 seconds, the dedicated scrapers in 2 to 4 seconds, and the generic scraper handled every shop we sent it to, using run_js on the one that needed it.
What I liked most is that the product is clearly designed for agents. You see it in the documentation, in the Markdown output, in the endpoint discovery that a model can read at runtime, and in the faulted scrapes that you don’t pay for.
The part that still depends on you is the agent itself: which tools you expose, what you write in their descriptions, and how much of each page you pass to the model. In our case, the difference between a slow, expensive run and a fast, cheap one was a single line that chose where to cut the Markdown. The next step for this agent would be a discovery tool so it can automatically pick the dedicated scraper for every shop that has one, instead of the two I gave it.
Let me know in the comments if you’ve already tried the Web API, and which agent you’d build with it.