Building a price comparison agent with the Oxylabs Web API

Oxylabs released a Web API with two endpoints, POST /v1/search and POST /v1/scrape, plus an MCP integration and an agent-oriented Markdown documentation page, aimed at AI agents that need live web data. In a hands-on test building a price comparison agent in Claude, the Best Buy product page for the Sony WH-1000XM5 returned 1,234,158 characters of HTML versus 10,318 characters of Markdown, about 120 times smaller, while enabling run_js: true grew the Markdown to 91,368 characters and surfaced the $228 price at a cost of 40.7 seconds versus 2.3 seconds without rendering. The API also accepts a two-letter location parameter, returning Amazon, camelcamelcamel and sony.com for a US search of "sony wh-1000xm5 price" and idealo.de, billiger.de and geizhals.de for DE, and offers AI parsing via output: ["json"] with either a natural-language prompt or an OpenAPI schema.

Every one of us who has tried to build an AI agent that fetches live data from the web has seen the limitations of the standard fetch tool included in the harness. You get blocked by anti-bot, the page is so big that your context is getting bloated by useless HTML, and so on. Today, for this issue of “Tool of the Week”, we’re testing the Oxylabs Web API https://oxylabs.go2cloud.org/aff c?offer id=7&aff id=665&url id=222 , a new product from Oxylabs that helps in building reliable agents, and we’ll see it live by building together a live price comparison agent in Claude. What the Oxylabs Web API is The Web API is composed of two endpoints, each with a different task, as you can see from the quick start guide https://developers.oxylabs.io/get-started/quick-start-web-api?utm source=thewebscrapingclub&utm medium=tool-review : - POST /v1/search takes a query and returns ranked results title, URL, a few truncated snippets called outline , plus related searches . - POST /v1/scrape takes a URL and returns the page as HTML, Markdown, JSON or a screenshot. On top of the generic scraper there’s a long list of dedicated scrapers, under /v1/scrape/{target}/{collection} , for sites like Amazon, Google and Best Buy, where Oxylabs maintains the parser for you. We also have an MCP integration https://developers.oxylabs.io/ai-workflows/mcp and the relative SKILL for the agent https://developers.oxylabs.io/ai-workflows/agent-skills so that it knows what to call and how when you need live web data. Additionally, there’s a page in the docs written for coding agents, For Agents https://developers.oxylabs.io/products/web-api/for-agents.md?utm source=thewebscrapingclub&utm medium=tool-review , that contains all the instructions in one Markdown file: both endpoint schemas, a working Python client, the error rules, and a checklist. In my opinion, this is the most interesting part of the documentation because it shows who Oxylabs believes the customer is. Every docs page also has a .md version, and you can append ?ask=<question &goal=<goal to get an answer generated from the docs. But let’s see the main features together. Markdown instead of HTML As every API thought for agents, also Oxylabs Web API can return both Markdown and plain HTML files. Markdowns are, of course, the preferred option if you’re working with LLMs. On the Best Buy product page of the Sony WH-1000XM5 the headphones I used for all the tests , the HTML returned by /v1/scrape is 1,234,158 characters. The Markdown of the same page is 10,318 characters, about 120 times smaller. Handling such a large HTML page will cost tokens and bloat all the context you’re working on. JavaScript rendering with run js Setting run js: true renders the page in a real browser before the results. On the same Best Buy page, the Markdown grew from 10,318 to 91,368 characters and contained the $228 price, which was not available in the plain HTML page. The cost is time: 2.3 seconds without rendering, 40.7 seconds with it. Geolocation Both endpoints accept a location parameter with a two-letter country code. On /search it changes the results in the way you’d expect for a shopping query. Searching “sony wh-1000xm5 price” with US returned Amazon, camelcamelcamel and sony.com, while with DE it returned idealo.de, billiger.de and geizhals.de. Using IT , we got idealo.it, sony.it and trovaprezzi.it. AI parsing AI parsing extracts JSON from any page in the same request that fetches it. You set output: "json" and pass either a natural language prompt or an OpenAPI schema . The docs say the schema mode “guarantees the response matches your defined types, nested structures, and required fields”, and recommend it for production. I wanted to see how stable it is, so I ran both modes three times on the Best Buy page, with and without JavaScript. This is the schema I used: { "type": "object", "properties": { "product name": {"type": "string"}, "price": {"type": "number", "description": "Current price the buyer pays, as a number"}, "currency": {"type": "string", "description": "ISO 4217 code"}, "availability": {"type": "string", "enum": "in stock", "out of stock", "preorder", "unknown" }, "seller": {"type": "string", "nullable": true} }, "required": "product name", "price", "currency", "availability" } With run js , the schema mode returned the complete object in 3 runs out of 3, including seller: "Best Buy" and availability: "in stock" . Without JavaScript, it returned price: null in 2 runs out of 3, even if price is marked as required. So the schema guarantees the shape of the output, not that every field is filled, and your code still needs to check for nulls. AI parsing is also the most expensive feature: $3 per 1,000 parsed pages, on top of the scrape. Dedicated scrapers and endpoint discovery One of the most interesting features for agents is having the list of managed APIs with a simple command. GET /v1/scrapers returns the list of every scrape endpoint, and OPTIONS on one of them returns its parameters, with type, constraints and a description. When I called it, the list had 351 endpoints, 160 of them synchronous, from Amazon and Google to grocery chains like Kroger and Publix, plus YouTube, ChatGPT, Perplexity and Gemini. python from client import describe scraper, list scrapers scrapers = list scrapers "scrapers" print len scrapers 351 print describe scraper "scrape/amazon/product" "query" string required min len: 10 max len: 18 . ASIN of the product, e.g. B0CX23V2ZK . An agent can read this at runtime instead of using a fixed list of websites and options. For amazon/product , for example, OPTIONS tells you that location is a postal code and not a country, and that you can force the currency . The output of a dedicated scraper is already parsed. amazon/product with an ASIN returned 50 fields in about 3 seconds price, buybox, stock, rating, reviews count and so on . bestbuy/product with the SKU returned title, price, is sold out and rating in 2.2 seconds, and walmart/product returned the price together with the marketplace seller that offers it. MCP server and agent skills If you don’t want to write code, Oxylabs gives you two ready integrations. The MCP server https://developers.oxylabs.io/ai-workflows/mcp?utm source=thewebscrapingclub&utm medium=tool-review uv tool install git+https://github.com/oxylabs/web-api-mcp exposes seven tools: search , scrape , extract , check scrape , read scraped for large pages that are saved to disk and read in chunks , list scrapers and scrape target . The agent skills https://developers.oxylabs.io/ai-workflows/agent-skills?utm source=thewebscrapingclub&utm medium=tool-review npx skills add oxylabs/web-api-skills give a coding agent the instructions on how to use the API, without a running server. For this article, I preferred to write my own tools because I wanted to control which endpoint the agent can call and see every request it makes. Building the price comparison agent The documentation says that “Installing beats building”: if the web access is for your coding agent, install the skills or the MCP server and you’re done. The same page says to build a client “when the project’s code has to make the calls itself”. Our agent is a program that runs on its own, so we’re in the second case. But I could still have connected it to the official MCP server instead of writing five tools, but I preferred writing some light wrappers. With the Claude API, the model can’t make HTTP calls on its own. It can only ask to use a tool that you declared, and your code runs it. So my tools are just thin wrappers, a few lines each, around a single Web API call, created for mainly these four reasons. The first is visibility. Every call of my tools goes through client.py , which logs status, latency, request id and size. All the numbers in this article come from that log, and with the MCP server the calls happen inside another process. The second is control. As you’ll see in the UK run, two details changed the result: a single retry on a failing page and where to cut a long page before passing it to the model. With my own code, I was able to change both in one line. The third is the tool descriptions. They’re what the model reads to choose a tool, and in mine I wrote the rules we found in the tests above, like “if the price is missing, call it again with run js=true”. The fourth is cost. The agent can call only five endpoints, so it can’t choose the more expensive ones Google with JavaScript, ChatGPT, or Perplexity scrapers for a price check. Now the tutorial. The goal is an agent that receives a product name and a country, finds the product in three online shops of that country and returns a JSON array with price, currency, availability and the tool it used for each shop. It stops at three results, because the point here is to see how the agent uses the tools, not to build the biggest price list. I used Claude Opus 5.5 through the Anthropic Python SDK, with a plain tool use loop. There are higher-level options, like the SDK’s tool runner or connecting Claude to the Oxylabs MCP server, but with a manual loop you can print every tool call and see why the agent does what it does. The only dependencies are anthropic , requests and python-dotenv , which were already installed in our environment. You need two keys in a .env file: OXYLABS WEB API KEY and ANTHROPIC API KEY . The tools The agent gets five tools, and each one maps to a single Web API call: - web search calls /v1/search - scrape page calls the generic /v1/scrape on any URL and returns Markdown, with optional run js - extract product calls the generic /v1/scrape with AI parsing and the schema above - amazon product and bestbuy product call the dedicated scrapers by ASIN and SKU The descriptions are where you put the knowledge from the tests above, because they’re what the model reads when it chooses a tool. This is the definition of scrape page : { "name": "scrape page", "description": "Read any URL as Markdown Oxylabs generic /v1/scrape . Fast without JavaScript. " "If the price or the main content is missing, call it again with run js=true " " real browser render, 20-40 seconds .", "input schema": { "type": "object", "properties": { "url": {"type": "string"}, "run js": {"type": "boolean", "default": False}, }, "required": "url" , }, } And this is the code that runs the tools. Pages longer than 60,000 characters are cut, starting from the first H1 heading, and the agent is told about the cut. I’ll explain in the UK run why the cut starts from the H1 and not from the top of the page. php def run tool name: str, args: dict - dict: if name == "web search": out = search args "query" , max results=10, location=args.get "location" return {"results": {"title": r "title" , "url": r "url" , "outline": r.get "outline", :1 } for r in out "results" } if name == "scrape page": Oxylabs already retries a failed scrape server-side before answering 500, so the agent gets the error once and decides what to do usually run js=true . out = scrape args "url" , output= "markdown" , run js=bool args.get "run js" , attempts=1 md = out "results" 0 .get "markdown" or "" result = {"chars": len md , "markdown": md :MAX PAGE CHARS } if len md MAX PAGE CHARS: Rendered pages can start with 100K+ chars of menus: start from the first H1 the product title . h1 = md.find "\n " start = h1 if h1 0 else 0 result "markdown" = md start:start + MAX PAGE CHARS result "note" = f"{len md } chars, showing {start}-{start + MAX PAGE CHARS} from the first H1 " return result if name == "extract product": out = scrape args "url" , output= "json" , json={"schema": PRICE SCHEMA}, run js=bool args.get "run js" return out "results" 0 .get "json" or {} if name == "amazon product": out = scrape target "amazon/product", {"query": args "asin" , "domain": args.get "domain", "com" , "output": "json" } j = out "results" 0 .get "json" or {} keep = "title", "price", "price buybox", "currency", "stock", "rating", "reviews count", "url" return {k: j.get k for k in keep} if name == "bestbuy product": out = scrape target "bestbuy/product", {"query": args "sku" , "output": "json" } return out "results" 0 .get "json" or {} raise ValueError f"unknown tool {name}" For scrape page I turned off the client retries attempts=1 . In my first version the client retried a failing page three times, and since Oxylabs had already retried on its side, the agent waited more than two minutes before seeing the error. Now it gets the error at the first failure and can switch to run js immediately. The system prompt The system prompt contains the rules from the For Agents page, adapted to prices: You are a price research agent. The user gives you a product and a country. Find the current price of that exact product in 3 different online shops of that country, then stop. Rules: - Start with one web search. Choose shop product pages, not reviews, forums or price trackers. - Read prices only from pages you scraped. Never use search snippets or your memory for a price. - For Amazon use amazon product, for Best Buy use bestbuy product. For other shops use scrape page first. - If scrape page fails or returns a page without the product price, call it again with run js=true. - Use extract product only when the Markdown is ambiguous. - If a page fails twice, report it as failed and pick another shop. - Stop as soon as you have 3 prices. - If the user lists shop URLs, skip the search and read exactly those pages. Final answer: a JSON array with one object per shop: {"shop", "url", "price", "currency", "availability", "tool used", "run js"} followed by one short line on anything that failed. The rule about search snippets isn’t there by chance, and we’ll see why in a moment. The loop The loop sends the conversation to Claude, runs every tool call in the response, sends back the results and repeats until the model stops asking for tools. A tool error goes back as a tool result with is error: true , so the model can react to it like any other result. for turn in range 20 : response = client.messages.create model=MODEL, max tokens=16000, system=SYSTEM, tools=TOOLS, messages=messages, messages.append {"role": "assistant", "content": response.content} if response.stop reason = "tool use": final = "".join b.text for b in response.content if b.type == "text" break results = for block in response.content: if block.type = "tool use": continue try: output, is error = run tool block.name, block.input , False except WebApiError, KeyError, IndexError, ValueError as exc: output, is error = {"error": str exc }, True results.append {"type": "tool result", "tool use id": block.id, "content": json.dumps output, ensure ascii=False , "is error": is error} messages.append {"role": "user", "content": results} The full script, with the logging I removed here, is agent.py in the oxylabs web api agent folder of our GitHub repository https://github.com/TheWebScrapingClub/TheScrapingClubFree/tree/main/Oxylabs%20Web%20API . You run it with python agent.py "Sony WH-1000XM5" US , or you add up to three shop URLs after the country to skip the search. Running the agent United States The first run, on the US market, took 22 seconds and four tool calls. The agent searched once, then called the three shops in parallel in the same turn: turn 0 web search {"query": "Sony WH-1000XM5 headphones buy", "location": "US"} - 1.7s turn 1 amazon product {"asin": "B09XS7JWHH"} - 3.1s, 199.99 USD, In Stock turn 1 bestbuy product {"sku": "6505727"} - 2.4s, 228 USD, is sold out: false turn 1 scrape page {"url": "https://electronics.sony.com/audio/headphones/headband/p/wh1000xm5-b"} - 5.3s, 23,023 chars Amazon at $199.99 and Best Buy at $228 came from the dedicated scrapers. The Sony store came from the generic scraper, with $299.99 marked down from $399.99 . The agent then added a note on its own. The search snippet for the Sony page said “Sale Price $228.00 Save $171.99”, so maybe the Markdown without JavaScript was stale and the page should be checked fully loaded. I checked it manually and the Sony page shows $299.99 with and without run js 4.1 seconds against 23.6 , so the price in the Markdown was right, and the snippet was wrong. That’s exactly why the docs tell you to never answer from outline , and why that rule is in the system prompt. Italy On the Italian market location: IT the agent searched “Sony WH-1000XM5 prezzo”, used the Amazon dedicated scraper with domain: it and read Unieuro and the Sony Store Italia with the generic scraper. Everything was in the static Markdown: Amazon.it at €216.49, Unieuro at €199.00 and the Sony Store at €259.00. Total time was 25 seconds with four tool calls, and the agent also noted that Unieuro adds an extra 5% discount in the cart, which isn’t in the €199 price. United Kingdom, where JavaScript is needed In the first two markets, every page the agent read with the generic scraper had the price in the static Markdown. To see the JavaScript fallback in action, I needed a shop without a dedicated scraper that doesn’t work without rendering. To find one, I ran the same search in seven countries, scraped every shop page without JavaScript and checked which ones had no price. Argos UK was the clearest case. The scrape without JavaScript failed every time I tried it four times, between 22 and 43 seconds each , always with the 500 REQUEST FAILED AFTER TOO MANY RETRIES that isn’t charged. With run js the page came back in about 22 seconds, with the price. So for this run I passed the agent three UK URLs: Argos, Currys and Amazon UK. php turn 0 scrape page {"url": "https://www.argos.co.uk/product/2847319"} - 21.9s error: 500 REQUEST FAILED AFTER TOO MANY RETRIES turn 0 scrape page {"url": "https://www.currys.co.uk/products/sony-wh1000xm5-...-10238756.html"} - 4.4s, 41,021 chars turn 0 amazon product {"asin": "B09Y2MYL5C", "domain": "co.uk"} - 2.8s, 199 GBP, In stock turn 1 scrape page {"url": "https://www.argos.co.uk/product/2847319", "run js": true} - 22.6s 172,012 chars, showing 137143-197143 from the first H1 The agent read the error, followed the rule in the system prompt and asked for the same page with run js: true . The final answer had Currys at £199.00 from the static Markdown, Argos at £229.99 from the rendered Markdown and Amazon UK at £199.00 from the dedicated scraper. Total time was 62 seconds with four tool calls. For Argos the agent also wrote that it couldn’t confirm availability, because the page shows an “Add to trolley” button but the stock depends on the postcode. This run didn’t work at the first attempt, and the reason is worth telling. The rendered Argos page is 172,012 characters of Markdown, and the product title starts at character 137,143. Everything before it is menus, mega menus and navigation links. In my first version the tool cut the page at the first 60,000 characters, so the agent received only the menu. It then called extract product with run js another 42 seconds to get the price, and it couldn’t read the availability. After I changed the cut to start from the first H1, the same run went from 153 to 62 seconds and from 130,000 to 62,000 input tokens. With rendered pages, the Markdown conversion doesn’t remove the navigation, so it’s up to you to decide which part of the page the model reads. The same first run also failed on Amazon UK with a 500 PARSE NOT SUPPORTED . That one was my error: I had passed the amazon.com ASIN of the headphones, and on amazon.co.uk the product has a different one B09Y2MYL5C, the same as on amazon.it . What it costs Oxylabs prices https://oxylabs.io/pricing?utm source=thewebscrapingclub&utm medium=tool-review start with a Starter plan at $30 per month. At that tier, /search costs $1 per 1,000 searches, the generic /scrape starts from $0.50 per 1,000 results, custom domains with JavaScript cost $1.50, Amazon without JavaScript costs $0.50 and AI parsing $3 per 1,000 parsed pages. Volume discounts bring the generic scrape down to $0.25 per 1,000 above 10 million results. So one run of our agent, with a search and three scrapes, costs a fraction of a cent on the Oxylabs side, even when one of the pages needs JavaScript. The bigger cost is the LLM. The US run used 20,472 input tokens and 848 output tokens, which at the Claude Opus 5.5 price $4 per million input tokens and $20 per million output tokens is about 10 cents. The UK run, with the rendered Argos page, used 61,759 input tokens, about 26 cents. Most of those tokens are page Markdown, so the 120 times reduction from HTML to Markdown is what makes this agent affordable at all, and what you cut from the Markdown before passing it to the model is the next place to save money. Final considerations The Oxylabs Web API https://oxylabs.go2cloud.org/aff c?offer id=7&aff id=665&url id=222 does what the quick start promises: a search endpoint and a scrape endpoint that an agent can use without knowing anything about proxies or rendering. In my tests, the searches answered in under 2 seconds, the dedicated scrapers in 2 to 4 seconds, and the generic scraper handled every shop we sent it to, using run js on the one that needed it. What I liked most is that the product is clearly designed for agents. You see it in the documentation, in the Markdown output, in the endpoint discovery that a model can read at runtime, and in the faulted scrapes that you don’t pay for. The part that still depends on you is the agent itself: which tools you expose, what you write in their descriptions, and how much of each page you pass to the model. In our case, the difference between a slow, expensive run and a fast, cheap one was a single line that chose where to cut the Markdown. The next step for this agent would be a discovery tool so it can automatically pick the dedicated scraper for every shop that has one, instead of the two I gave it. Let me know in the comments if you’ve already tried the Web API, and which agent you’d build with it.