{"slug": "how-to-build-a-price-monitoring-agent-with-pydantic-ai-and-zenrows", "title": "How to Build a Price Monitoring Agent with Pydantic AI and ZenRows", "summary": "A developer built a price monitoring agent using ZenRows and Pydantic AI that achieves a 99.93% success rate against protected e-commerce sites like Amazon and Walmart. The system addresses two common failure modes: retrieval failures where protected sites return blank pages or block requests, and extraction failures where LLM output drifts in data types across runs. The tutorial demonstrates a two-step pipeline that retrieves product pages via ZenRows and validates LLM output against a defined schema using Pydantic AI.", "body_md": "Most price monitoring systems look simple on paper: point a scraper at a product page, ask an LLM to extract the price, and store the result in a database. The problem arises when the same workflow runs continuously against heavily protected targets such as Amazon and Walmart. At scale, reliability becomes the real challenge. The why comes down to two failures that happen quietly: retrieval and extraction. Retrieval fails when a protected site returns a blank JavaScript shell, incomplete data, and a block page instead of the product data. Extraction fails when the LLM returns a field in a data type or shape that your downstream database write action does not expect. Both failures pass for normal output, so the pipeline keeps running and drops records downstream without raising an error.\n\nThis tutorial shows you how you can build a price monitoring agent using ZenRows and Pydantic AI. ZenRows retrieves the page and returns product details with a [99.93% success rate](https://www.zenrows.com/) against protected websites, while Pydantic AI extracts and validates LLM output against a defined schema. The tutorial walks through the two-step pipeline from a single product page to a multi-site price monitoring run across Amazon and Walmart, two heavily protected e-commerce sites.\n\nTwo failure modes break the price monitoring pipeline: retrieval and extraction.\n\nYou send a request to the protected site, and the response looks successful at the HTTP level. But the page still contains no usable product data or encounters [an error 1020](https://www.zenrows.com/blog/cloudflare-error-1020#what-is-error-1020). The code below shows this failed retrieval directly.\n\n``` python\nimport requests\n\n# Demo page that behaves like a protected target\nPROTECTED_URL = \"https://www.scrapingcourse.com/antibot-challenge\"\n\nresp = requests.get(PROTECTED_URL, timeout=30)\n\nprint(f\"status code: {resp.status_code}\")\nprint(f\"bytes returned: {len(resp.text)}\")\nprint(\"contains 'price'?\", \"price\" in resp.text.lower())\nprint(resp.text[:200])\n```\n\nThe request returns a 403, so the pipeline does not receive the product page required for price monitoring.\n\n```\nbenny@Mac price_monitoring % python3 test.py \nstatus code: 403\nbytes returned: 5507\ncontains 'price'? False\nJust a moment... None:\n if not isinstance(price, float):\n raise TypeError(\n f\"expected float, got {type(price).__name__}\"\n )\n\nfor i, response in enumerate(raw_llm_responses, start=1):\n price = response[\"price\"]\n\n print(\n f\"run {i}: price={price!r}, \"\n f\"type={type(price).__name__}\"\n )\n\nfor i, response in enumerate(raw_llm_responses, start=1):\n try:\n write_to_db(response[\"price\"])\n print(f\"run {i}: write OK\")\n\n except TypeError as e:\n print(f\"run {i}: write FAILED -> {e}\")\n```\n\nBelow is the output of the extraction script. It returns valid-looking data, but the output schema can drift between runs. Without validation, downstream writes fail when the data type does not match the expected format.\n\n```\nbenny@Mac price_monitoring % python3 test_llm.py\nrun 1: price=29.99, type=float\nrun 2: price='$29.99', type=str\nrun 1: write OK\nrun 2: write FAILED -> expected float, got str\nbenny@Mac price_monitoring %\n```\n\nSuch inconsistencies can break validation logic and downstream workflows, causing pipelines to fail unpredictably during automated or scheduled executions. Next, let's take a look at how ZenRows handles the retrieval problem. However, before we start, let's see everything you need to follow along.\n\nUse Python 3.10 or newer for this tutorial. If you have an older Python version, you need to create a dedicated virtual environment with a current Python installation to avoid dependency conflicts.\n\n`python -m pip install \"pydantic-ai-slim[anthropic]\" requests python-dotenv`\n\n. This tutorial was tested locally with Pydantic AI 0.8.1 and Anthropic SDK 0.111.0.\nOne way to address retrieval failures is to use a scraping API to retrieve data from protected pages and pass the page content to the extraction step.\n\n[ZenRows web scraping API](https://www.zenrows.com/products/universal-scraper) handles the retrieval layer for protected pages. It renders the pages, uses proxies, and returns markdown in a single request. You can then pass this output to your extraction agent for the next stage of your pipeline, which you will see in the next section of this piece. The code below shows this flow with `js_render`\n\nand `premium_proxy`\n\n.\n\n``` python\nimport os\nimport requests\nfrom dotenv import load_dotenv\n\nload_dotenv()\n\nZENROWS_API_KEY = os.environ[\"ZENROWS_API_KEY\"]\n\n# Target page (NO markdown formatting)\nurl = \"https://www.scrapingcourse.com/ecommerce/\"\n\nparams = {\n \"url\": url,\n \"apikey\": ZENROWS_API_KEY,\n \"js_render\": \"true\",\n \"premium_proxy\": \"true\",\n \"proxy_country\": \"us\",\n \"response_type\": \"markdown\",\n}\n\nresponse = requests.get(\n \"https://api.zenrows.com/v1/\",\n params=params,\n timeout=60\n)\n\nprint(f\"status code: {response.status_code}\")\nprint(f\"bytes returned: {len(response.text)}\")\nprint(\"\\n\".join(response.text.splitlines()[:30]))\n```\n\nZenRows manages anti-bot bypass via `\"js_render\": \"true\"`\n\nwhich ensures ZenRows loads the page like a real user browser, and `\"premium_proxy\": \"true\"`\n\nwhich enables residential proxies. For this use case, set `proxy_country=us`\n\nto route requests through a US IP address. That keeps our price comparisons consistent across runs. The response type is `markdown`\n\n, which is easier for LLMs to parse and read. Your response after running the code should be similar to the output below.\n\n```\nbenny@Mac price_monitoring % python3 zenrows_store.py\nstatus code: 200\nbytes returned: 6777\n[Skip to navigation](http://www.scrapingcourse.com#site-navigation) [Skip to content](http://www.scrapingcourse.com#content)\n\n[Ecommerce Test Site to Learn Web Scraping](https://www.scrapingcourse.com/ecommerce/)\n\nScrapingCourse.com\n\nSearch for: Search\n\nMenu\n\n- [Shop](https://www.scrapingcourse.com/ecommerce/)\n\n\n\n- [Home](https://www.scrapingcourse.com/ecommerce/)\n- [Cart](https://www.scrapingcourse.com/ecommerce/cart/)\n- [Checkout](https://www.scrapingcourse.com/ecommerce/checkout/)\n- [My account](https://www.scrapingcourse.com/ecommerce/my-account/)\n\n\n\n- [$0.00 0 items](https://www.scrapingcourse.com/ecommerce/cart/ \"View your shopping cart\")\n- No products in the cart.\n\n# Shop\n\nDefault sorting Sort by popularity Sort by latest Sort by price: low to high Sort by price: high to low\n\nShowing 1-16 of 188 results\n\nbenny@Mac price_monitoring % ;\n```\n\nNow, let's define your price schema to ensure your LLM reads the extracted content correctly and maps each product field into a predictable format for downstream processing.\n\nNow that ZenRows has retrieved the data from the website, [Pydantic AI](https://pydantic.dev/docs/validation/latest/get-started/) turns the markdown into a validated price record. You define the schema first, then run the agent against it. For a price monitoring agent, Pydantic protects your pipeline from messy extraction output.\n\nYou will start by defining your schema, which acts as a contract between the extraction layer and the rest of your pipeline. In the code below, that is `PriceRecord`\n\n. After that, you build the agent against the schema. So if the field has the wrong type, Pydantic AI reprompts the model with the validation error and returns the right output.\n\n``` python\nfrom datetime import datetime, timezone\nfrom pydantic import BaseModel, Field\nfrom dotenv import load_dotenv\n\nload_dotenv()\n\nclass PriceRecord(BaseModel):\n name: str\n price: float\n currency: str\n marketplace: str\n availability: str\n scraped_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))\n\nfrom pydantic_ai import Agent\n\nprice_agent = Agent(\n \"anthropic:claude-sonnet-4-6\",\n output_type=PriceRecord,\n instructions=(\n \"Extract the product price record from the Markdown. \"\n \"price is a number with no currency symbol. \"\n \"currency is the ISO code, for example USD. \"\n \"marketplace is the site name, for example Amazon.\"\n ),\n)\n```\n\nNow, let's run it against our ZenRows output.\n\n``` python\nimport os\nimport requests\nfrom datetime import datetime, timezone\nfrom typing import Optional\n\nfrom dotenv import load_dotenv\nfrom pydantic import BaseModel, Field\nfrom pydantic_ai import Agent\n\nload_dotenv()\n\nZENROWS_API_KEY = os.environ[\"ZENROWS_API_KEY\"]\n\nPRODUCT_URL = \"https://www.scrapingcourse.com/ecommerce/\"\n\n# --- Step 1 retrieve with ZenRows ---\n\ndef fetch_markdown(url: str) -> Optional[str]:\n params = {\n \"url\": url,\n \"apikey\": ZENROWS_API_KEY,\n \"js_render\": \"true\",\n \"premium_proxy\": \"true\",\n \"proxy_country\": \"us\",\n \"response_type\": \"markdown\",\n\n }\n\n resp = requests.get(\n \"https://api.zenrows.com/v1/\",\n params=params,\n timeout=60,\n )\n\n if resp.status_code != 200:\n print(f\"retrieval blocked: {resp.status_code}\")\n return None\n\n return resp.text\n\n# --- Step 2 schema + agent ---\n\nclass PriceRecord(BaseModel):\n name: str\n price: float\n currency: str\n marketplace: str\n availability: str\n scraped_at: datetime = Field(\n default_factory=lambda: datetime.now(timezone.utc)\n )\n\nprice_agent = Agent(\n \"anthropic:claude-sonnet-4-6\",\n output_type=PriceRecord,\n instructions=(\n \"Extract the product price record from the Markdown. \"\n \"price is a number with no currency symbol. \"\n \"currency is the ISO code, for example USD. \"\n \"marketplace is the site name, for example ScrapingCourse.\"\n ),\n)\n\n# --- Step 3 run pipeline ---\n\nmarkdown = fetch_markdown(PRODUCT_URL)\n\nif markdown is None:\n raise Exception(\"No markdown returned from ZenRows\")\n\nprint(f\"markdown length: {len(markdown)}\")\n\nresult = price_agent.run_sync(markdown)\n\nrecord = result.output\n\nprint(\"\\n--- extracted record ---\")\nprint(record.model_dump())\n\nprint(\"\\n--- type checks ---\")\nprint(\n \"price is float? \",\n isinstance(record.price, float),\n \"->\",\n repr(record.price)\n)\n\nprint(\n \"name is str? \",\n isinstance(record.name, str)\n)\n\nprint(\n \"scraped_at set? \",\n record.scraped_at.isoformat()\n)\n```\n\nFrom the output below, you can see that the ZenRows returned page content instead of a block or challenge response, hence the markdown length of 6750. The LLM also converted the messy markdown into your expected fields and shows the schema validation.\n\n```\nvenv) benny@Mac price_monitoring % python zenrows_pydantic.py\nmarkdown length: 6750\n\n--- extracted record ---\n{'name': 'Abominable Hoodie', 'price': 69.0, 'currency': 'USD', 'marketplace': 'ScrapingCourse', 'availability': 'In Stock', 'scraped_at': datetime.datetime(2026, 6, 22, 23, 5, 8, 154880, tzinfo=datetime.timezone.utc)}\n\n--- type checks ---\nprice is float? True -> 69.0\nname is str? True\nscraped_at set? 2026-06-22T23:05:08.154880+00:00\n(venv) benny@Mac price_monitoring %\n```\n\nThe script below uses the same retrieve-then-extract framework as in the previous two sections. However, now it is a loop over multiple URLs, collecting a validated PriceRecord from each site.\n\nTo demonstrate multi-marketplace price monitoring and ZenRows' capability, this tutorial runs the workflow against Amazon and Walmart. Amazon and Walmart are among the most aggressively protected e-commerce sites on the web, making them a meaningful test of retrieval reliability. Successfully extracting product data from these targets demonstrates ZenRows' ability to handle real-world protected websites, not just scraper-friendly demo pages.\n\n``` python\nimport os\nimport requests\nfrom datetime import datetime, timezone\nfrom typing import Optional\n\nfrom dotenv import load_dotenv\nfrom pydantic import BaseModel, Field\nfrom pydantic_ai import Agent, UnexpectedModelBehavior\n\nload_dotenv()\n\nZENROWS_API_KEY = os.getenv(\"ZENROWS_API_KEY\", \"\")\n\nMIN_MARKDOWN_LENGTH = 200 # a real product page dwarfs any block or error blob\n\n# The same product across three marketplaces, one product page per URL,\n# all on US domains so the prices share a currency.\nTARGETS = [\n (\"Amazon\", \"https://www.amazon.com/Apple-iPhone-Version-256GB-Titanium/dp/B0DHJDPYYR\"),\n (\"Walmart\", \"https://www.walmart.com/ip/Restored-Apple-iPhone-16-Pro-Carrier-Unlocked-256GB-Black-Titanium-Refurbished/13323059232\"),\n]\n\n# --- Step 1: retrieve with ZenRows ---\n\ndef fetch_markdown(url: str) -> Optional[str]:\n params = {\n \"url\": url,\n \"apikey\": ZENROWS_API_KEY,\n \"js_render\": \"true\",\n \"premium_proxy\": \"true\",\n \"proxy_country\": \"us\",\n \"response_type\": \"markdown\",\n\n }\n resp = requests.get(\n \"https://api.zenrows.com/v1/\",\n params=params,\n timeout=60,\n )\n\n # ZenRows returns a non-200 when it cannot deliver the page.\n if resp.status_code != 200:\n return None\n\n markdown = resp.text.strip()\n\n # Too short to be a product page. A block or error blob lands here.\n if len(markdown) < MIN_MARKDOWN_LENGTH:\n return None\n\n # An error came back as JSON instead of Markdown.\n if markdown.startswith(\"{\") and '\"code\"' in markdown:\n return None\n\n return markdown\n\n# --- Step 2: schema + agent ---\n\nclass PriceRecord(BaseModel):\n name: str\n price: float\n currency: str\n marketplace: str\n availability: str\n scraped_at: datetime = Field(\n default_factory=lambda: datetime.now(timezone.utc)\n )\n\nprice_agent = Agent(\n \"anthropic:claude-sonnet-4-6\",\n output_type=PriceRecord,\n instructions=(\n \"Extract the product price record from the Markdown. \"\n \"price is a number with no currency symbol. \"\n \"currency is the ISO code, for example USD.\"\n ),\n)\n\n# --- Step 3: run the pipeline across every marketplace ---\n\ndef run_monitor() -> tuple[list[PriceRecord], int]:\n results: list[PriceRecord] = []\n failed = 0\n\n for marketplace, url in TARGETS:\n print(f\"{marketplace}: {url}\")\n\n # Retrieval failure: ZenRows returns a blocked response. Log and skip.\n markdown = fetch_markdown(url)\n if markdown is None:\n print(\" retrieval blocked, skipping\")\n failed += 1\n continue\n\n # Extraction failure: the model never returns a valid record and\n # Pydantic AI exhausts its retries. Log and skip.\n try:\n result = price_agent.run_sync(\n f\"Marketplace: {marketplace}\\n\\n{markdown}\"\n )\n except UnexpectedModelBehavior as e:\n print(f\" extraction failed after retries, skipping ({e})\")\n failed += 1\n continue\n\n record = result.output\n record.marketplace = marketplace # use the known label, not the LLM's casing\n results.append(record)\n print(f\" ok -> {record.name} {record.price} {record.currency}\")\n\n return results, failed\n\nif __name__ == \"__main__\":\n records, failed_count = run_monitor()\n\n print(f\"\\nvalidated records: {len(records)}\")\n print(f\"failed URLs: {failed_count}\")\n\n for record in records:\n print(record.model_dump())\n\n # The decision: cheapest marketplace first.\n print()\n for record in sorted(records, key=lambda r: r.price):\n print(f\"{record.marketplace:<10} {record.price} {record.currency}\")\n```\n\nFrom our output below, you will notice that both sites ran and the product's information was retrieved from Amazon and Walmart. This tells us how reliable ZenRows is for retrieval and scraping.\n\n```\nvenv) benny@Mac price_monitoring % python zenrows_multi_copy.py\n/Users/benny/Desktop/price_monitoring/venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020\n warnings.warn(\nAmazon: https://www.amazon.com/Apple-iPhone-Version-256GB-Titanium/dp/B0DHJDPYYR\n ok -> Apple iPhone 16 Pro, 256GB, Black Titanium - Unlocked (Renewed) 745.93 USD\nWalmart: https://www.walmart.com/ip/Restored-Apple-iPhone-16-Pro-Carrier-Unlocked-256GB-Black-Titanium-Refurbished/13323059232\n ok -> Restored Apple iPhone 16 Pro - Carrier Unlocked - 256GB Black Titanium (Refurbished) 697.87 USD\n\nvalidated records: 2\nfailed URLs: 0\n{'name': 'Apple iPhone 16 Pro, 256GB, Black Titanium - Unlocked (Renewed)', 'price': 745.93, 'currency': 'USD', 'marketplace': 'Amazon', 'availability': 'In Stock', 'scraped_at': datetime.datetime(2026, 6, 22, 23, 27, 51, 184428, tzinfo=datetime.timezone.utc)}\n{'name': 'Restored Apple iPhone 16 Pro - Carrier Unlocked - 256GB Black Titanium (Refurbished)', 'price': 697.87, 'currency': 'USD', 'marketplace': 'Walmart', 'availability': 'In Stock', 'scraped_at': datetime.datetime(2026, 6, 22, 23, 28, 6, 863848, tzinfo=datetime.timezone.utc)}\n\nWalmart 697.87 USD\nAmazon 745.93 USD\n(venv) benny@Mac price_monitoring %\n```\n\nYou can find the code snippets in this article in this [GitHub repository.](https://github.com/Bennykillua/price_monitoring_agent_with_pydanticaai_and_zenrows)\n\nWith ZenRows handling retrieval and Pydantic AI validating each extraction, this pipeline remains reliable even when working with protected targets like Amazon and Walmart, which traditional scrapers often fail to scrape or return incomplete data for. ZenRows' strength lies in its ability to reliably access protected sites, with a [99.93% success rate](https://www.zenrows.com/) against sites that block everything else. The result is a list of typed product records across three marketplaces in a consistent shape, ready to insert into a database or feed an alerting layer without any official integration.\n\n[Try ZenRows for free](https://app.zenrows.com/register?prod=universal_scraper) to build more reliable price-monitoring workflows!", "url": "https://wpnews.pro/news/how-to-build-a-price-monitoring-agent-with-pydantic-ai-and-zenrows", "canonical_source": "https://dev.to/zenrows/how-to-build-a-price-monitoring-agent-with-pydantic-ai-and-zenrows-3183", "published_at": "2026-07-22 20:46:40+00:00", "updated_at": "2026-07-22 21:02:14.136364+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "large-language-models", "artificial-intelligence"], "entities": ["ZenRows", "Pydantic AI", "Amazon", "Walmart"], "alternates": {"html": "https://wpnews.pro/news/how-to-build-a-price-monitoring-agent-with-pydantic-ai-and-zenrows", "markdown": "https://wpnews.pro/news/how-to-build-a-price-monitoring-agent-with-pydantic-ai-and-zenrows.md", "text": "https://wpnews.pro/news/how-to-build-a-price-monitoring-agent-with-pydantic-ai-and-zenrows.txt", "jsonld": "https://wpnews.pro/news/how-to-build-a-price-monitoring-agent-with-pydantic-ai-and-zenrows.jsonld"}}