Your price-monitoring script probably logs thousands of prices a day into a dashboard few people open after the first week. Then a competitor cuts prices across a whole category by 20%, and your team might not notice until a week later. What's usually missing is a step that decides whether a change is noise, a promotion, or a broken page. Shoppers also ask ChatGPT and Google AI Mode where to buy, and most scripts don't check the prices in those answers.
This guide builds that decision step as an AI agent in Python, and adds a check of the prices in AI answers. The Decodo Web Data API fetches the competitor pages, Claude decides, and the demo reaches its first decision within 10 minutes of cloning the guide's repo.
The Decodo Web Data API gives you these endpoints with a single API key. The documentation lists the parameters for each one:
| Endpoint | What it does | Role in this build |
|---|---|---|
/map |
Lists URLs from a page and its sitemap in one call | New-SKU discovery |
/crawl |
Scrapes a section of a site in one job, as HTML or Markdown | Price tracking, daily |
/scrape |
Fetches one page, or asks AI search tools such as ChatGPT and Google AI Mode | New-SKU pages, price rechecks, and prices in AI answers |
/search |
Returns live search results | Optional promotion checks |
Proxy rotation, JavaScript rendering, and anti-bot handling are built into each endpoint, so your code focuses on prices, not proxies or headless browsers. Every endpoint runs on our proxy network of 125M+ IPs, with a 99.99% success rate. The Web Data API was built for AI agents that need live web data.
The Decodo dashboard Playground runs requests without writing code. Here it maps ScrapeMe's shop page sorted by newest:
Each stage is handled by Decodo, Python, Claude, or you:
Python extracts or verifies every price, computes the numbers, and enforces your price floor, the lowest price you allow. Its simple rules also log routine events, such as a price change under 5%, so the agent sees only events that need judgment. The agent works only with these numbers, which Python has already checked.
The stack is Python 3.12+, Decodo's Python SDK, SQLite, Pydantic, selectolax, and the Anthropic SDK, with no agent framework. The target is ScrapeMe, a public WooCommerce test store that anyone can use to reproduce the demo.
To set up the project:
The repo's commands do the work, and each step below explains one:
| Command | What it does | Explained in |
|---|---|---|
uv run price-agent discover |
Finds new products with /map |
Step 1 |
uv run price-agent track |
Crawls your batches with /crawl , finds what changed, and decides |
Steps 2 to 4 |
uv run price-agent eval |
Tests the agent's decisions | Step 4 |
uv run price-agent ai-shelf |
Checks the prices in AI answers | Step 5 |
The code in Steps 1 and 2 shows the Decodo calls behind these commands. You can run it on its own to try an endpoint, but the commands already include it.
New SKUs from any section of a store usually appear first on its shop page sorted by newest. Mapping that page finds them, even outside the sections your daily crawl covers.
uv run price-agent discover maps the first page of the shop, both in its default order and sorted by newest. For each URL that isn't in the local catalog yet, it fetches the page with /scrape and creates a new_product event. The first run only stores the catalog, so new products appear from the second run.
Behind the command, one synchronous /map call returns the product URLs, and its path filters keep only product pages. To try it on its own, save this code as try_api.py in the repo folder and run uv run --env-file .env python try_api.py:
import os
from decodo import DecodoClient, DecodoConfig, WebScrapingApiConfig
config = DecodoConfig(
web_scraping_api=WebScrapingApiConfig(api_key=os.environ["DECODO_API_KEY"])
)
api = DecodoClient(config).web_scraping_api
result = api.map({
"url": "https://scrapeme.live/shop/?orderby=date", # newest products first
"select_paths": ["^/shop/[^/]+/?$"], # product pages only
"exclude_paths": ["^/shop/page/", "^/cart", "^/my-account", "^/product-tag/"],
"geo": "United Kingdom", "locale": "en-GB",
})["results"][0]
urls = [link["url"] for link in result["content"]["links"]]
print(len(urls), urls[:5])
ScrapeMe's shop page sorted by latest lists its newest products first:
ScrapeMe's catalog is static. To simulate 2 new arrivals, this run first removes Stakataka and Naganadel, outlined above, from the local catalog:
$ uv run price-agent discover
/map: 32 product URLs on the 2 listing pages, 2 new
/scrape: 2 of 2 new URLs had product data
2 change events: {'new_product': 2}
[rule: log_only] e1 Stakataka: New competitor product with no SKU of ours to compare yet.
[rule: log_only] e2 Naganadel: New competitor product with no SKU of ours to compare yet.
No event needs judgment. No LLM call.
For your own target, set newest_listing in config.toml to the store's page sorted by newest.
uv run price-agent track crawls the catalog in batches by URL path, such as products whose names start with A or B. It runs one /crawl job per batch and saves every page before the next batch starts. /crawl takes a seed URL plus crawl filters and returns 202 Accepted with a crawl_id.
To try the crawl on its own, add this loop to the end of try_api.py and run the file again. It reuses the api client from Step 1:
import time
for batch in ["^/shop/[A-B][^/]*/?$", "^/shop/C[^/]*/?$"]:
crawl_id = api.start_crawl({
"url": "https://scrapeme.live/shop/",
"sitemap": "only", # take product URLs from the sitemap
"select_paths": [batch],
"exclude_paths": ["^/shop/page/", "^/cart", "^/my-account", "^/product-tag/"],
"limit": 100, # your page limit for this job
"output": "html", # html keeps the JSON-LD that Step 3 parses
"geo": "United Kingdom", "locale": "en-GB",
})["crawl_id"]
while api.get_crawl_status(crawl_id)["status"] == "pending":
time.sleep(10)
pages = list(api.get_crawl_results(crawl_id)) # follows every `next` link
print(batch, len(pages), "pages, first:", pages[0]["url"])
track also extracts each page's offers and saves them to SQLite right away. These settings matter:
You pay only for pages fetched successfully. A page without product data, such as a login page, counts as a missing page, not a removed product. When missing pages pass a threshold you set in config.toml, Python alerts a human.
A coverage check compares the URLs /map found with the pages fetched. A freshness check flags a competitor page from our_prices.csv if no crawl has fetched it within a time limit you set. After the first run sets the baseline, the agent gets nothing on a day with no changes:
$ uv run price-agent track
/crawl 667c7e31-dafb-4daa-8bc7-bcc80250d768: done, 72 product pages saved, 0 failed
/crawl 01c47591-a081-4fae-a215-4c019b28cec9: done, 59 product pages saved, 0 failed
total: 131 product pages saved, 0 faulted, 0 without product data
coverage: 0 of 131 expected catalog URLs not fetched
freshness: 3 of 3 tracked pages seen, oldest 0h ago
0 change events: {}
Nothing changed. No LLM call.
The repo's GitHub Actions workflow can run /crawl daily and commit prices.db back, because each new runner starts with no price history.
For SKUs in volatile eCommerce categories, add an hourly cron job that crawls only their batches, using the --shards option.
A daily crawl of the 2 demo batches, 131 products, comes to about 3.9K pages a month.
Price extraction reads Schema.org Product JSON-LD first, including product variants, with a CSS selector per store as the fallback. Without any changes, the same code read 200 priced size variants from 19 pages that /crawl fetched from a real Shopify store.
The diff stores a missing or zero price as unknown, not as a drop to zero.
Because ScrapeMe's prices are static, the repo includes a test fixture. It replays 5 unchanged days from a real /crawl of ScrapeMe, then a 6th day with deliberate test changes. They cover a 3% drop, a category promotion, a stock-out, a misplaced decimal, a missing price, and a new product:
$ uv run price-agent demo
9 change events: {'price_changed': 7, 'stock_changed': 1, 'new_product': 1}
Of the 9 events, 7 are real market changes, and 2 are deliberate store-page errors for the agent to recognize. With just the Decodo key, uv run price-agent demo --no-agent prints the events as JSON.
The diff links each competitor URL to your SKU, price, and price floor from our_prices.csv. Its core column marks your key value items (KVIs), the products shoppers use to judge your prices. Here's the Charizard event the agent receives, showing only the fields it uses to decide:
{
"id": "e2", "kind": "price_changed", "name": "Charizard",
"offer_key": "https://scrapeme.live/shop/Charizard#8840",
"category": "Fire", "our_sku": "OUR-006", "core_sku": true, "currency": "GBP",
"old_price": "156.00", "new_price": "127.92", "delta_pct": -18.0,
"our_price": "159.00", "gap_pct": -19.5, "below_floor": true,
"history": ["156.00", "156.00", "156.00", "156.00", "156.00", "127.92"],
"flags": []
}
A negative gap_pct means the competitor is cheaper. below_floor says whether matching would go below your price floor. flags marks a suspected data problem, such as a store page that briefly shows a wrong price. The agent rechecks a flagged event against the live page first.
The agent gets the events that need judgment and a plain-English policy, policy.md, as its system prompt. One rule tells it to match a competitor's price on a KVI when that competitor is more than 5% cheaper. The agent acts only through 4 tools with strict: true schemas, defined in src/price_agent/agent.py:
def _tool(name: str, description: str, properties: dict) -> dict:
return {"name": name, "description": description, "strict": True,
"input_schema": {"type": "object", "properties": properties,
"required": list(properties),
"additionalProperties": False}}
IDS = {"type": "array", "items": {"type": "string"}}
TOOLS = [
_tool("log_only",
"Record events as noise or as a data issue. No one is notified.",
{"event_ids": IDS, "reason": {"type": "string"}}),
_tool("alert_human",
"Notify the pricing team about events that need a person's attention.",
{"event_ids": IDS,
"urgency": {"type": "string", "enum": ["low", "normal", "high"]},
"reason": {"type": "string"},
"suggested_response": {"type": "string"}}),
_tool("propose_strategy_change",
"Write a pricing proposal for a human to approve. Changes nothing live.",
{"event_ids": IDS, "our_sku": {"type": "string"},
"change_type": {"type": "string",
"enum": ["match_price", "hold_price", "review_rule"]},
"proposed_price": {"type": "string", "description": "Decimal, e.g. 37.00"},
"reason": {"type": "string"}}),
_tool("reverify",
"Re-fetch one event's page live and re-run extraction before you act on it.",
{"event_id": {"type": "string"}, "reason": {"type": "string"}}),
]
Python adds the checks a schema can't express. For example, a proposal below your price floor goes back to the agent as a tool error. reverify re-fetches the page through /scrape with the crawl's geo and locale and returns the live price.
On the demo's test day, rules logged 2 events, and the agent detected both store-page errors and decided the rest:
[rule: log_only] e6 Pikachu: Moved -3.0%, under the 5% noise threshold.
[rule: log_only] e9 Mewtwo: New competitor product with no SKU of ours to compare yet.
[reverify] e1 Bulbasaur: snapshot 630.00, live 63.00
[reverify] e5 Mew: snapshot None, live 136.00
[propose_strategy_change] e7
Core SKU: competitor Squirtle went out of stock while cheaper than us
(gap_pct -6.5), so hold our 139.00 and cancel any planned match.
[alert_human] e2, e3, e4, e8
Fire category promotion pattern: 4 SKUs dropped 10% or more in one run
(Charizard -18.0%, Charmander -18.0%, Growlithe -17.0%, Vulpix -17.0%).
Charizard (OUR-006, core) is 19.5% below us and below floor, and
Charmander (OUR-004) is also below floor (gap -19.7%), so matching would
breach the floor.
[log_only] e1, e5
Data issues, not market signals: live pages do not reproduce the crawl
changes. e1 live price is 63.00 vs the 630.00 snapshot (extreme_jump), and
e5 live price is 136.00 vs the null snapshot (price_missing_or_zero).
Test the decisions. uv run price-agent eval sends 17 events through the same rules and agent, and compares each decision with an expected decision written by hand.
The 17 events are the 9 above, the 2 new products from Step 1, and 6 wrong prices in AI answers. Of these, 4 came from live runs, and 2 are edge cases. Rules decide 6, and the agent decides the other 11:
model run match reverify cost $ misses
claude-haiku-5-5 1 17/17 True 0.0038
claude-haiku-5-5 2 17/17 True 0.0036
claude-haiku-5-5 3 17/17 True 0.0034
claude-opus-5-5 1 17/17 True 0.1396
claude-opus-5-5 2 17/17 True 0.1221
claude-opus-5-5 3 17/17 True 0.1262
In the table, reverify means the agent rechecked every flagged event. With the same 17 events and 3 runs, Opus 5.5 made the same decisions at about 36 times the cost. Haiku is the default.
The 17 events make a small, fast regression test. Re-run eval after each policy change.
Keep people in control. The agent proposes, and a human approves, so your team stays in control of every live price. Teams usually give the agent more autonomy in stages, starting with small, reversible changes on low-volume SKUs above the price floor. Every alert and proposal includes its evidence for a reviewer to check.
In Slack, the promotion alert lists each SKU's price change and price history:
This step uses a real product that appears in AI answers, Sony's WH-1000XM6 headphones, and treats Sony's store as your store.
Our /scrape templates are ready-made integrations for specific sites, such as the ChatGPT scraper and the Amazon scraper. Here they ask ChatGPT, Google AI Mode, and Gemini where to buy the headphones, twice each. Claude Haiku then extracts each retailer's price from the answers.
Python keeps only prices that appear in the answer with a currency symbol. It reads each store's live price from product-page JSON-LD or our Amazon and Walmart templates. Each AI price then appears next to the store's live price, and a dash marks a store the answer gave no price for:
$ uv run price-agent ai-shelf
Sony Amazon Best Buy Walmart
live 459.99 458.00 458.00 458.00
google_ai_mode 1 379.99 378.00 398.00 378.00
google_ai_mode 2 - - 458.00 458.00
chatgpt 1 459.99 - 398.00 359.00
chatgpt 2 - - 378.00 458.00
gemini 1 459.99 - 458.00 -
gemini 2 378.00 - 458.00 -
8 of 15 quotes for these 4 stores below the live price, 0 above
1 change events: {'ai_misquote': 1}
[alert_human] a1
ChatGPT quotes Best Buy's Sony WH-1000XM6 at $378.00, but the live page
shows $458, a 17.5% gap. The quote is also below our $459.99. Per policy,
the quote is not a buyable price, so we should not match it.
Answers change each time you ask. The agent creates an ai_misquote event only when every answer from one AI search tool shows a wrong price for a store. Here, only ChatGPT's 2 answers both priced Best Buy too low.
Across 7 checks like this one, ChatGPT showed 31 of 44 prices for these stores more than 1% below the store's live price. Google AI Mode did so for 11 of 25. No AI price was more than 1% higher than the live price.
ChatGPT's own app, logged out, showed the same pattern. Its Best Buy price, $398.00, matches chatgpt 1 above and is $60 below the live price.
The policy matches only live prices, because those are the ones a shopper can buy. When an AI answer shows a wrong price for a competitor, the agent sends a low-urgency alert. It suggests checking the live page before you accept a price-match request based on that AI price.
A wrong price of 5% or more for one of your own KVIs is high urgency. That alert tells the team to check the product data your store sends to Google Merchant Center, which Google AI Mode and Gemini listings use.
Check 4 things before you use the agent on a real store:
You can extend the pipeline in 3 ways:
Price monitoring is most useful when it ends in a decision. Let the Web Data API collect the pages, keep extraction and the diff in Python, and give Claude only the decisions that need judgment. As store pages and AI answers change, check the coverage alerts and re-run eval after each policy change.
Start on the Web Data API free plan, then clone the repo and run uv run price-agent demo to watch the agent decide. Next, set one section of a competitor's store in config.toml, your SKUs and price floors in our_prices.csv, and your rules in policy.md.