{"slug": "building-a-self-healing-scraper-let-ai-fix-your-selectors-when-sites-change", "title": "Building a Self-Healing Scraper: Let AI Fix Your Selectors When Sites Change", "summary": "A developer has outlined a self-healing web scraping architecture that uses an LLM only as a fallback when hardcoded CSS selectors break, rather than for every request. The approach validates extracted data against a Pydantic schema to detect silent failures, then trims HTML by 60–80% and asks a model to recover the value and generate a replacement selector, which is verified and saved for subsequent runs. The design keeps the expensive model call on the exception path so stable sites never incur LLM costs.", "body_md": "Your scraper doesn't fail with a stack trace. It fails silently.\n\nA site ships a redesign, renames one CSS class, and your `.css(\"span.price\")` quietly starts returning `None`. Nothing crashes. The pipeline keeps running, writing rows of empty prices into your database, and you find out a week later when someone downstream asks why the dashboard is flat.\n\nThe hard part of scraping was never writing the first version. It's keeping it alive as the target keeps moving. In this tutorial we'll build a scraper that notices when its own selectors break, uses an LLM to find the data again **and generate a fresh selector**, then saves that selector so the next run is cheap again.\n\nThe key idea, before we write any code: the LLM is not doing the scraping. It only runs when the normal path breaks. That one constraint is what keeps this practical instead of a way to set money on fire.\n\nHere's the whole control flow:\n\n```\nrequest → try SAVED selectors\n │\n ┌───────┴────────┐\n works fails validation\n │ │\n return trim HTML → ask LLM for value + new selector\n │\n verify the new selector actually works\n │\n save it → return\n```\n\nThe expensive branch — trimming HTML, calling a model — is the exception, not the rule. On a stable site it never fires. It only kicks in on the day a site changes, and after it heals once, every following request is back on the fast path.\n\nStart with something completely ordinary. `httpx` to fetch, `parsel` to parse, hardcoded selectors.\n\n``` python\nimport httpx\nfrom parsel import Selector\n\nDEFAULT_SELECTORS = {\n \"title\": \"h1.product-title\",\n \"price\": \"span.price\",\n}\n\ndef extract(html: str, selectors: dict) -> dict:\n tree = Selector(html)\n return {\n field: tree.css(f\"{sel}::text\").get()\n for field, sel in selectors.items()\n }\n```\n\nNothing new here. This is the scraper you already write. The interesting part is how we decide whether its output is any good.\n\nMost people check for breakage with `if result is None`. That catches a selector that matches nothing, but it misses the nastier case: a selector that matches the *wrong* element and returns plausible-looking garbage.\n\nA schema is a much stronger health check. Define what a valid result looks like with `pydantic`, and let validation be your detector:\n\n``` python\nfrom pydantic import BaseModel, ValidationError, field_validator\n\nclass Product(BaseModel):\n title: str\n price: float\n\n @field_validator(\"title\")\n @classmethod\n def title_not_blank(cls, v: str) -> str:\n if not v or not v.strip():\n raise ValueError(\"empty title\")\n return v.strip()\n\ndef validate(raw: dict) -> Product | None:\n clean = {}\n for key, value in raw.items():\n if value is None:\n return None\n if key == \"price\":\n value = \"\".join(c for c in value if c.isdigit() or c == \".\")\n clean[key] = value\n try:\n return Product(**clean)\n except ValidationError:\n return None\n```\n\nNow \"is my scraper healthy?\" has a precise answer: `validate(extract(...))` returns a `Product`, or it returns `None`. A price of `\"Add to cart\"` fails the `float` coercion. An empty title fails the validator. That `None` is our trigger.\n\nWhen validation fails, we hand the problem to a model. Two things matter here: cost control and trust.\n\n**Cost control** means never sending raw HTML. A product page is mostly `