Building a Self-Healing Scraper: Let AI Fix Your Selectors When Sites Change

A developer has outlined a self-healing web scraping architecture that uses an LLM only as a fallback when hardcoded CSS selectors break, rather than for every request. The approach validates extracted data against a Pydantic schema to detect silent failures, then trims HTML by 60–80% and asks a model to recover the value and generate a replacement selector, which is verified and saved for subsequent runs. The design keeps the expensive model call on the exception path so stable sites never incur LLM costs.

Your scraper doesn't fail with a stack trace. It fails silently. A site ships a redesign, renames one CSS class, and your .css "span.price" quietly starts returning None . Nothing crashes. The pipeline keeps running, writing rows of empty prices into your database, and you find out a week later when someone downstream asks why the dashboard is flat. The hard part of scraping was never writing the first version. It's keeping it alive as the target keeps moving. In this tutorial we'll build a scraper that notices when its own selectors break, uses an LLM to find the data again and generate a fresh selector , then saves that selector so the next run is cheap again. The key idea, before we write any code: the LLM is not doing the scraping. It only runs when the normal path breaks. That one constraint is what keeps this practical instead of a way to set money on fire. Here's the whole control flow: request → try SAVED selectors │ ┌───────┴────────┐ works fails validation │ │ return trim HTML → ask LLM for value + new selector │ verify the new selector actually works │ save it → return The expensive branch — trimming HTML, calling a model — is the exception, not the rule. On a stable site it never fires. It only kicks in on the day a site changes, and after it heals once, every following request is back on the fast path. Start with something completely ordinary. httpx to fetch, parsel to parse, hardcoded selectors. python import httpx from parsel import Selector DEFAULT SELECTORS = { "title": "h1.product-title", "price": "span.price", } def extract html: str, selectors: dict - dict: tree = Selector html return { field: tree.css f"{sel}::text" .get for field, sel in selectors.items } Nothing new here. This is the scraper you already write. The interesting part is how we decide whether its output is any good. Most people check for breakage with if result is None . That catches a selector that matches nothing, but it misses the nastier case: a selector that matches the wrong element and returns plausible-looking garbage. A schema is a much stronger health check. Define what a valid result looks like with pydantic , and let validation be your detector: python from pydantic import BaseModel, ValidationError, field validator class Product BaseModel : title: str price: float @field validator "title" @classmethod def title not blank cls, v: str - str: if not v or not v.strip : raise ValueError "empty title" return v.strip def validate raw: dict - Product | None: clean = {} for key, value in raw.items : if value is None: return None if key == "price": value = "".join c for c in value if c.isdigit or c == "." clean key = value try: return Product clean except ValidationError: return None Now "is my scraper healthy?" has a precise answer: validate extract ... returns a Product , or it returns None . A price of "Add to cart" fails the float coercion. An empty title fails the validator. That None is our trigger. When validation fails, we hand the problem to a model. Two things matter here: cost control and trust. Cost control means never sending raw HTML. A product page is mostly <script , <style , and inline SVG — none of which the model needs, all of which you pay for by the token. Strip it first: php from bs4 import BeautifulSoup def trim html html: str - str: soup = BeautifulSoup html, "html.parser" for tag in soup "script", "style", "svg", "noscript", "head" : tag.decompose return str soup On a typical product page this cuts the payload by 60–80%. Trust means the prompt asks for two things — the value and a selector — so we can verify the answer instead of taking it on faith: python import json from openai import OpenAI client = OpenAI reads OPENAI API KEY HEAL PROMPT = """You are a web-scraping repair tool. Find one field in the HTML below and return a CSS selector that reaches it. Field: {field} What it is: {hint} Return ONLY JSON, no prose: {{"value": "<exact text you found ", "selector": "<CSS selector for the element "}} HTML: {html} """ def heal field field: str, hint: str, html: str - dict: resp = client.chat.completions.create model="gpt-4o-mini", cheap model is fine — see below temperature=0, response format={"type": "json object"}, messages= { "role": "user", "content": HEAL PROMPT.format field=field, hint=hint, html=html , } , return json.loads resp.choices 0 .message.content The model can absolutely hallucinate a selector that doesn't exist, or a value that isn't on the page. That's fine, because we never trust it directly. We verify. This is the section that makes the whole thing "self-healing" rather than "call an LLM every time." Before saving a healed selector, we run it against the real HTML and confirm it produces the value the model claimed. Only a selector that actually works gets persisted: python from parsel import Selector def verify html: str, selector: str, expected: str - bool: got = Selector html .css f"{selector}::text" .get return got is not None and expected.strip in got.strip And here's the loop that ties every piece together: python import json from pathlib import Path SELECTOR STORE = Path "selectors.json" FIELD HINTS = { "title": "the product's name or title", "price": "the product's price, digits only", } def load selectors - dict: if SELECTOR STORE.exists : return json.loads SELECTOR STORE.read text return dict DEFAULT SELECTORS def save selectors selectors: dict - None: SELECTOR STORE.write text json.dumps selectors, indent=2 def scrape html: str - Product: selectors = load selectors Fast path: the selectors we already have product = validate extract html, selectors if product is not None: return product Slow path: heal trimmed = trim html html new selectors = dict selectors values = {} for field in Product.model fields: result = heal field field, FIELD HINTS field , trimmed if verify html, result "selector" , result "value" : new selectors field = result "selector" keep only verified fixes values field = result "value" product = validate values if product is None: raise ValueError "healing failed: LLM output did not match schema" save selectors new selectors <-- next run is cheap again return product Trace one lifecycle: span.price misses, validation returns None , the healing branch fires, the model finds the price under its new markup, the selector verifies, and it's written to selectors.json . You pay for the LLM exactly once per break, not once per request. That's the whole trick. This pattern is genuinely useful, and it is not magic. Where it bites: Cost still scales with churn. Healing only on failure is cheap on a stable site and expensive on a site that reshuffles its DOM daily. If your target ships A/B tests that rotate markup constantly, you'll be in the healing branch a lot. Measure your break rate before you assume this is free. Validation is not optional. Everything here depends on the schema being strict enough to catch a wrong answer. A loose schema title: str with no other checks will happily accept the model's hallucination and persist a broken selector. Your validator is the only thing standing between "self-healing" and "self-corrupting." Spend time on it. It fixes layout, not access. Self-healing repairs selectors when the structure changes. It does nothing about anti-bot blocking, JavaScript that hasn't rendered yet, or rate limits. Those are separate problems and this pattern won't touch them. Sometimes you shouldn't. If the site is stable, plain selectors are simpler and free — don't add an LLM to a problem you don't have. And if there's an official API, that's almost always cheaper and more reliable than healing a scraper against a hostile frontend. Which LLM should I use? A cheap one. gpt-4o-mini , Claude Haiku, or any small model is fine, because your pydantic validation catches bad output regardless of which model produced it. Reach for a bigger model only if the small one keeps failing verification on genuinely complex pages. Does this work on JavaScript-heavy sites? Only after the page is rendered. Healing fixes selectors; it doesn't execute JavaScript. Put a headless browser Playwright in front to get rendered HTML, then feed that into the same loop. What about anti-bot protection? Different problem. Self-healing handles layout changes, not blocking. If you're getting challenged or banned, that's an access issue to solve before extraction ever runs. How much does one heal actually cost? A trimmed product page is usually a few thousand tokens. At small-model prices that's a fraction of a cent per field, and it happens once per break — not per request. The expensive scenario is a high break rate, not a high request count. Can I use XPath instead of CSS selectors? Yes. Same loop — ask the model for an XPath, swap .css for .xpath in extract and verify . Nothing else changes. A self-healing scraper isn't a smarter scraper — it's a scraper with a fallback that costs money, guarded by validation so it only spends that money when it has to, and a memory so it doesn't spend it twice for the same break. That combination — cheap by default, self-repairing on change, honest about its limits — is what turns a scraper from something you babysit into something that mostly runs itself. The full runnable code is in the repo below. Clone it, point it at a site you actually maintain, break a selector on purpose, and watch it heal.