`\n\n, omit the SKU entirely, or use inline styles. A few pages even dumped the price into a `data-*`\n\nattribute inside a script tag.\n\nParsing this with BeautifulSoup and CSS selectors worked on 80% of the pages, but that last 20% caused silent failures. I spent days writing custom parsers that became unmaintainable.\n\nI tried patterns like `/(Price:)\\s*<[^>]+>([^<]+)<\\/b>/i`\n\n. It worked on one page but broke on another where the `
`\n\nwas nested differently. Regex is brittle for HTML—we all know this, but sometimes we pretend we don't.\n\nI wrote a set of rules: “if `.name`\n\nexists, use that; else try `[itemprop=\"name\"]`\n\n; else fallback to first ``\n\n.” Every new page meant new rules. The rule count exploded, and I still missed edge cases.\n\nI fed entire HTML blocks to GPT‑4 with a prompt like “extract name, price, SKU, description, stock status.” It worked beautifully—but it cost $0.03 per product. For 10,000 products that’s $300. And latency was 2–3 seconds per call. Not feasible for a one‑time migration.\n\nI used a smaller, cheaper model (like Llama 3.1 8B via Ollama, or a service that wraps similar models) and asked it to output JSON according to a predefined schema. The trick was to *show* it the schema and only ask for the fields I needed, with clear instructions on how to handle missing data.\n\nHere’s the core idea:\n\nI wrote a Python script using `requests`\n\nand `json`\n\n. For the LLM, I used Ollama with `llama3.1:8b`\n\nrunning locally, but you can swap in any API that supports chat completions.\n\n``` python\nimport requests\nimport json\nimport re\nfrom typing import Optional, Dict\n\nLLM_URL = \"http://localhost:11434/api/generate\" # Ollama endpoint\nMODEL = \"llama3.1:8b\"\n\ndef extract_product(html: str) -> Optional[Dict]:\n schema = {\n \"name\": \"string (required)\",\n \"price\": \"float (required, in USD)\",\n \"sku\": \"string (optional)\",\n \"description\": \"string (optional)\",\n \"in_stock\": \"boolean (optional)\"\n }\n prompt = f\"\"\"You are an HTML extraction expert. Given a product card's HTML, return a JSON object with these fields:\n{schema}\n\nReturn ONLY valid JSON. If a field is missing, use null.\n\nExamples:\nHTML:
WidgetPrice: $10.00
\nJSON: {{\"name\": \"Widget\", \"price\": 10.00, \"sku\": null, \"description\": null, \"in_stock\": null}}\n\nHTML: {html}\nJSON:\"\"\"\n response = requests.post(\n LLM_URL,\n json={\n \"model\": MODEL,\n \"prompt\": prompt,\n \"stream\": False,\n \"temperature\": 0.1\n }\n )\n text = response.json()[\"response\"]\n # Clean markdown code fences if present\n match = re.search(r'\\{.*\\}', text, re.DOTALL)\n if match:\n try:\n return json.loads(match.group())\n except json.JSONDecodeError:\n return None\n return None\n\n# Test with our HTML\nhtml_sample = \"\"\"\n
Widget Alpha\n
Price: $29.99\n
SKU: WID-001
\n
A handy gadget
with extra features
\n
In Stock\n