How Scraping AI Extracts Structured Data from Any Webpage Without CSS Selectors

Pig Data has released a Python SDK called scraping-ai that extracts structured data from webpages without relying on fragile CSS selectors. The tool uses a pipeline of dynamic rendering, Markdown distillation, and LLM-based semantic schema matching to output validated JSON, making it resilient to frontend changes. The SDK integrates with AI agent frameworks like LangChain and LlamaIndex.

Stop maintaining fragile CSS selectors. Turn any webpage into validated JSON with the Python SDK. NOTE TL;DR The problem:Traditional scrapers break when a site changes its CSS classes, such as .price becoming . 3xP9z . That means more maintenance and broken data pipelines.The solution:The scraping-ai Python SDK uses semantic extraction instead of relying on fixed DOM selectors: URL → Dynamic Render → Markdown Distillation → LLM Schema Matching → Validated JSON .AI agents & RAG:JSON Schema output makes it easy to use Scraping AI as a web tool with LangChain or LlamaIndex.Python SDK: pip install scraping-ai Try it free:Get200 free tokenswith no credit card required: https://pig-data.jp/service/scraping-ai/ Most web scrapers built with BeautifulSoup, Cheerio, or Selenium depend on one basic assumption: The structure of the website won't change. For example: The fragile approach BeautifulSoup soup = BeautifulSoup html content, "html.parser" title = soup.select one ".product-container .title-wrapper h1.title" .text price = soup.select one ".price-box span.current-price" .text This works until the site changes its frontend. Maybe the company moves to Tailwind CSS. Maybe it replaces its component library. Maybe a developer renames a class during a redesign. Your selectors stop matching. Sometimes you get an obvious error. Other times, you just get empty data. Either way, your data pipeline needs fixing. Instead of telling your scraper how to navigate the DOM , you describe what data you want : { "title": "string", "price": "number", "in stock": "boolean" } The extraction engine finds the relevant information on the page and maps it to your schema. You don't need to know which CSS class contains the price. You just need to define what a price is. The extraction process has four main steps: ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ 1. Smart Render │ ─────▶│ 2. Markdown │ ─────▶│ 3. LLM Semantic │ │ Auto JS Exec │ │ Distillation │ │ Schema Match │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ ▼ ┌─────────────────┐ │ 4. Validated │ │ JSON Output │ └─────────────────┘ Simple HTML pages can be rendered directly. For JavaScript-heavy sites, including React, Next.js, and Vue applications, Scraping AI can use headless Chromium to execute client-side JavaScript and render the page before extraction. A raw webpage can contain a lot of content that isn't useful for extraction: inline SVGs, tracking pixels, CSS, scripts, and other presentation-related markup. Scraping AI converts the page into a cleaner Markdown representation while keeping the text, structure, and context needed for extraction. The engine uses models such as GPT-4o and Gemini to understand the page and match its content to your schema. The location of the data doesn't have to be consistent. A price could appear in a product card, a table cell, or a header. The model looks at the meaning of the content rather than relying on a specific CSS selector. The extracted data is validated against a JSON Schema before it's returned to your application. That gives your Python code structured, typed output instead of another block of raw HTML to parse. Install the SDK: pip install scraping-ai Here's a basic extraction with error handling: python from scraping ai import ScrapingAIClient client = ScrapingAIClient api key="YOUR API KEY" try: data = client.extract url="https://example.com/products/headphones", schema={ "title": "string", "price": "number", "in stock": "boolean", "rating": "number" } print data.results except Exception as e: print f"Extraction error handled gracefully: {e}" For higher-throughput workloads, you can use the async client: python import asyncio from scraping ai import AsyncScrapingAIClient async def main : async with AsyncScrapingAIClient api key="YOUR API KEY" as client: data = await client.extract url="https://example.com/products/headphones", schema={ "title": "string", "price": "number" } print data.results asyncio.run main The result is structured JSON: { "results": { "data": { "title": "Wireless Noise Cancelling Headphones", "price": 89.99, "in stock": true, "rating": 4.7 }, "target url": "https://example.com/products/headphones" } } If you're building an LLM agent or RAG pipeline, you can expose Scraping AI as a web extraction tool. For example: python from langchain.tools import tool from scraping ai import ScrapingAIClient client = ScrapingAIClient api key="YOUR API KEY" @tool def web data extractor url: str, required schema description: str - dict: """Fetch clean, structured JSON from a URL.""" result = client.extract url=url, schema={ "extracted info": "string", "summary": "string" } return result.results The agent gets structured data instead of having to reason over a page full of HTML, styles, scripts, and other noise. Scraping AI isn't a replacement for every scraping tool. A few things to keep in mind: The point is not to replace traditional scraping everywhere. It's to reduce the maintenance work that comes with extracting structured data from websites that keep changing. Get 200 free tokens with no credit card required. Install the SDK: pip install scraping-ai API Documentation: https://pig-data.jp/service/scraping-ai/docs/ https://pig-data.jp/service/scraping-ai/docs/ Scraping AI https://pig-data.jp/service/scraping-ai/ https://pig-data.jp/service/scraping-ai/ is developed and operated by indigodata Inc. , an AI venture subsidiary of SMS DataTech Co., Ltd. in Tokyo, Japan. The product is based on PigData's experience with 500+ enterprise data extraction projects and provides a self-serve LLM extraction API for developers. I’m a software developer at Indigodata, the team behind Scraping AI. I'm sharing the architecture behind how we built this because dealing with broken CSS selectors is a pain we've all faced. Note: This article was co-authored with my colleague Harsh Tripathi and originally published on Medium . I’m sharing our team's work here with the Dev.to community