Stop maintaining fragile CSS selectors. Turn any webpage into validated JSON with the Python SDK.
[!NOTE]
TL;DR
The problem:Traditional scrapers break when a site changes its CSS classes, such as.price
becoming._3xP9z
. That means more maintenance and broken data pipelines.The solution:Thescraping-ai
Python SDK uses semantic extraction instead of relying on fixed DOM selectors:URL → Dynamic Render → Markdown Distillation → LLM Schema Matching → Validated JSON
.AI agents & RAG:JSON Schema output makes it easy to use Scraping AI as a web tool with LangChain or LlamaIndex.Python SDK:pip install scraping-ai
Try it free:Get200 free tokenswith no credit card required:[https://pig-data.jp/service/scraping-ai/]
Most web scrapers built with BeautifulSoup, Cheerio, or Selenium depend on one basic assumption:
The structure of the website won't change.
For example:
soup = BeautifulSoup(html_content, "html.parser")
title = soup.select_one(
".product-container > .title-wrapper > h1.title"
).text
price = soup.select_one(
".price-box span.current-price"
).text
This works until the site changes its frontend.
Maybe the company moves to Tailwind CSS. Maybe it replaces its component library. Maybe a developer renames a class during a redesign.
Your selectors stop matching. Sometimes you get an obvious error. Other times, you just get empty data.
Either way, your data pipeline needs fixing.
Instead of telling your scraper how to navigate the DOM, you describe what data you want:
{
"title": "string",
"price": "number",
"in_stock": "boolean"
}
The extraction engine finds the relevant information on the page and maps it to your schema.
You don't need to know which CSS class contains the price. You just need to define what a price is.
The extraction process has four main steps:
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ 1. Smart Render │ ─────▶│ 2. Markdown │ ─────▶│ 3. LLM Semantic │
│ (Auto JS Exec) │ │ Distillation │ │ Schema Match │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│
▼
┌─────────────────┐
│ 4. Validated │
│ JSON Output │
└─────────────────┘
Simple HTML pages can be rendered directly.
For JavaScript-heavy sites, including React, Next.js, and Vue applications, Scraping AI can use headless Chromium to execute client-side JavaScript and render the page before extraction.
A raw webpage can contain a lot of content that isn't useful for extraction: inline SVGs, tracking pixels, CSS, scripts, and other presentation-related markup.
Scraping AI converts the page into a cleaner Markdown representation while keeping the text, structure, and context needed for extraction.
The engine uses models such as GPT-4o and Gemini to understand the page and match its content to your schema.
The location of the data doesn't have to be consistent.
A price could appear in a product card, a table cell, or a header. The model looks at the meaning of the content rather than relying on a specific CSS selector.
The extracted data is validated against a JSON Schema before it's returned to your application.
That gives your Python code structured, typed output instead of another block of raw HTML to parse.
Install the SDK:
pip install scraping-ai
Here's a basic extraction with error handling:
from scraping_ai import ScrapingAIClient
client = ScrapingAIClient(api_key="YOUR_API_KEY")
try:
data = client.extract(
url="https://example.com/products/headphones",
schema={
"title": "string",
"price": "number",
"in_stock": "boolean",
"rating": "number"
}
)
print(data.results)
except Exception as e:
print(f"Extraction error handled gracefully: {e}")
For higher-throughput workloads, you can use the async client:
import asyncio
from scraping_ai import AsyncScrapingAIClient
async def main():
async with AsyncScrapingAIClient(
api_key="YOUR_API_KEY"
) as client:
data = await client.extract(
url="https://example.com/products/headphones",
schema={
"title": "string",
"price": "number"
}
)
print(data.results)
asyncio.run(main())
The result is structured JSON:
{
"results": [
{
"data": {
"title": "Wireless Noise Cancelling Headphones",
"price": 89.99,
"in_stock": true,
"rating": 4.7
},
"target_url": "https://example.com/products/headphones"
}
]
}
If you're building an LLM agent or RAG pipeline, you can expose Scraping AI as a web extraction tool.
For example:
from langchain.tools import tool
from scraping_ai import ScrapingAIClient
client = ScrapingAIClient(api_key="YOUR_API_KEY")
@tool
def web_data_extractor(
url: str,
required_schema_description: str
) -> dict:
"""Fetch clean, structured JSON from a URL."""
result = client.extract(
url=url,
schema={
"extracted_info": "string",
"summary": "string"
}
)
return result.results
The agent gets structured data instead of having to reason over a page full of HTML, styles, scripts, and other noise.
Scraping AI isn't a replacement for every scraping tool.
A few things to keep in mind:
The point is not to replace traditional scraping everywhere. It's to reduce the maintenance work that comes with extracting structured data from websites that keep changing.
Get 200 free tokens with no credit card required.
Install the SDK:
pip install scraping-ai
API Documentation: https://pig-data.jp/service/scraping-ai/docs/
Scraping AI (https://pig-data.jp/service/scraping-ai/) is developed and operated by indigodata Inc., an AI venture subsidiary of SMS DataTech Co., Ltd. in Tokyo, Japan.
The product is based on PigData's experience with 500+ enterprise data extraction projects and provides a self-serve LLM extraction API for developers.
I’m a software developer at Indigodata, the team behind Scraping AI. I'm sharing the architecture behind how we built this because dealing with broken CSS selectors is a pain we've all faced.
Note: This article was co-authored with my colleague Harsh Tripathi and originally published on [Medium]. I’m sharing our team's work here with the Dev.to community!