Scraping e-commerce and real estate sites like Amazon or Walmart the traditional way means writing selectors, managing proxies, and watching your parser break every time the site updates its layout. The maintenance is usually the real cost, not the initial scrape.
Spidra skips the selector step entirely, so all you have to do is define the fields you want, either by using a JSON Schema or writing a prompt in plain text. The AI then extracts them from the page regardless of how the underlying HTML is structured.
Here is what that looks like end to end, using a real Amazon product page as the target.
Define the fields you want #
There are two ways to tell Spidra what data to return. You can create a JSON Schema via the free JSON Schema Generator tool or via the Spidra dashboard.
Or write a prompt. For this guide, we use this schema below:
{
"type": "object",
"required": [
"amazon_choice", "availability_status", "badge", "brand",
"buybox_seller", "category_breadcrumb", "is_available",
"manufacturer", "parent_asin", "price_currency_code",
"price_currency_symbol", "product_description", "product_images",
"product_model_number", "product_name", "product_price",
"product_price_before_discount", "product_url", "rating_score",
"review_count", "root_bs_rank", "sku"
],
"properties": {
"amazon_choice": { "type": "boolean" },
"availability_status": { "type": "string" },
"badge": { "type": "string" },
"brand": { "type": "string" },
"buybox_seller": { "type": "string" },
"category_breadcrumb": { "type": "array", "items": { "type": "string" } },
"is_available": { "type": "boolean" },
"manufacturer": { "type": "string" },
"parent_asin": { "type": "string" },
"price_currency_code": { "type": "string" },
"price_currency_symbol": { "type": "string" },
"product_description": { "type": "string" },
"product_images": { "type": "array", "items": { "type": "string" } },
"product_model_number": { "type": "string" },
"product_name": { "type": "string" },
"product_price": { "type": "number" },
"product_price_before_discount": { "type": "number" },
"product_url": { "type": "string" },
"rating_score": { "type": "number" },
"review_count": { "type": "integer" },
"root_bs_rank": { "type": "integer" },
"sku": { "type": "string" }
}
}
We then use the prompt below to guide the AI from making the mistakes that actually show up on a page this dense:
Extract each field from its correct, distinct source, don't reuse values across fields or pattern-match wrong ones. If product_description would exactly match product_name, return an empty string instead of duplicating it. category_breadcrumb must start from the top-level Amazon department (e.g. "Electronics") and end in an actual leaf category name, never an interface or connector term like "USB." review_count and root_bs_rank must come from different page elements, identical or oddly round values (e.g. 150000) are a sign one was pulled wrong. sku = ASIN; product_model_number = manufacturer's model number, don't swap them. Strip invisible Unicode chars and extra whitespace from all strings.
Turn URL into structured data #
Start by signing up for a new account and you'll get to the Spidra playground:
Paste the URL, attach the schema and prompt, and run the extraction. Here is a real result against a Logitech K120 keyboard listing on Amazon:
{
"amazon_choice": false,
"availability_status": "Only 1 left in stock, order soon.",
"badge": "Amazon's Choice",
"brand": "Logitech",
"buybox_seller": "Amazon Resale",
"category_breadcrumb": [
"Electronics",
"Computers & Accessories",
"Computer Accessories & Peripherals",
"Keyboards, Mice & Accessories",
"Keyboards"
],
"is_available": true,
"manufacturer": "Logitech",
"price_currency_code": "USD",
"price_currency_symbol": "$",
"product_description": "Logitech K120 Wired Keyboard. Comfortable, quiet typing with a sleek yet sturdy design and plug-and-play USB connection.",
"product_images": [
"https://m.media-amazon.com/images/I/61j3wQheLXL._AC_SX425_.jpg"
],
"product_model_number": "920-002478",
"product_name": "Logitech K120 Wired Keyboard for Windows, USB Plug-and-Play",
"product_price": 9.73,
"product_price_before_discount": 9.73,
"product_url": "https://www.amazon.com/Logitech-920-002478-K120-USB-Keyboard/dp/B003ELVLKU",
"rating_score": 4.6,
"review_count": 7888,
"root_bs_rank": 2,
"sku": "B003ELVLKU"
}
Every field landed in the right place. sku
holds the ASIN, product_model_number
holds the actual manufacturer number, and category_breadcrumb
runs cleanly from "Electronics" down to "Keyboards" without stalling on a connector type. product_price
and product_price_before_discount
came back identical here because there was no active discount on this listing at the time of the scrape, not because a value got duplicated by mistake.
Get structured data from popular websites with the Spidra API #
The same schema and prompt work directly against the REST API, no dashboard needed. Submit the request and it returns a jobId
you poll for the result:
curl -X POST https://api.spidra.io/api/scrape \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [{"url": "https://www.amazon.com/Logitech-920-002478-K120-USB-Keyboard/dp/B003ELVLKU"}],
"prompt": "Extract each field from its correct, distinct source, do not reuse values across fields or pattern-match wrong ones. If product_description would exactly match product_name, return an empty string instead of duplicating it.",
"output": "json",
"useProxy": true,
"proxyCountry": "us",
"schema": {
"type": "object",
"required": [
"amazon_choice", "availability_status", "badge", "brand",
"buybox_seller", "category_breadcrumb", "is_available",
"manufacturer", "parent_asin", "price_currency_code",
"price_currency_symbol", "product_description", "product_images",
"product_model_number", "product_name", "product_price",
"product_price_before_discount", "product_url", "rating_score",
"review_count", "root_bs_rank", "sku"
],
"properties": {
"amazon_choice": { "type": "boolean" },
"availability_status": { "type": "string" },
"badge": { "type": "string" },
"brand": { "type": "string" },
"buybox_seller": { "type": "string" },
"category_breadcrumb": { "type": "array", "items": { "type": "string" } },
"is_available": { "type": "boolean" },
"manufacturer": { "type": "string" },
"parent_asin": { "type": "string" },
"price_currency_code": { "type": "string" },
"price_currency_symbol": { "type": "string" },
"product_description": { "type": "string" },
"product_images": { "type": "array", "items": { "type": "string" } },
"product_model_number": { "type": "string" },
"product_name": { "type": "string" },
"product_price": { "type": "number" },
"product_price_before_discount": { "type": "number" },
"product_url": { "type": "string" },
"rating_score": { "type": "number" },
"review_count": { "type": "integer" },
"root_bs_rank": { "type": "integer" },
"sku": { "type": "string" }
}
}
}'
The response comes back with a jobId
. You can then poll GET /api/scrape/{jobId}
until the status is completed
, and result.content
holds the same JSON shown above.
Scaling beyond a single product #
The same schema and prompt work unchanged in a batch request against up to 50 product URLs at once, or against any other site entirely.
Nothing above is Amazon-specific except the field names, the same approach works for Walmart products, or any structured page you need data from repeatedly.
Sign up free at app.spidra.io to try it, 300 credits, no card required.