Build a Web Scraping Pipeline for LLMs with Crawl4AI

Crawl4AI 0.9.3, released 2026-08-31, enables building a web scraping pipeline that converts pages into clean Markdown for RAG and structured JSON for agents without LLM calls. The tutorial, verified on macOS 26 with Python 3.11 and Playwright 1.62.0, demonstrates crawling quotes.toscrape.com pages 1-3, using a CSS schema to extract quotes, authors, and tags, and writing outputs to Markdown and JSONL files. The pipeline uses a single CrawlerRunConfig for both jobs, ensuring each page is fetched once.

Build a Web Scraping Pipeline for LLMs with Crawl4AI Crawl pages once, get clean Markdown for RAG and structured JSON for agents, with no LLM calls. Mariana Souza https://sourcefeed.dev/u/mariana souza What you'll build A Python script that crawls a list of pages with Crawl4AI https://docs.crawl4ai.com/ , strips navigation and boilerplate, and writes two outputs per run: clean Markdown files you can drop into a RAG index, and a JSONL file of structured records pulled from the same pages with a CSS schema. No LLM calls, no API keys. Prerequisites Verified on macOS 26 Apple Silicon with: - Python 3.11 Crawl4AI requires 3.10+; PyPI classifiers go up to 3.13, so avoid 3.14 for now - Crawl4AI 0.9.3 released 2026-08-31 - Playwright 1.62.0, pulled in as a dependency; crawl4ai-setup downloads Chromium Headless Shell 151 about 95 MB On Linux, Chromium needs system libraries. Run python -m playwright install --with-deps chromium after the install step if crawl4ai-setup complains. No accounts needed. The target site, quotes.toscrape.com https://quotes.toscrape.com/ , exists for scraping practice. 1. Install Crawl4AI and the browser python3.11 -m venv .venv && source .venv/bin/activate pip install -U crawl4ai crawl4ai-setup crawl4ai-doctor crawl4ai-setup installs the headless browser and initialises a local SQLite cache under ~/.crawl4ai/ . crawl4ai-doctor does a real test crawl of crawl4ai.com and should end with: COMPLETE ● ✅ Crawling test passed 2. Write the pipeline Save this as pipeline.py . It uses one CrawlerRunConfig for both jobs: markdown generator produces the cleaned Markdown, extraction strategy produces the JSON. Both run against the same fetched HTML, so each page loads once. python import asyncio import json from pathlib import Path from crawl4ai import AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig, DefaultMarkdownGenerator, JsonCssExtractionStrategy, PruningContentFilter, URLS = "https://quotes.toscrape.com/page/1/", "https://quotes.toscrape.com/page/2/", "https://quotes.toscrape.com/page/3/", OUT = Path "out" SCHEMA = { "name": "quotes", "baseSelector": "div.quote", "fields": {"name": "text", "selector": "span.text", "type": "text"}, {"name": "author", "selector": "small.author", "type": "text"}, { "name": "tags", "selector": "div.tags a.tag", "type": "list", "fields": {"name": "tag", "type": "text"} , }, , } async def main : OUT.mkdir exist ok=True run cfg = CrawlerRunConfig cache mode=CacheMode.BYPASS, excluded tags= "nav", "footer", "header", "aside" , exclude external links=True, markdown generator=DefaultMarkdownGenerator content filter=PruningContentFilter threshold=0.45, threshold type="dynamic" , options={"ignore links": True}, , extraction strategy=JsonCssExtractionStrategy SCHEMA , records = async with AsyncWebCrawler config=BrowserConfig headless=True as crawler: results = await crawler.arun many URLS, config=run cfg for r in results: if not r.success: print f" FAIL {r.url}: {r.error message}" continue slug = r.url.rstrip "/" .split "/" -1 OUT / f"{slug}.md" .write text r.markdown.fit markdown items = json.loads r.extracted content for item in items: item "tags" = t "tag" for t in item "tags" records.append {"source url": r.url, item} print f" OK {r.url}: {len r.markdown.fit markdown } chars, {len items } quotes" with OUT / "quotes.jsonl" .open "w" as f: for rec in records: f.write json.dumps rec, ensure ascii=False + "\n" print f"Wrote {len records } records to {OUT/'quotes.jsonl'}" if name == " main ": asyncio.run main The cleaning happens in three layers. excluded tags drops whole DOM regions before conversion, which is the cheapest win: on page 1 it cut raw Markdown from 4,375 to 1,619 characters. PruningContentFilter then scores the remaining blocks by text density and link ratio and drops low scorers; its output lands in result.markdown.fit markdown , while the unfiltered version stays in raw markdown . ignore links strips anchor URLs from the Markdown so you don't pay tokens for them. JsonCssExtractionStrategy takes a schema: baseSelector matches one element per record, and each field's selector is scoped inside it. A list field returns a list of dicts {"tag": "change"}, ... , which is why the script flattens it to plain strings before writing. 3. Run it python pipeline.py arun many runs the three URLs concurrently in one browser. The whole run takes about 1.5 seconds on a laptop. Verify it works Crawl4AI prints its own FETCH , SCRAPE and EXTRACT progress lines. Below them you should see: OK https://quotes.toscrape.com/page/1/: 1410 chars, 10 quotes OK https://quotes.toscrape.com/page/3/: 1584 chars, 10 quotes OK https://quotes.toscrape.com/page/2/: 3241 chars, 10 quotes Wrote 30 records to out/quotes.jsonl Order varies because the pages finish concurrently. Check the outputs: ls out/ head -c 300 out/1.md head -1 out/quotes.jsonl 1.md 2.md 3.md quotes.jsonl Quotes to Scrape Login “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” by Albert Einstein about Tags: change deep-thoughts thinking world {"source url": "https://quotes.toscrape.com/page/1/", "text": "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”", "author": "Albert Einstein", "tags": "change", "deep-thoughts", "thinking", "world" } The Markdown has no nav, footer, or link URLs; the JSONL has one record per quote with tags as a flat list. Each .md file is ready to chunk and embed, and each JSONL line is ready to load as tool output or a metadata-filtered document. Troubleshooting playwright. impl. errors.Error: BrowserType.launch: Executable doesn't exist at .../chromium headless shell-1234/... You skipped crawl4ai-setup , or ran it in a different virtualenv. Activate the venv you installed Crawl4AI into and run crawl4ai-setup again. If it still fails, run python -m playwright install chromium directly. Host system is missing dependencies to run browsers Linux only Chromium needs shared libraries that minimal images and CI runners don't ship. Run python -m playwright install --with-deps chromium needs sudo on most distros and retry. TypeError: the JSON object must be str, bytes or bytearray, not NoneType result.extracted content is None because no extraction strategy was set on the CrawlerRunConfig you passed. Check you're passing config=run cfg to arun many , not a fresh config. fit markdown is an empty string fit markdown is only populated when DefaultMarkdownGenerator has a content filter . Without one, use result.markdown.raw markdown instead. Next steps Tune PruningContentFilter per site: raise threshold toward 0.6 on link-heavy pages, or set min word threshold to drop short nodes on this site a value of 10 also drops the author lines, so test before trusting it . For query-driven trimming, swap in BM25ContentFilter user query="..." from the same module. To discover pages instead of listing them, add a deep crawl strategy such as BFSDeepCrawlStrategy max depth=2 to the run config. And when a site's markup is too messy for CSS selectors, LLMExtractionStrategy accepts a Pydantic model and does the same job with a model call per page. All of these are documented under docs.crawl4ai.com https://docs.crawl4ai.com/ . Sources & further reading - Crawl4AI Installation and Setup https://docs.crawl4ai.com/core/installation/ — docs.crawl4ai.com - Crawl4AI Markdown Generation Basics https://docs.crawl4ai.com/core/markdown-generation/ — docs.crawl4ai.com - Crawl4AI Extracting JSON No LLM https://docs.crawl4ai.com/extraction/no-llm-strategies/ — docs.crawl4ai.com - Crawl4AI Multi-URL Crawling https://docs.crawl4ai.com/advanced/multi-url-crawling/ — docs.crawl4ai.com - Crawl4AI 0.9.3 on PyPI https://pypi.org/project/Crawl4AI/ — pypi.org - Crawl4AI CHANGELOG https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md — github.com Mariana Souza https://sourcefeed.dev/u/mariana souza · Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. Discussion 0 No comments yet Be the first to weigh in.