# Build a Web Scraping Pipeline for LLMs with Crawl4AI > Source: > Published: 2026-09-01 11:40:53+00:00 # Build a Web Scraping Pipeline for LLMs with Crawl4AI Crawl pages once, get clean Markdown for RAG and structured JSON for agents, with no LLM calls. [Mariana Souza](https://sourcefeed.dev/u/mariana_souza) ## What you'll build A Python script that crawls a list of pages with [Crawl4AI](https://docs.crawl4ai.com/), strips navigation and boilerplate, and writes two outputs per run: clean Markdown files you can drop into a RAG index, and a JSONL file of structured records pulled from the same pages with a CSS schema. No LLM calls, no API keys. ## Prerequisites Verified on macOS 26 (Apple Silicon) with: - Python 3.11 (Crawl4AI requires 3.10+; PyPI classifiers go up to 3.13, so avoid 3.14 for now) - Crawl4AI 0.9.3 (released 2026-08-31) - Playwright 1.62.0, pulled in as a dependency; `crawl4ai-setup` downloads Chromium Headless Shell 151 (about 95 MB) On Linux, Chromium needs system libraries. Run `python -m playwright install --with-deps chromium` after the install step if `crawl4ai-setup` complains. No accounts needed. The target site, [quotes.toscrape.com](https://quotes.toscrape.com/), exists for scraping practice. ## 1. Install Crawl4AI and the browser ``` python3.11 -m venv .venv && source .venv/bin/activate pip install -U crawl4ai crawl4ai-setup crawl4ai-doctor ``` `crawl4ai-setup` installs the headless browser and initialises a local SQLite cache under `~/.crawl4ai/` . `crawl4ai-doctor` does a real test crawl of crawl4ai.com and should end with: ``` [COMPLETE] ● ✅ Crawling test passed! ``` ## 2. Write the pipeline Save this as `pipeline.py` . It uses one `CrawlerRunConfig` for both jobs: `markdown_generator` produces the cleaned Markdown, `extraction_strategy` produces the JSON. Both run against the same fetched HTML, so each page loads once. ``` python import asyncio import json from pathlib import Path from crawl4ai import ( AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig, DefaultMarkdownGenerator, JsonCssExtractionStrategy, PruningContentFilter, ) URLS = [ "https://quotes.toscrape.com/page/1/", "https://quotes.toscrape.com/page/2/", "https://quotes.toscrape.com/page/3/", ] OUT = Path("out") SCHEMA = { "name": "quotes", "baseSelector": "div.quote", "fields": [ {"name": "text", "selector": "span.text", "type": "text"}, {"name": "author", "selector": "small.author", "type": "text"}, { "name": "tags", "selector": "div.tags a.tag", "type": "list", "fields": [{"name": "tag", "type": "text"}], }, ], } async def main(): OUT.mkdir(exist_ok=True) run_cfg = CrawlerRunConfig( cache_mode=CacheMode.BYPASS, excluded_tags=["nav", "footer", "header", "aside"], exclude_external_links=True, markdown_generator=DefaultMarkdownGenerator( content_filter=PruningContentFilter(threshold=0.45, threshold_type="dynamic"), options={"ignore_links": True}, ), extraction_strategy=JsonCssExtractionStrategy(SCHEMA), ) records = [] async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler: results = await crawler.arun_many(URLS, config=run_cfg) for r in results: if not r.success: print(f"[FAIL] {r.url}: {r.error_message}") continue slug = r.url.rstrip("/").split("/")[-1] (OUT / f"{slug}.md").write_text(r.markdown.fit_markdown) items = json.loads(r.extracted_content) for item in items: item["tags"] = [t["tag"] for t in item["tags"]] records.append({"source_url": r.url, **item}) print(f"[OK] {r.url}: {len(r.markdown.fit_markdown)} chars, {len(items)} quotes") with (OUT / "quotes.jsonl").open("w") as f: for rec in records: f.write(json.dumps(rec, ensure_ascii=False) + "\n") print(f"Wrote {len(records)} records to {OUT/'quotes.jsonl'}") if __name__ == "__main__": asyncio.run(main()) ``` The cleaning happens in three layers. `excluded_tags` drops whole DOM regions before conversion, which is the cheapest win: on page 1 it cut raw Markdown from 4,375 to 1,619 characters. `PruningContentFilter` then scores the remaining blocks by text density and link ratio and drops low scorers; its output lands in `result.markdown.fit_markdown` , while the unfiltered version stays in `raw_markdown` . `ignore_links` strips anchor URLs from the Markdown so you don't pay tokens for them. `JsonCssExtractionStrategy` takes a schema: `baseSelector` matches one element per record, and each field's `selector` is scoped inside it. A `list` field returns a list of dicts (`[{"tag": "change"}, ...]` ), which is why the script flattens it to plain strings before writing. ## 3. Run it ``` python pipeline.py ``` `arun_many` runs the three URLs concurrently in one browser. The whole run takes about 1.5 seconds on a laptop. ## Verify it works Crawl4AI prints its own `[FETCH]` , `[SCRAPE]` and `[EXTRACT]` progress lines. Below them you should see: ``` [OK] https://quotes.toscrape.com/page/1/: 1410 chars, 10 quotes [OK] https://quotes.toscrape.com/page/3/: 1584 chars, 10 quotes [OK] https://quotes.toscrape.com/page/2/: 3241 chars, 10 quotes Wrote 30 records to out/quotes.jsonl ``` Order varies because the pages finish concurrently. Check the outputs: ``` ls out/ head -c 300 out/1.md head -1 out/quotes.jsonl 1.md 2.md 3.md quotes.jsonl # Quotes to Scrape Login “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” by Albert Einstein (about) Tags: change deep-thoughts thinking world {"source_url": "https://quotes.toscrape.com/page/1/", "text": "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”", "author": "Albert Einstein", "tags": ["change", "deep-thoughts", "thinking", "world"]} ``` The Markdown has no nav, footer, or link URLs; the JSONL has one record per quote with tags as a flat list. Each `.md` file is ready to chunk and embed, and each JSONL line is ready to load as tool output or a metadata-filtered document. ## Troubleshooting `playwright._impl._errors.Error: BrowserType.launch: Executable doesn't exist at .../chromium_headless_shell-1234/...` You skipped `crawl4ai-setup` , or ran it in a different virtualenv. Activate the venv you installed Crawl4AI into and run `crawl4ai-setup` again. If it still fails, run `python -m playwright install chromium` directly. `Host system is missing dependencies to run browsers` (Linux only) Chromium needs shared libraries that minimal images and CI runners don't ship. Run `python -m playwright install --with-deps chromium` (needs sudo on most distros) and retry. `TypeError: the JSON object must be str, bytes or bytearray, not NoneType` `result.extracted_content` is `None` because no `extraction_strategy` was set on the `CrawlerRunConfig` you passed. Check you're passing `config=run_cfg` to `arun_many` , not a fresh config. `fit_markdown` is an empty string `fit_markdown` is only populated when `DefaultMarkdownGenerator` has a `content_filter` . Without one, use `result.markdown.raw_markdown` instead. ## Next steps Tune `PruningContentFilter` per site: raise `threshold` toward 0.6 on link-heavy pages, or set `min_word_threshold` to drop short nodes (on this site a value of 10 also drops the author lines, so test before trusting it). For query-driven trimming, swap in `BM25ContentFilter(user_query="...")` from the same module. To discover pages instead of listing them, add a `deep_crawl_strategy` such as `BFSDeepCrawlStrategy(max_depth=2)` to the run config. And when a site's markup is too messy for CSS selectors, `LLMExtractionStrategy` accepts a Pydantic model and does the same job with a model call per page. All of these are documented under [docs.crawl4ai.com](https://docs.crawl4ai.com/). ## Sources & further reading - [Crawl4AI Installation and Setup](https://docs.crawl4ai.com/core/installation/)— docs.crawl4ai.com - [Crawl4AI Markdown Generation Basics](https://docs.crawl4ai.com/core/markdown-generation/)— docs.crawl4ai.com - [Crawl4AI Extracting JSON (No LLM)](https://docs.crawl4ai.com/extraction/no-llm-strategies/)— docs.crawl4ai.com - [Crawl4AI Multi-URL Crawling](https://docs.crawl4ai.com/advanced/multi-url-crawling/)— docs.crawl4ai.com - [Crawl4AI 0.9.3 on PyPI](https://pypi.org/project/Crawl4AI/)— pypi.org - [Crawl4AI CHANGELOG](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md)— github.com [Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. ## Discussion 0 No comments yet Be the first to weigh in.