{"slug": "build-a-web-scraping-pipeline-for-llms-with-crawl4ai", "title": "Build a Web Scraping Pipeline for LLMs with Crawl4AI", "summary": "Crawl4AI 0.9.3, released 2026-08-31, enables building a web scraping pipeline that converts pages into clean Markdown for RAG and structured JSON for agents without LLM calls. The tutorial, verified on macOS 26 with Python 3.11 and Playwright 1.62.0, demonstrates crawling quotes.toscrape.com pages 1-3, using a CSS schema to extract quotes, authors, and tags, and writing outputs to Markdown and JSONL files. The pipeline uses a single CrawlerRunConfig for both jobs, ensuring each page is fetched once.", "body_md": "# Build a Web Scraping Pipeline for LLMs with Crawl4AI\n\nCrawl pages once, get clean Markdown for RAG and structured JSON for agents, with no LLM calls.\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)\n\n## What you'll build\n\nA Python script that crawls a list of pages with [Crawl4AI](https://docs.crawl4ai.com/), strips navigation and boilerplate, and writes two outputs per run: clean Markdown files you can drop into a RAG index, and a JSONL file of structured records pulled from the same pages with a CSS schema. No LLM calls, no API keys.\n\n## Prerequisites\n\nVerified on macOS 26 (Apple Silicon) with:\n\n- Python 3.11 (Crawl4AI requires 3.10+; PyPI classifiers go up to 3.13, so avoid 3.14 for now)\n- Crawl4AI 0.9.3 (released 2026-08-31)\n- Playwright 1.62.0, pulled in as a dependency;\n`crawl4ai-setup`\n\ndownloads Chromium Headless Shell 151 (about 95 MB)\n\nOn Linux, Chromium needs system libraries. Run `python -m playwright install --with-deps chromium`\n\nafter the install step if `crawl4ai-setup`\n\ncomplains. No accounts needed. The target site, [quotes.toscrape.com](https://quotes.toscrape.com/), exists for scraping practice.\n\n## 1. Install Crawl4AI and the browser\n\n```\npython3.11 -m venv .venv && source .venv/bin/activate\npip install -U crawl4ai\ncrawl4ai-setup\ncrawl4ai-doctor\n```\n\n`crawl4ai-setup`\n\ninstalls the headless browser and initialises a local SQLite cache under `~/.crawl4ai/`\n\n. `crawl4ai-doctor`\n\ndoes a real test crawl of crawl4ai.com and should end with:\n\n```\n[COMPLETE] ● ✅ Crawling test passed!\n```\n\n## 2. Write the pipeline\n\nSave this as `pipeline.py`\n\n. It uses one `CrawlerRunConfig`\n\nfor both jobs: `markdown_generator`\n\nproduces the cleaned Markdown, `extraction_strategy`\n\nproduces the JSON. Both run against the same fetched HTML, so each page loads once.\n\n``` python\nimport asyncio\nimport json\nfrom pathlib import Path\n\nfrom crawl4ai import (\n    AsyncWebCrawler,\n    BrowserConfig,\n    CacheMode,\n    CrawlerRunConfig,\n    DefaultMarkdownGenerator,\n    JsonCssExtractionStrategy,\n    PruningContentFilter,\n)\n\nURLS = [\n    \"https://quotes.toscrape.com/page/1/\",\n    \"https://quotes.toscrape.com/page/2/\",\n    \"https://quotes.toscrape.com/page/3/\",\n]\nOUT = Path(\"out\")\n\nSCHEMA = {\n    \"name\": \"quotes\",\n    \"baseSelector\": \"div.quote\",\n    \"fields\": [\n        {\"name\": \"text\", \"selector\": \"span.text\", \"type\": \"text\"},\n        {\"name\": \"author\", \"selector\": \"small.author\", \"type\": \"text\"},\n        {\n            \"name\": \"tags\",\n            \"selector\": \"div.tags a.tag\",\n            \"type\": \"list\",\n            \"fields\": [{\"name\": \"tag\", \"type\": \"text\"}],\n        },\n    ],\n}\n\nasync def main():\n    OUT.mkdir(exist_ok=True)\n\n    run_cfg = CrawlerRunConfig(\n        cache_mode=CacheMode.BYPASS,\n        excluded_tags=[\"nav\", \"footer\", \"header\", \"aside\"],\n        exclude_external_links=True,\n        markdown_generator=DefaultMarkdownGenerator(\n            content_filter=PruningContentFilter(threshold=0.45, threshold_type=\"dynamic\"),\n            options={\"ignore_links\": True},\n        ),\n        extraction_strategy=JsonCssExtractionStrategy(SCHEMA),\n    )\n\n    records = []\n    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:\n        results = await crawler.arun_many(URLS, config=run_cfg)\n        for r in results:\n            if not r.success:\n                print(f\"[FAIL] {r.url}: {r.error_message}\")\n                continue\n            slug = r.url.rstrip(\"/\").split(\"/\")[-1]\n            (OUT / f\"{slug}.md\").write_text(r.markdown.fit_markdown)\n            items = json.loads(r.extracted_content)\n            for item in items:\n                item[\"tags\"] = [t[\"tag\"] for t in item[\"tags\"]]\n                records.append({\"source_url\": r.url, **item})\n            print(f\"[OK] {r.url}: {len(r.markdown.fit_markdown)} chars, {len(items)} quotes\")\n\n    with (OUT / \"quotes.jsonl\").open(\"w\") as f:\n        for rec in records:\n            f.write(json.dumps(rec, ensure_ascii=False) + \"\\n\")\n    print(f\"Wrote {len(records)} records to {OUT/'quotes.jsonl'}\")\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\nThe cleaning happens in three layers. `excluded_tags`\n\ndrops whole DOM regions before conversion, which is the cheapest win: on page 1 it cut raw Markdown from 4,375 to 1,619 characters. `PruningContentFilter`\n\nthen scores the remaining blocks by text density and link ratio and drops low scorers; its output lands in `result.markdown.fit_markdown`\n\n, while the unfiltered version stays in `raw_markdown`\n\n. `ignore_links`\n\nstrips anchor URLs from the Markdown so you don't pay tokens for them.\n\n`JsonCssExtractionStrategy`\n\ntakes a schema: `baseSelector`\n\nmatches one element per record, and each field's `selector`\n\nis scoped inside it. A `list`\n\nfield returns a list of dicts (`[{\"tag\": \"change\"}, ...]`\n\n), which is why the script flattens it to plain strings before writing.\n\n## 3. Run it\n\n```\npython pipeline.py\n```\n\n`arun_many`\n\nruns the three URLs concurrently in one browser. The whole run takes about 1.5 seconds on a laptop.\n\n## Verify it works\n\nCrawl4AI prints its own `[FETCH]`\n\n, `[SCRAPE]`\n\nand `[EXTRACT]`\n\nprogress lines. Below them you should see:\n\n```\n[OK] https://quotes.toscrape.com/page/1/: 1410 chars, 10 quotes\n[OK] https://quotes.toscrape.com/page/3/: 1584 chars, 10 quotes\n[OK] https://quotes.toscrape.com/page/2/: 3241 chars, 10 quotes\nWrote 30 records to out/quotes.jsonl\n```\n\nOrder varies because the pages finish concurrently. Check the outputs:\n\n```\nls out/\nhead -c 300 out/1.md\nhead -1 out/quotes.jsonl\n1.md  2.md  3.md  quotes.jsonl\n#  Quotes to Scrape\nLogin\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” by Albert Einstein (about)\nTags: change deep-thoughts thinking world\n{\"source_url\": \"https://quotes.toscrape.com/page/1/\", \"text\": \"“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\", \"author\": \"Albert Einstein\", \"tags\": [\"change\", \"deep-thoughts\", \"thinking\", \"world\"]}\n```\n\nThe Markdown has no nav, footer, or link URLs; the JSONL has one record per quote with tags as a flat list. Each `.md`\n\nfile is ready to chunk and embed, and each JSONL line is ready to load as tool output or a metadata-filtered document.\n\n## Troubleshooting\n\n`playwright._impl._errors.Error: BrowserType.launch: Executable doesn't exist at .../chromium_headless_shell-1234/...`\n\nYou skipped `crawl4ai-setup`\n\n, or ran it in a different virtualenv. Activate the venv you installed Crawl4AI into and run `crawl4ai-setup`\n\nagain. If it still fails, run `python -m playwright install chromium`\n\ndirectly.\n\n`Host system is missing dependencies to run browsers`\n\n(Linux only)\nChromium needs shared libraries that minimal images and CI runners don't ship. Run `python -m playwright install --with-deps chromium`\n\n(needs sudo on most distros) and retry.\n\n`TypeError: the JSON object must be str, bytes or bytearray, not NoneType`\n\n`result.extracted_content`\n\nis `None`\n\nbecause no `extraction_strategy`\n\nwas set on the `CrawlerRunConfig`\n\nyou passed. Check you're passing `config=run_cfg`\n\nto `arun_many`\n\n, not a fresh config.\n\n`fit_markdown`\n\nis an empty string\n`fit_markdown`\n\nis only populated when `DefaultMarkdownGenerator`\n\nhas a `content_filter`\n\n. Without one, use `result.markdown.raw_markdown`\n\ninstead.\n\n## Next steps\n\nTune `PruningContentFilter`\n\nper site: raise `threshold`\n\ntoward 0.6 on link-heavy pages, or set `min_word_threshold`\n\nto drop short nodes (on this site a value of 10 also drops the author lines, so test before trusting it). For query-driven trimming, swap in `BM25ContentFilter(user_query=\"...\")`\n\nfrom the same module. To discover pages instead of listing them, add a `deep_crawl_strategy`\n\nsuch as `BFSDeepCrawlStrategy(max_depth=2)`\n\nto the run config. And when a site's markup is too messy for CSS selectors, `LLMExtractionStrategy`\n\naccepts a Pydantic model and does the same job with a model call per page. All of these are documented under [docs.crawl4ai.com](https://docs.crawl4ai.com/).\n\n## Sources & further reading\n\n-\n[Crawl4AI Installation and Setup](https://docs.crawl4ai.com/core/installation/)— docs.crawl4ai.com -\n[Crawl4AI Markdown Generation Basics](https://docs.crawl4ai.com/core/markdown-generation/)— docs.crawl4ai.com -\n[Crawl4AI Extracting JSON (No LLM)](https://docs.crawl4ai.com/extraction/no-llm-strategies/)— docs.crawl4ai.com -\n[Crawl4AI Multi-URL Crawling](https://docs.crawl4ai.com/advanced/multi-url-crawling/)— docs.crawl4ai.com -\n[Crawl4AI 0.9.3 on PyPI](https://pypi.org/project/Crawl4AI/)— pypi.org -\n[Crawl4AI CHANGELOG](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md)— github.com\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor\n\nMariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/build-a-web-scraping-pipeline-for-llms-with-crawl4ai", "canonical_source": "https://sourcefeed.dev/a/build-a-web-scraping-pipeline-for-llms-with-crawl4ai", "published_at": "2026-09-01 11:40:53+00:00", "updated_at": "2026-09-01 11:53:20.104961+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "large-language-models"], "entities": ["Crawl4AI", "Playwright", "Chromium Headless Shell", "quotes.toscrape.com", "Mariana Souza", "Python"], "alternates": {"html": "https://wpnews.pro/news/build-a-web-scraping-pipeline-for-llms-with-crawl4ai", "markdown": "https://wpnews.pro/news/build-a-web-scraping-pipeline-for-llms-with-crawl4ai.md", "text": "https://wpnews.pro/news/build-a-web-scraping-pipeline-for-llms-with-crawl4ai.txt", "jsonld": "https://wpnews.pro/news/build-a-web-scraping-pipeline-for-llms-with-crawl4ai.jsonld"}}