Crawl4AI: Open-source web crawler and scraper for LLMs

Crawl4AI, an open-source web crawler and scraper that converts web pages into LLM-ready Markdown, has released version 0.9.1 with 12 bug fixes. The tool, which has over 50,000 GitHub stars, is designed for RAG, agents, and data pipelines, offering cost-effective web extraction without requiring accounts or API tokens.

Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions. 👉 Apply here for early access We’ll be onboarding in phases and working closely with early users. Limited slots. Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community. ✨ Check out latest update v0.9.1 -recent-updates ✨ New in v0.9.1 : Patch release with 12 bug fixes across Docker, browser, and core. Adds preserve classes / preserve tags whitelist for PruningContentFilter, fixes Windows browser crash, Docker auth gate UI, HTTP timeout unit mismatch, and more. Release notes → https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.1.md ✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Release notes → https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.0.md ✨ Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret , adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. Release notes → https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.7.md ✨ Previous v0.8.0: Crash Recovery & Prefetch Mode Deep crawl crash recovery with resume state and on state change callbacks for long-running crawls. New prefetch=True mode for 5-10x faster URL discovery. Release notes → https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.0.md ✨ Previous v0.7.8: Stability & Bug Fix Release 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. Release notes → https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.8.md 🤓 My Personal Story I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters. In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub. I made it open source for availability , anyone can use it without a gate. Now I’m building the platform for affordability , anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing. Why developers pick Crawl4AI LLM ready output , smart Markdown with headings, tables, code, citation hints Fast in practice , async browser pool, caching, minimal hops Full control , sessions, proxies, cookies, user scripts, hooks Adaptive intelligence , learns site patterns, explores only what matters Deploy anywhere , zero keys, CLI and Docker, cloud friendly - Install Crawl4AI: Install the package pip install -U crawl4ai For pre release versions pip install crawl4ai --pre Run post-installation setup crawl4ai-setup Verify your installation crawl4ai-doctor If you encounter any browser-related issues, you can install them manually: python -m playwright install --with-deps chromium - Run a simple web crawl with Python: python import asyncio from crawl4ai import async def main : async with AsyncWebCrawler as crawler: result = await crawler.arun url="https://www.nbcnews.com/business", print result.markdown if name == " main ": asyncio.run main - Or use the new command-line interface: Basic crawl with markdown output crwl https://www.nbcnews.com/business -o markdown Deep crawl with BFS strategy, max 10 pages crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10 Use LLM extraction with a specific question crwl https://www.example.com/products -q "Extract all product prices" 🎉 Sponsorship Program Now Open After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support forstartupsandenterprises. Be among the first 50Founding Sponsorsfor permanent recognition in our Hall of Fame. Crawl4AI is the 1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community — while giving you direct access to premium benefits. 🌱 Believer $5/mo — Join the movement for data democratization 🚀 Builder $50/mo — Priority support & early access to features 💼 Growing Team $500/mo — Bi-weekly syncs & optimization help 🏢 Data Infrastructure Partner $2000/mo — Full partnership with dedicated support Custom arrangements available - see SPONSORS.md /unclecode/crawl4ai/blob/main/SPONSORS.md for details & contact Why sponsor? No rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI. 📝 Markdown Generation - 🧹 Clean Markdown : Generates clean, structured Markdown with accurate formatting. - 🎯 Fit Markdown : Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing. - 🔗 Citations and References : Converts page links into a numbered reference list with clean citations. - 🛠️ Custom Strategies : Users can create their own Markdown generation strategies tailored to specific needs. - 📚 BM25 Algorithm : Employs BM25-based filtering for extracting core information and removing irrelevant content. 📊 Structured Data Extraction - 🤖 LLM-Driven Extraction : Supports all LLMs open-source and proprietary for structured data extraction. - 🧱 Chunking Strategies : Implements chunking topic-based, regex, sentence-level for targeted content processing. - 🌌 Cosine Similarity : Find relevant content chunks based on user queries for semantic extraction. - 🔎 CSS-Based Extraction : Fast schema-based data extraction using XPath and CSS selectors. - 🔧 Schema Definition : Define custom schemas for extracting structured JSON from repetitive patterns. 🌐 Browser Integration - 🖥️ Managed Browser : Use user-owned browsers with full control, avoiding bot detection. - 🔄 Remote Browser Control : Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction. - 👤 Browser Profiler : Create and manage persistent profiles with saved authentication states, cookies, and settings. - 🔒 Session Management : Preserve browser states and reuse them for multi-step crawling. - 🧩 Proxy Support : Seamlessly connect to proxies with authentication for secure access. - ⚙️ Full Browser Control : Modify headers, cookies, user agents, and more for tailored crawling setups. - 🌍 Multi-Browser Support : Compatible with Chromium, Firefox, and WebKit. - 📐 Dynamic Viewport Adjustment : Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements. 🔎 Crawling & Scraping - 🖼️ Media Support : Extract images, audio, videos, and responsive image formats like srcset and picture . - 🚀 Dynamic Crawling : Execute JS and wait for async or sync for dynamic content extraction. - 📸 Screenshots : Capture page screenshots during crawling for debugging or analysis. - 📂 Raw Data Crawling : Directly process raw HTML raw: or local files file:// . - 🔗 Comprehensive Link Extraction : Extracts internal, external links, and embedded iframe content. - 🛠️ Customizable Hooks : Define hooks at every step to customize crawling behavior supports both string and function-based APIs . - 💾 Caching : Cache data for improved speed and to avoid redundant fetches. - 📄 Metadata Extraction : Retrieve structured metadata from web pages. - 📡 IFrame Content Extraction : Seamless extraction from embedded iframe content. - 🕵️ Lazy Load Handling : Waits for images to fully load, ensuring no content is missed due to lazy loading. - 🔄 Full-Page Scanning : Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages. 🚀 Deployment - 🐳 Dockerized Setup : Optimized Docker image with FastAPI server for easy deployment. - 🔑 Secure Authentication : Built-in JWT token authentication for API security. - 🔄 API Gateway : One-click deployment with secure token authentication for API-based workflows. - 🌐 Scalable Architecture : Designed for mass-scale production and optimized server performance. - ☁️ Cloud Deployment : Ready-to-deploy configurations for major cloud platforms. 🎯 Additional Features - 🕶️ Stealth Mode : Avoid bot detection by mimicking real users. - 🏷️ Tag-Based Content Extraction : Refine crawling based on custom tags, headers, or metadata. - 🔗 Link Analysis : Extract and analyze all links for detailed data exploration. - 🛡️ Error Handling : Robust error management for seamless execution. - 🔐 CORS & Static Serving : Supports filesystem-based caching and cross-origin requests. - 📖 Clear Documentation : Simplified and updated guides for onboarding and advanced usage. - 🙌 Community Recognition : Acknowledges contributors and pull requests for transparency. ✨ Visit our Documentation Website https://docs.crawl4ai.com/ Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker. 🐍 Using pip Choose the installation option that best fits your needs: For basic web crawling and scraping tasks: pip install crawl4ai crawl4ai-setup Setup the browser By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling. 👉 Note : When you install Crawl4AI, the crawl4ai-setup should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods: - Through the command line: playwright install - If the above doesn't work, try this more specific command: python -m playwright install chromium This second method has proven to be more reliable in some cases. The sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium: pip install crawl4ai sync For contributors who plan to modify the source code: git clone https://github.com/unclecode/crawl4ai.git cd crawl4ai pip install -e . Basic installation in editable mode Install optional features: pip install -e ". torch " With PyTorch features pip install -e ". transformer " With Transformer features pip install -e ". cosine " With cosine similarity features pip install -e ". sync " With synchronous crawling Selenium pip install -e ". all " Install all optional features 🐳 Docker Deployment 🚀 Now Available Our completely redesigned Docker implementation is here This new solution makes deployment more efficient and seamless than ever. The new Docker implementation includes: Real-time Monitoring Dashboard with live system metrics and browser pool visibility Browser pooling with page pre-warming for faster response times Interactive playground to test and generate request code MCP integration for direct connection to AI tools like Claude Code Comprehensive API endpoints including HTML extraction, screenshots, PDF generation, and JavaScript execution Multi-architecture support with automatic detection AMD64/ARM64 Optimized resources with improved memory management Pull and run the latest release docker pull unclecode/crawl4ai:latest docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest Visit the monitoring dashboard at http://localhost:11235/dashboard Or the playground at http://localhost:11235/playground Run a quick test works for both Docker options : python import requests Submit a crawl job response = requests.post "http://localhost:11235/crawl", json={"urls": "https://example.com" , "priority": 10} if response.status code == 200: print "Crawl job submitted successfully." if "results" in response.json : results = response.json "results" print "Crawl job completed. Results:" for result in results: print result else: task id = response.json "task id" print f"Crawl job submitted. Task ID:: {task id}" result = requests.get f"http://localhost:11235/task/{task id}" For more examples, see our Docker Examples https://github.com/unclecode/crawl4ai/blob/main/docs/examples/docker example.py . For advanced configuration, monitoring features, and production deployment, see our Self-Hosting Guide https://docs.crawl4ai.com/core/self-hosting/ . You can check the project structure in the directory docs/examples https://github.com/unclecode/crawl4ai/tree/main/docs/examples . Over there, you can find a variety of examples; here, some popular examples are shared. 📝 Heuristic Markdown Generation with Clean and Fit Markdown python import asyncio from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode from crawl4ai.content filter strategy import PruningContentFilter, BM25ContentFilter from crawl4ai.markdown generation strategy import DefaultMarkdownGenerator async def main : browser config = BrowserConfig headless=True, verbose=True, run config = CrawlerRunConfig cache mode=CacheMode.ENABLED, markdown generator=DefaultMarkdownGenerator content filter=PruningContentFilter threshold=0.48, threshold type="fixed", min word threshold=0 , markdown generator=DefaultMarkdownGenerator content filter=BM25ContentFilter user query="WHEN WE FOCUS BASED ON A USER QUERY", bm25 threshold=1.0 , async with AsyncWebCrawler config=browser config as crawler: result = await crawler.arun url="https://docs.micronaut.io/4.9.9/guide/", config=run config print len result.markdown.raw markdown print len result.markdown.fit markdown if name == " main ": asyncio.run main 🖥️ Executing JavaScript & Extract Structured Data without LLMs python import asyncio from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode from crawl4ai import JsonCssExtractionStrategy import json async def main : schema = { "name": "KidoCode Courses", "baseSelector": "section.charge-methodology .w-tab-content div", "fields": { "name": "section title", "selector": "h3.heading-50", "type": "text", }, { "name": "section description", "selector": ".charge-content", "type": "text", }, { "name": "course name", "selector": ".text-block-93", "type": "text", }, { "name": "course description", "selector": ".course-content-text", "type": "text", }, { "name": "course icon", "selector": ".image-92", "type": "attribute", "attribute": "src" } } extraction strategy = JsonCssExtractionStrategy schema, verbose=True browser config = BrowserConfig headless=False, verbose=True run config = CrawlerRunConfig extraction strategy=extraction strategy, js code= """ async = {const tabs = document.querySelectorAll "section.charge-methodology .tabs-menu-3 div" ;for let tab of tabs {tab.scrollIntoView ;tab.click ;await new Promise r = setTimeout r, 500 ;}} ;""" , cache mode=CacheMode.BYPASS async with AsyncWebCrawler config=browser config as crawler: result = await crawler.arun url="https://www.kidocode.com/degrees/technology", config=run config companies = json.loads result.extracted content print f"Successfully extracted {len companies } companies" print json.dumps companies 0 , indent=2 if name == " main ": asyncio.run main 📚 Extracting Structured Data with LLMs python import os import asyncio from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, LLMConfig from crawl4ai import LLMExtractionStrategy from pydantic import BaseModel, Field class OpenAIModelFee BaseModel : model name: str = Field ..., description="Name of the OpenAI model." input fee: str = Field ..., description="Fee for input token for the OpenAI model." output fee: str = Field ..., description="Fee for output token for the OpenAI model." async def main : browser config = BrowserConfig verbose=True run config = CrawlerRunConfig word count threshold=1, extraction strategy=LLMExtractionStrategy Here you can use any provider that Litellm library supports, for instance: ollama/qwen2 provider="ollama/qwen2", api token="no-token", llm config = LLMConfig provider="openai/gpt-4o", api token=os.getenv 'OPENAI API KEY' , schema=OpenAIModelFee.schema , extraction type="schema", instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens. Do not miss any models in the entire content. One extracted model JSON format should look like this: {"model name": "GPT-4", "input fee": "US$10.00 / 1M tokens", "output fee": "US$30.00 / 1M tokens"}.""" , cache mode=CacheMode.BYPASS, async with AsyncWebCrawler config=browser config as crawler: result = await crawler.arun url='https://openai.com/api/pricing/', config=run config print result.extracted content if name == " main ": asyncio.run main 🤖 Using Your own Browser with Custom User Profile python import os, sys from pathlib import Path import asyncio, time from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode async def test news crawl : Create a persistent user data directory user data dir = os.path.join Path.home , ".crawl4ai", "browser profile" os.makedirs user data dir, exist ok=True browser config = BrowserConfig verbose=True, headless=True, user data dir=user data dir, use persistent context=True, run config = CrawlerRunConfig cache mode=CacheMode.BYPASS async with AsyncWebCrawler config=browser config as crawler: url = "ADDRESS OF A CHALLENGING WEBSITE" result = await crawler.arun url, config=run config, magic=True, print f"Successfully crawled {url}" print f"Content length: {len result.markdown }" Version 0.9.1 Release Highlights - Bug Fixes & PruningContentFilter Whitelist A patch release with 12 bug fixes and one new feature. The new preserve classes / preserve tags parameters for PruningContentFilter let you whitelist CSS classes or HTML tags that should never be pruned — useful for protecting short metadata elements like author names and timestamps. Bug fixes span Docker auth gate UI, supervisord/redis dirs, FastAPI compatibility, redis auth , browser Windows channel crash, context snapshot leak , core HTTP timeout unit mismatch, best-first ordering , and extraction html2text table attributes . pip install -U crawl4ai Version 0.9.0 Release Highlights - Secure-by-Default Docker Server A major, secure-by-default release of the Docker API server. The out-of-the-box deployment is hardened with defense in depth: authentication is on by default, the server binds loopback unless you give it a token, and the network request body is treated as an untrusted trust boundary. pip install -U crawl4ai Version 0.8.7 Release Highlights - Security Hardening, DomainMapper & Community Fixes A security-hardening release. Fixes critical Docker API vulnerabilities AST sandbox escape RCE, hook sandbox RCE, hardcoded JWT secret, SSRF on webhook and crawl endpoints, arbitrary file write, monitor auth bypass, stored XSS, and unauthenticated JS execution , adds the DomainMapper feature, and ships a batch of scraping, deep-crawl, and LLM fixes. If you self-host the Docker API, upgrade immediately. pip install -U crawl4ai Version 0.8.6 - Security Hotfix: litellm Supply Chain Fix Replaced litellm dependency with unclecode-litellm due to a PyPI supply chain compromise affecting the original package. If you're on v0.8.5 or earlier, upgrade immediately. pip install -U crawl4ai Version 0.8.5 Release Highlights - Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes Our biggest release since v0.8.0. Anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes. - 🛡️ Anti-Bot Detection & Proxy Escalation :- 3-tier detection: known vendors, generic block indicators, structural integrity checks - Automatic retry with proxy chain and fallback fetch function python from crawl4ai import CrawlerRunConfig from crawl4ai.async configs import ProxyConfig config = CrawlerRunConfig proxy config= ProxyConfig.DIRECT, ProxyConfig server="http://my-proxy:8080" , max retries=2, fallback fetch function=my web unlocker, - 🌑 Shadow DOM Flattening :- Extract content hidden inside shadow DOM components config = CrawlerRunConfig flatten shadow dom=True - 🛑 Deep Crawl Cancellation :- Stop long crawls gracefully with cancel or should cancel callback - Works with BFS, DFS, and BestFirst strategies - Stop long crawls gracefully with - ⚙️ Config Defaults API : set defaults / get defaults / reset defaults on BrowserConfig and CrawlerRunConfig - 🔒 Critical Security Fixes :- RCE via deserialization in Docker /crawl endpoint — removed eval , added allowlist - Redis CVE-2025-49844 CVSS 10.0 — upgraded to 7.2.7 - RCE via deserialization in Docker - 60+ Bug Fixes across browser management, proxy, deep crawling, extraction, CLI, and Docker Version 0.8.0 Release Highlights - Crash Recovery & Prefetch Mode This release introduces crash recovery for deep crawls, a new prefetch mode for fast URL discovery, and critical security fixes for Docker deployments. - 🔄 Deep Crawl Crash Recovery : on state change callback fires after each URL for real-time state persistence resume state parameter to continue from a saved checkpoint- JSON-serializable state for Redis/database storage - Works with BFS, DFS, and Best-First strategies python from crawl4ai.deep crawling import BFSDeepCrawlStrategy strategy = BFSDeepCrawlStrategy max depth=3, resume state=saved state, Continue from checkpoint on state change=save to redis, Called after each URL - ⚡ Prefetch Mode for Fast URL Discovery : prefetch=True skips markdown, extraction, and media processing- 5-10x faster than full processing - Perfect for two-phase crawling: discover first, process selectively config = CrawlerRunConfig prefetch=True result = await crawler.arun "https://example.com", config=config Returns HTML and links only - no markdown generation - 🔒 Security Fixes Docker API :- Hooks disabled by default CRAWL4AI HOOKS ENABLED=false file:// URLs blocked on API endpoints to prevent LFI import removed from hook execution sandbox - Hooks disabled by default Version 0.7.8 Release Highlights - Stability & Bug Fix Release This release focuses on stability with 11 bug fixes addressing issues reported by the community. No new features, but significant improvements to reliability. - 🐳 Docker API Fixes :- Fixed ContentRelevanceFilter deserialization in deep crawl requests 1642 - Fixed ProxyConfig JSON serialization in BrowserConfig.to dict 1629 - Fixed .cache folder permissions in Docker image 1638 - Fixed - 🤖 LLM Extraction Improvements :- Configurable rate limiter backoff with new LLMConfig parameters 1269 : python from crawl4ai import LLMConfig config = LLMConfig provider="openai/gpt-4o-mini", backoff base delay=5, Wait 5s on first retry backoff max attempts=5, Try up to 5 times backoff exponential factor=3 Multiply delay by 3 each attempt - HTML input format support for LLMExtractionStrategy 1178 : python from crawl4ai import LLMExtractionStrategy strategy = LLMExtractionStrategy llm config=config, instruction="Extract table data", input format="html" Now supports: "html", "markdown", "fit markdown" - Fixed raw HTML URL variable - extraction strategies now receive "Raw HTML" instead of HTML blob 1116 - Configurable rate limiter backoff with new - 🔗 URL Handling :- Fixed relative URL resolution after JavaScript redirects 1268 - Fixed import statement formatting in extracted code 1181 - 📦 Dependency Updates :- Replaced deprecated PyPDF2 with pypdf 1412 - Pydantic v2 ConfigDict compatibility - no more deprecation warnings 678 - 🧠 AdaptiveCrawler :- Fixed query expansion to actually use LLM instead of hardcoded mock data 1621 Version 0.7.7 Release Highlights - The Self-Hosting & Monitoring Update - 📊 Real-time Monitoring Dashboard : Interactive web UI with live system metrics and browser pool visibility Access the monitoring dashboard Visit: http://localhost:11235/dashboard Real-time metrics include: - System health CPU, memory, network, uptime - Active and completed request tracking - Browser pool management permanent/hot/cold - Janitor cleanup events - Error monitoring with full context - 🔌 Comprehensive Monitor API : Complete REST API for programmatic access to all monitoring data python import httpx async with httpx.AsyncClient as client: System health health = await client.get "http://localhost:11235/monitor/health" Request tracking requests = await client.get "http://localhost:11235/monitor/requests" Browser pool status browsers = await client.get "http://localhost:11235/monitor/browsers" Endpoint statistics stats = await client.get "http://localhost:11235/monitor/endpoints/stats" - ⚡ WebSocket Streaming : Real-time updates every 2 seconds for custom dashboards - 🔥 Smart Browser Pool : 3-tier architecture permanent/hot/cold with automatic promotion and cleanup - 🧹 Janitor System : Automatic resource management with event logging - 🎮 Control Actions : Manual browser management kill, restart, cleanup via API - 📈 Production Metrics : 6 critical metrics for operational excellence with Prometheus integration - 🐛 Critical Bug Fixes :- Fixed async LLM extraction blocking issue 1055 - Enhanced DFS deep crawl strategy 1607 - Fixed sitemap parsing in AsyncUrlSeeder 1598 - Resolved browser viewport configuration 1495 - Fixed CDP timing with exponential backoff 1528 - Security update for pyOpenSSL =25.3.0 Version 0.7.5 Release Highlights - The Docker Hooks & Security Update - 🔧 Docker Hooks System : Complete pipeline customization with user-provided Python functions at 8 key points - ✨ Function-Based Hooks API NEW : Write hooks as regular Python functions with full IDE support: python from crawl4ai import hooks to string from crawl4ai.docker client import Crawl4aiDockerClient Define hooks as regular Python functions async def on page context created page, context, kwargs : """Block images to speed up crawling""" await context.route " / .{png,jpg,jpeg,gif,webp}", lambda route: route.abort await page.set viewport size {"width": 1920, "height": 1080} return page async def before goto page, context, url, kwargs : """Add custom headers""" await page.set extra http headers {'X-Crawl4AI': 'v0.7.5'} return page Option 1: Use hooks to string utility for REST API hooks code = hooks to string { "on page context created": on page context created, "before goto": before goto } Option 2: Docker client with automatic conversion Recommended client = Crawl4aiDockerClient base url="http://localhost:11235" results = await client.crawl urls= "https://httpbin.org/html" , hooks={ "on page context created": on page context created, "before goto": before goto } ✓ Full IDE support, type checking, and reusability - 🤖 Enhanced LLM Integration : Custom providers with temperature control and base url configuration - 🔒 HTTPS Preservation : Secure internal link handling with preserve https for internal links=True - 🐍 Python 3.10+ Support : Modern language features and enhanced performance - 🛠️ Bug Fixes : Resolved multiple community-reported issues including URL processing, JWT authentication, and proxy configuration Version 0.7.4 Release Highlights - The Intelligent Table Extraction & Performance Update - 🚀 LLMTableExtraction : Revolutionary table extraction with intelligent chunking for massive tables: python from crawl4ai import LLMTableExtraction, LLMConfig Configure intelligent table extraction table strategy = LLMTableExtraction llm config=LLMConfig provider="openai/gpt-4.1-mini" , enable chunking=True, Handle massive tables chunk token threshold=5000, Smart chunking threshold overlap threshold=100, Maintain context between chunks extraction type="structured" Get structured data output config = CrawlerRunConfig table extraction strategy=table strategy result = await crawler.arun "https://complex-tables-site.com", config=config Tables are automatically chunked, processed, and merged for table in result.tables: print f"Extracted table: {len table 'data' } rows" - ⚡ Dispatcher Bug Fix : Fixed sequential processing bottleneck in arun many for fast-completing tasks - 🧹 Memory Management Refactor : Consolidated memory utilities into main utils module for cleaner architecture - 🔧 Browser Manager Fixes : Resolved race conditions in concurrent page creation with thread-safe locking - 🔗 Advanced URL Processing : Better handling of raw:// URLs and base tag link resolution - 🛡️ Enhanced Proxy Support : Flexible proxy configuration supporting both dict and string formats Version 0.7.3 Release Highlights - The Multi-Config Intelligence Update - 🕵️ Undetected Browser Support : Bypass sophisticated bot detection systems: python from crawl4ai import AsyncWebCrawler, BrowserConfig browser config = BrowserConfig browser type="undetected", Use undetected Chrome headless=True, Can run headless with stealth extra args= "--disable-blink-features=AutomationControlled", "--disable-web-security" async with AsyncWebCrawler config=browser config as crawler: result = await crawler.arun "https://protected-site.com" Successfully bypass Cloudflare, Akamai, and custom bot detection - 🎨 Multi-URL Configuration : Different strategies for different URL patterns in one batch: python from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode configs = Documentation sites - aggressive caching CrawlerRunConfig url matcher= " docs ", " documentation " , cache mode=CacheMode.WRITE ONLY, markdown generator options={"include links": True} , News/blog sites - fresh content CrawlerRunConfig url matcher=lambda url: 'blog' in url or 'news' in url, cache mode=CacheMode.BYPASS , Fallback for everything else CrawlerRunConfig results = await crawler.arun many urls, config=configs Each URL gets the perfect configuration automatically - 🧠 Memory Monitoring : Track and optimize memory usage during crawling: python from crawl4ai.memory utils import MemoryMonitor monitor = MemoryMonitor monitor.start monitoring results = await crawler.arun many large url list report = monitor.get report print f"Peak memory: {report 'peak mb' :.1f} MB" print f"Efficiency: {report 'efficiency' :.1f}%" Get optimization recommendations - 📊 Enhanced Table Extraction : Direct DataFrame conversion from web tables: result = await crawler.arun "https://site-with-tables.com" New way - direct table access if result.tables: import pandas as pd for table in result.tables: df = pd.DataFrame table 'data' print f"Table: {df.shape 0 } rows × {df.shape 1 } columns" - 💰 GitHub Sponsors : 4-tier sponsorship system for project sustainability - 🐳 Docker LLM Flexibility : Configure providers via environment variables Version 0.7.0 Release Highlights - The Adaptive Intelligence Update - 🧠 Adaptive Crawling : Your crawler now learns and adapts to website patterns automatically: config = AdaptiveConfig confidence threshold=0.7, Min confidence to stop crawling max depth=5, Maximum crawl depth max pages=20, Maximum number of pages to crawl strategy="statistical" async with AsyncWebCrawler as crawler: adaptive crawler = AdaptiveCrawler crawler, config state = await adaptive crawler.digest start url="https://news.example.com", query="latest news content" Crawler learns patterns and improves extraction over time - 🌊 Virtual Scroll Support : Complete content extraction from infinite scroll pages: scroll config = VirtualScrollConfig container selector=" data-testid='feed' ", scroll count=20, scroll by="container height", wait after scroll=1.0 result = await crawler.arun url, config=CrawlerRunConfig virtual scroll config=scroll config - 🔗 Intelligent Link Analysis : 3-layer scoring system for smart link prioritization: link config = LinkPreviewConfig query="machine learning tutorials", score threshold=0.3, concurrent requests=10 result = await crawler.arun url, config=CrawlerRunConfig link preview config=link config, score links=True Links ranked by relevance and quality - 🎣 Async URL Seeder : Discover thousands of URLs in seconds: seeder = AsyncUrlSeeder SeedingConfig source="sitemap+cc", pattern=" /blog/ ", query="python tutorials", score threshold=0.4 urls = await seeder.discover "https://example.com" - ⚡ Performance Boost : Up to 3x faster with optimized resource handling and memory efficiency Read the full details in our 0.7.0 Release Notes https://docs.crawl4ai.com/blog/release-v0.7.0 or check the CHANGELOG https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md . Crawl4AI follows standard Python version numbering conventions PEP 440 to help users understand the stability and features of each release. 📈 Version Numbers Explained Our version numbers follow this pattern: MAJOR.MINOR.PATCH e.g., 0.4.3 We use different suffixes to indicate development stages: dev 0.4.3dev1 : Development versions, unstable a 0.4.3a1 : Alpha releases, experimental features b 0.4.3b1 : Beta releases, feature complete but needs testing rc 0.4.3 : Release candidates, potential final version - Regular installation stable version : pip install -U crawl4ai - Install pre-release versions: pip install crawl4ai --pre - Install specific version: pip install crawl4ai==0.4.3b1 We use pre-releases to: - Test new features in real-world scenarios - Gather feedback before final releases - Ensure stability for production users - Allow early adopters to try new features For production environments, we recommend using the stable version. For testing new features, you can opt-in to pre-releases using the --pre flag. 🚨 Documentation Update Alert: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide For current documentation, including installation instructions, advanced features, and API reference, visit our Documentation Website https://docs.crawl4ai.com/ . To check our development plans and upcoming features, visit our Roadmap https://github.com/unclecode/crawl4ai/blob/main/ROADMAP.md . 📈 Development TODOs - 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction - 1. Question-Based Crawler: Natural language driven web discovery and content extraction - 2. Knowledge-Optimal Crawler: Smart crawling that maximizes knowledge while minimizing data extraction - 3. Agentic Crawler: Autonomous system for complex multi-step crawling operations - 4. Automated Schema Generator: Convert natural language to extraction schemas - 5. Domain-Specific Scrapers: Pre-configured extractors for common platforms academic, e-commerce - 6. Web Embedding Index: Semantic search infrastructure for crawled content - 7. Interactive Playground: Web UI for testing, comparing strategies with AI assistance - 8. Performance Monitor: Real-time insights into crawler operations - 9. Cloud Integration: One-click deployment solutions across cloud providers - 10. Sponsorship Program: Structured support system with tiered benefits - 11. Educational Content: "How to Crawl" video series and interactive tutorials We welcome contributions from the open-source community. Check out our contribution guidelines https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTORS.md for more information. I'll help modify the license section with badges. For the halftone effect, here's a version with it: Here's the updated license section: This project is licensed under the Apache License 2.0, attribution is recommended via the badges below. See the Apache 2.0 License https://github.com/unclecode/crawl4ai/blob/main/LICENSE file for details. When using Crawl4AI, you must include one of the following attribution methods: 📈 1. Badge Attribution Recommended Add one of these badges to your README, documentation, or website: | Theme | Badge | |---|---| Disco Theme Animated | | Night Theme Dark with Neon | | Dark Theme Classic | | Light Theme Classic | HTML code for adding the badges: php < -- Disco Theme Animated -- <a href="https://github.com/unclecode/crawl4ai" <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-disco.svg" alt="Powered by Crawl4AI" width="200"/ </a < -- Night Theme Dark with Neon -- <a href="https://github.com/unclecode/crawl4ai" <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-night.svg" alt="Powered by Crawl4AI" width="200"/ </a < -- Dark Theme Classic -- <a href="https://github.com/unclecode/crawl4ai" <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-dark.svg" alt="Powered by Crawl4AI" width="200"/ </a < -- Light Theme Classic -- <a href="https://github.com/unclecode/crawl4ai" <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-light.svg" alt="Powered by Crawl4AI" width="200"/ </a < -- Simple Shield Badge -- <a href="https://github.com/unclecode/crawl4ai" <img src="https://img.shields.io/badge/Powered%20by-Crawl4AI-blue?style=flat-square" alt="Powered by Crawl4AI"/ </a 📖 2. Text Attribution Add this line to your documentation: This project uses Crawl4AI https://github.com/unclecode/crawl4ai https://github.com/unclecode/crawl4ai for web data extraction. If you use Crawl4AI in your research or project, please cite: @software{crawl4ai2024, author = {UncleCode}, title = {Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper}, year = {2024}, publisher = {GitHub}, journal = {GitHub Repository}, howpublished = {\url{https://github.com/unclecode/crawl4ai}}, commit = {Please use the commit hash you're working with} } Text citation format: UncleCode. 2024 . Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper Computer software . GitHub. https://github.com/unclecode/crawl4ai For questions, suggestions, or feedback, feel free to reach out: - GitHub: unclecode https://github.com/unclecode - Twitter: @unclecode https://twitter.com/unclecode - Website: crawl4ai.com https://crawl4ai.com Happy Crawling 🕸️🚀 Our mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy. We envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement. 🔑 Key Opportunities Data Capitalization : Transform digital footprints into measurable, valuable assets. Authentic AI Data : Provide AI systems with real human insights. Shared Economy : Create a fair data marketplace that benefits data creators. 🚀 Development Pathway Open-Source Tools : Community-driven platforms for transparent data extraction. Digital Asset Structuring : Tools to organize and value digital knowledge. Ethical Data Marketplace : A secure, fair platform for exchanging structured data. For more details, see our full mission statement /unclecode/crawl4ai/blob/main/MISSION.md . These companies provide core infrastructure and technology that power Crawl4AI’s capabilities — from web access and proxy networks to AI tooling and data pipelines. Our enterprise sponsors support Crawl4AI and help scale it to power production-grade data pipelines. Interested in partnering with Crawl4AI? Whether you’re a proxy provider, AI infrastructure company, cloud platform, or an organization looking to support the Crawl4AI ecosystem, we’d love to hear from you. 📩 Contact: hello@crawl4ai.com mailto:hello@crawl4ai.com A heartfelt thanks to our individual supporters Every contribution helps us keep our opensource mission alive and thriving Want to join them? Sponsor Crawl4AI →