AIArticle
Four June security releases and a breaking 0.9.0 turned the open-source LLM crawler's Docker API into a credible Firecrawl alternative.
Crawl4AI is back on GitHub's trending page, which happens every few months to a project with around 80,000 stars. What's changed since the last time is the reason to look again. In June, the maintainers shipped four security releases in under three weeks, ending in a breaking 0.9.0 that turned the Docker server from "open by default" into "locked down unless you say otherwise." That's the version you should be running, and the path to it says a lot about where self-hosted AI crawling stands.
What it does that Scrapy doesn't #
The pitch has been the same since 2024: point it at a URL, get back Markdown an LLM can read without you paying for tokens spent on nav bars and cookie banners. It runs Playwright under asyncio, so JavaScript-heavy pages render properly, and it gives you two Markdown variants per page. raw_markdown
is the whole page. fit_markdown
is the page after a content filter, and it's empty until you attach one. PruningContentFilter
scores DOM nodes heuristically and drops the low-value ones; ScrapingBee measured a 62% character cut on a docs homepage with it. BM25ContentFilter
keeps the chunks that match a query, which is the right tool when you're feeding a specific question into a RAG pipeline rather than archiving a site.
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
from crawl4ai.content_filter_strategy import PruningContentFilter
async def main():
md = DefaultMarkdownGenerator(
content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed")
)
cfg = CrawlerRunConfig(markdown_generator=md, cache_mode=CacheMode.BYPASS)
async with AsyncWebCrawler() as crawler:
r = await crawler.arun("https://docs.python.org/3/", config=cfg)
print(len(r.markdown.raw_markdown), len(r.markdown.fit_markdown))
asyncio.run(main())
Scrapy has been the Python default since 2008 and is still faster on static HTML, because it never boots a browser. Crawl4AI's bet is that the expensive part of an AI pipeline is no longer the fetch. It's the tokens you spend on junk afterward, and the engineering hours you spend writing per-site selectors. On that framing, a Playwright-based crawler that emits clean Markdown wins even when it's slower per page.
The feature most people haven't used yet #
Extraction comes in three tiers, and the order matters for your bill. JsonCssExtractionStrategy
is deterministic and free: give it a schema of CSS or XPath selectors and it returns JSON. Regex extraction handles emails, dates, and phone numbers without a model. LLMExtractionStrategy
takes a Pydantic model and lets any provider fill it in, including a local Ollama endpoint with api_token=None
. The sane workflow is to reach for CSS first, use the LLM strategy to bootstrap a schema on a messy page, then freeze that schema back into CSS so the recurring crawl costs nothing.
Adaptive crawling is the piece I'd point agent builders at. AdaptiveCrawler.digest(start_url=..., query=...)
follows links and stops when three scores, coverage, consistency, and saturation, cross a confidence_threshold
that defaults to 0.7. The default statistical strategy is term-based and runs offline; the embedding strategy needs an embedding_llm_config
and expands the query semantically. It's a bounded answer to a problem every agent framework has: an LLM tool call that says "research X" needs a crawler that knows when to quit, and max_pages
alone is a blunt instrument.
Deep crawling got the ops features it was missing this year. 0.8.0 added resume_state
and on_state_change
so a crashed BFS run picks up where it died, plus a two-phase prefetch mode the changelog credits with 5-10x faster URL discovery. 0.8.5 added Shadow DOM flattening and a three-tier anti-bot response that retries, escalates through a proxy chain, and falls back.
The Docker server grew up the hard way #
Crawl4AI ships as two products: a pip library and a Docker image that wraps it in a FastAPI server on port 11235, with /crawl
, /md
, /screenshot
, /pdf
, an async job queue, a monitoring dashboard, and MCP endpoints at /mcp/sse
and /mcp/ws
for Claude Code and other agent clients. The server is what made it a Firecrawl alternative. It was also, until June, a soft target.
The changelog for 0.8.7 (June 1) lists nine fixes, three of them CVSS 9.8: a sandbox escape in the computed-fields evaluator (CVE-2026-53753), an RCE through hook builtins, and a hardcoded JWT secret. Arbitrary file write via output_path
, SSRF through webhook URLs and crawl endpoints, and an unauthenticated /execute_js
round out the list. 0.8.8 and 0.8.9 (both June 4) closed an IPv6-mapped-IPv4 bypass of the SSRF blocklist and a second SSRF through proxy settings (CVE-2026-53755). 0.9.0 (June 18) then fixed another RCE via Chromium launch-argument injection (CVE-2026-57572) and a streaming-endpoint SSRF (CVE-2026-57573), and made the breaking changes: auth on by default, loopback-only binding until you set CRAWL4AI_API_TOKEN
, request-supplied js_code
, proxy_config
, and extra_args
rejected, Python hook strings replaced with declarative hooks, CORS deny-by-default, Redis password-protected.
None of this touched the pip library. If you import AsyncWebCrawler
in your own process, you were never exposed. But if you ran docker run -p 11235:11235 unclecode/crawl4ai
on a box with a public IP before June, treat it as compromised until proven otherwise; every one of those SSRF findings was unauthenticated, and the cloud metadata endpoint is exactly what they reach.
My read is that this is the maturity curve every "URL to Markdown" server was going to hit. Firecrawl absorbs the same risk class inside a managed fleet, and you pay for that: Apify's January comparison prices Firecrawl's 100k-page tier at $83 a month against roughly $70 of AWS for a self-hosted Crawl4AI doing the same volume. The dollar gap is small. What you're paying Firecrawl for is that someone else patches the SSRF filter; with Crawl4AI, that's you.
Where it fits, and where it doesn't #
Adopt the library if your pipeline is Python and you want Markdown generation, structured extraction, and deep crawling in-process with no vendor. The 0.9.2 patch (July 15) fixed a page leak in MemoryAdaptiveDispatcher
, so use that or newer for arun_many
workloads. Adopt the Docker server if you need a language-neutral endpoint or an MCP tool for agents, and set the token before you expose a port.
Don't adopt it as an unblocking service. It ships no managed proxies and won't guarantee a way past Cloudflare or DataDome; for those targets you're buying residential proxies or a commercial unblocker anyway, and the crawler is the cheap part. And don't expect a hosted convenience layer: there's no SaaS, no SDK for anything but Python, and the LangChain and LlamaIndex s are community-maintained.
The verdict: Crawl4AI is the best free way to turn a JavaScript-rendered page into LLM-ready Markdown, and after June it's finally safe to run as a server. Upgrade to 0.9.x, wire PruningContentFilter
into your default config, and push extraction toward CSS schemas before you push it toward a model.
Sources & further reading #
unclecode/crawl4ai— github.com - Crawl4AI CHANGELOG— github.com - Self-Hosting Guide - Crawl4AI Documentation— docs.crawl4ai.com - Adaptive Crawling - Crawl4AI Documentation— docs.crawl4ai.com - Crawl4AI vs. Firecrawl— blog.apify.com - The complete Crawl4AI guide for LLM-ready data and AI web crawling— scrapingbee.com - CVE-2026-53753: Crawl4AI Sandbox Escape RCE Vulnerability— sentinelone.com - CVE-2026-57573 - Unauthenticated SSRF - Crawl4AI Docker API Server prior to 0.9.0— ionix.io
Priya Nair· AI & Developer Experience Writer
Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.
Discussion 0 #
No comments yet
Be the first to weigh in.