Coding agents are great at research until they need to read a real web page. Crawl4AI is an open-source crawler and scraper that turns websites into clean, LLM-ready Markdown, and it ships an MCP server so Claude Code can scrape, extract and search on its own.
This guide covers both ways to use it: the hosted MCP server (one command in Claude Code) and the free open-source library you run yourself.
Sign up at crawl4ai.com and verify your email. Per the README, the free tier is 1,000 credits every month, 10 requests a minute and 2 at a time, with no card. Paid plans start at $10 a month (120 a minute, 10 at a time). Hosted requests are billed against your key, so keep CRAWL4AI_KEY out of shared scripts.
export CRAWL4AI_KEY="your-key-here"
claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp \
--header "Authorization: Bearer $CRAWL4AI_KEY"
The README links setup for Codex, Cursor and OpenCode too.
Start Claude Code and ask for a page in plain language:
Use crawl4ai to scrape https://docs.crawl4ai.com and summarize the installation options as a table.
The same key gives you /scrape (page to Markdown), /extract (page to JSON, no LLM key of your own needed), /search (web search), /serp (Google's results page as data, in beta) and /answer (a direct answer with sources, experimental). JS-heavy pages and bot walls are handled for you on the hosted side.
curl -s https://api.crawl4ai.com/scrape \
-H "Authorization: Bearer $CRAWL4AI_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown
The hosted API is optional. The open-source crawler runs free on your machine.
pip install -U crawl4ai
crawl4ai-setup # installs and sets up the browser
crawl4ai-doctor # checks the installation
If the browser setup fails, install Chromium by hand with python -m playwright install --with-deps chromium.
import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://news.ycombinator.com")
print(result.markdown)
asyncio.run(main())
Save result.markdown per page and you have clean input for a RAG index. For tighter output, the "fit Markdown" filters (PruningContentFilterLXML, BM25ContentFilter, LLMContentFilter) strip menus, footers and boilerplate.
crwl https://news.ycombinator.com -o markdown
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
If you want MCP without the hosted API, run your own server. It needs a token; without one it only answers inside its container.
export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)"
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g \
-e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \
unclecode/crawl4ai:latest
Allow about 10 seconds for it to start, then test:
curl -s http://localhost:11235/md \
-H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown
The server exposes /md, /html, /crawl, /screenshot, /pdf and /execute_js, plus a dashboard at http://localhost:11235/dashboard and a playground at /playground. The README says the self-hosted server also supports MCP for Claude Code and other agents; the exact connection steps are in the self-hosting guide.
/search, /serp and /answer.
Whichever you choose, respect each site's terms and robots rules when crawling.
Each page has who it is for, the steps, and a prompt you can paste into your coding agent.
Use case: Give Claude Code a web scraping tool over MCP
Use case: Turn any web page into clean Markdown for RAG
Use case: Scrape a page with one curl call
Get one email every Monday with the week's hottest AI repos and exactly how to use them: subscribe free at dedupedai.com