How to give Claude Code a web scraping tool with Crawl4AI (MCP)

A walkthrough shows how to connect Crawl4AI, an open-source crawler that converts web pages into LLM-ready Markdown, to Claude Code through its MCP server. Users can either point Claude Code at the hosted MCP endpoint with a CRAWL4AI_KEY (free tier: 1,000 credits a month, 10 requests a minute, 2 concurrent) or run the open-source library and Docker server locally, exposing endpoints like /md, /crawl and /execute_js. The hosted API adds /scrape, /extract, /search, /serp and /answer, while the self-hosted server also supports MCP for coding agents.

Coding agents are great at research until they need to read a real web page. Crawl4AI https://github.com/unclecode/crawl4ai is an open-source crawler and scraper that turns websites into clean, LLM-ready Markdown, and it ships an MCP server so Claude Code can scrape, extract and search on its own. This guide covers both ways to use it: the hosted MCP server one command in Claude Code and the free open-source library you run yourself. Sign up at crawl4ai.com and verify your email. Per the README, the free tier is 1,000 credits every month, 10 requests a minute and 2 at a time, with no card. Paid plans start at $10 a month 120 a minute, 10 at a time . Hosted requests are billed against your key, so keep CRAWL4AI KEY out of shared scripts. export CRAWL4AI KEY="your-key-here" claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp \ --header "Authorization: Bearer $CRAWL4AI KEY" The README links setup for Codex, Cursor and OpenCode too. Start Claude Code and ask for a page in plain language: Use crawl4ai to scrape https://docs.crawl4ai.com and summarize the installation options as a table. The same key gives you /scrape page to Markdown , /extract page to JSON, no LLM key of your own needed , /search web search , /serp Google's results page as data, in beta and /answer a direct answer with sources, experimental . JS-heavy pages and bot walls are handled for you on the hosted side. curl -s https://api.crawl4ai.com/scrape \ -H "Authorization: Bearer $CRAWL4AI KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown The hosted API is optional. The open-source crawler runs free on your machine. pip install -U crawl4ai crawl4ai-setup installs and sets up the browser crawl4ai-doctor checks the installation If the browser setup fails, install Chromium by hand with python -m playwright install --with-deps chromium . python import asyncio from crawl4ai import AsyncWebCrawler async def main : async with AsyncWebCrawler as crawler: result = await crawler.arun url="https://news.ycombinator.com" print result.markdown asyncio.run main Save result.markdown per page and you have clean input for a RAG index. For tighter output, the "fit Markdown" filters PruningContentFilterLXML , BM25ContentFilter , LLMContentFilter strip menus, footers and boilerplate. crwl https://news.ycombinator.com -o markdown crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10 If you want MCP without the hosted API, run your own server. It needs a token; without one it only answers inside its container. export CRAWL4AI API TOKEN="$ openssl rand -hex 32 " docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g \ -e CRAWL4AI API TOKEN="$CRAWL4AI API TOKEN" \ unclecode/crawl4ai:latest Allow about 10 seconds for it to start, then test: curl -s http://localhost:11235/md \ -H "Authorization: Bearer $CRAWL4AI API TOKEN" \ -H "Content-Type: application/json" \ -d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown The server exposes /md , /html , /crawl , /screenshot , /pdf and /execute js , plus a dashboard at http://localhost:11235/dashboard and a playground at /playground . The README says the self-hosted server also supports MCP for Claude Code and other agents; the exact connection steps are in the self-hosting guide https://docs.crawl4ai.com/core/self-hosting/ . /search , /serp and /answer . Whichever you choose, respect each site's terms and robots rules when crawling. Each page has who it is for, the steps, and a prompt you can paste into your coding agent. Use case: Give Claude Code a web scraping tool over MCP https://dedupedai.com/use-cases/unclecode-crawl4ai/give-claude-code-web-scraping-tool/ Use case: Turn any web page into clean Markdown for RAG https://dedupedai.com/use-cases/unclecode-crawl4ai/turn-website-into-markdown-for-rag/ Use case: Scrape a page with one curl call https://dedupedai.com/use-cases/unclecode-crawl4ai/scrape-page-via-hosted-api/ Get one email every Monday with the week's hottest AI repos and exactly how to use them: subscribe free at dedupedai.com https://dedupedai.com/join/?utm source=devto&utm medium=article&utm campaign=howto crawl4ai