{"slug": "how-to-give-claude-code-a-web-scraping-tool-with-crawl4ai-mcp", "title": "How to give Claude Code a web scraping tool with Crawl4AI (MCP)", "summary": "A walkthrough shows how to connect Crawl4AI, an open-source crawler that converts web pages into LLM-ready Markdown, to Claude Code through its MCP server. Users can either point Claude Code at the hosted MCP endpoint with a CRAWL4AI_KEY (free tier: 1,000 credits a month, 10 requests a minute, 2 concurrent) or run the open-source library and Docker server locally, exposing endpoints like /md, /crawl and /execute_js. The hosted API adds /scrape, /extract, /search, /serp and /answer, while the self-hosted server also supports MCP for coding agents.", "body_md": "Coding agents are great at research until they need to read a real web page. [Crawl4AI](https://github.com/unclecode/crawl4ai) is an open-source crawler and scraper that turns websites into clean, LLM-ready Markdown, and it ships an MCP server so Claude Code can scrape, extract and search on its own.\n\nThis guide covers both ways to use it: the hosted MCP server (one command in Claude Code) and the free open-source library you run yourself.\n\nSign up at crawl4ai.com and verify your email. Per the README, the free tier is 1,000 credits every month, 10 requests a minute and 2 at a time, with no card. Paid plans start at $10 a month (120 a minute, 10 at a time). Hosted requests are billed against your key, so keep `CRAWL4AI_KEY` out of shared scripts.\n\n```\nexport CRAWL4AI_KEY=\"your-key-here\"\nclaude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp \\\n  --header \"Authorization: Bearer $CRAWL4AI_KEY\"\n```\n\nThe README links setup for Codex, Cursor and OpenCode too.\n\nStart Claude Code and ask for a page in plain language:\n\n```\nUse crawl4ai to scrape https://docs.crawl4ai.com and summarize the installation options as a table.\n```\n\nThe same key gives you `/scrape` (page to Markdown), `/extract` (page to JSON, no LLM key of your own needed), `/search` (web search), `/serp` (Google's results page as data, in beta) and `/answer` (a direct answer with sources, experimental). JS-heavy pages and bot walls are handled for you on the hosted side.\n\n```\ncurl -s https://api.crawl4ai.com/scrape \\\n  -H \"Authorization: Bearer $CRAWL4AI_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"url\": \"https://news.ycombinator.com\"}' | jq -r .markdown\n```\n\nThe hosted API is optional. The open-source crawler runs free on your machine.\n\n```\npip install -U crawl4ai\ncrawl4ai-setup      # installs and sets up the browser\ncrawl4ai-doctor     # checks the installation\n```\n\nIf the browser setup fails, install Chromium by hand with `python -m playwright install --with-deps chromium`.\n\n``` python\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler\n\nasync def main():\n    async with AsyncWebCrawler() as crawler:\n        result = await crawler.arun(url=\"https://news.ycombinator.com\")\n        print(result.markdown)\n\nasyncio.run(main())\n```\n\nSave `result.markdown` per page and you have clean input for a RAG index. For tighter output, the \"fit Markdown\" filters (`PruningContentFilterLXML`, `BM25ContentFilter`, `LLMContentFilter`) strip menus, footers and boilerplate.\n\n```\ncrwl https://news.ycombinator.com -o markdown\ncrwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10\n```\n\nIf you want MCP without the hosted API, run your own server. It needs a token; without one it only answers inside its container.\n\n```\nexport CRAWL4AI_API_TOKEN=\"$(openssl rand -hex 32)\"\ndocker run -d -p 11235:11235 --name crawl4ai --shm-size=1g \\\n  -e CRAWL4AI_API_TOKEN=\"$CRAWL4AI_API_TOKEN\" \\\n  unclecode/crawl4ai:latest\n```\n\nAllow about 10 seconds for it to start, then test:\n\n```\ncurl -s http://localhost:11235/md \\\n  -H \"Authorization: Bearer $CRAWL4AI_API_TOKEN\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"url\": \"https://news.ycombinator.com\"}' | jq -r .markdown\n```\n\nThe server exposes `/md`, `/html`, `/crawl`, `/screenshot`, `/pdf` and `/execute_js`, plus a dashboard at `http://localhost:11235/dashboard` and a playground at `/playground`. The README says the self-hosted server also supports MCP for Claude Code and other agents; the exact connection steps are in the [self-hosting guide](https://docs.crawl4ai.com/core/self-hosting/).\n\n`/search`, `/serp` and `/answer`.\nWhichever you choose, respect each site's terms and robots rules when crawling.\n\nEach page has who it is for, the steps, and a prompt you can paste into your coding agent.\n\nUse case: [Give Claude Code a web scraping tool over MCP](https://dedupedai.com/use-cases/unclecode-crawl4ai/give-claude-code-web-scraping-tool/)\n\nUse case: [Turn any web page into clean Markdown for RAG](https://dedupedai.com/use-cases/unclecode-crawl4ai/turn-website-into-markdown-for-rag/)\n\nUse case: [Scrape a page with one curl call](https://dedupedai.com/use-cases/unclecode-crawl4ai/scrape-page-via-hosted-api/)\n\n**Get one email every Monday with the week's hottest AI repos and exactly how to use them: [subscribe free at dedupedai.com](https://dedupedai.com/join/?utm_source=devto&utm_medium=article&utm_campaign=howto_crawl4ai)**", "url": "https://wpnews.pro/news/how-to-give-claude-code-a-web-scraping-tool-with-crawl4ai-mcp", "canonical_source": "https://dev.to/dedupedai/how-to-give-claude-code-a-web-scraping-tool-with-crawl4ai-mcp-ege", "published_at": "2026-10-25 03:33:22+00:00", "updated_at": "2026-10-25 03:41:03.032796+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "agent-protocols", "ai-crawlers"], "entities": ["Crawl4AI", "Claude Code", "Anthropic", "unclecode", "Codex", "Cursor", "OpenCode", "Playwright"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-give-claude-code-a-web-scraping-tool-with-crawl4ai-mcp", "markdown": "https://wpnews.pro/news/how-to-give-claude-code-a-web-scraping-tool-with-crawl4ai-mcp.md", "text": "https://wpnews.pro/news/how-to-give-claude-code-a-web-scraping-tool-with-crawl4ai-mcp.txt", "jsonld": "https://wpnews.pro/news/how-to-give-claude-code-a-web-scraping-tool-with-crawl4ai-mcp.jsonld"}}