{"slug": "i-built-an-open-source-mcp-server-so-ai-agents-can-read-the-web-even-sites-that", "title": "I built an open-source MCP server so AI agents can read the web - even sites that block bots", "summary": "A developer open-sourced HatFetch, an MIT-licensed MCP server that gives AI agents tools to scrape, crawl, and screenshot web pages as clean Markdown, even on sites that block bots. The tool escalates from lightweight fetches to headless browser rendering when needed, and returns honest errors on aggressive anti-bot systems rather than silently returning CAPTCHA pages. The developer, who runs the residential proxy service ProxyHat, acknowledges that no tool bypasses all anti-bot measures in 2026.", "body_md": "If you've tried to give an LLM agent web access, you've hit this wall: point it at the real web and a plain `fetch`\n\nbreaks two ways.\n\n`<div id=\"root\">`\n\nshell — the model gets nothing.`403`\n\nor a Cloudflare \"just a moment\" page.I kept hitting this building agent workflows, so I open-sourced the tool I made:\n\n**HatFetch** — an MIT-licensed MCP server that gives Claude, Cursor, or any MCP\n\nclient three tools (`scrape`\n\n, `crawl`\n\n, `screenshot`\n\n) returning pages as clean Markdown.\n\nMost tools either always spin up a heavy headless browser (slow) or never do (blocked). HatFetch escalates:\n\n```\n// add to your MCP client (Claude Desktop, Cursor, ...)\n{ \"mcpServers\": { \"hatfetch\": { \"command\": \"npx\", \"args\": [\"-y\", \"hatfetch\"] } } }\n```\n\nThen ask your agent: \"read and summarize it.\"\n\nBeing honest about what it can't do\n\nNo tool gets past 100% of anti-bot in 2026, and I won't pretend otherwise.\n\nAggressive Cloudflare / DataDome / PerimeterX still win sometimes. When they do, HatFetch returns an honest error instead of handing your model a CAPTCHA page as if it were content — that silent false-success is, IMO, the real sin in a lot of scraping tooling.\n\nIt's reliable on: server-rendered sites, JavaScript apps, geo-restricted pages, and IP-reputation / rate-limit blocks. It's a coin flip on the very hardest targets.\n\nThe pieces (all MIT)\n\nDisclosure: I run ProxyHat, a residential proxy service, and there's a\n\nfirst-class integration — but everything works with any HTTP proxy via\n\nPROXY_URL, or none at all.\n\nFeedback very welcome, especially on the escalation heuristics — when is it worth spending a browser render?", "url": "https://wpnews.pro/news/i-built-an-open-source-mcp-server-so-ai-agents-can-read-the-web-even-sites-that", "canonical_source": "https://dev.to/proxybrad/i-built-an-open-source-mcp-server-so-ai-agents-can-read-the-web-even-sites-that-block-bots-1oo", "published_at": "2026-07-13 13:40:23+00:00", "updated_at": "2026-07-13 13:46:21.078248+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["HatFetch", "ProxyHat", "Claude", "Cursor", "MCP", "Cloudflare", "DataDome", "PerimeterX"], "alternates": {"html": "https://wpnews.pro/news/i-built-an-open-source-mcp-server-so-ai-agents-can-read-the-web-even-sites-that", "markdown": "https://wpnews.pro/news/i-built-an-open-source-mcp-server-so-ai-agents-can-read-the-web-even-sites-that.md", "text": "https://wpnews.pro/news/i-built-an-open-source-mcp-server-so-ai-agents-can-read-the-web-even-sites-that.txt", "jsonld": "https://wpnews.pro/news/i-built-an-open-source-mcp-server-so-ai-agents-can-read-the-web-even-sites-that.jsonld"}}