I built an open-source MCP server so AI agents can read the web - even sites that block bots

A developer open-sourced HatFetch, an MIT-licensed MCP server that gives AI agents tools to scrape, crawl, and screenshot web pages as clean Markdown, even on sites that block bots. The tool escalates from lightweight fetches to headless browser rendering when needed, and returns honest errors on aggressive anti-bot systems rather than silently returning CAPTCHA pages. The developer, who runs the residential proxy service ProxyHat, acknowledges that no tool bypasses all anti-bot measures in 2026.

If you've tried to give an LLM agent web access, you've hit this wall: point it at the real web and a plain fetch breaks two ways. <div id="root" shell — the model gets nothing. 403 or a Cloudflare "just a moment" page.I kept hitting this building agent workflows, so I open-sourced the tool I made: HatFetch — an MIT-licensed MCP server that gives Claude, Cursor, or any MCP client three tools scrape , crawl , screenshot returning pages as clean Markdown. Most tools either always spin up a heavy headless browser slow or never do blocked . HatFetch escalates: // add to your MCP client Claude Desktop, Cursor, ... { "mcpServers": { "hatfetch": { "command": "npx", "args": "-y", "hatfetch" } } } Then ask your agent: "read and summarize it." Being honest about what it can't do No tool gets past 100% of anti-bot in 2026, and I won't pretend otherwise. Aggressive Cloudflare / DataDome / PerimeterX still win sometimes. When they do, HatFetch returns an honest error instead of handing your model a CAPTCHA page as if it were content — that silent false-success is, IMO, the real sin in a lot of scraping tooling. It's reliable on: server-rendered sites, JavaScript apps, geo-restricted pages, and IP-reputation / rate-limit blocks. It's a coin flip on the very hardest targets. The pieces all MIT Disclosure: I run ProxyHat, a residential proxy service, and there's a first-class integration — but everything works with any HTTP proxy via PROXY URL, or none at all. Feedback very welcome, especially on the escalation heuristics — when is it worth spending a browser render?