If you've tried to give an LLM agent web access, you've hit this wall: point it at the real web and a plain fetch
breaks two ways.
<div id="root">
shell — the model gets nothing.403
or a Cloudflare "just a moment" page.I kept hitting this building agent workflows, so I open-sourced the tool I made:
HatFetch — an MIT-licensed MCP server that gives Claude, Cursor, or any MCP
client three tools (scrape
, crawl
, screenshot
) returning pages as clean Markdown.
Most tools either always spin up a heavy headless browser (slow) or never do (blocked). HatFetch escalates:
// add to your MCP client (Claude Desktop, Cursor, ...)
{ "mcpServers": { "hatfetch": { "command": "npx", "args": ["-y", "hatfetch"] } } }
Then ask your agent: "read and summarize it."
Being honest about what it can't do
No tool gets past 100% of anti-bot in 2026, and I won't pretend otherwise.
Aggressive Cloudflare / DataDome / PerimeterX still win sometimes. When they do, HatFetch returns an honest error instead of handing your model a CAPTCHA page as if it were content — that silent false-success is, IMO, the real sin in a lot of scraping tooling.
It's reliable on: server-rendered sites, JavaScript apps, geo-restricted pages, and IP-reputation / rate-limit blocks. It's a coin flip on the very hardest targets.
The pieces (all MIT)
Disclosure: I run ProxyHat, a residential proxy service, and there's a
first-class integration — but everything works with any HTTP proxy via
PROXY_URL, or none at all.
Feedback very welcome, especially on the escalation heuristics — when is it worth spending a browser render?