# I built an open-source MCP server so AI agents can read the web - even sites that block bots

> Source: <https://dev.to/proxybrad/i-built-an-open-source-mcp-server-so-ai-agents-can-read-the-web-even-sites-that-block-bots-1oo>
> Published: 2026-07-13 13:40:23+00:00

If you've tried to give an LLM agent web access, you've hit this wall: point it at the real web and a plain `fetch`

breaks two ways.

`<div id="root">`

shell — the model gets nothing.`403`

or a Cloudflare "just a moment" page.I kept hitting this building agent workflows, so I open-sourced the tool I made:

**HatFetch** — an MIT-licensed MCP server that gives Claude, Cursor, or any MCP

client three tools (`scrape`

, `crawl`

, `screenshot`

) returning pages as clean Markdown.

Most tools either always spin up a heavy headless browser (slow) or never do (blocked). HatFetch escalates:

```
// add to your MCP client (Claude Desktop, Cursor, ...)
{ "mcpServers": { "hatfetch": { "command": "npx", "args": ["-y", "hatfetch"] } } }
```

Then ask your agent: "read and summarize it."

Being honest about what it can't do

No tool gets past 100% of anti-bot in 2026, and I won't pretend otherwise.

Aggressive Cloudflare / DataDome / PerimeterX still win sometimes. When they do, HatFetch returns an honest error instead of handing your model a CAPTCHA page as if it were content — that silent false-success is, IMO, the real sin in a lot of scraping tooling.

It's reliable on: server-rendered sites, JavaScript apps, geo-restricted pages, and IP-reputation / rate-limit blocks. It's a coin flip on the very hardest targets.

The pieces (all MIT)

Disclosure: I run ProxyHat, a residential proxy service, and there's a

first-class integration — but everything works with any HTTP proxy via

PROXY_URL, or none at all.

Feedback very welcome, especially on the escalation heuristics — when is it worth spending a browser render?