My agent was confidently wrong #
As of July 1, 2025, Cloudflare blocks AI crawlers by default on every new domain, and it has since turned away hundreds of billions of bot requests. Most of the coverage was about the big fight, AI labs against publishers over training data. That wasn’t my problem.
My problem was an agent that kept coming back confidently wrong. It would read a source, check a price, fetch a page I asked for, and give me an answer that sounded right and wasn’t. It was reading the block page instead of the real page: Cloudflare’s “Just a moment” screen, summarized back in the same confident voice it uses for everything else.
Nothing in the response said the fetch had failed. So I measured how often it happens, and we built something to fix it.
A 403 with a body is worse than a 403 with nothing #
The reason this is easy to miss is that nothing in the system thinks anything went wrong.
When you make a plain fetch()
, httpx.get()
, or requests.get()
to a bot-protected site, you usually don’t get an exception. You get a 403, sometimes even a 200, with a full HTML body attached: Cloudflare’s “Just a moment” screen, DataDome’s “please enable JavaScript,” PerimeterX’s “Press & Hold,” Amazon’s “Continue shopping” wall.
To your retry logic, that looks like success; the request finished. To a language model, the body is just text to read, so it reads it. Hand that 27KB block page to a model and ask for the jobs on it, and it will list them for you, none of which exist. The wrong answer is impossible to tell apart from the right one.
A loud failure, a timeout or a clean 429 with an empty body, would trigger a retry or a skip and you’d move on. The quiet block page slips through because it looks like content.
Twelve sites your agent can’t read right now #
On July 10 I sent an ordinary browser request to a set of popular targets to see which ones Cloudflare turns away. Twelve of the ones people ask about most came back with a hard 403:
| Use case | Sites returning a Cloudflare 403 |
|---|---|
| Hiring | Indeed, Glassdoor, ZipRecruiter |
| Sales and market research | Crunchbase, Product Hunt, Capterra |
| Freelance | Upwork, Fiverr |
| Commerce and consumer | StockX, DoorDash, Coinbase, Udemy |
If you are building for recruiting, sales research, lead generation, or shopping, you are already running into this. You probably won’t notice. The agent returns a normal-looking answer, built from a page it never actually read.
The benchmark #
Eight sites, one for each of the major anti-bot vendors, each hit three ways. Naive is curl
with an ordinary desktop Chrome user-agent, roughly what an untuned fetch
gets. Fortress is the open-source stealth browser we build. Tilion Cloud is the hosted layer on top of it.
| Site | Anti-bot | Naive | Fortress | Tilion Cloud |
|---|---|---|---|---|
| Indeed | Cloudflare | 403 | jobs | |
| Zillow | PerimeterX | 403 | 878 listings | |
| Walmart | Akamai | 307 | page | |
| StockX | Cloudflare | challenge | GraphQL API | |
| Booking | proprietary | 202 | app loaded | |
| Amazon | CloudFront WAF | 404 | click-wall | |
| Etsy | DataDome | 403 | DataDome | |
| G2 | DataDome | 403 | DataDome |
The naive fetch got real content zero times out of eight. It got a summarizable body eight times out of eight. And it got a signal the agent could use to know it had been blocked zero times out of eight. That last number is the one that made me write this. Nothing in the response says “you failed.”
The open-source Fortress build gets through five of the eight today: the Cloudflare, CloudFront, and PerimeterX targets. It returns clean structured data from Indeed, Zillow, and StockX. DataDome (Etsy, G2) and Amazon’s click-wall are the three it doesn’t clear, and those are the ones Tilion Cloud is built for.
Same URL, side by side #
Point both at indeed.com/jobs?q=software+engineer
. The naive fetch returns 27KB of “Just a moment. Enable JavaScript and cookies to continue.” Fortress returns the actual listings: Senior Software Developer, TherapyNotes, Remote, $110k to $135k; Principal Backend Engineer (AI), Cotiviti, $217k to $258k, full descriptions and all.
Point both at zillow.com/homes/for_sale/San-Francisco
. The naive fetch returns a 403, “Access to this page has been denied.” Fortress returns 878 live listings with price, beds and baths, sqft, address, and listing agent.
The agent on the naive path raises no error on either. It quietly summarizes the block page and moves on.
Getting through is a ladder #
It isn’t one trick. Which move you need depends on which system you are up against.
A single stealth fetch, waiting patiently through the JS challenge, is enough for Cloudflare, CloudFront, and PerimeterX. That covered Indeed, Zillow, and Walmart. Rendering the whole page to structured markdown picks up the SPA and JS-gated content. And when the good data lives behind the scenes, a recon pass drives the page and watches its private API traffic. On StockX that solved the Cloudflare challenge and caught POST /api/graphql
coming back 200, which is the site’s own pricing backend. That is better than scraping the HTML, because you get the real JSON.
DataDome and click-walls are the rung the open-source build doesn’t reach. DataDome closes the door before the app even boots, so no XHR ever fires and there is nothing to read. Amazon hides the product behind a click. That is the part that took the longest, and it is where Tilion Cloud comes in.
Tilion Cloud clears the hard tier #
Tilion Cloud is the hosted layer we run on top of Fortress. It keeps warm, persisted browser profiles around, matches the fingerprint and TLS of a real desktop Chrome, and handles the interstitials inline, which is what DataDome and Amazon’s click-wall are looking for. On the same eight sites, it clears all of them, Etsy and G2 included.
Fortress itself stays open source, and the five-of-eight result above runs on the public build.
The fix: fail loud, then use it #
Getting through matters. Noticing when you didn’t matters more. Before an agent embeds a page, cites a source, or compares a price, it needs one cheap check: is this a real page or a block screen? The challenge pages have signatures that stay put: Cloudflare’s “Just a moment,” DataDome’s “enable JavaScript,” PerimeterX’s “Press & Hold,” Amazon’s “Continue shopping.”
Fortress returns blocked: true
with empty text when it can’t get through, so your agent knows to stop instead of summarizing a challenge screen.
fortress.fetch_protected_page("https://www.zillow.com/homes/for_sale/San-Francisco_rb/")
fortress.fetch_protected_page("https://www.g2.com/products/notion/reviews")
To wire it into an agent, point it at the Fortress MCP. One install pulls the engine with it:
pip install "tilion[mcp]" # pulls the Fortress engine automatically
tilion-mcp # stdio transport (or: python -m tilion.mcp)
Then add the server to your MCP config in Claude Desktop, Cursor, Cline, or Windsurf:
{ "mcpServers": { "fortress": { "command": "tilion-mcp" } } }
Get Fortress #
Fortress is open source and free to run locally. Tilion Cloud, for the DataDome tier, is on the waitlist. Star the repo, and leave your email for Cloud.