{"slug": "show-hn-we-beat-cloudflare-s-bot-detection-open-source-stealth-browser", "title": "Show HN: We beat Cloudflare's bot detection (open-source stealth browser)", "summary": "An open-source stealth browser called Fortress successfully bypasses Cloudflare's bot detection on multiple high-traffic sites including Indeed, Zillow, and StockX, solving a problem where AI agents silently read block pages instead of real content. The project's creator found that naive HTTP requests to bot-protected sites return 403 or 200 status codes with full HTML bodies that language models summarize as if they were legitimate pages, producing confidently wrong answers without any failure signal.", "body_md": "## My agent was confidently wrong\n\nAs of July 1, 2025, Cloudflare blocks AI crawlers by default on every new domain, and it has since turned away hundreds of billions of bot requests. Most of the coverage was about the big fight, AI labs against publishers over training data. That wasn’t my problem.\n\nMy problem was an agent that kept coming back confidently wrong. It would read a source, check a price, fetch a page I asked for, and give me an answer that sounded right and wasn’t. It was reading the block page instead of the real page: Cloudflare’s “Just a moment” screen, summarized back in the same confident voice it uses for everything else.\n\nNothing in the response said the fetch had failed. So I measured how often it happens, and we built something to fix it.\n\n## A 403 with a body is worse than a 403 with nothing\n\nThe reason this is easy to miss is that nothing in the system thinks anything went wrong.\n\nWhen you make a plain `fetch()`\n\n, `httpx.get()`\n\n, or `requests.get()`\n\nto a bot-protected site, you usually don’t get an exception. You get a 403, sometimes even a 200, with a full HTML body attached: Cloudflare’s “Just a moment” screen, DataDome’s “please enable JavaScript,” PerimeterX’s “Press & Hold,” Amazon’s “Continue shopping” wall.\n\nTo your retry logic, that looks like success; the request finished. To a language model, the body is just text to read, so it reads it. Hand that 27KB block page to a model and ask for the jobs on it, and it will list them for you, none of which exist. The wrong answer is impossible to tell apart from the right one.\n\nA loud failure, a timeout or a clean 429 with an empty body, would trigger a retry or a skip and you’d move on. The quiet block page slips through because it looks like content.\n\n## Twelve sites your agent can’t read right now\n\nOn July 10 I sent an ordinary browser request to a set of popular targets to see which ones Cloudflare turns away. Twelve of the ones people ask about most came back with a hard 403:\n\n| Use case | Sites returning a Cloudflare 403 |\n|---|---|\n| Hiring | Indeed, Glassdoor, ZipRecruiter |\n| Sales and market research | Crunchbase, Product Hunt, Capterra |\n| Freelance | Upwork, Fiverr |\n| Commerce and consumer | StockX, DoorDash, Coinbase, Udemy |\n\nIf you are building for recruiting, sales research, lead generation, or shopping, you are already running into this. You probably won’t notice. The agent returns a normal-looking answer, built from a page it never actually read.\n\n## The benchmark\n\nEight sites, one for each of the major anti-bot vendors, each hit three ways. Naive is `curl`\n\nwith an ordinary desktop Chrome user-agent, roughly what an untuned `fetch`\n\ngets. Fortress is the open-source stealth browser we build. Tilion Cloud is the hosted layer on top of it.\n\n| Site | Anti-bot | Naive | Fortress | Tilion Cloud |\n|---|---|---|---|---|\n| Indeed | Cloudflare | 403 | jobs | |\n| Zillow | PerimeterX | 403 | 878 listings | |\n| Walmart | Akamai | 307 | page | |\n| StockX | Cloudflare | challenge | GraphQL API | |\n| Booking | proprietary | 202 | app loaded | |\n| Amazon | CloudFront WAF | 404 | click-wall | |\n| Etsy | DataDome | 403 | DataDome | |\n| G2 | DataDome | 403 | DataDome |\n\nThe naive fetch got real content zero times out of eight. It got a summarizable body eight times out of eight. And it got a signal the agent could use to know it had been blocked zero times out of eight. That last number is the one that made me write this. Nothing in the response says “you failed.”\n\nThe open-source Fortress build gets through five of the eight today: the Cloudflare, CloudFront, and PerimeterX targets. It returns clean structured data from Indeed, Zillow, and StockX. DataDome (Etsy, G2) and Amazon’s click-wall are the three it doesn’t clear, and those are the ones Tilion Cloud is built for.\n\n## Same URL, side by side\n\nPoint both at `indeed.com/jobs?q=software+engineer`\n\n. The naive fetch returns 27KB of “Just a moment. Enable JavaScript and cookies to continue.” Fortress returns the actual listings: Senior Software Developer, TherapyNotes, Remote, $110k to $135k; Principal Backend Engineer (AI), Cotiviti, $217k to $258k, full descriptions and all.\n\nPoint both at `zillow.com/homes/for_sale/San-Francisco`\n\n. The naive fetch returns a 403, “Access to this page has been denied.” Fortress returns 878 live listings with price, beds and baths, sqft, address, and listing agent.\n\nThe agent on the naive path raises no error on either. It quietly summarizes the block page and moves on.\n\n## Getting through is a ladder\n\nIt isn’t one trick. Which move you need depends on which system you are up against.\n\nA single stealth fetch, waiting patiently through the JS challenge, is enough for Cloudflare, CloudFront, and PerimeterX. That covered Indeed, Zillow, and Walmart. Rendering the whole page to structured markdown picks up the SPA and JS-gated content. And when the good data lives behind the scenes, a recon pass drives the page and watches its private API traffic. On StockX that solved the Cloudflare challenge and caught `POST /api/graphql`\n\ncoming back 200, which is the site’s own pricing backend. That is better than scraping the HTML, because you get the real JSON.\n\nDataDome and click-walls are the rung the open-source build doesn’t reach. DataDome closes the door before the app even boots, so no XHR ever fires and there is nothing to read. Amazon hides the product behind a click. That is the part that took the longest, and it is where Tilion Cloud comes in.\n\n## Tilion Cloud clears the hard tier\n\nTilion Cloud is the hosted layer we run on top of Fortress. It keeps warm, persisted browser profiles around, matches the fingerprint and TLS of a real desktop Chrome, and handles the interstitials inline, which is what DataDome and Amazon’s click-wall are looking for. On the same eight sites, it clears all of them, Etsy and G2 included.\n\nFortress itself stays open source, and the five-of-eight result above runs on the public build.\n\n## The fix: fail loud, then use it\n\nGetting through matters. Noticing when you didn’t matters more. Before an agent embeds a page, cites a source, or compares a price, it needs one cheap check: is this a real page or a block screen? The challenge pages have signatures that stay put: Cloudflare’s “Just a moment,” DataDome’s “enable JavaScript,” PerimeterX’s “Press & Hold,” Amazon’s “Continue shopping.”\n\nFortress returns `blocked: true`\n\nwith empty text when it can’t get through, so your agent knows to stop instead of summarizing a challenge screen.\n\n```\nfortress.fetch_protected_page(\"https://www.zillow.com/homes/for_sale/San-Francisco_rb/\")\n# { blocked: False, title: \"San Francisco CA Homes For Sale\",\n#   text: \"$1,595,000, 2bd 2ba, 466 19th Ave ... (878 listings)\" }\n\nfortress.fetch_protected_page(\"https://www.g2.com/products/notion/reviews\")\n# { blocked: True, text: \"\" }   # it says so, instead of handing you the wall\n```\n\nTo wire it into an agent, point it at the Fortress MCP. One install pulls the engine with it:\n\n```\npip install \"tilion[mcp]\"   # pulls the Fortress engine automatically\ntilion-mcp                  # stdio transport (or: python -m tilion.mcp)\n```\n\nThen add the server to your MCP config in Claude Desktop, Cursor, Cline, or Windsurf:\n\n```\n{ \"mcpServers\": { \"fortress\": { \"command\": \"tilion-mcp\" } } }\n```\n\n## Get Fortress\n\nFortress is open source and free to run locally. Tilion Cloud, for the DataDome tier, is on the waitlist. Star the repo, and leave your email for Cloud.\n\n[GitHub](https://github.com/tiliondev/fortress)", "url": "https://wpnews.pro/news/show-hn-we-beat-cloudflare-s-bot-detection-open-source-stealth-browser", "canonical_source": "https://tilion.dev/blog/cloudflare-blocks-agents", "published_at": "2026-07-11 00:26:16+00:00", "updated_at": "2026-07-11 01:05:14.858431+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": ["Cloudflare", "Indeed", "Zillow", "StockX", "DataDome", "PerimeterX", "Amazon", "Fortress"], "alternates": {"html": "https://wpnews.pro/news/show-hn-we-beat-cloudflare-s-bot-detection-open-source-stealth-browser", "markdown": "https://wpnews.pro/news/show-hn-we-beat-cloudflare-s-bot-detection-open-source-stealth-browser.md", "text": "https://wpnews.pro/news/show-hn-we-beat-cloudflare-s-bot-detection-open-source-stealth-browser.txt", "jsonld": "https://wpnews.pro/news/show-hn-we-beat-cloudflare-s-bot-detection-open-source-stealth-browser.jsonld"}}