Websites are locking AI crawlers out as agent bot traffic surges in 2026

A robots.txt analysis of the internet's top 2,000 domains, published on Dev.to and run on August 2, 2026, found that 292 of 1,530 domains with a reachable robots.txt file — 19.1% — now disallow at least one of the four major AI crawlers: GPTBot, ClaudeBot, Google-Extended or PerplexityBot, with Amazon, Instagram, X, Pinterest and TikTok blocking all five major AI crawlers including Bytespider. Cloudflare is making the shift a default, blocking bots it classifies as "Training" or "Agent" on any ad-carrying page for new domains onboarding from September 15, 2026, while search indexing bots stay welcome. Cloudflare Radar data shows Anthropic's ClaudeBot crawled 23,951 pages per referral in the first quarter of 2026 (a separate measurement put it at 20,583:1), falling to 1,917:1 by July, while OpenAI's GPTBot went from 1,276:1 in January-to-March to roughly 251:1 by July, as bots now account for 57.4% of web traffic versus 42.6% from humans.

AI crawlers now account for over a fifth of web traffic, but they send almost nothing back. Site owners have stopped tolerating the trade. Amazon blocks GPTBot, ClaudeBot, Google-Extended, PerplexityBot and Bytespider, all five of the major AI crawlers, according to a robots.txt analysis covering the internet's top 2,000 domains. Instagram, X, Pinterest and TikTok sit alongside it on the list of platforms shutting the door entirely. The study, published on Dev.to and run on August 2, 2026, found that of 1,530 domains with a reachable robots.txt file, 292, or 19.1%, now disallow at least one of the four major AI crawlers: GPTBot, ClaudeBot, Google-Extended or PerplexityBot. That is a sharp jump from a year earlier, when blocking was still a fringe move rather than a default setting. Cloudflare made the shift official rather than optional. Starting September 15, 2026, any new domain onboarding to Cloudflare has bots it classifies as "Training" or "Agent" blocked by default on any page that carries advertising, while bots doing search indexing stay welcome. The company extended that default to all existing free-tier customers and to new sites from existing customers too. Cloudflare sorts crawlers into three buckets: Search, for indexing; Training, for scraping data to build models; and Agent, for bots acting on a user's behalf in real time. Get flagged as the wrong one on an ad-funded page, and you're out. Here's the part that should worry OpenAI, Anthropic and Perplexity more than the blocklists themselves: the crawlers are taking far more than they're giving back. According to Cloudflare's own Radar data, Anthropic's ClaudeBot crawled 23,951 pages for every single referral it sent back to a publisher in the first quarter of 2026. A separate measurement for that same period put the figure at 20,583:1. By July, Cloudflare's numbers had ClaudeBot's ratio down to 1,917:1, still a landslide compared with what a referral used to look like from a search engine. OpenAI's GPTBot fares better but tells the same story: a 1,276:1 ratio in the January-to-March window, improving to roughly 251:1 by July, up from 1,104:1 a year earlier. Google's crawlers, and Microsoft's and Apple's, complicate the picture further, because Googlebot, Applebot and Bingbot are multi-purpose. Cloudflare applies the most restrictive rule a site sets to any crawler that serves more than one function, so a site trying to block AI training can end up blocking Googlebot's search indexing in the same stroke. Ad-supported publishers have a simple arithmetic problem. A crawler that visits thousands of times to extract training data, or to answer a live agent query, and almost never sends a human back to click an ad, is a cost with no revenue attached. That calculus didn't exist when the main bots on the web were Googlebot and Bingbot, both of which drove clicks in return for being let in. AI crawlers broke that bargain, and publishers are now writing the correction into their robots.txt files one disallow line at a time. The Dev.to study found the decision is rarely made crawler by crawler: 75% of the sites that block anything block three or more of the four major bots at once, which suggests editors and engineering teams are setting policy in one move rather than weighing each AI company's crawler on its own behavior. Cloudflare's Matthew Prince Says Bots Now Outnumber Humans Online https://startupfortune.com/cloudflares-matthew-prince-says-bots-now-outnumber-humans-online/ Cloudflare Radar data shows bots now account for 57.4% of web traffic versus 42.6% from humans, arriving nearly two years ahead of CEO Matthew Prince's own forecast. Prince told Fortune the web's future is "pay to crawl," and Cloudflare has already rolled out a Disallow AI Training setting letting sites block AI scrapers while staying searchable. - how bots now outnumber humans on the internet https://startupfortune.com/cloudflares-matthew-prince-says-bots-now-outnumber-humans-online/ - should AI companies pay for training data https://startupfortune.com/cloudflares-matthew-prince-says-bots-now-outnumber-humans-online/ When sites are selective, GPTBot draws the most singling-out: 19 sites blocked only GPTBot, compared with eight for ClaudeBot, five for PerplexityBot and three for Google-Extended. That's a reputational tax OpenAI is paying on top of the technical one. None of this stops a determined scraper. Robots.txt is a request, not a lock, and a crawler that ignores it faces no automatic penalty beyond a website operator's annoyance and, increasingly, a Cloudflare-enforced block at the network layer. That is the real teeth behind the September 15 change: Cloudflare isn't asking bots to behave, it's refusing their requests outright on the pages it controls. For a company like Anthropic that depends on live web access to power Claude's browsing and agent features, a blanket Cloudflare default aimed at "Agent" traffic is a different problem than a training-data fight. It's the difference between a model that can't learn from a page and a product that can't load one. The open web spent two decades optimizing for one kind of visitor: a person who might click something. AI crawlers don't click, they extract, and in 2026 the sites that make their money from clicks are the first to say no. Also read: Nvidia says the RTX 5090 is not dead but admits supply is tight https://startupfortune.com/nvidia-says-the-rtx-5090-is-not-dead-but-admits-supply-is-tight/ • Wall Street is finally asking whether the AI boom can pay for itself https://startupfortune.com/wall-street-is-finally-asking-whether-the-ai-boom-can-pay-for-itself/ • DeepSeek Ties Its New V4.1-Flash-1017 Model Tighter to Its Own Harness https://startupfortune.com/deepseek-ties-its-new-v41-flash-1017-model-tighter-to-its-own-harness/ This article is posted in Technology News https://startupfortune.com/category/technology/ , check it out for more related stories. Join the discussion Open in the community → https://startupfortune.com/community/ Almost there. Sign in and your reply posts straight away.