Cloudflare's September 15 default change gives AI crawler operators a blunt choice: separate search from training and agent traffic, or lose automatic access to ad-supported pages on new and free sites.
Cloudflare isn't waiting for AI companies to tidy up their crawler behavior on their own. Starting September 15, 2026, the company will block Training and Agent crawlers by default on ad-supported pages. That covers new domains joining Cloudflare and new sites added by existing customers - plus free-tier customers who haven't opted out.
That date matters. September 15 is still ahead, not already here. So the real story isn't a deadline that has arrived this week. It's the warning Cloudflare has put in front of AI companies, publishers and site owners while they still have time to change settings, split crawlers or accept the block.
Cloudflare's scale is what gives the rule weight. In a July 1 announcement, the company said more than 20% of the web sits behind its network. That includes 36% of the world's most-visited websites and more than 40% of the Fortune 500. If you run a site that depends on ads, that is the kind of middleman you suddenly care about.
The new system divides automated traffic into three uses: Search, Agent and Training. Search stays allowed by default because it still sends people back to websites. Training and Agent access do not. A crawler that mixes search indexing with model training gets treated according to its most restrictive behavior, which means it can be blocked when the site owner blocks Training.
[Bots now generate more web traffic than humans and startups are about to feel it](https://startupfortune.com/bots-now-generate-more-web-traffic-than-humans-and-startups-are-about-to-feel-it/)
For the first time in internet history, bots now generate more web traffic than humans, with automated requests hitting 57.4% of all HTTP traffic by mid-2026. Agentic AI traffic grew 7,851% year over year, according to HUMAN Security's 2026 benchmark report, arriving 18 months ahead of Cloudflare CEO Matthew Prince's own forecast. The shift... - [bot traffic inflating website metrics](https://startupfortune.com/bots-now-generate-more-web-traffic-than-humans-and-startups-are-about-to-feel-it/) - [how to detect fake bot traffic](https://startupfortune.com/bots-now-generate-more-web-traffic-than-humans-and-startups-are-about-to-feel-it/)
That's the pressure point.
Cloudflare's own documentation names Googlebot, Applebot and BingBot as examples of multi-purpose crawlers that can be affected by the stricter setting. The company isn't saying search should disappear. It is saying one crawler shouldn't ask for the privileges of search while quietly doing another job at the same time.
Cloudflare CEO Matthew Prince put the point plainly in the company's press release. "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," he said. Cloudflare's report says automated agents and bots now drive more than half of web requests, while 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025.
Those aren't small shifts. They change the bargain.
The Toll Booth Comes With A Market #
The blocking rule is only one part of Cloudflare's move. The company is also trying to turn last year's Pay Per Crawl idea into Pay Per Use, where publishers can be paid when their material helps form an AI answer, not only when a bot fetches a page.
That difference is worth taking seriously. A crawler can fetch a page once and have the answer appear many times. Or it can fetch a page over and over and never use it at all. Cloudflare says Ceramic.ai is testing a pay-per-query model: participating publishers get paid when their content appears in Ceramic search results. You.com is doing something similar. It lets agents pay on demand for specific premium content mid-task.
Anna Patterson, Ceramic.ai's founder and CEO, said in Cloudflare's announcement that the partnership lets Ceramic bring its pay-per-query model to millions of content owners on Cloudflare's network. That's the pitch. The reporting is just as important for publishers: Cloudflare says participating content owners can see the queries that surfaced their pages and the webpage and snippet involved. They also get their average ranking position in search results.
Cloudflare Starts Charging AI Companies for the Web Data They Once Took Free Cloudflare has moved from tracking AI crawlers to blocking them by default on ad-supported pages starting September 15, 2026, while shifting its payment model from per-crawl fees toward paying publishers only when their content shows up in a generated answer. The shift turns years of free AI scraping into a metered cost, backed by Cloudflare data... - cloudflare charging AI companies for web - how AI crawlers access published web
If you publish for a living, you know why that matters. AI search has made many sites visible in answers while starving them of the click that used to pay for the work. A dashboard won't solve that by itself, but it gives publishers something better than guessing who took what.
Google Is The Hardest Case #
Cloudflare's sharpest example is Google. In its July report, the company said Google has access to about twice as much information as leading AI companies because it uses a mixed-use crawler that makes it hard for site owners to stay in Google's search ecosystem without also taking part in Google's AI ecosystem.
Google has argued before that site owners can use Google-Extended to opt out of some AI training and product uses without leaving Google Search. That doesn't fully answer Cloudflare's complaint. Search now includes AI Overviews and AI Mode, and the old line between sending readers to a page and answering them on the results page is getting harder to see.
Robots.txt has been the web's polite agreement for roughly 30 years. It still matters, but it has always depended on crawler operators choosing to obey it. Cloudflare's version sits closer to the pipe. A bot can ignore a text file. It has a harder time ignoring a network-level block across a large slice of the web.
Don't overstate it. A blocked crawler can look for other routes, use new infrastructure or focus on sites outside Cloudflare. But Cloudflare has given publishers a practical lever they didn't have before, and it has put AI companies on notice before September 15: if your crawler has several jobs, say so clearly or expect the strictest rule to apply.
Also read: Google Ships Gemini 3.8 Flash, Its Third Flash Model in Six Weeks • New York City Schools Ban AI Chatbots for 600,000 Young Students • CrowdStrike and the DOJ Dismantled the Sality Botnet That Stole Crypto for Years