Cloud & InfraArticle
New per-purpose controls land on every plan, and September's defaults put bundled crawlers like Googlebot on notice.
A year after Cloudflare declared "Content Independence Day" and flipped AI crawler blocking on by default, it's back with the sequel — and this one matters more. The July 1 announcement replaces the one-click "Block AI Bots" switch with three independent controls — Search, Agent, and Training — available on every plan, including Free. And starting September 15, the defaults change: on pages that display ads, Training and Agent traffic gets blocked automatically, and multi-purpose crawlers that won't separate those behaviors get the most restrictive rule applied. That list includes Googlebot.
Read that last sentence again, because it's the actual story. Everything else in this release is sensible product iteration. That part is brinkmanship.
Why the blunt switch had to die #
The 2025 block was a hammer, and over a million customers swung it. It worked as a statement — no crawl without compensation — but it lumped together traffic that has nothing in common economically. A training scraper extracts value once and never comes back. An AI search crawler indexes you and can, at least in theory, send referrals. An agent fetching a page because a human asked ChatGPT to book a table or compare your pricing is arguably a customer, wearing a bot's user-agent string.
Blocking all three with one toggle meant e-commerce sites were turning away buying-intent agent traffic to stop training scrapers, and publishers courting AI search citations were invisible to answer engines. The new taxonomy fixes that: Search covers behavior that indexes content to answer questions later, Agent covers real-time action on a person's behalf (ChatGPT-User, or Claude and Gemini driving a browser), and Training covers crawls that feed model weights. You can now allow the first, meter the second, and block the third — which is roughly what most content businesses actually want.
This is the correct shape for the control plane. It's also two years late relative to how the traffic evolved, and the categories will blur at the edges: content ingested for an AI search index has a way of ending up in training corpora eventually, and Cloudflare's enforcement depends on classifying bots correctly at line speed. But directionally, purpose-based control beats user-agent whack-a-mole, and nobody else has the traffic position to make it stick.
The September defaults are aimed at Mountain View #
Here's the sharp edge. Google crawls with one bot for search indexing and — via its own opaque internal plumbing — feeds AI features from the same corpus. Google's crawler infrastructure offers Google-Extended as a training opt-out, but the core Googlebot crawl has always been bundled: refuse it and you vanish from search. That bundling is exactly what Cloudflare's "most restrictive applicable rule" targets. If a crawler mixes purposes and a site blocks Training, the whole crawler gets blocked — Googlebot, Applebot, BingBot, all named explicitly.
Cloudflare couldn't force crawler unbundling by asking nicely, so it's using default settings across the 20 percent of web domains behind its network as leverage. The ad-page heuristic is the tell: an ad, in Cloudflare's framing, is a signal the page was meant for human eyes, so that's where the human-displacing traffic gets cut first. Matthew Prince's line in the press release — now that the majority of web traffic is non-human, "we must act so a sustainable ecosystem emerges" — is the diplomatic version. The operational version is: split your crawler by purpose, or our customers' defaults will split you out entirely.
Whether Cloudflare holds that line is the thing to watch. Default-blocking Googlebot on ad-monetized pages is mutually assured destruction for its own customers' organic traffic, which is presumably why there's an opt-out window before September 15. My read: this is a negotiating posture more than a settled policy, and I'd expect either Google to ship purpose-separated crawling or Cloudflare to soften the default before the deadline. Either outcome is a win for site owners.
Quietly writing the standards nobody else shipped #
The less flashy pieces are Cloudflare doing what the IETF and the robots.txt lineage haven't managed at production scale. The Content Signals grammar in its managed robots.txt now carries a use
parameter — immediate
(interact, store nothing), reference
(index, excerpt, link back), full
(summarize and reproduce):
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
And there's an experiment in transitive trust built on RFC 7239's Forwarded
header — Forwarded: for="openai";use="reference"
— so an operator's identity and permissions survive passing through intermediaries. That's a real unsolved problem: agent traffic increasingly arrives via browser-automation clouds and proxy layers, and today a site owner's "allow OpenAI" decision evaporates the moment the request hops through one.
None of this is a ratified standard. It's standards-by-fiat, enforced by a single vendor's market position, with Verified-bot status as the carrot and de-listing as the stick — bots caught abusing signals lose verification, and bots that reproduce content in full can't get it at all. Purists should be uncomfortable; the IETF's AI-preferences work and efforts like RSL are the legitimate path. But robots.txt survived thirty years on voluntary compliance and died in two years of AI crawling. An honor system with a bouncer is still better than an honor system.
What to do before September 15 #
If you operate sites behind Cloudflare, don't let the defaults choose for you. Audit your zones' Security settings and AI Crawl Control now: decide per-property whether Agent traffic is a threat or a sales channel. A docs site probably wants agents in; a paywalled or ad-funded publication probably doesn't, unless payment flows through the Pay Per Crawl successor Cloudflare is now calling Pay Per Use. And if you run ads and rely on Google organic traffic, confirm exactly how the most-restrictive-rule default lands on your zone before the flip — this is not a setting to discover in an October traffic postmortem.
If you build agents or crawlers, the message is blunter: declare your purpose, verify your bot, and start emitting identity headers through your proxy chain, because anonymous scraping from residential IPs is now a strategy with a shrinking half-life and a growing invoice.
The web's crawl economics broke, no standards body moved fast enough, and a CDN with 20 percent of the web's domains decided that made it the regulator. That should bother you a little. It's also, right now, the only enforcement mechanism that works.
Sources & further reading #
Your site, your rules: new AI traffic options for all customers— blog.cloudflare.com - Cloudflare Allows the Agentic Internet to Flourish with a Simple Philosophy: Your Content, Your Rules— cloudflare.com - Cloudflare unveils per-purpose AI bot controls on all plans— gigazine.net - New options to manage AI traffic— developers.cloudflare.com
Ji-ho Choi· Security & Cloud Editor
Ji-ho covers the increasingly tangled overlap between cloud architecture and security, drawing on a background as a penetration tester to keep his reporting grounded in real-world attack paths. He never lets a vendor claim go unquestioned and insists that every buzzword come with a proof of concept.
Discussion 0 #
No comments yet
Be the first to weigh in.