Cloudflare finally lets you block AI training without nuking your search rankings

Cloudflare launched a "Disallow AI Training" setting that lets site owners block AI training crawlers while keeping their pages indexed for search, with Apple, Google, and Microsoft committing to honor the setting. Cloudflare reports that fewer than 1% of its sites block search bots while 17% actively try to block AI training, a conflict caused by mixed-use crawlers that serve both search and training. The company is classifying bots into Search, Training, and Agent categories and introduced an "Accountable" label requiring operators to offer training opt-outs, summary opt-outs, URL visibility, and a no-retaliation guarantee on search rankings.

Cloudflare finally lets you block AI training without nuking your search rankings Stop pretending that a robots.txt file is a magic shield. We all know it's basically a "please don't" sign that half the internet ignores. For the longest time, we were stuck in this hostage situation: either let the big tech giants scrape your soul for their next LLM, or block their crawlers and vanish from search results because they use the same bot for both. Cloudflare just dropped a "Disallow AI Training" setting that actually separates these two. Now you can stay indexed for search while telling the training bots to beat it. Apparently, Apple, Google, and Microsoft are actually playing ball and have committed to honoring this setting. Why is this a big deal? Because the math of the internet is broken. Less than 1% of Cloudflare sites block search bots—obviously, we all want the traffic—but 17% of sites are actively trying to block AI training. The problem is "mixed-use crawlers." If you block the bot to save your data from a training set, you accidentally delete your existence from the search index. Cloudflare is trying to fix this at the network level. Instead of hoping a bot reads your text file, the network identifies who is crawling and why, then blocks the ones that decide to ignore your preferences. They're tracking this on Radar so you can actually see who is lying to you. What does "Accountable" actually mean? Cloudflare created this "Accountable" label for bot operators who aren't just making vague promises. To get this badge, a company has to meet or commit to these specific requirements: - Opt-out of training: Provide a way to opt out via robots.txt or similar standards. - Opt-out of summaries: Give a way to opt out of AI summaries currently done directly with the operator, but Cloudflare says they'll let you do this through their dashboard by early next year . - URL visibility: Show exactly which pages were used for training and provide metrics on search appearances. - No retaliation: Guarantee that opting out of training won't tank your traditional search rankings. How the new security settings work Cloudflare is now classifying bots by their specific behavior rather than just their name. You can now toggle controls based on three distinct categories: - Search: Bots crawling to build an index. - Training: Bots crawling to fine-tune or train a model. - Agent: User-directed bots, like chat fetchers or browser-use agents, visiting on behalf of a human. Next Stop wasting time re-explaining your project history to Claude Code, Codex, and Cursor → https://promptcube3.com/en/threads/9428/ All Replies （3） I'm curious if this actually stops them. Does it work against the 403 errors I'm seeing with Python requests? I want to try this tonight. It's a huge relief, but I wonder if it handles the specific bypasses from GPT-4o... This burned me last year when my niche blog got scraped into oblivion. I wonder if it catches the newer Common Crawl bots?