{"slug": "cloudflare-threatens-to-cut-google-off-from-their-publishers-in-searches", "title": "Cloudflare Threatens to Cut Google Off from Their Publishers in Searches", "summary": "Cloudflare is threatening to block Google's multi-purpose crawlers from accessing ad-supported webpages starting September 15, 2026, unless Google provides a technical solution that allows publishers to remain discoverable in search without giving away content for free. The move targets Google's practice of using a single crawler for both search indexing and AI training, which has frustrated publishers who say their content is being stolen. Cloudflare's ultimatum could lead to a significant shift in how publishers interact with Google Search.", "body_md": "One of our big complaints about AI is that they steal our copyrighted content. On top of that, the intensity of site scraping is so high that even Cloudflare (and we pay pretty hefty monthly fees for their industrial-grade product), which has the best bot detection and blocking on the Internet, is having difficulty in this arms race. Readers may have noticed that from time to time, we have had to put up captchas to stop this theft. If we don’t, the degradation of performance results in time outs for admins and writers. That means we can’t put up new posts or free comments caught in moderation.\n\nApparently Cloudflare is as frustrated as we are. From Adweek in [Once Unimaginable, Publishers Are Preparing to Opt Out of Google Search.](https://www.adweek.com/media/publishers-opt-out-google-search/):\n\nThe nuclear option is gaining traction as web traffic collapses and Google refuses to negotiate with content creatorsFor decades, publishers have done everything in their power, from the legal to the not-explicitly illegal, to rank as highly in Google Search as possible…\n\nNow though, a handful of influential players in the digital media ecosystem have begun moving in the opposite direction, laying the groundwork for what was once unthinkable: removing themselves from Google Search.\n\nLast week, the content delivery network Cloudflare, which hosts roughly one-fifth of the websites in the world, gave Google an ultimatum.\n\nBeginning Sept. 15, all new websites signing up for Cloudflare, as well as all the customers on its free tier, will have the default settings in their bot management protocol set to block “multi-purpose crawlers” on any webpage that has ads. This means that any crawler that scrapes for both search indexing and AI training will be turned away at the door, unless the site owner decides otherwise.\n\n“We’ve been clear about what we want,” said Cloudflare chief strategy officer Stephanie Cohen. “We want a technical solution that allows you to be discoverable without having to give your content away for free.”\n\nWhile a handful of crawlers fit this description—Apple and Bing, among others—the primary, unnamed target of this action is Google, which infamously uses one crawler to both index sites and train its AI models….\n\no be fair, Google recently introduced an option called Google Extended, which nominally allows publishers to opt out of AI training without disappearing from Search. But publishers are wary that the program will penalize their search visibility, according to executives at two media companies.\n\nWhat this article does not mention is that Google has been hostile to independent publishers at least as early as 2014. We have been repeatedly downgraded in searches, as have sites like The Intercept and Truthout. We don’t see Google as at all helpful in finding new readers, but it would be a shame if they could not longer find articles. However, Kagi is vastly better, particularly on site-specific searches, so we continue to recommend it. Remember, if something is free, you are the product, and with Kagi you also get privacy. They do not retain search histories.\n\nConfirming our dim view of Google’s predation:\n\nGoogle pays Reddit millions for content access.\n\nYour website's content is an asset too.\n\nDon't let every AI crawler scrape it for free. Control who can access your content while keeping Google Search working.\n\nCloudflare AI Crawl Control + Pay Per Crawl are just the beginning.\n\n[pic.twitter.com/WRxXzEx6KD]— Cuong Thach (@cuongthach_)\n\n[July 10, 2026]\n\nAnd some powerhouses are effectively joining Cloudflare. Again from Adweek:\n\nUSA Today Inc., which encompasses not just USA Today but a nationwide network of news sites, is weighing its options on the matter, according to CEO Mike Reed.\n\nThe company, like many others, has responded to declines in search traffic by bolstering audience from other sources, like newsletters, social media, and events. Its traffic has remained relatively stable in recent years, hitting its goal of 1 billion pageviews every month for the last three years, according to Reed.\n\nStill, its monetization strategy going forward as it relates to AI will come from licensing agreements, which the company has struck with Meta, Microsoft, and Amazon, among others. Google, unlike its hyperscaler peers or pure-play AI firms like OpenAI and Anthropic, has not struck any licensing deals with any publishers.\n\nAs a result, USA Today Inc. is prepared to delist from Google in the next six to twelve months, according to Reed. Likewise, the creator network Beehiiv announced in a recent partnership with Cloudflare that its network of creators now has the ability to block the Google crawler.\n\n“I wouldn’t call it a big decision because we’re blocking other crawlers,” Reed said. “For those with licensing agreements, they get our content. For those without, we block them.”\n\nWhile USA Today Inc., Beehiiv, and Cloudflare are the first major players to take this step, executives at every major media company have a model for what it would look like if they blocked the Google Bot, according to one executive who wished to remain anonymous because of business engagements with Google.\n\nIt’s worth noting what these AI scrapers ought to be paying is considerable. For ECONNED, your humble blogger is due $3,000 in a class action settlement with Anthropic. How much do you imagine all of these AI thieves should be paying for the vastly larger body of work at Naked Capitalism?\n\nSo again, AI is evil, it is destroying the business of original content provider to generate regularly erroneous and colorless derivatives. And it is not good for critical thinking or retention. But informational junk food is proving to be popular.\n\n## 32 comments\n\n-\n**Acacia** Didn’t see this coming, but good on them!\n\nAnd if Google is downranking your site, why allow their bots in the door?\n\nThere are other, less scammy search services.\n\n[Reply](#comment-4441440)↓ -\n**TimH** And if Google is downranking your site, why allow their bots in the door?\n\nThere are other, less scammy search services.\n\nThat really doesn’t help when the majority of people still googlesearch, particularly insidious for Android OS.\n\n[Reply](#comment-4441476)↓ -\n**DJG, Reality Czar*** Cloudflare chief strategy officer Stephanie Cohen: “We want a technical solution that allows you to be discoverable without having to give your content away for free.”*This is why I have no patience with people who go on and on (and on) about the dangers of copyright and how information (an animate object) just wants to be free.\n\nThe alternative is just what Yves Smith is describing here — “scraping” is a polite word for copying and stealing. How do you think that writers make money (a pittance in most cases) off their work? Through copyright.\n\nCloudfare will benefit from copyright law, too, because copyright law is clear, well established, and has plenty of court cases to back it up.\n\nAnd, no, “fair use” doesn’t allow you to copy large chunks of text from someone’s essay and download photos off someone’s site just because You Be You.\n\n[Reply](#comment-4441494)↓-\n**Amr** The alternative is just what Yves Smith is describing here — “scraping” is a polite word for copying and stealing. How do you think that writers make money (a pittance in most cases) off their work? Through copyright.\n\n“Scraping” is a term that’s almost as old the modern internet itself and has has many legitimate uses. There are been plenty of now-defunct websites with valuable information that would have been lost forever if they had never been scraped by the Internet Archive and other smaller, independent groups like Archive Team. Since Twitter shut down their API, the only way to actually archive a tweet (say, to have a record for what a public figure says) is to use third-party scrapers. They become a major problem if they’re deployed recklessly enough to substantially slow down websites and increase hosting costs, as most AI labs have done.\n\nI have always taken Richard Stallman’s perspective that copyright was designed with readers, not publishers, in mind. In enforcing copyright, we’re making making the gamble that that ceding certain rights to publishers is beneficial, since there is an incentive to publish more. This does not mean the same thing as maximizing published content, since because of diminishing returns there comes a point where we’d have ceded too much.\n\nI’ll always be wary of extensions of copyright power, because even if we take the perspective of helping publishers it will be the largest publishers that stand to gain the most. I don’t think copyright extensions for works from the 1920s to prevent them from entering the public domain helps the “average” publisher instead of companies like Disney.\n\n[Reply](#comment-4441613)↓\n\n-\n-\n**.Tom**>\n\n*How much do you imagine all of these AI thieves should be paying for the vastly larger body of work at Naked Capitalism?*A\n\n*lot*more than $3000. That’s nothing to the likes of Google. When you combine volume, relevance, quality and the variety of perspectives and checks in comments… Idk. More than a master’s degree per user, lets say.Edit to add: Props to Cloudflare for caring and doing something.\n\n[Reply](#comment-4441497)↓-\n**das monde**[In the Weights is your new AI-centric vanity search](http://https://techcrunch.com/2026/06/20/in-the-weights-is-your-new-ai-centric-vanity-search/)The “weights” in question are the numerical parameters that shape an AI model’s training and output, so the website purports to measure how well “a model is able to recall someone without using tools like web search.”\n\n“Being in the weights means your existence was deemed important in the process of creating superhuman artificial intelligence,” the website says.\n\n[Reply](#comment-4441586)↓\n\n-\n-\n**Jason Boxman** At an employer I know, they’re hogwild for AI. They haven’t yet entered the “find out” period of cost per token, despite being a huge software infrastructure player in the enterprise market. Everyone is expected to, and is being pushed constantly, to use AI as much as possible for any task, in the mercurial quest of finding some purpose for it.\n\nIn upskill training material, it explains in one segment that asking for JSON/structured output is worthwhile, because it enables a company to ensure content adheres to internal content styling requirements.\n\nHello. Non-deterministic. It outputs “JSON like” slop. No guarantees that it is valid, so how does this accomplish anything?\n\nI hate this timeline. I want to opt out.\n\n[Reply](#comment-4441498)↓ -\n**Carolinian** We who once defended Google apologize. Power corrupts.\n\n[Reply](#comment-4441536)↓ -\n**Tom Stone** When all you give people is shit it doesn’t take them long to realize you are an asshole.\n\n[Reply](#comment-4441552)↓ -\n**AG** So what should I install as standard search engine instead of Big G?\n\nQwant, Yandex, Duck, didn´t work for me. I use them additionally.p.s. I remember the day at a company when head told us about this new site, “google”. To me it sounded like a joke. That was still altavista era…\n\n[Reply](#comment-4441557)↓ -\n**flora** When the Goog acts up I go to Mojeek(dot)com. Type ‘Naked Capitalism’ into the search bar and see all the links that come up.\n\n[Reply](#comment-4441631)↓-\n**AG** wow\n\nI search for “Russiagate” and the first 5 hits are German alter. sites.\n\nThank you![Reply](#comment-4441634)↓\n\n-\n-\n**elviejito** Thanks for the “Kagi” suggestion. Trying it out now.\n\n[Reply](#comment-4441658)↓ -\n**Dingleberry** Both Cloudflare and Google are Big Tech scum.\n\nLet them fight and kill each other, one can hope![Reply](#comment-4441667)↓ -\n**Rui** I once made the mistake of paying for Google ads for my small medical practice. After 6 months of not getting any new patient via the paid ads, I canceled it.\n\nRetaliation was swift. My practice disappeared from Google Maps. Even when doing maximum zoom in on my office location, it didn’t show. Doing a maps search for my business didn’t show it unless I spelled it out. None of this was true before I paid for the adverts.\n\nIt was unannounced gangster like behaviour to blackmail me into paying again. I never did again and never will again. But it really opened my eyes.\n\n[Reply](#comment-4441690)↓ -\n**Dadda** This just appeared on Hacker News:\n\nGhost Font An anti-AI font that can be read by humans but not leading AI models\n\nat\n\n[https://www.mixfont.com/ghost-font](https://www.mixfont.com/ghost-font)There’s a discussion and proposals for applications.\n\n[Reply](#comment-4441696)↓-\n**vao*** Ghost Font An anti-AI font that can be read by humans but not leading AI models**…that can be read*. I tried various examples, but definitely have a hard time and most often fail to discern those moving pixelated characters.**with more or less difficulties** by humansMy sight is defective, so there is that. As another example, I just cannot get those images made of jagged coloured patterns supposedly producing some kind of 3D picture if one focuses appropriately.\n\n[Reply](#comment-4441838)↓\n\n-\n-\n**Mario** For ECONNED, your humble blogger is due $3,000 in a class action settlement with Anthropic.\n\nWhich class action settlement is this? I’d like to know more, did you write about it?\n\n[Reply](#comment-4441767)↓", "url": "https://wpnews.pro/news/cloudflare-threatens-to-cut-google-off-from-their-publishers-in-searches", "canonical_source": "https://www.nakedcapitalism.com/2026/07/cloudflare-threatens-to-cut-google-off-from-their-publishers-in-searches-due-to-ai-scraping.html", "published_at": "2026-07-12 03:43:18+00:00", "updated_at": "2026-07-12 04:04:59.132443+00:00", "lang": "en", "topics": ["ai-policy", "ai-ethics", "ai-infrastructure"], "entities": ["Cloudflare", "Google", "Stephanie Cohen", "Apple", "Bing", "USA Today", "Mike Reed", "Kagi"], "alternates": {"html": "https://wpnews.pro/news/cloudflare-threatens-to-cut-google-off-from-their-publishers-in-searches", "markdown": "https://wpnews.pro/news/cloudflare-threatens-to-cut-google-off-from-their-publishers-in-searches.md", "text": "https://wpnews.pro/news/cloudflare-threatens-to-cut-google-off-from-their-publishers-in-searches.txt", "jsonld": "https://wpnews.pro/news/cloudflare-threatens-to-cut-google-off-from-their-publishers-in-searches.jsonld"}}