Connect with execs from Axios, The New York Times, Paramount and more.
VIEW PASSES This week’s Media Briefing looks at how AI crawlers now reach publisher content, one year after Cloudflare’s default bot changes.
- Why the black market scraping economy still wins.
- Netflix strikes video licensing agreements with Penske Media digital brands including Variety and Rolling Stone.
From default blocks to ‘mixed-use’ bots #
The AI crawling environment for publishers is fragmenting. On one side is a visible layer of declared “good bots” that go through infrastructure gatekeepers like Cloudflare, Fastly and Akamai. On the other is a much larger gray crawling economy that still sits largely outside meaningful oversight.
Last week, Cloudflare rolled out new rules for “mixed crawlers” — bots that can be used both for basic search and for AI training or agentic tasks — declaring a “deadline to end the free pass” for these kinds of bots, which it says disadvantages site owners and more transparent AI companies. On Sept. 15, the defaults will be set to allow for search but block training and agent use for pages with ads.
That update lands a year after Cloudflare switched its default settings to make it easier for publishers to block AI crawlers, and underlines how quickly the crawling landscape has shifted from a blunt allow/deny decision to a layered mix of declared, mixed-use and gray bots. In practice, that now includes tackling the mixed-use giants whose crawlers have historically been treated as pure search, including Google.
Cloudflare isn’t claiming it can force separation of the most famous mixed-use crawler of all. But it is pushing for a clampdown on mixed-use crawlers generally, which put content creators in an “impossible position” and won’t lead to a “flourishing agentic internet” if they’re left unchecked, stressed Cloudflare’s chief strategy officer Stephanie Cohen. “We understand how we got here with the mixed‑use crawlers, but it doesn’t work for the ecosystem,” she told Digiday. “…there has to be a better way than forcing people into the decision of all or nothing.”
When asked whether the new settings would let publishers block Google’s crawler for AI training without tanking their search presence, Cohen was careful not to overpromise. “The reason we did not make the change on the day is that we want to engage in active dialogue with a crawler that today is characterized as search, because its predominant use case historically was search,” she said. “In the future, it would be characterized as mixed‑use, which means it would get caught in the default. Between now and then [Sept. 15] the goal is to figure out a way…for people to remain discoverable and indexed in search, while not being forced to give their information away for free.”
Chris Dicker, CEO of Candr Media, argues the power of Cloudflare’s move lies less in individual settings toggles and more in what happens by inertia. If even a fraction of the roughly 20% of the web that runs through Cloudflare simply leaves the new defaults alone, he said, the effective supply of freely scrapable content shrinks and the marginal cost of crawling goes up — a hypothetical he uses to show how defaults can start to put a real price on AI access, even if most savvy publishers rush to re‑enable Googlebot.
Dicker also sees implications for the U.K. Competition and Markets Authority’s conduct requirements on Google. Until now, he noted, Google has been able to push back on calls to separate crawlers for search and AI. If Cloudflare succeeds in forcing other AI players to split or declare their crawlers, that defense weakens and gives the CMA more ammunition to revisit how strictly it enforces crawler separation.
Why the black market scraping economy still wins
The new AI settings reshape how compliant crawlers behave, yet the biggest leakage for publisher content remains a gray scraping economy that doesn’t bother to play by those rules.
“Cloudflare only governs what’s behind Cloudflare. Well-funded scrapers can route around it,” said Dicker. “They can use residential proxies or just buy data from second-hand brokers that have been scraped elsewhere.”
It takes one to know one. Before Frederick Jahn started helping publishers defend against stealth crawlers, he spent his time reverse‑engineering the very scraping tools now hammering their sites. His team at Centinel Analytica has “pretty much reverse-engineered any protection that is out there,” he said, which is why he’s blunt about how far most publishers still are from real leverage in AI licensing.
To Jahn, the fixation on so-called “polite” bots that declare themselves in robots.txt misses the real threat. On major news brands, he estimates that while 20%-30% of traffic is from identified crawlers, roughly a quarter of total traffic is “stealth” crawlers mimicking human users — traffic most publishers don’t even see as bots. This gray scraping economy is routed through shared proxy infrastructure and scraping‑as‑a‑service providers, making it nearly impossible to tie specific AI companies to specific crawls with certainty, he said.
That opacity is exactly what he used to rely on. Now, on the defensive side, rather than tweaking robots.txt and hoping for the best, Frederick argues publishers need to turn on the highest‑friction defences they can: client‑side “are you human?” challenges on first page load, systematic blocking of non‑essential crawlers, and rigorous benchmarking of any vendor that claims more than 90% detection. The aim isn’t perfection; it’s to make scraping so unreliable and expensive that intermediaries are forced into a technical arms race — and, eventually, into licensing talks on publishers’ terms.
What a ‘good bot’ business model looks like
If the gray market relies on opacity and arbitrage, the emerging class of “good bots” is betting on the opposite: declared access, transparent economics and infrastructure partners to enforce the rules. One example is Ceramic, the AI search company founded by former Google engineering vp Anna Patterson, which has just begun working with Cloudflare on a premium search API that pays publishers when their pages are used to answer AI queries. Patterson’s core claim is that Ceramic’s search infrastructure is efficient enough to flip the unit economics. Where she says rival search APIs typically charge between $5 and $14 per 1,000 queries, Ceramic’s own costs come in closer to $0.05 per 1,000. Because the system is built specifically for AI use — crawling pages once, then serving 100- to 300-word snippets that LLMs can ingest — it doesn’t have to hammer publishers’ servers for every new question.
In the Cloudflare-aligned premium tier, that efficiency may become the basis of a per-use revenue share. When an AI model calls Ceramic’s API, Ceramic returns snippets from across the web and logs which publishers’ content was read as part of the answer. Instead of trying to work out which single article “won” the generated response, Ceramic pays out on the simpler rule: if your snippet was sent to the model, you get paid, whether or not the output visibly cites you.
Patterson’s pitch is that most of the per‑query fee flows back to publishers, with early modelling based on paying per 1,000 snippets served. The numbers are still being tested in a pilot, but the structure is clear: AI customers pay Ceramic for fast, compliant access to recent web content; Ceramic absorbs the infrastructure cost and “waterfalls” most of the margin back to the publishers whose pages underpin the answers.
That creates a different set of incentives from the current gray scraping economy. For AI companies, the promise is cheaper, low‑latency retrieval without having to run their own crawlers through an increasingly hostile and legally risky landscape. For publishers, the offer is to be crawled once, have access routed through Cloudflare and similar gatekeepers, and receive a metered income stream tied to actual AI usage of their content, rather than hoping for occasional headline licensing deals.
Patterson argues that if enough major publishers adopt this model, it could start to compress the market for third‑party scrapers. It won’t eliminate stealth scraping — well‑funded actors can still route around Cloudflare — but it does offer a concrete alternative to the black‑market status quo, where most of the economic upside from AI training and agentic use never reaches the publishers whose work is being scraped.
As Patterson put it, “it’s an experimental time, and we’ll likely see more pretty interesting experiments in this direction.”
What we’ve heard: #
“We’re seeing increasing financialization within the advertising community, where quick, rapid metrics that demonstrate whether advertising is affecting top- and bottom-line growth are becoming pivotal.”
— IAB Europe chief economist, Daniel Knapp, on state of Europe’s $150 billion advertising market.
Numbers to know: #
€131 billion ($150 billion): Total Continental Europe ad spend, up 10.5% YoY, perIAB Europe.$3.8 billion: The value of the defamation suit filed by Trump Media in 2023, which wasthis week overturnedby a federal judge.$5.99: Monthly price of The Hill’snew subscription offering; $9.99 for its premium tier.71%: How much more often Google’s AI Overviews are now showing up on searches where people are likely to buy something, perSemrush data.$67 million:Amount U.K.’s Daily Mail publisheris seeking in legal costs after winning its case against Prince Harry, Elton John and more.
What we’ve covered: #
In Graphic Detail: The state of streaming highlights the power of creators
- Halfway through 2026, the state of streaming underlines just how dependent the category has become on big, personality-driven creators.
- Streaming platforms like Twitch and YouTube Gaming have become a major spoke in the broader creator economy wheel.
*Read more *here.
**Why brands are bringing creators to the World Cup sidelines **
- Brands aren’t just buying media around the tournament – they’re building entire creator-led activation machines.
- That ranges from local microcreators selling host cities to influencer-heavy sidelines content that turns every match into a days-long stream of social posts.
*Read more **here. *
Hot Ones creator Sean Evans on YouTube vs. TV, the interview boom and what comes next
- Digiday sat down with Evans to talk about how interview formats and platforms have evolved, what’s real versus hype around AI and CTV, and where the next wave of creator business is headed.
- “I do think that you’ll just see like a Wild Wild West, just passing the torch – these paradigm-shifting moments that used to happen in media would be once a generation,” he said.
*Read more *here.
Dentsu strikes Meta deal to build plumbing for mass influencer activation
- The holding company is set to integrate Meta’s Creator Marketplace and Partnership Ads into its operating system through an API partnership.
- It’s allowing users of its internal systems to manage social listening, creator selection and paid activation within the same dashboard, blending Dentsu’s own data with Meta’s platform.
*Read more *here.
What’s really driving Europe’s €131 billion ad boom
- Continental ad spend up 10.5% year over year to €131 billion — roughly the size of Morocco’s entire economy, according to the report’s author and IAB Europe chief economist Daniel Knapp.
- Video, in all its forms, from CTV to social video, is hoovering up ad dollars for media owners, now accounting for more than half of all display investment in Europe.
*Read more *here.
What we’re reading:
**Netflix strikes licensing deals with Penske Media’s PMX unit **
Netflix has agreed video licensing agreements with Penske Media brands including Variety, Rolling Stone and Billboard.French watchdog orders Meta to negotiate in news copyright case
France’s competition regulator has ordered Meta to negotiate in good faith with news publishers over copyright payments after complaints that it abused its market dominance.
Google Search lets creators know more about their reach
Google is introducing a Search Console feature called “platform properties” to help creators and website owners understand how users discover their content across platforms like Instagram, TikTok, X, and YouTube through Search.
How David Senra built the podcast CEOs can’t stop listening to
A deep dive into how Senra’s podcast has become a favorite among some of the world’s most successful executives by exploring the lessons, struggles, and decisions behind the lives of history’s great entrepreneurs.
BBC and Channel 4 want to create a streaming giant to rival Netflix The BBC’s new director general and former Google boss Matt Brittin has said Channel 4 and the BBC are in talks over how they scale a streaming rival to Netflix.
More in Media
[
In Graphic Detail: The state of streaming highlights the power of creators ](https://digiday.com/media/in-graphic-detail-the-state-of-streaming-highlights-the-power-of-creators/)
“Just Chatting” is the driving force behind views on major streaming platforms, thanks to the appeal of personality-driven creators
[
Hot Ones creator Sean Evans on YouTube vs. TV, the interview boom and what comes next ](https://digiday.com/media/hot-ones-creator-sean-evans-on-youtube-vs-tv-the-interview-boom-and-what-comes-next/)
Hot Ones host and TIME 100 top creator Sean Evans chats about the creator economy’s past, present, and future
[
Why brands are bringing creators to the World Cup sidelines ](https://digiday.com/media/why-brands-are-bringing-creators-to-the-world-cup-sidelines/)
Brands are bringing creators to the World Cup sidelines to boost engagement, tap into new audiences, and be a part of the cultural conversation.