# European publishers are getting hit harder by AI bot scraping, report finds > Source: > Published: 2026-08-14 04:01:00+00:00 Join us Sept. 14-16 in Miami to connect with top publishing leaders [SECURE YOUR SEAT](https://digiday.com/events/digiday-publishing-summit-sep-26/?source=display&utm-source=display) # European publishers are getting hit harder by AI bot scraping, report finds European publishers are getting hit harder than North American sites by AI bot scraping – and no one really knows why. [A recent report from TollBit](https://tollbit.com/state-of-the-bots/) found that European sites get scraped more by AI bots, receive fewer referrals, and have their robots.txt rules – which websites use to communicate to crawlers what they can and cannot scrape – ignored more often. Here are the main findings from that report, based on TollBit’s analysis identified AI bots from 40 scraping vendors across 3,906 publishers: - Median AI scrapes per site were four-times higher on European sites compared to North American sites. - European publishers receive just one human referral visit from AI apps for every 179 AI bot visits — a rate more than three times worse than for North American sites. - In the first half of 2026, just 0.05% of external referrals to European sites came from an AI app (and about 1 AI bot scrape for every 33 human visits), versus 0.16% in North America. And the situation is getting worse. AI scrapes on European sites were nearly 20% higher in June than in January 2026. Local news and sports sites saw the highest quarter-over-quarter growth. The scrape-to-referral ratio went from 150:1 in Q1 2026 to 227:1 in Q2 2026, according to the report. The median European site’s robots.txt instructions to not scrape were ignored nearly three times more often than North American sites. TollBit’s cofounder Olivia Joslin speculated that this disparity could be because Europe has more languages than North America, and as LLMs try to learn different languages and pull content from different sources, European publishers are getting hit more by bot scrapes. Grzegorz Piechota, researcher-in-residence at INMA, suggested the same, though his argument leans on usage data. Claude users in the U.S. account for more than a fifth (21.6%) of total Claude usage, and English-speaking countries make up less than a third of Claude users globally, [according to stats from Anthropic](https://www.anthropic.com/research/anthropic-economic-index-september-2025-report) from last September. It means, said Piechota, that most Claude usage, and by extension most scraping activity, is likely tied to non-English content, which would explain the heavier hit to European publishers. “Consumer demand for non-English language information is influencing the strategies of AI companies and their suppliers of data,”said Piechota. “For example, in recent years, Google, Open AI, and Microsoft announced research and initiatives to make AI more relevant to multi-language audiences, and especially those in Europe. As a result, suppliers of data for AI, such as Common Crawl, started to collect more non-English content, he said. European country domains made up only 4.75% of pages captured by Common Crawl in 2009. In 2026, they made up almost 29.98%, according to Piechota’s analysis. “The Big Tech and AI industry seems to be catching up with demand for high-quality information in multiple languages,” Piechota said. Another reason for the findings in TollBit’s report could be the mix of publishers in the company’s customer base and therefore the dataset, Piechota noted. Of the nearly 4,000 publishers included in TollBit’s analysis, 456 of which were European publishers, ranging from the Telegraph to Ringier to Styria. Cloudflare data, by contrast, does not show European publishers getting hit by bots more than North American sites, according to Lai Yi Ohlsen, head of Cloudflare’s internet tracking dashboard, Cloudflare Radar. The absolute number of bot requests is consistently higher for those in North America, she said. However, European publishers have a higher percentage of all their site requests coming from bots, compared to North American sites, according to Cloudflare data from customers in the media and publishing sector. In other words, European sites are attracting a higher share of bot traffic, but North American sites have more total bot activity. Data from DataDome, a cybersecurity and bot blocking company, also doesn’t show the same consistent gap between European and North American publishers as TollBit’s findings, according to vp of threat research, Jérôme Segura. There is a “massive” variance in AI bot scraping publisher by publisher, likely based on the prominence and size of a publisher and how much scraping is coming from bots that self-identify (and therefore easier to track), he said. “In our own data, some individual publishers we looked at (regardless of region) showed AI traffic as high as roughly 1 AI visit for every 10 human visits, underscoring just how significant a share of overall site traffic AI agents can represent once you look past regional averages,” he said. Every source has a theory and every theory is backed by a different dataset pointing in a different direction. Sure, the scrapes are real but the reasons behind them are still up for debate. ### More in Media #### [ An anatomy of the creator brand trip’s midlife crisis ](https://digiday.com/media/an-anatomy-of-the-creator-brand-trips-midlife-crisis/) The brand trip as we know it may be dead, but a newer, more genuine and engaging format is rising from its ashes. #### [ Walmart-backed gaming publication lays off editorial team ](https://digiday.com/media/walmart-backed-gaming-publication-lays-off-editorial-team/) Brandy Berthelson — who had been Restart’s editor-in-chief — said on Bluesky last Friday that the “entire editorial team” at Restart was laid off. #### [ Brands can’t just flip the ’employees as creators’ switch ](https://digiday.com/media/brands-cant-just-flip-the-employees-as-creators-switch/) Employee generated content is an increasingly enticing lever for brands to pull, but there needs to be a clear-cut strategy.