A new open-source tool called robots-check reveals that Cloudflare's AI Crawl Control feature silently appends its own rules to websites' robots.txt files, often blocking AI crawlers like GPTBot and G…
Common Crawl's analysis of over 500,000 llms.txt files found that 68% originated from a plugin or template, 22% had no links, and some sites are using the format like robots.txt despite it granting or…
A 24-hour census of a single MCP endpoint recorded 60 distinct named crawler agents making 6,309 requests, including roughly 600 that hit a /v1/buy/* route and received a valid HTTP 402 payment challe…
Cloudflare CFO Thomas Seifert told analysts on the company's second-quarter earnings call that non-human traffic could reach 1,000 times human traffic within five years, adding that 'humans will be a …
An engineer's analysis of 30 days of server logs reveals that 71% of AI crawler traffic is spoofed, with 916 of 1,290 attributable requests coming from sources not operated by the claimed vendors. The…
Common Crawl's July 2026 crawl contains 2.14 billion webpages, but a new analysis shows that 95% of the roughly 492,000 sites naming CCBot in robots.txt do so to block it, and the BBC is among major p…
A new web standard, AI-aware robots.txt, allows site owners to control which AI crawlers can access their content, preventing silent exclusion from AI training data and answer corpora. The article pro…
A developer discovered that AI crawlers like GPTBot, ClaudeBot, and PerplexityBot were methodically reading their technical blog. Instead of blocking them, they realized these bots distribute content …