A Cloudflare setting silently blocked AI crawlers from a digital marketing consultant's website, making it invisible to AI assistants for its entire existence, according to the consultant's own accoun…
Cloudflare's AI Crawl Control data revealed that nearly half of the requests claiming to be from OpenAI's ChatGPT-User agent were actually vulnerability scanners probing for secrets like SSH keys and …
A developer found that most AI crawler traffic on their site is fake, with only 468 of 1,459 requests actually fetching articles. Using Cloudflare's verifiedBotCategory, they showed that only 13% of G…
In a 30-day server-log census of dejan.ai's machine-readable files, Google, Anthropic and OpenAI crawlers fetched robots.txt 1,150 times, the OKF content bundle 23 times, and llms.txt 4 times (none fr…
TollBit's H1 2026 data found that ChatGPT-User ignored explicit robots.txt disallow instructions on 54% of its scrapes and reached disallowed pages on more European publisher sites than any tracked bo…
OpenAI's ChatGPT-User bot accessed disallowed pages on nearly half of European sites that explicitly blocked it, and TollBit's State of the Bots report for the first half of 2026 found that about 15% …
Cloudflare reported that 2% of bot traffic across 5,000+ websites using Agent Analytics and AI Chat Referral Tracking is AI-related, up 11% from the previous 90 days, while overall bot visits fell 2%.…
A developer added AI-readability checks to the open-source SEO tool open-seo, detecting whether a site blocks AI agents like GPTBot and ClaudeBot or supports the emerging llms.txt standard. The featur…
Cloudflare is updating its AI crawler controls to categorize bots by behavior—search, agent, or training—and will block training and agent crawlers by default on ad-supported sites starting September …
Website owners face a dilemma over whether to block AI crawlers or measure their value first, as the proliferation of AI bots increases server costs and raises questions about ROI. SEO experts recomme…
A developer discusses how AI crawlers and robots.txt policies are fragmenting the web's shared information environment. As websites selectively allow or block AI systems like ClaudeBot and GPTBot, ret…
AI crawlers now match Googlebot in traffic to a website that hit the front page of Hacker News, with both categories accounting for 35% of 240,060 bot visits over 30 days. AmazonBot was the most aggre…
Fastly released an open-source JavaScript service for its Compute platform that automatically converts web pages into clean Markdown for AI crawlers, reducing bandwidth and parsing overhead. The servi…