A developer shipping a static Astro site to Cloudflare Pages discovered three platform defaults that silently broke the site on launch day. Cloudflare's managed robots.txt injected AI crawler blocks, …
A 41-day field experiment by SEO engineer Vinicius Stanula found that JavaScript-injected navigation creates a discovery gap for AI crawlers. Pages linked only through JavaScript were not discovered b…
An engineer's analysis of 36 days of server logs reveals that Bing's crawler, bingbot, now hits his site 4.2 times more than Googlebot, and that ChatGPT's citations largely derive from Bing's index. C…
Starting September 15, 2026, Cloudflare will block Googlebot and Bingbot on sites set to block AI training, potentially affecting search indexing. Resoneo found that ChatGPT's own index serves sites w…
Cloudflare reported that 2% of bot traffic across 5,000+ websites using Agent Analytics and AI Chat Referral Tracking is AI-related, up 11% from the previous 90 days, while overall bot visits fell 2%.…
Developer Marco Bellingeri opened his website's robots.txt to all AI crawlers to boost visibility, but after receiving 33,561 pageviews in its first month, he realized the number was inflated by bots …
TIME is serving AI crawlers a stripped-down markdown version of its website with ads embedded, while human visitors receive the full HTML page, according to a test by an unnamed researcher. The markdo…
A Redditor in the r/SEO community reports that enabling Cloudflare's AI Training block causes Googlebot and Bingbot to receive HTTP 403 errors when fetching sitemaps, despite Cloudflare's official ann…
AI crawlers from OpenAI, Anthropic, and Perplexity do not execute JavaScript, leaving client-rendered single-page apps invisible to them, according to an audit by developer and product builder who run…
Cloudflare launched granular AI traffic controls this month, replacing its single 'block AI bots' toggle with three independently configurable categories—Search, Agent, and Training—available to all c…
Cloudflare replaced its one-click AI bot block with three per-purpose controls — Search, Agent, and Training — available on every plan starting July 1, and announced that from September 15, pages disp…
Cloudflare announced new crawler rules on July 1 that classify all automated traffic into Search, Agent, or Training behaviors, with mixed-use crawlers subject to the most restrictive rule, directly t…
Cloudflare will block AI training and agent crawlers by default on ad-supported pages starting September 15, affecting all new sites and existing free-tier accounts. The company introduced a three-cra…
Cloudflare announced plans to automatically block mixed-use web crawlers that index sites for search engines and also serve AI companies, starting September 15, 2026. The hosting platform will default…
Cloudflare is updating its AI crawler controls to categorize bots by behavior—search, agent, or training—and will block training and agent crawlers by default on ad-supported sites starting September …
A new analysis of server logs for CitationIQ.com found that 81.8% of claimed AI assistant traffic was fake, with only 6 of 33 requests verified as legitimate. Googlebot spoofing was even worse, with j…
A developer built Greater Than Services, a platform that audits websites for three pillars: Conversion Rate Optimization (CRO), Generative Engine Optimization (GEO), and Agentic Search Optimization (A…
Jeremy Howard from Answer.AI proposed llms.txt, a Markdown file placed at a domain root to describe a site for AI consumption. Unlike robots.txt or sitemap.xml, which are used by continuous crawlers, …
AI crawlers now match Googlebot in traffic to a website that hit the front page of Hacker News, with both categories accounting for 35% of 240,060 bot visits over 30 days. AmazonBot was the most aggre…
A developer discovered that AI crawlers like GPTBot, ClaudeBot, and PerplexityBot were methodically reading their technical blog. Instead of blocking them, they realized these bots distribute content …