Cloudflare announced Bot Preference Sync on August 21, a product that generates a website's robots.txt from three dashboard settings — Search, Agent, and Training — and prepends the entries between "#…
A developer who launched a Generative Engine Optimization (GEO) site tracked AI crawler traffic for 30 days using Cloudflare D1 logging. The data revealed that Bingbot dominates AI crawler volume, Cla…
OpenAI's ChatGPT-User bot accessed disallowed pages on nearly half of European sites that explicitly blocked it, and TollBit's State of the Bots report for the first half of 2026 found that about 15% …
TollBit's H1 2026 data found that ChatGPT-User ignored explicit robots.txt disallow instructions on 54% of its scrapes and reached disallowed pages on more European publisher sites than any tracked bo…
A developer released a Claude Code skill that diagnoses whether websites are visible to AI crawlers like ClaudeBot, GPTBot, and PerplexityBot, which do not run JavaScript. The tool fetches pages as ra…
AI crawlers are inflating web view counts by making hundreds of thousands of requests per day, skewing analytics for human engagement. A developer fixed the issue by moving tracking to client-side Jav…