A developer outlines a framework for deciding whether to block AI crawlers such as GPTBot, ClaudeBot, CCBot, PerplexityBot and Bytespider, arguing that a selective policy beats a blanket allow-or-bloc…
A developer-run census of 100 WooCommerce stores found that only 2 of 93 stores with a robots.txt file blocked any of nine major AI crawlers, and none blocked OpenAI's shopping crawlers, while product…
A developer's AI crawler dashboard recorded 20 requests for the site's /.env secrets file over seven days, all claiming to be ClaudeBot, while only 68% of ClaudeBot traffic that week could be verified…
A new free AI visibility audit tool scores websites from 0 to 100 on how clearly generative models parse their entity data, with breakdowns across ChatGPT Search, Google AI Overviews, Claude, and Perp…
A check of 32 well-known websites' public robots.txt files on 7 September 2026 found that 16 block at least one AI search crawler, and 10 block both OAI-SearchBot, used by ChatGPT for citations, and P…
AI answer engines including ChatGPT, Claude, and Perplexity rely on distinct crawlers — GPTBot, ClaudeBot, PerplexityBot, and Google-Extended — that many sites inadvertently block or fail to serve, ac…
A 2026-09-07 census of 2,771 sites serving a robots.txt found that 368 block at least one of the three crawlers feeding AI answers, and 200 block all three. PerplexityBot is blocked by 358 sites, Clau…
A developer building an answer engine optimization site for SearchD implemented llms.txt, JSON-LD, and AI crawler allowances, then tested whether each actually delivers the promised citation benefits.…
A developer detailed a free website SEO audit process that uncovered 31 broken pages and 900 phantom URLs on an inherited site, leading to improved crawl efficiency after fixes. The audit uses free to…
A Cloudflare setting silently blocked AI crawlers from a digital marketing consultant's website, making it invisible to AI assistants for its entire existence, according to the consultant's own accoun…
Search Engine Land's investigation found that WP Engine's managed WordPress hosting blocks or rate-limits certain AI crawlers, including GPTBot, ClaudeBot, and PerplexityBot, at the platform level, pr…
Indexflow-SEO, a lightweight SEO analyzer written in Rust, has been released as version 0.1.2 on crates.io, offering zero-dependency technical SEO validation and Generative Engine Optimization (GEO) a…
A developer warns that many AI crawlers, including GPTBot, ClaudeBot, and PerplexityBot, do not execute JavaScript, making client-rendered websites invisible to them. The post advises using curl to ch…
A developer found that most AI crawler traffic on their site is fake, with only 468 of 1,459 requests actually fetching articles. Using Cloudflare's verifiedBotCategory, they showed that only 13% of G…
A 41-day field experiment by SEO engineer Vinicius Stanula found that JavaScript-injected navigation creates a discovery gap for AI crawlers. Pages linked only through JavaScript were not discovered b…
GEO (Generative Engine Optimization) is the practice of optimizing content so that generative AI systems like ChatGPT, Perplexity, Gemini, and Google's AI Overviews cite and attribute it in their answ…
An engineer's checklist reveals why AI assistants often fail to cite web pages: crawlers like GPTBot, ClaudeBot, and PerplexityBot may be blocked by robots.txt or CDN settings, or pages may rely on cl…
TollBit's H1 2026 data found that ChatGPT-User ignored explicit robots.txt disallow instructions on 54% of its scrapes and reached disallowed pages on more European publisher sites than any tracked bo…
A ReadableByAI Index found that 25.5% of 145 YC Fall 2025 startup homepages are empty client-rendered shells to AI crawlers like GPTBot, ClaudeBot, and PerplexityBot, compared to 2.9% of 455 establish…
A developer built a tool that checks how visible websites are to AI engines like ChatGPT, Perplexity, and Gemini, revealing that most sites are failing at Generative Engine Optimization (GEO). The too…