A developer building a tool that checks how visible webpages are to AI crawlers found it reported a 403 error response from nytimes.com as if it were the real homepage, correctly flagging a missing H1…
A developer detailed how edge security rules and web application firewalls—not robots.txt—are the usual cause of AI crawlers like GPTBot being silently blocked from websites. The writeup explains that…
A site's edge sensor recorded 1,534 fetches from self-declared AI-training crawlers between July 28 and August 10, 2026, and exactly one of them touched a rights file, according to the site's own coun…
Website operators can block AI training and search crawlers such as GPTBot, ClaudeBot, Google-Extended and PerplexityBot with robots.txt entries, but user-triggered agents including ChatGPT-User, Goog…
A guide from soamee.com lays out a five-layer method for measuring traffic and leads arriving from ChatGPT and other AI assistants, starting with a custom "AI Assistants" channel group in GA4 that mat…
A developer outlines a framework for deciding whether to block AI crawlers such as GPTBot, ClaudeBot, CCBot, PerplexityBot and Bytespider, arguing that a selective policy beats a blanket allow-or-bloc…
A developer-run census of 100 WooCommerce stores found that only 2 of 93 stores with a robots.txt file blocked any of nine major AI crawlers, and none blocked OpenAI's shopping crawlers, while product…
A new free AI visibility audit tool scores websites from 0 to 100 on how clearly generative models parse their entity data, with breakdowns across ChatGPT Search, Google AI Overviews, Claude, and Perp…
A developer has published a WordPress mu-plugin that logs visits from nine documented AI crawler user agents, after finding that Google-Extended and Applebot-Extended never appear in access logs becau…
Websites can now block AI training crawlers such as GPTBot while remaining discoverable to search indexers by configuring granular robots.txt rules and HTTP headers, according to Cloudflare. Google-Ex…
AI answer engines including ChatGPT, Claude, and Perplexity rely on distinct crawlers — GPTBot, ClaudeBot, PerplexityBot, and Google-Extended — that many sites inadvertently block or fail to serve, ac…
A 2026-09-07 census of 2,771 sites serving a robots.txt found that 368 block at least one of the three crawlers feeding AI answers, and 200 block all three. PerplexityBot is blocked by 358 sites, Clau…
A developer building an answer engine optimization site for SearchD implemented llms.txt, JSON-LD, and AI crawler allowances, then tested whether each actually delivers the promised citation benefits.…
Buttondown replaced its raw user-agent-string textarea with a Crawling add-on in Settings → Archives that gives users one toggle per crawler category: Search engines (Googlebot, Bingbot, DuckDuckBot, …
The IETF's Web Bot Auth working group adopted draft-ietf-webbotauth-httpsig-protocol as its first working-group document in September 2026, folding in the separate key-directory draft so one document …
A new open-source tool called robots-check reveals that Cloudflare's AI Crawl Control feature silently appends its own rules to websites' robots.txt files, often blocking AI crawlers like GPTBot and G…
A developer detailed a free website SEO audit process that uncovered 31 broken pages and 900 phantom URLs on an inherited site, leading to improved crawl efficiency after fixes. The audit uses free to…
A Cloudflare setting silently blocked AI crawlers from a digital marketing consultant's website, making it invisible to AI assistants for its entire existence, according to the consultant's own accoun…
A study of 9,037 AI tools found that 10.5% explicitly block OpenAI's GPTBot in their robots.txt files, with Cloudflare-hosted sites more than four times as likely to block it as Vercel-hosted sites. T…
A developer's analysis reveals that AI crawlers like GPTBot do not execute JavaScript, meaning client-rendered websites appear empty to them. Testing across 500 million GPTBot fetches by Vercel and ME…