A site's edge sensor recorded 1,534 fetches from self-declared AI-training crawlers between July 28 and August 10, 2026, and exactly one of them touched a rights file, according to the site's own coun…
A guide from soamee.com lays out a five-layer method for measuring traffic and leads arriving from ChatGPT and other AI assistants, starting with a custom "AI Assistants" channel group in GA4 that mat…
A developer-run census of 100 WooCommerce stores found that only 2 of 93 stores with a robots.txt file blocked any of nine major AI crawlers, and none blocked OpenAI's shopping crawlers, while product…
A check of 32 well-known websites' public robots.txt files on 7 September 2026 found that 16 block at least one AI search crawler, and 10 block both OAI-SearchBot, used by ChatGPT for citations, and P…
A 2026-09-07 census of 2,771 sites serving a robots.txt found that 368 block at least one of the three crawlers feeding AI answers, and 200 block all three. PerplexityBot is blocked by 358 sites, Clau…
A developer building an answer engine optimization site for SearchD implemented llms.txt, JSON-LD, and AI crawler allowances, then tested whether each actually delivers the promised citation benefits.…
A Cloudflare setting silently blocked AI crawlers from a digital marketing consultant's website, making it invisible to AI assistants for its entire existence, according to the consultant's own accoun…
A developer's analysis reveals that AI crawlers like GPTBot do not execute JavaScript, meaning client-rendered websites appear empty to them. Testing across 500 million GPTBot fetches by Vercel and ME…
Cloudflare's managed robots.txt blocks eight AI crawlers, but a developer's analysis reveals it excludes OpenAI's OAI-SearchBot and Perplexity's crawler, meaning ChatGPT search and Perplexity can stil…
Small businesses should adopt a nuanced AI crawler access policy rather than blanket allow or block decisions, according to guidance from Cloudflare, OpenAI, Perplexity, and Anthropic. The recommendat…
In a 30-day server-log census of dejan.ai's machine-readable files, Google, Anthropic and OpenAI crawlers fetched robots.txt 1,150 times, the OKF content bundle 23 times, and llms.txt 4 times (none fr…
TollBit's H1 2026 data found that ChatGPT-User ignored explicit robots.txt disallow instructions on 54% of its scrapes and reached disallowed pages on more European publisher sites than any tracked bo…
A developer rebuilding their local business website found that technical health alone is insufficient for AI search and Local SEO, and structured data must reflect visible content. The site achieved a…
A scan of 59 Chinese cross-border brand websites found that none block OAI-SearchBot, 67.4% serve an llms.txt file, and only 1.9% carry FAQPage schema. However, four out of five llms.txt files contain…
TIME is serving AI crawlers a stripped-down markdown version of its website with ads embedded, while human visitors receive the full HTML page, according to a test by an unnamed researcher. The markdo…
A developer's guide explains how to configure Next.js sites for Generative Engine Optimization (GEO), distinguishing between training crawlers like GPTBot and search bots like OAI-SearchBot. The artic…
Rudrendu Paul and Sourav Nandy released LLMScout, a zero-dependency CLI that runs 21 checks against any website to determine how well AI systems can find and cite it. The tool covers technical SEO fun…
A developer found that their site's table of contents had been invisible for months due to a legacy anchor mismatch, and that fix was part of a broader effort to get articles cited by AI search engine…
OpenAI has shipped its version of attribution for ChatGPT, but the signal is arriving through the ads door, not the webmaster-tools door, according to a survey and analysis by Duane Forrester. The det…
A developer found that roughly 27% of sites unintentionally block AI crawlers like GPTBot and ClaudeBot at the CDN or WAF layer, even when robots.txt appears permissive. The developer warns that clien…