A developer has published a WordPress mu-plugin that logs visits from nine documented AI crawler user agents, after finding that Google-Extended and Applebot-Extended never appear in access logs becau…
Websites can now block AI training crawlers such as GPTBot while remaining discoverable to search indexers by configuring granular robots.txt rules and HTTP headers, according to Cloudflare. Google-Ex…
A new open-source tool called robots-check reveals that Cloudflare's AI Crawl Control feature silently appends its own rules to websites' robots.txt files, often blocking AI crawlers like GPTBot and G…
A developer detailed a free website SEO audit process that uncovered 31 broken pages and 900 phantom URLs on an inherited site, leading to improved crawl efficiency after fixes. The audit uses free to…
Indexflow-SEO, a lightweight SEO analyzer written in Rust, has been released as version 0.1.2 on crates.io, offering zero-dependency technical SEO validation and Generative Engine Optimization (GEO) a…
GreyNoise reported that automated scanners forged the web crawler identities of OpenAI, Anthropic, DeepSeek, Google and Perplexity to request credentials and secrets, with six forged AI crawler names …
A developer found that most AI crawler traffic on their site is fake, with only 468 of 1,459 requests actually fetching articles. Using Cloudflare's verifiedBotCategory, they showed that only 13% of G…
Google has rolled out new Publisher Center controls and expanded the Google-Extended user agent, allowing publishers to block their content from AI Overview citations via robots.txt while remaining in…
Cloudflare's managed robots.txt blocks eight AI crawlers, but a developer's analysis reveals it excludes OpenAI's OAI-SearchBot and Perplexity's crawler, meaning ChatGPT search and Perplexity can stil…
Prominara's 2026 GEO guidance recommends blocking GPTBot, ClaudeBot, and Google-Extended via robots.txt and X-Robots-Tag headers to prevent model training while allowing citation/search agents, preser…
A developer analyzed Cloudflare logs from their own site over a 23.5-hour window and split AI bot traffic into three categories: live answers (189 fetches), search indexing (541 fetches), and training…
Nearly one in six U.S. trending news queries now place Top Stories carousels inside Google AI Overviews, according to NewzDash CEO John Shehata, with 15.5% of U.S. and 17.46% of UK queries showing the…
New data from NewzDash shows that 15.5% of U.S. and 17.46% of U.K. news searches where Google displayed Top Stories placed the carousel inside AI Overviews, a single placement that never appeared alon…
A developer found that their site's table of contents had been invisible for months due to a legacy anchor mismatch, and that fix was part of a broader effort to get articles cited by AI search engine…
Isvisible.ai has launched a free tool that lets website owners check whether AI crawlers from ChatGPT, Claude, Perplexity, Google AI and others can access their site, returning an instant 0-100 visibi…
A growing share of people search on ChatGPT, Perplexity and Gemini before buying or choosing a supplier, making Generative Engine Optimization (GEO) essential for websites. GEO rests on concrete signa…
The team behind Chat Thing has launched a free Agent Readiness Checker that scans websites and scores how accessible, citable, and actionable they are for AI agents and LLMs, grouping findings into th…
A developer at a small digital marketing agency built installable Claude skills that package accumulated professional judgment rather than relying on static prompts. The system distinguishes between s…
Cloudflare announced plans to automatically block mixed-use web crawlers that index sites for search engines and also serve AI companies, starting September 15, 2026. The hosting platform will default…
A new web standard, AI-aware robots.txt, allows site owners to control which AI crawlers can access their content, preventing silent exclusion from AI training data and answer corpora. The article pro…