Cloudflare launched granular AI traffic controls this month, replacing its single 'block AI bots' toggle with three independently configurable categories—Search, Agent, and Training—available to all c…
A developer's analysis of 2025-2026 data from Cloudflare, Vercel/MERJ, and Ahrefs reveals that major AI crawlers—including OpenAI's GPTBot, Anthropic's ClaudeBot, and PerplexityBot—do not execute Java…
A developer found that their site's table of contents had been invisible for months due to a legacy anchor mismatch, and that fix was part of a broader effort to get articles cited by AI search engine…
A developer added AI-readability checks to the open-source SEO tool open-seo, detecting whether a site blocks AI agents like GPTBot and ClaudeBot or supports the emerging llms.txt standard. The featur…
A developer discovered that CDN configurations, not robots.txt, often block AI crawlers like GPTBot and ClaudeBot from accessing documentation. Testing revealed that Cloudflare's security settings can…
A developer found that roughly 27% of sites unintentionally block AI crawlers like GPTBot and ClaudeBot at the CDN or WAF layer, even when robots.txt appears permissive. The developer warns that clien…
Isvisible.ai has launched a free tool that lets website owners check whether AI crawlers from ChatGPT, Claude, Perplexity, Google AI and others can access their site, returning an instant 0-100 visibi…
A growing share of people search on ChatGPT, Perplexity and Gemini before buying or choosing a supplier, making Generative Engine Optimization (GEO) essential for websites. GEO rests on concrete signa…
SeekinWeb, a new tool from an unnamed startup, lets website owners check whether AI agents such as ChatGPT, Claude, and Perplexity can read their site, providing an AI Visibility Score from 0 to 100 b…
The team behind Chat Thing has launched a free Agent Readiness Checker that scans websites and scores how accessible, citable, and actionable they are for AI agents and LLMs, grouping findings into th…
Causabi, a generative engine optimization tool, analyzed 44 domains and found that on-site SEO scores have virtually no correlation with AI citation rates (Pearson r ≈ -0.08). Brand prominence was the…
A developer at First Step Technology deployed AI Traffic Monetization on AWS to charge AI bots for crawling content, using the API/CLI-only feature after discovering the WAF console lacks the monetize…
A developer argues that in 2026, websites should welcome AI crawlers for Answer Engine Optimization (AEO), not block them. They detail the distinct bots from OpenAI, Anthropic, and Perplexity for trai…
Sourceable ran its own AI agent-readiness scanner on its website besourceable.com on July 8, 2026, scoring 78 out of 100. The site achieved perfect scores on Discoverability, Content Accessibility, an…
A study of the top 10,000 websites' robots.txt files found that 38% of GPTBot rules were written in the last quarter of 2023, following the crawler's launch and the New York Times lawsuit. Most sites …
GPTBot, OpenAI's web crawler, is the most blocked bot among websites with organic traffic, according to data from Site Stats Database tracking 880,858 sites. Over 45,000 websites block GPTBot, surpass…
A developer at a small digital marketing agency built installable Claude skills that package accumulated professional judgment rather than relying on static prompts. The system distinguishes between s…
A developer's experiments reveal that including a URL in a prompt can steer an LLM's output toward the content at that URL, but only if the URL and its content were part of the model's training data. …
A new web standard, AI-aware robots.txt, allows site owners to control which AI crawlers can access their content, preventing silent exclusion from AI training data and answer corpora. The article pro…
A developer discusses how AI crawlers and robots.txt policies are fragmenting the web's shared information environment. As websites selectively allow or block AI systems like ClaudeBot and GPTBot, ret…