Steel, a browser-infrastructure startup for AI agents, found that competitors bid on its brand name in Google Ads while AI agents evaluating the product never see those ads, according to a first-perso…
A blogger has confirmed that their website is fully deindexed from Google after a process that took at least 3 months, following an initial blanket robots.txt disallow that had no effect. The blogger …
An autonomous AI agent running on a GPU-less AMD Ryzen 5 3600 server with 62GB of RAM has published ten articles and built a suite of free, open-source tools at k1r4.space, reporting that its main obs…
A developer outlines a framework for deciding whether to block AI crawlers such as GPTBot, ClaudeBot, CCBot, PerplexityBot and Bytespider, arguing that a selective policy beats a blanket allow-or-bloc…
A developer's server-log analysis of sightread.org found that requests to the llms.txt agent-guidance file ran at just 1–1.2% of requests to robots.txt over five days, with a single unidentified Chrom…
A developer building the FoxyInvoice invoicing app documented a distribution stack that serves crawler-ready static HTML for its Angular SPA, publishes an llms.txt file for AI answer engines, and push…
Cloudflare announced Bot Preference Sync on August 21, a product that generates a website's robots.txt from three dashboard settings — Search, Agent, and Training — and prepends the entries between "#…
Websites can now block AI training crawlers such as GPTBot while remaining discoverable to search indexers by configuring granular robots.txt rules and HTTP headers, according to Cloudflare. Google-Ex…
Cloudflare launched Disallow AI Training on September 15th, 2026, a setting that lets website owners publish a no-training preference in robots.txt while keeping Applebot, Googlebot and Bingbot availa…
Buttondown replaced its raw user-agent-string textarea with a Crawling add-on in Settings → Archives that gives users one toggle per crawler category: Search engines (Googlebot, Bingbot, DuckDuckBot, …
A developer's analysis reveals that AI crawlers like GPTBot do not execute JavaScript, meaning client-rendered websites appear empty to them. Testing across 500 million GPTBot fetches by Vercel and ME…
Cloudflare will block AI Training and Agent crawlers by default on ad-supported pages for new domains and free-tier customers starting September 15, 2026, according to a July 1 announcement. The compa…
Linux kernel developer Konstantin Ryabitsev said that coping with AI-generated 'slop' from LLM bots has become so expensive that even Linux infrastructure struggles, noting that kernel.org and related…
A developer warns that many AI crawlers, including GPTBot, ClaudeBot, and PerplexityBot, do not execute JavaScript, making client-rendered websites invisible to them. The post advises using curl to ch…
A 24-hour census of a single MCP endpoint recorded 60 distinct named crawler agents making 6,309 requests, including roughly 600 that hit a /v1/buy/* route and received a valid HTTP 402 payment challe…
A 41-day experiment reported by Search Engine Land found that JavaScript-powered navigation can hide website pages from AI crawlers. GPTBot and ClaudeBot discovered zero pages available only through J…
A developer shipping a static Astro site to Cloudflare Pages discovered three platform defaults that silently broke the site on launch day. Cloudflare's managed robots.txt injected AI crawler blocks, …
A 41-day field experiment by SEO engineer Vinicius Stanula found that JavaScript-injected navigation creates a discovery gap for AI crawlers. Pages linked only through JavaScript were not discovered b…
An engineer's analysis of 36 days of server logs reveals that Bing's crawler, bingbot, now hits his site 4.2 times more than Googlebot, and that ChatGPT's citations largely derive from Bing's index. C…
Starting September 15, 2026, Cloudflare will block Googlebot and Bingbot on sites set to block AI training, potentially affecting search indexing. Resoneo found that ChatGPT's own index serves sites w…