Developer Marco Bellingeri opened his website's robots.txt to all AI crawlers to boost visibility, but after receiving 33,561 pageviews in its first month, he realized the number was inflated by bots …
Applighter, a two-person team selling React Native templates, has published 118 blog posts without hiring any writers by building an automated pipeline that uses Claude Code for drafting and a webhook…
An engineer's analysis of 30 days of server logs reveals that 71% of AI crawler traffic is spoofed, with 916 of 1,290 attributable requests coming from sources not operated by the claimed vendors. The…
A developer audited ten websites they scrape for AI-related content and found that only two had licenses permitting commercial reuse, despite several having permissive robots.txt files. The audit reve…
A developer analyzed Cloudflare logs from their own site over a 23.5-hour window and split AI bot traffic into three categories: live answers (189 fetches), search indexing (541 fetches), and training…
Bruce Ediger, a software developer, describes the offensive security measures he deploys on his blog's VPS to deter malicious bots and scrapers, including a fake WordPress login, an endlessh tarpit on…
Major publishers including the BBC, The Guardian, and The New York Times are blocking OpenAI's GPTBot from accessing their content, signaling a broader shift in how news organizations assert control o…
A developer's guide explains how to configure Next.js sites for Generative Engine Optimization (GEO), distinguishing between training crawlers like GPTBot and search bots like OAI-SearchBot. The artic…
Rudrendu Paul and Sourav Nandy released LLMScout, a zero-dependency CLI that runs 21 checks against any website to determine how well AI systems can find and cite it. The tool covers technical SEO fun…
AI crawlers from OpenAI, Anthropic, and Perplexity do not execute JavaScript, leaving client-rendered single-page apps invisible to them, according to an audit by developer and product builder who run…
Content Signals, a proposed extension to robots.txt backed by the IETF AI Preferences working group and the IAB Tech Lab, adds directives to declare whether AI crawlers may search, ingest, or train on…
A developer's analysis of 2025-2026 data from Cloudflare, Vercel/MERJ, and Ahrefs reveals that major AI crawlers—including OpenAI's GPTBot, Anthropic's ClaudeBot, and PerplexityBot—do not execute Java…
A developer found that their site's table of contents had been invisible for months due to a legacy anchor mismatch, and that fix was part of a broader effort to get articles cited by AI search engine…
A developer added AI-readability checks to the open-source SEO tool open-seo, detecting whether a site blocks AI agents like GPTBot and ClaudeBot or supports the emerging llms.txt standard. The featur…
A developer discovered that CDN configurations, not robots.txt, often block AI crawlers like GPTBot and ClaudeBot from accessing documentation. Testing revealed that Cloudflare's security settings can…
A developer found that roughly 27% of sites unintentionally block AI crawlers like GPTBot and ClaudeBot at the CDN or WAF layer, even when robots.txt appears permissive. The developer warns that clien…
Isvisible.ai has launched a free tool that lets website owners check whether AI crawlers from ChatGPT, Claude, Perplexity, Google AI and others can access their site, returning an instant 0-100 visibi…
A growing share of people search on ChatGPT, Perplexity and Gemini before buying or choosing a supplier, making Generative Engine Optimization (GEO) essential for websites. GEO rests on concrete signa…
SeekinWeb, a new tool from an unnamed startup, lets website owners check whether AI agents such as ChatGPT, Claude, and Perplexity can read their site, providing an AI Visibility Score from 0 to 100 b…
The team behind Chat Thing has launched a free Agent Readiness Checker that scans websites and scores how accessible, citable, and actionable they are for AI agents and LLMs, grouping findings into th…