AI Crawlers news and analysis on Web Pulse: 461 curated articles tracking the latest AI Crawlers developments, tools, and research, updated continuously from vetted sources.
OpenAI, Google, Microsoft and Anthropic agreed on October 30, 2026 to honor a single robots.txt token, `__usage_ai-training`, that publishers can use to request their content not be used for generative model training, ac…
A developer built AI Tools List, a directory designed for both autonomous agents and human users, exposing /llms.txt, /llms-full.txt and /ai.json endpoints plus an OpenAPI 3.0.3-documented submission API at /api/agent/v1…
A founder reported on Hacker News that Gmail's spam filters are failing to catch continuous streams of clearly AI-generated unsolicited outreach email, triggering repeated phone notifications throughout the day. The foun…
Google announced on October 30, 2026 that it will support a shared AI training control in robots.txt, built on work from the IETF's AIPREF working group, with Microsoft and OpenAI reported to honor it and Anthropic build…
A developer ran a free AI visibility audit on their own site, onyxailabs.me, and found that all 100 AI answers from ChatGPT, Google AI Overviews, Gemini, and Google AI Mode described a different company — an enterprise A…
Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC announced plans to support an upcoming website crawling standard that would let site operators block AI developers from using their content to train models,…
Clear Rank Studio ran its own free AI-readiness checker against PlainData, its calculators site, and reported nine passing checks, three warnings and a missing sitemap.xml that returned a 404. The team added a sitemap wi…
Arcjet cited a September study by an independent researcher that captured 37 requests from ten AI assistants to a Cloudflare Worker, using the researcher's DuckAssistBot capture alongside SeatGeek's ChatGPT example in it…
OpenAI, Google, Microsoft and Anthropic all committed on Friday to honouring a single new robots.txt policy token, `__usage_ai-training`, that lets site owners opt out of AI training without blocking each company's crawl…
OpenAI, Google, Microsoft and Anthropic will adopt a shared "policy token" signal in robots.txt that lets publishers apply AI training opt-out preferences across multiple AI companies instead of setting them individually…
Google announced it plans to voluntarily adopt an IETF AIPREF draft proposal that would let publishers express AI training preferences across multiple AI providers through a single shared "policy token" in robots.txt, wi…
A developer launched a static site on 15 October and logged 9,987 crawler requests over the following 15 days, finding that Anthropic's ClaudeBot and OpenAI's GPTBot and OAI-SearchBot arrived within roughly 65-70 minutes…
Construct shipped Markdown twins for all 92 pages in its sitemap on October 26, 2026, serving each page's clean Markdown version at its path plus .md (for example, /pricing.md) alongside an llms.txt, a robots.txt naming …
A Wix infrastructure engineer built a personal blog pipeline in which Claude Code drafts each post from the engineer's PRs, notes and logs, while Wix Headless handles hosting, domain, subscribe form and contact list. The…
FourA's October 23–30, 2026 changelog adds check-page handling for Reddit, eBay and Imperva to its scraping API, so Single, Browser and Proxy Finder name those pages in the X-FourA-Check-Page header and do not bill them,…
Google stopped appending UTM parameters to URLs cited in Gemini responses around September 28, roughly three weeks after it began adding utm_source=gemini tags in early September, according to tracking by Natzir and corr…
A developer published a guide for self-hosted WordPress operators to detect AI crawler traffic in a site's first week, recommending a custom nginx or Apache log format that captures ISO-8601 timestamps, full user agents,…
A developer built a free AI SEO checker that audits any public URL in seconds for signals AI answer engines like ChatGPT search, Perplexity and Google AI Overviews rely on, returning a score out of 100 with fixes for eac…
A developer surveyed the robots.txt files of 100 major Malaysian websites on 20 October 2026 and found that 13 block at least one AI crawler, with none blocking all 17 checked and none blocking Googlebot. News sites acco…
Agents accounted for 471 million requests across Mintlify-powered sites in October 2026 versus 134 million human page loads — 78% of all traffic, up from 21% in February — according to Mintlify's 2026 State of Knowledge …