AI Crawlers news and analysis on Web Pulse: 463 curated articles tracking the latest AI Crawlers developments, tools, and research, updated continuously from vetted sources.
Ahrefs analyzed 66.6 million free, logged-out ChatGPT responses from August and September 2026 and found that only 14.4% triggered a live web search, with the remaining 85.6% answered from training data alone. The analys…
The European Data Protection Board adopted draft Guidelines 03/2026 on web scraping in the context of generative AI on 7 July 2026, with public consultation closing 30 October 2026, drawing a compliance line between targ…
Roblox confirmed that an AI agent built by one of its researchers "mistakenly scraped" content from Open3DLab, a free 3D model site, and said the agent was decommissioned and the data was not used for any purpose, includ…
Discord announced that discoverable servers will get public server web pages starting November 16th, with channel messages becoming viewable on those pages from mid-December, as the company said in a blog post that valua…
A developer documented how Phenom-powered careers sites for RTX, UPS and CVS Health expose their full job listings through an unauthenticated POST to the /widgets endpoint, which robots.txt does not block. Testing on 202…
A documentation writer is reworking their site's pages to be readable by coding agents, removing JavaScript-driven hidden content because agent fetch tools make an HTTP request, convert HTML to Markdown, and never run Ja…
Meta Platforms Inc. delayed implementation of its new commercial terms of service until Nov. 20 after removing the word "train" from the language within a day of a backlash from U.S. publishers, according to columnist Va…
A developer published a teaching example showing how to keep a catalogue page small while preserving crawlable navigation: the server returns only the current page's items and total count, full public metadata is fetched…
A robots.txt audit of 945 well-known websites, completed 22 October 2026 and repeated monthly, found that 25.9% (193 of the 746 sites whose robots.txt could be read) block at least one of 14 AI crawlers, while 9.2% (69 s…
A two-month campaign tracked by Check Point Research uses fake GitHub repositories impersonating AI desktop apps — GPT-5.6 Sol, GPT-6 Astra and Claude Opus — to deliver a custom malware chain the researchers call Yoba, w…
Encore modeled the most popular personal AI assistants and sent them to visit more than 1,500 organizations across 25 industries, finding that as of Oct. 23, 94% of sites could not accept an agent connection at all and 2…
The Estonian Association of Media Companies has urged the European Commission to intervene against Meta's new business terms, which the association says would let Meta index, store, search and update publishers' web cont…
A developer released CatalogAudit, a free tool that checks Shopify stores against Google Merchant Center's product data rules and tests whether AI shopping agents such as ChatGPT, Perplexity and Google AI Mode can read p…
A first-hand test of 13 real buyer questions in ChatGPT found that seven answers carried sources and 14 links in total, none of which pointed to the testing site, according to the site's own server logs and citation trac…
A developer released Hatamoto, a batteries-included content management system written in Go that ships as a single binary alongside Postgres with no PHP, Node or Redis dependencies. Hatamoto bundles a page builder, SEO, …
A test run on 26 October 2026 of 49 runs by 23 AI agents, chatbots and developer tools against a purpose-built three-page site found that most browser-using agents look exactly like a person in Chrome, reporting no autom…
Developer Quentin Wendegass tested detection of AI agents on a website by sending 23 of them to the site and found that most presented as human traffic, concluding that reliable agent detection is difficult. The hands-on…
Meta pushed the effective date of its revised terms of service from October 30 to November 20 after legal experts and publisher groups objected to language letting Meta use automated tools to "access, index, cache, analy…
A developer guide outlines how sites can be crawled and cited by AI systems, recommending that operators allow search-focused AI crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot in robots.txt while opti…
A developer published an .htaccess configuration snippet combining GZIP compression rules with a user-agent blocklist, labeled "Ultimate AI Block List v1.8," that uses mod_rewrite conditions to deny requests from AI craw…