AI Crawlers news and analysis on Web Pulse: 463 curated articles tracking the latest AI Crawlers developments, tools, and research, updated continuously from vetted sources.
Cloudflare data shows the share of 403 "Forbidden" responses received by agents and bots has roughly doubled since the summer, as large etailers including Amazon block agent shoppers from their sites. Roughly 25% of surv…
Canonry launched as an AI visibility tracking platform that monitors whether ChatGPT, Claude, Gemini, Perplexity, Copilot, Grok, Meta AI, and DeepSeek name a brand, cite its site, or omit it, alongside the competitors th…
Cloudflare introduced a system that continuously evaluates Verified bots and assigns them an explainable trust band based on whether their observed traffic respects site preferences such as robots.txt and whether they re…
Meta's Muse AI shopping agent reached approximately 2.18 million daily active users in the U.S. by October 5, 2026, after launching September 8, and traffic matching Muse appeared at more than half of the customers studi…
A technical guide explains that OpenAI, Anthropic and Perplexity each operate multiple crawler user agents with distinct purposes, and that robots.txt rules can be set independently per agent. Training crawlers such as G…
A developer published a method for isolating traffic from AI assistants such as ChatGPT, Perplexity, Gemini, Copilot, Claude and DeepSeek in Google Analytics 4 and BigQuery. The approach adds a custom "AI assistants" cha…
Lightpanda released version 1.0 of its headless browser, a browser engine written from scratch in Zig for automation, crawling and AI agents that executes JavaScript without graphical rendering. The company reports 10,00…
A paper from three authors in the Department of Computing at Imperial College London, titled "Retrieved but Not Read: Understanding Why Search-Augmented LLMs Fail to Use Available Evidence," found that search-augmented L…
A new advisory from the NSA, CISA and the FBI (AA26-251A) details industrial-scale knowledge distillation campaigns that route requests through native APIs, cloud providers, third-party aggregators and proxy relays it ca…
ChatGPT-attributed sessions in Google Analytics 4 fell 22% to 57% across Graphite-tracked sites starting Oct. 9, a drop Graphite CEO Ethan Smith attributes to ChatGPT opening links without the utm_source=chatgpt.com tag …
Security firm Signals Corps found 30 sites attempting indirect prompt injection out of 217 million HTTP page responses sampled from Common Crawl, according to threat researcher Sarah Andrews. Of the almost 8,000 sites co…
EmDash released version 1.3 on its blog, nine days after 1.2, with 87 changes from 16 community contributors and the core team, headlined by a readable revision history that shows each saved version in the editor with ch…
A research pilot launched in July 2026 between Cloudflare and OpenAI has indexed hundreds of millions of previously unknown web pages into ChatGPT's search index, using signals from about 1% of Cloudflare's global networ…
DataDome's sample of traffic attributed to Meta's Muse found that 8.9% of requests reached forms, login, payment, account-creation or cart endpoints, but Jérôme Segura, VP of Threat Research at DataDome, told Let's Data …
A study of the robots.txt files of 285 of the most visited websites, 261 of which returned readable files, found that 35% block training crawlers such as GPTBot while only 21% block search-oriented agents like OAI-Search…
A developer outlined a practical framework for measuring brand visibility in AI-generated answers, arguing that reliable results require separating mentions, citations, and recommendations rather than treating every appe…
Cloudflare and OpenAI's research pilot, run on a cohort of roughly 1% of Cloudflare's total network, has enabled OpenAI to discover and index hundreds of millions of new web pages for ChatGPT Search, Cloudflare reported.…
US companies are being targeted by fake AI campaigns that impersonate trusted brands including Claude, ChatGPT, and Copilot to spread malware and steal sensitive data, according to the report. Attackers use sponsored sea…
SEO veteran Jon Henshaw released the Coywolf Pack, a modular plugin for the EmDash website platform that adds publishing, SEO, and performance features, with tested pages scoring a perfect 100 on PageSpeed Insights. The …
Researchers known as the Swarm Chasers published a report on October 14 describing how they made contact with a fleet of AI agents running on Tencent Cloud that were using the public security-scanning service urlquery to…