AI Crawlers news and analysis on Web Pulse: 465 curated articles tracking the latest AI Crawlers developments, tools, and research, updated continuously from vetted sources.
DataDome's State of Bot & Agent Security Report, 2026 Edition found that 313 million AI requests reached login pages in the first half of 2026, equal to 51.7% of the 605.6 million requests to high-risk endpoints and abou…
A developer released AIFeed, a protocol that lets publishers sign a small JSON manifest with Ed25519 and anchor the key in DNS so AI crawlers can cryptographically verify content permissions rather than relying on robots…
A developer's AI crawler dashboard recorded 20 requests for the site's /.env secrets file over seven days, all claiming to be ClaudeBot, while only 68% of ClaudeBot traffic that week could be verified as coming from Anth…
MotherDuck published an analysis of Common Crawl's web archive, which has collected roughly 2.2 billion pages and about 150 TiB per monthly crawl since 2008, to measure how much personal, platform-hosted publishing has g…
Semrush released an expanded 2026 AI Visibility Index analyzing 126 million AI search prompts, finding that the brands AI systems mention and the websites they cite frequently diverge. Across ChatGPT and Google AI Mode, …
The website hosting the Radxa E52C data entry is blocking visitors with an Anubis proof-of-work challenge, a Hashcash-style scheme the site's administrator deployed to stop AI companies from aggressively scraping the ser…
Amazon has blocked Meta's Muse AI agent from shopping on its site, showing users a pop-up notice that "Continued access by an unauthorized AI agent violates Amazon's Conditions of Use," according to GeekWire. Meta launch…
Trendos Chief Commercial Officer Gintare Rimolaityte told a webinar that brands should map which sources AI answer engines cite by industry and engine rather than rely on search rankings alone, citing summer 2026 example…
A developer behind FreeModel ran 27 developer-style questions through four AI engines on a single day and analyzed the 613 citations returned, finding that the ten most-cited domains account for only 18.6% of citations w…
HUMAN Security reported AI-driven traffic grew 187% in 2025 and AI agents plus agentic browsers grew 7,851% year over year, according to the company's 2026 State of AI Traffic & Cyberthreat Benchmarks. The analysis argue…
Cloudflare blocks AI crawlers by default, Whitespark founder Darren Shaw warned during a Search Engine Journal Live session with Duda Director of Platform and Product Strategy Russ Jeffery, citing a Shopify site where Cl…
A new free AI visibility audit tool scores websites from 0 to 100 on how clearly generative models parse their entity data, with breakdowns across ChatGPT Search, Google AI Overviews, Claude, and Perplexity. The tool aud…
A developer audited 15 top-ranking guides on LLM SEO, AI visibility, and ranking in ChatGPT, finding that none of the five schema-markup guides cited a source for claims that structured data drives AI citations, while Ah…
Apify's RAG Web Browser retrieved 564,715 characters across 40 results versus Tavily's 67,472 characters across 40 results in a four-query benchmark on consumer computer price hikes tied to a memory shortage, according t…
A developer has published a WordPress mu-plugin that logs visits from nine documented AI crawler user agents, after finding that Google-Extended and Applebot-Extended never appear in access logs because neither sends HTT…
Newly unsealed internal emails and analysis in The New York Times' lawsuit against OpenAI and Microsoft show Microsoft recorded 83-93% drops in click-through rates for The New York Times' and Daily News' domains when com…
Vincent Bernat replaced GoAccess with GoatCounter, an open-source, privacy-friendly web analytics platform, on his NixOS-hosted blog after AI scrapers inflated GoAccess's visitor counts to around 2,000 per day. Bernat wr…
Jeremy Howard, the fast.ai founder and former Kaggle president, proposed llms.txt in September 2024 as a plain-text Markdown map that gives AI models a curated overview of a website's most important pages. Adoption has s…
Attackers used a single hardcoded Cloudflare API key to hijack Brevo's CDN edge for five and a half hours on September 14, serving fake Cloudflare verification pages and silently installing a backdoor WordPress plugin ac…
OpenAI's ad measurement system sets a first-party cookie called __obi on the .openai.com domain that is sent back to OpenAI from advertiser websites running its measurement pixel, letting the company tie browsing and pur…