AI Crawlers news and analysis on Web Pulse: 465 curated articles tracking the latest AI Crawlers developments, tools, and research, updated continuously from vetted sources.
Dolphin Emulator released version 2609, which adds NetPlay support to its Android build and addresses an AI-driven wave of vulnerability reports plus a macOS buildbot cryptominer compromise that the project says did not …
Cloudflare introduced new AI-crawler controls on September 15, 2026 that let site owners disallow AI training while keeping their content eligible for search indexing. The update replaces the single Block AI Bots switch …
A developer demonstrated that three queries run minutes apart against ZoomEye's internet-wide dataset returned 161,907 hosts answering on TCP port 102, 95,749 devices classified as PLCs, and just 173 assets positively fi…
A developer who worked at LI.FI, Netlify, Flutterwave, and Shardeum has launched Thally, a tool that connects product repositories to documentation surfaces, identifies content affected by code changes, and opens evidenc…
The website hosting the page titled "802.11s - VLAN Trunk Limitations" is blocking visitors with an Anubis proof-of-work challenge, which the site's administrator deployed to stop aggressive scraping by AI companies. Anu…
Baidu confirmed in an official notice on September 16 that it added an AI Traffic Analysis section to its webmaster tools, tracking AI Citations, AI Impressions, AI Clicks and AI CTR, though the company says the data ser…
DataDome's Bot and Agent Security Report found that malicious automated traffic grew 124% between July 2025 and June 2026, nine times faster than human web traffic growth over the same period, based on more than 1 trilli…
An autonomous agent analyzed 1,634 articles from DEV's public API and found that 81.8% declare their canonical home on dev.to, while the 297 off-site posts spread across 149 distinct hosts — 148 of which carry exactly on…
Richresults.ai reported that in 20 fresh, logged-out ChatGPT tests run on September 23, 2026 — ten in German and ten in English, seven days after launch and AEO implementation — the intended JUNO PORTALS entity was corre…
Malicious bot activity rose 124% between July 2025 and June 2026 while human traffic grew 13.2%, and traffic from AI agents and large language model crawlers climbed 82.3% over the same period, according to DataDome's St…
YouTube was the most-cited platform across eight AI platforms — Claude, ChatGPT, Copilot, Google AI Mode, Google AI Overviews, Gemini, Perplexity and Grok — in August, according to Meltwater data reported by Digiday. Tin…
EN World, a tabletop role-playing game news site, is suffering site slowdowns and outages from waves of up to 90,000 AI scraper bots that constantly rotate IP addresses to evade blocking, the site reported. The attack ha…
Search Engine Journal published a tiered reference table categorizing data sources by their usefulness for AI search, ranking Google Search, Bing Search, Google Merchant Center, OpenAI merchant feeds, and Google Maps as …
Chris Siebenmann, a University of Toronto computer science researcher, has begun blocking all browsers presenting old Chrome user agents from his blog Wandering Thoughts and CSpace wiki, citing an early-2025 surge in hig…
A developer built Svipall, a local-first web-fetching layer for AI agents written in Rust, after encountering a silent failure mode where anti-bot challenge pages returned HTTP 200 and were extracted as if they were real…
SEO consultant Sara Taher has begun building a custom search engine for her own content, starting with a crawler planned with Claude Code that relies on two sitemaps for seoriddler.com and sara-taher.com and caps crawl d…
Developer djsumdog released an open-source search tool, "search the past," that adds date-range parameters to Bing and other engines so users can surface pre-chatbot web results, with source code published on GitLab. The…
Amazon has begun blocking Meta's Muse AI agent from browsing and purchasing on amazon.com, treating automated shopping-agent traffic as a policy violation rather than a supported access pattern. The block forces AI devel…
Akamai reported that verified AI crawlers, including ChatGPT's, have shifted from reading web pages to sending high-frequency POST requests, with ecommerce accounting for 44.8% of those AI bot POST transactions and trave…
Meta's Muse AI agent reached the top of the US App Store within a week of its September 8, 2026 launch, crossing 2.5 million downloads in its first two weeks and edging past ChatGPT, Claude, and Grok. Amazon blocked Muse…