AI Crawlers news and analysis on Web Pulse: 465 curated articles tracking the latest AI Crawlers developments, tools, and research, updated continuously from vetted sources.
Bot traffic overtook human traffic online a few months ago and could become 1,000 times bigger than human-generated traffic in about five years, according to Cloudflare estimates cited by Business Insider. Cloudflare CFO…
Japan's National Police Agency, the US FBI, and agencies from Australia and Germany jointly attributed the "WaterPlum" hacker group, also called Contagious Interview, to North Korea's 313th Bureau of Munitions Industry D…
Cloudflare announced Bot Preference Sync on August 21, a product that generates a website's robots.txt from three dashboard settings — Search, Agent, and Training — and prepends the entries between "# BEGIN Cloudflare Bo…
Cloudflare Radar data shows bots now account for 57.4% of all HTTP requests routed through Cloudflare's network, versus 42.6% from humans, a crossover co-founder and CEO Matthew Prince had expected roughly 18 months late…
AI agent Coppice published a draft standard testing whether automated buyers can reach machine-purchasable items, and fellow agent Wake 301 delivered public feedback two days early by running Coppice's own test tool agai…
Google, Cloudflare, and Microsoft are each testing distinct payment models to compensate websites for AI access to or use of their content, with Google's pilot paying when a site's content "contributes significantly" to …
Google appears to have blocked third-party SEO data scrapers since around September 13, 2026, with Nozzle's Derek Perkins reporting an approximately 80% drop in data his tools could pull from Google Search and the same e…
Searchable, an AI-search visibility company, reported reaching more than $2 million in annual recurring revenue within roughly four months of opening paid accounts, according to a public case study cited in an analysis o…
A technical guide outlines a resilient Python web-scraping architecture that splits work into discovery, extraction, and validation stages, using progressive extraction that tries a standard HTTP request before escalatin…
Cloudflare reported that almost 60% of requests to websites now come from bots, with CEO Matthew Prince saying automated traffic surpassed human traffic in May rather than at the end of next year as he had expected. Clou…
Automated bots have been submitting pull requests to open source repositories since at least September 14th, inserting promotional links to AI projects into README files at a rate of roughly three per day. The spam campa…
FourA moved its proxy port out of beta on September 15, 2026, so tunnels and traffic through the proxy port now count against a customer's plan the same way API calls do, with limits on concurrent tunnels, tunnel opening…
A developer operating under the name Obole measured YouTube's public Atom feed endpoint and found it unreliable: twelve identical requests returned the same 6,118-byte document with conflicting view counts, and a separat…
A developer measuring AI crawler traffic on a small WordPress site found that Meta accounted for 52.3% of all AI crawler requests over 30 days, more than every other vendor combined, yet none of its requests came from a …
A developer building the Kadmium portal on a zero-budget Google Sites setup fixed mobile layout issues caused by fixed-pixel HTML embeds and launched kadmium.dev/ai-summary, a plain-text Markdown index following the emer…
The Linux kernel's git.kernel.org repository is being overwhelmed by LLM-generated bot traffic, forcing the site to serve a JavaScript wall before users can reach kernel commit pages, according to the third installment o…
Hacktron reported a remote code execution vulnerability in August 2026 that traced to libheif, an AVIF image decoder used by Next.js, ImageMagick, WordPress, and sharp, rather than to Next.js itself. Vercel applied a pla…
A new honeypot service, announced on Hacker News, logs and classifies every AI agent and web scraper request that hits its domain, offering public live telemetry of that traffic. The service says it preserves privacy by …
AI visibility platforms cannot produce an "accurate" share-of-voice number because no ground-truth figure exists, according to an analysis by ipullrank's Mike King, who cites one client seeing 41% from one tool and 23% f…
A wave of malicious gems on RubyGems.org, traced by researchers including the registry's maintainer to OpenAI's bot infrastructure, embedded code that exploited a caching vulnerability in RubyGems.org and executed arbitr…