AI Crawlers news and analysis on Web Pulse: 463 curated articles tracking the latest AI Crawlers developments, tools, and research, updated continuously from vetted sources.
A developer surveyed the robots.txt files of 100 major Malaysian websites on 20 October 2026 and found that 13 block at least one AI crawler, with none blocking all 17 checked and none blocking Googlebot. News sites acco…
Agents accounted for 471 million requests across Mintlify-powered sites in October 2026 versus 134 million human page loads — 78% of all traffic, up from 21% in February — according to Mintlify's 2026 State of Knowledge …
Reddit CEO Steve Huffman said the company will require all visitors to log in with a registered account, moving Reddit to "a primarily logged-in experience" to curb unauthorized scraping of its data for AI training and t…
The Copyright Alliance warned that Meta's revised commercial terms of service, due to take effect in November, still permit broad scraping of business-account websites and allow Meta to ignore or circumvent protocols lik…
A single threat actor impersonated roughly 32 AI bot identities, including OpenAI, Anthropic, Meta and Perplexity, while scanning enterprise web servers for secrets, according to Cequence Security researchers who publish…
A writer at The LAB replicated DataSocial's headless method for scraping metadata from TikTok's mobile API using GLM 5.3 on OpenRouter driven by the Hermes agent and the open-source Reverse Engineer Anything (REA) repo, …
The team behind veilvex.com, a four-language Astro static site, generates its llms.txt and robots.txt files at build time from the same i18n content files that render the pages, so prices and descriptions in the machine-…
A free tool announced on Show HN runs 16 checks on a submitted homepage to determine whether ChatGPT, Perplexity and Google AI can read, trust and cite the site, and reports which checks fail. The tool requires no signup…
Cloudflare launched a beta Web Search API through its AI Gateway, giving AI agents live web results from Ceramic.ai, Exa and Linkup via REST APIs and Cloudflare Workers, with unified logging, access controls and billing …
A developer released rankprobe, an MIT-licensed Python tool that audits a site for 36 SEO and AI-crawler checks and can fail a pull request in CI when issues like a stray noindex, an overlong meta description, or a robot…
A developer built a free, no-account llms.txt generator that reads a site's sitemap and produces a spec-style /llms.txt file in seconds, plus an optional llms-full.txt containing short excerpts of every page. The tool fo…
Vercel added an AI Bots managed ruleset to Vercel Firewall that lets customers log or block AI crawlers and AI assistants separately, the company said. AI crawlers are defined as bots that collect content for indexing an…
Skyfire has submitted KYAPay, an agent-identity protocol built around a digitally signed token that identifies an AI agent and the person or organization it acts for, to the IETF as six Internet-Drafts. The company says …
Meta has delayed an update to its ads and business terms that would have taken effect November 20, after criticism over language concerning scraping websites for AI training, company spokesman Andy Stone said. "We're com…
Roblox Corporation confirmed that an AI agent built by one of its researchers downloaded 145.5 GB of content, including 117.4 GB of adult material, from Open3DLab and its subsidiary sites Smutlab and SFMLab, and said the…
The UK Competition and Markets Authority is engaging directly with Meta "to establish the facts" over revised commercial terms that could let Meta crawl British businesses' websites for AI development, government ministe…
Decodo launched a Web Data API that combines scraping, search, mapping and crawling behind a single API key, aimed at AI startups building agents and retrieval pipelines. Decodo CEO Vaidotas Juknys said the product gives…
A study of the top 10,000 websites found that 29.8% of robots.txt files that block paths for all crawlers and name an AI crawler let that crawler past at least one of those blocks (329 of 1,105 files), according to the 2…
A first-party release check run by the desk between 10:49:33Z and 10:50:04Z on October 28, 2026 found that day-level release verification could not be completed at the source for four of six frontier lab surfaces. OpenAI…
Between May 5 and June 18, 2026, agents published more than 2,000 gems in the GemStuffer campaign, using .yardopts files to make RubyDoc.info execute supplied Ruby code that scraped public UK council portals and republis…