AI Crawlers news and analysis on Web Pulse: 465 curated articles tracking the latest AI Crawlers developments, tools, and research, updated continuously from vetted sources.
Clawptcha launched a reverse CAPTCHA service that is designed to be solved by bots rather than humans, requiring automated clients to factor a number and submit the factors to verify.clawptcha.com. Websites embed the wid…
Mdjango is a new drop-in Django app that serves a technical documentation site from a directory of markdown files, including header, section navigation, table of contents, prev/next links, full-text search, dark mode, an…
A developer operating under the name Obole, an AI running on a two-core ARM server, found that a custom "real reading" analytics metric was never once triggered by an identified human reader — the scroll-depth event fire…
A 2026 guide on auditing websites argues that site audits must shift from static annual checklists to ongoing, AI-informed diagnostic cycles covering four dimensions: technical infrastructure, content health, user experi…
Fastly Bot Management and ContentGuard now cover AI scraper traffic hitting both origin servers and CDN cache, according to Fastly, which reported that AI requests on its platform grew roughly 6.5x faster than human traf…
An AI agent's hand-kept machine-readable site map drew referrals from three separate AI answer engines in one month — ChatGPT, Microsoft's Bing and Copilot, and Perplexity — totaling eight visits (three, three, and two),…
Answer engine optimization (AEO) is an emerging practice of making content easier for AI systems to understand, cite, and recommend in generated answers, according to a guide that notes the terminology remains unsettled …
A San Francisco County Superior Court judge allowed three of Reddit's five claims against Anthropic to proceed in an order dated Sept. 17, overruling Anthropic's demurrer and advancing breach of contract, interference wi…
Microsoft Chief Scientist Kate Crawford warned that AI scraping amounts to "the largest theft of labor in human history," describing how machine-learning-powered scrapers evade detection and adapt to site layouts in ways…
AI agents accounted for 66% of measured web traffic across Mintlify-powered documentation in July 2026, according to Mintlify, driving demand for an emerging role the company calls the "knowledge engineer." The article c…
Newly unsealed court filings in The New York Times' copyright lawsuit against OpenAI and Microsoft reveal that Microsoft's director of applied science, Brent Hecht, called large-scale copying of online content for AI tra…
Google is piloting a program that pays participating publishers when their content "contributes significantly" to answers in the Gemini app, AI Overviews, and AI Mode, with at least dozens of publishers approached, accor…
A website hosting documentation for the XikeStor SKS8300-8X switch (also sold as the ONTi ONT-S508CL-8S, JT-COM JT-FG6700-8TFM, and Hicomdata HK-ES3058-8P-L) is blocking visitors with an Anubis proof-of-work challenge in…
Two independent investigations found that ZCode, a GLM-based coding agent, has been quietly uploading users' git history and full workspace snapshots to remote servers. The findings prompted a warning that users should c…
GrowNexus audited 25 major SaaS platforms to measure industry adoption of the llms.txt v2 specification, which was formally updated in August 2026 to add <link rel="describedby"> tags. The team found that the most signif…
An analysis of AI-generated source citations found that AI systems increasingly cite websites that appear to exist solely to be cited by AI, rather than established or niche publications. The author reports barely recogn…
A developer outlined a repeatable method for measuring whether AI answer engines such as ChatGPT, Perplexity, Claude, Gemini, and AI Overviews cite a website, arguing that citations differ from traditional search ranking…
A developer shipped Safe Space, a multilingual Flutter mental-health self-help app that routes AI chat through Google's Gemini first with Anthropic's Claude as a fallback, backed by mood journaling, CBT-style thought rec…
Search Engine Journal published a guide showing that prompting an AI chatbot such as ChatGPT to search for an exact text snippet can reveal whether a page is part of the retrieval pipeline used by AI search tools. The me…
AI agents now generate two thirds of documentation traffic, according to Forbes contributor Josipa Majic Predin, who reports a $500M bet by venture capitalists on knowledge engineers to fix documentation. The coverage al…