AI Crawlers news and analysis on Web Pulse: 463 curated articles tracking the latest AI Crawlers developments, tools, and research, updated continuously from vetted sources.
A Nozzle study found that Google served poisoned search results to AI scrapers on 8.4% of SERPs since September 13, up from 0.17% before, according to the newsletter. The study says Google deliberately returns different …
A website seeded with 50,000 fabricated statistics generated by GPT-4.1 for roughly $30 — later expanded with 15,000 listicles for about $20 more, for a total of about $50 — has drawn more than 10,000 AI sessions and lin…
WhatsApp removed 102 hidden HTML instruction blocks from its help pages on October 18, hours after WIRED asked about them, after a privacy firm's October scrape found text directing the company's AI chatbot to avoid ment…
A developer built a lightweight documentation crawler that runs inside an AI agent's tool loop, fetching web docs and converting them to clean Markdown for retrieval-augmented context. In a test run it pulled 60 pages of…
The maintainer of GEO Doctor released an MIT-licensed checker that diagnoses why AI search crawlers cannot read a website, testing four conditions: robots.txt rules blocking search crawlers such as OAI-SearchBot and Perp…
Meta removed the word "train" from a section of its business terms of service that had authorized automated crawling, ten days before the revised terms take effect on October 30, 2026, following criticism led by Engadget…
Meta's proposed commercial terms, slated to take effect October 30, shift liability for any mistakes made by its AI models onto enterprise customers, who would be "solely responsible for and assume any and all risks asso…
Meta's proposed commercial terms, slated to take effect October 30, shift liability for errors made by its AI models onto enterprise customers, who would be "solely responsible for and assume any and all risks associated…
Mojodojo.io's AI Discovery Index, tracking 73 newly published pages since 6 June 2026, found that Bingbot was first to reach 51% of pages and Anthropic's ClaudeBot first on 49%, with a median time to first visit of 14 ho…
AgentBadge now publishes every article on agentbadge.xyz as a typed JSON Knowledge Object (KO v1) with claims linked to evidence, Q&A and live actions, connected in a knowledge graph exposed through two MCP tools, search…
A developer's analysis finds that AI agents are increasingly blocked by major sites — Yelp, Delta, JPMorgan Chase, Charles Schwab and Amazon among them — as anti-bot infrastructure tuned for scrapers also trips legitimat…
Brave unveiled BAT Roadmap 4.0 on July 9, 2026, steering the Basic Attention Token toward a "transactional attention economy" built on autonomous machine payments, including support for x402 and the Machine Payments Prot…
Meta removed the word "train" from the crawling clause of its new Commercial Terms, which take effect Oct. 30, while keeping permission to crawl websites associated with a business's products or services, Nieman Lab repo…
DeepSeek opened its in-house search engine to developers on Oct. 17 through a Search API priced at $0.75 per 1,000 successful requests, with each result carrying an authority tier from 0 to 4 (-1 for unknown) and a reran…
Airlines, banks, and retailers are increasingly blocking third-party AI agents from booking, shopping, and accessing accounts, with Yelp requiring paid data licensing agreements and Delta, United, JPMorgan Chase, and Cha…
A survey of all 2,060 hosts listed in the x402 Bazaar found that only 949 (46%) served a front page an AI crawler could read when fetched on 25 September 2026 with a single request and no JavaScript. The remaining hosts …
Meta's rewritten Commercial Terms, effective October 30, 2026, grant the company the right to crawl websites "associated with" any business account — including Pages, ad accounts and WhatsApp Business — and use collected…
Eight of 10 AI search tools tested, including ChatGPT, Perplexity, Grok and Google AI, cited posts about a fabricated concept called the "Cactus Gravity Protocol" within seven days of the posts going live, according to a…
Bright Data's official Model Context Protocol server, Bright Data MCP, now exposes 69 tools for AI agents, including Google, Bing and Yandex search results as structured data, URL-to-Markdown or HTML scraping with bot de…
A developer released gsearch, an open-source MCP server that runs a real Firefox browser via Camoufox to return Google's live results page as typed JSON, including resolved links, the AI Overview, People also ask and rel…