AI Crawlers news and analysis on Web Pulse: 463 curated articles tracking the latest AI Crawlers developments, tools, and research, updated continuously from vetted sources.
A team at 402registry fetched 120 sites built with AI app builders the way AI crawlers like GPTBot, ClaudeBot and PerplexityBot do—a single plain HTTP request with no JavaScript—and found that 61% of the 104 that respond…
A 2026 arXiv study (2610.15382) that traced ChatGPT Web's internal execution flow via server-sent event (SSE) traces found that candidates retrieved more often across repeated runs are more likely to be shown as citation…
ForgeMesh released Pay-per-Crawl, a pay-per-crawl gate that charges AI crawlers, SEO crawlers and scripts in USDC on the Base network, across ten stacks including WordPress, Drupal, Laravel, Next.js, Node, Cloudflare, ng…
Meta revised its forthcoming commercial terms preview on Friday to remove the explicit word "train" and state that the consolidated crawling terms apply only to its ads and business products, responding to criticism from…
A robots.txt analysis of the internet's top 2,000 domains, published on Dev.to and run on August 2, 2026, found that 292 of 1,530 domains with a reachable robots.txt file — 19.1% — now disallow at least one of the four m…
Meta removed language from its commercial terms page that legal experts said could have let it scrape content from news sites and other domains to train its AI, following backlash. The deleted clause stated Meta "may use…
Nozzle found that 8.0% of the 17.8 million Google results pages it checked between Sept. 13 and Oct. 9 were missing most of each keyword's usual sites, up from 0.17% between Aug. 20 and Sept. 9, a pattern Nozzle calls "p…
Reza and the Serpair team published a findability checklist for newly launched startups, covering WebSite and Organization structured data, robots.txt rules for AI crawlers such as OAI-SearchBot and PerplexityBot, and fi…
An open-source project called CiteLadder has launched to track how AI answer engines including ChatGPT, Claude, Gemini and Google AI Overviews name brands and cite pages in response to buyer prompts. CiteLadder records v…
A developer maintaining a pinyin chart site found that Bing indexed the page as containing only four syllables because the 199-syllable table was rendered client-side, leaving crawlers with an empty table element. The fi…
Meta will roll out an updated commercial terms of service on Oct. 30 that authorizes the company "or any entity acting on our behalf" to use web scrapers, bots, crawlers and user agents to access, index, cache and crawl …
A developer operating under the name Omegatech released duckai_cli.js, an unofficial Node.js command-line scraper for DuckDuckGo's Duck.ai chat service, which has no public API. The tool uses puppeteer-real-browser to dr…
Meta's updated Commercial Terms, effective October 30, authorize the company to use automated scrapers, bots and crawlers — plus "test credentials" for registered content — to collect data from any website "associated wi…
A new version of Meta's Commercial Terms taking effect October 30 authorizes Meta to use automated software such as web scrapers, bots and crawlers to access, index, cache and crawl the domains, web pages and URLs associ…
Meta will update its commercial terms of service on October 30 to require business accounts to authorize automated scraping and crawling of their linked websites and to let Meta use that content to "train" its products, …
Developer-focused startups are shifting from generative engine optimization toward "agent-led growth," optimizing content so coding agents like Claude Code, Codex, Cursor, and OpenCode can find and implement their tools,…
A developer published a five-minute checklist for making websites readable by AI assistants such as ChatGPT, Claude and Perplexity, covering robots.txt user agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBo…
Meta has rewritten its Commercial Terms so that any business using Facebook, Instagram or WhatsApp for commercial purposes authorises Meta to send crawlers across that business's websites and use "any content" it finds t…
Meta introduced new terms of service allowing it to use content from news sites to train its artificial intelligence systems without charge, applying to business accounts on Facebook and Instagram and authorizing automat…
Meta's new Commercial Terms, effective October 30th, authorize the company to use automated scrapers, bots, crawlers and user agents to access, index, cache and crawl domains and web pages "associated" with businesses th…