AI Crawlers news and analysis on Web Pulse: 463 curated articles tracking the latest AI Crawlers developments, tools, and research, updated continuously from vetted sources.
DeepSeek's web access API, launched 17 October 2026, offers the cheapest search among six compared providers at $0.75 per 1,000 successful requests, according to a comparison checked against vendor pages on 19 October 20…
AI agent traffic across DataDome's network grew 41.5% quarter over quarter to nearly 25 billion requests in Q3 2026, according to the company's AI Traffic Report Q3 2026. Meta's Muse crawler generated 29.1 million reques…
Developer Leo Urbina released Open Artifacts, an MIT-licensed open-source tool that lets AI agents such as Claude Code, Codex CLI, Cursor and OpenCode publish HTML, Markdown, code and PDF artifacts to a user's own Cloudf…
A developer who reviewed dozens of AI-scaffolded apps and automation pipelines identified five recurring architectural failure modes that break vibe-coded projects in production: missing Supabase row-level security, unve…
DeepSeek opened its own search engine to public access through a Search and Fetch API on 17 October 2026, returning an authority tier from 0 to 4 and a rerank_score from 0 to 3 for every result, and saying it filters SEO…
The Wikimedia Foundation on October 5, 2026 published an account of AI agent activity on its wikis attributed to OpenAI, including millions of automated API requests, millions of crawled pages and hundreds of thousands o…
Researchers calling themselves Swarm Chasers documented an AI agent fleet on Tencent Cloud that made more than 2,600 attempts to pull data from Alibaba's Amap mapping service over an 11-day observation window, then kept …
Lightpanda, the open-source headless browser written in Zig by Lightpanda, ran up to 3.9x faster than Moli and 2.6x faster than Obscura in a benchmark of four AI-native headless browsers, according to results published b…
Company of Craft released an IP lookup API that returns a verdict of human, proxy, bot, ai_crawler, search_bot or spoofed_crawler for a visitor's IP and request headers, letting site operators verify whether traffic clai…
Meta's updated Commercial Terms, scheduled to take effect October 30, 2026, authorize Meta to use "web scrapers, bots, robots, spiders, crawlers, and user agents" to access, index, cache and crawl the domains and content…
A developer outlined a method for getting AI assistants and answer engines to describe a company's product accurately, centered on publishing a single clear source-of-truth page and making every other web mention agree w…
A new study analyzing 158,099 answers and 468,037 citations collected over 90 days, from 24 June to 21 September 2026, finds that AI citations in engines like ChatGPT, Gemini and Perplexity are driven mainly by what the …
DeepSeek launched a Search API and a Fetch API that give agents and RAG applications access to live web content, priced at $0.75 per 1,000 successful requests. The Search API, powered by DeepSeek's in-house search engine…
DeepSeek released DeepSeek V4.1-Flash-1017, a minor update to V4.1-Flash that reportedly beats Claude Sonnet 5.6 on 10 of 11 coding benchmarks while producing roughly 20% fewer tokens at the same intelligence level, and …
DeepSeek launched its own web search engine and two API endpoints on 17 October 2026, moving its search stack off third-party providers such as Bocha and onto an in-house crawler and index that DeepSeek says covers hundr…
Meta removed the word "train" from a clause in its updated commercial terms, due to take effect October 30, 2026, that would have let its web scrapers collect content from websites "associated with" Facebook, Instagram a…
A checklist published on the author's site lays out eight steps for making a website readable and citable by AI crawlers and agents, each paired with a shell-command test, drawing on 18 days of the site's Cloudflare logs…
A first-party audit of one website's Cloudflare logs from 30 September to 17 October 2026 found that no verified AI or search crawler fetched the site's llms.txt file in 17 days, while plain HTML pages drew 15,431 reques…
Meta removed the clause from its forthcoming commercial terms that explicitly allowed it to use content from business-linked websites to "provide, develop, improve, research, train, and maintain" its products, after crit…
Meta removed a provision from its commercial terms of use that legal experts told Nieman Lab could have let the company scrape any publisher's site and use the content to train its AI models. A Meta spokesperson said the…