The agent web crawler census: 60 bots, 6,309 requests, zero payments

A 24-hour census of a single MCP endpoint recorded 60 distinct named crawler agents making 6,309 requests, including roughly 600 that hit a /v1/buy/* route and received a valid HTTP 402 payment challenge, yet not one attached a payment. The endpoint's lifetime payment log shows only 2 payment attempts, both deliberate test probes with invalid signatures, and 0 real payments from autonomous agents. The author, who runs the endpoint, concludes the agent web has a fully-built supply side and no demand side.

The finding: the agent web has a fully-built supply side and no demand side. In this window, 60 distinct named crawlers discovered, probed, graded, indexed, health-checked and price-scraped this endpoint. Roughly 600 of those requests hit a /v1/buy/ route and received a valid HTTP 402 payment challenge. Not one attached a payment. Across the endpoint’s entire lifetime the payment log records 2 payment attempts — both of them our own test probes with deliberately invalid signatures. Real payments received from an autonomous agent: 0 . 19 of the 60 say so in their own User-Agent string : liveness-only, never invokes tools · reads-402-price-quotes-only-never-pays · no auth attempted · introspection-only . Who is actually out there | Category | Agents | Requests/24h | Paid | What they do | |---|---|---|---|---| | Liveness / uptime monitor | 14 | 1,396 | 0 | Checks that the endpoint is up and answering. Never calls a tool, never pays. | | Directory & index crawler | 16 | 718 | 0 | Ingests the catalog/manifest to list the service in a directory. Never pays. | | Security research | 5 | 50 | 0 | Scans MCP surfaces for injection, rug-pull and tool-poisoning risk. Never pays. | | Search engine | 3 | 46 | 0 | Conventional web indexing. Never pays. | | Ecosystem census / research | 9 | 39 | 0 | Longitudinal surveys of the agent ecosystem. Never pays. | | AI training / retrieval | 2 | 20 | 0 | Fetches page content for model training or retrieval. Never pays. | | Price-quote scraper | 4 | 17 | 0 | Reads the 402 challenge purely to record the price. Explicitly never pays. | | Contact / domain harvesting | 3 | 11 | 0 | Looks for operator contact details. Never pays. | | Misc utility | 3 | 10 | 0 | Favicons, text extraction, link checking. Never pays. | | SEO / backlink | 1 | 8 | 0 | Backlink-graph crawling. Never pays. | The full census Sorted by volume. Every row observed first-hand — nothing is copied from a third-party bot list. Operator links are the ones each agent published in its own UA string. | User agent | Category | Req/24h | Paid | |---|---|---|---| “liveness-only, never invokes tools” | x402-list-monitor/1.0 +https://x402-list.com https://x402-list.com x402-observer/1.0 uptime+trust monitor; +https://x402.fuchss.app/trust https://x402.fuchss.app/trust mcpbeat/0.1 +https://mcpbeat.com/bot/; liveness check https://mcpbeat.com/bot/ zevruna-monitor/1.0 +https://zevruna.com https://zevruna.com agent-tools.cloud-crawler/0.1 +https://agent-tools.cloud https://agent-tools.cloud GolemreachTrustBot/0.1 +https://golemreach.com/trust/bot https://golemreach.com/trust/bot Googlebot/2.1 +http://www.google.com/bot.html http://www.google.com/bot.html CCBot/2.0 https://commoncrawl.org/faq/ https://commoncrawl.org/faq/ ProofBench/0.1 +https://proofbench.dev/about/probe; MCP registry health probe https://proofbench.dev/about/probe aisec-registry/0.2 +https://sec.sqrx.io https://sec.sqrx.io FaviconAPI/1.0 +https://vemetric.com/favicon-api https://vemetric.com/favicon-api serpstatbot/2.1 advanced backlink tracking bot; https://serpstatbot.com/ https://serpstatbot.com/ TOLL402-Exact-Quote-Verifier/1.0 +https://toll402.com/insights/x402-discovery-crawl-methodology https://toll402.com/insights/x402-discovery-crawl-methodology AgentIndexBot/0.1 +https://agents.traderszone.net; polite ARD crawler https://agents.traderszone.net Claude-User claude-code/2.1.247; +https://support.anthropic.com/ https://support.anthropic.com/ Cleared-Harness/1.0 +https://clearedindex.com/harness https://clearedindex.com/harness AgentAlmanac-PriceBot/0.1 +https://agentalmanac.org reads-402-price-quotes-only-never-pays https://agentalmanac.org AIVE-MCP-EndpointProbe/1.0 +https://github.com/eXaive/aive-ingest; reachability check only, no auth attempted https://github.com/eXaive/aive-ingest api-forge-mcp-index/1.0 +https://api.temsor.com/mcp/index https://api.temsor.com/mcp/index MCPWitness/1.0 health probe; +https://mcpwitness.com https://mcpwitness.com mcpscan/1.0 +https://modc2.com/mcpscan; MCP index crawler https://modc2.com/mcpscan mcpqueen-grader/0.3 +https://mcpqueen.com https://mcpqueen.com mcp-observatory/0.1.0 +https://github.com/yhouta/mcp-observatory; public transparency log https://github.com/yhouta/mcp-observatory VerifyMCP-OwnersBot/1.0 +https://verifymcp.io/docs/build/owners-json https://verifymcp.io/docs/build/owners-json lastseen-schema-probe/1.0 +https://lastseen.dev; introspection-only https://lastseen.dev AIVE-MCP-Discover/1.0 +https://aive.global/mcp-trust/census; one server/discover POST per endpoint, no auth attempted https://aive.global/mcp-trust/census TOLL402-Safe-Origin-Verifier/1.0 +https://toll402.com/insights/x402-discovery-crawl-methodology https://toll402.com/insights/x402-discovery-crawl-methodology DomainArrivals-Evidence/1.0 +https://domainarrivals.com https://domainarrivals.com mcphq-probe/0.1 +https://mcphq.ai https://mcphq.ai Station70-Gatekeeper-Catalog/1.0 +https://station70.com; catalog research https://station70.com 402explorer/0.1 +https://discover.paygent.net/about https://discover.paygent.net/about hultra-link/1.0 +https://donnees.hultra.link/sondes.md https://donnees.hultra.link/sondes.md trafilatura/2.1.0 +https://github.com/adbar/trafilatura https://github.com/adbar/trafilatura Use it on your own traffic The same data, with a case-insensitive matcher per agent, is served free and unmetered — no key, no payment, no rate limit worth worrying about: curl -s https://fetchgate.dev/v1/agent-census.json | jq '.agents | {matcher, category}' Classify your own access log in about five lines: js const census = await await fetch "https://fetchgate.dev/v1/agent-census.json" .json ; const rules = census.agents.map a = new RegExp a.matcher, "i" , a.category ; const classify = ua = rules.find re = re.test ua ?. 1 ?? "unknown"; Licensed CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ — use it anywhere, just link back to this page. Method, and what this does not show Counts come from Cloudflare zone analytics for fetchgate.dev httpRequestsAdaptiveGroups over the 24 hours ending 2026-08-27T20:30:00Z. Payment counts come from the endpoint’s own payment-analytics dataset, which records every x402 verify/settle attempt at a single choke point. Honest limitations: - This is one endpoint . A busier or differently-listed origin will see a different mix. It is a real sample, not the whole population. - Categories are assigned from observed request paths plus each agent’s self-description. An agent that lies in its UA string is categorised by what it did, but a sufficiently well-disguised one would be missed. - 2,981 requests in this window were a single-hour vulnerability-scanner burst using a bare curl UA against paths like /admin.pl and / config . Those are excluded from the named-agent census above, because they are ordinary web background radiation and have nothing to do with agents. - “Never paid” means never paid this origin. It is not a claim that these operators never pay anyone. - A snapshot, not a live feed. It is refreshed by hand rather than by handing a Worker an account-scoped analytics token. So what? If you are building a machine-payable API, the practical reading is that getting discovered is solved and getting paid is not . Listing in every x402 and MCP directory works — it reliably produces crawlers, grades, uptime badges and index entries. None of that is demand. Budget your effort accordingly, and instrument the payment path itself so you can tell a price-scraper from a customer on day one rather than day five. Built from the same work This census is a by-product of running Fetchgate — a real x402 + MCP storefront. The paid datasets come from the same crawling and reconciliation work: x402 Services & Facilitator Registry https://fetchgate.dev/v1/products — 21 facilitators and 118 x402-payable services, reconciled across directories. $15 MCP Server Registry Snapshot https://fetchgate.dev/v1/products — 400 servers cross-referenced across 5 directories. $19