The agent web crawler census: 60 bots, 6,309 requests, zero payments

The finding: the agent web has a fully-built supply side and no demand side. In this window, 60 distinct named crawlers discovered, probed, graded, indexed, health-checked and price-scraped this endpoint. Roughly 600 of those requests hit a /v1/buy/*

route and received a valid HTTP 402 payment challenge. Not one attached a payment.

Across the endpoint’s entire lifetime the payment log records 2 payment attempts — both of them our own test probes with deliberately invalid signatures. Real payments received from an autonomous agent: 0.

19 of the 60 say so in their own User-Agent string: liveness-only, never invokes tools

· reads-402-price-quotes-only-never-pays

· no auth attempted

· introspection-only

.

Who is actually out there #

The full census #

Sorted by volume. Every row observed first-hand — nothing is copied from a third-party bot list. Operator links are the ones each agent published in its own UA string.

x402-list-monitor/1.0 (+https://x402-list.com)x402-observer/1.0 (uptime+trust monitor; +https://x402.fuchss.app/trust)mcpbeat/0.1 (+https://mcpbeat.com/bot/; liveness check)zevruna-monitor/1.0 (+https://zevruna.com)agent-tools.cloud-crawler/0.1 (+https://agent-tools.cloud)GolemreachTrustBot/0.1 (+https://golemreach.com/trust/bot)Googlebot/2.1 (+http://www.google.com/bot.html)CCBot/2.0 (https://commoncrawl.org/faq/)ProofBench/0.1 (+https://proofbench.dev/about/probe; MCP registry health probe)aisec-registry/0.2 (+https://sec.sqrx.io)FaviconAPI/1.0 (+https://vemetric.com/favicon-api)serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/)TOLL402-Exact-Quote-Verifier/1.0 (+https://toll402.com/insights/x402-discovery-crawl-methodology)AgentIndexBot/0.1 (+https://agents.traderszone.net; polite ARD crawler)Claude-User (claude-code/2.1.247; +https://support.anthropic.com/)Cleared-Harness/1.0 (+https://clearedindex.com/harness)AgentAlmanac-PriceBot/0.1 (+https://agentalmanac.org) reads-402-price-quotes-only-never-paysAIVE-MCP-EndpointProbe/1.0 (+https://github.com/eXaive/aive-ingest; reachability check only, no auth attempted)api-forge-mcp-index/1.0 (+https://api.temsor.com/mcp/index)MCPWitness/1.0 (health probe; +https://mcpwitness.com)mcpscan/1.0 (+https://modc2.com/mcpscan; MCP index crawler)mcpqueen-grader/0.3 (+https://mcpqueen.com)mcp-observatory/0.1.0 (+https://github.com/yhouta/mcp-observatory; public transparency log)VerifyMCP-OwnersBot/1.0 (+https://verifymcp.io/docs/build/owners-json)lastseen-schema-probe/1.0 (+https://lastseen.dev; introspection-only)AIVE-MCP-Discover/1.0 (+https://aive.global/mcp-trust/census; one server/discover POST per endpoint, no auth attempted)TOLL402-Safe-Origin-Verifier/1.0 (+https://toll402.com/insights/x402-discovery-crawl-methodology)DomainArrivals-Evidence/1.0 (+https://domainarrivals.com)mcphq-probe/0.1 (+https://mcphq.ai)Station70-Gatekeeper-Catalog/1.0 (+https://station70.com; catalog research)402explorer/0.1 (+https://discover.paygent.net/about)hultra-link/1.0 (+https://donnees.hultra.link/sondes.md)trafilatura/2.1.0 (+https://github.com/adbar/trafilatura)## Use it on your own traffic

The same data, with a case-insensitive matcher per agent, is served free and unmetered — no key, no payment, no rate limit worth worrying about:

curl -s https://fetchgate.dev/v1/agent-census.json | jq '.agents[] | {matcher, category}'

Classify your own access log in about five lines:

const census = await (await fetch("https://fetchgate.dev/v1/agent-census.json")).json();
const rules = census.agents.map(a => [new RegExp(a.matcher, "i"), a.category]);

const classify = (ua) =>
  rules.find(([re]) => re.test(ua))?.[1] ?? "unknown";

Licensed CC BY 4.0 — use it anywhere, just link back to this page.

Method, and what this does not show #

Counts come from Cloudflare zone analytics for fetchgate.dev

(httpRequestsAdaptiveGroups

) over the 24 hours ending 2026-08-27T20:30:00Z. Payment counts come from the endpoint’s own payment-analytics dataset, which records every x402 verify/settle attempt at a single choke point.

Honest limitations:

• This is one endpoint. A busier or differently-listed origin will see a different mix. It is a real sample, not the whole population. - Categories are assigned from observed request paths plus each agent’s self-description. An agent that lies in its UA string is categorised by what it did, but a sufficiently well-disguised one would be missed.
• 2,981 requests in this window were a single-hour vulnerability-scanner burst using a bare curl

UA against paths like/admin.pl

and/_config

. Those are excluded from the named-agent census above, because they are ordinary web background radiation and have nothing to do with agents. - “Never paid” means never paid thisorigin. It is not a claim that these operators never pay anyone. - A snapshot, not a live feed. It is refreshed by hand rather than by handing a Worker an account-scoped analytics token.

So what? #

If you are building a machine-payable API, the practical reading is that getting discovered is solved and getting paid is not. Listing in every x402 and MCP directory works — it reliably produces crawlers, grades, uptime badges and index entries. None of that is demand. Budget your effort accordingly, and instrument the payment path itself so you can tell a price-scraper from a customer on day one rather than day five.

Built from the same work #

This census is a by-product of running Fetchgate — a real x402 + MCP storefront. The paid datasets come from the same crawling and reconciliation work:

x402 Services & Facilitator Registry— 21 facilitators and 118 x402-payable services, reconciled across directories. $15MCP Server Registry Snapshot— 400 servers cross-referenced across 5 directories. $19