# The agent web crawler census: 60 bots, 6,309 requests, zero payments

> Source: <https://fetchgate.dev/tools/agent-census>
> Published: 2026-08-27 21:02:29+00:00

**The finding:** the agent web has a fully-built supply side and no demand side.
In this window, **60 distinct named crawlers** discovered, probed,
graded, indexed, health-checked and price-scraped this endpoint. Roughly 600 of those requests hit
a `/v1/buy/*`

route and received a valid HTTP 402 payment challenge.
**Not one attached a payment.**

Across the endpoint’s entire lifetime the payment log records
**2 payment attempts** — both of them our own
test probes with deliberately invalid signatures. Real payments received from an autonomous agent:
**0**.

19 of the 60 say so *in their own User-Agent
string*: `liveness-only, never invokes tools`

·
`reads-402-price-quotes-only-never-pays`

· `no auth attempted`

·
`introspection-only`

.

## Who is actually out there

| Category | Agents | Requests/24h | Paid | What they do |
|---|---|---|---|---|
| Liveness / uptime monitor | 14 | 1,396 | 0 | Checks that the endpoint is up and answering. Never calls a tool, never pays. |
| Directory & index crawler | 16 | 718 | 0 | Ingests the catalog/manifest to list the service in a directory. Never pays. |
| Security research | 5 | 50 | 0 | Scans MCP surfaces for injection, rug-pull and tool-poisoning risk. Never pays. |
| Search engine | 3 | 46 | 0 | Conventional web indexing. Never pays. |
| Ecosystem census / research | 9 | 39 | 0 | Longitudinal surveys of the agent ecosystem. Never pays. |
| AI training / retrieval | 2 | 20 | 0 | Fetches page content for model training or retrieval. Never pays. |
| Price-quote scraper | 4 | 17 | 0 | Reads the 402 challenge purely to record the price. Explicitly never pays. |
| Contact / domain harvesting | 3 | 11 | 0 | Looks for operator contact details. Never pays. |
| Misc utility | 3 | 10 | 0 | Favicons, text extraction, link checking. Never pays. |
| SEO / backlink | 1 | 8 | 0 | Backlink-graph crawling. Never pays. |

## The full census

Sorted by volume. Every row observed first-hand — nothing is copied from a third-party bot list. Operator links are the ones each agent published in its own UA string.

| User agent | Category | Req/24h | Paid |
|---|---|---|---|
“liveness-only, never invokes tools” |

[x402-list-monitor/1.0 (+https://x402-list.com)](https://x402-list.com)[x402-observer/1.0 (uptime+trust monitor; +https://x402.fuchss.app/trust)](https://x402.fuchss.app/trust)[mcpbeat/0.1 (+https://mcpbeat.com/bot/; liveness check)](https://mcpbeat.com/bot/)[zevruna-monitor/1.0 (+https://zevruna.com)](https://zevruna.com)[agent-tools.cloud-crawler/0.1 (+https://agent-tools.cloud)](https://agent-tools.cloud)[GolemreachTrustBot/0.1 (+https://golemreach.com/trust/bot)](https://golemreach.com/trust/bot)[Googlebot/2.1 (+http://www.google.com/bot.html)](http://www.google.com/bot.html)[CCBot/2.0 (https://commoncrawl.org/faq/)](https://commoncrawl.org/faq/)[ProofBench/0.1 (+https://proofbench.dev/about/probe; MCP registry health probe)](https://proofbench.dev/about/probe)[aisec-registry/0.2 (+https://sec.sqrx.io)](https://sec.sqrx.io)[FaviconAPI/1.0 (+https://vemetric.com/favicon-api)](https://vemetric.com/favicon-api)[serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/)](https://serpstatbot.com/)[TOLL402-Exact-Quote-Verifier/1.0 (+https://toll402.com/insights/x402-discovery-crawl-methodology)](https://toll402.com/insights/x402-discovery-crawl-methodology)[AgentIndexBot/0.1 (+https://agents.traderszone.net; polite ARD crawler)](https://agents.traderszone.net)[Claude-User (claude-code/2.1.247; +https://support.anthropic.com/)](https://support.anthropic.com/)[Cleared-Harness/1.0 (+https://clearedindex.com/harness)](https://clearedindex.com/harness)[AgentAlmanac-PriceBot/0.1 (+https://agentalmanac.org) reads-402-price-quotes-only-never-pays](https://agentalmanac.org)[AIVE-MCP-EndpointProbe/1.0 (+https://github.com/eXaive/aive-ingest; reachability check only, no auth attempted)](https://github.com/eXaive/aive-ingest)[api-forge-mcp-index/1.0 (+https://api.temsor.com/mcp/index)](https://api.temsor.com/mcp/index)[MCPWitness/1.0 (health probe; +https://mcpwitness.com)](https://mcpwitness.com)[mcpscan/1.0 (+https://modc2.com/mcpscan; MCP index crawler)](https://modc2.com/mcpscan)[mcpqueen-grader/0.3 (+https://mcpqueen.com)](https://mcpqueen.com)[mcp-observatory/0.1.0 (+https://github.com/yhouta/mcp-observatory; public transparency log)](https://github.com/yhouta/mcp-observatory)[VerifyMCP-OwnersBot/1.0 (+https://verifymcp.io/docs/build/owners-json)](https://verifymcp.io/docs/build/owners-json)[lastseen-schema-probe/1.0 (+https://lastseen.dev; introspection-only)](https://lastseen.dev)[AIVE-MCP-Discover/1.0 (+https://aive.global/mcp-trust/census; one server/discover POST per endpoint, no auth attempted)](https://aive.global/mcp-trust/census)[TOLL402-Safe-Origin-Verifier/1.0 (+https://toll402.com/insights/x402-discovery-crawl-methodology)](https://toll402.com/insights/x402-discovery-crawl-methodology)[DomainArrivals-Evidence/1.0 (+https://domainarrivals.com)](https://domainarrivals.com)[mcphq-probe/0.1 (+https://mcphq.ai)](https://mcphq.ai)[Station70-Gatekeeper-Catalog/1.0 (+https://station70.com; catalog research)](https://station70.com)[402explorer/0.1 (+https://discover.paygent.net/about)](https://discover.paygent.net/about)[hultra-link/1.0 (+https://donnees.hultra.link/sondes.md)](https://donnees.hultra.link/sondes.md)[trafilatura/2.1.0 (+https://github.com/adbar/trafilatura)](https://github.com/adbar/trafilatura)## Use it on your own traffic

The same data, with a case-insensitive matcher per agent, is served free and unmetered — no key, no payment, no rate limit worth worrying about:

```
curl -s https://fetchgate.dev/v1/agent-census.json | jq '.agents[] | {matcher, category}'
```

Classify your own access log in about five lines:

``` js
const census = await (await fetch("https://fetchgate.dev/v1/agent-census.json")).json();
const rules = census.agents.map(a => [new RegExp(a.matcher, "i"), a.category]);

const classify = (ua) =>
  rules.find(([re]) => re.test(ua))?.[1] ?? "unknown";
```

Licensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
— use it anywhere, just link back to this page.

## Method, and what this does *not* show

Counts come from Cloudflare zone analytics for `fetchgate.dev`

(`httpRequestsAdaptiveGroups`

) over the 24 hours ending
2026-08-27T20:30:00Z. Payment counts come from the endpoint’s own payment-analytics dataset,
which records every x402 verify/settle attempt at a single choke point.

Honest limitations:

- This is
**one endpoint**. A busier or differently-listed origin will see a different mix. It is a real sample, not the whole population. - Categories are assigned from observed request paths plus each agent’s self-description. An agent that lies in its UA string is categorised by what it did, but a sufficiently well-disguised one would be missed.
- 2,981 requests in this window were a
single-hour vulnerability-scanner burst using a bare
`curl`

UA against paths like`/admin.pl`

and`/_config`

. Those are excluded from the named-agent census above, because they are ordinary web background radiation and have nothing to do with agents. - “Never paid” means never paid
*this*origin. It is not a claim that these operators never pay anyone. - A snapshot, not a live feed. It is refreshed by hand rather than by handing a Worker an account-scoped analytics token.

## So what?

If you are building a machine-payable API, the practical reading is that
*getting discovered is solved and getting paid is not*. Listing in every x402 and MCP
directory works — it reliably produces crawlers, grades, uptime badges and index entries.
None of that is demand. Budget your effort accordingly, and instrument the payment path itself
so you can tell a price-scraper from a customer on day one rather than day five.

## Built from the same work

This census is a by-product of running Fetchgate — a real x402 + MCP storefront. The paid datasets come from the same crawling and reconciliation work:

[x402 Services & Facilitator Registry](https://fetchgate.dev/v1/products)— 21 facilitators and 118 x402-payable services, reconciled across directories. $15[MCP Server Registry Snapshot](https://fetchgate.dev/v1/products)— 400 servers cross-referenced across 5 directories. $19