{"slug": "the-agent-web-crawler-census-60-bots-6309-requests-zero-payments", "title": "The agent web crawler census: 60 bots, 6,309 requests, zero payments", "summary": "A 24-hour census of a single MCP endpoint recorded 60 distinct named crawler agents making 6,309 requests, including roughly 600 that hit a /v1/buy/* route and received a valid HTTP 402 payment challenge, yet not one attached a payment. The endpoint's lifetime payment log shows only 2 payment attempts, both deliberate test probes with invalid signatures, and 0 real payments from autonomous agents. The author, who runs the endpoint, concludes the agent web has a fully-built supply side and no demand side.", "body_md": "**The finding:** the agent web has a fully-built supply side and no demand side.\nIn this window, **60 distinct named crawlers** discovered, probed,\ngraded, indexed, health-checked and price-scraped this endpoint. Roughly 600 of those requests hit\na `/v1/buy/*`\n\nroute and received a valid HTTP 402 payment challenge.\n**Not one attached a payment.**\n\nAcross the endpoint’s entire lifetime the payment log records\n**2 payment attempts** — both of them our own\ntest probes with deliberately invalid signatures. Real payments received from an autonomous agent:\n**0**.\n\n19 of the 60 say so *in their own User-Agent\nstring*: `liveness-only, never invokes tools`\n\n·\n`reads-402-price-quotes-only-never-pays`\n\n· `no auth attempted`\n\n·\n`introspection-only`\n\n.\n\n## Who is actually out there\n\n| Category | Agents | Requests/24h | Paid | What they do |\n|---|---|---|---|---|\n| Liveness / uptime monitor | 14 | 1,396 | 0 | Checks that the endpoint is up and answering. Never calls a tool, never pays. |\n| Directory & index crawler | 16 | 718 | 0 | Ingests the catalog/manifest to list the service in a directory. Never pays. |\n| Security research | 5 | 50 | 0 | Scans MCP surfaces for injection, rug-pull and tool-poisoning risk. Never pays. |\n| Search engine | 3 | 46 | 0 | Conventional web indexing. Never pays. |\n| Ecosystem census / research | 9 | 39 | 0 | Longitudinal surveys of the agent ecosystem. Never pays. |\n| AI training / retrieval | 2 | 20 | 0 | Fetches page content for model training or retrieval. Never pays. |\n| Price-quote scraper | 4 | 17 | 0 | Reads the 402 challenge purely to record the price. Explicitly never pays. |\n| Contact / domain harvesting | 3 | 11 | 0 | Looks for operator contact details. Never pays. |\n| Misc utility | 3 | 10 | 0 | Favicons, text extraction, link checking. Never pays. |\n| SEO / backlink | 1 | 8 | 0 | Backlink-graph crawling. Never pays. |\n\n## The full census\n\nSorted by volume. Every row observed first-hand — nothing is copied from a third-party bot list. Operator links are the ones each agent published in its own UA string.\n\n| User agent | Category | Req/24h | Paid |\n|---|---|---|---|\n“liveness-only, never invokes tools” |\n\n[x402-list-monitor/1.0 (+https://x402-list.com)](https://x402-list.com)[x402-observer/1.0 (uptime+trust monitor; +https://x402.fuchss.app/trust)](https://x402.fuchss.app/trust)[mcpbeat/0.1 (+https://mcpbeat.com/bot/; liveness check)](https://mcpbeat.com/bot/)[zevruna-monitor/1.0 (+https://zevruna.com)](https://zevruna.com)[agent-tools.cloud-crawler/0.1 (+https://agent-tools.cloud)](https://agent-tools.cloud)[GolemreachTrustBot/0.1 (+https://golemreach.com/trust/bot)](https://golemreach.com/trust/bot)[Googlebot/2.1 (+http://www.google.com/bot.html)](http://www.google.com/bot.html)[CCBot/2.0 (https://commoncrawl.org/faq/)](https://commoncrawl.org/faq/)[ProofBench/0.1 (+https://proofbench.dev/about/probe; MCP registry health probe)](https://proofbench.dev/about/probe)[aisec-registry/0.2 (+https://sec.sqrx.io)](https://sec.sqrx.io)[FaviconAPI/1.0 (+https://vemetric.com/favicon-api)](https://vemetric.com/favicon-api)[serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/)](https://serpstatbot.com/)[TOLL402-Exact-Quote-Verifier/1.0 (+https://toll402.com/insights/x402-discovery-crawl-methodology)](https://toll402.com/insights/x402-discovery-crawl-methodology)[AgentIndexBot/0.1 (+https://agents.traderszone.net; polite ARD crawler)](https://agents.traderszone.net)[Claude-User (claude-code/2.1.247; +https://support.anthropic.com/)](https://support.anthropic.com/)[Cleared-Harness/1.0 (+https://clearedindex.com/harness)](https://clearedindex.com/harness)[AgentAlmanac-PriceBot/0.1 (+https://agentalmanac.org) reads-402-price-quotes-only-never-pays](https://agentalmanac.org)[AIVE-MCP-EndpointProbe/1.0 (+https://github.com/eXaive/aive-ingest; reachability check only, no auth attempted)](https://github.com/eXaive/aive-ingest)[api-forge-mcp-index/1.0 (+https://api.temsor.com/mcp/index)](https://api.temsor.com/mcp/index)[MCPWitness/1.0 (health probe; +https://mcpwitness.com)](https://mcpwitness.com)[mcpscan/1.0 (+https://modc2.com/mcpscan; MCP index crawler)](https://modc2.com/mcpscan)[mcpqueen-grader/0.3 (+https://mcpqueen.com)](https://mcpqueen.com)[mcp-observatory/0.1.0 (+https://github.com/yhouta/mcp-observatory; public transparency log)](https://github.com/yhouta/mcp-observatory)[VerifyMCP-OwnersBot/1.0 (+https://verifymcp.io/docs/build/owners-json)](https://verifymcp.io/docs/build/owners-json)[lastseen-schema-probe/1.0 (+https://lastseen.dev; introspection-only)](https://lastseen.dev)[AIVE-MCP-Discover/1.0 (+https://aive.global/mcp-trust/census; one server/discover POST per endpoint, no auth attempted)](https://aive.global/mcp-trust/census)[TOLL402-Safe-Origin-Verifier/1.0 (+https://toll402.com/insights/x402-discovery-crawl-methodology)](https://toll402.com/insights/x402-discovery-crawl-methodology)[DomainArrivals-Evidence/1.0 (+https://domainarrivals.com)](https://domainarrivals.com)[mcphq-probe/0.1 (+https://mcphq.ai)](https://mcphq.ai)[Station70-Gatekeeper-Catalog/1.0 (+https://station70.com; catalog research)](https://station70.com)[402explorer/0.1 (+https://discover.paygent.net/about)](https://discover.paygent.net/about)[hultra-link/1.0 (+https://donnees.hultra.link/sondes.md)](https://donnees.hultra.link/sondes.md)[trafilatura/2.1.0 (+https://github.com/adbar/trafilatura)](https://github.com/adbar/trafilatura)## Use it on your own traffic\n\nThe same data, with a case-insensitive matcher per agent, is served free and unmetered — no key, no payment, no rate limit worth worrying about:\n\n```\ncurl -s https://fetchgate.dev/v1/agent-census.json | jq '.agents[] | {matcher, category}'\n```\n\nClassify your own access log in about five lines:\n\n``` js\nconst census = await (await fetch(\"https://fetchgate.dev/v1/agent-census.json\")).json();\nconst rules = census.agents.map(a => [new RegExp(a.matcher, \"i\"), a.category]);\n\nconst classify = (ua) =>\n  rules.find(([re]) => re.test(ua))?.[1] ?? \"unknown\";\n```\n\nLicensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)\n— use it anywhere, just link back to this page.\n\n## Method, and what this does *not* show\n\nCounts come from Cloudflare zone analytics for `fetchgate.dev`\n\n(`httpRequestsAdaptiveGroups`\n\n) over the 24 hours ending\n2026-08-27T20:30:00Z. Payment counts come from the endpoint’s own payment-analytics dataset,\nwhich records every x402 verify/settle attempt at a single choke point.\n\nHonest limitations:\n\n- This is\n**one endpoint**. A busier or differently-listed origin will see a different mix. It is a real sample, not the whole population. - Categories are assigned from observed request paths plus each agent’s self-description. An agent that lies in its UA string is categorised by what it did, but a sufficiently well-disguised one would be missed.\n- 2,981 requests in this window were a\nsingle-hour vulnerability-scanner burst using a bare\n`curl`\n\nUA against paths like`/admin.pl`\n\nand`/_config`\n\n. Those are excluded from the named-agent census above, because they are ordinary web background radiation and have nothing to do with agents. - “Never paid” means never paid\n*this*origin. It is not a claim that these operators never pay anyone. - A snapshot, not a live feed. It is refreshed by hand rather than by handing a Worker an account-scoped analytics token.\n\n## So what?\n\nIf you are building a machine-payable API, the practical reading is that\n*getting discovered is solved and getting paid is not*. Listing in every x402 and MCP\ndirectory works — it reliably produces crawlers, grades, uptime badges and index entries.\nNone of that is demand. Budget your effort accordingly, and instrument the payment path itself\nso you can tell a price-scraper from a customer on day one rather than day five.\n\n## Built from the same work\n\nThis census is a by-product of running Fetchgate — a real x402 + MCP storefront. The paid datasets come from the same crawling and reconciliation work:\n\n[x402 Services & Facilitator Registry](https://fetchgate.dev/v1/products)— 21 facilitators and 118 x402-payable services, reconciled across directories. $15[MCP Server Registry Snapshot](https://fetchgate.dev/v1/products)— 400 servers cross-referenced across 5 directories. $19", "url": "https://wpnews.pro/news/the-agent-web-crawler-census-60-bots-6309-requests-zero-payments", "canonical_source": "https://fetchgate.dev/tools/agent-census", "published_at": "2026-08-27 21:02:29+00:00", "updated_at": "2026-08-27 21:18:46.638800+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "ai-tools"], "entities": ["x402-list.com", "mcpbeat.com", "Googlebot", "CCBot", "Anthropic", "Claude", "ProofBench", "AgentAlmanac"], "alternates": {"html": "https://wpnews.pro/news/the-agent-web-crawler-census-60-bots-6309-requests-zero-payments", "markdown": "https://wpnews.pro/news/the-agent-web-crawler-census-60-bots-6309-requests-zero-payments.md", "text": "https://wpnews.pro/news/the-agent-web-crawler-census-60-bots-6309-requests-zero-payments.txt", "jsonld": "https://wpnews.pro/news/the-agent-web-crawler-census-60-bots-6309-requests-zero-payments.jsonld"}}