{"slug": "how-to-charge-ai-crawlers-per-page-with-http-402-and-keep-your-site-free-for", "title": "How to charge AI crawlers per page with HTTP 402 (and keep your site free for people)", "summary": "A developer built AIPayGen, a free gateway that lets site owners charge AI crawlers per page using the long-dormant HTTP 402 Payment Required status code and the x402 payment flow. Requests from crawlers like GPTBot, ClaudeBot and PerplexityBot are routed to the gateway, which quotes a USDC price on Base, verifies payment via a facilitator, and then fetches the page from the origin using a secret header, while human visitors and search engines are served normally. The developer acknowledges that user agents can be spoofed and that most crawlers cannot yet pay, so the setup currently functions largely as a block with a price tag.", "body_md": "If you run a website, a growing share of your traffic isn't people. It's AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Bytespider and a long tail of others, fetching pages to train models or answer questions.\n\nUntil now you had two choices:\n\n1. \n**Let them in for free.** You pay the bandwidth; they get the content.\n2. \n**Block them in robots.txt.** You save the bandwidth, and you disappear from AI answers.\n\nThis post shows a third option: **answer them with `402 Payment Required` and a price**, so a crawler that wants the page can pay a fraction of a cent for it, straight to your wallet. People and search engines see no difference.\n\n## \n  \n  \n  The status code that waited 28 years\n\nHTTP has had `402 Payment Required` since 1997, marked \"reserved for future use\". The web never used it because charging a cent was impossible: a card payment costs around 30 cents plus a percentage.\n\n[x402](https://aipaygen.com/x402) finally gives 402 a working flow:\n\n1. \n**Ask.** The client sends a normal request.\n2. \n**Price.** The server answers`402` with a`payment-required` header: amount, token (USDC), network (Base), the wallet to pay.\n3. \n**Pay.** The client signs a payment authorization for exactly that amount and retries the same request with it attached.\n4. \n**Deliver.** A facilitator verifies the signature and settles it on-chain in about a second; the server returns the page.\n\nNo accounts, no API keys, no subscriptions. That's exactly what an automated client needs.\n\nYou can see a real 402 quote from your browser on [this demo page](https://aipaygen.com/x402#demo).\n\n## \n  \n  \n  The idea: route AI crawlers to a payment gateway, on the same URL\n\nThe site itself shouldn't have to implement payments. So the setup is:\n\n- Requests from known AI crawlers (or anything already carrying an x402 payment) are passed to a gateway.\n- Unpaid, the gateway answers with your price.\n- Paid, the gateway verifies the payment, fetches the page from your site with a secret header, settles the payment to **your** wallet, and returns the page.\n- Everyone else is served by your site as usual.\n\nThe money goes crawler -> your wallet. If your site errors, nobody is charged.\n\n## \n  \n  \n  The code: a Cloudflare Worker\n\nIf your site is on Cloudflare, put this Worker on your site's route:\n\n(The full list of crawlers is longer; the page linked below has it.)\n\nThree details worth calling out:\n\n- \n**The secret header closes the loop.** Without it, a crawler could hit your origin directly with a browser user agent. With it, your origin only serves gateway traffic that has been paid for.\n- \n**Search engines are not on the list.** Googlebot and Bingbot aren't AI-training crawlers, so your rankings don't change.\n- \n**Same URL, both ways.** A crawler that understands x402 just retries the URL it asked for with a payment. Nothing special to discover.\n\n## \n  \n  \n  The code: Express\n\nNot on Cloudflare? The same thing as middleware (Node 18+):\n\n## \n  \n  \n  Getting a route id\n\nI built this as a free gateway, [AIPayGen](https://aipaygen.com). On the [pay-per-crawl page](https://aipaygen.com/pay-per-crawl) you enter your site, a price per page and your wallet address on Base, and it gives you the route plus the Worker code with your id filled in. No fees from us; the payment goes directly to you.\n\nThen check it:\n\n## \n  \n  \n  Honest limits\n\n- \n**User agents can be spoofed.** A crawler that pretends to be Chrome gets the free page, same as with robots.txt. This targets the crawlers that identify themselves, which is most of the big ones.\n- \n**Most crawlers can't pay yet.** Today this mostly works like a block with a price tag. But agent frameworks are adding x402 support quickly, and this gives them a way to say yes instead of being shut out.\n- \n**You need a wallet on Base** to receive USDC.\n\nIf you try it, I'd love to hear which crawlers hit your site the most and what price per page feels fair.", "url": "https://wpnews.pro/news/how-to-charge-ai-crawlers-per-page-with-http-402-and-keep-your-site-free-for", "canonical_source": "https://dev.to/damien829/how-to-charge-ai-crawlers-per-page-with-http-402-and-keep-your-site-free-for-people-15af", "published_at": "2026-11-15 04:16:41+00:00", "updated_at": "2026-11-15 04:21:36.480749+00:00", "lang": "en", "topics": ["ai-crawlers", "agent-protocols", "ai-infrastructure", "ai-tools"], "entities": ["AIPayGen", "Cloudflare", "GPTBot", "ClaudeBot", "PerplexityBot", "Bytespider", "x402", "Base"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-charge-ai-crawlers-per-page-with-http-402-and-keep-your-site-free-for", "markdown": "https://wpnews.pro/news/how-to-charge-ai-crawlers-per-page-with-http-402-and-keep-your-site-free-for.md", "text": "https://wpnews.pro/news/how-to-charge-ai-crawlers-per-page-with-http-402-and-keep-your-site-free-for.txt", "jsonld": "https://wpnews.pro/news/how-to-charge-ai-crawlers-per-page-with-http-402-and-keep-your-site-free-for.jsonld"}}