How to charge AI crawlers per page with HTTP 402 (and keep your site free for people)

A developer built AIPayGen, a free gateway that lets site owners charge AI crawlers per page using the long-dormant HTTP 402 Payment Required status code and the x402 payment flow. Requests from crawlers like GPTBot, ClaudeBot and PerplexityBot are routed to the gateway, which quotes a USDC price on Base, verifies payment via a facilitator, and then fetches the page from the origin using a secret header, while human visitors and search engines are served normally. The developer acknowledges that user agents can be spoofed and that most crawlers cannot yet pay, so the setup currently functions largely as a block with a price tag.

If you run a website, a growing share of your traffic isn't people. It's AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Bytespider and a long tail of others, fetching pages to train models or answer questions. Until now you had two choices: 1. Let them in for free. You pay the bandwidth; they get the content. 2. Block them in robots.txt. You save the bandwidth, and you disappear from AI answers. This post shows a third option: answer them with 402 Payment Required and a price , so a crawler that wants the page can pay a fraction of a cent for it, straight to your wallet. People and search engines see no difference. The status code that waited 28 years HTTP has had 402 Payment Required since 1997, marked "reserved for future use". The web never used it because charging a cent was impossible: a card payment costs around 30 cents plus a percentage. x402 https://aipaygen.com/x402 finally gives 402 a working flow: 1. Ask. The client sends a normal request. 2. Price. The server answers 402 with a payment-required header: amount, token USDC , network Base , the wallet to pay. 3. Pay. The client signs a payment authorization for exactly that amount and retries the same request with it attached. 4. Deliver. A facilitator verifies the signature and settles it on-chain in about a second; the server returns the page. No accounts, no API keys, no subscriptions. That's exactly what an automated client needs. You can see a real 402 quote from your browser on this demo page https://aipaygen.com/x402 demo . The idea: route AI crawlers to a payment gateway, on the same URL The site itself shouldn't have to implement payments. So the setup is: - Requests from known AI crawlers or anything already carrying an x402 payment are passed to a gateway. - Unpaid, the gateway answers with your price. - Paid, the gateway verifies the payment, fetches the page from your site with a secret header, settles the payment to your wallet, and returns the page. - Everyone else is served by your site as usual. The money goes crawler - your wallet. If your site errors, nobody is charged. The code: a Cloudflare Worker If your site is on Cloudflare, put this Worker on your site's route: The full list of crawlers is longer; the page linked below has it. Three details worth calling out: - The secret header closes the loop. Without it, a crawler could hit your origin directly with a browser user agent. With it, your origin only serves gateway traffic that has been paid for. - Search engines are not on the list. Googlebot and Bingbot aren't AI-training crawlers, so your rankings don't change. - Same URL, both ways. A crawler that understands x402 just retries the URL it asked for with a payment. Nothing special to discover. The code: Express Not on Cloudflare? The same thing as middleware Node 18+ : Getting a route id I built this as a free gateway, AIPayGen https://aipaygen.com . On the pay-per-crawl page https://aipaygen.com/pay-per-crawl you enter your site, a price per page and your wallet address on Base, and it gives you the route plus the Worker code with your id filled in. No fees from us; the payment goes directly to you. Then check it: Honest limits - User agents can be spoofed. A crawler that pretends to be Chrome gets the free page, same as with robots.txt. This targets the crawlers that identify themselves, which is most of the big ones. - Most crawlers can't pay yet. Today this mostly works like a block with a price tag. But agent frameworks are adding x402 support quickly, and this gives them a way to say yes instead of being shut out. - You need a wallet on Base to receive USDC. If you try it, I'd love to hear which crawlers hit your site the most and what price per page feels fair.