If you run a website, a growing share of your traffic isn't people. It's AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Bytespider and a long tail of others, fetching pages to train models or answer questions. Until now you had two choices:
Let them in for free. You pay the bandwidth; they get the content. 2. Block them in robots.txt. You save the bandwidth, and you disappear from AI answers.
This post shows a third option: answer them with 402 Payment Required and a price, so a crawler that wants the page can pay a fraction of a cent for it, straight to your wallet. People and search engines see no difference.
#
The status code that waited 28 years
HTTP has had 402 Payment Required since 1997, marked "reserved for future use". The web never used it because charging a cent was impossible: a card payment costs around 30 cents plus a percentage.
x402 finally gives 402 a working flow:
1.
Ask. The client sends a normal request.
2.
Price. The server answers402 with apayment-required header: amount, token (USDC), network (Base), the wallet to pay.
3.
Pay. The client signs a payment authorization for exactly that amount and retries the same request with it attached.
4.
Deliver. A facilitator verifies the signature and settles it on-chain in about a second; the server returns the page.
No accounts, no API keys, no subscriptions. That's exactly what an automated client needs.
You can see a real 402 quote from your browser on this demo page.
#
The idea: route AI crawlers to a payment gateway, on the same URL
The site itself shouldn't have to implement payments. So the setup is:
- Requests from known AI crawlers (or anything already carrying an x402 payment) are passed to a gateway.
- Unpaid, the gateway answers with your price.
- Paid, the gateway verifies the payment, fetches the page from your site with a secret header, settles the payment to your wallet, and returns the page.
- Everyone else is served by your site as usual.
The money goes crawler -> your wallet. If your site errors, nobody is charged.
#
The code: a Cloudflare Worker
If your site is on Cloudflare, put this Worker on your site's route: (The full list of crawlers is longer; the page linked below has it.)
Three details worth calling out:
The secret header closes the loop. Without it, a crawler could hit your origin directly with a browser user agent. With it, your origin only serves gateway traffic that has been paid for. #
Search engines are not on the list. Googlebot and Bingbot aren't AI-training crawlers, so your rankings don't change. #
Same URL, both ways. A crawler that understands x402 just retries the URL it asked for with a payment. Nothing special to discover.
#
The code: Express
Not on Cloudflare? The same thing as middleware (Node 18+):
#
Getting a route id
I built this as a free gateway, AIPayGen. On the pay-per-crawl page you enter your site, a price per page and your wallet address on Base, and it gives you the route plus the Worker code with your id filled in. No fees from us; the payment goes directly to you.
Then check it:
#
Honest limits
User agents can be spoofed. A crawler that pretends to be Chrome gets the free page, same as with robots.txt. This targets the crawlers that identify themselves, which is most of the big ones. #
Most crawlers can't pay yet. Today this mostly works like a block with a price tag. But agent frameworks are adding x402 support quickly, and this gives them a way to say yes instead of being shut out. #
You need a wallet on Base to receive USDC.
If you try it, I'd love to hear which crawlers hit your site the most and what price per page feels fair.