# How to charge AI crawlers per page with HTTP 402 (and keep your site free for people) > Source: > Published: 2026-11-15 04:16:41+00:00 If you run a website, a growing share of your traffic isn't people. It's AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Bytespider and a long tail of others, fetching pages to train models or answer questions. Until now you had two choices: 1. **Let them in for free.** You pay the bandwidth; they get the content. 2. **Block them in robots.txt.** You save the bandwidth, and you disappear from AI answers. This post shows a third option: **answer them with `402 Payment Required` and a price**, so a crawler that wants the page can pay a fraction of a cent for it, straight to your wallet. People and search engines see no difference. ## The status code that waited 28 years HTTP has had `402 Payment Required` since 1997, marked "reserved for future use". The web never used it because charging a cent was impossible: a card payment costs around 30 cents plus a percentage. [x402](https://aipaygen.com/x402) finally gives 402 a working flow: 1. **Ask.** The client sends a normal request. 2. **Price.** The server answers`402` with a`payment-required` header: amount, token (USDC), network (Base), the wallet to pay. 3. **Pay.** The client signs a payment authorization for exactly that amount and retries the same request with it attached. 4. **Deliver.** A facilitator verifies the signature and settles it on-chain in about a second; the server returns the page. No accounts, no API keys, no subscriptions. That's exactly what an automated client needs. You can see a real 402 quote from your browser on [this demo page](https://aipaygen.com/x402#demo). ## The idea: route AI crawlers to a payment gateway, on the same URL The site itself shouldn't have to implement payments. So the setup is: - Requests from known AI crawlers (or anything already carrying an x402 payment) are passed to a gateway. - Unpaid, the gateway answers with your price. - Paid, the gateway verifies the payment, fetches the page from your site with a secret header, settles the payment to **your** wallet, and returns the page. - Everyone else is served by your site as usual. The money goes crawler -> your wallet. If your site errors, nobody is charged. ## The code: a Cloudflare Worker If your site is on Cloudflare, put this Worker on your site's route: (The full list of crawlers is longer; the page linked below has it.) Three details worth calling out: - **The secret header closes the loop.** Without it, a crawler could hit your origin directly with a browser user agent. With it, your origin only serves gateway traffic that has been paid for. - **Search engines are not on the list.** Googlebot and Bingbot aren't AI-training crawlers, so your rankings don't change. - **Same URL, both ways.** A crawler that understands x402 just retries the URL it asked for with a payment. Nothing special to discover. ## The code: Express Not on Cloudflare? The same thing as middleware (Node 18+): ## Getting a route id I built this as a free gateway, [AIPayGen](https://aipaygen.com). On the [pay-per-crawl page](https://aipaygen.com/pay-per-crawl) you enter your site, a price per page and your wallet address on Base, and it gives you the route plus the Worker code with your id filled in. No fees from us; the payment goes directly to you. Then check it: ## Honest limits - **User agents can be spoofed.** A crawler that pretends to be Chrome gets the free page, same as with robots.txt. This targets the crawlers that identify themselves, which is most of the big ones. - **Most crawlers can't pay yet.** Today this mostly works like a block with a price tag. But agent frameworks are adding x402 support quickly, and this gives them a way to say yes instead of being shut out. - **You need a wallet on Base** to receive USDC. If you try it, I'd love to hear which crawlers hit your site the most and what price per page feels fair.