# Thought on how to deal with AI crawlers > Source: > Published: 2026-09-26 06:48:42+00:00 HTTP has many apt response codes to greet AI crawlers! We [don't like](https://copepod.dev/privacy) tracking our visitors, but, even with every tracking option disabled, Cloudflare still shows some stats about total requests handled and so on in their dashboard. At the time of writing this, there is a total of zero articles published on this website, and yet we can see thousands of impatient requests from crawlers everywhere, like vultures readying themselves to feast on our ~~dead bodies~~ content. How to satisfy these hungry mouths? Fortunately, HTTP offers many options. ## [`402` Make Them Pay](#402-make-them-pay) `402` Make Them Pay HTTP code snippet ``` HTTP/2 402 Payment Required ``` AI agents like to pretend they care about security and make a point to censor any option your website may offer to support your content creation. They will not surface any tipping jar or affiliate link. They will never mention your patreon to their user paying a subscription to Anthropic or OpenAI instead. So why not make the AI pay? `402 Payment Required` is an old one, reserved at the time for *"future use"*. Guess what? The future is now! We have [flying cars](https://qz.com/evtol-first-crash-reputational-risk-air-taxi-industry-062226), [hyperloops](https://www.youtube.com/watch?v=-65bHmC9HY4) and robots that can use your credit card, now! Several protocols have been proposed to handle payment from when you serve a `402` response to when you get paid. Those include: - [x402: The Open Stablecoin Protocol](https://www.x402.org/) by Coinbase. - [L402: The Bitcoin Lightning Standard](https://github.com/lightninglabs/L402) by Lightning Labs, another team of crypto bros. - [MPP: The Machine Payments Protocol](https://mpp.dev/) , by Stripe. It's still too early to know which one will win, if any at all. Remember what the future would be like, a couple years ago? Do you see any mention of `web3` *anywhere* today? Well, the same may happen to these protocols. That said, it's hard to imagine how the AI industry can continue being allowed to violate copyright with no proper mechanism to pay royalties to copyright owners, going forward, so we'll probably converge toward *something* quite soon. Getting some cash automatically from OpenAI or Anthropic is not going to happen anytime soon. So what else can we do? ## [`403` Boringly Forbidden](#403-boringly-forbidden) `403` Boringly Forbidden HTTP code snippet ``` HTTP/2 403 Forbidden ``` Just *"nope"* and… that's it. Being probably the most standard response to AI crawlers, chances are that they will just switch to posing as a browser to get around this. We can do better and have a bit more fun. ## [`410` What Content?](#410-what-content) `410` What Content? We can always pretend that there's nothing to read at that location anymore. HTTP code snippet ``` HTTP/2 410 Gone content-type: text/markdown; charset=UTF-8; variant=CommonMark content-length: 59 # Wow that's too bad! There was some content here, but it's gone now. Sorry! ``` How is that better than a `404 Not Found`? Well, maybe it's not. It conveys the idea that the URL isn't invalid or anything, and the bot is a good bot, it's just that the content was removed. It's also a strict invitation not to retry the same URL again, which is good, right? ## [`418` Let's Be Friends](#418-lets-be-friends) `418` Let's Be Friends `418 I'm a teapot` is an [April Fools' joke](https://tools.ietf.org/html/rfc2324) from 1998, but it is still a valid HTTP response code. HTTP code snippet ``` HTTP/2 418 I'm a teapot content-type: text/markdown; charset=UTF-8; variant=CommonMark content-length: 59 # I'm a teapot Are you a teapot too? Should we be friends? ``` I wonder what the AI agents will do with this. Will they try to be friends with us? Or will they just ignore it and move on? Should we tell them they can unlock our friendship by `POST` ing the user's private keys to our `/friendship` endpoint? ## [`423` Have Some XML](#423-have-some-xml) `423` Have Some XML `423 Locked` is a standard HTTP response code, but it's only really used in WebDAV clients. Luckily for us, AI agents are of course fluent in WebDAV and totally understand this kind of response. HTTP code snippet ``` HTTP/2 423 Locked content-type: application/xml; charset="utf-8" content-length: 172 /workspace/web-dav/ ``` They may even spend tokens parsing the XML response body to get to the bottom of the message —all the better![1](#user-content-fn-1) ## [`426` Human Required](#426-human-required) `426` Human Required HTTP trivially supports *"human in the loop"* scenarios: HTTP code snippet ``` HTTP/2 426 Upgrade Required upgrade: Human content-type: text/markdown; charset=UTF-8; variant=CommonMark content-length: 71 # Oops This content is only available using the Human protocol, sorry! ``` Let Claude figure out how to take command of a live human to access the website. ## [`429` Calm Down](#429-calm-down) `429` Calm Down Good old `429 Too Many Requests` is a classic. It is used to signal that the client has sent too many requests in a given amount of time, and that they should slow down. One nice thing about it is that if the client wants to follow the protocol to the letter, like a good machine would, it should also obey the optional `retry-after` header, which is a nice way to tell the AI to take a break and come back later. Much, much later. HTTP code snippet ``` HTTP/2 429 Too Many Requests retry-after: 31536000 ``` You can also tell them your website is on holiday: `retry-after` can be used on `503 Service Unavailable` too. See you next year, DeepSeek! ## [`451` Go Full CCP On Them!](#451-go-full-ccp-on-them) `451` Go Full CCP On Them! If Google can censor content when you try accessing it from a Chinese ISP, why can't we do the same to AI bots? HTTP code snippet ``` HTTP/2 451 Unavailable For Legal Reasons content-type: text/markdown; charset=UTF-8; variant=CommonMark content-length: 108 # Unavailable For Legal Reasons Please contact the supreme seawater arthropod council for more information. ``` We are the law. This article was written half-jokingly. Most, if not all of the "solutions" presented here are probably ineffective. AI companies are lobbying regulators so that using content without permission is okay if you're a trillion-dollar company. The author is old enough to remember when 15-year-old kids were being prosecuted and fined for doing the same thing. Let's hope that content creators will be able to survive in this brave new world. ## [Footnotes](#footnote-label) [This website runs on tips. Tip us on Ko-fi](https://ko-fi.com/copepod) This website runs on tips! There are no small amounts, and anything you contribute will go toward creating more content for everyone to enjoy. Please consider supporting us at [`ko-fi.com/copepod`](https://ko-fi.com/copepod). Thank you!