Thought on how to deal with AI crawlers A blog post proposes using HTTP status codes to handle AI crawlers, noting that a site with zero published articles still receives thousands of impatient crawler requests. The post suggests 402 Payment Required with emerging payment protocols x402 by Coinbase, L402 by Lightning Labs, and MPP by Stripe, alongside 403 Forbidden, 410 Gone, 418 I'm a teapot, and 423 responses as alternatives. The author argues it is hard to imagine the AI industry continuing to violate copyright without a mechanism to pay royalties to copyright owners. HTTP has many apt response codes to greet AI crawlers We don't like https://copepod.dev/privacy tracking our visitors, but, even with every tracking option disabled, Cloudflare still shows some stats about total requests handled and so on in their dashboard. At the time of writing this, there is a total of zero articles published on this website, and yet we can see thousands of impatient requests from crawlers everywhere, like vultures readying themselves to feast on our ~~dead bodies~~ content. How to satisfy these hungry mouths? Fortunately, HTTP offers many options. 402 Make Them Pay 402-make-them-pay 402 Make Them Pay HTTP code snippet HTTP/2 402 Payment Required AI agents like to pretend they care about security and make a point to censor any option your website may offer to support your content creation. They will not surface any tipping jar or affiliate link. They will never mention your patreon to their user paying a subscription to Anthropic or OpenAI instead. So why not make the AI pay? 402 Payment Required is an old one, reserved at the time for "future use" . Guess what? The future is now We have flying cars https://qz.com/evtol-first-crash-reputational-risk-air-taxi-industry-062226 , hyperloops https://www.youtube.com/watch?v=-65bHmC9HY4 and robots that can use your credit card, now Several protocols have been proposed to handle payment from when you serve a 402 response to when you get paid. Those include: - x402: The Open Stablecoin Protocol https://www.x402.org/ by Coinbase. - L402: The Bitcoin Lightning Standard https://github.com/lightninglabs/L402 by Lightning Labs, another team of crypto bros. - MPP: The Machine Payments Protocol https://mpp.dev/ , by Stripe. It's still too early to know which one will win, if any at all. Remember what the future would be like, a couple years ago? Do you see any mention of web3 anywhere today? Well, the same may happen to these protocols. That said, it's hard to imagine how the AI industry can continue being allowed to violate copyright with no proper mechanism to pay royalties to copyright owners, going forward, so we'll probably converge toward something quite soon. Getting some cash automatically from OpenAI or Anthropic is not going to happen anytime soon. So what else can we do? 403 Boringly Forbidden 403-boringly-forbidden 403 Boringly Forbidden HTTP code snippet HTTP/2 403 Forbidden Just "nope" and… that's it. Being probably the most standard response to AI crawlers, chances are that they will just switch to posing as a browser to get around this. We can do better and have a bit more fun. 410 What Content? 410-what-content 410 What Content? We can always pretend that there's nothing to read at that location anymore. HTTP code snippet HTTP/2 410 Gone content-type: text/markdown; charset=UTF-8; variant=CommonMark content-length: 59 Wow that's too bad There was some content here, but it's gone now. Sorry How is that better than a 404 Not Found ? Well, maybe it's not. It conveys the idea that the URL isn't invalid or anything, and the bot is a good bot, it's just that the content was removed. It's also a strict invitation not to retry the same URL again, which is good, right? 418 Let's Be Friends 418-lets-be-friends 418 Let's Be Friends 418 I'm a teapot is an April Fools' joke https://tools.ietf.org/html/rfc2324 from 1998, but it is still a valid HTTP response code. HTTP code snippet HTTP/2 418 I'm a teapot content-type: text/markdown; charset=UTF-8; variant=CommonMark content-length: 59 I'm a teapot Are you a teapot too? Should we be friends? I wonder what the AI agents will do with this. Will they try to be friends with us? Or will they just ignore it and move on? Should we tell them they can unlock our friendship by POST ing the user's private keys to our /friendship endpoint? 423 Have Some XML 423-have-some-xml 423 Have Some XML 423 Locked is a standard HTTP response code, but it's only really used in WebDAV clients. Luckily for us, AI agents are of course fluent in WebDAV and totally understand this kind of response. HTTP code snippet HTTP/2 423 Locked content-type: application/xml; charset="utf-8" content-length: 172