{"slug": "thought-on-how-to-deal-with-ai-crawlers", "title": "Thought on how to deal with AI crawlers", "summary": "A blog post proposes using HTTP status codes to handle AI crawlers, noting that a site with zero published articles still receives thousands of impatient crawler requests. The post suggests 402 Payment Required with emerging payment protocols x402 by Coinbase, L402 by Lightning Labs, and MPP by Stripe, alongside 403 Forbidden, 410 Gone, 418 I'm a teapot, and 423 responses as alternatives. The author argues it is hard to imagine the AI industry continuing to violate copyright without a mechanism to pay royalties to copyright owners.", "body_md": "HTTP has many apt response codes to greet AI crawlers!\n\nWe [don't like](https://copepod.dev/privacy) tracking our visitors, but, even with every tracking option disabled, Cloudflare still shows some stats about total requests handled and so on in their dashboard.\n\nAt the time of writing this, there is a total of zero articles published on this website, and yet we can see thousands of impatient requests from crawlers everywhere, like vultures readying themselves to feast on our ~~dead bodies~~ content.\n\nHow to satisfy these hungry mouths?\n\nFortunately, HTTP offers many options.\n\n## [`402` Make Them Pay](#402-make-them-pay)\n\n`402` Make Them Pay\nHTTP code snippet\n\n```\nHTTP/2 402 Payment Required\n```\n\nAI agents like to pretend they care about security and make a point to censor any option your website may offer to support your content creation. They will not surface any tipping jar or affiliate link. They will never mention your patreon to their user paying a subscription to Anthropic or OpenAI instead.\n\nSo why not make the AI pay?\n\n`402 Payment Required` is an old one, reserved at the time for *\"future use\"*. Guess what? The future is now! We have [flying cars](https://qz.com/evtol-first-crash-reputational-risk-air-taxi-industry-062226), [hyperloops](https://www.youtube.com/watch?v=-65bHmC9HY4) and robots that can use your credit card, now!\n\nSeveral protocols have been proposed to handle payment from when you serve a `402` response to when you get paid. Those include:\n\n- [x402: The Open Stablecoin Protocol](https://www.x402.org/) by Coinbase.\n- [L402: The Bitcoin Lightning Standard](https://github.com/lightninglabs/L402) by Lightning Labs, another team of crypto bros.\n- [MPP: The Machine Payments Protocol](https://mpp.dev/) , by Stripe.\n\nIt's still too early to know which one will win, if any at all. Remember what the future would be like, a couple years ago? Do you see any mention of `web3` *anywhere* today? Well, the same may happen to these protocols.\n\nThat said, it's hard to imagine how the AI industry can continue being allowed to violate copyright with no proper mechanism to pay royalties to copyright owners, going forward, so we'll probably converge toward *something* quite soon.\n\nGetting some cash automatically from OpenAI or Anthropic is not going to happen anytime soon. So what else can we do?\n\n## [`403` Boringly Forbidden](#403-boringly-forbidden)\n\n`403` Boringly Forbidden\nHTTP code snippet\n\n```\nHTTP/2 403 Forbidden\n```\n\nJust *\"nope\"* and… that's it.\n\nBeing probably the most standard response to AI crawlers, chances are that they will just switch to posing as a browser to get around this. We can do better and have a bit more fun.\n\n## [`410` What Content?](#410-what-content)\n\n`410` What Content?\nWe can always pretend that there's nothing to read at that location anymore.\n\nHTTP code snippet\n\n```\nHTTP/2 410 Gone\ncontent-type: text/markdown; charset=UTF-8; variant=CommonMark\ncontent-length: 59\n\n# Wow that's too bad!\nThere was some content here, but it's gone now. Sorry!\n```\n\nHow is that better than a `404 Not Found`? Well, maybe it's not. It conveys the idea that the URL isn't invalid or anything, and the bot is a good bot, it's just that the content was removed. It's also a strict invitation not to retry the same URL again, which is good, right?\n\n## [`418` Let's Be Friends](#418-lets-be-friends)\n\n`418` Let's Be Friends\n`418 I'm a teapot` is an [April Fools' joke](https://tools.ietf.org/html/rfc2324) from 1998, but it is still a valid HTTP response code.\n\nHTTP code snippet\n\n```\nHTTP/2 418 I'm a teapot\ncontent-type: text/markdown; charset=UTF-8; variant=CommonMark\ncontent-length: 59\n\n# I'm a teapot\nAre you a teapot too? Should we be friends?\n```\n\nI wonder what the AI agents will do with this. Will they try to be friends with us? Or will they just ignore it and move on? Should we tell them they can unlock our friendship by `POST` ing the user's private keys to our `/friendship` endpoint?\n\n## [`423` Have Some XML](#423-have-some-xml)\n\n`423` Have Some XML\n`423 Locked` is a standard HTTP response code, but it's only really used in WebDAV clients. Luckily for us, AI agents are of course fluent in WebDAV and totally understand this kind of response.\n\nHTTP code snippet\n\n```\nHTTP/2 423 Locked\ncontent-type: application/xml; charset=\"utf-8\"\ncontent-length: 172\n\n<?xml version=\"1.0\" encoding=\"utf-8\" ?>\n<D:error xmlns:D=\"DAV:\">\n  <D:lock-token-submitted>\n    <D:href>/workspace/web-dav/</D:href>\n  </D:lock-token-submitted>\n</D:error>\n```\n\nThey may even spend tokens parsing the XML response body to get to the bottom of the message —all the better![1](#user-content-fn-1)\n\n## [`426` Human Required](#426-human-required)\n\n`426` Human Required\nHTTP trivially supports *\"human in the loop\"* scenarios:\n\nHTTP code snippet\n\n```\nHTTP/2 426 Upgrade Required\nupgrade: Human\ncontent-type: text/markdown; charset=UTF-8; variant=CommonMark\ncontent-length: 71\n\n# Oops\nThis content is only available using the Human protocol, sorry!\n```\n\nLet Claude figure out how to take command of a live human to access the website.\n\n## [`429` Calm Down](#429-calm-down)\n\n`429` Calm Down\nGood old `429 Too Many Requests` is a classic. It is used to signal that the client has sent too many requests in a given amount of time, and that they should slow down. One nice thing about it is that if the client wants to follow the protocol to the letter, like a good machine would, it should also obey the optional `retry-after` header, which is a nice way to tell the AI to take a break and come back later. Much, much later.\n\nHTTP code snippet\n\n```\nHTTP/2 429 Too Many Requests\nretry-after: 31536000\n```\n\nYou can also tell them your website is on holiday: `retry-after` can be used on `503 Service Unavailable` too.\n\nSee you next year, DeepSeek!\n\n## [`451` Go Full CCP On Them!](#451-go-full-ccp-on-them)\n\n`451` Go Full CCP On Them!\nIf Google can censor content when you try accessing it from a Chinese ISP, why can't we do the same to AI bots?\n\nHTTP code snippet\n\n```\nHTTP/2 451 Unavailable For Legal Reasons\ncontent-type: text/markdown; charset=UTF-8; variant=CommonMark\ncontent-length: 108\n\n# Unavailable For Legal Reasons\nPlease contact the supreme seawater arthropod council for more information.\n```\n\nWe are the law.\n\nThis article was written half-jokingly. Most, if not all of the \"solutions\" presented here are probably ineffective.\n\nAI companies are lobbying regulators so that using content without permission is okay if you're a trillion-dollar company. The author is old enough to remember when 15-year-old kids were being prosecuted and fined for doing the same thing. Let's hope that content creators will be able to survive in this brave new world.\n\n## [Footnotes](#footnote-label)\n\n[This website runs on tips. Tip us on Ko-fi](https://ko-fi.com/copepod)\n\nThis website runs on tips! There are no small amounts, and anything you contribute will go toward creating more content for everyone to enjoy.\n\nPlease consider supporting us at [`ko-fi.com/copepod`](https://ko-fi.com/copepod). Thank you!", "url": "https://wpnews.pro/news/thought-on-how-to-deal-with-ai-crawlers", "canonical_source": "https://copepod.dev/human-required", "published_at": "2026-09-26 06:48:42+00:00", "updated_at": "2026-09-26 07:01:40.295754+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-agents", "agent-protocols"], "entities": ["Cloudflare", "Coinbase", "Lightning Labs", "Stripe", "x402", "L402", "MPP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/thought-on-how-to-deal-with-ai-crawlers", "markdown": "https://wpnews.pro/news/thought-on-how-to-deal-with-ai-crawlers.md", "text": "https://wpnews.pro/news/thought-on-how-to-deal-with-ai-crawlers.txt", "jsonld": "https://wpnews.pro/news/thought-on-how-to-deal-with-ai-crawlers.jsonld"}}