06:48
2026-09-26
copepod.dev
ai-crawlers
Thought on how to deal with AI crawlers
A blog post proposes using HTTP status codes to handle AI crawlers, noting that a site with zero published articles still receives thousands of impatient crawler requests. The post suggests 402 Paymen…