03:15
2026-09-09
github.com
developer-tools
Robots-check: see what your robots.txt is serving (Cloudflare AI Crawl Control)
A new open-source tool called robots-check reveals that Cloudflare's AI Crawl Control feature silently appends its own rules to websites' robots.txt files, often blocking AI crawlers like GPTBot and Gā¦