19:25
2026-08-17
dev.to
artificial-intelligence
Your robots.txt probably does not block the AI crawler you think it blocks
A developer discovered that many robots.txt files fail to block AI crawlers due to a subtlety in RFC 9309, where naming a specific user-agent group disables wildcard rules for that bot. The developer …