16:24
2026-08-11
dev.to
ai-ethics
A permissive robots.txt is not a licence
A developer audited ten websites they scrape for AI-related content and found that only two had licenses permitting commercial reuse, despite several having permissive robots.txt files. The audit reve…