13:35
2026-10-15
dev.to
ai-crawlers
Which AI crawlers do 261 top websites block? Study data
A study of the robots.txt files of 285 of the most visited websites, 261 of which returned readable files, found that 35% block training crawlers such as GPTBot while only 21% block search-oriented ag…