10.7% of top sites block AI search crawlers but still allow Googlebot
A Tranco list analysis of the top 5,000 websites found that 10.7% of sites with a robots.txt block at least one AI search crawler while still allowing Googlebot, including major platforms like Faceboo…