Cloudflare Blocks Mixed-Use Crawlers on Monetized Pages
Cloudflare announced it will default to blocking mixed-use crawlers from accessing ad-supported customer websites, citing broken crawl-for-traffic bargains with AI companies. The move affects roughly 20 percent of the we…