{"slug": "cloudflare-blocks-mixed-use-crawlers-on-monetized-pages", "title": "Cloudflare Blocks Mixed-Use Crawlers on Monetized Pages", "summary": "Cloudflare announced it will default to blocking mixed-use crawlers from accessing ad-supported customer websites, citing broken crawl-for-traffic bargains with AI companies. The move affects roughly 20 percent of the web and includes options for managed robots.txt controls and a potential pay-per-crawl feature for publishers.", "body_md": "Editorial analysis: Changes to default crawl policies shift the economics of web-data collection and model training, with direct implications for dataset sourcing, provenance tracking, and model costs. Per Cloudflare's blog and reporting by The Register and CJR, **Cloudflare** announced it will default to blocking mixed-use crawlers from accessing ad-supported customer websites, and will offer managed `robots.txt` controls plus an option to restrict crawls to monetized pages (Cloudflare blog). Reporting by CJR and the Transparency Coalition says Cloudflare is testing a \"pay-per-crawl\" feature that would let publishers charge AI companies for crawl access. Cloudflare-hosted traffic reaches roughly **20 percent** of the web, CJR reports. Per Cloudflare's blog, crawl-to-referral ratios in June 2025 were roughly **Google 14:1**, **OpenAI 1,700:1**, and **Anthropic 73,000:1**, figures Cloudflare uses to argue the historic crawl-for-traffic bargain has broken down.", "url": "https://wpnews.pro/news/cloudflare-blocks-mixed-use-crawlers-on-monetized-pages", "canonical_source": "https://letsdatascience.com/news/cloudflare-blocks-mixed-use-crawlers-on-monetized-pages-5a9acadb", "published_at": "2026-07-01 13:00:00+00:00", "updated_at": "2026-07-01 13:56:05.944285+00:00", "lang": "en", "topics": ["ai-policy", "ai-ethics", "ai-infrastructure", "ai-research", "ai-tools"], "entities": ["Cloudflare", "Google", "OpenAI", "Anthropic", "The Register", "CJR", "Transparency Coalition"], "alternates": {"html": "https://wpnews.pro/news/cloudflare-blocks-mixed-use-crawlers-on-monetized-pages", "markdown": "https://wpnews.pro/news/cloudflare-blocks-mixed-use-crawlers-on-monetized-pages.md", "text": "https://wpnews.pro/news/cloudflare-blocks-mixed-use-crawlers-on-monetized-pages.txt", "jsonld": "https://wpnews.pro/news/cloudflare-blocks-mixed-use-crawlers-on-monetized-pages.jsonld"}}