Cloudflare CEO says the web's future is "pay to crawl" as bots overtake human traffic

Cloudflare CEO Matthew Prince announced that bot traffic has surpassed human traffic on the internet, years earlier than his 2027 prediction, driven by a surge in AI agents. Prince stated the web's future will require a "pay to crawl" model, as his company works on infrastructure to support charging AI crawlers for site access.

Cloudflare CEO says the web's future is "pay to crawl" as bots overtake human traffic Bot traffic has overtaken human traffic on the internet, according to Cloudflare CEO Matthew Prince. Prince says https://x.com/eastdakota/status/2062212701414187452 he didn't expect this tipping point until late 2027, but rapid growth in automated traffic from AI agents moved the timeline up fast. The shift happened over the past few months. The data is still rough, but the trend is unmistakable, Prince adds. Prince notes that bot, crawler, and agent all mean the same thing; the label just depends on whether you see them as good or bad. As for the future of the web, "clearly it's going to be pay to crawl," he writes https://x.com/eastdakota/status/2062309790609473608 . Cloudflare launched a platform last summer https://the-decoder.com/cloudflare-aims-to-save-the-world-wide-web-by-blocking-ai-crawlers-without-explicit-consent/ that lets site owners gate AI crawlers and charge for access, but it hasn't gained traction. Prince says his company is still working on "protocols and infrastructure to support the volume it'll require." Meanwhile, Google's AI Overviews and AI Mode have racked up billions of users https://the-decoder.com/google-lets-sites-opt-out-of-ai-search-results-knowing-most-have-nowhere-else-to-go/ . AI News Without the Hype – Curated by Humans Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section. Subscribe now Prince via X https://x.com/eastdakota/status/2062212701414187452