Cloudflare has spent a year arming websites against AI crawlers. Now it wants to help one crawl better. The company said on Wednesday it is running a research pilot with OpenAI to test whether its network data can make AI search more accurate.
Cloudflare sits in front of more than a fifth of the web. That vantage point lets it watch how pages change and how traffic behaves in real time. The pilot feeds those signals, such as content freshness and traffic quality, into OpenAI’s search and answer system.
Why freshness matters #
AI answers rise or fall on the pages behind them. When an index lags, a chatbot serves stale or plain wrong information. OpenAI wants Cloudflare’s live view of the web to help ChatGPT discover new content sooner.
OpenAI has also signed content deals to feed ChatGPT. This pilot targets a different gap. It tackles discovery, not licensing.
“Up-to-date information is important for delivering accurate answers to people using ChatGPT,” said Nick Ryder, OpenAI’s VP of research. He framed the test as a way to find content more efficiently.
A notable turn #
The move sits oddly against Cloudflare’s recent record. Last year it let sites block AI crawlers unless the AI firms paid publishers. It built a browser protocol to tell humans and bots apart. Its slogan was blunt: your content, your rules.
This pilot points the other way. Rather than wall content off, Cloudflare now helps an AI engine reach it. CEO Matthew Prince cast it as efficiency, a way to “make AI search more efficient and help people get quality answers faster.”
Why it matters #
Search keeps shifting from links to answers. Publishers already fear losing traffic as chatbots summarise their work, a worry that pushed regulators to force opt-outs. Cloudflare now sits on both sides of that line. It sells the tools to block crawlers, and it tests the plumbing to feed them. How the pilot lands could shape who reaches the open web, and on whose terms.
Get the TNW newsletter #
Get the most important tech news in your inbox each week.