Cloudflare spent a year blocking AI crawlers. Now it’s helping OpenAI index the web

Cloudflare announced a research pilot with OpenAI to use its network data to improve AI search accuracy, despite having spent the past year helping websites block AI crawlers. The pilot feeds signals like content freshness and traffic quality into OpenAI's search system to help ChatGPT discover new content sooner.

Cloudflare has spent a year arming websites against AI crawlers. Now it wants to help one crawl better. The company said on Wednesday it is running a research pilot with OpenAI to test whether its network data can make AI search more accurate. Cloudflare sits in front of more than a fifth of the web. That vantage point lets it watch how pages change and how traffic behaves in real time. The pilot feeds those signals, such as content freshness and traffic quality, into OpenAI’s search and answer system. Why freshness matters AI answers rise or fall on the pages behind them. When an index lags, a chatbot serves stale or plain wrong information https://thenextweb.com/news/google-ai-overviews-scp-horror-fiction-real . OpenAI wants Cloudflare’s live view of the web to help ChatGPT discover new content sooner. OpenAI has also signed content deals https://thenextweb.com/news/getty-images-openai-chatgpt-deal-shares-soar to feed ChatGPT. This pilot targets a different gap. It tackles discovery, not licensing. “Up-to-date information is important for delivering accurate answers to people using ChatGPT,” said Nick Ryder, OpenAI’s VP of research. He framed the test as a way to find content more efficiently. A notable turn The move sits oddly against Cloudflare’s recent record. Last year it let sites block AI crawlers https://thenextweb.com/news/cloudflare-block-ai-crawlers-pay-publishers unless the AI firms paid publishers. It built a browser protocol https://thenextweb.com/news/cloudflare-pact-browser-privacy-bot-traffic-protocol to tell humans and bots apart. Its slogan was blunt: your content, your rules. This pilot points the other way. Rather than wall content off, Cloudflare now helps an AI engine reach it. CEO Matthew Prince cast it as efficiency, a way to “make AI search more efficient and help people get quality answers faster.” Why it matters Search keeps shifting from links to answers. Publishers already fear losing traffic as chatbots summarise their work, a worry that pushed regulators to force opt-outs https://thenextweb.com/news/uk-forces-google-to-let-publishers-opt-out-of-ai-search-results-without-losing-their-ranking . Cloudflare now sits on both sides of that line. It sells the tools to block crawlers, and it tests the plumbing to feed them. How the pilot lands could shape who reaches the open web, and on whose terms. Get the TNW newsletter Get the most important tech news in your inbox each week.