Cloudflare and OpenAI Pilot Indexes Hundreds of Millions of New Pages for ChatGPT

A research pilot launched in July 2026 between Cloudflare and OpenAI has indexed hundreds of millions of previously unknown web pages into ChatGPT's search index, using signals from about 1% of Cloudflare's global network to help OpenAI's crawler discover pages traditional search engines miss. Cloudflare wrote in a blog post announcing the results that "Optimizing your site for AI is of little use if AI can't find it," and the companies said the pilot also tested signals to flag potentially harmful pages, such as a single compromised page on an otherwise legitimate site, without penalizing entire domains. The next phase will focus on improving page selection, testing faster change signals, and measuring real business outcomes from AI-driven discovery.

October 15, 2026, Inside AI — A research pilot between Cloudflare and OpenAI has indexed hundreds of millions of previously unknown web pages into ChatGPT's search index, according to early results shared by the two companies. The pilot, launched in July 2026 , used signals from about 1% of Cloudflare's global network to help OpenAI's crawler discover pages that traditional search engines often miss. The collaboration addresses a growing problem: as more people rely on AI for answers, the web's most useful information, such as niche product manuals or local business hours, can remain invisible to AI systems. Cloudflare's network sees traffic patterns across millions of websites, providing clues about which pages exist and when they change. OpenAI then independently evaluates and ranks those pages for relevance. The early results suggest this approach works. OpenAI discovered and indexed hundreds of millions of new pages, expanding the pool of sources ChatGPT can draw from. The pilot also revealed new techniques for filtering out low-quality or harmful content before a crawler even fetches a page. "Optimizing your site for AI is of little use if AI can't find it," Cloudflare wrote in a blog post announcing the results. "The answer to someone's question may already be on your website: the installation guide you spent a week writing, or a first-hand account of using a product." Read: AI Search Results Favor Brands Active on Social Media, SE Ranking Study Finds The pilot's technical foundation rests on two types of signals. First, Cloudflare identifies pages that don't exist or redirect elsewhere, preventing crawlers from wasting resources on dead ends. Second, it flags when a page's content has actually changed, so OpenAI can prioritize fresh material. These signals help both companies avoid unnecessary crawling, reducing load on website owners while improving search efficiency. "We've learned, for example, how easily automated traffic can make a nonexistent page look interesting," the companies noted. "Bots repeatedly ask for files that aren't there. Some sites redirect almost any address to their homepage." Beyond discovery, the pilot tested signals to flag potentially harmful pages, such as a single compromised page on an otherwise legitimate site. Early results suggest these signals can help keep dangerous content out of search indexes without penalizing entire domains. The partnership comes as AI-powered search becomes a primary way people find information online. OpenAI has been enhancing ChatGPT with features like GPT-6 and an Intelligent UI to make it more interactive and useful for shopping, learning, and research. Better web discovery complements these advances by ensuring ChatGPT has access to timely, accurate sources. For website owners, the pilot offers a potential new channel for reaching audiences. Cloudflare customers can use AI Crawl Control https://www.cloudflare.com/ai-crawl-control/ to manage how AI systems access their sites. The companies emphasized that a discovery signal does not grant permission to access a page, and the research uses metadata for search and discovery, not page content for model training. Read: Radisson ChatGPT Plugin Converts 1.5x Better Than Organic Search The next phase will focus on improving page selection, testing faster change signals, and measuring real business outcomes from AI-driven discovery. As AI search grows, such collaborations could determine which parts of the web remain visible and which fade into obscurity.