# Cloudflare’s 1% Pilot Feeds ChatGPT Search [2026] > Source: > Published: 2026-10-24 08:15:51+00:00 Cloudflare and OpenAI say a research pilot covering about 1% of Cloudflare’s network helped ChatGPT Search find and index hundreds of millions of web pages it had never seen, according to [Search Engine Journal’s report on the October 15 results](https://www.searchenginejournal.com/chatgpt-search-can-draw-on-more-pages-after-a-cloudflare-pilot/593720/). The Cloudflare OpenAI ChatGPT Search pilot started in July 2026 and remains a research effort, not a product. Two other Cloudflare moves landed in the same three weeks. On October 2 the company opened a Web Search API in beta, with three third-party providers and list prices that differ by a factor of 28. In mid-October it announced that Verified Bots will face continuous behavior checks, starting with whether they honor robots.txt and back off after an HTTP 429 response. Taken together, they show where Cloudflare wants to sit as AI search matures: between the crawlers and the sites they read, selling visibility to one side and control to the other. Cloudflare has [already worked with OpenAI on edge vulnerability fixes](https://shattered.io/cloudflare-openai-gpt-5-6-cyber-vulnerability-remediation-2026/), so the search pilot extends an existing relationship rather than starting one. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. [Add now](https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F) ## What Cloudflare and OpenAI Actually Reported on October 15 The announcement came in a Cloudflare blog post co-written with OpenAI staff. Cloudflare first described the research on July 8, then ran the pilot with a participating cohort representing roughly 1% of its total network. By October 15 the companies said the signals had helped OpenAI’s search crawler discover and index hundreds of millions of previously unknown pages for ChatGPT Search. The headline number is deliberately vague. Hundreds of millions is the only count the companies gave. [Coverage of the announcement](https://letsdatascience.com/news/cloudflare-and-openai-report-pilot-added-hundreds-of-million-29c404ee) repeats the same phrase, and no outlet has found a more precise figure. That matters because Cloudflare says its network reaches within 50ms of 95% of the world’s Internet-connected population, so a 1% cohort is still a slice of a very large surface. The pilot’s stated purpose was narrow. OpenAI wanted to learn whether network-level insights could help its crawler spend its time on pages worth indexing. Cloudflare wanted to show that the traffic it sees can serve that purpose without handing over page content. The companies also said the pilot used web metadata for search and discovery, not shared page content for model training. The examples in the post were deliberately mundane: a user guide for a particular washing machine, current passport requirements and fees, a restaurant’s menu and hours. Cloudflare’s point is that such pages carry real value for a searcher even when few other sites link to them. A link-based crawler, which finds pages by following links from pages it already knows, tends to miss exactly that kind of page. ## How the Discovery Signals Work Cloudflare sits in front of a large share of the web, so it sees which pages receive real traffic, which pages changed, and which URLs lead nowhere. It can pass a distilled version of that picture to OpenAI’s crawler. The inputs described so far are content freshness, traffic quality, and evidence that a page has actually changed. ### What the Signals Cover The signals run in two directions. Some point the crawler toward pages worth fetching. Others tell it where not to waste requests, such as dead URLs, sites that redirect almost any address to the homepage, and addresses that duplicate other content. A change signal also tells the crawler whether a known page deserves a revisit or would return the same bytes as last time. The companies are also testing page-level safety signals. Their example is a fake login form an attacker plants on one page of an otherwise legitimate site. The goal is to flag that single page for exclusion from the index and leave the rest of the domain alone. Early results suggest the signals can help identify pages to keep out of an index. ### What the Signals Do Not Do A discovery signal does not grant access. Domain owners’ controls still apply, so a page that blocks OpenAI’s search crawler stays blocked even if Cloudflare’s network flags it as fresh and busy. Cloudflare’s post says this directly, and Search Engine Journal highlights it as the key limit on the program. The signals also do not decide rankings. OpenAI evaluates and ranks discovered pages itself, so appearing in the index guarantees nothing about being cited in an answer. Publishers who read hundreds of millions of new pages as hundreds of millions of new citations are reading the announcement wrongly. ## Why a 1% Cohort Is Both Small and Meaningful One percent sounds modest, but Cloudflare’s own site puts the network in 335+ cities and says it powers 45% of the Fortune 500. A cohort of that size still carries a wide mix of sites, from big retailers to small business pages. [MACCUS framed the result](https://maccus.fr/en/cloudflare-openai-chatgpt-search/) as Cloudflare helping ChatGPT find hundreds of millions of pages it had never seen, which captures the scale better than any percentage. The limited cohort also cuts the other way. If a 1% sample surfaced that many unknown pages, the pool of undiscovered pages on the remaining 99% is plausibly much larger. That is an inference, not a reported figure, and the companies have not claimed it. What the sample does show is that link-based discovery leaves large gaps, and that network telemetry can fill some of them. Cloudflare has said it wants to offer the capability more widely. According to Search Engine Journal, the company signaled in earlier posts that it intends broad access sometime in the rest of 2026, but the October 15 announcement set no date. The next phase, per the companies, covers better page selection, faster and more reliable change signals, and measurement of what happens after discovery. ## What the Companies Have Not Disclosed The gaps in the announcement matter as much as the claims. The post does not say how often newly indexed pages were cited in ChatGPT answers. It does not report how many visits those pages received from ChatGPT. It does not name which sites joined the cohort, or whether the newly found pages sit on those sites. It also gives no crawl-volume or server-load figures, even though reducing unnecessary load for Cloudflare customers is one of the stated goals. The post concedes that finding new pages can initially increase crawling while those pages are evaluated and indexed. For a site owner, the first visible effect of the program could be more OpenAI crawler traffic in server logs, not more referral traffic. Cloudflare says the companies are now working to determine how better discovery might translate into business outcomes. Until they publish numbers, the honest reading is that the pilot proves reach, not value. Reach is the cheap half of the equation. Whether a discovered page earns a click is the half publishers care about. ## Cloudflare Web Search API Pricing: Three Providers, a 28x Spread Cloudflare’s Web Search API reached beta on October 2, 2026. It lets agents and applications query the live web instead of guessing URLs or relying on a model’s training cutoff. It runs through AI Gateway, so each search appears in gateway logs and bills against AI Gateway credits at the provider’s list price, with no Cloudflare markup. Developers can also bring their own provider API key. Three providers launched with the service: Ceramic.ai, Exa, and Linkup. Cloudflare says all three support Zero Data Retention for requests made through its platform and have committed to its verified-bot crawling standards. Those standards require a provider’s crawlers to identify themselves, respect robots.txt, and return source links with results. | Provider | List price per 1,000 requests | Cost for 1 million requests | Multiple of cheapest | |---|---|---|---| | Ceramic.ai | $0.25 | $250 | 1x | | Linkup | $5.00 | $5,000 | 20x | | Exa | $7.00 | $7,000 | 28x | | Cloudflare markup via AI Gateway credits | $0 | $0 | n/a | | Bring-your-own provider key | Provider’s own rate | Provider’s own rate | n/a | The per-1,000 prices come from Cloudflare’s October 2 documentation. The one-million-request column is simple multiplication. A search-heavy agent that makes 1 million queries a month pays $250 at the low end and $7,000 at the high end, so provider choice outweighs every other cost lever in the pipeline. Price likely tracks result quality and index depth, but the documentation does not publish benchmark comparisons, so the cheaper provider is not automatically the worse buy. Rival model makers are building their own search layers too. [DeepSeek opened its own search API](https://shattered.io/deepseek-v4-1-flash-1017-deepswe-77-search-api-2026/) alongside a new Flash release, which means Cloudflare’s marketplace will compete with first-party options from the labs themselves. ## How OpenAI’s Own Search Pricing Compares OpenAI’s API pricing page, updated October 23, reportedly lists web search at $10 per 1,000 calls, with search-content tokens free and normal model token charges still applying. That figure appears in a single source in our research, so treat it as a reported number and check the page before budgeting. A separate OpenAI help-center rate card for ChatGPT Enterprise words the same $10 differently, as web runs plus search-content tokens billed at model rates. If the $10 figure holds, the first-party tool costs 40 times the cheapest Cloudflare-routed provider and about 1.4 times the priciest. The comparison is imperfect. OpenAI’s tool bundles retrieval with its own models and ranking, while Cloudflare’s API returns search results to whatever model the developer chooses. A team already committed to OpenAI’s models may pay the premium for convenience, and a team mixing models will favor the neutral layer. OpenAI also says API usage bills separately from ChatGPT subscriptions, including Plus, Business, Enterprise, and Edu. A reader who pays for a ChatGPT plan gets no API search credit from it. Consumer-facing ChatGPT Search itself runs across Free, Go, Plus, Pro, Business, Enterprise, and Edu plans, subject to plan usage limits, and works for users who are not signed in. ## Verified Bots Get a Behavior Test, Not Just an ID Check Cloudflare’s mid-October post, written by Chris Tsoukaladelis, argues that identity and trust are different things. A driver’s license shows who someone is and that they passed a test, not that they will drive well later. To earn Verified Bot status, a bot must identify itself accurately and securely, do only what it says it will do, and avoid abusive behavior. Bots sign requests with Web Bot Auth, and an open catalogue called BotBase records them. The new system targets the third criterion. Cloudflare now measures how often a crawler respects site preferences and turns those observations into what it calls an explainable trust band. The band shows how consistently a bot behaves and which observed behaviors informed the assessment. Cloudflare stresses that it is not a permanent judgment of the bot or its operator. ### The robots.txt and HTTP 429 Tests The first signal asks whether a bot’s observed traffic matches the preferences a site publishes in robots.txt, in line with RFC 9309. Cloudflare says isolated violations may not matter. It looks for repeated patterns that disregard site preferences. The company also notes that robots.txt sets minimum rules, and that anything unstated is not automatically allowed. The second signal covers bots that stay inside allowed paths yet hammer a server. A site can answer with an HTTP 429 (Too Many Requests) response, which tells the visitor to slow down, sometimes with a wait time. Cloudflare checks whether the bot’s traffic adapts afterward. Bots that consistently slow or pause gain trust. Bots that keep sending at the same pace lose it. Cloudflare is already using the system internally to evaluate Verified Bots. It plans to publish the scores on Cloudflare Radar and to open the system to all customers, who would then pick the trust threshold that decides how Verified Bot traffic gets handled. None of that public-facing part has shipped as of October 24, 2026, and Cloudflare has not published a scoring rubric or any actual trust values. ## The Accountability Loop Cloudflare Is Building Cloudflare describes the trust system as one leg of a three-part loop. Bot Preference Sync lets site owners publish their preferences and align them with configured policies. The new behavior checks show whether Verified Bots honor those preferences over time. Bot Management then lets owners decide how each kind of traffic is handled. The pieces connect to the search news. The Web Search API’s provider requirements and the Verified Bots program share one rulebook: identify yourself, respect robots.txt, and slow down when told. A search provider that wants to sell through Cloudflare must meet that bar, and a crawler that fails it can lose the verified label that gives it easy access to Cloudflare-protected sites. Cloudflare has used the stick before. Its post recalls that it de-listed Perplexity’s bot from the Verified Bot list in 2025, after the bot sought to access resources on sites that had opted out of scraping and then tried to circumvent controls when blocked. The new system aims to catch such drift earlier, and to catch the milder cases that come from a bug or a misconfiguration rather than bad intent. ## What Publishers Should Set in robots.txt Right Now OpenAI runs separate crawlers for search and for training, and the robots.txt directives are separate too. According to Search Engine Journal’s reading of OpenAI’s crawler documentation, the choices break down as follows. | robots.txt setting | Effect on ChatGPT Search | Effect on training use | |---|---|---| | Allow OAI-SearchBot, block GPTBot | Pages stay eligible for ChatGPT Search results | Signals that crawled content should not be used for training | | Block OAI-SearchBot, allow GPTBot | Pages drop out of search results, though links may still appear as navigational links | Content may still be collected for training | | Allow both | Pages stay eligible | OpenAI may use data from one crawl for both purposes | | Block both | Pages drop out of search results | No training crawl | The first row is the configuration OpenAI itself offers as an example. For Cloudflare customers, AI Crawl Control, available on every plan, shows which AI services are accessing a site and lets owners allow or block each crawler. OpenAI separately says a site becomes eligible for ChatGPT Search by allowing OAI-SearchBot and permitting traffic from its published search-bot IP addresses. Developers who want to test the Web Search API from their own code can call it from a Worker through the AI binding or from the REST endpoint. Our walkthrough on how to [deploy a secure Cloudflare Workers API](https://shattered.io/deploy-secure-cloudflare-workers-api-2026/) covers the token and routing basics that any such call needs. ## Market Impact: Edge Telemetry Becomes a Product For years the edge was a place to cache files and filter attacks. The pilot treats the edge as a sensor network whose output has resale value to search operators. If OpenAI’s results hold up, every large search and answer engine has a reason to ask for the same arrangement, and every CDN that sees large traffic volumes has a reason to offer it. The economics favor Cloudflare in one specific way. Discovery signals are a byproduct of traffic it already carries, so the marginal cost is low. Pairing them with bot controls lets Cloudflare sell to both parties in the same transaction. Publishers get a say in which crawlers read them, and AI companies get cleaner paths to content. That position is hard for a pure compute vendor to copy, much as [Cloudflare R2’s egress pricing against AWS S3](https://shattered.io/cloudflare-r2-vs-aws-s3-pricing-2026/) showed how a network owner can undercut storage vendors on data movement. Cloudflare’s AI Search product, which reached general availability during Birthday Week, adds a third line. It now supports visual search over image pixels, OCR for scanned PDFs, files up to 10 MiB, and any chat model. Usage-based billing for it begins November 1, 2026, with included monthly allowances for ingestion, storage, and semantic and full-text queries. The date is about a week away, so teams already using AI Search should check their allowances before the meter starts. Publishers face a harder choice. More discovery brings more crawling and no guaranteed traffic, and the companies have yet to show referral numbers. Sites that depend on search visibility will likely welcome the signals. Sites that monetize through direct visits may decide the trade is poor. Watch how the broader [cloud computing market](https://shattered.io/category/cloud-computing/) prices these edge services once more customers can choose. ## Competitive Comparison: Where Cloudflare’s Edge Rivals Stand Cloudflare competes with edge and serverless vendors on compute, and our pricing comparisons show how the fight has played out. We broke down [Cloudflare Workers against AWS Lambda@Edge](https://shattered.io/cloudflare-workers-vs-aws-lambda-edge-2026/) on latency and cost, and ran similar numbers for [Workers versus Fastly Compute](https://shattered.io/cloudflare-workers-vs-fastly-compute-pricing-2026/). In each case the question was who runs code closest to users at the lowest price. The search pilot moves the contest to a different axis. We found no public announcement of an equivalent discovery arrangement between a major search operator and any of those rivals. A hyperscaler can sell more GPUs or functions. It cannot easily sell a view of what real visitors request across a large slice of the web. Vercel, which we compared in [Workers versus Vercel Edge](https://shattered.io/cloudflare-workers-vs-vercel-edge-2026/), and Fastly both operate edge networks, so they could pursue similar programs. On bot controls, Cloudflare has the advantage of scale in its own telemetry. An earlier Cloudflare post on the agentic web reported more than 500 billion verified-bot requests per week at the time. A behavior score built on that volume has more statistical weight than one built by a smaller vendor. The weakness is that a score only matters if bot operators care about it, and that depends on how many sites actually set thresholds. ## Historical Context: From Crawler Blocks to Search Partnerships Cloudflare’s relationship with AI crawlers has swung between enforcement and cooperation. The Perplexity de-listing in 2025 was the enforcement end. The OpenAI pilot is the cooperative end. The Verified Bots accountability system sits between them, offering a way to keep cooperating while keeping the option to withdraw trust. | Date | Event | Status | |---|---|---| | 2025 | Cloudflare de-lists Perplexity’s bot from its Verified Bot list | Completed | | July 8, 2026 | Cloudflare announces the OpenAI discovery research | Completed | | October 2, 2026 | Web Search API opens in beta through AI Gateway | Beta | | October 15, 2026 | Cloudflare and OpenAI publish early pilot results | Research, ongoing | | Mid-October 2026 | Verified Bots continuous accountability announced | Used internally | | November 1, 2026 | AI Search usage-based billing begins | Scheduled | | Rest of 2026 | Broader access to discovery signals, per Cloudflare’s stated intent | No date set | The timeline shows how quickly the pieces arrived. Within three weeks Cloudflare launched a search marketplace, reported pilot results, and rewrote how it judges good bots. The sequence reads like a planned rollout, though Cloudflare has not said the pieces were designed to ship together. ## What the Companies Say in Their Own Words OpenAI framed the pilot as an experiment in efficiency. Nick Ryder, identified in the coverage as OpenAI’s vice president of research, said, [per Tech Times](https://www.techtimes.com/articles/329138/20261016/chatgpt-search-gained-hundreds-millions-pages-link-crawlers-never-found.htm): “Up-to-date information is important for delivering accurate answers to people using ChatGPT. Piloting this with Cloudflare will allow us to explore whether network-level insights can help us discover content more efficiently.” Cloudflare’s [October 15 blog post](https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/) describes the result in more upbeat terms: “The pilot has brought previously unknown pages into OpenAI’s search index, giving ChatGPT more sources to draw on to help people find relevant, timely information.” The same post states that “Cloudflare’s network provides signals that can help OpenAI’s search crawler discover more useful pages across the web to surface to ChatGPT users.” The post also draws the line on responsibility. In its words, “OpenAI then independently evaluates and ranks how those pages appear based on how relevant and helpful they are.” That sentence is the most important one for publishers. It places ranking, and therefore traffic, entirely on OpenAI’s side of the table. Read side by side, the statements show a careful split. OpenAI talks about efficiency and accuracy. Cloudflare talks about reach and sources. Neither mentions revenue, referral traffic, or payment to publishers, and that silence is where the next round of negotiation is likely to happen. ## Five Predictions for the Next Quarter These are analysis, not reported facts, and each could miss. 1. **Cloudflare widens the discovery program beyond the 1% cohort.** The company has said it intends broad access during the rest of 2026, and a pilot with one partner is a thin base for a product. Expect an opt-in for more customers before the year ends, possibly with a second search operator attached. 2. **Radar publishes the first Verified Bot trust bands.** Cloudflare already runs the system internally and has committed to public reporting. Publishing is cheap and builds pressure on bot operators, so it should come before customer-configurable thresholds. 3. **Web Search API prices converge.** A 28x spread between providers is hard to sustain on a marketplace that shows prices side by side. The cheapest provider’s price will probably hold, while the most expensive will either cut or differentiate on quality. 4. **Publishers split their robots.txt policies.** The allow-search, block-training setup is already the documented example, and tools like AI Crawl Control make it easy. More sites will adopt it as they learn that search visibility and training use are separate decisions. 5. **Rival edge vendors announce their own search-signal deals.** The pilot gives them a template. A vendor with large traffic share and a willing search partner can copy the structure, so expect at least one similar announcement. ## FAQ: Cloudflare, OpenAI and ChatGPT Search **What did the Cloudflare OpenAI ChatGPT Search pilot achieve?** The companies say it helped OpenAI discover and index hundreds of millions of previously unknown web pages for ChatGPT Search. It used a cohort of about 1% of Cloudflare’s network and began in July 2026. They have not published an exact page count, citation rate, or referral figure. **Does Cloudflare share my page content with OpenAI?** According to the companies, the pilot uses web metadata for search and discovery, not shared page content for model training. A signal about a page also does not grant permission to access it, so your robots.txt and Cloudflare controls still apply. **How much does the Cloudflare Web Search API cost?** Cloudflare adds no markup. Billed through AI Gateway credits, searches cost each provider’s list price: $0.25 per 1,000 requests for Ceramic.ai, $5.00 for Linkup, and $7.00 for Exa. The service has been in beta since October 2, 2026. **What is a Verified Bot trust band?** It is an explainable range showing how consistently a Verified Bot respects site preferences such as robots.txt and HTTP 429 slow-down requests. Cloudflare says it is not a permanent judgment. Public reporting on Radar and customer-set thresholds are planned, not live. **How do I keep my site in ChatGPT Search but out of training?** Allow OAI-SearchBot and block GPTBot in robots.txt. OpenAI documents this setup as an example. Also permit traffic from OpenAI’s published search-bot IP addresses, and use AI Crawl Control on Cloudflare to confirm which crawlers reach you. **Will more discovered pages mean more traffic to my site?** Not necessarily. OpenAI ranks pages on its own, and the announcement reports no referral data. The post warns that finding new pages can initially increase crawling, so the first change you see may be extra crawler requests in your logs. **Is the pilot available to all Cloudflare customers?** No. It remains a research pilot with a participating cohort. Cloudflare has said it intends to offer wider access, but no date or product name has been announced.