Cloudflare has recently launched a beta Web Search API through AI Gateway. Available through REST APIs and Cloudflare Workers, it provides unified logging, access controls, and billing, and supports bring-your-own-provider keys.
The API enables AI agents to retrieve live web results from Ceramic.ai, Exa, and Linkup. It provides a unified interface, integrates search billing with AI Gateway, logs search requests alongside model inference, and requires providers to follow verified-bot requirements and return source links.
Cloudflare’s Web Search API allows AI applications to retrieve current web content, including recent news, API changes, and technical documentation, and provide the results as context to language models during inference. Michelle Chen, group product manager at Cloudflare, Sam Else, senior director at Cloudflare, and Gabriel Massadas, senior systems engineer at Cloudflare, explain the challenge that the company is trying to address:
When you use an agent and it needs to fetch a live web page, the agent usually just guesses the URL of the page and then makes a tool call to curl it. This is why you’ll sometimes see web fetches come back with a 404 Not Found, which happens if the agent incorrectly guesses the URL of that information. As you can imagine, it’s not super efficient to randomly guess URLs all the time.
Applications and external services can access web search through a REST endpoint by providing an AI Gateway authentication token and specifying a search provider in the request payload, with the endpoint returning the search results. Chen, Else, and Massadas add:
The crawler used by the web search provider must comply with Cloudflare’s publicly stated requirements for "Verified bots" as defined in our developer documentation, and web search responses must include a link to the location of crawled content. These rules are net-positive for a fair Internet, and allow creators to decide what they want to do with their data.
In a popular Hacker News thread, the community discusses search quality, data-retention restrictions, and whether a unified API offers meaningful advantages over existing alternatives. Simon Willison writes:
My number one question about search APIs is always if they allow you to store and resyndicate results you get from them. If I'm running an agent system but I'm not allowed to store the responses - or provide a "share transcript" button - that's a pretty significant limitation. The answer to that question is inevitably buried deep in the terms.
Maximilian Goisser, CTO and co-founder of Erato Labs, notes: I think Cloudflare is (for companies already using it) approaching the status of trusted main cloud supplier (which usually would be AWS, GCP, Azure) via which the majority of cloud costs are billed (so you don't have to go through a fresh procurement process).
Cloudflare states it will identify providers that support Zero Data Retention and offers search at providers’ list prices without additional markup. The company plans to add native server tools to AI Gateway, starting with web search. Until then, developers can integrate web search into their workflows using Cloudflare Workers.