We tested 12 search APIs for AI agents. Even the best one isn't enough

← All posts

• The best single provider had the known answer in its top five for 68% of our searches. All 12 providers together reached 84%.
• Most of the gain came from the second provider, from 68% to 77%. From the fifth provider on, each one added less than a point.
• For one search in seven, only one to three providers had the known answer in their top five.

We also picked the mix on four fifths of the searches and tested it on the rest. Two providers then reached 75.5% and four reached 82.6%.

The rows are grouped by provider family, by how much their results overlap. Google-results group: serper, firecrawl, tinyfish, tavily and serpapi. brave and you.com form the second family. Own index: exa, perplexity, parallel, ceramic and seltz. Our next post covers the families in detail.

At Telem.ai, we want every search your agent makes to find the best answer on the web, whichever provider holds it. You shouldn't have to track which provider is good at what, because we take that part off your hands. Your agent shouldn't miss an answer because it asked the wrong web search provider.

Let's start with a simple question: is one good provider enough? Better yet, is the best provider enough? We tested a dozen providers on the same agent searches and counted what each one found. The web is wider than any one provider, and even the best one sees only part of it.

How we tested #

We ran a series of experiments on real searches from an AI agent and checked which providers came back with the answer. Every extra provider costs money and gives your agent more to read, so we wanted to know how much each one adds. Methods at the end has the full setup.

Most of the gain comes from the second provider #

On its own, the best provider had the known answer in its top five 68% of the time. About a third of the searches came back without the known answer in their top five. No single provider stood out: the top three finished within a point of each other, and which one comes first depends on which searches you test.

The second provider is where we saw the big jump: about 9 points, from 68% to 77%. A third added about 3 more and a fourth about 2. From the fifth provider on, each one added less than a point.

Each provider ran in its best mode from our tests. At default settings the line is almost the same: 68%, 77% and 80% for one to three providers, and 83% with all 12.

You don't need the perfect pair, either. The dashed line in Fig. 1 averages over every possible mix. One provider picked at random found the page 55% of the time. Two picked at random found it 68% of the time, about as often as the best provider alone. That's handy if you'd rather not test 12 providers yourself.

The answers only a few providers had #

Fig. 2 sorts the searches by how many providers had the known answer in their top five. It runs from all 12 on the left down to none on the right. Start at the right end: for 16% of searches, none of the 12 had it. Adding providers won't fix those. The agent has to search a different way, or open pages and read them.

Just left of that is where your choice of provider decides things. For 14% of searches, only one, two or three providers had the page. If your agent's one provider isn't among them, it never sees the answer. At the far left, only 5.5% of searches were easy enough for all 12 providers.

What's next #

This is the gap we built Telem to close. On its own, the best provider missed the known answer for about a third of these searches. A second provider found 9 more in 100, and some answers sat with only one to three providers.

This test only checks whether the known answer shows up in the top five results. The agent still has to read them and get the answer right. Every extra provider also gives it more to read, which we'll come back to in a later post. One agent wrote all these searches, one per question, so your agent's numbers will look a bit different.

So where do those extra answers come from? Our next post looks at how little the providers' results overlap. After that, we look at what happens when an agent asks another provider after a miss.

Wanna try it out? Explore a demo in our playground.

Methods and setup #

Searches

We used 363 test questions with known answers. Every known answer was recorded before we searched: either a specific page, or a value such as a number, name or date. Each search is the first one our test agent wrote for that question in September, sent again word for word. The agent is opencode, an open-source coding agent, running a language model, with Telem as its web search tool. The questions come from public sets (the RealTime QA news quizzes, WIT, Exa's company benchmark, MIRACL, FinSearchComp, LitSearch, FinanceBench and Needle in the Web) and from our own questions about posts on data silo sites (platforms whose content is hard to reach through search). We don't publish the second group, because those questions can name private people.

News quizzes are 41% of the questions. Without them the gain from more providers is larger: 56% for the best provider, 66% for two, 72% for three and 78% for all 12.

Measure

Known answer in the top five. One of a provider's first five results is the question's known page, or shows its known value in the title or snippet. A simple rule checks it, with no judge. A mix of providers counts a search when any provider in it has the known answer in its top five.

The two lines

Best possible pick: for each number of providers, the set that found the most, chosen after seeing the results. Picked at random: the average over every possible set of that size. When we chose the mix on four fifths of the searches and scored it on the rest, one to four providers reached 64.5%, 75.5%, 79.9% and 82.6%. The one-provider figure drops because three providers tie near the top, so each split picks a different one.

Configuration

Telem sends agent searches to all of these providers, and we are a customer of each. For this test each provider ran in its best mode from our earlier tests.

Providers: brave, ceramic, exa, firecrawl, parallel, perplexity, seltz, serpapi, serper, tavily, tinyfish, you.com.

Telem.ai Team.