Artificial Analysis has released the "Search Index," a benchmark that measures how well search API providers work for AI agents across quality, cost, and speed.
The initial lineup includes Parallel, Exa, Firecrawl, You.com, Tavily, Keenable, and Brave. Each one is tested with the same model (GPT-5.6 Luna) in a standardized agent setup. Only the search provider changes. The agent runs on Stirrup, an open-source framework from Artificial Analysis, and gets 25 runs per task to search and pull up web pages.
The index combines three equally weighted benchmarks. DeepSearchQA has 900 research questions, each requiring multiple search queries. A BrowseComp subset tests 200 hard-to-find facts that need multi-step browsing. AA-Omniscience covers 600 questions across six knowledge domains. A tool-free baseline, where the model answers on its own, provides the comparison point.
Better search quality also lowers total costs. The model uses fewer tokens when it gets good results up front. With Parallel Search (advanced), token use drops by over 40 percent compared to the Basic version. Per-task search costs go up, but total cost comes in lower ($0.084 vs. $0.11).
Raw speed per query doesn't always mean faster results overall. Parallel Search (turbo) clocks the shortest response time per query (0.51 seconds vs. 1.03 seconds for Basic), but its lower quality (67 vs. 73) forces the agent to run more passes. Total time per task winds up about the same.
Artificial Analysis says Parallel, Firecrawl, and Parallel (turbo) hit the best mix of cost and performance. Other providers can apply to join the benchmark. The full methodology is public.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
Subscribe now