# Show HN: Sourcery – Benchmark for search APIs on the sources they return > Source: > Published: 2026-08-20 20:21:41+00:00 Sourcery Search Benchmark: Search API Retrieval Eval Finding the best search for your agents.Benchmarks of search APIs tend to grade providers by judging the answer a model writes after retrieval. In my view that is a judge of the model more than of the search provider itself. With Sourcery I wanted to evaluate these providers controlling for the LLM, judging on the fetched results rather than the output. All eight providers got the same 204 questions in a single run, then 3 models, Claude Sonnet 5, GPT-5.6 Terra and GLM 5.2, read every page and scored it. Disclaimer Independent Nobody paid me to run this, no provider saw it before it went up, and there are no affiliate links on this page. Some of it ran on credits I had from hackathons, friends, etc. Self-funded The prices on this page are what I got charged. There are good pay-as-you-go plans and free tiers to try each of these. I spent around $40.08 across the eight providers. Checkable Every question-rating pair can be checked in the explorer, with the sentence the judge wrote to justify its rating. Read them there. Pooling Every page is graded once, so if several providers returned the same link I only ran it past the judges once, to control cost. There is a real methodological argument for pooling too. A page is detached from whoever returned it before anyone grades it, so there is no provider name for a model to be swayed by. 12,954 question-page pairs collapse to 7,496 unique ones, and grading the unique set meant 42% fewer judge calls. Where several providers returned the same URL, the longest extraction is the one that got graded, so extraction quality is averaged away on those pages. Would not pool next time with a larger budget. Context management To preserve cost and context size, each result is truncated to 1600 characters before a judge sees it. That is a realistic slice of an agent's context budget and it was also the cheapest way to make this many ratings affordable, but it cuts hardest against the providers that return whole documents, so a bigger slice could reorder the bottom of the table. The reason I do not think it is only measuring truncation is Exa and Brave, which run past the budget on 93% and 77% of their results and still place fourth and second. More on that in the limits. Experimental design This run was designed as a follow-up to the first one on Sourcery. Making it agnostic to the model's answer was my takeaway from run 1, which found very small differences between search APIs largely because the models already knew a lot of the answers. Run 2 overhauls the scoring, widens the field of both providers and judge models, and raises the latency threshold so the slower providers get to finish. Looking at you, Bright Data. I thought it important to be upfront about the limits of this run. If there is something you would want done differently, DM me on X. There is more in the methodology, and what this cannot tell you. How to pick your provider What you're building decides the best provider for you. Three of these eight are tied on the score below and they still aren't interchangeable: one returns snippets, one returns prose with the links stripped out, one returns excerpts with the links left in. So pick on the job at hand, and weigh what your code can check: it can see an empty result and handle it, but it is much harder to account for a confident near-miss. Clear of the field on the score and on the cheapest tier. Fully answers 49.9% of the time, about 1.4× anyone else. But to be clear, with Perplexity you are getting excerpts not whole pages. Only 40% of its results even reach the 1600-character budget, against 98% of Firecrawl's. So if your agents need to read a document end to end rather than a summary, providers like Tavily, Firecrawl and Exa are your best bets. Keeps structure and strips navigation: 89% of its pages have headings, with only 7% having links. Top-cluster relevance, and it goes toe to toe with Parallel while having a much better developer experience and dashboard. The best option when you are feeding text to a model and link noise costs you context. Third, close enough to Brave and Exa that the order between the three of them is noise, on the cheapest tier at $0.005 a query, and quick. The thing that sets it apart is that alongside Perplexity it is the only arm that does not drop between the base and the hard half, while everyone else gets worse exactly where you needed the help. 89% of its pages keep headings and 36% keep links, so it sits between Exa's stripped prose and Tavily's whole documents. Ranks second on relevance and returns real page text on 1% of results, and its content is search snippets joined together. Fine for grounding a summary, but wrong for anything that has to read a page. On the 12 questions with no answer, it produced the fewest convincing near-misses, beating Perplexity on all 12. That is 12 questions, so read it as a signal rather than a rank. Most expensive of the eight. Full page markdown on 98% of results, with headings, links and images intact. Outscores Firecrawl and costs less. If your agent needs the document rather than a summary of it, start here. The order changes on the 12 questions with no answer. These were designed to rank which providers are most likely to cause models to hallucinate. Perplexity returns the most convincing near-misses, with Firecrawl being the best for this. Results Each provider gave back 8 pages per question, then a judge saw all 8 together and answered one question: could you answer from these alone? Zero means nothing on that set was useful; three means a full answer is in there. That is one verdict on the whole set, not an average of page scores. The pages are also rated one at a time, separately, and that number appears further down as the page rating. Perplexity is clear of everyone else here. Below it there are two clear clusters, and inside either one the order doesn't mean much. Ask again next week and they could come back in a different sequence, but the gap between the two groups is the substantial one. Exa to Tavily is four tenths of a point on the 0–3 scale, bigger than anything inside either cluster. The ordering holds on the individual page scores too. Perplexity returns a page that fully answers the question 49.9% of the time; Bright Data manages 14.9%. Base vs. hard questions I wrote half the set to be hard: multi-hop, ambiguous, or needing a source that isn't the first result. This chart is the page rating, not the set rating above: it scores each page on its own rather than the eight together. That is why the order is different: Exa's individual pages beat Parallel's, while Parallel's eight fit together better and win on the set. The benchmark search is on questions your model could nearly answer on its own, without really needing the retrieval. The difference shows up on the hard ones, where model intelligence can't carry the weight. That is also where the weaker four pull away downward. Cost The three providers on the cheapest tier take three of the top four places, with Serper, the outright cheapest, placing sixth. Serper matches Tavily in quality at a fifty-fourth of the cost. It returns links and snippets and never page content, so if you already run a scraper, it may be a good fit. Speed Speed is the other factor worth weighing. For every query I recorded how long the provider took to return its results. Brave answers in a median of 0.6s per search, about 3 times quicker than the closest competition; Serper, the cheapest arm, takes 3.2s. Bright Data is the slowest at 34s, and its slowest one search in ten takes 75s or worse. Returned content Brave returns real page text on 1% of results, which is why the picking section lists it as summaries-only. Exa keeps headings and drops links, which makes it the cleanest thing to put in an agent's context. The other axis is volume: an agent gives each source a slice of its context, and this run allows 1600 characters, which is a realistic budget. 98% of Firecrawl's pages run past it; 40% of Perplexity's do. Firecrawl returns whole documents and Perplexity returns excerpts, and past the budget that difference stops reaching the model. In real cases I find the higher markdown volume better, because it takes that to carry the full context of a page. Why this disagrees with answer-graded indexes Artificial Analysis published a search index as I was writing this, and the two grade different things. They grade what an agent eventually produces: up to 25 turns and roughly nine to thirteen searches per task, and then the final answer gets scored, so a weak first result set just gets repaired by searching again. Sourcery grades what one search hands back: 8 results, one shot, no recovery. I did it that way to sit as close to real agentic use as I could. Their questions are static obscure facts, much like run 1 of Sourcery; mine lean fresh. One caveat worth naming: neither index rewards Firecrawl's extraction. Their settings disable its scraping by name and route everyone through one generic extractor. This run asked for full page markdown, got it, and then metered every source to the same context budget, which most of those pages exceed. Their index carries no confidence intervals and their top five span two points. On their published sample sizes that gap is inside the noise, which is the same reason three providers here are drawn as a tie. Scope matters in this table. “Set rating” grades a provider's eight results together and is the headline. Every other quality column grades pages one at a time, which is why a provider can lead on one and not the other. Both scales are 0–3. “Fully answers” is the share of a provider's pages the judges scored 3. “Over budget” counts how often a page ran past the 1600-character per-source allowance, which measures how much text a provider returns. Serper is n/a because it returns no page text. “Median latency” is how long one search took, measured from my machine on the day. * based on the quoted price on the website. Notes from the run Getting these numbers was messier than the table makes it look, and the parts that went wrong say as much about these products as the ranking does. Bright Data throttles above one concurrent request, so its arm ran at a crawl and needed a second repair pass that went past midnight, which is why this run is stamped across two days instead of one. Tavily's plan ran out around question 150 and the rest of its questions had to wait. Firecrawl went through 3,493 credits in a day, 70% of a month's allowance and $25 of prepaid credit, because it bills web and news results separately. I also tried a keyless arm as a free-floor control and dropped it after eight consecutive captcha failures and no successes at all. Need to work on that one. One thing I would tell anyone repeating this: run it before you read it. A batching bug cost about $19 of retrieval that produced nothing usable. Of 204 questions across eight providers, 1,631 of 1,632 fetches came back clean; the single miss is Firecrawl timing out at 300 seconds, which I checked against their API by hand before recording it.