# Perplexity launches a 190M-document retrieval test and keeps the answers private > Source: > Published: 2026-09-09 20:29:21+00:00 # Perplexity launches a 190M-document retrieval test and keeps the answers private **Q2D-Web uses 69,721 agent-written queries from production traffic and accepts open-weight models for a public leaderboard.** By [Ryan Merket](/author/ryan-merket) ยท Published Primary source: [X](https://x.com/perplexity_ai/status/2097782467210166601) ## Why it matters Agents can only cite evidence their retrievers find. Q2D-Web gives developers a production-shaped test, while its private data leaves Perplexity controlling the scoreboard. [Perplexity](https://x.com/perplexity_ai/status/2097782467210166601) launched Q2D-Web on September 9th, a benchmark designed to test the retrieval systems that decide which web pages an AI agent gets to read before it writes an answer. The benchmark pairs about 190 million web documents with 69,721 machine-written search queries derived from roughly 23,000 production searches. Perplexity collected the searches over nine months, removed queries flagged as containing personally identifiable information and retained the ten most common languages in the sample, according to the accompanying [technical paper](https://arxiv.org/abs/2609.08887), which was submitted on September 8th. Maximilian Schall, Sedigheh Eslami, Markus Krimmel, Antoine Chaffin, Louis Milliken, Bo Wang and Denis Bykov authored the paper for Perplexity. Several members of the group also built Perplexity's pplx-embed family, giving the researchers a direct stake in improving the first retrieval stage behind web-search agents. Q2D-Web addresses a widening gap between how retrieval models are commonly tested and how AI agents actually use them. Standard search benchmarks tend to use human-written queries. A production agent rewrites a user's request, incorporates earlier conversation turns, decomposes the task and issues several narrower searches. Those machine-written queries have a different distribution and often require exact dates, entities, versions or quantities. The first retriever in that pipeline determines the pool of documents available to every later reranker and language model. Evidence missed at that stage cannot appear in the final answer, regardless of how capable the downstream model is. ### Production traffic, with controlled access Perplexity divided the benchmark's queries into 12,365 primary queries and 57,356 supporting queries. Primary queries stay relatively close to the user's request. Supporting queries collect background information, alternative phrasings or details about adjacent entities. About 65.8% of the queries are in English, with the rest spread across Spanish, German, Russian, French, Portuguese, Italian, Korean, Japanese and Chinese. The corpus came from the top 5,000 production retrieval results for each query. Perplexity then used MinHash-LSH to deduplicate pages with highly similar text, leaving about 190 million documents. That construction method concentrates plausible distractors: pages that discuss the correct topic while missing a required date, product version or named entity. Perplexity is keeping the corpus, queries and relevance labels private. Developers can submit publicly available Hugging Face retrieval models through an evaluation form, and Perplexity will publish scores on a [public leaderboard](https://huggingface.co/spaces/perplexity-ai/q2d-web-leaderboard). The researchers argue that releasing the test set would allow it to leak into training corpora and eventually reward memorization. That design preserves the benchmark as a blind test, while making Perplexity the permanent evaluator. Outside researchers cannot independently reproduce a full run or inspect individual examples where the scoring system may have made a questionable relevance judgment. ### Three definitions of relevance Q2D-Web scores models against three relevance sets. The first treats pages cited by a production agent as relevant. The second uses up to 50 results per query from Perplexity's internal web-ranking stack. The combined set unions those signals and adds judgments from [DeepSeek-V4-Flash](/models/azure/deepseek-v4-flash) on previously unlabeled documents. Each method carries a different bias. Citations omit useful pages once an agent has enough evidence. Production rankings reflect Perplexity's existing retrieval and reranking choices. LLM judges can favor certain writing styles or document formats. Perplexity reports an average of 99.6 positive judgments per query in the combined set, substantially deeper labeling than benchmarks that attach one or a handful of positive documents to each query. The researchers also separated the older models used to build the evaluation pool from the neural retrievers being tested. Models released before January 1st, 2025 contributed to pooling and label generation, while every evaluated neural checkpoint was released after that cutoff. BM25 was the exception because it participated in construction and served as the lexical baseline. ### Perplexity's model wins Perplexity's test, mostly Perplexity evaluated 13 lexical, dense and late-interaction retrievers, using Recall@1000 as the primary metric. The metric asks how many relevant documents a model retrieves within its first 1,000 results, fitting the benchmark's focus on candidate generation rather than the final ordering presented to a user. Perplexity's [pplx-embed-v1-4b](https://huggingface.co/perplexity-ai/pplx-embed-v1-4b) led the web-ranking and combined evaluations, scoring 65.73 and 69.11, respectively. Nvidia's Nemotron-3-Embed-8B led the citation set at 61.68. Nemotron also beat the Perplexity model on Recall@100 and nDCG@10 under the combined labels, meaning it placed relevant pages nearer the top even though it recovered fewer across the first 1,000 results. The results remain a vendor-run evaluation built from Perplexity traffic and partly labeled by Perplexity's production stack. The multiple judgment sets and release-date holdout reduce obvious home-field effects, but they do not turn the leaderboard into independent validation. One finding complicates the usual model ranking. BM25 posted the lowest combined Recall@1000, at 44.77, yet recovered 14,621 relevant documents missed by every other retriever. Perplexity's 4 billion-parameter model found 884 such unique positives. The result suggests that retrieval systems may gain from combining architecturally different models instead of selecting a single leaderboard winner. Q2D-Web is also expensive to run. Perplexity says a full evaluation of pplx-embed-v1-4b required 4,608 Nvidia H200 GPU-hours. The researchers built a reciprocal-rank-[fusion](/models/openrouter/fusion) subcorpus containing 31.7% of the documents that preserved the full-corpus model order, reducing the same evaluation to roughly 1,500 H200 GPU-hours. Scores on the smaller corpus rose by three to seven points, so the sampled test is suited to ranking models rather than comparing its absolute numbers with full-corpus results. The benchmark gives retrieval developers a test shaped by deployed agent behavior rather than tidy academic queries. Its value will depend on whether Perplexity keeps the hidden evaluation stable, adds outside models promptly and discloses enough methodology for researchers to understand why a score moved.