{"slug": "your-queries-can-reorder-the-retrieval-leaderboard", "title": "Your Queries Can Reorder the Retrieval Leaderboard", "summary": "A developer's investigation into retrieval leaderboards reveals that rankings are highly sensitive to query wording, meaning models topping benchmarks may not perform best on an application's own data. The analysis shows that LLM-generated queries are significantly longer than human-authored ones, which can skew results, and concludes that leaderboards should only be used to shortlist candidates, with final model selection requiring custom evaluation.", "body_md": "**Key Take-aways:**\n\n- A retrieval leaderboard can be used as a proxy for selecting the best embedding model for a semantic search-powered application\n- But, as we have found, these rankings are sensitive to the exact wording of queries in the data.\n- If you want to optimize retrieval quality, treat leaderboard rankings as a shortlist to run your own testing rather than a final decision.\n\nHere’s a decision a lot of us have faced: choosing an embedding model for your application’s semantic retrieval system. A typical approach might be to search for “best embedding model” which might lead you to find the leaderboards that track retrieval quality: for example, the [Retrieval Embedding Benchmark (RTEB)](https://huggingface.co/spaces/embedding-benchmark/RTEB) and the broader [Massive Text Embedding Benchmark (MTEB)](https://huggingface.co/spaces/mteb/leaderboard). Perhaps you look at some domain-specific results. You take the model at the top of the leaderboard, wire it into your application, and the results are… good enough, but not great.\n\nSure, maybe you had been given the advice to create your own evaluation using your own data… but that sounds like a lot more work than just selecting from a table. Just pick a model that looks ok and be done with it, right? Easy.\n\nSo what’s wrong with the easy path? A leaderboard scores every model against one benchmark or a fixed combination of benchmarks. Someone else built those benchmark datasets from end to end. Other than domain, your application may have little in common with the datasets, and what you want to know is which model works best on your own queries and your own documents.\n\nIn one study, the authors found that humans submitting search queries to an LLM had longer queries than those submitting through a traditional search box (6.06 words vs 4.19 words) ([Wazzan et al., 2024](https://arxiv.org/abs/2401.10184)).\n\nMany retrieval benchmark datasets are built using queries written by humans. But your search application may not involve human-authored search queries at all. Telemetry reported in another study put typical LLM queries near 23 words ([SOCi, via Localogy](https://www.localogy.com/2026/02/soci-study-llm-queries-are-6x-longer-than-search-queries/)). If an LLM sits anywhere in your pipeline — a retrieval-augmented generation (RAG) query rewriter, or an agent composing a tool call ([Ma et al., 2023](https://arxiv.org/abs/2305.14283)) — the query reaching your retriever is LLM-authored and in an LLM’s style: longer, and more elaborated.\n\nYour queries likely look different in form from those in a benchmark dataset; how might differences between queries change an embedding model’s retrieval performance? Can these differences change the leaderboard? That is what I set out to explore in this blog. My conclusion is that a leaderboard should be treated as a way to shortlist model candidates and no more. For optimal retrieval performance in your application, the verdict has to come from your own data.\n\n## How a retrieval leaderboard is built\n\nA retrieval benchmark dataset needs three fixed ingredients: a *corpus* of documents, a set of *queries*, and *relevance labels* to indicate whether a specific result is a relevant result for a given query (a.k.a. *qrels* - the answer key). Each model on the leaderboard embeds every document and every query, and for each query it retrieves the nearest documents in embedding space.\n\nThe ranked lists of results for each query are scored using *[NDCG@10](https://en.wikipedia.org/wiki/Discounted_cumulative_gain)* (normalized discounted cumulative gain at rank 10). In plain terms it asks: of the ten documents the model put on top, how many are on the answer key, and how high did the model rank them? More credit is given for putting relevant documents near the top than near the bottom, and the score is normalized so that a perfect ranking is 1.0 and every query is comparable. Average that score across all queries, rank the models by the average and bing-bang-boom you have the leaderboard.\n\nThe usefulness of a leaderboard to *you* rests on an assumption of transfer that the model which wins on the benchmark’s queries and labels will also win on yours. That holds only if your queries resemble the benchmark’s queries and your documents resemble its corpus.\n\n## What I did\n\nThe focus of my exploration was on varying the *form* of the queries. For each dataset used, I held the corpus and qrels fixed, performed transformations on the queries, and re-scored every model.\n\nThe original queries from each dataset were transformed in three ways: a *paraphrase* that preserves length and meaning, a compressed *terse* keyword form, and an expanded *verbose* elaboration. The intent was to emulate what may actually happen to a query in a production system — an agent restating a request (*paraphrase*), a user typing a couple of keywords (*terse*), and an LLM expanding a short question into a full information need (*verbose*). Four LLMs from four different families (gemini-2.5-pro, claude-opus-4-8, gpt-5.1-2025-11-13, meta-llama/llama-3.3-70b-instruct) rewrote each query independently, so I could tell whether an effect holds across generators or is just an artifact of one model’s writing style.\n\nI used three datasets, each with a different number of labels for documents associated with each query:\n\n- [ChatDoctor](https://huggingface.co/datasets/embedding-benchmark/ChatDoctor_HealthCareMagic) labels one document per query,\n- [CUREv1](https://huggingface.co/datasets/clinia/CUREv1) labels ~40 documents per query, and\n- [TREC-COVID](https://huggingface.co/datasets/mteb/trec-covid) labels several hundred per query. TREC-COVID also ships three human-written phrasings of every query topic - keyword, question, and narrative.\n\nSometimes I may make reference to these label densities: a dataset with few labelled documents per query has *sparse* labels, and when there are many labelled documents per query the labels are *dense*. A single labelled document per query is a special case of sparse labels referred to as the “golden document”, the single document which contains the answer or is the “absolute best match.” In a dataset with only one labelled relevant document per query we are left to assume that every other search result is not relevant to the query, whereas in a densely labelled dataset a list of search results may include several relevant documents, perhaps with some documents labelled as more relevant than others. NDCG@10 only uses positive relevance labels - documents explicitly labelled as non-relevant are treated the same as unlabelled documents, which do not affect the metric.\n\nI included 21 embedding models (see the list in Figure 1 below) spanning from small self-hosted bi-encoders, to hosted LLM-backbone embedders like qwen3-embedding-8b and gemini-embedding-001.\n\nTo compare two leaderboards I used *[Kendall’s τ](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient)*, a measure of how much two rankings agree: 1 means identical order, -1 means reversed order, and anything in between indicates that some reshuffling has occurred. A τ of 0 means the two lists have no association with one another, in terms of their order. I set a bar of 0.9 as an indication of a real reshuffle - a convention borrowed from information retrieval which views two rankings that agree at τ ≥ 0.9 as essentially telling you the same thing ([Voorhees, 2000](https://dl.acm.org/doi/10.1145/290941.291017)). For a leaderboard with 21 models, ten neighbour swaps, each moving a different adjacent pair, results in a τ of 0.905 against the original order.\n\nI made sure to reproduce each model’s *published* score on the original human queries before trusting any scores using the transformed queries. I was able to do this for the models with comparable published anchors - everything for ChatDoctor and TREC-COVID reproduced within 0.01, and 16 of 17 on CUREv1 were within 0.01 (the last was just outside of the 0.01 tolerance). Reproducing these published scores provided confidence in the differences below being attributable to the rewrite rather than to our plumbing. If you want to check any of this, the code and the per-query score files are at `query_form_experiment` ([GitHub repo here](https://github.com/georgian-io/query_form_experiment)). The numbers in this post recompute from a clone without re-running anything paid.\n\n## How you phrase the query changes which model wins\n\nTo give you an idea for what each query transformation looks like, here is an example CUREv1 information need in the four forms (word counts in brackets):\n\nEvery term from the original form is kept in the verbose form, which adds “standard of care” and “clinical guidelines” scope. The added scope pulls off-intent documents into the top-10 and moves the ranking.\n\nHere is how these transformations reordered the leaderboard for each dataset.\n\nEach cell is the Kendall’s τ between the leaderboard under the transformed query form vs the same dataset’s original human-query leaderboard, averaged over the four LLMs . A Kendall’s τ of 1 would mean that transforming the queries left the ranking untouched. Starting with the paraphrase row - same length, same meaning, just different words - the leaderboard order changes for all three datasets. If a leaderboard’s ordering were a durable fact about the models, rewording the questions without changing what they ask should have left it alone.\n\nThe terse and verbose transformations reorder things even more. The verbose transform is the harshest on TREC-COVID at 0.607 and the gentlest on ChatDoctor at 0.938; the same instruction, the same four generators, pointing in opposite directions. I think that the pattern I see here — ChatDoctor’s reordering being less sensitive to query-form changes than the other datasets — may actually relate to how it’s evaluated compared to the others. I’ll come back to it in the next section.\n\nFocusing on TREC-COVID for a moment, it is the only dataset I studied which ships human-written alternatives to its own queries, which gives me a control to come back to. It also has the most labels of the three (several hundred per query) so I can actually see what a rewrite does across the whole top-10 instead of to a single document.\n\nI took the verbose rewrite and ranked the same 21 models twice: once on the benchmark’s original questions, and once on the elaborated version of those same questions.\n\nElaboration acts like a *leveller*. The models that do best on the benchmark’s short questions suffer the most in retrieval performance, and some of the weakest models seem to gain a lot. qwen3-embedding-8b is a standout case, moving from 15th on the benchmark’s original questions up to the top spot once those questions are transformed into verbose queries. Four of the top six on the original board move down.\n\nI’m not trying to argue that these leaderboards are wrong, but that they represent the results of answering in one query style. If your pipeline hands long, elaborated queries to the retriever, the ranking you should care about may look closer to the right-hand column of Figure 1 than the left.\n\nRelated work points the same way: embedding *rankings* shift under meaning-preserving paraphrase ([PTEB; Frank & Afli, 2025](https://arxiv.org/abs/2510.06730)); that leaderboards are not robust even to the *instruction prompt* paired with a model, so favourable prompt selection can lift essentially any model to first place ([Kostiuk & Enevoldsen, 2026](https://arxiv.org/abs/2605.22544)); and, pre-dating LLMs, that meaning-preserving query variation costs retrieval pipelines around 20% effectiveness ([Penha et al., 2022](https://arxiv.org/abs/2111.13057)).\n\nEvery rewrite so far was written by an LLM, which invites a fair objection: perhaps the results of this experiment say more about how language models write than about retrieval. TREC-COVID lets me evaluate this objection, because each topic also ships a keyword query and a paragraph-length narrative written by people, which means different forms of the same information need can be judged against the same labels.\n\nEach form of information need reorders the board, including the human ones provided in the dataset. narrative is a paragraph-length variant of question, written by a human, and it still reorders the board with τ = 0.726. Each of the LLM rewrites is somewhat harsher than the human form it imitates, so the LLM transforms overstate the effect a little but they are not creating it.\n\nBy construction those three human phrasings express one information need: the benchmark’s authors wrote them that way. The one-sentence question is generally the easiest form, keyword-form degrades performance of models roughly the same, while the narrative-form costs the *strongest* models the most.\n\nHaving three phrasings rather than two also lets us ask whether this is really a story about length. Not quite. No two of the three rank the models the same way, and the two extremes (keyword vs narrative) sit at 0.716, no further apart than either is from the question form. Short and long phrasings reorder different pairs of models, and the disagreement is about the same size whichever pair you pick.\n\nCompression and elaboration are on the same axis but run in opposite directions. Compressing a long narrative back into a short question—what a pipeline does when it summarizes a user’s request—improves 19 of the 21 models, by an average of 0.110 NDCG@10. And it isn’t even-handed; measured against an independent score of model strength, the stronger a model already is, the more compression will improve it (+0.44). Elaboration is that same relationship with the sign flipped, and the LLM’s verbose rewrite is sharper still at -0.78. Short queries suit the strong models; long ones level the field. That is the pattern in Figures 1 and 2.\n\nThere is a subtler point hiding in what benchmark results provide. Think of a single information need as having a whole distribution of query variants: transform a query a hundred ways, preserving the intent/information need, retrieve for each, and you get a spread of NDCG@10 scores. Each query in a benchmark hands you a single sample from that distribution, and it could be either a favourable phrasing or an unlucky one. The spread is wide. A single need can elicit thousands of distinct human phrasings ([UQV100, Bailey et al., 2016](https://dl.acm.org/doi/10.1145/2911451.2914671)). How wide the spread is depends on the model ([Campese et al., 2025](https://arxiv.org/abs/2508.07975)).\n\nWhich query sample/form you get affects some models more than others. This is what I see in the reordering experiments: elaboration costs the strongest models the most and lifts several of the weakest, so any benchmark’s own phrasing can flatter some models and shortchange others. My takeaway is that the order a leaderboard reports isn’t a property of the models alone, but a property of the models paired with the particular style and sampling of queries in the benchmark.\n\n## An incidental finding: more relevance labels helps to differentiate models\n\nThe datasets I used to explore query forms differ in how densely they are labelled. ChatDoctor marks one relevant document per query, CUREv1 about forty, TREC-COVID several hundred. At the beginning of this experiment I started with ChatDoctor and expanded with the others after noticing that its sparse single document-per-query answer key affects what an experiment can see at all.\n\nI’ll try to make that more clear by starting with what a single label can tell you. If a single document per query is marked relevant, NDCG@10 knows where that document landed and nothing else. The other nine positions are unjudged, and the metric reads them all as equally irrelevant. Relevance need not be equated exactly with being *the* gold document. There may well be a single *best* answer, but the other results can still be better or worse than junk. Take two models that both rank the judged document first:\n\n*Figure 4. The same two rankings, scored against a one-document answer key and a three-document one.*\n\nWhen 3 docs are labelled as relevant, if two of them do not land in the top-10 results it cuts the score considerably. Specifically, what separated the scores was the extra *relevant* labels. Plain NDCG@10 treats an unjudged document and a judged not-relevant document identically. However, labelling not-relevant docs can be useful within other metrics.\n\nBack to my own results for query rewrites against all three datasets:\n\nThe paraphrase query transformation, arguably the gentlest transform compared to terse and verbose, leads to a Kendall’s τ of approx 0.9 on all datasets. A slight reordering. The leaderboard is not shuffled more drastically for ChatDoctor when the query is transformed to be terse or verbose. However, on the two datasets with more densely-labelled query result sets, the terse and verbose query rewrites increasingly reorder the leaderboard. This is not a controlled ablation. These datasets differ in ways beyond the labelling, so this is a pattern across three points rather than a controlled result, but the direction is the one the arithmetic above predicts.\n\nIn practical terms, if you are interpreting benchmark results, or if you find yourself building an evaluation of your own, the number of relevant labelled documents-per-query can factor into how finely you can tell models apart. The leaderboard for a benchmark with a sparse answer key may not differentiate the retrieval performance of each model.\n\n## The leaderboard shortlists - your data decides\n\nAnchoring back to our start, if you’ve selected a model based on it being at the top of a retrieval leaderboard and were underwhelmed with the results, maybe it’s time that you reflect on how the queries in the benchmark dataset compare to your own application.\n\nLeaderboards can be directionally helpful, but relative performance rankings warrant a degree of skepticism when transferred to other data. Choosing a model for your own application will likely involve using some leaderboards to select candidate models, run the candidates over your own queries and your own documents, and label enough of what the queries return to tell them apart. The caveat is that your own evaluation inherits whatever query shapes you put into it. If your application is changed in such a way that queries take on a different shape or style, then a benchmark built from one shape is likely to have the same blind spot as the public one you are replacing.\n\nWhat I’ve aimed to establish is that leaderboard rank is contingent on the style and form of queries in the benchmark dataset, and the same query put another way can hurt retrieval in one model and help another. Evaluating retrieval performance on your own data will provide you with the specific leaderboard you need to choose the model best-suited for your application.\n\nThe evaluation of a group of models from a leaderboard on your own data is not something I’ve demonstrated end to end. The hard part isn’t running the models over your data. It’s deciding what counts as relevant, and applying that consistently across thousands of query-document pairs. Labelling a “golden document” for a few hundred queries is relatively easy - but creating a densely labelled query relevance dataset is a more difficult task. We’ve been building an internal tool to help scale this process using an LLM judge, which also facilitates the measurement of human-to-judge agreement to make it trustworthy. It’s the subject of my next post.\n\n*Paul Inder is an AI Tech Lead at Georgian, where he works with portfolio companies to implement agentic systems.*\n\n*Grateful to Kshitij Jain, Siddharth Arya and Aditya Shukla for their thoughtful feedback on this piece.*\n\n*This blog is provided for informational purposes only and should not be relied upon as legal, business, investment, or tax advice. Nothing in this blog constitutes investment advice, nor is it intended for use by any investors or prospective investors in any Georgian funds. This blog may include links to external websites or information obtained from third-party sources. Georgian has not independently verified and makes no representations regarding the accuracy or completeness of such information, whether current or ongoing.If this content includes third-party advertisements, Georgian has not reviewed such materials and does not endorse any advertising content or the companies referenced.*\n\n*Any investments or portfolio companies mentioned are for illustrative purposes only and may not be representative of all investments made by funds managed by Georgian. Please contact Georgian for more information.*\n\n## Additional Reading:\n\n- Muennighoff, Tazi, Magne, Reimers. *MTEB: Massive Text Embedding Benchmark.* EACL 2023.[arXiv](https://arxiv.org/abs/2210.07316)\n- *RTEB - Retrieval Embedding Benchmark.*[MTEB docs](https://docs.mteb.org/) ·[explainer](https://mrlcftech.substack.com/p/rteb-a-real-test-of-how-well-models)\n- Wazzan, MacNeil, Souvenir. *Comparing Traditional and LLM-based Search for Image Geolocation.* CHIIR 2024.[arXiv](https://arxiv.org/abs/2401.10184)\n- Boland / SOCi 2026 Local Visibility Index. *LLM Queries Are 6x Longer Than Traditional Search.*[Localogy](https://www.localogy.com/2026/02/soci-study-llm-queries-are-6x-longer-than-search-queries/) (industry)\n- Ma, Gong, He, Zhao, Duan. *Query Rewriting for Retrieval-Augmented Large Language Models.* EMNLP 2023.[arXiv](https://arxiv.org/abs/2305.14283)\n- Voorhees. *Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness.* IP&M 36(5) 2000.[ACM](https://dl.acm.org/doi/10.1145/290941.291017)\n- Frank & Afli. *PTEB: Robust Text Embedding Evaluation via Stochastic Paraphrasing.* 2025.[arXiv](https://arxiv.org/abs/2510.06730)\n- Kostiuk & Enevoldsen. *One Prompt is Not Enough: Instruction Sensitivity Undermines Embedding Evaluation.* 2026.[arXiv](https://arxiv.org/abs/2605.22544)\n- Penha, Câmara, Hauff. *Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators.* ECIR 2022.[arXiv](https://arxiv.org/abs/2111.13057)\n- Bailey, Moffat, Scholer, Thomas. *UQV100: A Test Collection with Query Variability.* SIGIR 2016.[ACM](https://dl.acm.org/doi/10.1145/2911451.2914671)\n- Campese, Moschitti, Lauriola. *Improving Document Retrieval Coherence for Semantically Equivalent Queries.* 2025.[arXiv](https://arxiv.org/abs/2508.07975)", "url": "https://wpnews.pro/news/your-queries-can-reorder-the-retrieval-leaderboard", "canonical_source": "https://georgianailab.substack.com/p/your-queries-can-reorder-the-retrieval", "published_at": "2026-09-03 14:58:15+00:00", "updated_at": "2026-09-08 21:58:09.769021+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-tools", "developer-tools"], "entities": ["RTEB", "MTEB", "Hugging Face", "SOCi", "Localogy"], "alternates": {"html": "https://wpnews.pro/news/your-queries-can-reorder-the-retrieval-leaderboard", "markdown": "https://wpnews.pro/news/your-queries-can-reorder-the-retrieval-leaderboard.md", "text": "https://wpnews.pro/news/your-queries-can-reorder-the-retrieval-leaderboard.txt", "jsonld": "https://wpnews.pro/news/your-queries-can-reorder-the-retrieval-leaderboard.jsonld"}}