{"slug": "agents-shouldnt-run-your-whole-retrieval-stack", "title": "Agents Shouldn’t Run Your Whole Retrieval Stack", "summary": "The Georgian AI Lab benchmarked six agentic search tool configurations and found that embedding re-ranking inside the search tool lifts evidence recall by roughly 27% versus the same tool without one, with the best setups beating primitive keyword and semantic search tools. The team reports that about 99% of successful agent trajectories open with broad semantic queries before specializing, with keyword search making up roughly 58% of subsequent queries. They argue the retrieval control plane should be split so agents choose search methods while the search tool handles re-ranking.", "body_md": "*Key Take-aways:*\n\n- ***Split the retrieval control plane between agent and search tool.** The agent should decide between search methods; your search tool should re-rank documents.*\n- ***Given named tools, agents come up with intelligent search patterns. ~** 99% of successful trajectories open with broad semantic search queries; then specialize - with keyword search being ~58% of queries thereafter.*\n- ***Re-ranking** **survives the transition to agentic search**. A re-ranker inside the search tool lifts evidence recall by ~27% relative to the same tool without one.*\n\nThe Georgian AI Lab regularly helps teams get retrieval systems into production. We’ve often  observed that teams have layered search stacks, including multiple retrievers, a rank fusion step[1](#footnote-1) between these retrievers and finally a cross-encoder[2](#footnote-2) to rerank and shortlist the most relevant documents. This is the RAG (retrieval augmented generation) pipeline that we’ve seen commonly deployed across the industry.\n\nRecently, we have begun to see a gradual shift away from static RAG pipelines towards iterative agentic-search: hand an agent a `search()` tool to interact with your corpus and let it repeatedly search and refine its query until it finds the desired information, the same way humans search. Teams in this space such as [sid.ai](https://turbopuffer.com/blog/reinforcement-learning-sid-ai) and [mixedbread](https://www.mixedbread.com/blog/toast-1) have demonstrated the capabilities of fine-tuning specialized agents for this iterative search pattern.\n\nHowever, we noticed something strange about the design of agentic search harnesses. Agentic search is framed as replacing the RAG pipeline with a loop and *primitive* search tools (return top k documents along some dimension of similarity). While making search tools for agents as basic and atomic as possible seems logical from a first principles point of view, we found it surprising how little benchmarking has been done for different tool shapes.*Instead of giving the agent 2 basic search tools* `(keyword_search()`  *and* `semantic_search())`*, what happens when the search tool our agent uses in a loop does rank fusion between different methods internally? Even further, what if the agent’s search tool is an entire rank fusion + re-ranking pipeline?*\n\nTo answer this, we benchmarked 6 different tool configurations to map out how the agent uses them and which ones balance cost and performance the best.\n\nWhile all of these agentic search setups perform much better than RAG with no looping, there is in fact a lot of variance in both cost and performance when we just change how the agent’s search tool looks. Notably, the top two configurations of our experimental setup both beat the basic search tools default approach comfortably. Giving an agent the full RAG pipeline (fusion + re-ranking) behind a search tool vs giving the agent multiple search retrieval tools that each re-rank internally, produce similar results on retrieval. **The common ingredient between the best configurations in our study was re-ranking inside of the tool boundary.** Our guess as to why agents with primitive search tools can’t keep up is that when rank fusion and re-ranking is not present within the search tool, the agent bears the load of decisions that were previously decomposed as steps in a retrieval pipeline.\n\nAfter observing how agents behave under many of these different tools and comparing what happens when decisions are a part of the tool vs the agent’s reasoning, we built a mental model for how retrieval really functions and how agentic search fits into that model. In this blog, we aim to explain this mental model to you, and how it may assist you in designing your own agentic retrieval stack.\n\n# Retrieval Control Plane\n\nWe define the **retrieval control plane (RCP)** as the set of decisions a retrieval system must make: which retriever to use, how to combine retrievers and how to rank candidates. In traditional RAG setups (without loops), your retrieval step actually takes control of your entire **RCP**.\n\nThe switch to agentic search introduces an allocation-of-control problem over this **RCP** i.e., we need to now decide who is responsible for the different parts of the retrieval stack. In an agentic search setup with even a basic tool, the decisions within your **RCP** are split between your tool and the agent’s reasoning. As you add more layers within that tool, you take control of your **RCP** away from your agent and hand it to a pre-configured tool instead. One of our core findings was that agentic search performs better as we lean further into the tool to re-rank candidates - taking over a core element of the **RCP**.\n\nIn one of the most performant configurations on our benchmark (rank fusion + rerank): the tool actually contains most of the **RCP** under the tool boundary; retrieval decisions are made without the agent’s input:\n\nAs you consider how to best allocate the **RCP** for your own agentic retrieval use case, I think there are three fundamental questions that need to be addressed:\n\n1. Who decides which retrieval method is best?\n2. Who decides how to balance multiple-retrievers?\n3. Who re-ranks documents with the search query in mind?\n\nThe answer to each of these questions will be either the tool that you configured while building your stack or your agent who is actively searching. As an illustrative example of how you can follow this mental model to determine which tool may be best for your agent, we want to walk you through our experiments assessing and answering each of the above questions, first in isolation, and then concluding with what this means for the **RCP** overall.\n\n# Experimentation\n\nIn our experiments we have an agent that is allowed to call a search tool repeatedly (capped at 12 calls). Between run configurations, we simply swap whether retrieval decisions are fixed and hidden within the `search()` tool, or whether the agent makes them actively at inference time.\n\nWe use multi-hop Q&A as a representative example of what agentic search queries demand. The agent reasons over the question and searches broadly to collect information towards the ultimate question being asked. [BrowseComp-Plus](https://github.com/texttron/BrowseComp-Plus) closely matches this shape; the dataset evaluates multi-hop queries against a fixed, curated corpus of ~100K human-verified documents. The corpus is derived from live-web documents and frozen in place after document validation for a more controlled evaluation over retrieval tasks. Each question is paired with a set of human-verified labels for both evidence documents and answers to allow us to evaluate retrieval and responses simultaneously. All configurations run GPT 5.6 Luna as the agent with a maximum of 12 search calls, over 400 queries from the BrowseComp-Plus dataset.\n\n# Question 1: Who decides which Retrieval method is right for a task?\n\nWe looked at a simple demonstration. In searching against a text corpus, should you use a lexical heavy strategy in BM25 or semantic heavy strategy in using a vector embedding model?\n\nIn the agentic search framework, there is compelling work which argues that [the lighter search module (BM25) is enough](https://arxiv.org/pdf/2605.10848) because the ability to search repeatedly makes up for the shortcomings of the method. We remind you that the goal of our experiments is not to argue which retrieval method is best for this dataset. Instead we want to explore whether the choice should rest with the designer of the search tool or with the agent, the actual user of the tool.\n\nThese are the exact configurations we compared:\n\n1. **Generic semantic:** The agent gets one tool named ‘search’ whose description says nothing about what is behind it, the implementation is using dense retrieval (using[qwen3-embedding-4b](https://huggingface.co/Qwen/Qwen3-Embedding-4B) model).\n2. **Generic BM25:** The agent gets the same tool and the same description as configuration 1, with BM25 search behind it instead.\n3. **Two named tools:** The agent gets two tools, semantic_search and keyword_search, each with tool descriptions that reveal what the tool is and a one-liner on how to use them.\n\nAll three return the top 5 documents per call, with no fusion and no reranking. The system prompt is identical across all three.\n\nIf we tried to make a decision between different retrieval methods and embed that into our retrieval tool, we would see no clear answer here; dense-versus-BM25 difference is negligible on this corpus (0.448 against 0.455), with a confidence interval that includes zero. However, the two-tools setup, where the agent controls which retrieval method to use, recovers more evidence than either approach alone. The dual tools setup does cost more: it is about a third of a search call longer and spends 7 to 15% more on tokens than the single-retriever configurations. The extra turns are used to retrieve more evidence by balancing both tools together.\n\n## What the agent does with two tools\n\nIn line with an investigation [by Vercel and Braintrust](https://vercel.com/blog/testing-if-bash-is-all-you-need) into a similar dual-tools approach (their setting compared SQL queries and bash commands to navigate a corpus of GitHub issues), we found that agents can adapt their search queries and search patterns when given access to multiple tools:\n\nWhen given access to both tools, the agent implicitly decides to write long descriptive strings for the embedding index, and short keyword-heavy strings for BM25. In light of [Georgian’s recent post](https://georgianailab.substack.com/p/your-queries-can-reorder-the-retrieval) detailing how the written style of queries can impact the retrieval quality of embedding models, it is particularly important to note when agents choose to write queries of different shapes. Here is a search trace from the agent with access to both tools to demonstrate the difference in search queries going to the two tools:\n\nSo what does this mean for our retrieval control question, should the retrieval method choice sit within our search tool or with the agent?\n\n**Finding: Lift the retrieval method decision out of the black box tool.** In our experiments, the agent queries the two tools differently and exploits the strengths of each of them in the specific cases where they are most useful, recovering more evidence than either single retriever. Although this method is slightly more costly and uses up more tokens, the alternative of fixing one search method for an entire corpus may be restrictive for retrieval.\n\n# Question 2: Should retrievers be fused inside a tool call or orchestrated across calls?\n\nAlthough a choice between retrieval methods is sometimes made, practitioners often instead fuse multiple retrieval methods together to combine their respective strengths. The most common fusion technique is [Reciprocal Rank Fusion (RRF)](https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf): send the query through both methods, merge the ranked lists by rank position and hand back one blended top-k. The question is whether RRF  remains the preferred  way to fuse different retrieval choices, or can we instead rely on the agent to mediate between the different retrieval methods itself.\n\nWe compared two configurations here:\n\n1. **RRF** : The agent gets a single tool named ‘`search` ’ with the same generic description as before. Behind it, each query goes to both retrievers at depth 50, the two ranked lists are fused with RRF and the top 5 come back.\n2. **Two named tools** : The agent gets`semantic_search` and`keyword_search` with the descriptions shown earlier, each returning its own top 5 -*the same as configuration 3 from above* .\n\nSimilarity of performance makes this decision a little more complicated - so we turn to other goals beyond accuracy: cost and observability.\n\nWhen the agent mediates between two tools, it spends about half a search call more than fusion and 15% more tokens per question for the same recall. In the two-tools configuration we are not paying for more performance, but for observability over our retrieval tasks. The decision to reach for a lexical instead of a semantic tool is a line in the agent’s trajectory - whereas rank fusion obscures **(i)** why a certain query may have formed certain orderings within each retrieval tool and **(ii)** how RRF created the final 5 documents to be revealed. The agentic search philosophy makes agent recall quality easier to trace back to the usage of the retrieval tool and thus easier to correct. We think that’s worth the extra cost in this case. Also, we believe language model costs will fall in future; especially with the current frequency of release of open-weight models accelerating that trend. We think the traces we gain will remain valuable.\n\n## Agent orchestration also unlocks a move fusion cannot make\n\nFusion always sends the same query to both retrievers, so it cannot express “search by meaning, read what comes back, then search for the name that was just found.” We see this pattern in our search trace above (question #1211) and time and time again once the agent has access to both tools: 36% of all keyword queries contain a proper noun or quoted phrase that appeared in an earlier semantic snippet and not in the original question.\n\nWe can even quantitatively plot this search pattern:\n\n**Finding: debatable, but we argue that this should be managed by the agent.** Fusion and dual-tools agents more or less tie on recall and on accuracy. Fusion is cheaper, but does not give you the query adaptation and observability that agent orchestration does. The winner of this decision depends on your preference. We favor observability and thus lean towards agent decision making as the better option. We expect the higher cost of the dual-tool setup to drop in the future.\n\n# Question 3: Does an agentic search loop make reranking redundant?\n\nRe-ranking involves looking at a large retrieved set of documents — hundreds of them — cross-encoding each document with the query in order to rank the top 5 that are truly most relevant to your question out of your candidates. Now to rerank you have to see the candidates, but repeatedly feeding hundreds of documents averaging five thousand words each (length based on the BrowseComp-plus corpus) into the primary agent’s context is the textbook setup for [context rot](https://www.trychroma.com/research/context-rot). Context rot is where the agent has far too much irrelevant information within its context and performance falls. The primary agent sees only 5 documents intentionally. The mechanism of short-listing the 5 has to sit within the tool where we can afford to look at everything.\n\nThis leaves the alternative version of the re-ranking argument: maybe an iterating agent does not need a reranker at all. Instead, give the loop enough turns and it should eventually surface what a reranker would have shortlisted anyway.\n\nWe compare two pairs of configurations (with and without re-rankers):\n\n1. **The RRF pair** : the RRF configuration from the last section — the same generic search tool with a deeper pipeline behind it — RRF surfaces 50 documents and a cross-encoder picks 5 that the agent is allowed to see.\n2. **The two-tools pair** : the two named tools from the last section, with the same descriptions, where each tool now retrieves 50 candidates and a re-ranker filters them down to 5 before returning.\n\nThe agent sees 5 documents per call in all four setups. The only thing added is a cross-encoder ([cohere rerank-4-fast](https://cohere.com/blog/rerank-4)) inside the tool.\n\nBoth architectures (RRF and two-tools) see a significant uplift from reranking - landing at similar performance after the uplift. The agentic-search loop does not replace re-ranking, in fact it can still gain significantly from it.\n\nThe cross-encoder adds its own cost. However, the intelligence it provides helps save some agentic search turns: both reranked configurations converge more than a full search call sooner. Thus, it appears that the added feature partially offsets its own cost.\n\n**Finding: keep the re-ranker within the tool call.** This is an interesting result that we did not see highlighted in other agentic search work we looked at. Reranking is load-bearing under both the old architecture and the new one. Agentic search does  not replace reranking because the agent cannot rank a hundred candidates that it can not afford to read.\n\n# So who ends up making the decisions?\n\nWe found that the retrieval control plane should not dissolve into the agent, nor should it stay completely within the `search()` tool, abstracted away from the agent. Different retrieval decisions must be made by different parties.\n\nOn the BrowseComp-Plus dataset, here is what we think would be most effective\n\n**Figure 4.** Agentic search consumes the choice of retriever and the choice of how retrievers combine. It does not consume ranking. What is left below the line is a deep candidate pool and a cross-encoder.\n\nWe believe that the retrieval control plane has been well designed for mature RAG applications. In moving towards agentic search implementations, we delegate a lot of the decisions that sat under the previous retrieval pipeline to the agent instead. For a more effective balance of decisions, we found that an agent should be given multiple retrieval tools and be allowed to make decisions between them; however, the agent should still be supported with a re-ranker sitting within the tool.\n\nIf you are building a search agent - or have a monolithic `search()` tool call, consider trying this:\n\n1. Give your agent access to multiple retrieval methods on the same dataset, your agent can adapt to use individual ones when appropriate.\n2. Expose each retrieval method as named tools: better observability over your agent’s decisions and retrieval. However, if you are cost-sensitive, you can just as well fuse them inside your search tool.\n3. Keep a re-ranker within your search tool even when relying on agent iterations. It is the single largest effect here and it is too costly and harmful for context to try to replicate reranking within your agent ( *at least as of now)* .\n\nThese results are from one corpus, one model, and one task shape; your team’s dataset may look very different. Consider exploring how **your** retrieval control plane is distributed and design deeper tools for your search agent on your own dataset. If you are building a search agent over your own documents and are seeing something different from what we found, we would be interested in hearing about it.\n\n# Additional Reading:\n\nIf interested in learning more about changes in the retrieval paradigm based on smarter, iterative reasoning agents, consider reading the following work that helped inspire our work.\n\n- Earlier work released by Vercel and Braintrust demonstrating the benefit of handing a retrieval agent multiple retrieval tools (though their “multiple-tools” are SQL and bash tools for navigating structured data)\n- How to evaluate retrieval systems using LLM as a judge:\n\n- Recent paper that opens up the retrieval black box to the agent and allows access to multiple-retrieval tools further tuned with SFT and RL.\n- This blog covers a search agent published by [sid.ai](http://sid.ai) (in collaboration with turbo-puffer) with a focus on allowing it to make more decisions that are hidden within the black box retrieval tool with further RL training.\n- During our investigation we noticed agents use different writing styles when sending queries to different tools. Georgian has explored the impact of query writing style on embedding leaderboards.\n\n*Siddharth Arya is an AI Engineer at Georgian, where he works on evaluation and agentic AI systems.*\n\n*Grateful to Paul Inder, Kshitij Jain, David Tingle and Kartik Gupta for their thoughtful feedback on this piece.*\n\n*This blog is provided for informational purposes only and should not be relied upon as legal, business, investment, or tax advice. Nothing in this blog constitutes investment advice, nor is it intended for use by any investors or prospective investors in any Georgian funds. This blog may include links to external websites or information obtained from third-party sources. Georgian has not independently verified and makes no representations regarding the accuracy or completeness of such information, whether current or ongoing. If this content includes third-party advertisements, Georgian has not reviewed such materials and does not endorse any advertising content or the companies referenced.*\n\n*Any investments or portfolio companies mentioned are for illustrative purposes only and may not be representative of all investments made by funds managed by Georgian. Please contact Georgian for more information.*\n\n[1](#footnote-anchor-1)\n\n**Rank Fusion** is a method used to combine multiple ranked lists of search results into a single, unified list. For more context: [https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf](https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf)\n\n[2](#footnote-anchor-2)\n\nWe use **cross-encoder** and **reranker** interchangeably in this context: A **reranker** is a model that jointly embeds queries and candidate documents in order to score how well said document addresses said query. You run this over your query and call candidate documents to rank the best one. For more context:", "url": "https://wpnews.pro/news/agents-shouldnt-run-your-whole-retrieval-stack", "canonical_source": "https://georgianailab.substack.com/p/agents-shouldnt-run-your-whole-retrieval", "published_at": "2026-09-10 15:12:34+00:00", "updated_at": "2026-09-10 16:14:52.719592+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "natural-language-processing", "ai-tools"], "entities": ["Georgian AI Lab", "sid.ai", "mixedbread"], "alternates": {"html": "https://wpnews.pro/news/agents-shouldnt-run-your-whole-retrieval-stack", "markdown": "https://wpnews.pro/news/agents-shouldnt-run-your-whole-retrieval-stack.md", "text": "https://wpnews.pro/news/agents-shouldnt-run-your-whole-retrieval-stack.txt", "jsonld": "https://wpnews.pro/news/agents-shouldnt-run-your-whole-retrieval-stack.jsonld"}}