Agentic RAG: What Happens When Retrieval Becomes a Decision Instead of a Step

Agentic RAG transforms retrieval from a fixed pipeline step into a decision-making loop where an agent retrieves, evaluates, and retries if necessary. This approach, implemented via frameworks like LangGraph, handles complex queries that single-pass retrieval cannot, but at the cost of 3-10x token usage and 2-5x latency.

An earlier post on RAG described it as handing a model the right reference right before it answers. That description was accurate, and it was also describing the simplest possible version of the idea: retrieve once, generate once, done. Agentic RAG is what you get when retrieval stops being a fixed step in a pipeline and becomes something an agent decides to do, evaluates, and redoes if the first attempt was not good enough. It is the same retrieval concept from before, placed inside the same think, act, observe loop covered in the agent posts, with the "act" step specifically being a search. Traditional RAG runs the same three steps for every query, regardless of what that query actually needs: embed the question, retrieve the closest matching chunks, hand them to the model to generate an answer. This works well for direct lookup questions, where the answer genuinely lives in one or two retrieved passages. It has no good answer for a question that needs comparing numbers from two different documents, or one where the first retrieval attempt returns irrelevant chunks because the query was phrased ambiguously. A fixed pipeline has no mechanism to notice that its own retrieval was weak, it just generates an answer from whatever came back, confidently, regardless of whether that evidence was actually sufficient. The core change is a control loop wrapped around retrieval. Instead of retrieve once and generate, an agentic RAG system retrieves, reasons about whether what it got back is actually enough to answer the question, and either proceeds to generate or goes back and tries again, a different query, a different source, a different tool entirely. This is the exact same loop shape from the earlier agent posts, applied specifically to the retrieval step. Where a fixed pipeline treats retrieval as a single deterministic action, agentic RAG treats it as one possible action the agent can choose to take, evaluate, and repeat. A few named patterns are worth knowing, because they show up constantly in how this actually gets built. Corrective RAG explicitly scores the relevance of retrieved chunks before generation, and if that score is low, triggers a fallback, often a web search, rather than feeding weak evidence to the model anyway. Self-RAG has the model reflect on its own retrieved context and its own draft answer, checking whether the response is actually grounded in what was retrieved before finalizing it. Adaptive and multi-hop patterns handle questions that genuinely require several rounds of retrieval, where each round's results inform what to search for next, closer to research than lookup. None of these are exotic, they are all specific implementations of the same underlying idea: do not trust the first retrieval blindly, build in a checkpoint that can catch a bad one. This is also exactly where a framework like LangGraph earns its place. A fixed RAG pipeline fits naturally into a linear chain, which is why plain LCEL-style composition works fine for it. The moment retrieval needs to loop back on itself, retrieve, evaluate, maybe retrieve again, that is precisely the shape a linear chain cannot express and a graph with a conditional edge can. An agent node evaluates retrieved chunks, a conditional edge routes either forward to generation or back to a retrieval node with a reformulated query, and that backward edge is the loop that makes the whole pattern possible, the same mechanism covered in the LangGraph post, just applied to a retrieval-specific agent instead of a general tool-calling one. None of this is free, and pretending otherwise would be dishonest. Adding an evaluation and retry loop around retrieval genuinely increases both cost and latency, current estimates put agentic RAG in the range of three to ten times the token usage and two to five times the latency of a single-pass retrieval pipeline, because you are now potentially running multiple retrieval rounds and extra reasoning steps the simple version never needed. That is not a rounding error, it is a real operational tradeoff, and it only makes sense to pay for use cases where the fixed pipeline's blind spot actually matters, ambiguous or multi-part questions, high-stakes domains where a wrong answer built on weak evidence is genuinely costly, not for straightforward lookup questions where one good retrieval pass was always going to be enough. ⁘ Agentic RAG is not a strictly better version of RAG, it is RAG with a loop wrapped around the retrieval step, and that loop costs real tokens and real latency every time it runs. ⁘ The actual mechanism worth remembering is evaluation before generation, checking whether retrieved evidence is sufficient rather than assuming it is, and only that checkpoint is what justifies the added cost. ⁘ If most of your queries are direct lookups with a clear answer in one or two documents, a fixed pipeline is not a lesser choice, it is the correctly scoped one for that problem. ⁘ Reach for agentic RAG specifically when queries are multi-part, ambiguous, or high-stakes enough that a confidently wrong answer built on a bad first retrieval is a real cost worth paying extra latency and tokens to avoid. Agentic RAG is not a separate technology from RAG or from agents, it is the intersection of both ideas already covered separately: the retrieval mechanism from the RAG post, placed inside the loop structure from the agent posts, with a graph like LangGraph providing the conditional edge that lets a weak retrieval trigger a retry instead of getting passed straight to generation. The value is real for the specific class of question a fixed pipeline cannot handle well. The cost is also real, and skipping the honest tradeoff is how a genuinely useful pattern turns into something reached for by default when a simpler pipeline would have done the job for a fraction of the price.