Standard RAG vs. Agentic RAG: Moving Retrieval From Pipeline Stage to Runtime Decision

A developer explains the shift from standard RAG to agentic RAG, where retrieval decisions are made at runtime by a planner rather than fixed at design time. The post argues that standard RAG is suitable for homogeneous corpora and lookup tasks, while agentic RAG handles complex, multi-source queries by dynamically orchestrating retrieval.

Standard RAG assumes the user's question maps onto one vector search. One query in, one embedding, one top-k lookup, one answer. That assumption holds up in demos, because demos ask demo questions. "What's our parental leave policy?" is one document. Retrieve it, stuff it into the prompt, done. Then you ship, and a real user types: "Did the carrier rate change we approved in Q2 actually reduce our cost per shipment in the Northeast, and does that hold if I exclude the Boston depot?" That question needs a policy document, a rate table, a transactional aggregate, and a filtered re-computation. Your retriever will embed the whole sentence, find the three chunks nearest to it in vector space, and hand the model text that is topically adjacent and factually useless. The model, being a good sport, will answer anyway. The problem isn't the embedding model or the chunk size. You hardcoded how many times to retrieve, and where to retrieve from, at design time, for a question you hadn't read yet. Agentic RAG moves that decision to runtime. Planners, memory, MCP servers, sub-agents: all of it is implementation detail hanging off that one change. STANDARD RAG — fixed pipeline, one pass ┌──────┐ 1. prompt+query ┌─────────────┐ │ User │ ───────────────────► │ Chat UI │ └──────┘ └──────┬──────┘ ▲ │ 2. query │ 6. response ▼ │ ┌─────────────┐ │ │ Retriever │ │ └──────┬──────┘ │ │ 3. fetch top-k, one shot │ ▼ │ ┌───────────────────────────┐ │ │ Knowledge Sources │ │ │ docs · PDFs · code · DB │ │ │ APIs · web index │ │ └───────────┬───────────────┘ │ │ 4. chunks │ ┌──────▼──────┐ └──────────────────────────│ LLM │ └─────────────┘ 5. prompt + query + enhanced context The defining property is that the model is never consulted about retrieval. It receives context and produces text, and retrieval already finished by the time it runs. That's a design choice with real advantages. One embedding call plus one vector query is cheap and predictable, latency sits in a tight distribution, and you can cache aggressively. Failures are legible: a bad answer means bad chunks, and you can go read the chunks. Your eval harness is a fixed input and a fixed output, so it works. Standard RAG is the right architecture when your corpus is homogeneous and your users mostly want lookup rather than synthesis. Don't let anyone talk you out of it for those workloads. AGENTIC RAG — planner decides retrieval at runtime ┌──────┐ 1. prompt+query ┌─────────────┐ │ User │ ─────────────────► │ Chat UI │ └──────┘ └──────┬──────┘ ▲ │ 2. query │ 6. response ▼ │ ┌──────────────────────┐ ┌──────────────┐ │ │ Aggregator / │◄────►│ Planning │ │ │ Orchestrator Agent │ 3. │ ReAct · CoT │ │ └───────┬──────────────┘ └──────────────┘ │ │ ▲ │ │ └──────┐ ┌──────────────┐ │ │ 4. fan-out └─►│ Memory │ │ ▼ │ short · long │ │ ┌───────────────┼───────────────┐ └──────────────┘ │ ┌────▼────┐ ┌────▼────┐ ┌─────▼───┐ │ │ Agent 1 │ │ Agent 2 │ │ Agent 3 │ │ └────┬────┘ └────┬────┘ └────┬────┘ │ │ MCP servers / tool layer │ │ ┌────▼────┐ ┌────▼────┐ ┌────▼────┐ │ │ SQL DB │ │ Search │ │ Data │ │ │ │ │ index │ │ Explorer│ │ └─────────┘ └─────────┘ └─────────┘ │ │ 5. prompt + query + enhanced context │ ┌──────▼──────┐ └─────────────────────│ LLM │ └─────────────┘ ◄── loop back to 3 if context insufficient Four things are different. 1. A planner sits between the query and the retriever. The aggregator agent decomposes the question before anything gets embedded. ReAct and chain-of-thought are the mechanism that turns one user sentence into a retrieval plan. The Northeast shipping question decomposes into four steps: fetch the Q2 rate change record, aggregate shipment costs by region and date, re-aggregate with the Boston depot excluded, compare. 2. Retrieval fans out across heterogeneous backends. Standard RAG normalizes everything into one vector index at ingestion time. Agentic RAG queries systems in their native language at query time: SQL against the warehouse, semantic search against the document index, a time-series query against event data. You don't embed a rate table, you query it. 3. MCP is the integration seam. When your retrievers are MCP servers rather than bespoke functions, the tool surface is declarative and swappable. You add a data source by registering a server instead of shipping a new agent build. It's a plugin architecture applied to retrieval. 4. Memory carries retrieval state across turns. Short-term memory holds what this session already fetched, so hop three doesn't re-retrieve what hop one found. Long-term memory holds durable facts about the user and their prior queries. Without it, a multi-hop system re-derives the same context every turn and you pay for it twice. The structural difference is the loop back to the planner. Standard RAG has one retrieval pass. Agentic RAG has a loop with a termination condition, and that loop is where both the value and the risk live. | Dimension | Standard RAG | Agentic RAG | |---|---|---| | Retrieval decision | Design time | Runtime | | LLM calls per query | One | One per hop, plus planning | | Latency | One round trip, tight variance | Accumulates per hop, long tail | | Cost per query | Flat | Variable, roughly linear in hops | | Determinism | High | Low, the same query can take a different plan | | Debugging | Inspect the chunks | Reconstruct the trace | | Eval strategy | Answer quality on fixed input | Answer quality and trajectory quality | | Multi-hop questions | Fails silently | Handles | | Structured data | Poor, embeddings flatten schema | Native, via SQL tools | | Failure mode | Confident answer from wrong chunks | Loops, fans out, burns budget | The determinism row is the one that costs most in practice. Standard RAG failures are boring and reproducible. Agentic RAG failures are interesting, which is a bad property for a production system: the same question asked twice can take different paths and produce different answers, and neither is wrong so much as differently sourced. Reproducibility was doing quiet work in your incident response process, and you give it up when you move the decision to runtime. Before adding another hop, check whether the context you already have is enough to answer, and put a hard ceiling on the case where it isn't. That check does more for a production agentic RAG loop than a better planner does. python MAX HOPS = 4 TOKEN BUDGET = 20 000 def agentic retrieve query: str, tools: dict - RetrievalResult: plan = planner.decompose query ReAct-style sub-questions ctx, trace, spent = , , 0 for hop in range MAX HOPS : gap = planner.next gap query, ctx what's still missing? if gap is None: break sufficiency gate: stop early tool = router.select gap, tools sql | search | timeseries chunks = tool.invoke gap spent += count tokens chunks if spent TOKEN BUDGET: trace.append "halt", "budget exceeded", hop break ctx.extend chunks trace.append tool.name, gap, len chunks return RetrievalResult context=dedupe ctx , trace=trace, emit this. always. exhausted=len trace = MAX HOPS, flag for review queue next gap returning None is the sufficiency gate. It lets a single-hop question cost one hop instead of four. Without it, your agentic system pays multi-hop prices on lookup questions, which are most of your traffic, and that shows up on the bill. MAX HOPS and TOKEN BUDGET are not optional. An unbounded retrieval loop against a live SQL tool is a denial-of-service vector pointed at your own warehouse, and a badly worded user question is enough to trigger it. trace is what makes the system debuggable. Emit it on every request, store it next to the answer, and make it queryable. When agentic RAG fails, the answer tells you nothing about why. The most common mistake is adopting agentic RAG for a single-hop corpus. If your knowledge base is a few thousand support articles and users ask article-shaped questions, a planner adds latency and cost to reach the same document standard RAG would have found in one pass. Route on question shape rather than on architectural fashion. Skipping the router is close behind. A planner handed eleven tools and no routing heuristic will explore all of them. Constrain the tool set per query class before the planner ever sees it. Then there's treating the two architectures as either/or. A classifier in front of both paths usually works better: the cheap single-hop route for lookup questions, the agentic route for questions that decompose. If most of your traffic is lookup, most of your traffic shouldn't be paying for a planner. Evaluation is the expensive mistake. In standard RAG, answer quality is a decent proxy for system health. In agentic RAG it isn't, because a correct answer reached through six wasteful hops is a cost and latency problem waiting for your traffic to grow. Track hops per query, tool selection precision, redundant retrieval rate, and how often the sufficiency gate fires. Finally, teams forget which argument actually closes the case, and it isn't multi-hop reasoning. Embeddings are a lossy representation of a relational schema. If your answers require joins, aggregations, or filters, no chunking strategy saves you, and you need a tool that speaks SQL. MAX HOPS and a token budget give you a worst case you can name.