Short answer
Agentic RAG is retrieval-augmented generation where the agent decides when to retrieve, what to query, and whether the results are good enough — sometimes searching again — instead of running a fixed retrieve-then-generate pipeline. Retrieval stops being a pre-step bolted onto the prompt and becomes a tool the orchestrator can call, evaluate, and re-call until the context is sufficient to answer.
This page goes deep on the retrieval layer specifically. For how retrieval sits among the other five components of an agent system, see agentic AI architecture; this page is the part it points to.
Classic RAG is a straight line: take the user's question, embed it, search a vector index, stuff the top chunks into the prompt, generate. It runs once, the same way, every time. That is enough for a lot of question-answering, and you should not reach past it without a reason.
Agentic RAG turns that line into a loop with a decision at the top of it. The agent treats retrieval as one of its tools. It chooses whether to search at all, writes its own query (often rewriting the user's), inspects what came back, and decides whether to answer, search again with a better query, or try a different source. Retrieval becomes controlled rather than scripted.
The difference is not the vector store or the embeddings — both architectures use those. The difference is who is in charge of retrieval: a fixed pipeline, or the model.
| Dimension | Classic RAG | Agentic RAG |
|---|---|---|
| When to retrieve | ||
| Always, once, before generation | The agent decides — maybe zero, maybe several times | |
| What to query | ||
| The user's question, embedded as-is | A query the agent writes or rewrites for the index | |
| Result sufficiency | ||
| Assumed — top-k goes straight to the prompt | Judged — the agent grades relevance and can retry | |
| Sources | ||
| Usually one index | Can route across indexes or tools (search, SQL, API) | |
| Control flow | ||
| Linear pipeline, runs identically each time | A loop with a stop condition | |
| Cost / latency | ||
| Low and predictable | Higher and variable — extra model turns per search | |
| Best for | ||
| Direct lookups over one corpus | Multi-step questions, mixed sources, recall that needs checking |
Read the table as a cost curve, not a verdict. Agentic RAG buys recall and robustness with extra model turns. If classic RAG answers your questions, the loop is overhead you do not need.
You cannot reason about the agentic version without the baseline it is built on. A RAG pipeline has two phases.
Indexing (offline): split source documents into chunks, embed each chunk into a vector with an embedding model, and store those vectors — plus the original text and metadata — in a vector index. This runs ahead of time and is refreshed as the source changes.
Retrieval and generation (per request): embed the incoming question, find the nearest chunks by vector similarity (often re-ranked, sometimes combined with keyword search as hybrid retrieval), and pass those chunks to the model as grounding context alongside the question. The model answers from the supplied text rather than from its parametric memory.
Everything load-bearing about RAG lives in this baseline: chunking strategy, embedding quality, the index, re-ranking, and — the part teams skip — carrying each chunk's source through to the answer so it can be cited. Agentic RAG does not replace any of this. It wraps a decision-maker around the retrieval step.
Three capabilities move a pipeline from classic to agentic. None of them is exotic; together they change who is in control.
Frameworks expose these as first-class building blocks. LangGraph documents an "agentic RAG" pattern where the retriever is a tool the agent calls, grades, and re-queries; LlamaIndex offers router and sub-question query engines that let the model choose an index or decompose a question. Treat these as confirmation that the shape is standard, not as a requirement — the capabilities matter more than any one library.
Stop treating retrieval as a phase and treat it as a tool in the agent's action layer. The orchestrator runs its normal bounded loop; one of the tools it can invoke is search(query)
.
A single iteration looks like this: the model receives the goal, decides a search is warranted, emits a search
tool call with a query it wrote, the orchestrator runs the retrieval and returns the chunks, and the model reads them and either answers — citing sources — or calls search
again with a refined query. The same turn-and-tool-call budget that bounds any agent loop bounds the number of retrievals, which is what stops a hard question from triggering an unbounded search spiral.
This is why agentic RAG belongs to the same architecture as everything else an agent does, not to a separate "RAG system." Retrieval is in the tools/action layer; the decision to use it is in the orchestrator; the bound on it is in the operational plane. For how those layers fit together — and where retrieval sits among them — see agentic AI architecture, and for the loop that bounds the tool calls, agentic AI design patterns. When the search tool lives behind a standard interface, the Model Context Protocol (MCP) is a common way to expose it so the same retrieval tool is reusable across agents.
Agentic RAG still rests on a vector index, and the index choice is mostly orthogonal to whether retrieval is agentic — both architectures need it. Keep this decision boring and reversible. The realistic options in 2026:
Pick on data gravity, hybrid-search needs, and operational appetite — not on a benchmark leaderboard. The vector layer is the easiest piece to swap later; the retrieval logic and provenance handling are the parts worth getting right.
Provenance is non-negotiable. Carry each chunk's source identifier from the index through retrieval into the answer, so every claim can be traced back. An answer you cannot attribute is one you cannot ship — and the same discipline is what makes evaluation possible.
Agentic RAG has more to evaluate than classic RAG because both the retrieval and the agent's decisions can fail. Split it:
Because the same question can take a different retrieval path on different runs, you pin quality with an eval harness rather than manual spot-checks — the same discipline used for any non-deterministic agent. The full treatment is in how to evaluate an LLM agent; the short rule is that you cannot tune what you do not measure, and agentic RAG gives you two layers to measure.
The clearest instance of "retrieval as a tool the agent decides to call" on aiarch.dev is not the vector search above the fold — it is the coach reaching for live documentation mid-conversation. src/lib/docsMcp.ts
gives the in-product coach two tools that query the public, no-auth aws-knowledge
and cloudflare-docs
MCP servers over Streamable HTTP. The model decides whether a question needs a live doc lookup at all — most turns don't — and only then emits the tool call; nothing runs on a fixed schedule.
Building that client surfaced the same "results aren't uniform" problem this page's evaluating-agentic-RAG section warns about, at the transport level rather than the content level. Cloudflare's endpoint always replies as Server-Sent Events, even for a single one-shot call, so we parse the SSE data:
line before touching the payload; its tool text itself is pseudo-XML result blocks, not JSON. AWS replies plain JSON, but double-encodes — the MCP text block is itself a JSON string that has to be parsed a second time. Neither server needs an initialize
handshake or session id for a single tools/call
— undocumented under revision 2025-11-25
, so it had to be confirmed by calling both endpoints directly. Revision 2026-07-28
, published on that date, removes the handshake and sessions from the protocol entirely and instead requires three headers and two _meta
fields on every request; both servers were measured accepting either shape on 28 Jul 2026, so docsMcp.ts
sends the newer one and falls back once, on a 400
, to the older rather than pinning an era. AWS also enforces an undocumented roughly one-request-per-15-seconds per-IP limit, so it makes one soft-failing attempt per lookup rather than hammering it — that 400
fallback is the only second request it sends. The whole client never throws: a network or parse failure returns an empty result list with a reason, so the coach can tell the learner the live lookup is unavailable instead of erroring the turn or guessing an answer — the same reliability posture this page argues an agentic retrieval loop needs. See the MCP guide for the protocol itself, and the bounded agentic loop pattern for how the tool-call budget bounds this alongside every other tool the coach can reach for.
Agentic RAG is retrieval-augmented generation where an agent controls retrieval rather than a fixed pipeline. The agent decides when to search, writes or rewrites the query, judges whether the results are sufficient, and can search again before answering. Retrieval becomes a tool the agent calls inside its loop, instead of a single retrieve-then-generate step run the same way every time.
Classic RAG runs one fixed sequence: embed the question, search the index, put the top chunks in the prompt, generate. Agentic RAG wraps a decision around that step — the model chooses whether to retrieve, what to query, and whether the results are good enough, sometimes iterating. Same vector index and embeddings; the difference is that the agent, not the pipeline, is in charge of retrieval.
A RAG pipeline has two phases. Offline indexing splits documents into chunks, embeds each into a vector, and stores them in a vector index. Per request, it embeds the question, retrieves the nearest chunks by similarity (often re-ranked or combined with keyword search), and passes them to the model as grounding context so it answers from supplied text rather than parametric memory. Agentic RAG builds a decision layer on top of this baseline.
Use classic RAG when questions are direct lookups over a single corpus — it is cheaper, faster, and predictable. Reach for agentic RAG when questions are multi-step, span multiple sources, or when retrieval quality is shaky enough that the agent needs to grade results and retry. The agentic loop buys recall and robustness with extra model turns; if the baseline already answers your questions, that cost is overhead you do not need. Choose on data gravity and operations, not benchmarks. pgvector is the strong default if you already run Postgres. Pinecone is fully managed and serverless. Weaviate and Qdrant offer first-class hybrid search. Cloudflare Vectorize is edge-native and pairs with Cloudflare AI Search. Managed RAG services like AWS Bedrock Knowledge Bases handle indexing and retrieval for you. The vector layer is the easiest piece to swap later, so keep the choice reversible.
Evaluate three things. Retrieval quality with recall and precision over labelled question-to-chunk pairs. Groundedness — whether the answer is actually supported by the retrieved chunks, which citations make checkable. And the agent's decisions — whether it retrieved, queried, and stopped well — using a held-out set of scenarios with an LLM-as-judge. Because runs are non-deterministic, you pin quality with an eval harness rather than manual spot-checks.
docs/DESIGN.md
, src/lib/rag.ts
).tools/call
method against live Vendor products and API shapes change; treat the mapping as a design template, not a guaranteed signature. Corrections: hello@aiarch.dev.
Originally published at aiarch.dev/agentic-rag, where it is kept up to date.
Want the skeleton instead of the essay? aiarch-templates has the src/lib/ seams, a threshold-gated eval stub and a cost-model skeleton. It is deliberately empty — it fixes the shape and you write the implementation. Apache-2.0.