If you've shipped a "chat with your docs" prototype in a weekend, congratulations — you've built Naive RAG. If you've then watched it hallucinate on multi-hop questions, choke on tables, and confidently cite the wrong PDF in production... also congratulations. You've discovered why "RAG" is not a single architecture. It's a design space.
This article is the map I wish I had before I rebuilt the same pipeline four times.
Let's get into it.
The "hello world" RAG loop looks like this:
User Query → Embed → Vector Search (top-k) → Stuff into Prompt → LLM → Answer
It works great in a demo with 20 PDFs. Then someone asks a real question and things fall apart:
| Failure Mode | What Actually Happens |
|---|---|
| Chunking artifacts | |
| A table gets split mid-row; the answer is technically "retrieved" but semantically garbage | |
| Semantic drift | |
| The query embedding is close to lexically similar chunks, not answer-relevant ones | |
| Multi-hop failure | |
| "Compare Q3 revenue to Q2 and explain the delta" needs two retrievals and a reasoning step — vanilla RAG does one retrieval, once | |
| No relevance filtering | |
| Top-k always returns k chunks, even if none of them are actually relevant | |
| No verification | |
| The LLM generates fluently even when the retrieved context doesn't support the claim — silent hallucination | |
| Static k | |
| A simple FAQ question and a complex synthesis question get the same fixed number of retrieved chunks |
None of this means "RAG is broken." It means naive RAG is the MVP, not the destination. Everything below is what production teams reach for next.
Before the architecture zoo, it helps to zoom out. Most RAG systems fall into one of three evolutionary stages.
┌─────────┐ ┌──────────┐ ┌────────────┐ ┌──────────┐
│ Query │ --> │ Embed │ --> │ Vector DB │ --> │ LLM │ --> Answer
└─────────┘ └──────────┘ │ (top-k) │ └──────────┘
└────────────┘
Single embed → single retrieve → single generate. No feedback loops, no query understanding, no correction. This is your baseline, not your product.
Advanced RAG keeps the linear shape but adds two optimization stages:
Pre-retrieval: improve the query before it hits the index.
Post-retrieval: improve the context before it hits the LLM.
Query --> [Rewrite / HyDE] --> Retrieve --> [Re-rank / Filter] --> LLM --> Answer
This is the single highest-ROI upgrade most teams should make before reaching for anything fancier.
Modular RAG stops treating the pipeline as a fixed chain and starts treating it as a graph of interchangeable modules: retrieval, routing, memory, fusion, task adapters — wired together however the problem demands, including loops.
┌─────────────┐
│ Router │
└──────┬───────┘
┌──────────────┼──────────────┐
v v v
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Vector DB │ │ Graph DB │ │ Web/API │
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
└──────────────┼──────────────┘
v
┌───────────────┐
│ Fusion/Rerank │
└───────┬───────┘
v
┌───────────────┐
│ Memory Store │◄──┐ (feedback loop)
└───────┬───────┘ │
v │
┌───────────────┐ │
│ LLM │───┘
└───────────────┘
Every pattern in the next section is really a specific configuration of Modular RAG's building blocks.
The classic. Pure dense vector similarity search — embeddings in, cosine/dot-product similarity out.
[Query] --embed--> [Query Vector]
│
v
┌──────────────────┐
│ Vector Index │
│ (HNSW / IVF) │
└─────────┬─────────┘
v
top-k chunks
v
[LLM] --> Answer
Use when: semantically rich, unstructured text corpora (docs, wikis, support articles) where exact keyword matches don't matter much.
Pros: simple, fast to stand up, well-supported tooling (pgvector, Pinecone, Qdrant, Weaviate).
Cons: blind to exact-match needs (SKUs, error codes, acronyms); no relevance guarantee; single-shot.
Dense search alone fails on exact-match terms (ERR_402
, product codes, proper nouns embeddings weren't trained to distinguish). Hybrid RAG fuses dense vector search with sparse keyword search (BM25), typically combined via Reciprocal Rank Fusion (RRF).
┌──────────────┐
┌───────────►│ Dense Search │───────────┐
│ │ (embeddings) │ │
[Query]─┤ └──────────────┘ v
│ ┌───────────┐
│ ┌──────────────┐ │ RRF │──> [LLM] --> Answer
└───────────►│ Sparse Search │─────►│ Fusion │
│ (BM25) │ └───────────┘
└──────────────┘
Use when: mixed corpora with both semantic and exact-match retrieval needs — technical docs, legal text, e-commerce catalogs.
Pros: best-of-both-worlds recall; handles rare/OOV terms embeddings miss.
Cons: two indexes to maintain; fusion tuning (RRF constant k
, weighting) adds a knob to babysit.
Vector search treats every chunk as an island. GraphRAG builds a knowledge graph (entities + relationships, often in Neo4j) alongside — or instead of — the vector index, so retrieval can traverse relationships, not just similarity.
[Query] --> [Entity Extraction] --> [Graph Traversal]
│
┌─────────────────┼─────────────────┐
v v v
(Entity A)──relates──(Entity B)──relates──(Entity C)
│ │
└──────────── subgraph ────────────────┘
v
[Context Assembly]
v
[LLM] --> Answer
Use when: questions require multi-hop reasoning over relationships — "Which suppliers does Company X depend on that are also linked to Region Y?" Vector search can't answer that; graph traversal can.
Pros: captures relational/structural knowledge; strong for compliance, org-chart, and dependency-mapping queries.
Cons: expensive to build and maintain (entity extraction + graph construction pipeline); overkill for simple lookup tasks.
CRAG adds a quality gate after retrieval: a lightweight evaluator grades each retrieved chunk (correct / ambiguous / incorrect). If confidence is low, it triggers an external fallback — like a web search via Tavily or DuckDuckGo — instead of letting the LLM generate from garbage context.
[Query] --> Retrieve --> [Relevance Grader]
│
┌─────────────────┼─────────────────┐
v v v
CORRECT AMBIGUOUS INCORRECT
│ (refine + web) │
│ │ (discard, web search)
└────────┬────────┴────────┬─────────┘
v v
[Knowledge Refinement / External Search]
v
[LLM] --> Answer
Use when: your corpus has coverage gaps and you need graceful degradation instead of confident hallucination.
Pros: dramatically reduces hallucination from irrelevant retrieval; self-healing.
Cons: extra latency (grading step + possible external call); grader quality becomes a new dependency to tune.
Self-RAG pushes reflection to the generation side. The model is trained/prompted to emit reflection tokens that grade its own output: is retrieval even needed? Is the generated answer supported by the retrieved passages? Is it useful?
[Query] --> [Retrieve?] --yes--> Retrieve --> Generate --> [Self-Critique]
│no │
v ┌──────────────────┼──────────────────┐
Generate directly v v v
"Supported" "Partially" "Not Supported"
│ │ │
v v v
Return Regenerate Re-retrieve
w/ more context
Use when: you need built-in hallucination detection without bolting on a separate verifier service.
Pros: tighter faithfulness guarantees; can skip retrieval entirely when unnecessary (saves latency/cost).
Cons: best results need a fine-tuned or carefully prompted critique step; harder to implement well with an off-the-shelf general model.
Not every query deserves the same amount of machinery. Adaptive RAG routes queries by complexity tier — a classifier decides whether a query needs no retrieval, single-step retrieval, or full multi-step agentic reasoning.
┌───────────────────┐
[Query] --> │ Complexity Router │
└──────────┬─────────┘
┌─────────────────────┼─────────────────────┐
v v v
SIMPLE (no RAG) MODERATE (Standard RAG) COMPLEX (Agentic)
│ │ │
[LLM only] [Retrieve → Gen] [Multi-step reasoning
+ tools + iteration]
└─────────────────────┴─────────────────────┘
v
Answer
Use when: you're serving a wide mix of query types (chit-chat + FAQ + deep analytical questions) and can't afford full agentic overhead on every single request.
Pros: big latency/cost savings; right-sizes compute per query.
Cons: router accuracy becomes a critical bottleneck — a misclassified complex query gets under-served.
Retrieval becomes one tool among many, orchestrated by an agent (or a team of specialized agents) that plans, calls tools, observes results, and iterates until it's satisfied — think ReAct-style loops or multi-agent handoffs.
┌─────────────────────────────────────────┐
│ Orchestrator Agent │
└──────────────────┬──────────────────────┘
v
┌──────────────┼──────────────┐
v v v
[Retrieval Agent] [SQL Agent] [Web Search Agent]
│ │ │
└──────────────┼──────────────┘
v
[Synthesis / Reflection]
│
(loop if answer incomplete)
v
Final Answer
Use when: questions require multi-step reasoning across heterogeneous sources — databases, APIs, documents, live web — with planning in between.
Pros: most flexible and capable pattern; handles genuinely hard, multi-source tasks.
Cons: highest latency and cost; harder to debug (non-deterministic loops); needs strong guardrails against runaway tool-calling.
Real documents aren't pure text — they have diagrams, tables, charts, and scanned images. Multi-Modal RAG embeds text and visual content into a shared (or jointly-indexed) space so retrieval can pull the right diagram, not just the paragraph near it.
┌─────────────┐ ┌──────────────┐
[Document]-->│ Text Chunks │ │ Images/Tables/ │
│ │ │ Diagrams │
└──────┬──────┘ └───────┬──────┘
v v
[Text Embedder] [Vision Embedder]
│ │
└────────────┬───────────┘
v
┌────────────────────┐
│ Joint Vector Index │
└──────────┬─────────┘
v
[Query] --> Retrieve (text + visual)
v
[Multi-Modal LLM] --> Answer
Use when: your corpus is PDFs with architecture diagrams, financial tables, engineering schematics, or scanned forms.
Pros: unlocks information that pure-text pipelines silently drop; matches how humans actually read technical documents.
Cons: immature tooling relative to text-only RAG; multi-modal embedding models are heavier and pricier to run at scale.
Here's a compact, runnable pattern combining Hybrid RAG (dense + BM25 via RRF) with a re-ranking post-retrieval step — arguably the highest-leverage upgrade you can make to a naive pipeline.
from langchain_community.retrievers import BM25Retriever
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain.retrievers import EnsembleRetriever
from langchain.retrievers.document_compressors import CohereRerank
from langchain.retrievers import ContextualCompressionRetriever
docs = [...] # your pre-chunked Document objects
bm25_retriever = BM25Retriever.from_documents(docs)
bm25_retriever.k = 10
vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())
dense_retriever = vectorstore.as_retriever(search_kwargs={"k": 10})
hybrid_retriever = EnsembleRetriever(
retrievers=[bm25_retriever, dense_retriever],
weights=[0.4, 0.6], # tune based on your corpus
)
reranker = CohereRerank(model="rerank-english-v3.0", top_n=4)
compression_retriever = ContextualCompressionRetriever(
base_compressor=reranker,
base_retriever=hybrid_retriever,
)
query = "What caused the Q3 latency regression in the payments service?"
final_docs = compression_retriever.invoke(query)
for doc in final_docs:
print(doc.page_content[:200], "\n---")
Why this matters: the ensemble step catches what pure embeddings miss (exact error codes, service names), and the re-ranker throws away the noise that top-k alone would have shipped straight into your prompt. This one change routinely moves retrieval precision more than swapping embedding models does.
A minimal CRAG-style relevance gate, for comparison:
def grade_relevance(query: str, doc: str, llm) -> str:
prompt = f"""Query: {query}
Retrieved passage: {doc}
Is this passage relevant and sufficient to answer the query?
Respond with exactly one word: CORRECT, AMBIGUOUS, or INCORRECT."""
return llm.invoke(prompt).content.strip().upper()
def corrective_retrieve(query, retriever, web_search_fn, llm):
docs = retriever.invoke(query)
grades = [grade_relevance(query, d.page_content, llm) for d in docs]
if all(g == "INCORRECT" for g in grades):
return web_search_fn(query) # fallback to external search
return [d for d, g in zip(docs, grades) if g != "INCORRECT"]
| Architecture | Complexity | Latency | Cost | Best Use Cases |
|---|---|---|---|---|
| Standard (Dense) RAG | ||||
| Low | Low | Low | Homogeneous unstructured text corpora, FAQs, docs | |
| Hybrid RAG | ||||
| Medium | Low–Med | Low–Med | Mixed content with exact-match terms (codes, IDs, jargon) | |
| GraphRAG | ||||
| High | Medium | Med–High | Relationship-heavy domains: compliance, org data, dependency graphs | |
| Corrective RAG (CRAG) | ||||
| Medium | Medium | Medium | Incomplete/noisy corpora; hallucination-sensitive applications | |
| Self-RAG | ||||
| High | Medium | Medium | Faithfulness-critical answers (medical, legal, financial) | |
| Adaptive RAG | ||||
| Medium–High | Variable (optimized) | Variable (optimized) | Mixed-complexity query traffic at scale | |
| Agentic / Multi-Agent RAG | ||||
| Very High | High | High | Multi-source, multi-step research and analysis tasks | |
| Multi-Modal RAG | ||||
| High | Medium–High | High | Technical/engineering docs, financial reports, scanned forms |
The real skill here isn't memorizing eight architectures. It's diagnosing which failure mode you actually have and reaching for the smallest pattern that fixes it.