If you've built an LLM application, you've probably discovered a problem pretty quickly: The model doesn't know your application's data.
Your product documentation might change tomorrow. Your company's API may be private. Your database may contain information that never appeared in the model's training data.
You could fine-tune a model, but that isn't always the right answer.
This is where Retrieval-Augmented Generation (RAG) comes in.
The Short Version
A basic RAG pipeline looks like this:
User Query
↓
Retriever
↓
Relevant Chunks
↓
Context
↓
LLM
↓
Answer
The retriever finds useful information.
The LLM turns that information into an answer.
Simple, right?
The interesting engineering problems start after that.
A More Realistic Pipeline
A production RAG system might look closer to:
Documents
↓
Parsing
↓
Chunking
↓
Embeddings
↓
Index
↓
Query Processing
↓
Retrieval
↓
Reranking
↓
Context Construction
↓
LLM
↓
Answer + Sources
There are really two workflows here.
The first prepares your knowledge.
The second runs every time somebody asks a question.
Your data might come from:
Markdown
PDFs
Websites
Databases
Internal documentation
APIs
Cloud storage
The raw files usually need to be cleaned and structured before retrieval.
If your source material is messy, retrieval starts with a disadvantage. Long documents are usually divided into smaller passages.
Consider API documentation containing sections for authentication, pagination, rate limits, and webhooks.
If somebody asks: "What's the API rate limit?"
Returning the entire documentation page is unnecessary.
A focused chunk is more useful.
But chunking has a trade-off.
Too large → unnecessary context.
Too small → missing context.
There isn't one perfect chunk size for every application. Test it against your actual documents and queries.
An embedding model converts chunks into numerical representations.
The point isn't the numbers themselves.
The useful property is that semantically related text can be represented similarly.
For example: Query:
"What's the maximum number of API requests?"
Document:
"Requests are limited to 100 calls per minute."
The wording isn't identical, but the meaning is closely related.
At query time, the user's question is used to find relevant chunks.
There are several ways to do this.
Keyword retrieval
Good when exact words matter.
Vector retrieval
Good when semantic meaning matters.
Hybrid retrieval
Combines both.
For developer-facing systems, hybrid retrieval can be particularly useful because technical searches often contain exact identifiers alongside natural-language questions. The first retrieval step may return more results than the LLM actually needs.
A reranker can score those candidates again.
The result might go from:
20 retrieved chunks
↓
5 strongest chunks
↓
LLM
This helps keep irrelevant material out of the final context.
At this point, the application combines:
System instructions
Retrieved context
User question
The model can now generate an answer using the retrieved information.
This is the "generation" part of RAG.
The Important Part: Retrieval Quality
Here's the part that's easy to underestimate.
If your retriever returns the wrong documents, the LLM doesn't magically know that. It may generate a confident answer from bad context.
That's why a RAG evaluation should ask two separate questions:
Did we retrieve the right information?
and
Did the model use that information correctly?
Treating those as one problem makes debugging much harder.
Why Hybrid Search Often Makes Sense
Imagine a developer searches:
"ERR_AUTH_401 API v3"
Semantic search may understand the general concept, but exact identifiers can be important.
Keyword search handles exact terms well.
Semantic search handles meaning well.
Combining them can provide a better retrieval strategy for mixed queries.
RAG vs Fine-Tuning
A useful mental model:
RAG = give the model knowledge at runtime.
Fine-tuning = change how the model behaves.
If your documentation changes every week, RAG is often much more practical than repeatedly retraining a model to memorize the latest version. Agentic RAG
Traditional RAG normally performs retrieval according to a predefined pipeline.
Agentic RAG gives an AI agent more control.
For example, a user could ask: "Compare our 2025 and 2026 security policies and explain the important changes."
An agent might:
Search the 2025 policy.
Search the 2026 policy.
Identify relevant sections.
Compare the retrieved content.
Generate the final response.
This is more flexible, but also more complex to build, monitor, and evaluate.
Common Failure Modes
Some RAG problems show up repeatedly:
Wrong chunks retrieved
Poor chunk boundaries
Outdated documents
Missing metadata
Weak exact-term matching
Too much context
Too little context
Incorrect permissions
Unsupported model-generated claims
So when a RAG application fails, don't immediately replace the LLM.
First inspect the retrieval pipeline.
Production Considerations
A production RAG system should think about:
Access control
Document freshness
Metadata filtering
Latency
Token costs
Retrieval evaluation
Answer evaluation
Monitoring
Prompt injection risks
The vector database is only one part of the architecture.
Final Takeaway RAG is often described as:
Retrieve → Augment → Generate.
That's accurate, but it hides most of the engineering work.
Reliable RAG depends on the quality of the entire retrieval pipeline.
I put together a longer practical guide covering the architecture, embeddings, chunking, retrieval, RAG vs fine-tuning, traditional vs agentic RAG, challenges, and optimization:
Full [(RAG guide — AI Tools Vault)]