Retrieval-Augmented Generation (RAG) is often described as a simple pipeline:
Query → Retrieve documents → Send context to an LLM → Generate answer
In production, however, retrieval is rarely that simple.
The retriever can return irrelevant documents. Important information may be buried in the middle of a document. A query may be too vague for semantic search. Retrieved chunks may lose their surrounding context. And sometimes the model does not need retrieval at all.
The quality of a RAG system therefore depends heavily on how information is retrieved, filtered, ranked, compressed, and presented to the model.
This guide covers nine techniques that address different parts of the RAG pipeline:
Retrieve candidates. Reranking finds the best.
A vector database may search through hundreds or thousands of documents and return the top 20 candidate chunks.
But the first result from vector search is not necessarily the best result.
For example, suppose the correct answer is ranked at position #19.
If the application only sends the top 3–5 chunks to the LLM, the correct information never reaches the model.
100 Pages
↓
Vector Search
↓
20 Candidate Chunks
│
┌───────────┴───────────┐
↓ ↓
Without Reranking With Reranking
↓ ↓
Top 3–5 Chunks Reranker
↓ ↓
│ Top 5 Relevant
│ Chunks
↓ ↓
↓ ↓
LLM LLM
Without reranking, the most relevant chunk might be ranked #19 and never reach the model.
With reranking, a reranker evaluates the retrieved candidates using both the query and the content of each chunk, promoting the most relevant results to the top.
The reranker can move a previously low-ranked but highly relevant chunk to the top.
A reranker essentially asks:
Which of these retrieved chunks actually answers the user's question best?
Unlike basic vector similarity, a reranker can inspect the relationship between the entire query and the retrieved document.
This helps it:
Common approaches include:
Cross-Encoder
Bi-Encoder + Rerank Model
LLM-based Reranker
Suppose a developer asks:
How can I make a Node.js API handle thousands of simultaneous connections?
A basic vector search might initially return chunks about HTTP status codes, API authentication, or general Node.js syntax.
A reranker can compare each candidate directly against the question and prioritize content discussing connection handling, asynchronous I/O, event loops, connection pooling, and horizontal scaling.
Retrieve more. Rerank intelligently. Let the LLM see the best context, not merely the first context.
Meaning + Keywords = Better Retrieval
Semantic vector search and keyword search solve different problems.
Vector search understands meaning.
Keyword search understands exact words.
Using only one can cause important documents to be missed.
Consider this query:
"Redis connection timeout"
A semantic search might return:
These documents may be semantically related, but the exact phrase Redis connection timeout
might not appear.
A keyword search such as BM25 can find:
But keyword search may fail when the document uses different terminology.
Hybrid search combines both approaches:
User Query
│
┌─────────┴─────────┐
↓ ↓
Vector Search Keyword Search
(Semantic) (BM25)
│ │
└─────────┬─────────┘
↓
Merge & Rank
↓
Final Results
The results from both searches are combined and ranked.
Popular approaches include:
Vector search is good at understanding intent:
"cache performance troubleshooting"
Keyword search is good at exact terms:
"Redis MISCONF"
A production search system often needs both.
Hybrid search is particularly useful for:
Don't choose between meaning and keywords. Use both.
Good chunks → Better Retrieval → Better Answers
Chunking is one of the most important decisions in a RAG system.
Documents are usually too large to embed and retrieve as a single unit, so they must be divided into smaller pieces.
But chunk size matters.
If a chunk is too large:
If a chunk is too small:
The goal is:
Keep chunks as small as possible for precision, but as large as necessary for completeness.
The document is divided into chunks of a fixed number of tokens.
For example:
Document
↓
400 tokens
↓
400 tokens
↓
400 tokens
↓
...
An overlap can be added between chunks.
Instead of splitting at arbitrary token boundaries, the system splits around sentence boundaries.
Sentence 1
Sentence 2
Sentence 3
Sentence 4
Sentence 5
Sentence 6
Sentence lengths can vary significantly.
Semantic chunking groups sentences or paragraphs based on their meaning.
Conceptually:
Topic A
├── Authentication configuration
├── Token validation
└── Session management
Topic B
├── Database indexing
├── Query planning
└── Connection pooling
Small chunks are used for retrieval, but the larger parent section is returned to the LLM.
Large Parent Document
│
┌──────────┼──────────┐
↓ ↓ ↓
Small Small Small
Chunk Chunk Chunk
The small chunks provide retrieval precision while the parent document provides context.
A moving window is used to create overlapping chunks.
Window 1
████████
Window 2
████████
Window 3
████████
This preserves more context across chunk boundaries.
More chunks mean more storage and potentially more retrieval cost.
The document's structure is used to determine chunk boundaries.
For example:
↓
Chunk 1
## Token Validation
↓
Chunk 2
- Access token
- Refresh token
↓
Chunk 3
Configuration Table
↓
Chunk 4
Code Block
↓
Chunk 5
This works particularly well for:
There is no universally best chunking strategy.
The right strategy depends on the data.
A production system may combine multiple approaches:
Structure-aware splitting
+
Semantic grouping
+
Parent document retrieval
You should also experiment with chunk sizes such as:
256 tokens
512 tokens
1024 tokens
and measure actual retrieval performance.
Good chunks bring the right context. The right context helps the LLM produce the right answer.
One question. Multiple perspectives. Better results.
A single query can fail because documents may describe the same concept using completely different language.
Even if query expansion improves the wording, searching in only one direction can still miss relevant documents.
Instead of searching once, ask the LLM to generate multiple versions of the query.
For example:
Original question:
How does OAuth token refresh work?
The system might generate:
What is OAuth token refresh?
How does a refresh token work?
What happens when an access token expires?
How does an application obtain a new access token?
What is the OAuth refresh-token flow?
Each query is searched independently.
Original Question
↓
Generate Queries
↓
┌────────┬────────┬────────┐
↓ ↓ ↓ ↓
Search Search Search Search
└────────┴────────┴────────┘
↓
Merge & Rerank
↓
Final Chunks
Different documents use different terminology.
One document might say:
OAuth token refresh
while another says:
renewing an expired access token
and another says:
obtaining a new bearer token using a refresh credential
Multiple queries give the retriever more opportunities to find relevant information.
These concepts are related but not identical.
| Feature | Query Expansion | Multi-Query |
|---|---|---|
| Main goal | Better wording | Different viewpoints |
| Queries | Similar variations | More diverse queries |
| Focus | Query improvement | Retrieval coverage |
| Recall | Good | Often higher |
| Typical use | General search | Production RAG |
Multi-query retrieval is particularly useful for:
Don't ask once. Ask in multiple smart ways.
More angles give the retriever more chances to find the right information.
Small chunks = better search. Parent documents = better understanding.
Small chunks are useful because they make retrieval precise.
But small chunks have a problem:
They can lose context.
Consider retrieving this chunk:
"... it automatically retries failed operations ..."
The chunk might be relevant, but by itself it doesn't tell us what "it" refers to.
The original section might say:
"The job processor automatically retries failed operations when a worker temporarily loses access to the message queue."
The parent document provides the missing context.
Document
↓
Chunk 1
Chunk 2
Chunk 3
Chunk 4
...
The vector database retrieves the most relevant chunks.
Top Chunks:
1
2
8
9
10
Each chunk stores a reference to its parent section or document.
Chunk 8
↓
Parent Document / Section
Instead of giving the LLM only the tiny chunk, provide the relevant parent section.
Small chunks → Search
Parent document → Context
This creates a useful separation:
Retrieve small. Read big.
Parent document retrieval is useful when:
Store a parent_id
with each chunk.
For example:
Chunk:
{
id: "chunk_123",
parent_id: "section_42",
embedding: [...]
}
After retrieval, use parent_id
to fetch the larger context.
Chunks help you find information. Parent documents help the model understand it.
Too much context can be as bad as too little.
Imagine a retriever returns 40 chunks, but your LLM can effectively process only 8 useful chunks.
Sending all 40 creates several problems:
This is related to the lost-in-the-middle problem: important information can become harder for the model to use when surrounded by large amounts of irrelevant context.
40 Retrieved Chunks
↓
Compress
↓
Keep Relevant Information
↓
8 Clean Chunks
↓
LLM
The compressor attempts to remove everything that does not contribute meaningfully to answering the question.
Common targets include:
Summarize each chunk into a smaller representation.
Large chunk
↓
1–2 sentence summary
Keep the most important terms and phrases.
Remove information that appears repeatedly across retrieved documents.
Keep only the sentences that directly contribute to answering the query.
Score individual sentences or chunks and keep only high-scoring content.
Suppose retrieval returns 40 chunks.
After compression:
40 chunks
↓
8 chunks
↓
~75% token reduction
↓
Better focused context
The exact improvement depends on the data and compression method, but the goal is to make the context smaller without losing useful information.
More context is not always better. Relevant context is better.
Think before you search.
HyDE stands for Hypothetical Document Embeddings.
It addresses a common retrieval problem:
The user's query may be too short or vague to produce a strong embedding.
For example:
"message queue retries"
The query contains only a few terms.
A better search signal could be a hypothetical answer generated by an LLM.
Instead of embedding the original question:
User Question
↓
Embedding
↓
Vector Search
HyDE introduces an intermediate generation step:
User Question
↓
Generate Hypothetical Answer
↓
Embed Hypothetical Answer
↓
Vector Search
↓
Retrieve Documents
↓
LLM
For example, the user asks:
How does a message queue retry failed jobs?
The LLM might generate a hypothetical answer such as:
A message processing system can retry a failed job when the worker encounters a temporary error. Retry policies commonly use a maximum attempt count and exponential backoff before moving permanently failed messages to a dead-letter queue.
The hypothetical answer contains more meaningful domain terms than the original question.
The system embeds that hypothetical answer and uses the embedding to search the knowledge base.
The generated answer may contain:
This can improve semantic matching.
The hypothetical answer is not used as the final answer.
It is primarily a search representation.
The actual answer still comes from retrieved documents.
Question
↓
Hypothetical Answer
↓
Embedding
↓
Retrieve Real Documents
↓
Generate Final Answer
Query expansion usually creates multiple alternative queries.
HyDE generates a hypothetical document or answer and embeds that representation.
Query Expansion
→ Multiple queries
HyDE
→ One hypothetical answer
→ One embedding
HyDE can help with:
HyDE turns a weak question into a stronger search signal.
Why search every time? Let the model decide first.
Traditional RAG often retrieves documents for every query.
But not every question needs external retrieval.
For example:
What is the square root of 144?
Retrieving documents from a vector database would be unnecessary.
Always retrieving causes:
Self-RAG introduces a decision step.
User Question
↓
Should I retrieve?
↓
┌────────┴────────┐
NO YES
↓ ↓
Answer Retrieve
Directly ↓
Generate
The model first considers:
The model answers using its internal knowledge.
Example:
What is 15 × 8?
No retrieval is required.
The system retrieves relevant documents.
Example:
What changed in our company's API documentation this week?
Retrieval is useful because the information is recent and internal.
Typical cases include:
Typical cases include:
Self-RAG can:
Traditional RAG retrieves every time. Self-RAG decides whether retrieval is needed before acting.
Not every retrieved chunk is useful. CRAG checks the quality before trusting it.
A retriever is not perfect.
It can return:
If the LLM blindly trusts those chunks, it can produce a confident but incorrect answer.
CRAG introduces a quality-control step.
User Question
↓
Retrieve Documents
↓
Evaluate Retrieved Documents
↓
┌───────┴───────┐
GOOD BAD
↓ ↓
Use Docs Correct Retrieval
↓
Refine / Re-query
↓
Retrieve Again
↓
Final Context
↓
LLM
The retrieved documents are evaluated before they are trusted.
Potential criteria include:
If the documents are good enough, they can be passed to the LLM.
If they are poor, the system can attempt corrective actions.
Examples include:
Suppose the user asks:
Which database is a good choice for high-volume event analytics?
The retriever returns:
1. Introduction to relational databases
2. Key-value cache configuration
3. Columnar database architecture for analytics
4. Basic SQL CRUD operations
The evaluator can determine that only some of these documents directly address the question.
The system can then remove weak results and perform additional retrieval if necessary.
The goal is:
Retrieve
↓
Check
↓
Correct
↓
Answer
CRAG can:
CRAG is particularly useful for:
Normal RAG:
Retrieve → Answer
CRAG:
Retrieve → Evaluate → Correct → Answer
The fundamental difference is that CRAG does not blindly trust the retriever.
CRAG verifies the retrieved context before allowing the model to rely on it.
These techniques do not need to be used independently.
A production RAG system can combine several of them.
For example:
User Query
│
▼
Self-RAG Decision
/ \
NO YES
│ │
▼ ▼
Direct Answer Multi-Query
│
▼
Hybrid Search
Vector + BM25
│
▼
Retrieval
│
▼
Reranking
│
▼
CRAG Evaluation
/ \
GOOD BAD
│ │
│ Re-query/Correct
│ │
└───────┬─────────┘
▼
Parent Document Retrieval
│
▼
Context Compression
│
▼
LLM
│
▼
Final Answer
Not every application needs every component.
The correct architecture depends on:
Each technique solves a different failure mode.
| Problem | Technique |
|---|---|
| Correct chunk is retrieved but ranked too low | Reranking |
| Exact keywords and semantic meaning both matter | Hybrid Search |
| Documents are difficult to split correctly | Better Chunking |
| One query misses relevant terminology | Multi-Query Retrieval |
| Retrieved chunk lacks surrounding context | Parent Document Retrieval |
| Too many retrieved chunks overwhelm the model | Context Compression |
| Query is vague or lacks useful search terms | HyDE |
| Retrieval isn't necessary for every question | Self-RAG |
| Retriever returns poor or misleading documents | CRAG |
A practical system might start with something relatively simple:
Documents
↓
Structure-Aware Chunking
↓
Embeddings + Keyword Index
↓
Hybrid Search
↓
Reranking
↓
Parent Context
↓
Context Compression
↓
LLM
Then add more advanced techniques only where measurements show they are needed.
For example:
Self-RAG
can reduce unnecessary retrieval.
Multi-Query Retrieval
can improve recall for difficult questions.
HyDE
can help with vague queries.
CRAG
can add a validation and correction loop.
The biggest mistake when building RAG systems is treating retrieval as a single operation:
Query → Vector DB → LLM
Real-world retrieval is closer to a pipeline of decisions:
Should I retrieve?
↓
What should I search for?
↓
Where should I search?
↓
How should I split the documents?
↓
Which results are actually relevant?
↓
Which results should be ranked highest?
↓
How much context should I provide?
↓
Is the retrieved context trustworthy?
↓
Can the LLM answer from this context?
The quality of the final answer is often determined before the LLM generates a single token.
Better retrieval → Better context → Better answers.
And the goal isn't to build the most complicated RAG pipeline.
The goal is to build the simplest retrieval architecture that reliably provides the right context for your workload.