Building a reliable RAG system is not about connecting an LLM to a vector database. It requires a complete architecture covering data ingestion, retrieval, ranking, evaluation, and production operations.
This article is the first part of a five-part series where we will explore how modern RAG systems are designed, why naive implementations fail, and what it takes to build AI search systems that work reliably in production.
Why Most RAG Systems Fail in Production: The Hidden Architecture Problems Behind AI Search (you are here)
Building a Production RAG Pipeline: Document Processing, Chunking, and Metadata Design
Beyond Vector Search: Building Better RAG Retrieval with Hybrid Search and Reranking
Scaling RAG Systems: Production Architecture and Performance Optimization
Evaluating Production RAG Systems: Metrics, Monitoring, and Common Failure Patterns
Building a Retrieval-Augmented Generation system isn't hard.
Building one that people can trust is.
You've probably seen hundreds of tutorials:
Documents β Split into Chunks β Generate Embeddings β Store in Vector DB β Similarity Search β Send Context to LLM β Answer
It looks beautifully simple.
You can build a working chatbot in 30 minutes with LangChain, LlamaIndex, or Haystack.
Upload a PDF.
Generate embeddings.
Store them.
Ask questions.
Done.
For a demo, that's enough.
For production, that's the beginning of your problems.
Imagine this.
You built an internal AI assistant for your company's documentation.
Demo:
User: "How do I reset my password?"
Assistant: "Go to Settings β Security β Reset Password. You'll receive an email with a link."
Perfect. Everyone applauds. Project approved.
Two weeks later, support starts using it.
User: "Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?"
Assistant: "Yes."
Actual answer from the policy document:
"Active invoices must be closed before upgrading from Professional to Enterprise."
Nobody notices immediately.
Support gives wrong advice.
Customer tries to upgrade.
Billing rejects.
Support spends hours resolving.
Management asks:
"I thought AI was supposed to reduce support workload."
Nothing crashed.
Embeddings were correct.
Vector DB was running.
Similarity search returned relevant chunks.
LLM generated fluent text.
Every component worked.
Yet the system failed.
That is what makes production RAG difficult.
Failures rarely come from broken software.
They come from broken architecture.
Most tutorials focus on this:
Query
β
Vector Search
β
LLM
β
Answer
Real production systems look more like this:
Ingestion (Offline)
Upload
β
Object Storage (S3 / GCS)
β
OCR / Parser (PDF, HTML, Markdown)
β
Content Cleaning (headers, footers, page numbers)
β
Chunking (semantic + structure-aware)
β
Metadata Extraction (doc type, language, version, section)
β
Embedding Workers
β
βββ Vector DB
βββ BM25 / Keyword Index
----------------------------
Retrieval (Online)
User Query
β
Query Rewriting / Expansion
β
Metadata Filters (language, department, version)
β
Hybrid Search (Vector + BM25)
β
Reranking (Cross-Encoder)
β
Context Compression
β
Prompt Builder
β
LLM
β
Answer + Citations
The language model is often the smallest part of the architecture.
But it receives almost all attention.
The biggest misconception:
"Retrieval is a solved problem. Just use a vector DB."
It isn't.
Think about the Library of Congress.
Naive approach:
Throw all books into a giant pile.
Ask someone to find the right one by reading random pages.
Eventually, they might find it.
This is basically what many beginner RAG systems do:
Split everything into fixed-size chunks.
Embed.
Retrieve top-k by cosine similarity.
Feed to LLM.
Professional librarian:
Before searching, they narrow it down:
Language?
Publication year?
Author?
Category?
Edition?
Print or digital?
Academic or popular?
Only after reducing millions of possibilities to a few hundred, they start semantic search.
Professional retrieval works the same way.
Goal is not to search better.
Goal is to search less.
Another common mistake:
"Vector DB replaces traditional databases."
No.
A vector DB is not your source of truth.
Think about Google:
Same with RAG:
Your documents should live somewhere else:
Vector DB only stores an efficient representation for fast retrieval.
Delete your vectors β you can rebuild them.
Delete your original documents β you lost everything.
Many production systems fail because developers treat Vector DB as permanent storage.
Another mistake:
"Retrieval begins when the user submits a question."
No.
Retrieval begins the moment a document enters your system.
Imagine two companies.
Company A:
Employee uploads PDF.
System extracts plain text.
Splits every 500 tokens.
Generates embeddings.
Stores them.
Done.
Company B:
Same PDF arrives.
Before embeddings, system asks:
Is this real text or scanned images? (OCR needed?)
Does it contain tables?
Headers / footers?
Multiple languages?
Version numbers?
Duplicate pages?
References to other documents?
Sensitive information?
Structured sections (chapters, clauses)?
Only after understanding the document, indexing begins.
Which system produces better answers 6 months later?
Not the one with the better LLM.
The one with better ingestion.
If there's one idea to remember:
RAG is not an AI problem.
RAG is an information retrieval problem that happens to use AI.
That changes everything.
Instead of asking:
"Which LLM should I use?"
You start asking:
How should documents be represented?
How should information be organized?
What makes one chunk better than another?
Which metadata should be extracted?
Which ranking strategy should be used?
How do we measure retrieval quality?
How do we know retrieval failed?
Those matter more than choosing GPT-5, Claude, or Gemini.
A brilliant LLM cannot generate knowledge it never received.
Garbage retrieval β garbage answers. Every time.
Now let's see what a toy RAG really looks like, and why it fails.
We'll use Python + sentence-transformers
chromadb
(local vector DB).
No LangChain magic, just raw code, so you see what's happening.
We'll build:
A naive RAG (fixed-size chunks).
Show where it breaks.
Compare with a slightly better version (structure-aware chunks + metadata).
Imagine you have a simple knowledge base:
pricing_policy.md
onboarding_guide.md
support_rules.md
Example content from pricing_policy.md
:
## Upgrading Plans
Customers can upgrade from Professional to Enterprise.
**Important:** Active invoices must be closed before upgrading.
If you have any open invoices, contact billing@example.com.
## Downgrading Plans
Downgrading is allowed only if:
- No active trials
- No pending invoices
And a user query:
"Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?"
from pathlib import Path
from typing import List
import chromadb
from chromadb.config import Settings
from sentence_transformers import SentenceTransformer
documents_dir = Path("docs")
texts = []
for md_file in documents_dir.glob("*.md"):
texts.append(md_file.read_text(encoding="utf-8"))
def naive_chunk(text: str, chunk_size: int = 500, overlap: int = 50) -> List[str]:
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunks.append(text[start:end])
start += chunk_size - overlap
return chunks
all_chunks = []
for text in texts:
all_chunks.extend(naive_chunk(text))
embedder = SentenceTransformer("all-MiniLM-L6-v2")
chunk_embeddings = embedder.encode(all_chunks, convert_numpy=True)
chroma_client = chromadb.Client(Settings(
chroma_db_impl="duckdb+parquet",
persist_directory="./chroma_db"
))
collection = chroma_client.create_collection("naive_rag")
for i, chunk in enumerate(all_chunks):
collection.add(
ids=[f"chunk_{i}"],
embeddings=[chunk_embeddings[i].tolist()],
documents=[chunk]
)
def query_naive_rag(collection, query: str, top_k: int = 5) -> List[str]:
query_emb = embedder.encode([query], convert_numpy=True)[0].tolist()
result = collection.query(
query_embeddings=[query_emb],
n_results=top_k
)
return result["documents"][0]
query = "Can Enterprise customers upgrade directly from the Professional plan while keeping active invoices?"
retrieved = query_naive_rag(collection, query, top_k=5)
for i, chunk in enumerate(retrieved, 1):
print(f"Chunk {i}:\n{chunk}\n")
What you'll likely see:
Some chunks might contain:
"Customers can upgrade from Professional to Enterprise."
"Downgrading is allowed only if..."
Random fragments with "invoices" but not the right rule.
But the crucial sentence:
"Active invoices must be closed before upgrading from Professional to Enterprise."
might be:
Split across two chunks.
Buried in a chunk with unrelated text.
Not in top-k at all.
Then LLM will generate:
"Yes, they can upgrade."
Because the retrieved context doesn't say "No".
This is exactly how naive RAG fails.
Not because embedding is wrong.
Because chunking + retrieval are not designed for real questions.
Now let's do it more thoughtfully.
We'll:
Split by Markdown headings.
Keep section title + content together.
Add metadata: doc_name
, section
, language
.
import re
from typing import List, Dict
def structure_aware_chunk(text: str, doc_name: str) -> List[Dict]:
chunks = []
pattern = r"(^#{1,2}\s+.+$)"
parts = re.split(pattern, text, flags=re.MULTILINE)
current_heading = None
current_content = []
for i, part in enumerate(parts):
if i == 0 and part.strip() == "":
continue
if re.match(pattern, part):
current_heading = part.strip()
current_content = []
else:
if current_heading:
current_content.append(part.strip())
if current_heading and (i + 1 >= len(parts) or re.match(pattern, parts[i + 1])):
content_text = "\n\n".join(current_content).strip()
if content_text:
chunks.append({
"text": f"{current_heading}\n\n{content_text}",
"metadata": {
"doc_name": doc_name,
"section": current_heading,
"language": "en",
}
})
current_heading = None
current_content = []
return chunks
all_chunks = []
for md_file in documents_dir.glob("*.md"):
doc_name = md_file.name
text = md_file.read_text(encoding="utf-8")
all_chunks.extend(structure_aware_chunk(text, doc_name))
chunk_embeddings = embedder.encode([c["text"] for c in all_chunks], convert_numpy=True)
collection2 = chroma_client.create_collection("better_rag")
for i, chunk in enumerate(all_chunks):
collection2.add(
ids=[f"chunk_{i}"],
embeddings=[chunk_embeddings[i].tolist()],
documents=[chunk["text"]],
metadatas=[chunk["metadata"]]
)
def query_better_rag(collection, query: str, top_k: int = 5) -> List[Dict]:
query_emb = embedder.encode([query], convert_numpy=True)[0].tolist()
result = collection.query(
query_embeddings=[query_emb],
n_results=top_k
)
return [
{"text": doc, "metadata": meta}
for doc, meta in zip(result["documents"][0], result["metadatas"][0])
]
retrieved2 = query_better_rag(collection2, query, top_k=5)
for i, item in enumerate(retrieved2, 1):
print(f"Chunk {i} (from {item['metadata']['doc_name']}, section: {item['metadata']['section']}):")
print(item["text"])
print()
What improves:
Chunks now preserve sections:
Metadata tells you:
Retrieval more likely to find the exact section about upgrading + invoices.
Now context to LLM might look like:
## Upgrading Plans
Customers can upgrade from Professional to Enterprise.
Important: Active invoices must be closed before upgrading.
If you have any open invoices, contact billing@example.com.
Then LLM can answer:
"No. Active invoices must be closed before upgrading from Professional to Enterprise."
Same embedding model.
Same LLM.
But better chunking + metadata β better retrieval β better answer.
Even in this tiny example:
Fixed-size chunks β fragmentation β wrong answer.
Structure-aware chunks + metadata β better context β correct answer.
In production, differences are even bigger:
Millions of documents.
Many languages.
Many versions.
Complex queries.
If you don't design ingestion, chunking, and metadata properly, no LLM will save you.
If you've only followed tutorials, you probably think RAG is one pipeline:
Query
β
Retrieve
β
Generate
β
Answer
That's enough for a demo.
It's catastrophic for production.
In any real system, you actually have two separate pipelines:
Offline (ingestion) pipeline β processes documents.
Online (query) pipeline β handles user requests.
They have completely different goals, constraints, and failure modes.
Designing them as one monolithic flow is one of the first mistakes that leads to production failures.
The offline pipeline is what happens to documents before anyone ever asks a question.
Offline Pipeline (ingestion)
Documents (PDF, HTML, Markdown, JSON)
β
Parser (structure-aware)
β
Cleaner (headers, footers, page numbers)
β
Chunker (semantic / structure-aware)
β
Metadata Extractor
β
Embedder
β
Indexer
βββ Vector DB
βββ BM25 / Keyword Index
----------------------------
Online Pipeline (query)
User Query
β
Query Processor (rewrite, expand)
β
Metadata Filters
β
Hybrid Search (Vector + BM25)
β
Reranker (Cross-Encoder)
β
Context Compression
β
Prompt Builder
β
LLM
β
Answer + Citations
* Reads raw documents.
* Extracts structure: headings, tables, lists, code blocks, paragraphs.
* For PDFs: uses OCR if needed, understands layout.
* For Markdown / HTML: preserves semantic tags.
* Removes noise:
* Headers / footers ("Company", "Confidential").
* Page numbers.
* Watermarks.
* Repeated boilerplate text.
* Normalizes whitespace, encoding, line breaks.
* Fixes broken sentences or artifacts.
* Splits documents into meaningful chunks.
* Ensures:
* Headings are not cut in the middle.
* Tables stay intact.
* Paragraph boundaries are respected.
* Uses strategies like:
* Structure-aware chunking
* Semantic chunking
* Parent-child chunking
* (We'll cover these in detail in Chapter 5.)
* Extracts document-level metadata:
* `document_type`
* `language`
* `version`
* `department`
* `created_at`, `updated_at`
* Extracts chunk-level metadata:
* `section` / `heading`
* `page`
* `chapter`
* `parent_id`
* `hierarchy_path`
* Computes embeddings for each chunk.
* Optionally computes document-level embeddings too.
* Uses models appropriate for:
* Domain (technical, legal, general).
* Language (English, Russian, multilingual).
* Inserts chunks into:
* Vector DB (dense embeddings).
* BM25 / keyword index (sparse).
* (Optionally) graph index (for knowledge graphs).
It doesn't need to be fast.
Processing a document can take seconds or minutes.
Documents are not requested by users in real time.
It must be correct and robust.
If ingestion is wrong, retrieval will never be right.
Bad chunks, missing metadata, wrong parsing β garbage retrieval.
It can be batched or incremental.
This pipeline is where you decide:
How documents are represented.
How information is organized.
What metadata is available for filtering.
What kind of chunks the retriever will see.
If you ignore this pipeline and only focus on "query β retrieve β generate", you're building a demo, not a production system.
The online pipeline is what happens when a user asks a question.
Online Pipeline (query)
User Query
β
Query Processor (rewrite, expand, normalize)
β
Metadata Filters (language, department, version, etc.)
β
Hybrid Search (Vector + BM25)
β
Reranker (Cross-Encoder)
β
Context Compression
β
Prompt Builder
β
LLM
β
Answer + Citations
* Rewrites or expands the query:
* "upgrade plan" β "upgrade from Professional to Enterprise plan".
* Adds implicit context (user role, department).
* Normalizes:
* Lowercases, removes noise.
* Handles typos, slang, abbreviations.
* Applies filters based on:
* User's language.
* User's department / team.
* Document version (latest, stable, deprecated).
* Document type (policy, doc, article).
* Reduces the candidate set before search.
* Runs:
* Vector search (semantic similarity).
* BM25 / keyword search (exact term matching).
* Combines scores:
* `final_score = Ξ± * vector_score + Ξ² * bm25_score`.
* Takes top-K candidates (e.g., 50β100).
* Uses a cross-encoder to compute a more precise relevance score for each.
* Sorts and picks the best N (e.g., 5β10).
* Limits the number of chunks sent to the LLM.
* Drops low-relevance chunks.
* Optionally summarizes or extracts key sentences.
* Ensures the prompt stays within token limits.
* Constructs the final prompt:
* System instructions.
* Retrieved context.
* User query.
* Output format instructions (citations, confidence, etc.).
* Generates the answer based on the prompt.
* Ideally grounded in the retrieved context.
* Returns:
* The final answer.
* Citations (document title, section, page).
* Optionally: confidence score, latency, token usage.
It must be fast.
Users expect answers in < 1 second, ideally < 500ms.
It must be stable under load.
It must be observable.
This pipeline is where you decide:
How queries are interpreted.
How retrieval is constrained.
How context is prepared for the LLM.
How answers are presented and tracked.
Most tutorials only design the online pipeline:
Query
β
Vector Search
β
LLM
β
Answer
They assume:
Documents are already chunked.
Chunks are already embedded.
Indices already exist.
Metadata is perfect.
In production, that assumption is fatal.
Company A:
Uploads a PDF policy.
Extracts raw text.
Splits every 500 tokens.
Embeds.
Stores.
Company B:
Uploads the same PDF.
Detects structure: headings, sections, tables.
Cleans headers/footers, page numbers.
Creates section-based chunks.
Extracts metadata: document_type
, version
, language
.
Embeds with domain-specific model.
Stores in vector + BM25 indexes.
Six months later:
Company A's RAG returns fragmented, noisy chunks.
Company B's RAG returns structured, filtered, relevant chunks.
The difference is not the LLM.
It's the offline pipeline.
Company C:
Uses only vector search.
No metadata filters.
No reranking.
Sends top-50 chunks to LLM.
Company D:
Uses hybrid search (vector + BM25).
Applies metadata filters (language, version, department).
Reranks with cross-encoder.
Sends top-5 compressed chunks to LLM.
Under the same LLM, Company D's answers are:
More precise.
More grounded.
More consistent.
The difference is not the LLM.
It's the online pipeline.
Treat them as two independent systems:
Offline = "data engineering" system.
Online = "search engine" system.
You can:
Change chunking strategies without touching retrieval logic.
Add new metadata fields without changing the LLM.
Swap embedding models without rewriting the query pipeline.
Introduce reranking without changing ingestion.
But you must design both.
If one is weak, the whole system fails.
In the next articles, we'll go deeper into each component of a production RAG system:
Part 2: Building a Production RAG Pipeline
We will explore:
Ingestion pipelines (parsers, cleaners, and document processing).
Chunking strategies with practical examples.
Metadata design and how it improves retrieval quality.
Part 3: Advanced Retrieval for Production RAG
We will explore:
Embeddings and semantic representation.
Hybrid search combining vector search and keyword retrieval.
Reranking strategies.
Context compression techniques.
Part 4: Scaling RAG Systems in Production
We will explore:
Prompt construction patterns.
Large-scale RAG architecture.
Performance optimization.
Scaling pipelines for millions of documents.
Part 5: Evaluating and Improving Production RAG Systems
We will explore:
RAG evaluation methods.
Production monitoring.
Common failure patterns.
15 production mistakes with real-world scenarios.
A complete production RAG checklist.
Together, these articles cover the complete journey from a simple RAG prototype to a reliable production-grade AI system.