You're building a chatbot for a company's internal documentation — hundreds of pages of engineering specs, HR policies, product guides. You plug in ChatGPT, write a clean system prompt,launch a demo. Someone asks "What's the reimbursement limit for travel expenses?" and the model confidently answers with a number. The number is wrong. It's not from your docs — it's from the model's general training data, some pattern absorbed from the internet years ago that roughly sounds like a reasonable reimbursement policy.
This is the fundamental problem with raw LLMs in production: they don't know your data. They know what was in their training set — a snapshot of text from months or years ago, with no awareness of your internal docs, your product specs, your latest pricing, your customer records, or anything else that lives inside your organization. And when they don't know something, they don't say "I don't know" — they say something plausible-sounding that might be completely fabricated.
The fix has a name: Retrieval-Augmented Generation, almost universally abbreviated as RAG. The idea is elegant: before asking the model to answer a question, find the relevant pieces of your actual data and hand them to the model as context. Now, instead of guessing, the model is summarizing and reasoning over the real information you provided.
The gap between "RAG sounds simple" and "RAG actually works reliably in production" is where most teams spend weeks of engineering time. Understanding the full pipeline — every
stage, every design decision, every place it quietly breaks — is what this article is about.
Before code, you need the right mental model, because RAG is counterintuitive until it clicks.
Think about how you answer questions as a consultant. If someone asks you a question about a client's contract terms, you don't try to recall everything from memory — you pull up the contract, skim for the relevant section and answer based on what's in front of you. You're not generating from memory; you're reading and summarizing from a source.
RAG turns an LLM into that consultant. You build a system that:
Without RAG:
User question → LLM (guesses from training data) → Response (possibly hallucinated)
With RAG:
User question → Retrieve relevant docs → LLM (reads your actual docs) → Grounded response
This is why RAG is the dominant pattern for enterprise AI applications. It doesn't eliminate the LLM's tendency to hallucinate — but it gives the model true ground to stand on,and makes it much easier to audit: if the answer is wrong, you can trace it back to exactly which chunk was retrieved and why.
The full pipeline has two distinct phases that most beginners conflate into one. Let's separate them.
Almost every RAG confusion I've seen in teams comes from mixing up what happens before a user ever sends a message versus what happens when they do. These are two fundamentally different processes with different triggers, different costs, and different failure modes.
This is the preparation phase. You run it ahead of time — sometimes once, sometimes on a schedule, sometimes triggered by new document uploads. It does not happen in response to user queries.
[ Your Raw Documents ]
│
▼
[ Document ] ← Pull documents in from wherever they live
│
▼
[ Text Splitter ] ← Break documents into manageable chunks
│
▼
[ Embedding Model ] ← Convert each chunk into a numeric vector
│
▼
[ Vector Store ] ← Store vectors + original text in a searchable database
The result: a vector store that contains your entire knowledge base in a format where "find me content similar to this question" is a fast, cheap operation.
This is the live path. It runs every time a user sends a message.
[ User Question ]
│
▼
[ Embedding Model ] ← Convert the question into a vector
│
▼
[ Vector Store Retriever ] ← Find the chunks most similar to the question
│
▼
[ Prompt Template ] ← Inject retrieved chunks + question into a prompt
│
▼
[ LLM ] ← Generates answer grounded in the retrieved context
│
▼
[ Response to User ]
These two phases have completely different performance characteristics. Indexing is slow and expensive per document but runs offline. Retrieval + generation must be fast — it's the live user experience. Every design decision in RAG falls into one of these two phases. Keep this separation in mind as we go through each component.
The first problem in any real RAG system is always the same: your data doesn't live in a clean text file. It lives in PDFs, Word documents, Notion pages, Confluence wikis,Google Drive folders, databases, Slack threads, web pages, and half a dozen other formats
— each requiring different handling to extract usable text.
Document s in LangChain are the abstraction that handles this — they give you a consistent load()
interface regardless of the source, returning a list of Document
objects (each containing a page_content
string and a metadata
dict).
from langchain_community.document_s import (
PyPDF, # PDFs — extracts text page by page
WebBase, # Web pages — fetches and strips HTML
Text, # Plain .txt files
CSV, # Spreadsheets — each row becomes a Document
UnstructuredWordDocument, # .docx files
NotionDirectory, # Exported Notion workspace
)
pdf_ = PyPDF("engineering_spec.pdf")
pdf_docs = pdf_.load()
web_ = WebBase("https://docs.mycompany.com/api-reference")
web_docs = web_.load()
csv_ = CSV("product_faq.csv", source_column="url")
csv_docs = csv_.load()
The metadata
dict is not an afterthought — it's critical for production systems. Every document you ingest should carry metadata: its source URL, file name, creation date, author, document type, whatever is relevant. You'll use this metadata later for filtering ("only retrieve from documents tagged HR policy") and for citations in your responses ("according to engineering_spec.pdf, page 12").
for doc in pdf_docs[:3]:
print(f"Content length: {len(doc.page_content)} chars")
print(f"Metadata: {doc.metadata}")
print(f"Preview: {doc.page_content[:200]}")
print("---")
⚠️
Heads up:PDFs are the most common source and the most problematic. Scanned PDFs
(images of text rather than actual text) will return empty strings with most s —
you'll need OCR processing before the text is even extractable. Always validate that
your is actually getting text, not silence.
Here's where most RAG beginners make their biggest architectural mistake, because this stage looks trivial and isn't.
You have your documents loaded. They might be entire 80-page PDFs or multi-thousand-word web pages. You can't stuff all of that into a prompt — LLMs have context limits, and even if you could, you don't want to. You want to retrieve the specific relevant section, not an entire document.
So you split documents into chunks — smaller pieces of text that can be independently retrieved and included in a prompt. The decisions you make here — how big each chunk is, how they overlap, how they respect the document's natural structure — have a bigger impact on RAG quality than almost any other single variable.
This is a genuine tension with no universal right answer:
When you split at token/character boundaries, you'll slice through sentences, sometimes mid-thought. The sentence that contains the key information might end up half in chunk 12 and half in chunk 13, and neither chunk retrieves well. Overlap addresses this by
repeating the last N tokens of each chunk at the start of the next.
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=500, # Target max characters per chunk
chunk_overlap=50, # Last 50 chars of chunk N repeated at start of chunk N+1
separators=["\n\n", "\n", ". ", " ", ""] # Try these in order
)
chunks = splitter.split_documents(pdf_docs)
print(f"Original: {len(pdf_docs)} pages")
print(f"After splitting: {len(chunks)} chunks")
for i, chunk in enumerate(chunks[:3]):
print(f"\n--- Chunk {i} ({len(chunk.page_content)} chars) ---")
print(chunk.page_content)
RecursiveCharacterTextSplitter
is the general-purpose default, but different content
types benefit from different strategies:
from langchain.text_splitter import (
RecursiveCharacterTextSplitter, # General prose — paragraphs, articles, docs
MarkdownHeaderTextSplitter, # Markdown — splits on headings, preserves structure
Language, # Code — splits on functions/classes, not mid-line
)
md_splitter = MarkdownHeaderTextSplitter(
headers_to_split_on=[
("#", "h1"),
("##", "h2"),
("###", "h3"),
]
)
code_splitter = RecursiveCharacterTextSplitter.from_language(
language=Language.PYTHON,
chunk_size=1000,
chunk_overlap=100,
)
💡
Pro tip:Add the source metadata to every chunk explicitly — once you've split
200 documents into 5,000 chunks, you need to know which chunk came from where.
LangChain's splitters preserve metadata from the parentDocument
, but always verify.
You now have thousands of text chunks. The next problem: how do you find the chunks relevant to a specific user question, fast, at query time?
The answer is semantic search, and it works through embeddings.
Remember from the first article: each chunk of text can be converted into a vector — a list of hundreds or thousands of numbers — that represents its meaning in mathematical space. This conversion is done by an embedding model (a different, smaller model than your LLM — its only job is text → vector).
The key property: semantically similar text produces mathematically similar vectors. "How do I reset my password?" and "Steps to change my account credentials" are worded completely differently but mean similar things — their embedding vectors will be close together in vector space, close enough to find each other via similarity search.
This is why RAG can retrieve relevant content even when the user's question uses completely different words from your documentation. You're not searching for matching keywords; you're searching for matching meaning.
from langchain_openai import OpenAIEmbeddings
from langchain_community.embeddings import HuggingFaceEmbeddings
openai_embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector = openai_embeddings.embed_query("How do I reset my password?")
Embeddings are only useful if you can search through them efficiently. A vector store is a database built for this — you insert vectors + their associated text, and at query time you give it a query vector and it returns the N most similar stored vectors.
from langchain_community.vectorstores import (
FAISS, # Facebook AI Similarity Search — in-memory, great for dev/small scale
Chroma, # Lightweight persistent vector store — good for local development
Pinecone, # Managed cloud vector database — production scale
Weaviate, # Open-source cloud/self-hosted — good for complex filtering
pgvector, # Postgres extension — great if you're already on Postgres
)
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(
documents=chunks, # Your split documents from Chapter 4
embedding=embeddings # The model that converts text → vector
)
vectorstore.save_local("my_knowledge_base")
vectorstore = FAISS.load_local("my_knowledge_base", embeddings,
allow_dangerous_deserialization=True)
⚠️
Heads up:Embedding your entire document set costs money and takes time —
OpenAI's embedding API charges per token, and embedding 10,000 chunks can take minutes.
Always persist your vector store to disk (or use a managed service) so the indexing
phase runs once, not on every app restart.
A retriever is the interface that sits in front of your vector store and answers one question: given this query, what chunks should I surface? It's what Phase 2 of the pipeline calls when a user sends a message.
The simplest retriever is just a vector store wrapper:
retriever = vectorstore.as_retriever(
search_type="similarity", # Find the N most semantically similar chunks
search_kwargs={"k": 4} # Return the top 4 chunks
)
relevant_chunks = retriever.invoke("What is the reimbursement limit for travel?")
for chunk in relevant_chunks:
print(chunk.page_content[:300])
print(f"Source: {chunk.metadata.get('source', 'unknown')}")
print("---")
But "top 4 by similarity" is actually a pretty naive strategy, and there are several retrieval approaches that meaningfully improve quality in practice.
Pure similarity search has a hidden flaw: if your knowledge base has five paragraphs that all say the same thing slightly differently, you'll retrieve all five of them instead of
getting diverse, complementary information. MMR trades off between relevance (is this chunk similar to the query?) and diversity (is this chunk different from what I've already retrieved?).
retriever = vectorstore.as_retriever(
search_type="mmr", # Maximum Marginal Relevance
search_kwargs={
"k": 4, # Return 4 chunks
"fetch_k": 20, # Consider top 20 by similarity first
"lambda_mult": 0.7 # 0 = pure diversity, 1 = pure similarity
}
)
Don't retrieve bad chunks just to hit your k
target. If nothing in your knowledge base is actually relevant to the query, it's better to return nothing than to return vaguely-related content that confuses the model.
retriever = vectorstore.as_retriever(
search_type="similarity_score_threshold",
search_kwargs={
"score_threshold": 0.7, # Only return chunks above this similarity score
"k": 4
}
)
Vector search finds semantically similar content, but sometimes you need to constrain which documents are even eligible — only search HR policies, or only search documents updated in the last 30 days, or only search content in English. This is where metadata filters come in.
retriever = vectorstore.as_retriever(
search_kwargs={
"k": 4,
"filter": {"department": "HR", "document_type": "policy"}
}
)
One underrated technique: the user's question, as phrased, might not be the best query for semantic search. Multi-query retrieval uses the LLM itself to generate multiple alternative phrasings of the question, runs all of them as separate queries, and takes the union of results. This catches relevant chunks that a single query might miss due to phrasing mismatches.
from langchain.retrievers import MultiQueryRetriever
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(temperature=0)
multi_retriever = MultiQueryRetriever.from_llm(
retriever=vectorstore.as_retriever(search_kwargs={"k": 3}),
llm=llm
)
Here's a complete, minimal, working RAG chain using LangChain's modern interface. This is the skeleton of nearly every document Q&A system you'll ever build.
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_community.document_s import PyPDF
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
= PyPDF("company_handbook.pdf")
raw_docs = .load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(raw_docs)
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(chunks, embeddings)
vectorstore.save_local("handbook_index")
vectorstore = FAISS.load_local("handbook_index", embeddings,
allow_dangerous_deserialization=True)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
rag_prompt = ChatPromptTemplate.from_messages([
("system",
"You are a helpful assistant answering questions about company policy. "
"Use ONLY the following retrieved context to answer the question. "
"If the answer is not in the context, say: 'I don't have that information.' "
"Do not use any knowledge outside of what's provided below.\n\n"
"Context:\n{context}"),
("user", "{question}")
])
llm = ChatOpenAI(model="gpt-4o", temperature=0) # temp=0 for consistent, factual answers
def format_docs(docs):
return "\n\n".join(
f"[Source: {doc.metadata.get('source', 'unknown')}, "
f"Page: {doc.metadata.get('page', '?')}]\n{doc.page_content}"
for doc in docs
)
rag_chain = (
{
"context": retriever | format_docs, # retrieve → format as string
"question": RunnablePassthrough() # pass question through unchanged
}
| rag_prompt # fill the template
| llm # call the model
| StrOutputParser() # parse response to plain string
)
response = rag_chain.invoke("What is the reimbursement limit for travel expenses?") print(response)
This chain is clean, auditable, and straightforward to swap out individual pieces — change the , the splitter settings, the retriever strategy, or the underlying LLM without touching the rest of the pipeline.
RAG systems fail in consistent, diagnosable ways. Knowing the failure modes ahead of time saves days of debugging.
"The model ignores the retrieved context and makes things up anyway." Your system prompt isn't firm enough. Add explicit grounding instructions: "Answer only from the context below. If the answer is not in the context, say so." Setting temperature=0
on the LLM also helps with consistency.
"Retrieval returns irrelevant chunks."
This is the most common failure, and it almost always traces to chunking. Chunks are too large (low retrieval precision), too small (not enough context per chunk), or split at bad boundaries (sentences split mid-idea). Try smaller chunks, increase overlap, or switch to a structure-aware splitter (markdown header, code-aware).
"The right information is in the document but retrieval never finds it." Your query phrasing and your document phrasing are too semantically distant. Try the multi-query retriever to cast a wider net. Also check: was this section actually indexed? s silently fail on scanned PDFs, complex tables, and certain layouts.
"The model's answer is correct but I can't trace where it came from." Always include source metadata in your formatted context and instruct the model to cite it. Without attribution, RAG is a black box — with attribution, it's auditable.
"It works in testing but gives bad answers on long documents at scale." Your k
value (number of retrieved chunks) is hitting the LLM's context window limit. Either reduce k
, reduce chunk size, or use a model with a larger context window. Also check for embedding model mismatch — if you indexed with one embedding model and query with another, the vector space is incompatible.
You now have a mental model of the complete RAG pipeline from ingestion to response — not just what the pieces are called, but why each one exists and what breaks when it's wrong.
The natural next step is building something real with this. Start with a single clean PDF and a handful of test questions whose correct answers you already know. Run the full pipeline, check whether your retriever surfaces the right chunks for each question, then tune chunk size and overlap until retrieval quality is solid. Get that working before touching retriever strategies or advanced techniques.
Once retrieval is reliable, the rest — getting the model to answer well, adding conversation memory, citing sources — falls into place quickly. The retrieval quality is the foundation. Everything else is downstream from it.
RAG isn't magic. It's plumbing — well-designed plumbing that lets a language model answer questions about your actual data instead of guessing. Build the plumbing carefully, and the magic takes care of itself.