RAG Explained: How to Give Your LLM a Memory It Can Actually Trust

A developer explains Retrieval-Augmented Generation (RAG), a technique that gives LLMs access to actual data before answering, reducing hallucinations. The pipeline involves indexing documents into a vector store during ingestion, then retrieving relevant chunks at query time to ground the model's response.

You're building a chatbot for a company's internal documentation — hundreds of pages of engineering specs, HR policies, product guides. You plug in ChatGPT, write a clean system prompt,launch a demo. Someone asks "What's the reimbursement limit for travel expenses?" and the model confidently answers with a number. The number is wrong. It's not from your docs — it's from the model's general training data, some pattern absorbed from the internet years ago that roughly sounds like a reasonable reimbursement policy. This is the fundamental problem with raw LLMs in production: they don't know your data. They know what was in their training set — a snapshot of text from months or years ago, with no awareness of your internal docs, your product specs, your latest pricing, your customer records, or anything else that lives inside your organization. And when they don't know something, they don't say "I don't know" — they say something plausible-sounding that might be completely fabricated. The fix has a name: Retrieval-Augmented Generation , almost universally abbreviated as RAG . The idea is elegant: before asking the model to answer a question, find the relevant pieces of your actual data and hand them to the model as context. Now, instead of guessing, the model is summarizing and reasoning over the real information you provided. The gap between "RAG sounds simple" and "RAG actually works reliably in production" is where most teams spend weeks of engineering time. Understanding the full pipeline — every stage, every design decision, every place it quietly breaks — is what this article is about. Before code, you need the right mental model, because RAG is counterintuitive until it clicks. Think about how you answer questions as a consultant. If someone asks you a question about a client's contract terms, you don't try to recall everything from memory — you pull up the contract, skim for the relevant section and answer based on what's in front of you. You're not generating from memory; you're reading and summarizing from a source. RAG turns an LLM into that consultant. You build a system that: Without RAG: User question → LLM guesses from training data → Response possibly hallucinated With RAG: User question → Retrieve relevant docs → LLM reads your actual docs → Grounded response This is why RAG is the dominant pattern for enterprise AI applications. It doesn't eliminate the LLM's tendency to hallucinate — but it gives the model true ground to stand on,and makes it much easier to audit: if the answer is wrong, you can trace it back to exactly which chunk was retrieved and why. The full pipeline has two distinct phases that most beginners conflate into one. Let's separate them. Almost every RAG confusion I've seen in teams comes from mixing up what happens before a user ever sends a message versus what happens when they do. These are two fundamentally different processes with different triggers, different costs, and different failure modes. This is the preparation phase. You run it ahead of time — sometimes once, sometimes on a schedule, sometimes triggered by new document uploads. It does not happen in response to user queries. Your Raw Documents │ ▼ Document Loader ← Pull documents in from wherever they live │ ▼ Text Splitter ← Break documents into manageable chunks │ ▼ Embedding Model ← Convert each chunk into a numeric vector │ ▼ Vector Store ← Store vectors + original text in a searchable database The result: a vector store that contains your entire knowledge base in a format where "find me content similar to this question" is a fast, cheap operation. This is the live path. It runs every time a user sends a message. User Question │ ▼ Embedding Model ← Convert the question into a vector │ ▼ Vector Store Retriever ← Find the chunks most similar to the question │ ▼ Prompt Template ← Inject retrieved chunks + question into a prompt │ ▼ LLM ← Generates answer grounded in the retrieved context │ ▼ Response to User These two phases have completely different performance characteristics. Indexing is slow and expensive per document but runs offline. Retrieval + generation must be fast — it's the live user experience. Every design decision in RAG falls into one of these two phases. Keep this separation in mind as we go through each component. The first problem in any real RAG system is always the same: your data doesn't live in a clean text file. It lives in PDFs, Word documents, Notion pages, Confluence wikis,Google Drive folders, databases, Slack threads, web pages, and half a dozen other formats — each requiring different handling to extract usable text. Document loaders in LangChain are the abstraction that handles this — they give you a consistent load interface regardless of the source, returning a list of Document objects each containing a page content string and a metadata dict . from langchain community.document loaders import PyPDFLoader, PDFs — extracts text page by page WebBaseLoader, Web pages — fetches and strips HTML TextLoader, Plain .txt files CSVLoader, Spreadsheets — each row becomes a Document UnstructuredWordDocumentLoader, .docx files NotionDirectoryLoader, Exported Notion workspace Every loader shares the same interface — .load returns List Document Load a PDF — returns one Document per page, with page number in metadata pdf loader = PyPDFLoader "engineering spec.pdf" pdf docs = pdf loader.load pdf docs 0 .page content → "Chapter 1: System Architecture..." pdf docs 0 .metadata → {"source": "engineering spec.pdf", "page": 0} Load a web page — strips HTML, extracts main text content web loader = WebBaseLoader "https://docs.mycompany.com/api-reference" web docs = web loader.load Load a CSV — useful for structured data like FAQs or product catalogs csv loader = CSVLoader "product faq.csv", source column="url" csv docs = csv loader.load The metadata dict is not an afterthought — it's critical for production systems. Every document you ingest should carry metadata: its source URL, file name, creation date, author, document type, whatever is relevant. You'll use this metadata later for filtering "only retrieve from documents tagged HR policy" and for citations in your responses "according to engineering spec.pdf, page 12" . Always inspect what your loader actually produced before moving on. Loaders fail silently in surprising ways — PDFs with scanned images return empty page content strings; HTML with JavaScript-rendered content may come back blank. for doc in pdf docs :3 : print f"Content length: {len doc.page content } chars" print f"Metadata: {doc.metadata}" print f"Preview: {doc.page content :200 }" print "---" ⚠️ Heads up:PDFs are the most common source and the most problematic. Scanned PDFs images of text rather than actual text will return empty strings with most loaders — you'll need OCR processing before the text is even extractable. Always validate that your loader is actually getting text, not silence. Here's where most RAG beginners make their biggest architectural mistake, because this stage looks trivial and isn't. You have your documents loaded. They might be entire 80-page PDFs or multi-thousand-word web pages. You can't stuff all of that into a prompt — LLMs have context limits, and even if you could, you don't want to . You want to retrieve the specific relevant section, not an entire document. So you split documents into chunks — smaller pieces of text that can be independently retrieved and included in a prompt. The decisions you make here — how big each chunk is, how they overlap, how they respect the document's natural structure — have a bigger impact on RAG quality than almost any other single variable. This is a genuine tension with no universal right answer: When you split at token/character boundaries, you'll slice through sentences, sometimes mid-thought. The sentence that contains the key information might end up half in chunk 12 and half in chunk 13, and neither chunk retrieves well. Overlap addresses this by repeating the last N tokens of each chunk at the start of the next. python from langchain.text splitter import RecursiveCharacterTextSplitter The workhorse of LangChain splitting — tries to split on natural boundaries first paragraphs → sentences → words before falling back to raw characters. splitter = RecursiveCharacterTextSplitter chunk size=500, Target max characters per chunk chunk overlap=50, Last 50 chars of chunk N repeated at start of chunk N+1 separators= "\n\n", "\n", ". ", " ", "" Try these in order chunks = splitter.split documents pdf docs print f"Original: {len pdf docs } pages" print f"After splitting: {len chunks } chunks" Good habit: sanity-check your chunks before continuing for i, chunk in enumerate chunks :3 : print f"\n--- Chunk {i} {len chunk.page content } chars ---" print chunk.page content RecursiveCharacterTextSplitter is the general-purpose default, but different content types benefit from different strategies: from langchain.text splitter import RecursiveCharacterTextSplitter, General prose — paragraphs, articles, docs MarkdownHeaderTextSplitter, Markdown — splits on headings, preserves structure Language, Code — splits on functions/classes, not mid-line For technical documentation in Markdown — splits on headers rather than character count md splitter = MarkdownHeaderTextSplitter headers to split on= " ", "h1" , " ", "h2" , " ", "h3" , Each resulting chunk carries header metadata — great for citations later For code repositories — respects function/class boundaries code splitter = RecursiveCharacterTextSplitter.from language language=Language.PYTHON, chunk size=1000, chunk overlap=100, 💡 Pro tip:Add the source metadata to every chunk explicitly — once you've split 200 documents into 5,000 chunks, you need to know which chunk came from where. LangChain's splitters preserve metadata from the parent Document , but always verify. You now have thousands of text chunks. The next problem: how do you find the chunks relevant to a specific user question, fast, at query time? The answer is semantic search , and it works through embeddings . Remember from the first article: each chunk of text can be converted into a vector — a list of hundreds or thousands of numbers — that represents its meaning in mathematical space. This conversion is done by an embedding model a different, smaller model than your LLM — its only job is text → vector . The key property: semantically similar text produces mathematically similar vectors. "How do I reset my password?" and "Steps to change my account credentials" are worded completely differently but mean similar things — their embedding vectors will be close together in vector space, close enough to find each other via similarity search. This is why RAG can retrieve relevant content even when the user's question uses completely different words from your documentation. You're not searching for matching keywords; you're searching for matching meaning . python from langchain openai import OpenAIEmbeddings from langchain community.embeddings import HuggingFaceEmbeddings OpenAI's embedding model — cloud API, very good quality openai embeddings = OpenAIEmbeddings model="text-embedding-3-small" Or a local HuggingFace model — free, runs on your hardware, great for cost-sensitive or privacy-sensitive use cases your documents never leave your infrastructure local embeddings = HuggingFaceEmbeddings model name="sentence-transformers/all-MiniLM-L6-v2" What actually happens under the hood you usually don't call this directly : vector = openai embeddings.embed query "How do I reset my password?" vector is a list of ~1500 floats — the "meaning fingerprint" of that sentence print f"Vector dimensions: {len vector }" e.g., 1536 for text-embedding-3-small Embeddings are only useful if you can search through them efficiently. A vector store is a database built for this — you insert vectors + their associated text, and at query time you give it a query vector and it returns the N most similar stored vectors. from langchain community.vectorstores import FAISS, Facebook AI Similarity Search — in-memory, great for dev/small scale Chroma, Lightweight persistent vector store — good for local development Pinecone, Managed cloud vector database — production scale Weaviate, Open-source cloud/self-hosted — good for complex filtering pgvector, Postgres extension — great if you're already on Postgres Building a FAISS vector store from chunks — this runs your embedding model on every chunk and stores the result. This is the slow, expensive indexing step. from langchain community.vectorstores import FAISS from langchain openai import OpenAIEmbeddings embeddings = OpenAIEmbeddings This call: embeds every chunk + builds the search index vectorstore = FAISS.from documents documents=chunks, Your split documents from Chapter 4 embedding=embeddings The model that converts text → vector Persist to disk so you don't re-embed every time your app restarts vectorstore.save local "my knowledge base" Load it back later — no re-embedding needed vectorstore = FAISS.load local "my knowledge base", embeddings, allow dangerous deserialization=True ⚠️ Heads up:Embedding your entire document set costs money and takes time — OpenAI's embedding API charges per token, and embedding 10,000 chunks can take minutes. Always persist your vector store to disk or use a managed service so the indexing phase runs once, not on every app restart. A retriever is the interface that sits in front of your vector store and answers one question: given this query, what chunks should I surface? It's what Phase 2 of the pipeline calls when a user sends a message. The simplest retriever is just a vector store wrapper: The most basic retriever — pure semantic similarity search retriever = vectorstore.as retriever search type="similarity", Find the N most semantically similar chunks search kwargs={"k": 4} Return the top 4 chunks Test it — this is what runs at query time: relevant chunks = retriever.invoke "What is the reimbursement limit for travel?" for chunk in relevant chunks: print chunk.page content :300 print f"Source: {chunk.metadata.get 'source', 'unknown' }" print "---" But "top 4 by similarity" is actually a pretty naive strategy, and there are several retrieval approaches that meaningfully improve quality in practice. Pure similarity search has a hidden flaw: if your knowledge base has five paragraphs that all say the same thing slightly differently, you'll retrieve all five of them instead of getting diverse, complementary information. MMR trades off between relevance is this chunk similar to the query? and diversity is this chunk different from what I've already retrieved? . retriever = vectorstore.as retriever search type="mmr", Maximum Marginal Relevance search kwargs={ "k": 4, Return 4 chunks "fetch k": 20, Consider top 20 by similarity first "lambda mult": 0.7 0 = pure diversity, 1 = pure similarity } Don't retrieve bad chunks just to hit your k target. If nothing in your knowledge base is actually relevant to the query, it's better to return nothing than to return vaguely-related content that confuses the model. retriever = vectorstore.as retriever search type="similarity score threshold", search kwargs={ "score threshold": 0.7, Only return chunks above this similarity score "k": 4 } Vector search finds semantically similar content, but sometimes you need to constrain which documents are even eligible — only search HR policies, or only search documents updated in the last 30 days, or only search content in English. This is where metadata filters come in. retriever = vectorstore.as retriever search kwargs={ "k": 4, "filter": {"department": "HR", "document type": "policy"} Only consider chunks where metadata matches these conditions } One underrated technique: the user's question, as phrased, might not be the best query for semantic search. Multi-query retrieval uses the LLM itself to generate multiple alternative phrasings of the question, runs all of them as separate queries, and takes the union of results. This catches relevant chunks that a single query might miss due to phrasing mismatches. python from langchain.retrievers import MultiQueryRetriever from langchain openai import ChatOpenAI llm = ChatOpenAI temperature=0 multi retriever = MultiQueryRetriever.from llm retriever=vectorstore.as retriever search kwargs={"k": 3} , llm=llm Under the hood: generates 3-5 alternative phrasings of the query, retrieves for each, deduplicates results Here's a complete, minimal, working RAG chain using LangChain's modern interface. This is the skeleton of nearly every document Q&A system you'll ever build. python from langchain openai import ChatOpenAI, OpenAIEmbeddings from langchain community.vectorstores import FAISS from langchain community.document loaders import PyPDFLoader from langchain.text splitter import RecursiveCharacterTextSplitter from langchain.prompts import ChatPromptTemplate from langchain core.runnables import RunnablePassthrough from langchain core.output parsers import StrOutputParser ─── PHASE 1: INDEXING run once, offline ─────────────────────────────────── 1. Load documents loader = PyPDFLoader "company handbook.pdf" raw docs = loader.load 2. Split into chunks splitter = RecursiveCharacterTextSplitter chunk size=500, chunk overlap=50 chunks = splitter.split documents raw docs 3. Embed and store embeddings = OpenAIEmbeddings vectorstore = FAISS.from documents chunks, embeddings vectorstore.save local "handbook index" ─── PHASE 2: RETRIEVAL + GENERATION runs per query ──────────────────────── Load the pre-built index skip re-embedding vectorstore = FAISS.load local "handbook index", embeddings, allow dangerous deserialization=True retriever = vectorstore.as retriever search kwargs={"k": 4} The prompt that instructs the model to answer ONLY from retrieved context rag prompt = ChatPromptTemplate.from messages "system", "You are a helpful assistant answering questions about company policy. " "Use ONLY the following retrieved context to answer the question. " "If the answer is not in the context, say: 'I don't have that information.' " "Do not use any knowledge outside of what's provided below.\n\n" "Context:\n{context}" , "user", "{question}" llm = ChatOpenAI model="gpt-4o", temperature=0 temp=0 for consistent, factual answers def format docs docs : Combine retrieved chunks into a single context string return "\n\n".join f" Source: {doc.metadata.get 'source', 'unknown' }, " f"Page: {doc.metadata.get 'page', '?' } \n{doc.page content}" for doc in docs Build the chain using LangChain Expression Language LCEL rag chain = { "context": retriever | format docs, retrieve → format as string "question": RunnablePassthrough pass question through unchanged } | rag prompt fill the template | llm call the model | StrOutputParser parse response to plain string Use it response = rag chain.invoke "What is the reimbursement limit for travel expenses?" print response "According to the handbook, the travel reimbursement limit is $350 per night for hotels and $75 per day for meals..." This chain is clean, auditable, and straightforward to swap out individual pieces — change the loader, the splitter settings, the retriever strategy, or the underlying LLM without touching the rest of the pipeline. RAG systems fail in consistent, diagnosable ways. Knowing the failure modes ahead of time saves days of debugging. "The model ignores the retrieved context and makes things up anyway." Your system prompt isn't firm enough. Add explicit grounding instructions: "Answer only from the context below. If the answer is not in the context, say so." Setting temperature=0 on the LLM also helps with consistency. "Retrieval returns irrelevant chunks." This is the most common failure, and it almost always traces to chunking. Chunks are too large low retrieval precision , too small not enough context per chunk , or split at bad boundaries sentences split mid-idea . Try smaller chunks, increase overlap, or switch to a structure-aware splitter markdown header, code-aware . "The right information is in the document but retrieval never finds it." Your query phrasing and your document phrasing are too semantically distant. Try the multi-query retriever to cast a wider net. Also check: was this section actually indexed? Loaders silently fail on scanned PDFs, complex tables, and certain layouts. "The model's answer is correct but I can't trace where it came from." Always include source metadata in your formatted context and instruct the model to cite it. Without attribution, RAG is a black box — with attribution, it's auditable. "It works in testing but gives bad answers on long documents at scale." Your k value number of retrieved chunks is hitting the LLM's context window limit. Either reduce k , reduce chunk size, or use a model with a larger context window. Also check for embedding model mismatch — if you indexed with one embedding model and query with another, the vector space is incompatible. You now have a mental model of the complete RAG pipeline from ingestion to response — not just what the pieces are called, but why each one exists and what breaks when it's wrong. The natural next step is building something real with this. Start with a single clean PDF and a handful of test questions whose correct answers you already know. Run the full pipeline, check whether your retriever surfaces the right chunks for each question, then tune chunk size and overlap until retrieval quality is solid. Get that working before touching retriever strategies or advanced techniques. Once retrieval is reliable, the rest — getting the model to answer well, adding conversation memory, citing sources — falls into place quickly. The retrieval quality is the foundation. Everything else is downstream from it. RAG isn't magic. It's plumbing — well-designed plumbing that lets a language model answer questions about your actual data instead of guessing. Build the plumbing carefully, and the magic takes care of itself.