Mastering Retrieval Augmented Generation (RAG): The Complete End-to-End Guide

Retrieval Augmented Generation (RAG) combines a retriever that fetches relevant information from external knowledge sources at query time with a generator LLM that uses that context to produce grounded answers, addressing LLM limitations such as outdated training data, hallucinations, and lack of access to internal documents. RAG is already in production at thousands of companies for applications like internal chatbots, customer support deflection, contract analysis, and code-aware chat, and it offers benefits including easy updates, reduced hallucinations, citations, cost efficiency, data privacy, modularity, and explainability.

Large Language Models LLMs like GPT-4, Claude, Llama 3, and Gemini are astonishing. They can write essays, generate code, translate languages, and hold coherent conversations. But if you have tried to use them in a real business context, you have probably hit the same three walls that everyone else does: Every LLM is trained on data up to a certain date. Ask GPT-4 about a product launched last week, and it either says “I don’t know” or — worse — invents an answer. This is fine for chit-chat, catastrophic for enterprise use. LLMs are next-token predictors, not truth engines. When they don’t know something, they don’t stop — they guess fluently . A hallucinated legal citation, medical dose, or financial figure isn’t just embarrassing; it’s a liability. Your internal wiki, customer support tickets, PDFs of contracts, Confluence pages, Jira comments — none of that is in the LLM. And you probably don’t want to fine-tune a model every time a document changes. RAG Retrieval Augmented Generation is the elegant, practical, and now-industry-standard answer to all three problems. Retrieval Augmented Generation RAG is a technique that combines two components: 1. A Retriever — that fetches relevant information from an external knowledge source documents, databases, APIs at query time. 2. A Generator — an LLM that uses the retrieved information as context to produce a grounded, accurate answer. Imagine a very smart student taking an open-book exam. Without RAG → the student answers from memory alone may forget, may guess . With RAG → the student first flips to the exact right page of the textbook, reads it, then answers. The student LLM is the same. The difference is having the right page open in front of them . RAG = Retrieve relevant context + Feed it to an LLM + Generate a grounded answer. That’s it. Everything else in this article is a variation, optimization, or evaluation of that flow. Let’s be concrete about what RAG fixes: RAG doesn’t make the LLM smarter. It makes the LLM better informed — and in enterprise settings, that’s usually what you actually need. Update a document → the next query uses the new version. No retraining. Answers are anchored to retrieved sources, dramatically reducing hallucinations. Every answer can be shown with its citations. Auditors love this. So do users. Fine-tuning a 70B model is expensive. Adding a PDF to a vector DB is free. Your data never leaves your infrastructure if you self-host the vector DB and the model . GDPR / HIPAA-friendly. Swap the retriever, swap the LLM, swap the embedding model — each piece is independent. You can literally show the retrieved chunks that produced the answer. RAG isn’t a lab curiosity. It’s already in production at thousands of companies. Here are the patterns you’ll see everywhere: An internal chatbot that answers: “ What’s our parental leave policy?” by searching Confluence, Google Drive, and Notion. Ingest all past tickets, product manuals, and FAQs. Deflect 40–70% of L1 tickets with grounded, cited answers. Query thousands of contracts: “ Find all agreements where the termination clause requires more than 90 days notice.” Ground answers in latest medical guidelines and journals — never in the LLM’s training data. Analyze earnings reports, SEC filings, and news to answer: “ What did the CFO say about margins last quarter?” Code-aware chat over your monorepo: “ How do we implement retries in this codebase?” Beyond keyword: “ A waterproof jacket good for hiking in cold rain under $200.” Personalized tutors grounded in the specific curriculum, not the internet. These three are often confused. Let’s untangle them. What it is: Find documents whose meaning matches the query using embeddings — not just keyword match. Output: A ranked list of documents. Analogy: A very smart Google. What it is: Semantic Search + an LLM that reads the results and writes an answer. Output: A natural language answer with optionally citations. Analogy: A smart Google that also reads the top links and summarizes them for you . What it is: Update the LLM’s weights on your domain data so the knowledge is baked in. Output: A new, specialized model. Analogy: Sending your intern to a 6-month bootcamp. Rule of thumb: Fine-tuning teaches the model new skills. RAG gives the model new knowledge. A production RAG system has two distinct phases: Let’s walk through each stage with the “why” behind the “what”. Get the raw text out of PDFs, Word docs, HTML, Markdown, databases, APIs, etc. LLMs have context limits; vector search works best on focused passages. Split the text into ~200–1000 token chunks. Convert each chunk into a high-dimensional vector using an embedding model. Similar meanings → nearby vectors. Save the vectors plus the original text and metadata in a vector database that supports fast similarity search. At query time, embed the user’s question and find the top-k most similar chunks. Stuff those chunks into a prompt: “ Answer the question using ONLY this context: {chunks}” . Send the prompt to the LLM. Return the answer plus citations . Every step has knobs. Tuning those knobs is the art of RAG. If your document is 100 pages long and you embed it as a single vector, that vector represents an average meaning — useless for precise retrieval. Chunking creates many focused vectors, each representing one idea. Too small < 100 tokens : Loses context. “The company reported a loss.” Which company? Too large 1500 tokens : Dilutes relevance. LLM may miss the key sentence. Sweet spot: ~300–800 tokens with 50–100 token overlap. python from langchain.text splitter import RecursiveCharacterTextSplittertext = """Retrieval Augmented Generation RAG combines the strengths of retrieval-basedand generation-based approaches. It first retrieves relevant documents from aknowledge base and then uses a language model to generate an answer.RAG helps reduce hallucinations because the model is grounded in real sources.It also allows for easy updates: change the documents, and the answers update."""splitter = RecursiveCharacterTextSplitter chunk size=200, chunk overlap=40, separators= "\n\n", "\n", ". ", " ", "" , chunks = splitter.split text text for i, c in enumerate chunks : print f"--- Chunk {i} ---\n{c}\n" An embedding is a list of numbers e.g., 1536 floats that represents the meaning of a piece of text. "dog" → 0.12, -0.44, 0.98, … "puppy" → 0.14, -0.41, 0.95, … very close to "dog" "submarine" → -0.71, 0.03, 0.22, … far from "dog" The magic: semantically similar texts → geometrically close vectors. python pip install openaifrom openai import OpenAIclient = OpenAI api key="YOUR KEY" def embed text: str - list float : resp = client.embeddings.create model="text-embedding-3-small", input=text, return resp.data 0 .embeddingvec = embed "What is Retrieval Augmented Generation?" print f"Dimension: {len vec }" 1536print f"First 5: {vec :5 }" python pip install sentence-transformersfrom sentence transformers import SentenceTransformermodel = SentenceTransformer "BAAI/bge-small-en-v1.5" free, local, fastvecs = model.encode "RAG stands for Retrieval Augmented Generation.", "Dogs are loyal companions.", print vecs.shape 2, 384 Popular open embedding models: BAAI/bge-large-en-v1.5 - top-tier English intfloat/e5-large-v2 - strong general-purpose sentence-transformers/all-MiniLM-L6-v2 - tiny, fast, decent nomic-ai/nomic-embed-text-v1.5 - long context 8k A vector DB stores vectors and supports Approximate Nearest Neighbor ANN search : given a query vector, find the top-k most similar stored vectors — fast, at billion-scale. Cosine similarity — most common; measures angle between vectors. Dot product — fast; good when vectors are normalized. Euclidean L2 distance — geometric distance. pip install chromadb sentence-transformersimport chromadbfrom chromadb.utils import embedding functionsclient = chromadb.PersistentClient path="./chroma store" embed fn = embedding functions.SentenceTransformerEmbeddingFunction model name="all-MiniLM-L6-v2" collection = client.get or create collection name="rag docs", embedding function=embed fn, docs = "RAG combines retrieval and generation for grounded answers.", "Vector databases store embeddings for similarity search.", "Chunking splits long documents into retrievable passages.", "LLMs like GPT-4 can hallucinate without grounding.", collection.add documents=docs, ids= f"doc {i}" for i in range len docs , metadatas= {"source": "handbook", "idx": i} for i in range len docs , results = collection.query query texts= "How do we prevent LLM hallucinations?" , n results=2, print results "documents" Retrieval is the most impactful stage of RAG. Garbage retrieval → garbage generation, no matter how good your LLM is. Given a query $q$ and a corpus of chunks C = {c 1, c 2, …, c n}, return the top-k chunks that maximize: relevance q, c i Relevance is usually measured by embedding similarity, but can also include keyword overlap, metadata filters, recency, etc. Dense retrieval: Uses embeddings. Great for semantic meaning “car” ≈ “automobile” . Sparse retrieval: Uses keyword statistics BM25, TF-IDF . Great for exact matches, product codes, names. Hybrid retrieval: Combines both. Almost always beats either alone. Vector search doesn’t have to be blind. You can pre-filter by metadata: collection.query query texts= "quarterly revenue" , n results=5, where={"year": 2025, "doc type": "earnings call"}, This is huge for multi-tenant apps, security scoping, and time-based filtering. Typical k = 3–10. Vector search at scale doesn’t do exact nearest neighbor — that’s O n per query and would take seconds for a million vectors. Instead, we use Approximate Nearest Neighbor ANN algorithms. The dominant algorithm today. Builds a multi-layer graph: Top layers are sparse long-range links . Bottom layer contains all points. Search descends from the top, greedily hopping toward the query. Result: logarithmic search time with 95% recall. Cluster vectors into buckets. At query time, search only the nearest few buckets. Compress vectors into small codes to save memory. Often combined with IVF IVF-PQ for billion-scale search. You rarely tune ANN parameters directly, but knowing they exist helps you understand: Why recall isn’t 100%. Why ef search or nprobe parameters exist. Why raising them improves quality but slows things down. The default. Embed query → find top-k nearest chunks. results = collection.query query texts= "What is RAG?" , n results=3 Balances relevance with diversity . Prevents returning 5 nearly identical chunks. python from langchain community.vectorstores import Chromafrom langchain openai import OpenAIEmbeddingsvs = Chroma persist directory="./cdb", embedding function=OpenAIEmbeddings docs = vs.max marginal relevance search "How does RAG reduce hallucinations?", k=4, final results fetch k=20, candidates to consider lambda mult=0.5, 0 = max diversity, 1 = max relevance python pip install rank bm25from rank bm25 import BM25Okapicorpus = doc.split for doc in docs bm25 = BM25Okapi corpus query = "vector database indexing".split scores = bm25.get scores query top idx = sorted range len scores , key=lambda i: -scores i :3 Combine BM25 and vector scores with a weightdef hybrid score bm25 score, vec score, alpha=0.5 : return alpha vec score + 1 - alpha bm25 score Weaviate, Qdrant, and Elasticsearch support hybrid search natively. Generate several rephrasings of the user’s query, retrieve for each, then merge. python from langchain.retrievers.multi query import MultiQueryRetrieverfrom langchain openai import ChatOpenAIllm = ChatOpenAI model="gpt-4o-mini", temperature=0 retriever = MultiQueryRetriever.from llm retriever=vs.as retriever , llm=llm, docs = retriever.invoke "How does RAG reduce hallucinations?" Embed small chunks for precise matching, but return their larger parent chunks for richer context. 1. Ask the LLM to hallucinate a fake answer to the query. 2. Embed the fake answer. 3. Use that embedding to retrieve. Surprisingly effective — the fake answer often looks more like the target chunk than the raw query. A minimal RAG prompt: You are a helpful assistant. Answer the user's question using ONLY theprovided context. If the answer is not in the context, say "I don't know."Context:{retrieved chunks}Question: {user question}Answer: Number the chunks so the LLM can cite them: 1 … 2 … Explicitly forbid hallucination : “If not in context, say you don’t know.” Ask for citations : “Cite the chunk numbers you used.” Cap total context tokens to avoid overflow. Include metadata source, date inside each chunk header. python from openai import OpenAIclient = OpenAI def build prompt question: str, chunks: list str - str: ctx = "\n\n".join f" {i+1} {c}" for i, c in enumerate chunks return f"""You are a precise assistant. Use ONLY the context below.If the answer is not present, reply "I don't know".Cite sources like 1 , 2 .Context:{ctx}Question: {question}Answer:"""def rag answer question: str, chunks: list str - str: prompt = build prompt question, chunks resp = client.chat.completions.create model="gpt-4o-mini", messages= {"role": "user", "content": prompt} , temperature=0.1, return resp.choices 0 .message.content Now let’s put it all together into a working end-to-end system. pip install langchain langchain-openai langchain-community \chromadb pypdf sentence-transformers tiktoken python import osos.environ "OPENAI API KEY" = "sk-…" python from langchain community.document loaders import PyPDFLoaderfrom langchain.text splitter import RecursiveCharacterTextSplitterfrom langchain openai import OpenAIEmbeddings, ChatOpenAIfrom langchain community.vectorstores import Chromafrom langchain.chains import RetrievalQA 1. LOADloader = PyPDFLoader "company handbook.pdf" pages = loader.load 2. CHUNKsplitter = RecursiveCharacterTextSplitter chunk size=600, chunk overlap=100 chunks = splitter.split documents pages 3. EMBED + STOREembeddings = OpenAIEmbeddings model="text-embedding-3-small" vectorstore = Chroma.from documents chunks, embeddings, persist directory="./chroma db", 4. RETRIEVE + GENERATEllm = ChatOpenAI model="gpt-4o-mini", temperature=0 qa = RetrievalQA.from chain type llm=llm, retriever=vectorstore.as retriever search kwargs={"k": 4} , return source documents=True, result = qa.invoke {"query": "What is our remote work policy?"} print "Answer:", result "result" print "\nSources:" for doc in result "source documents" : print "-", doc.metadata.get "source" , "page", doc.metadata.get "page" That’s a working RAG system in ~25 lines. Everything else is optimization. Both are Python frameworks for building LLM apps. Both do RAG well. They differ in philosophy. Strengths: Massive ecosystem, agents, tools, integrations with everything. Weakness: Can feel over-abstracted; API churn. Best for: Complex chains, agent workflows, mixed tool use. Strengths: Laser-focused on RAG. Best-in-class indexing primitives, query engines, evaluation. Weakness: Less flexible for non-RAG tasks. Best for: Data-heavy RAG apps, structured document QA. python pip install llama-indexfrom llama index.core import VectorStoreIndex, SimpleDirectoryReaderfrom llama index.llms.openai import OpenAIfrom llama index.embeddings.openai import OpenAIEmbeddingfrom llama index.core import SettingsSettings.llm = OpenAI model="gpt-4o-mini" Settings.embed model = OpenAIEmbedding model="text-embedding-3-small" documents = SimpleDirectoryReader "./data" .load data index = VectorStoreIndex.from documents documents query engine = index.as query engine similarity top k=4 resp = query engine.query "Summarize the RAG chapter." print resp Notice how much less ceremony there is — that’s LlamaIndex’s design goal. Building a RAG-first product? LlamaIndex . Building a multi-tool agent where RAG is one capability? LangChain . Doing both? Use them together — they compose fine. Real-world data is messy. Here’s how to handle it. python from langchain community.document loaders import PyPDFLoader, UnstructuredPDFLoader Simple text extractiondocs = PyPDFLoader "file.pdf" .load Better: preserves tables, structuredocs = UnstructuredPDFLoader "file.pdf", mode="elements" .load python from langchain community.document loaders import WebBaseLoaderdocs = WebBaseLoader "https://example.com/blog/post" .load python from langchain community.document loaders import CSVLoaderdocs = CSVLoader "data.csv" .load Each has a dedicated loader in langchain community.document loaders. Real production systems usually build incremental sync on top: track last-modified timestamps, re-embed only changed docs. python from langchain community.document loaders import DirectoryLoader, TextLoaderdocs = DirectoryLoader "./docs", glob=" / .md", loader cls=TextLoader .load Two patterns: 1. Text-to-SQL agent generates SQL from natural language . 2. Row-to-Document embed each row’s textual columns . Always attach metadata during loading: for doc in docs: doc.metadata.update { "source": doc.metadata.get "source", "unknown" , "team": "engineering", "ingested at": "2026-07-28", "access level": "internal", } Metadata enables filtering, security, and citations. Let’s build a slightly more serious index. python from langchain openai import OpenAIEmbeddingsfrom langchain community.vectorstores import Chromafrom langchain.text splitter import RecursiveCharacterTextSplitterfrom langchain community.document loaders import DirectoryLoader, PyPDFLoaderloader = DirectoryLoader "./corpus", glob=" / .pdf", loader cls=PyPDFLoader, show progress=True, raw docs = loader.load splitter = RecursiveCharacterTextSplitter chunk size=800, chunk overlap=120, chunks = splitter.split documents raw docs for c in chunks: c.metadata "team" = "product" c.metadata "version" = "v2"emb = OpenAIEmbeddings model="text-embedding-3-large" vs = Chroma.from documents chunks, emb, persist directory="./prod chroma" retriever = vs.as retriever search type="mmr", search kwargs={ "k": 5, "fetch k": 25, "lambda mult": 0.6, "filter": {"team": "product"}, }, docs = retriever.invoke "What changed in v2?" python from langchain.retrievers import EnsembleRetrieverfrom langchain community.retrievers import BM25Retrieverbm25 = BM25Retriever.from documents chunks bm25.k = 5dense = vs.as retriever search kwargs={"k": 5} hybrid = EnsembleRetriever retrievers= bm25, dense , weights= 0.4, 0.6 , docs = hybrid.invoke "SKU-2025-A pricing change" Hybrid typically boosts recall on queries containing exact terms IDs, names, codes . Trade-offs to consider: Dimension 384 vs 1536 vs 3072 — bigger = more storage/compute. Context length — how many tokens per chunk it can handle. Domain — general vs. code-specialized vs. multilingual. Cost — API vs. self-host. python from sentence transformers import SentenceTransformer, utilpairs = "What is RAG?", "Retrieval Augmented Generation combines retrieval and LLMs." , "What is RAG?", "The Eiffel Tower is in Paris." , for name in "all-MiniLM-L6-v2", "BAAI/bge-small-en-v1.5" : m = SentenceTransformer name for q, d in pairs: e1, e2 = m.encode q, d print f"{name:35s} sim={util.cos sim e1, e2 .item :.3f} '{d :40 }...'" Always use the same embedding model for indexing and querying. Mixing them silently destroys retrieval quality. Let’s write a clean, production-ready augmented generation function. python from openai import OpenAIfrom langchain community.vectorstores import Chromafrom langchain openai import OpenAIEmbeddingsclient = OpenAI vs = Chroma persist directory="./prod chroma", embedding function=OpenAIEmbeddings model="text-embedding-3-large" SYSTEM PROMPT = """You are a precise, honest assistant.Rules:1. Answer using ONLY the provided context.2. If the answer isn't in the context, reply exactly: "I don't know based on the provided documents."3. Cite sources inline like 1 , 2 using the chunk numbers.4. Be concise. Do not invent facts."""def format context docs - str: parts = for i, d in enumerate docs, start=1 : src = d.metadata.get "source", "unknown" page = d.metadata.get "page", "?" parts.append f" {i} source: {src}, page: {page} \n{d.page content}" return "\n\n".join parts def rag question: str, k: int = 5 - dict: docs = vs.similarity search question, k=k context = format context docs resp = client.chat.completions.create model="gpt-4o-mini", temperature=0.1, messages= {"role": "system", "content": SYSTEM PROMPT}, {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}, , return { "answer": resp.choices 0 .message.content, "sources": {"source": d.metadata.get "source" , "page": d.metadata.get "page" } for d in docs , }out = rag "What are the key benefits of RAG?" print out "answer" print "\nSources:", out "sources" This pattern — retrieve → format → prompt → generate → return with citations — is the beating heart of virtually every RAG app in production. You can’t improve what you don’t measure. RAG evaluation has two axes : 1. Retrieval quality — did we fetch the right chunks? 2. Generation quality — did the LLM produce a good answer given those chunks? Create ~50–200 golden examples: eval set = { "question": "What is our maternity leave policy?", "ground truth": "26 weeks of paid leave for birthing parents.", "relevant doc ids": "hr policy.pdf p12", "hr policy.pdf p13" , }, ... Hit Rate @ k — was any relevant doc in the top-k? MRR Mean Reciprocal Rank — how high was the first relevant doc? Recall @ k — fraction of relevant docs retrieved. Precision @ k — fraction of retrieved docs that were relevant. NDCG — quality-weighted ranking. python def hit rate retrieved ids, relevant ids : return int any r in relevant ids for r in retrieved ids def mrr retrieved ids, relevant ids : for i, r in enumerate retrieved ids, start=1 : if r in relevant ids: return 1 / i return 0.0def recall at k retrieved ids, relevant ids, k : top = set retrieved ids :k return len top & set relevant ids / max len relevant ids , 1 Generation is trickier — there’s no single “correct” answer. Common approaches: BLEU / ROUGE / METEOR — n-gram overlap. Weak for open-ended answers. BERTScore — semantic similarity via embeddings. Better. Use a strong LLM GPT-4, Claude to grade answers on: Faithfulness — is the answer supported by the context? Answer relevance — does it actually answer the question? Context relevance — was the retrieved context useful? Correctness — vs. a golden reference if you have one . python def llm judge question, answer, context, model="gpt-4o" : prompt = f"""Grade this answer on 3 axes 1-5 :- Faithfulness only uses the context - Relevance answers the question - ClarityQuestion: {question}Context: {context}Answer: {answer}Return JSON: {{"faithfulness": int, "relevance": int, "clarity": int, "reason": str}}""" resp = client.chat.completions.create model=model, messages= {"role": "user", "content": prompt} , response format={"type": "json object"}, temperature=0, return resp.choices 0 .message.content There’s no universally best retriever. Match the method to the query pattern. methods = { "dense": vs.as retriever search kwargs={"k": 5} , "mmr": vs.as retriever search type="mmr", search kwargs={"k": 5, "fetch k": 20} , "hybrid": hybrid, from earlier}for name, r in methods.items : hits, mrrs = , for row in eval set: docs = r.invoke row "question" ids = d.metadata.get "chunk id" for d in docs hits.append hit rate ids, row "relevant doc ids" mrrs.append mrr ids, row "relevant doc ids" print f"{name:8s} HitRate={sum hits /len hits :.2f} MRR={sum mrrs /len mrrs :.2f}" Iterate. Measure. Pick what wins on your data. RAGAS is an open-source library that automates RAG evaluation using LLM-as-judge with well-defined metrics. Faithfulness — is the answer grounded in the retrieved context? Answer Relevancy — does the answer address the question? Context Precision — how much of the retrieved context is relevant? Context Recall — did we retrieve everything needed? Answer Correctness — vs. a ground-truth answer. python pip install ragas datasetsfrom datasets import Datasetfrom ragas import evaluatefrom ragas.metrics import faithfulness, answer relevancy, context precision, context recall, data = { "question": "What is RAG?", "How does chunking help retrieval?", , "answer": "RAG combines retrieval with generation to produce grounded answers.", "Chunking splits documents into focused passages for precise retrieval.", , "contexts": "Retrieval Augmented Generation combines retrieval with an LLM..." , "Chunking creates smaller passages so the vector search can find precise matches..." , , "ground truth": "RAG retrieves external context and feeds it to an LLM for grounded generation.", "Chunking breaks documents into small passages so each embedding captures one idea.", ,}ds = Dataset.from dict data scores = evaluate ds, metrics= faithfulness, answer relevancy, context precision, context recall , print scores 1. Build a curated eval set of ~100 Q/A/context/ground-truth rows. 2. Run RAGAS after every change new chunker, new embedder, new prompt . 3. Track scores over time — treat regressions like test failures. Users are messy. Their queries are: Too short → pricing? Too vague → Better queries → better retrieval → better answers. It’s often the single highest-ROI improvement in a RAG system after basic hygiene. python from openai import OpenAIclient = OpenAI REWRITE PROMPT = """Rewrite the user's question to be a clear, standalone,search-friendly query. Fix spelling, expand acronyms, and add relevant terms.Return ONLY the rewritten query.Original: {q}Rewritten:"""def rewrite query: str - str: resp = client.chat.completions.create model="gpt-4o-mini", messages= {"role": "user", "content": REWRITE PROMPT.format q=query } , temperature=0, return resp.choices 0 .message.content.strip print rewrite "pricing?" → "What are the current pricing plans and subscription tiers?" EXPAND PROMPT = """Generate 3 different rephrasings of this question thatwould help retrieve relevant documents from a knowledge base.Return each on its own line.Question: {q}"""def expand query: str - list str : resp = client.chat.completions.create model="gpt-4o-mini", messages= {"role": "user", "content": EXPAND PROMPT.format q=query } , temperature=0.3, return line.strip "-• \t" for line in resp.choices 0 .message.content.splitlines if line.strip queries = expand "How do I reset my password?" Retrieve for each, merge, deduplicate. DECOMPOSE PROMPT = """Break this complex question into simpler sub-questions.Return one per line.Question: {q}"""def decompose query: str - list str : resp = client.chat.completions.create model="gpt-4o-mini", messages= {"role": "user", "content": DECOMPOSE PROMPT.format q=query } , temperature=0, return l.strip "-• \t" for l in resp.choices 0 .message.content.splitlines if l.strip print decompose "What's our refund policy and how does it compare to competitors?" → "What is our refund policy?", "What are our competitors' refund policies?", ... HYDE PROMPT = """Write a short, factual paragraph that would perfectly answerthis question, as if you had access to authoritative sources.Question: {q}Paragraph:"""def hyde retrieve query: str, k: int = 5 : resp = client.chat.completions.create model="gpt-4o-mini", messages= {"role": "user", "content": HYDE PROMPT.format q=query } , temperature=0.2, fake answer = resp.choices 0 .message.content return vs.similarity search fake answer, k=k CONTEXTUAL PROMPT = """Given the chat history, rewrite the follow-up questionto be a standalone question.Chat history:{history}Follow-up: {q}Standalone question:"""def contextualize history: str, q: str - str: resp = client.chat.completions.create model="gpt-4o-mini", messages= {"role": "user", "content": CONTEXTUAL PROMPT.format history=history, q=q } , temperature=0, return resp.choices 0 .message.content.strip Even with good retrieval, the top-k list is often noisy. Re-ranking fixes this by using a more expensive but more accurate model to re-score the candidates. Stage 1 Retriever : Fast, bi-encoder embedding search. Recall-focused. Stage 2 Re-Ranker : Slow, cross-encoder that scores query, doc pairs directly. Precision-focused. A bi-encoder embeds query and doc independently , then compares vectors. A cross-encoder feeds both together into a transformer, which can reason about their interaction — much more accurate, but too slow to run on millions of docs. Two-stage is the best of both. BAAI/bge-reranker-large — open source, strong. Cohere Rerank — hosted API, very good. ColBERT / ColBERTv2 — late-interaction, fast. LLM-as-reranker — use GPT-4 to score. Highest quality, highest cost. python pip install sentence-transformersfrom sentence transformers import CrossEncoderreranker = CrossEncoder "BAAI/bge-reranker-base" def rerank query: str, docs, top n: int = 5 : pairs = query, d.page content for d in docs scores = reranker.predict pairs ranked = sorted zip docs, scores , key=lambda x: -x 1 return d for d, s in ranked :top n Two-stage retrievalcandidates = vs.similarity search "What is RAG?", k=30 top docs = rerank "What is RAG?", candidates, top n=5 pip install cohereimport cohereco = cohere.Client "YOUR COHERE KEY" def cohere rerank query, docs, top n=5 : texts = d.page content for d in docs resp = co.rerank model="rerank-english-v3.0", query=query, documents=texts, top n=top n return docs r.index for r in resp.results RERANK PROMPT = """Given a query and a passage, rate how relevant the passageis to answering the query on a scale from 0 to 10. Return only the number.Query: {q}Passage: {p}Score:"""def llm score query, passage : resp = client.chat.completions.create model="gpt-4o-mini", messages= {"role": "user", "content": RERANK PROMPT.format q=query, p=passage } , temperature=0, try: return float resp.choices 0 .message.content.strip except ValueError: return 0.0def llm rerank query, docs, top n=5 : scored = d, llm score query, d.page content for d in docs scored.sort key=lambda x: -x 1 return d for d, s in scored :top n php def advanced rag user query: str, history: str = "" - dict: 1. Rewrite / contextualize query = contextualize history, user query if history else rewrite user query 2. Multi-query expansion expanded = query + expand query 3. Retrieve broadly candidates = seen = set for q in expanded: for d in vs.similarity search q, k=15 : key = d.page content :100 if key not in seen: candidates.append d seen.add key 4. Re-rank top docs = rerank query, candidates, top n=5 5. Generate context = format context top docs resp = client.chat.completions.create model="gpt-4o-mini", temperature=0.1, messages= {"role": "system", "content": SYSTEM PROMPT}, {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {user query}"}, , return { "answer": resp.choices 0 .message.content, "rewritten query": query, "sources": d.metadata for d in top docs , } This one function embodies most of what a serious RAG system does at query time. RAG is not just a technique — it’s the bridge that turns generic LLMs into trustworthy, up-to-date, domain-specific assistants. Master it, and you master the single most valuable pattern in applied AI today. Mastering Retrieval Augmented Generation RAG : The Complete End-to-End Guide https://pub.towardsai.net/mastering-retrieval-augmented-generation-rag-the-complete-end-to-end-guide-754248a787eb was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.