{"slug": "mastering-retrieval-augmented-generation-rag-the-complete-end-to-end-guide", "title": "Mastering Retrieval Augmented Generation (RAG): The Complete End-to-End Guide", "summary": "Retrieval Augmented Generation (RAG) combines a retriever that fetches relevant information from external knowledge sources at query time with a generator LLM that uses that context to produce grounded answers, addressing LLM limitations such as outdated training data, hallucinations, and lack of access to internal documents. RAG is already in production at thousands of companies for applications like internal chatbots, customer support deflection, contract analysis, and code-aware chat, and it offers benefits including easy updates, reduced hallucinations, citations, cost efficiency, data privacy, modularity, and explainability.", "body_md": "Large Language Models (LLMs) like GPT-4, Claude, Llama 3, and Gemini are astonishing. They can write essays, generate code, translate languages, and hold coherent conversations. But if you have tried to use them in a real business context, you have probably hit the same three walls that everyone else does:\n\nEvery LLM is trained on data up to a certain date. Ask GPT-4 about a product launched last week, and it either says “I don’t know” or — worse — invents an answer. This is fine for chit-chat, catastrophic for enterprise use.\n\nLLMs are next-token predictors, not truth engines. When they don’t know something, they don’t stop — they *guess fluently*. A hallucinated legal citation, medical dose, or financial figure isn’t just embarrassing; it’s a liability.\n\nYour internal wiki, customer support tickets, PDFs of contracts, Confluence pages, Jira comments — none of that is in the LLM. And you probably don’t want to fine-tune a model every time a document changes.\n\n**RAG (Retrieval Augmented Generation)** is the elegant, practical, and now-industry-standard answer to all three problems.\n\n**Retrieval Augmented Generation (RAG)** is a technique that combines two components:\n\n1. **A Retriever** — that fetches relevant information from an external knowledge source (documents, databases, APIs) at query time.\n\n2. **A Generator** — an LLM that uses the retrieved information as context to produce a grounded, accurate answer.\n\nImagine a very smart student taking an open-book exam.**Without RAG** → the student answers from memory alone (may forget, may guess).**With RAG **→ the student first flips to the *exact right page* of the textbook, reads it, then answers.\n\nThe student (LLM) is the same. The difference is **having the right page open in front of them**.\n\nRAG = Retrieve relevant context + Feed it to an LLM + Generate a grounded answer.\n\nThat’s it. Everything else in this article is a variation, optimization, or evaluation of that flow.\n\nLet’s be concrete about what RAG fixes:\n\nRAG doesn’t make the LLM smarter. It makes the LLM **better informed** — and in enterprise settings, that’s usually what you actually need.\n\nUpdate a document → the next query uses the new version. No retraining.\n\nAnswers are anchored to retrieved sources, dramatically reducing hallucinations.\n\nEvery answer can be shown with its citations. Auditors love this. So do users.\n\nFine-tuning a 70B model is expensive. Adding a PDF to a vector DB is free.\n\nYour data never leaves your infrastructure (if you self-host the vector DB and the model). GDPR / HIPAA-friendly.\n\nSwap the retriever, swap the LLM, swap the embedding model — each piece is independent.\n\nYou can literally show the retrieved chunks that produced the answer.\n\nRAG isn’t a lab curiosity. It’s already in production at thousands of companies. Here are the patterns you’ll see everywhere:\n\nAn internal chatbot that answers: “*What’s our parental leave policy?”* by searching Confluence, Google Drive, and Notion.\n\nIngest all past tickets, product manuals, and FAQs. Deflect 40–70% of L1 tickets with grounded, cited answers.\n\nQuery thousands of contracts: “*Find all agreements where the termination clause requires more than 90 days notice.”*\n\nGround answers in latest medical guidelines and journals — never in the LLM’s training data.\n\nAnalyze earnings reports, SEC filings, and news to answer: “*What did the CFO say about margins last quarter?”*\n\nCode-aware chat over your monorepo: “*How do we implement retries in this codebase?”*\n\nBeyond keyword: “*A waterproof jacket good for hiking in cold rain under $200.”*\n\nPersonalized tutors grounded in the specific curriculum, not the internet.\n\nThese three are often confused. Let’s untangle them.\n\n**What it is:** Find documents whose *meaning* matches the query (using embeddings) — not just keyword match.**Output:** A ranked list of documents.**Analogy:** A very smart Google.\n\n**What it is:** Semantic Search **+** an LLM that reads the results and writes an answer.**Output:** A natural language answer with (optionally) citations.**Analogy:** A smart Google that also *reads the top links and summarizes them for you*.\n\n**What it is:** Update the LLM’s *weights* on your domain data so the knowledge is baked in.**Output:** A new, specialized model.**Analogy:** Sending your intern to a 6-month bootcamp.\n\n**Rule of thumb:** *Fine-tuning teaches the model new skills. RAG gives the model new knowledge.*\n\nA production RAG system has two distinct phases:\n\nLet’s walk through each stage with the “why” behind the “what”.\n\nGet the raw text out of PDFs, Word docs, HTML, Markdown, databases, APIs, etc.\n\nLLMs have context limits; vector search works best on focused passages. Split the text into ~200–1000 token chunks.\n\nConvert each chunk into a high-dimensional vector using an embedding model. Similar meanings → nearby vectors.\n\nSave the vectors (plus the original text and metadata) in a vector database that supports fast similarity search.\n\nAt query time, embed the user’s question and find the top-k most similar chunks.\n\nStuff those chunks into a prompt: “*Answer the question using ONLY this context: {chunks}”*.\n\nSend the prompt to the LLM. Return the answer (plus citations).\n\nEvery step has knobs. Tuning those knobs is the art of RAG.\n\nIf your document is 100 pages long and you embed it as a single vector, that vector represents an *average* meaning — useless for precise retrieval. Chunking creates *many* focused vectors, each representing one idea.\n\n**Too small (< 100 tokens):** Loses context. “The company reported a loss.” Which company?**Too large (> 1500 tokens):** Dilutes relevance. LLM may miss the key sentence.**Sweet spot:** ~300–800 tokens with 50–100 token overlap.\n\n``` python\nfrom langchain.text_splitter import RecursiveCharacterTextSplittertext = \"\"\"Retrieval Augmented Generation (RAG) combines the strengths of retrieval-basedand generation-based approaches. It first retrieves relevant documents from aknowledge base and then uses a language model to generate an answer.RAG helps reduce hallucinations because the model is grounded in real sources.It also allows for easy updates: change the documents, and the answers update.\"\"\"splitter = RecursiveCharacterTextSplitter(    chunk_size=200,    chunk_overlap=40,    separators=[\"\\n\\n\", \"\\n\", \". \", \" \", \"\"],)chunks = splitter.split_text(text)for i, c in enumerate(chunks):    print(f\"--- Chunk {i} ---\\n{c}\\n\")\n```\n\nAn embedding is a list of numbers (e.g., 1536 floats) that represents the *meaning* of a piece of text.\n\n```\n\"dog\" → [0.12, -0.44, 0.98, …]\"puppy\" → [0.14, -0.41, 0.95, …] (very close to \"dog\")\"submarine\" → [-0.71, 0.03, 0.22, …] (far from \"dog\")\n```\n\nThe magic: **semantically similar texts → geometrically close vectors.**\n\n``` python\n# pip install openaifrom openai import OpenAIclient = OpenAI(api_key=\"YOUR_KEY\")def embed(text: str) -> list[float]:    resp = client.embeddings.create(        model=\"text-embedding-3-small\",        input=text,    )    return resp.data[0].embeddingvec = embed(\"What is Retrieval Augmented Generation?\")print(f\"Dimension: {len(vec)}\")     # 1536print(f\"First 5:   {vec[:5]}\")\npython\n# pip install sentence-transformersfrom sentence_transformers import SentenceTransformermodel = SentenceTransformer(\"BAAI/bge-small-en-v1.5\")   # free, local, fastvecs = model.encode([    \"RAG stands for Retrieval Augmented Generation.\",    \"Dogs are loyal companions.\",])print(vecs.shape)   # (2, 384)\n```\n\nPopular open embedding models:\n\n```\n`BAAI/bge-large-en-v1.5` - top-tier English`intfloat/e5-large-v2` - strong general-purpose`sentence-transformers/all-MiniLM-L6-v2` - tiny, fast, decent`nomic-ai/nomic-embed-text-v1.5` - long context (8k)\n```\n\nA vector DB stores vectors and supports **Approximate Nearest Neighbor (ANN) search**: given a query vector, find the top-k most similar stored vectors — fast, at billion-scale.\n\n**Cosine similarity** — most common; measures angle between vectors.**Dot product** — fast; good when vectors are normalized.**Euclidean (L2) distance** — geometric distance.\n\n```\n# pip install chromadb sentence-transformersimport chromadbfrom chromadb.utils import embedding_functionsclient = chromadb.PersistentClient(path=\"./chroma_store\")embed_fn = embedding_functions.SentenceTransformerEmbeddingFunction(    model_name=\"all-MiniLM-L6-v2\")collection = client.get_or_create_collection(    name=\"rag_docs\",    embedding_function=embed_fn,)docs = [    \"RAG combines retrieval and generation for grounded answers.\",    \"Vector databases store embeddings for similarity search.\",    \"Chunking splits long documents into retrievable passages.\",    \"LLMs like GPT-4 can hallucinate without grounding.\",]collection.add(    documents=docs,    ids=[f\"doc_{i}\" for i in range(len(docs))],    metadatas=[{\"source\": \"handbook\", \"idx\": i} for i in range(len(docs))],)results = collection.query(    query_texts=[\"How do we prevent LLM hallucinations?\"],    n_results=2,)print(results[\"documents\"])\n```\n\nRetrieval is the *most impactful* stage of RAG. Garbage retrieval → garbage generation, no matter how good your LLM is.\n\nGiven a query $q$ and a corpus of chunks C = {c_1, c_2, …, c_n}, return the top-k chunks that maximize: relevance(q, c_i)\n\n*Relevance* is usually measured by embedding similarity, but can also include keyword overlap, metadata filters, recency, etc.\n\n**Dense retrieval:** Uses embeddings. Great for semantic meaning (“car” ≈ “automobile”).**Sparse retrieval:** Uses keyword statistics (BM25, TF-IDF). Great for exact matches, product codes, names.**Hybrid retrieval:** Combines both. Almost always beats either alone.\n\nVector search doesn’t have to be blind. You can pre-filter by metadata:\n\n```\ncollection.query(    query_texts=[\"quarterly revenue\"],    n_results=5,    where={\"year\": 2025, \"doc_type\": \"earnings_call\"},)\n```\n\nThis is huge for multi-tenant apps, security scoping, and time-based filtering.\n\nTypical k = 3–10.\n\nVector search at scale doesn’t do exact nearest neighbor — that’s O(n) per query and would take seconds for a million vectors. Instead, we use **Approximate Nearest Neighbor (ANN)** algorithms.\n\nThe dominant algorithm today. Builds a multi-layer graph:\n\nTop layers are sparse (long-range links).\n\nBottom layer contains all points.\n\nSearch descends from the top, greedily hopping toward the query.\n\nResult: **logarithmic search time** with >95% recall.\n\nCluster vectors into buckets. At query time, search only the nearest few buckets.\n\nCompress vectors into small codes to save memory. Often combined with IVF (IVF-PQ) for billion-scale search.\n\nYou rarely tune ANN parameters directly, but knowing they exist helps you understand:\n\nWhy recall isn’t 100%.\n\nWhy ef_search or nprobe parameters exist.\n\nWhy raising them improves quality but slows things down.\n\nThe default. Embed query → find top-k nearest chunks.\n\n```\nresults = collection.query(query_texts=[\"What is RAG?\"], n_results=3)\n```\n\nBalances **relevance** with **diversity**. Prevents returning 5 nearly identical chunks.\n\n``` python\nfrom langchain_community.vectorstores import Chromafrom langchain_openai import OpenAIEmbeddingsvs = Chroma(persist_directory=\"./cdb\", embedding_function=OpenAIEmbeddings())docs = vs.max_marginal_relevance_search(    \"How does RAG reduce hallucinations?\",    k=4,        # final results    fetch_k=20, # candidates to consider    lambda_mult=0.5,  # 0 = max diversity, 1 = max relevance)\npython\n# pip install rank_bm25from rank_bm25 import BM25Okapicorpus = [doc.split() for doc in docs]bm25 = BM25Okapi(corpus)query = \"vector database indexing\".split()scores = bm25.get_scores(query)top_idx = sorted(range(len(scores)), key=lambda i: -scores[i])[:3]\n# Combine BM25 and vector scores with a weightdef hybrid_score(bm25_score, vec_score, alpha=0.5):    return alpha * vec_score + (1 - alpha) * bm25_score\n```\n\nWeaviate, Qdrant, and Elasticsearch support hybrid search natively.\n\nGenerate several rephrasings of the user’s query, retrieve for each, then merge.\n\n``` python\nfrom langchain.retrievers.multi_query import MultiQueryRetrieverfrom langchain_openai import ChatOpenAIllm = ChatOpenAI(model=\"gpt-4o-mini\", temperature=0)retriever = MultiQueryRetriever.from_llm(    retriever=vs.as_retriever(),    llm=llm,)docs = retriever.invoke(\"How does RAG reduce hallucinations?\")\n```\n\nEmbed *small* chunks for precise matching, but return their *larger parent chunks* for richer context.\n\n1. Ask the LLM to *hallucinate* a fake answer to the query.\n\n2. Embed the fake answer.\n\n3. Use *that* embedding to retrieve.\n\nSurprisingly effective — the fake answer often looks more like the target chunk than the raw query.\n\nA minimal RAG prompt:\n\n```\nYou are a helpful assistant. Answer the user's question using ONLY theprovided context. If the answer is not in the context, say \"I don't know.\"Context:{retrieved_chunks}Question: {user_question}Answer:\n```\n\n**Number the chunks** so the LLM can cite them: [1] … [2] …**Explicitly forbid hallucination**: “If not in context, say you don’t know.”** Ask for citations**: “Cite the chunk numbers you used.”** Cap total context tokens** to avoid overflow.**Include metadata** (source, date) inside each chunk header.\n\n``` python\nfrom openai import OpenAIclient = OpenAI()def build_prompt(question: str, chunks: list[str]) -> str:    ctx = \"\\n\\n\".join(f\"[{i+1}] {c}\" for i, c in enumerate(chunks))    return f\"\"\"You are a precise assistant. Use ONLY the context below.If the answer is not present, reply \"I don't know\".Cite sources like [1], [2].Context:{ctx}Question: {question}Answer:\"\"\"def rag_answer(question: str, chunks: list[str]) -> str:    prompt = build_prompt(question, chunks)    resp = client.chat.completions.create(        model=\"gpt-4o-mini\",        messages=[{\"role\": \"user\", \"content\": prompt}],        temperature=0.1,    )    return resp.choices[0].message.content\n```\n\nNow let’s put it all together into a working end-to-end system.\n\n```\npip install langchain langchain-openai langchain-community \\chromadb pypdf sentence-transformers tiktoken\npython\nimport osos.environ[\"OPENAI_API_KEY\"] = \"sk-…\"\npython\nfrom langchain_community.document_loaders import PyPDFLoaderfrom langchain.text_splitter import RecursiveCharacterTextSplitterfrom langchain_openai import OpenAIEmbeddings, ChatOpenAIfrom langchain_community.vectorstores import Chromafrom langchain.chains import RetrievalQA# 1. LOADloader = PyPDFLoader(\"company_handbook.pdf\")pages = loader.load()# 2. CHUNKsplitter = RecursiveCharacterTextSplitter(chunk_size=600, chunk_overlap=100)chunks = splitter.split_documents(pages)# 3. EMBED + STOREembeddings = OpenAIEmbeddings(model=\"text-embedding-3-small\")vectorstore = Chroma.from_documents(    chunks,    embeddings,    persist_directory=\"./chroma_db\",)# 4. RETRIEVE + GENERATEllm = ChatOpenAI(model=\"gpt-4o-mini\", temperature=0)qa = RetrievalQA.from_chain_type(    llm=llm,    retriever=vectorstore.as_retriever(search_kwargs={\"k\": 4}),    return_source_documents=True,)result = qa.invoke({\"query\": \"What is our remote work policy?\"})print(\"Answer:\", result[\"result\"])print(\"\\nSources:\")for doc in result[\"source_documents\"]:    print(\"-\", doc.metadata.get(\"source\"), \"page\", doc.metadata.get(\"page\"))\n```\n\nThat’s a working RAG system in ~25 lines. Everything else is optimization.\n\nBoth are Python frameworks for building LLM apps. Both do RAG well. They differ in philosophy.\n\n**Strengths:** Massive ecosystem, agents, tools, integrations with everything.**Weakness:** Can feel over-abstracted; API churn.**Best for:** Complex chains, agent workflows, mixed tool use.\n\n**Strengths:** Laser-focused on RAG. Best-in-class indexing primitives, query engines, evaluation.**Weakness:** Less flexible for non-RAG tasks.**Best for:** Data-heavy RAG apps, structured document QA.\n\n``` python\n# pip install llama-indexfrom llama_index.core import VectorStoreIndex, SimpleDirectoryReaderfrom llama_index.llms.openai import OpenAIfrom llama_index.embeddings.openai import OpenAIEmbeddingfrom llama_index.core import SettingsSettings.llm = OpenAI(model=\"gpt-4o-mini\")Settings.embed_model = OpenAIEmbedding(model=\"text-embedding-3-small\")documents = SimpleDirectoryReader(\"./data\").load_data()index = VectorStoreIndex.from_documents(documents)query_engine = index.as_query_engine(similarity_top_k=4)resp = query_engine.query(\"Summarize the RAG chapter.\")print(resp)\n```\n\nNotice how much less ceremony there is — that’s LlamaIndex’s design goal.\n\nBuilding a **RAG-first** product? **LlamaIndex**.\n\nBuilding a **multi-tool agent** where RAG is one capability? **LangChain**.\n\nDoing both? Use them together — they compose fine.\n\nReal-world data is messy. Here’s how to handle it.\n\n``` python\nfrom langchain_community.document_loaders import PyPDFLoader, UnstructuredPDFLoader# Simple text extractiondocs = PyPDFLoader(\"file.pdf\").load()# Better: preserves tables, structuredocs = UnstructuredPDFLoader(\"file.pdf\", mode=\"elements\").load()\npython\nfrom langchain_community.document_loaders import WebBaseLoaderdocs = WebBaseLoader([\"https://example.com/blog/post\"]).load()\npython\nfrom langchain_community.document_loaders import CSVLoaderdocs = CSVLoader(\"data.csv\").load()\n```\n\nEach has a dedicated loader in langchain_community.document_loaders. Real production systems usually build **incremental sync** on top: track last-modified timestamps, re-embed only changed docs.\n\n``` python\nfrom langchain_community.document_loaders import DirectoryLoader, TextLoaderdocs = DirectoryLoader(\"./docs\", glob=\"**/*.md\", loader_cls=TextLoader).load()\n```\n\nTwo patterns:\n\n1. **Text-to-SQL** (agent generates SQL from natural language).\n\n2. **Row-to-Document** (embed each row’s textual columns).\n\nAlways attach metadata during loading:\n\n```\nfor doc in docs:    doc.metadata.update({        \"source\": doc.metadata.get(\"source\", \"unknown\"),        \"team\": \"engineering\",        \"ingested_at\": \"2026-07-28\",        \"access_level\": \"internal\",    })\n```\n\nMetadata enables filtering, security, and citations.\n\nLet’s build a slightly more serious index.\n\n``` python\nfrom langchain_openai import OpenAIEmbeddingsfrom langchain_community.vectorstores import Chromafrom langchain.text_splitter import RecursiveCharacterTextSplitterfrom langchain_community.document_loaders import DirectoryLoader, PyPDFLoaderloader = DirectoryLoader(    \"./corpus\",    glob=\"**/*.pdf\",    loader_cls=PyPDFLoader,    show_progress=True,)raw_docs = loader.load()splitter = RecursiveCharacterTextSplitter(    chunk_size=800,    chunk_overlap=120,)chunks = splitter.split_documents(raw_docs)for c in chunks:    c.metadata[\"team\"] = \"product\"    c.metadata[\"version\"] = \"v2\"emb = OpenAIEmbeddings(model=\"text-embedding-3-large\")vs = Chroma.from_documents(    chunks, emb, persist_directory=\"./prod_chroma\")\nretriever = vs.as_retriever(    search_type=\"mmr\",    search_kwargs={        \"k\": 5,        \"fetch_k\": 25,        \"lambda_mult\": 0.6,        \"filter\": {\"team\": \"product\"},    },)docs = retriever.invoke(\"What changed in v2?\")\npython\nfrom langchain.retrievers import EnsembleRetrieverfrom langchain_community.retrievers import BM25Retrieverbm25 = BM25Retriever.from_documents(chunks)bm25.k = 5dense = vs.as_retriever(search_kwargs={\"k\": 5})hybrid = EnsembleRetriever(    retrievers=[bm25, dense],    weights=[0.4, 0.6],)docs = hybrid.invoke(\"SKU-2025-A pricing change\")\n```\n\nHybrid typically boosts recall on queries containing exact terms (IDs, names, codes).\n\nTrade-offs to consider:**Dimension** (384 vs 1536 vs 3072) — bigger = more storage/compute.**Context length** — how many tokens per chunk it can handle.**Domain **— general vs. code-specialized vs. multilingual.** Cost** — API vs. self-host.\n\n``` python\nfrom sentence_transformers import SentenceTransformer, utilpairs = [    (\"What is RAG?\", \"Retrieval Augmented Generation combines retrieval and LLMs.\"),    (\"What is RAG?\", \"The Eiffel Tower is in Paris.\"),]for name in [\"all-MiniLM-L6-v2\", \"BAAI/bge-small-en-v1.5\"]:    m = SentenceTransformer(name)    for q, d in pairs:        e1, e2 = m.encode([q, d])        print(f\"{name:35s} sim={util.cos_sim(e1, e2).item():.3f}  '{d[:40]}...'\")\n```\n\n**Always use the same embedding model** for indexing and querying. Mixing them silently destroys retrieval quality.\n\nLet’s write a clean, production-ready augmented generation function.\n\n``` python\nfrom openai import OpenAIfrom langchain_community.vectorstores import Chromafrom langchain_openai import OpenAIEmbeddingsclient = OpenAI()vs = Chroma(persist_directory=\"./prod_chroma\",            embedding_function=OpenAIEmbeddings(model=\"text-embedding-3-large\"))SYSTEM_PROMPT = \"\"\"You are a precise, honest assistant.Rules:1. Answer using ONLY the provided context.2. If the answer isn't in the context, reply exactly: \"I don't know based on the provided documents.\"3. Cite sources inline like [1], [2] using the chunk numbers.4. Be concise. Do not invent facts.\"\"\"def format_context(docs) -> str:    parts = []    for i, d in enumerate(docs, start=1):        src = d.metadata.get(\"source\", \"unknown\")        page = d.metadata.get(\"page\", \"?\")        parts.append(f\"[{i}] (source: {src}, page: {page})\\n{d.page_content}\")    return \"\\n\\n\".join(parts)def rag(question: str, k: int = 5) -> dict:    docs = vs.similarity_search(question, k=k)    context = format_context(docs)    resp = client.chat.completions.create(        model=\"gpt-4o-mini\",        temperature=0.1,        messages=[            {\"role\": \"system\", \"content\": SYSTEM_PROMPT},            {\"role\": \"user\",             \"content\": f\"Context:\\n{context}\\n\\nQuestion: {question}\"},        ],    )    return {        \"answer\": resp.choices[0].message.content,        \"sources\": [            {\"source\": d.metadata.get(\"source\"),             \"page\": d.metadata.get(\"page\")}            for d in docs        ],    }out = rag(\"What are the key benefits of RAG?\")print(out[\"answer\"])print(\"\\nSources:\", out[\"sources\"])\n```\n\nThis pattern — retrieve → format → prompt → generate → return with citations — is the beating heart of virtually every RAG app in production.\n\nYou can’t improve what you don’t measure. RAG evaluation has **two axes**:\n\n1. **Retrieval quality** — did we fetch the right chunks?\n\n2. **Generation quality** — did the LLM produce a good answer given those chunks?\n\nCreate ~50–200 golden examples:\n\n```\neval_set = [    {        \"question\": \"What is our maternity leave policy?\",        \"ground_truth\": \"26 weeks of paid leave for birthing parents.\",        \"relevant_doc_ids\": [\"hr_policy.pdf#p12\", \"hr_policy.pdf#p13\"],    },    # ...]\n```\n\n**Hit Rate @ k** — was any relevant doc in the top-k?**MRR (Mean Reciprocal Rank)** — how high was the first relevant doc?**Recall @ k** — fraction of relevant docs retrieved.**Precision @ k** — fraction of retrieved docs that were relevant.**NDCG** — quality-weighted ranking.\n\n``` python\ndef hit_rate(retrieved_ids, relevant_ids):    return int(any(r in relevant_ids for r in retrieved_ids))def mrr(retrieved_ids, relevant_ids):    for i, r in enumerate(retrieved_ids, start=1):        if r in relevant_ids:            return 1 / i    return 0.0def recall_at_k(retrieved_ids, relevant_ids, k):    top = set(retrieved_ids[:k])    return len(top & set(relevant_ids)) / max(len(relevant_ids), 1)\n```\n\nGeneration is trickier — there’s no single “correct” answer. Common approaches:\n\n**BLEU / ROUGE / METEOR** — n-gram overlap. Weak for open-ended answers.**BERTScore** — semantic similarity via embeddings. Better.\n\nUse a strong LLM (GPT-4, Claude) to grade answers on:**Faithfulness** — is the answer supported by the context?**Answer relevance** — does it actually answer the question?**Context relevance** — was the retrieved context useful?**Correctness** — vs. a golden reference (if you have one).\n\n``` python\ndef llm_judge(question, answer, context, model=\"gpt-4o\"):    prompt = f\"\"\"Grade this answer on 3 axes (1-5):- Faithfulness (only uses the context)- Relevance (answers the question)- ClarityQuestion: {question}Context: {context}Answer: {answer}Return JSON: {{\"faithfulness\": int, \"relevance\": int, \"clarity\": int, \"reason\": str}}\"\"\"    resp = client.chat.completions.create(        model=model,        messages=[{\"role\": \"user\", \"content\": prompt}],        response_format={\"type\": \"json_object\"},        temperature=0,    )    return resp.choices[0].message.content\n```\n\nThere’s no universally best retriever. Match the method to the query pattern.\n\n```\nmethods = {    \"dense\": vs.as_retriever(search_kwargs={\"k\": 5}),    \"mmr\":   vs.as_retriever(search_type=\"mmr\", search_kwargs={\"k\": 5, \"fetch_k\": 20}),    \"hybrid\": hybrid,   # from earlier}for name, r in methods.items():    hits, mrrs = [], []    for row in eval_set:        docs = r.invoke(row[\"question\"])        ids = [d.metadata.get(\"chunk_id\") for d in docs]        hits.append(hit_rate(ids, row[\"relevant_doc_ids\"]))        mrrs.append(mrr(ids, row[\"relevant_doc_ids\"]))    print(f\"{name:8s}  HitRate={sum(hits)/len(hits):.2f}  MRR={sum(mrrs)/len(mrrs):.2f}\")\n```\n\nIterate. Measure. Pick what wins on *your* data.\n\n**RAGAS** is an open-source library that automates RAG evaluation using LLM-as-judge with well-defined metrics.\n\n**Faithfulness** — is the answer grounded in the retrieved context?**Answer Relevancy** — does the answer address the question?**Context Precision** — how much of the retrieved context is relevant?**Context Recall** — did we retrieve everything needed?**Answer Correctness** — vs. a ground-truth answer.\n\n``` python\n# pip install ragas datasetsfrom datasets import Datasetfrom ragas import evaluatefrom ragas.metrics import (    faithfulness,    answer_relevancy,    context_precision,    context_recall,)data = {    \"question\": [        \"What is RAG?\",        \"How does chunking help retrieval?\",    ],    \"answer\": [        \"RAG combines retrieval with generation to produce grounded answers.\",        \"Chunking splits documents into focused passages for precise retrieval.\",    ],    \"contexts\": [        [\"Retrieval Augmented Generation combines retrieval with an LLM...\"],        [\"Chunking creates smaller passages so the vector search can find precise matches...\"],    ],    \"ground_truth\": [        \"RAG retrieves external context and feeds it to an LLM for grounded generation.\",        \"Chunking breaks documents into small passages so each embedding captures one idea.\",    ],}ds = Dataset.from_dict(data)scores = evaluate(    ds,    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],)print(scores)\n```\n\n1. Build a curated eval set of ~100 Q/A/context/ground-truth rows.\n\n2. Run RAGAS after **every** change (new chunker, new embedder, new prompt).\n\n3. Track scores over time — treat regressions like test failures.\n\nUsers are messy. Their queries are:\n\nToo short → *pricing?*Too vague →\n\nBetter queries → better retrieval → better answers. It’s often the single highest-ROI improvement in a RAG system after basic hygiene.\n\n``` python\nfrom openai import OpenAIclient = OpenAI()REWRITE_PROMPT = \"\"\"Rewrite the user's question to be a clear, standalone,search-friendly query. Fix spelling, expand acronyms, and add relevant terms.Return ONLY the rewritten query.Original: {q}Rewritten:\"\"\"def rewrite(query: str) -> str:    resp = client.chat.completions.create(        model=\"gpt-4o-mini\",        messages=[{\"role\": \"user\", \"content\": REWRITE_PROMPT.format(q=query)}],        temperature=0,    )    return resp.choices[0].message.content.strip()print(rewrite(\"pricing?\"))# → \"What are the current pricing plans and subscription tiers?\"\nEXPAND_PROMPT = \"\"\"Generate 3 different rephrasings of this question thatwould help retrieve relevant documents from a knowledge base.Return each on its own line.Question: {q}\"\"\"def expand(query: str) -> list[str]:    resp = client.chat.completions.create(        model=\"gpt-4o-mini\",        messages=[{\"role\": \"user\", \"content\": EXPAND_PROMPT.format(q=query)}],        temperature=0.3,    )    return [line.strip(\"-•* \\t\") for line in resp.choices[0].message.content.splitlines() if line.strip()]queries = expand(\"How do I reset my password?\")\n```\n\nRetrieve for each, merge, deduplicate.\n\n```\nDECOMPOSE_PROMPT = \"\"\"Break this complex question into simpler sub-questions.Return one per line.Question: {q}\"\"\"def decompose(query: str) -> list[str]:    resp = client.chat.completions.create(        model=\"gpt-4o-mini\",        messages=[{\"role\": \"user\", \"content\": DECOMPOSE_PROMPT.format(q=query)}],        temperature=0,    )    return [l.strip(\"-•* \\t\") for l in resp.choices[0].message.content.splitlines() if l.strip()]print(decompose(\"What's our refund policy and how does it compare to competitors?\"))# → [\"What is our refund policy?\", \"What are our competitors' refund policies?\", ...]\nHYDE_PROMPT = \"\"\"Write a short, factual paragraph that would perfectly answerthis question, as if you had access to authoritative sources.Question: {q}Paragraph:\"\"\"def hyde_retrieve(query: str, k: int = 5):    resp = client.chat.completions.create(        model=\"gpt-4o-mini\",        messages=[{\"role\": \"user\", \"content\": HYDE_PROMPT.format(q=query)}],        temperature=0.2,    )    fake_answer = resp.choices[0].message.content    return vs.similarity_search(fake_answer, k=k)\nCONTEXTUAL_PROMPT = \"\"\"Given the chat history, rewrite the follow-up questionto be a standalone question.Chat history:{history}Follow-up: {q}Standalone question:\"\"\"def contextualize(history: str, q: str) -> str:    resp = client.chat.completions.create(        model=\"gpt-4o-mini\",        messages=[{\"role\": \"user\",                   \"content\": CONTEXTUAL_PROMPT.format(history=history, q=q)}],        temperature=0,    )    return resp.choices[0].message.content.strip()\n```\n\nEven with good retrieval, the top-k list is often noisy. Re-ranking fixes this by using a **more expensive but more accurate** model to re-score the candidates.\n\n**Stage 1 (Retriever):** Fast, bi-encoder embedding search. Recall-focused.**Stage 2 (Re-Ranker):** Slow, cross-encoder that scores (query, doc) pairs directly. Precision-focused.\n\nA bi-encoder embeds query and doc *independently*, then compares vectors. A cross-encoder feeds *both together* into a transformer, which can reason about their interaction — much more accurate, but too slow to run on millions of docs. Two-stage is the best of both.\n\n**BAAI/bge-reranker-large** — open source, strong.**Cohere Rerank** — hosted API, very good.**ColBERT / ColBERTv2** — late-interaction, fast.**LLM-as-reranker** — use GPT-4 to score. Highest quality, highest cost.\n\n``` python\n# pip install sentence-transformersfrom sentence_transformers import CrossEncoderreranker = CrossEncoder(\"BAAI/bge-reranker-base\")def rerank(query: str, docs, top_n: int = 5):    pairs = [(query, d.page_content) for d in docs]    scores = reranker.predict(pairs)    ranked = sorted(zip(docs, scores), key=lambda x: -x[1])    return [d for d, s in ranked[:top_n]]# Two-stage retrievalcandidates = vs.similarity_search(\"What is RAG?\", k=30)top_docs = rerank(\"What is RAG?\", candidates, top_n=5)\n# pip install cohereimport cohereco = cohere.Client(\"YOUR_COHERE_KEY\")def cohere_rerank(query, docs, top_n=5):    texts = [d.page_content for d in docs]    resp = co.rerank(model=\"rerank-english-v3.0\",                     query=query, documents=texts, top_n=top_n)    return [docs[r.index] for r in resp.results]\nRERANK_PROMPT = \"\"\"Given a query and a passage, rate how relevant the passageis to answering the query on a scale from 0 to 10. Return only the number.Query: {q}Passage: {p}Score:\"\"\"def llm_score(query, passage):    resp = client.chat.completions.create(        model=\"gpt-4o-mini\",        messages=[{\"role\": \"user\",                   \"content\": RERANK_PROMPT.format(q=query, p=passage)}],        temperature=0,    )    try:        return float(resp.choices[0].message.content.strip())    except ValueError:        return 0.0def llm_rerank(query, docs, top_n=5):    scored = [(d, llm_score(query, d.page_content)) for d in docs]    scored.sort(key=lambda x: -x[1])    return [d for d, s in scored[:top_n]]\nphp\ndef advanced_rag(user_query: str, history: str = \"\") -> dict:    # 1. Rewrite / contextualize    query = contextualize(history, user_query) if history else rewrite(user_query)    # 2. Multi-query expansion    expanded = [query] + expand(query)    # 3. Retrieve broadly    candidates = []    seen = set()    for q in expanded:        for d in vs.similarity_search(q, k=15):            key = d.page_content[:100]            if key not in seen:                candidates.append(d)                seen.add(key)    # 4. Re-rank    top_docs = rerank(query, candidates, top_n=5)    # 5. Generate    context = format_context(top_docs)    resp = client.chat.completions.create(        model=\"gpt-4o-mini\",        temperature=0.1,        messages=[            {\"role\": \"system\", \"content\": SYSTEM_PROMPT},            {\"role\": \"user\",             \"content\": f\"Context:\\n{context}\\n\\nQuestion: {user_query}\"},        ],    )    return {        \"answer\": resp.choices[0].message.content,        \"rewritten_query\": query,        \"sources\": [d.metadata for d in top_docs],    }\n```\n\nThis one function embodies most of what a serious RAG system does at query time.\n\nRAG is not just a technique — it’s the bridge that turns generic LLMs into trustworthy, up-to-date, domain-specific assistants. Master it, and you master the single most valuable pattern in applied AI today.\n\n[Mastering Retrieval Augmented Generation (RAG): The Complete End-to-End Guide](https://pub.towardsai.net/mastering-retrieval-augmented-generation-rag-the-complete-end-to-end-guide-754248a787eb) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/mastering-retrieval-augmented-generation-rag-the-complete-end-to-end-guide", "canonical_source": "https://pub.towardsai.net/mastering-retrieval-augmented-generation-rag-the-complete-end-to-end-guide-754248a787eb?source=rss----98111c9905da---4", "published_at": "2026-08-19 21:01:02+00:00", "updated_at": "2026-08-19 21:42:42.355003+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "generative-ai", "ai-products"], "entities": ["GPT-4", "Claude", "Llama 3", "Gemini", "Confluence", "Google Drive", "Notion"], "alternates": {"html": "https://wpnews.pro/news/mastering-retrieval-augmented-generation-rag-the-complete-end-to-end-guide", "markdown": "https://wpnews.pro/news/mastering-retrieval-augmented-generation-rag-the-complete-end-to-end-guide.md", "text": "https://wpnews.pro/news/mastering-retrieval-augmented-generation-rag-the-complete-end-to-end-guide.txt", "jsonld": "https://wpnews.pro/news/mastering-retrieval-augmented-generation-rag-the-complete-end-to-end-guide.jsonld"}}