{"slug": "your-rag-pipeline-can-be-fast-and-still-be-wrong-a-developer-s-guide-to", "title": "Your RAG Pipeline Can Be Fast and Still Be Wrong: A Developer's Guide to Embedding Evaluation", "summary": "Shrijith Venkatramana, a developer building the AI code review tool LiveReview, argues that RAG pipelines can return results in milliseconds yet still surface the wrong documents, and that embedding evaluation should be treated as an information-retrieval ranking problem. He outlines building a benchmark of real user queries with graded relevance judgments, including multilingual, typo-laden, and vague queries, to compare embedding models and decide when fine-tuning is warranted.", "body_md": "*Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. [Star us](https://github.com/HexmosTech/LiveReview/) to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nA vector database can return 20 documents in 15 milliseconds and still give your RAG system a terrible answer.\n\nThe hard part of retrieval is rarely generating vectors. It is deciding whether the vectors are putting the *right* documents near the query.\n\nThat makes embedding evaluation an information-retrieval problem.\n\nOnce you see it that way, three metrics become particularly useful:\n\nThen there is a second problem.\n\nWhat happens when your users search in Hindi for English documentation? Or type a product name in Latin script inside a Hindi sentence? Or use internal terminology that never appeared in the embedding model's training data?\n\nAnd finally: when should you fine-tune the embedding model instead of changing something else?\n\nLet's build the answer from the ground up.\n\nIt is tempting to think of an embedding as a semantic fingerprint.\n\nYou encode:\n\n```\n\"How do I rotate an API key?\"\n```\n\nand get a vector such as:\n\n```\n[0.18, -0.07, 0.42, ...]\n```\n\nYou encode every document chunk in your corpus, store those vectors, and retrieve the closest ones.\n\nUsually the similarity is cosine similarity:\n\n```\ncos(q, d) = (q . d) / (||q|| * ||d||)\n```\n\nWhen vectors are normalized to unit length, cosine similarity becomes equivalent to the dot product:\n\n```\ncos(q, d) = q . d\n```\n\nSo the system is doing something conceptually simple:\n\n```\nquery\n  |\n  v\nembedding\n  |\n  v\nrank documents by similarity\n  |\n  v\ntop-k documents\n```\n\nThe important word is **rank**.\n\nYour application does not really care whether document A has a cosine similarity of `0.83` and document B has `0.79`.\n\nIt cares whether A should appear before B.\n\nThis distinction has been present in information retrieval for decades.\n\nIn 2002, Kalervo Järvelin and Jaana Kekäläinen published work on evaluating retrieval systems using *graded relevance*: a document can be irrelevant, somewhat useful, highly useful, and so on. Their motivation was practical: retrieval systems produce too many results, so evaluation has to reward systems that put highly relevant material near the top.\n\nModern embedding search has the same problem.\n\nThe vector database is merely the mechanism.\n\nYour real system is:\n\n``` php\nsemantic query -> ranking -> useful context\n```\n\nThat means you need a test set.\n\nSuppose you have a documentation system with 100,000 chunks.\n\nYou collect 500 real user queries:\n\n```\n\"how to rotate api keys\"\n\"why does websocket authentication fail\"\n\"can I run the agent without public ports\"\n\"delete an organization\"\n...\n```\n\nFor each query, you identify the documents that actually answer it.\n\nFor example:\n\n```\nQuery:\n\"how to rotate api keys\"\n\nRelevant:\ndoc_1842\ndoc_7119\ndoc_9320\n```\n\nThat gives you a small retrieval benchmark.\n\nYou can now compare embedding models using the exact same queries.\n\nThis is much more useful than saying:\n\nModel A \"looks more semantic.\"\n\nA useful benchmark should contain the ugly cases.\n\nInclude short queries.\n\nInclude vague queries.\n\nInclude terminology that only your users understand.\n\nInclude queries with multiple correct documents.\n\nInclude queries where the correct answer is buried inside a large document.\n\nInclude typos.\n\nInclude multilingual queries.\n\nAnd keep a separate test set that you do not use while tuning the model.\n\nA simple evaluation dataset might look like:\n\n```\nquery_id: 17\nquery: \"rotate api key\"\nrelevant_docs:\n  - doc_1842\n  - doc_7119\n\nquery_id: 18\nquery: \"websocket auth failure\"\nrelevant_docs:\n  - doc_921\n```\n\nNow your embedding problem becomes measurable.\n\nSuppose there are five relevant documents for a query:\n\n```\nRelevant = {A, B, C, D, E}\n```\n\nYour embedding search returns:\n\n```\nTop 5 = [X, A, Y, Z, B]\n```\n\nYou found 2 of the 5 relevant documents.\n\nSo:\n\n```\nRecall@5 = 2 / 5 = 0.40\n```\n\nIn general:\n\n```\nRecall@k =\n    relevant documents retrieved in top-k\n    -------------------------------------\n    total relevant documents\n```\n\nThe intuition is:\n\nRecall@k measures how much of the answer space you managed to bring into view.\n\nConsider a RAG system retrieving 10 chunks.\n\nIf the answer requires information from one specific chunk and that chunk never appears in the top 10, your LLM cannot recover it.\n\nA stronger language model does not magically fix missing context.\n\nImagine 1,000 evaluation queries, with an average of 4 relevant chunks each.\n\nThat is approximately:\n\n```\n1,000 * 4 = 4,000 relevant chunks\n```\n\nIf Recall@20 is 0.80, your retriever brought back roughly:\n\n```\n4,000 * 0.80 = 3,200\n```\n\nrelevant chunks.\n\nAbout 800 relevant chunks were missed.\n\nThat is a concrete failure budget.\n\nIt treats all relevant documents equally.\n\nSuppose the relevant set is:\n\n```\n[A, B, C]\n```\n\nand two systems return:\n\n```\nSystem 1: [A, X, Y, Z, B]\nSystem 2: [X, A, B, Y, Z]\n```\n\nAt `k=5`, both have identical recall.\n\nYet System 1 puts the strongest result first.\n\nRecall tells you whether useful material entered the candidate set.\n\nIt says much less about ordering.\n\nThat is where MRR and NDCG enter.\n\nMRR stands for Mean Reciprocal Rank.\n\nFor one query:\n\n```\nreciprocal rank = 1 / rank_of_first_relevant_result\n```\n\nSo if the first relevant result appears at:\n\n``` php\nrank 1 -> 1.00\nrank 2 -> 0.50\nrank 3 -> 0.33\nrank 10 -> 0.10\n```\n\nFor multiple queries, take the average.\n\nExample:\n\n``` php\nQuery 1 -> first relevant at rank 1 -> 1.00\nQuery 2 -> first relevant at rank 2 -> 0.50\nQuery 3 -> first relevant at rank 4 -> 0.25\nQuery 4 -> no relevant result      -> 0.00\n```\n\nThen:\n\n```\nMRR = (1.00 + 0.50 + 0.25 + 0.00) / 4\n    = 0.4375\n```\n\nMRR is especially useful when your query usually has one obvious answer.\n\nThink of questions such as:\n\n```\n\"what is the default timeout?\"\n\"where is the config file?\"\n\"how do I reset my password?\"\n\"what command starts the server?\"\n```\n\nFor these, getting the correct result at rank 1 is much better than getting it at rank 15.\n\nBut MRR ignores everything after the first relevant document.\n\nConsider:\n\n```\nSystem A: [A, X, X, X, X]\nSystem B: [A, B, C, D, E]\n```\n\nIf A is relevant, both have:\n\n```\nRR = 1.0\n```\n\nMRR sees them as identical.\n\nThat is clearly wrong for many RAG applications.\n\nIf your answer depends on several pieces of evidence, you care about the whole ranking.\n\nThis is where NDCG becomes useful.\n\nNDCG stands for Normalized Discounted Cumulative Gain.\n\nThe useful idea is simpler than the name.\n\nYou assign each retrieved result a relevance grade.\n\n```\n0 = irrelevant\n1 = somewhat useful\n2 = useful\n3 = directly answers the question\n```\n\nNow imagine your ranking is:\n\n``` php\nrank 1 -> relevance 3\nrank 2 -> relevance 2\nrank 3 -> relevance 0\nrank 4 -> relevance 1\nrank 5 -> relevance 0\n```\n\nNDCG gives more credit to highly relevant documents near the top.\n\nThe underlying DCG calculation is:\n\n```\nDCG@k =\n    sum from i=1 to k of\n    (2^rel_i - 1) / log2(i + 1)\n```\n\nThe pieces have intuitive meanings.\n\n```\n2^rel_i - 1\n```\n\nmakes relevance 3 much more valuable than relevance 1.\n\nAnd:\n\n```\nlog2(i + 1)\n```\n\ndiscounts lower-ranked results.\n\nSo relevance at rank 1 counts more than relevance at rank 10.\n\nThen normalize against the ideal ordering:\n\n```\nNDCG@k = DCG@k / ideal_DCG@k\n```\n\nTherefore:\n\n```\nNDCG@k = 1.0\n```\n\nmeans your results are ordered exactly like the ideal ranking.\n\nSuppose the ideal relevance grades are:\n\n```\n[3, 2, 2, 1, 0]\n```\n\nYour system returns:\n\n```\n[1, 0, 3, 2, 0]\n```\n\nThe system still retrieved good documents.\n\nRecall might look decent.\n\nMRR might also look decent because a relevant document appears early.\n\nBut NDCG drops because the grade-3 answer was pushed to rank 3.\n\nThat is often exactly what you want to punish in a RAG pipeline.\n\nThink of them as three different questions:\n\n``` php\nRecall@k -> Did we retrieve enough of the answer?\n\nMRR      -> Did we find an answer quickly?\n\nNDCG     -> Did we order useful answers correctly?\n```\n\nA good retrieval benchmark often uses all three.\n\nA simple Python implementation makes the distinction concrete:\n\n``` python\nimport math\n\ndef recall_at_k(ranked_ids, relevant_ids, k):\n    relevant_ids = set(relevant_ids)\n    retrieved = set(ranked_ids[:k])\n    return len(retrieved & relevant_ids) / len(relevant_ids)\n\ndef reciprocal_rank(ranked_ids, relevant_ids):\n    relevant_ids = set(relevant_ids)\n\n    for rank, doc_id in enumerate(ranked_ids, start=1):\n        if doc_id in relevant_ids:\n            return 1.0 / rank\n\n    return 0.0\n\ndef ndcg_at_k(relevances, k):\n    def dcg(values):\n        return sum(\n            (2 ** rel - 1) / math.log2(i + 2)\n            for i, rel in enumerate(values[:k])\n        )\n\n    actual = dcg(relevances)\n    ideal = dcg(sorted(relevances, reverse=True))\n\n    return actual / ideal if ideal else 0.0\n```\n\nNotice something important:\n\n**NDCG requires graded judgments.**\n\nIf every document is simply relevant or irrelevant, NDCG still works, but you are throwing away useful information.\n\nFor many production systems, a practical judgment scheme is:\n\n```\n0 = does not answer the query\n1 = related but insufficient\n2 = useful evidence\n3 = directly answers the query\n```\n\nThat is often enough to make ranking failures visible.\n\nNow suppose your documentation is multilingual.\n\nA user asks:\n\n```\n\"How do I reset my password?\"\n```\n\nThe relevant document is written in Hindi:\n\n```\n\"पासवर्ड भूल जाने पर आप अपना पासवर्ड रीसेट कर सकते हैं...\"\n```\n\nA genuinely multilingual embedding model should place those texts near each other in vector space.\n\nThis is a harder problem than ordinary monolingual similarity because the model has to learn a language-independent semantic representation.\n\nThis became a major research direction before today's LLM era.\n\nMikel Artetxe and Holger Schwenk, for example, developed LASER, a multilingual sentence-embedding system covering 93 languages. Their work explicitly evaluated cross-lingual similarity and showed that a shared embedding space could support semantic search across languages.\n\nBut \"multilingual\" does not mean \"equally good in every language pair.\"\n\nYou should test that yourself.\n\nFor example, your evaluation matrix might look like:\n\n| Query | Document | Recall@10 | \n|---|---|---|\n| English | English | 0.91 | \n| Hindi | Hindi | 0.87 | \n| English | Hindi | 0.64 | \n| Hindi | English | 0.59 | \n| Tamil | English | 0.52 | \n| Hinglish | English | 0.47 | \n\nThese numbers are illustrative, but the structure of the test is important.\n\nThe aggregate score could hide a serious problem.\n\nSuppose:\n\n```\nEnglish-English: 0.92\nEnglish-Hindi:   0.61\nHindi-English:   0.58\n```\n\nand 90% of your benchmark queries are English-English.\n\nYour overall recall could still look excellent.\n\nYour Hindi users would experience something very different.\n\nThere are several recurring cases:\n\n**Different scripts**\n\n```\nEnglish: \"insurance claim\"\nHindi:    \"बीमा दावा\"\n```\n\n**Transliteration**\n\n```\n\"बीमा\"\n\"beema\"\n\"bima\"\n```\n\n**Code-switching**\n\n```\n\"UPI ka transaction fail ho raha hai\"\n```\n\n**Named entities**\n\nA company, API, medicine, place, or product name may appear unchanged across languages.\n\n**Domain vocabulary**\n\nYour users may mix natural language with terms such as:\n\n```\nOAuth\nJWT\nwebhook\nprotobuf\nnginx\n```\n\nThese are neither cleanly English nor cleanly Hindi.\n\nFor multilingual retrieval, evaluate by language pair and query type rather than relying on one global score.\n\nThe broader lesson came through later benchmark work too. The MTEB benchmark evaluated embedding models across many tasks and languages and found that no single embedding method dominated every task. Embedding quality is task-dependent.\n\nThat observation matters enormously when somebody says:\n\n\"We use the best embedding model.\"\n\nBest for what?\n\nFine-tuning is attractive because it feels like the direct solution:\n\n```\nretrieval is bad\n        |\n        v\nfine-tune embedding model\n        |\n        v\nretrieval gets better\n```\n\nSometimes that is exactly right.\n\nOften it is premature.\n\nThe first question should be:\n\nWhat kind of retrieval error are you trying to fix?\n\nConsider this failure:\n\n```\nQuery:\n\"How do I rotate an API key?\"\n\nReturned:\n\"API authentication concepts\"\n\"API key configuration\"\n\"Creating API credentials\"\n```\n\nThe system understands the general topic but misses the specific operation.\n\nThis could be an embedding-model problem.\n\nNow consider another failure:\n\n```\nQuery:\n\"How do I rotate an API key?\"\n\nRelevant chunk exists.\n\nThe embedding retriever finds the correct document.\n\nBut your chunk is 3,000 tokens long and the relevant sentence is buried near the bottom.\n```\n\nFine-tuning the embedding model will not solve the real problem.\n\nYou have a chunking problem.\n\nOr perhaps:\n\n```\nQuery:\n\"delete organization\"\n```\n\nThe system retrieves documentation for both:\n\n```\ndelete organization\ndelete organization member\n```\n\nThe semantic representation may be perfectly reasonable.\n\nYou may need metadata filters, query rewriting, or a reranker.\n\nStart with the cheapest intervention.\n\n```\n1. Check chunking\n2. Check metadata filters\n3. Check query preprocessing\n4. Try a stronger embedding model\n5. Add a reranker\n6. Fine-tune the embedding model\n```\n\nThe exact order can change, but the principle is:\n\nFix the smallest component that explains the failure.\n\nSuppose you have 5,000 real query-document judgments.\n\nYou repeatedly see:\n\n```\n\"policy renewal\"\n```\n\nbeing ranked below:\n\n```\n\"policy purchase\"\n```\n\neven though your users consider renewal documents clearly relevant.\n\nThat tells you something useful.\n\nYour production relevance function differs from the semantic relationships learned by the base embedding model.\n\nNow fine-tuning has a concrete target.\n\nYou can train on examples such as:\n\n```\nquery:\n\"policy renewal\"\n\npositive:\n\"Renewing an existing insurance policy\"\n\nhard negative:\n\"Purchasing a new insurance policy\"\n```\n\nThe hard negative matters.\n\nRandom negatives are usually too easy.\n\nIf the model already knows that:\n\n```\n\"dog food\"\n```\n\nis unrelated to:\n\n```\n\"database replication\"\n```\n\ntraining on that pair teaches it almost nothing.\n\nYou want confusing examples:\n\n```\npositive:\n\"reset password for an existing account\"\n\nhard negative:\n\"change account email address\"\n```\n\nThe model needs to learn the boundary.\n\nSuppose you want 2,000 evaluation queries.\n\nYou inspect the top 20 retrieved chunks for each query.\n\nThat is:\n\n```\n2,000 * 20 = 40,000 judgments\n```\n\nAt just 10 seconds per judgment:\n\n```\n40,000 * 10 seconds = 400,000 seconds\n                   ~= 111 hours\n```\n\nThe expensive part of embedding work is often not GPU training.\n\nIt is producing trustworthy relevance data.\n\nThat means your benchmark itself is an asset.\n\nOnce you have it, you can evaluate:\n\n```\nEmbedding A\nEmbedding B\nEmbedding C\nEmbedding C + reranker\nFine-tuned C\nFine-tuned C + reranker\n```\n\nusing the same dataset.\n\nSuppose you store 1 million chunks with 1,536-dimensional float32 embeddings.\n\nRaw vector storage is approximately:\n\n```\n1,000,000 * 1,536 * 4 bytes\n= 6.144 GB\n```\n\nbefore vector-index overhead, metadata, replication, and backups.\n\nA 768-dimensional representation cuts the raw vector storage roughly in half.\n\nThis is why embedding evaluation is not purely about \"which model has the highest score?\"\n\nYou are optimizing a system with several variables:\n\n```\nretrieval quality\nlatency\nembedding generation cost\nstorage\nindex size\nreranking cost\nengineering complexity\n```\n\nSuppose a new model improves Recall@20 from:\n\n``` php\n0.84 -> 0.87\n```\n\nbut doubles embedding generation cost and increases query latency.\n\nThat improvement might still be worthwhile.\n\nOr it might be irrelevant if a cheap reranker gets you:\n\n``` php\n0.84 -> 0.91\n```\n\nat acceptable latency.\n\nYou need measurements rather than intuition.\n\nA production embedding evaluation system does not need to be elaborate.\n\nStart with perhaps 300-1,000 real queries.\n\nFor each query, store:\n\n```\nquery\nlanguage\nrelevant document IDs\noptional relevance grades\n```\n\nThen run every candidate retriever over the same benchmark.\n\nTrack at least:\n\n```\nRecall@5\nRecall@10\nRecall@20\nMRR\nNDCG@10\n```\n\nFor multilingual systems, break them down by language pair.\n\nFor domain-heavy systems, break them down by query category.\n\n```\n                Recall@10   MRR    NDCG@10\n--------------------------------------------\nAuthentication    0.91      0.88     0.86\nBilling           0.87      0.79     0.81\nDeployment        0.83      0.76     0.77\nMultilingual      0.62      0.51     0.55\n```\n\nNow you have something far more actionable than:\n\n\"The embeddings seem pretty good.\"\n\nYou can also investigate individual failures.\n\nFor every bad query, log:\n\n```\nquery\ntop-k documents\nsimilarity scores\nrelevance labels\nlanguage\nchunk metadata\n```\n\nThen ask:\n\n```\nWas the answer absent from top-k?\nWas the right document present but badly ranked?\nWas the chunk itself poor?\nWas the query ambiguous?\nWas this a language-specific failure?\n```\n\nThose questions lead to different engineering fixes.\n\nThat is the main mindset shift.\n\n**Embedding evaluation is not a model leaderboard exercise.**\n\nIt is debugging a ranking system.\n\nThe embedding model is one component inside that system.\n\nAnd once you have a real benchmark, the fine-tuning question becomes much easier.\n\nYou can make the change, rerun the exact same queries, and ask:\n\n```\nDid Recall@20 improve?\n\nDid NDCG improve?\n\nDid multilingual retrieval improve?\n\nDid we make another language worse?\n\nDid latency or cost change?\n\nDid the improvement survive on the held-out test set?\n```\n\nThat is much more reliable than choosing an embedding model because its benchmark score looks good.\n\nEmbeddings turn semantic similarity into geometry.\n\nRetrieval turns that geometry into a ranking.\n\nEvaluation tells you whether that ranking actually serves your users.\n\nRecall@k tells you whether the useful information entered the candidate set.\n\nMRR tells you how quickly the first useful result appears.\n\nNDCG tells you whether the most useful results were placed where users can actually benefit from them.\n\nMultilingual evaluation tells you whether a single aggregate number is hiding failures across languages.\n\nAnd a good benchmark tells you when fine-tuning is justified rather than merely tempting.\n\nThe most useful question is therefore not:\n\n\"Which embedding model should I use?\"\n\nIt is:\n\n\"What retrieval behavior do my users actually need, and can I measure whether my system produces it?\"\n\nWhat retrieval metric or failure mode has mattered most in the RAG systems you have built?\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n⭐ Star it on GitHub: \n\nLiveReview is an AI code reviewer that scores every hunk of a diff by **blast radius**: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.\n\n*LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.*\n\n| The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | \n|---|---|---|\n\n**Here's the goal:**\n\n**Click below to try LiveReview with your codebase:**", "url": "https://wpnews.pro/news/your-rag-pipeline-can-be-fast-and-still-be-wrong-a-developer-s-guide-to", "canonical_source": "https://dev.to/shrsv/your-rag-pipeline-can-be-fast-and-still-be-wrong-a-developers-guide-to-embedding-evaluation-3lip", "published_at": "2026-09-19 18:33:16+00:00", "updated_at": "2026-09-19 19:23:05.802209+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "natural-language-processing", "ai-products"], "entities": ["Shrijith Venkatramana", "LiveReview", "HexmosTech", "Kalervo Järvelin", "Jaana Kekäläinen"], "alternates": {"html": "https://wpnews.pro/news/your-rag-pipeline-can-be-fast-and-still-be-wrong-a-developer-s-guide-to", "markdown": "https://wpnews.pro/news/your-rag-pipeline-can-be-fast-and-still-be-wrong-a-developer-s-guide-to.md", "text": "https://wpnews.pro/news/your-rag-pipeline-can-be-fast-and-still-be-wrong-a-developer-s-guide-to.txt", "jsonld": "https://wpnews.pro/news/your-rag-pipeline-can-be-fast-and-still-be-wrong-a-developer-s-guide-to.jsonld"}}