# Ragas vs Truvec for RAG Retrieval Evaluation > Source: > Published: 2026-10-12 12:30:00+00:00 I have built and debugged many RAG systems, and the most common failure I saw was not because of bad prompts or weak LLM models. Most failures were due to the right passage that was never retrieved or retrieved at the wrong rank. This post starts with a short guide on how retrieval evaluation works, then compares Ragas and Truvec and explains where each one fits. A retrieval augmented generation (RAG) system answers questions in two steps, retrieval finds the passages (chunks) of your documents that are relevant to the question and generation gives those chunks to an LLM to write the answer. The retrieval step has many moving parts such as: **Chunking:** How documents are split into passages (size, overlap, by paragraph, by Markdown header, by function…). **The embedding model:** Turns each chunk and each question into a vector. **The retrieval algorithm:** How the closest chunks are found for example cosine similarity **Vector storage:** How the vectors are stored (Qdrant, Weaviate, PGVector…). **The reranker:** How the closest retrieved chunks are reordered **Top k:** How many of the chunks to pass to the LLM. If the chunk with the answer is at rank 9 and you send the top 5, the LLM will not be able to answer the question correctly. because the generation step only works with what retrieval hands it. **What a retrieval evaluation needs** To evaluate a retrieval process, you need three things. The documents or corpus, your system will search, chunked the way production chunks them. A golden dataset, test questions linked to the chunks that contain the answer (the ground truth) and the metrics to score where the expected chunks landed in each result. **The core retrieval metrics** To explain these metrics, let’s take a concrete example: Supposing you have a pdf about your company’s internal processes and you want to evaluate how you can find the relevant chunks for a given question. You already chunked the pdf into 50 chunks and stored them in a vector database and created a golden dataset with questions and their expected chunks. Take a question whose answer lives in chunks #14 and #15. The retriever configured with top_k=5, returned in order these chunks #22, #14, #8, #41, #3. This means that the first expected chunk #14 is at rank 2. The table below calculates some of the core retrieval metrics for this example: These are deterministic retrieval metrics, the same chunks, questions and model always give the same score. **Ground truth or LLM-as-judge (or JEV-as-judge)** There are two broad ways to decide whether retrieved chunks are relevant: ***Reference-based or ground truth IDs*:** you know in advance which chunks answer each question. Scoring is exact and cheap, and you can compare models with confidence. The cost is building the golden dataset. ***LLM-as-judge:*** an LLM reads the question, the retrieved chunks and the reference answer, and decides what’s relevant. It needs less setup and works on any pipeline, but every score depends on a judge model, its prompt and its own mistakes, and each run costs LLM calls. Ragas leans on the second approach. Truvec uses the first one. That difference explains most of what follows. [Ragas](https://github.com/vibrantlabsai/ragas) is an open-source Python library for evaluating LLM applications, and it’s one of the most widely used tools for RAG evaluation. Using Ragas you can compute metrics such as: **Context precision and context recall:** how relevant the retrieved contexts are, and how much of the reference they cover. **Faithfulness: **whether the answer’s claims are supported by the retrieved contexts. **Response relevancy:** whether the answer addresses the question. Other metrics, such as noise sensitivity and context entity recall. Most of these metrics use an LLM as judge. Recent versions also include non-LLM and ID based variants of context precision and recall when you supply reference contexts. Ragas can also generate synthetic test sets from your documents. **Where you can use Ragas:** You can use Ragas to evaluate a pipeline you already built, end to end, including rerankers, hybrid search and query rewriting. and calculate metrics like faithfulness, hallucinations, relevance. **What it asks of you:** With Ragas you should build and run the pipeline for every configuration you want to test. Comparing five embedding models means indexing your corpus five times, with five provider accounts and API keys. The LLM-judged evaluations depends on the judge model and cost tokens on each run. [Truvec](https://truvec.dev) is a hosted platform that evaluates the retrieval step on your own documents. You upload a sample of documents (Truvec chunks them based on your preferences) or chunks that you already have, and the platform builds a golden dataset for you, runs selected embedding models against it and scores where the expected chunks land. **Where you can use Truvec:** **Side-by-side model comparison.** Pick two or more models and run one comparison. Truvec supports multiple embedding models from OpenAI, Cohere, Voyage AI, Google, and open-source models (BGE, Nomic, mxbai, GTE, Qwen3 Embedding). You don't need any provider API key: Truvec runs the models on its own accounts and your plan includes the usage. **Calculate rank-based metrics:** hit rate, recall, precision, MRR, nDCG and F1 at your top k, plus diagnostics: hit rate at every cut-off, rank of the first relevant chunk, and the score gap between the right chunk and the best distractor. **Calculate cost,** speed and storage: index cost, cost per 1,000 queries, query embedding latency, vector search p50/p95 and vector storage for each model, computed from real token counts. **The Retrieval Inspector:** For every question, see the ranked chunks, marked Ground truth or Distractor, and the Expected but not retrieved chunks. **A Playground** to run any question against any supported model, with no API setup. **PDF reports** with a recommendation, a leaderboard, a head-to-head analysis between models and a question-by-question matrix, for the decision record. Truvec does not evaluate the answer your LLM writes, nor does it run your own pipeline. **Use Truvec when:** You’re choosing an embedding model for a new project and want evidence on your own documents, not a leaderboard rank. You’re considering a migration to a new embedding model and need to know if it’s better enough to justify re-indexing everything. Your answers are often wrong or vague and you suspect retrieval. The Misses filter usually tells you in ten minutes whether the problem is the model, the chunking, PDF extraction or the test questions. You want to test a different chunking setup quickly without rebuilding your pipeline. You need a report a non-specialist can read. **Use Ragas when:** Retrieval is solid and you need to grade the generated answers: faithfulness, hallucinations, relevance. You want to score the exact pipeline you run, including rerankers and query rewriting, inside your codebase. Use both when you’re building a serious RAG product. A workflow that works well: **Prototype retrieval in Truvec:** pick the embedding model and chunking with a golden dataset and a comparison. Build the pipeline with the winning setup. Evaluate end to end with Ragas, focusing on the generation metrics. When a new embedding model comes out or your corpus changes, go back to Truvec and re-run the comparison with your current model as the baseline. When a RAG pipeline quality drops, you don’t know whether the retriever, the prompt or the LLM is to blame. Isolating retrieval metrics like hit rate@5 helps you identify the issue. It also makes decisions cheaper. Comparing six embedding models with an LLM judge means six indexing pipelines and thousands of judge calls. Comparing them on a locked golden dataset is deterministic. Originally published at [https://truvec.dev](https://truvec.dev) on October 9, 2026.