Evaluate and Debug RAG Pipelines with Ragas

Ragas 0.4.3 introduces an evaluation harness that scores RAG pipeline outputs on faithfulness, context precision, and answer relevancy, using LLM judges to identify whether failures originate from the retriever or the generator. The tutorial, authored by Rachel Goldstein, demonstrates setup with Python 3.10–3.13, OpenAI API key, and specific package versions, costing about a cent per full run with gpt-4o-mini and text-embedding-3-small.

Evaluate and Debug RAG Pipelines with Ragas Score faithfulness, context precision, and answer relevancy to tell whether your retriever or your generator is failing. Rachel Goldstein https://sourcefeed.dev/u/rachel goldstein What you'll build A small evaluation harness that scores your RAG pipeline's outputs on three Ragas https://docs.ragas.io/ metrics faithfulness, context precision, answer relevancy and tells you whether a bad answer came from the retriever or the generator. Prerequisites - Python 3.10–3.13. Verified on 3.13; Ragas declares support for 3.9+. - Ragas 0.4.3 and openai 1.109.1, the current stable releases this was tested against. The code below uses the ragas.metrics.collections API introduced in the 0.3/0.4 line. It will not run on ragas 0.2.x. - langchain-community pinned below 0.4 see step 1; ragas 0.4.3 breaks with 0.4.x . - An OpenAI https://platform.openai.com/ API key with a small amount of credit. The judge calls in this tutorial use gpt-4o-mini and text-embedding-3-small ; a full run costs about a cent. - Commands are for macOS/Linux. On Windows, activate the venv with .venv\Scripts\activate instead. A note on how this works: these metrics are LLM-judged. Ragas sends your question, contexts, and answer to a judge model that extracts claims and checks them. That's why you need an API key even though you're evaluating outputs you already have. 1. Set up the project mkdir rag-eval && cd rag-eval python3 -m venv .venv && source .venv/bin/activate pip install ragas==0.4.3 openai "langchain-community<0.4" export OPENAI API KEY="sk-..." The langchain-community<0.4 pin matters. Ragas 0.4.3 imports a module that langchain-community 0.4.x removed, and without the pin the install succeeds but every import fails see Troubleshooting . 2. Capture pipeline outputs as an eval dataset Ragas evaluates records with four fields: user input the question , retrieved contexts the chunks your retriever returned , response what your generator produced , and reference a known-good answer, needed only for context precision . In a real app you'd log these from your pipeline; here we hand-craft three samples with known failure modes so you can see each metric react. Save as samples.json : { "id": "good", "user input": "What port does PostgreSQL listen on by default?", "retrieved contexts": "PostgreSQL listens on TCP port 5432 by default. The port is set by the port parameter in postgresql.conf.", "To change the listen address, edit the listen addresses parameter in postgresql.conf and restart the server." , "response": "PostgreSQL listens on port 5432 by default. You can change it via the port parameter in postgresql.conf.", "reference": "PostgreSQL's default port is 5432, configured by the port parameter in postgresql.conf." }, { "id": "hallucinated-answer", "user input": "How do I enable WAL archiving in PostgreSQL?", "retrieved contexts": "To enable WAL archiving, set wal level to replica or higher and set archive mode to on in postgresql.conf.", "The archive command parameter defines the shell command used to copy a completed WAL segment to archive storage." , "response": "Set wal level to replica and archive mode to on in postgresql.conf. Note that archive mode defaults to on since PostgreSQL 16, so most installations already archive WAL automatically.", "reference": "Enable WAL archiving by setting wal level to replica or higher, archive mode to on, and configuring archive command in postgresql.conf." }, { "id": "bad-retrieval", "user input": "What is the maximum identifier length in PostgreSQL?", "retrieved contexts": "VACUUM reclaims storage occupied by dead tuples and is run automatically by the autovacuum daemon.", "The pg dump utility creates logical backups of a single PostgreSQL database in script or archive formats." , "response": "PostgreSQL identifiers are limited to 63 bytes by default, set by the NAMEDATALEN constant at compile time.", "reference": "The maximum identifier length in PostgreSQL is 63 bytes, determined by NAMEDATALEN minus one." } Sample two retrieves the right chunks but the generator invents a claim archive mode does not default to on . Sample three answers correctly from the model's own knowledge while the retriever returned junk, a failure that stays invisible until the model gets a question it can't answer from memory. 3. Write the scoring script Save as eval rag.py : python import asyncio import json from openai import AsyncOpenAI from ragas.embeddings.base import embedding factory from ragas.llms import llm factory from ragas.metrics.collections import AnswerRelevancy, ContextPrecision, Faithfulness with open "samples.json" as f: samples = json.load f async def main : client = AsyncOpenAI reads OPENAI API KEY llm = llm factory "gpt-4o-mini", client=client embeddings = embedding factory "openai", model="text-embedding-3-small", client=client faithfulness = Faithfulness llm=llm context precision = ContextPrecision llm=llm answer relevancy = AnswerRelevancy llm=llm, embeddings=embeddings print f"{'sample':<22}{'faithfulness': 14}{'ctx precision': 15}{'relevancy': 11}" for s in samples: faith, precision, relevancy = await asyncio.gather faithfulness.ascore user input=s "user input" , response=s "response" , retrieved contexts=s "retrieved contexts" , , context precision.ascore user input=s "user input" , reference=s "reference" , retrieved contexts=s "retrieved contexts" , , answer relevancy.ascore user input=s "user input" , response=s "response" , , print f"{s 'id' :<22}{faith.value: 14.2f}" f"{precision.value: 15.2f}{relevancy.value: 11.2f}" asyncio.run main Each metric only sees the fields it needs, which is the point: faithfulness checks the response against the retrieved contexts, context precision checks the contexts against the reference, and answer relevancy ignores the contexts entirely. The three metrics for one sample run concurrently via asyncio.gather since they're independent API calls. 4. Read the scores like a debugger Each metric isolates one component, so a low score points at a specific fix: - Low faithfulness, high context precision: the generator is inventing claims the context doesn't support. Tighten the prompt "answer only from the provided context" or use a stronger generation model. - Low context precision: the retriever is the problem. Look at chunk size, embedding model, top-k, or add a reranker. Faithfulness may still be high if the model faithfully summarizes the wrong chunks. - Low answer relevancy: the response dodges the question. Usually a prompt problem over-long boilerplate answers or the query needs rewriting before retrieval. High relevancy with low faithfulness and low precision is the dangerous quadrant: confident, on-topic answers built from nothing. Verify it works python eval rag.py Expected output judge-based scores wobble a few points between runs; the pattern is what you're checking : sample faithfulness ctx precision relevancy good 1.00 1.00 0.97 hallucinated-answer 0.67 1.00 0.93 bad-retrieval 0.00 0.00 0.91 The run takes 20–60 seconds. Each sample's scores match its planted failure: the hallucinated answer loses a third of its faithfulness score one unsupported claim out of three , and the bad-retrieval sample shows 0.00 precision while relevancy stays high because the answer itself reads fine. Troubleshooting ModuleNotFoundError: No module named 'langchain community.chat models.vertexai' on any ragas import. Ragas 0.4.3 depends on langchain-community without an upper bound, and 0.4.x removed this module. Fix: pip install "langchain-community<0.4" resolves to 0.3.31 . OpenAIError: The api key client option must be set either by passing api key to the client or by setting the OPENAI API KEY environment variable . The key isn't in the environment of the shell running the script. Re-run export OPENAI API KEY="sk-..." in the same terminal, or pass AsyncOpenAI api key=... explicitly. ValueError: Collections metrics only support modern embeddings. Found: LangchainEmbeddingsWrapper. You called the deprecated no-argument embedding factory from an older tutorial. Metrics from ragas.metrics.collections require the modern interface: embedding factory "openai", model="text-embedding-3-small", client=client . TypeError: AnswerRelevancy. init missing 1 required positional argument: 'embeddings' . Unlike the other two metrics, AnswerRelevancy embeds generated questions to compare against the original, so it needs both llm= and embeddings= . Next steps Add ContextRecall and FactualCorrectness both in ragas.metrics.collections , both need reference to catch retrievers that miss chunks rather than fetch wrong ones. Replace the hand-written samples with real traces logged from your pipeline, or generate a synthetic test set from your documents with ragas.testset.TestsetGenerator . Once scores are stable, wire eval rag.py into CI and fail the build when faithfulness drops below your baseline, so a prompt tweak can't silently reintroduce hallucinations. Sources & further reading - Evaluate a simple RAG system https://docs.ragas.io/en/stable/getstarted/rag eval/ — docs.ragas.io - Faithfulness metric https://docs.ragas.io/en/stable/concepts/metrics/available metrics/faithfulness/ — docs.ragas.io - Context Precision metric https://docs.ragas.io/en/stable/concepts/metrics/available metrics/context precision/ — docs.ragas.io - Answer Relevancy metric https://docs.ragas.io/en/stable/concepts/metrics/available metrics/answer relevance/ — docs.ragas.io - ragas 0.4.3 https://pypi.org/project/ragas/ — pypi.org Rachel Goldstein https://sourcefeed.dev/u/rachel goldstein · Dev Tools Editor Rachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop. Discussion 0 No comments yet Be the first to weigh in.