Ragas: how to actually measure your RAG pipeline (faithfulness, context precision/recall, relevancy)

A developer published a hands-on guide to using Ragas 0.4.x, an Apache-2.0 open-source library for evaluating RAG and LLM applications, to replace subjective "vibe checks" with trackable metrics. The guide covers four LLM-judged metrics — context recall, context precision, faithfulness and answer relevancy — each scored 0 to 1 and targeting a distinct pipeline failure mode, and walks through building a 20-50 question golden dataset with reference answers. It also flags breaking API changes from 0.3 to 0.4, including async metrics in ragas.metrics.collections, MetricResult return values, llm_factory replacing LangchainLLMWrapper, and the @experiment decorator superseding evaluate().

Thursday AI Tool/Workflow, part two. This morning was about parsing PDFs cleanly with Docling. This afternoon: Ragas , the open-source Apache-2.0 library for evaluating RAG and LLM apps, so you can tell whether a change actually made your pipeline better. Most RAG tuning goes like this: change the chunk size, ask the bot three questions, decide it "feels better", ship. Then a week later someone finds the question that broke. Ragas replaces the vibe check with numbers you can track: did retrieval find the right context, did the answer stick to that context, did it actually answer the question? Here's exactly how to use it on a real pipeline. Ragas has changed a lot, and many tutorials online use APIs that are now deprecated. Everything below targets Ragas 0.4.x 0.4.3 is the latest release on PyPI as I write . The main changes from 0.3 to 0.4, per the official migration guide: ragas.metrics.collections and are called with keyword arguments: await metric.ascore user input=..., response=... . MetricResult .value plus an optional .reason instead of a bare float. llm factory "model-name", client=... . LangchainLLMWrapper and LlamaIndexLLMWrapper are deprecated. evaluate still works but is deprecated in favour of the @experiment decorator. The old SingleTurnSample + single turn ascore pattern is legacy. The project also moved on GitHub: explodinggradients/ragas now points to vibrantlabsai/ragas . Docs are still at docs.ragas.io. pip install ragas openai The default examples use OpenAI as the judge model, so set OPENAI API KEY . llm factory also supports other providers Anthropic, Google, Azure, Bedrock and more by passing that provider's client. One thing worth knowing: Ragas collects minimal anonymized usage analytics. To opt out, set RAGAS DO NOT TRACK=true . Each metric looks at a different part of the pipeline, which is what makes them useful for debugging. All of them score from 0 to 1, higher is better, and all four use an LLM as the judge. | Metric | Plain-words question | Needs | Points at | |---|---|---|---| | Context recall | Did we retrieve everything needed to answer? | user input , retrieved contexts , reference | Retrieval missing info | | Context precision | Are the useful chunks ranked at the top? | user input , retrieved contexts , reference | Retrieval noise / ranking | | Faithfulness | Is every claim in the answer backed by the retrieved context? | user input , response , retrieved contexts | Generation making things up | | Answer relevancy | Does the answer address the question that was asked? | user input , response + embeddings | Off-topic or padded answers | How they're computed, from the docs: strictness and averages their embedding cosine similarity to the real question. It deliberately does You don't need a giant benchmark. Start with 20 to 50 real questions your users actually ask, each with a short reference answer written by someone who knows the docs. That's the slow, valuable part. Example hypothetical : an HR policy bot, built on the Docling chunks from this morning's post. {"id": "leave-01", "user input": "How many unused leave days can I carry over?", "reference": "Up to 10 days of unused annual leave carry over to the next year."} {"id": "leave-02", "user input": "Do carried-over days expire?", "reference": "Carried-over days must be used by 31 March or they lapse."} {"id": "exp-01", "user input": "What's the limit for client dinners?", "reference": "Client dinners are reimbursed up to the per-person limit in the Expense Policy, with itemised receipts."} Save that as golden.jsonl . Then run your real pipeline on each question and record what it retrieved and what it answered. Ragas evaluates outputs; it doesn't run your app for you. python collect.py - run the pipeline under test and save its outputs import json from my rag import answer question your app: returns {"answer": str, "contexts": str, ... } def collect config: str, path: str = "golden.jsonl" - list dict : rows = for line in open path : q = json.loads line out = answer question q "user input" , config=config rows.append { q, "response": out "answer" , "retrieved contexts": out "contexts" , the chunk texts, in rank order } return rows Keep retrieved contexts in the order your retriever returned them. Context precision depends on that order. If you have no questions yet, Ragas also has a TestsetGenerator that builds synthetic questions from your documents. Treat those as a starting point and add real user questions as soon as you have them. python eval rag.py import asyncio import json import statistics import sys from openai import AsyncOpenAI from ragas.embeddings import embedding factory from ragas.llms import llm factory from ragas.metrics.collections import AnswerRelevancy, ContextPrecision, ContextRecall, Faithfulness, from collect import collect client = AsyncOpenAI llm = llm factory "gpt-4o-mini", client=client the judge model emb = embedding factory "openai", model="text-embedding-3-small", client=client faithfulness = Faithfulness llm=llm precision = ContextPrecision llm=llm recall = ContextRecall llm=llm relevancy = AnswerRelevancy llm=llm, embeddings=emb METRICS = "faithfulness", "context precision", "context recall", "answer relevancy" async def score row row: dict - dict: q, ctx = row "user input" , row "retrieved contexts" f, p, r, a = await asyncio.gather faithfulness.ascore user input=q, response=row "response" , retrieved contexts=ctx , precision.ascore user input=q, reference=row "reference" , retrieved contexts=ctx , recall.ascore user input=q, retrieved contexts=ctx, reference=row "reference" , relevancy.ascore user input=q, response=row "response" , return {"id": row "id" , "faithfulness": f.value, "context precision": p.value, "context recall": r.value, "answer relevancy": a.value} async def main config: str - None: rows = collect config scores = await asyncio.gather score row r for r in rows summary = {m: round statistics.mean s m for s in scores , 3 for m in METRICS} json.dump {"config": config, "summary": summary, "rows": scores}, open f"results-{config}.json", "w" , indent=2 print config, summary if name == " main ": asyncio.run main sys.argv 1 Run it: python eval rag.py baseline . You get a summary per metric plus per-question scores in results-baseline.json . For a single spot check there's also a synchronous metric.score ... , and MetricResult.reason is worth printing when a score surprises you. Most existing tutorials and the "Evaluate a simple RAG system" guide use evaluate with an EvaluationDataset . It still runs in 0.4 with a deprecation warning but expects the legacy metric classes from ragas.metrics , not the collections ones: python from ragas import EvaluationDataset, evaluate from ragas.metrics import Faithfulness, LLMContextPrecisionWithReference, LLMContextRecall, ResponseRelevancy, dataset = EvaluationDataset.from list collect "baseline" user input, retrieved contexts, response, reference result = evaluate dataset=dataset, metrics= LLMContextRecall , LLMContextPrecisionWithReference , Faithfulness , ResponseRelevancy , llm=llm, embeddings=emb, print result mean score per metric df = result.to pandas one row per question For new code, use the collections metrics above or wrap them in @experiment , which also saves each run to CSV under experiments/ . Keep evaluate for existing code until you migrate. The value of four metrics is that the pattern tells you where to look: Don't stop at the averages. Sort rows by the lowest score and read those questions. That's where the fix usually is. The Ragas "Evaluate and Improve a RAG App" guide follows exactly this loop: its baseline BM25 RAG passed 43 of 66 questions 65.2% , the error analysis showed retrieval was the bottleneck, and an agentic retrieval version passed 58 of 66 87.9% . Example workflow hypothetical . You want to compare your current 800-token chunks with Docling's heading-aware chunks. Same golden set, same judge model, one change: python eval rag.py chunks-800 python eval rag.py docling-hybrid python compare.py import json a = json.load open "results-chunks-800.json" b = json.load open "results-docling-hybrid.json" for m in a "summary" : print f"{m:18} {a 'summary' m :.3f} - {b 'summary' m :.3f}" per-question regressions are more useful than the averages by id = {r "id" : r for r in a "rows" } for r in b "rows" : drop = by id r "id" "context recall" - r "context recall" if drop 0.2: print "recall dropped:", r "id" , round drop, 2 Change one thing at a time the Ragas experimentation docs say the same , otherwise you won't know which change moved the score. Once you have a baseline, make the eval run on every pull request that touches retrieval, chunking or prompts: python check regression.py import json, sys TOLERANCE = 0.05 my choice; tune it once you know how noisy your scores are base = json.load open "baseline.json" "summary" new = json.load open sys.argv 1 "summary" failed = m for m in base if new m < base m - TOLERANCE for m in base: print f"{m:18} base {base m :.3f} new {new m :.3f}" sys.exit 1 if failed else 0 .github/workflows/rag-eval.yml excerpt on: pull request: paths: "rag/ ", "prompts/ " jobs: eval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: pip install -r requirements.txt pin the ragas version here - run: python eval rag.py candidate env: OPENAI API KEY: ${{ secrets.OPENAI API KEY }} RAGAS DO NOT TRACK: "true" - run: python check regression.py results-candidate.json Commit baseline.json and update it on purpose when you accept a new baseline. To save money on reruns, Ragas supports exact-match caching: llm factory "gpt-4o-mini", client=client, cache=DiskCacheBackend with from ragas.cache import DiskCacheBackend . ground truths , instructor llm factory or single turn ascore , you're reading pre-0.4 code reference , a single string . Change one thing. Rerun the same questions. Let the numbers argue with your gut. How do you check today whether a RAG change made things better or worse?