{"slug": "ragas-how-to-actually-measure-your-rag-pipeline-faithfulness-context-precision", "title": "Ragas: how to actually measure your RAG pipeline (faithfulness, context precision/recall, relevancy)", "summary": "A developer published a hands-on guide to using Ragas 0.4.x, an Apache-2.0 open-source library for evaluating RAG and LLM applications, to replace subjective \"vibe checks\" with trackable metrics. The guide covers four LLM-judged metrics — context recall, context precision, faithfulness and answer relevancy — each scored 0 to 1 and targeting a distinct pipeline failure mode, and walks through building a 20-50 question golden dataset with reference answers. It also flags breaking API changes from 0.3 to 0.4, including async metrics in ragas.metrics.collections, MetricResult return values, llm_factory replacing LangchainLLMWrapper, and the @experiment decorator superseding evaluate().", "body_md": "Thursday AI Tool/Workflow, part two. This morning was about parsing PDFs cleanly with Docling. This afternoon: **Ragas**, the open-source (Apache-2.0) library for evaluating RAG and LLM apps, so you can tell whether a change actually made your pipeline better.\n\nMost RAG tuning goes like this: change the chunk size, ask the bot three questions, decide it \"feels better\", ship. Then a week later someone finds the question that broke. Ragas replaces the vibe check with numbers you can track: did retrieval find the right context, did the answer stick to that context, did it actually answer the question?\n\nHere's exactly how to use it on a real pipeline.\n\nRagas has changed a lot, and many tutorials online use APIs that are now deprecated. Everything below targets **Ragas 0.4.x** (0.4.3 is the latest release on PyPI as I write). The main changes from 0.3 to 0.4, per the official migration guide:\n\n`ragas.metrics.collections` and are called with keyword arguments: `await metric.ascore(user_input=..., response=...)`.` MetricResult` (`.value` plus an optional `.reason`) instead of a bare float.` llm_factory(\"model-name\", client=...)`. `LangchainLLMWrapper` and `LlamaIndexLLMWrapper` are deprecated.`evaluate()` still works but is deprecated in favour of the `@experiment` decorator. The old `SingleTurnSample` + `single_turn_ascore()` pattern is legacy.\nThe project also moved on GitHub: `explodinggradients/ragas` now points to `vibrantlabsai/ragas`. Docs are still at docs.ragas.io.\n\n```\npip install ragas openai\n```\n\nThe default examples use OpenAI as the judge model, so set `OPENAI_API_KEY`. `llm_factory` also supports other providers (Anthropic, Google, Azure, Bedrock and more) by passing that provider's client.\n\nOne thing worth knowing: Ragas collects minimal anonymized usage analytics. To opt out, set `RAGAS_DO_NOT_TRACK=true`.\n\nEach metric looks at a different part of the pipeline, which is what makes them useful for debugging. All of them score from 0 to 1, higher is better, and all four use an LLM as the judge.\n\n| Metric | Plain-words question | Needs | Points at | \n|---|---|---|---|\n| **Context recall** | Did we retrieve everything needed to answer? | `user_input` ,`retrieved_contexts` ,`reference` | Retrieval missing info | \n| **Context precision** | Are the useful chunks ranked at the top? | `user_input` ,`retrieved_contexts` ,`reference` | Retrieval noise / ranking | \n| **Faithfulness** | Is every claim in the answer backed by the retrieved context? | `user_input` ,`response` ,`retrieved_contexts` | Generation making things up | \n| **Answer relevancy** | Does the answer address the question that was asked? | `user_input` ,`response` (+ embeddings) | Off-topic or padded answers | \n\nHow they're computed, from the docs:\n\n`strictness`) and averages their embedding cosine similarity to the real question. It deliberately does You don't need a giant benchmark. Start with 20 to 50 real questions your users actually ask, each with a short reference answer written by someone who knows the docs. That's the slow, valuable part.\n\n*Example (hypothetical): an HR policy bot, built on the Docling chunks from this morning's post.*\n\n```\n{\"id\": \"leave-01\", \"user_input\": \"How many unused leave days can I carry over?\", \"reference\": \"Up to 10 days of unused annual leave carry over to the next year.\"}\n{\"id\": \"leave-02\", \"user_input\": \"Do carried-over days expire?\", \"reference\": \"Carried-over days must be used by 31 March or they lapse.\"}\n{\"id\": \"exp-01\", \"user_input\": \"What's the limit for client dinners?\", \"reference\": \"Client dinners are reimbursed up to the per-person limit in the Expense Policy, with itemised receipts.\"}\n```\n\nSave that as `golden.jsonl`. Then run your *real* pipeline on each question and record what it retrieved and what it answered. Ragas evaluates outputs; it doesn't run your app for you.\n\n``` python\n# collect.py - run the pipeline under test and save its outputs\nimport json\nfrom my_rag import answer_question  # your app: returns {\"answer\": str, \"contexts\": [str, ...]}\n\ndef collect(config: str, path: str = \"golden.jsonl\") -> list[dict]:\n    rows = []\n    for line in open(path):\n        q = json.loads(line)\n        out = answer_question(q[\"user_input\"], config=config)\n        rows.append({\n            **q,\n            \"response\": out[\"answer\"],\n            \"retrieved_contexts\": out[\"contexts\"],  # the chunk texts, in rank order\n        })\n    return rows\n```\n\nKeep `retrieved_contexts` in the order your retriever returned them. Context precision depends on that order.\n\nIf you have no questions yet, Ragas also has a `TestsetGenerator` that builds synthetic questions from your documents. Treat those as a starting point and add real user questions as soon as you have them.\n\n``` python\n# eval_rag.py\nimport asyncio\nimport json\nimport statistics\nimport sys\n\nfrom openai import AsyncOpenAI\nfrom ragas.embeddings import embedding_factory\nfrom ragas.llms import llm_factory\nfrom ragas.metrics.collections import (\n    AnswerRelevancy,\n    ContextPrecision,\n    ContextRecall,\n    Faithfulness,\n)\n\nfrom collect import collect\n\nclient = AsyncOpenAI()\nllm = llm_factory(\"gpt-4o-mini\", client=client)  # the judge model\nemb = embedding_factory(\"openai\", model=\"text-embedding-3-small\", client=client)\n\nfaithfulness = Faithfulness(llm=llm)\nprecision = ContextPrecision(llm=llm)\nrecall = ContextRecall(llm=llm)\nrelevancy = AnswerRelevancy(llm=llm, embeddings=emb)\n\nMETRICS = [\"faithfulness\", \"context_precision\", \"context_recall\", \"answer_relevancy\"]\n\nasync def score_row(row: dict) -> dict:\n    q, ctx = row[\"user_input\"], row[\"retrieved_contexts\"]\n    f, p, r, a = await asyncio.gather(\n        faithfulness.ascore(user_input=q, response=row[\"response\"], retrieved_contexts=ctx),\n        precision.ascore(user_input=q, reference=row[\"reference\"], retrieved_contexts=ctx),\n        recall.ascore(user_input=q, retrieved_contexts=ctx, reference=row[\"reference\"]),\n        relevancy.ascore(user_input=q, response=row[\"response\"]),\n    )\n    return {\"id\": row[\"id\"], \"faithfulness\": f.value, \"context_precision\": p.value,\n            \"context_recall\": r.value, \"answer_relevancy\": a.value}\n\nasync def main(config: str) -> None:\n    rows = collect(config)\n    scores = await asyncio.gather(*(score_row(r) for r in rows))\n    summary = {m: round(statistics.mean(s[m] for s in scores), 3) for m in METRICS}\n    json.dump({\"config\": config, \"summary\": summary, \"rows\": scores},\n              open(f\"results-{config}.json\", \"w\"), indent=2)\n    print(config, summary)\n\nif __name__ == \"__main__\":\n    asyncio.run(main(sys.argv[1]))\n```\n\nRun it: `python eval_rag.py baseline`. You get a summary per metric plus per-question scores in `results-baseline.json`. For a single spot check there's also a synchronous `metric.score(...)`, and `MetricResult.reason` is worth printing when a score surprises you.\n\nMost existing tutorials and the \"Evaluate a simple RAG system\" guide use `evaluate()` with an `EvaluationDataset`. It still runs in 0.4 (with a deprecation warning) but expects the **legacy** metric classes from `ragas.metrics`, not the collections ones:\n\n``` python\nfrom ragas import EvaluationDataset, evaluate\nfrom ragas.metrics import (\n    Faithfulness,\n    LLMContextPrecisionWithReference,\n    LLMContextRecall,\n    ResponseRelevancy,\n)\n\ndataset = EvaluationDataset.from_list(collect(\"baseline\"))  # user_input, retrieved_contexts, response, reference\nresult = evaluate(\n    dataset=dataset,\n    metrics=[LLMContextRecall(), LLMContextPrecisionWithReference(), Faithfulness(), ResponseRelevancy()],\n    llm=llm,\n    embeddings=emb,\n)\nprint(result)               # mean score per metric\ndf = result.to_pandas()     # one row per question\n```\n\nFor new code, use the collections metrics above (or wrap them in `@experiment`, which also saves each run to CSV under `experiments/`). Keep `evaluate()` for existing code until you migrate.\n\nThe value of four metrics is that the *pattern* tells you where to look:\n\nDon't stop at the averages. Sort `rows` by the lowest score and read those questions. That's where the fix usually is. The Ragas \"Evaluate and Improve a RAG App\" guide follows exactly this loop: its baseline BM25 RAG passed 43 of 66 questions (65.2%), the error analysis showed retrieval was the bottleneck, and an agentic retrieval version passed 58 of 66 (87.9%).\n\n*Example workflow (hypothetical).* You want to compare your current 800-token chunks with Docling's heading-aware chunks. Same golden set, same judge model, one change:\n\n```\npython eval_rag.py chunks-800\npython eval_rag.py docling-hybrid\npython\n# compare.py\nimport json\na = json.load(open(\"results-chunks-800.json\"))\nb = json.load(open(\"results-docling-hybrid.json\"))\n\nfor m in a[\"summary\"]:\n    print(f\"{m:18} {a['summary'][m]:.3f} -> {b['summary'][m]:.3f}\")\n\n# per-question regressions are more useful than the averages\nby_id = {r[\"id\"]: r for r in a[\"rows\"]}\nfor r in b[\"rows\"]:\n    drop = by_id[r[\"id\"]][\"context_recall\"] - r[\"context_recall\"]\n    if drop > 0.2:\n        print(\"recall dropped:\", r[\"id\"], round(drop, 2))\n```\n\nChange one thing at a time (the Ragas experimentation docs say the same), otherwise you won't know which change moved the score.\n\nOnce you have a baseline, make the eval run on every pull request that touches retrieval, chunking or prompts:\n\n``` python\n# check_regression.py\nimport json, sys\nTOLERANCE = 0.05  # my choice; tune it once you know how noisy your scores are\n\nbase = json.load(open(\"baseline.json\"))[\"summary\"]\nnew = json.load(open(sys.argv[1]))[\"summary\"]\nfailed = [m for m in base if new[m] < base[m] - TOLERANCE]\nfor m in base:\n    print(f\"{m:18} base {base[m]:.3f}  new {new[m]:.3f}\")\nsys.exit(1 if failed else 0)\n# .github/workflows/rag-eval.yml (excerpt)\non:\n  pull_request:\n    paths: [\"rag/**\", \"prompts/**\"]\njobs:\n  eval:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - run: pip install -r requirements.txt   # pin the ragas version here\n      - run: python eval_rag.py candidate\n        env:\n          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n          RAGAS_DO_NOT_TRACK: \"true\"\n      - run: python check_regression.py results-candidate.json\n```\n\nCommit `baseline.json` and update it on purpose when you accept a new baseline. To save money on reruns, Ragas supports exact-match caching: `llm_factory(\"gpt-4o-mini\", client=client, cache=DiskCacheBackend())` with `from ragas.cache import DiskCacheBackend`.\n\n`ground_truths`, `instructor_llm_factory` or `single_turn_ascore`, you're reading pre-0.4 code (` reference`, a single string).\nChange one thing. Rerun the same questions. Let the numbers argue with your gut.\n\n*How do you check today whether a RAG change made things better or worse?*", "url": "https://wpnews.pro/news/ragas-how-to-actually-measure-your-rag-pipeline-faithfulness-context-precision", "canonical_source": "https://dev.to/m_t_ramkrushna/ragas-how-to-actually-measure-your-rag-pipeline-faithfulness-context-precisionrecall-relevancy-26", "published_at": "2026-10-29 09:09:21+00:00", "updated_at": "2026-10-29 09:17:06.146298+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "artificial-intelligence", "ai-agents", "developer-tools"], "entities": ["Ragas", "Docling", "OpenAI", "Anthropic", "Google", "Azure", "Amazon Bedrock", "vibrantlabsai"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ragas-how-to-actually-measure-your-rag-pipeline-faithfulness-context-precision", "markdown": "https://wpnews.pro/news/ragas-how-to-actually-measure-your-rag-pipeline-faithfulness-context-precision.md", "text": "https://wpnews.pro/news/ragas-how-to-actually-measure-your-rag-pipeline-faithfulness-context-precision.txt", "jsonld": "https://wpnews.pro/news/ragas-how-to-actually-measure-your-rag-pipeline-faithfulness-context-precision.jsonld"}}