# Ragas: how to actually measure your RAG pipeline (faithfulness, context precision/recall, relevancy)

> Source: <https://dev.to/m_t_ramkrushna/ragas-how-to-actually-measure-your-rag-pipeline-faithfulness-context-precisionrecall-relevancy-26>
> Published: 2026-10-29 09:09:21+00:00

Thursday AI Tool/Workflow, part two. This morning was about parsing PDFs cleanly with Docling. This afternoon: **Ragas**, the open-source (Apache-2.0) library for evaluating RAG and LLM apps, so you can tell whether a change actually made your pipeline better.

Most RAG tuning goes like this: change the chunk size, ask the bot three questions, decide it "feels better", ship. Then a week later someone finds the question that broke. Ragas replaces the vibe check with numbers you can track: did retrieval find the right context, did the answer stick to that context, did it actually answer the question?

Here's exactly how to use it on a real pipeline.

Ragas has changed a lot, and many tutorials online use APIs that are now deprecated. Everything below targets **Ragas 0.4.x** (0.4.3 is the latest release on PyPI as I write). The main changes from 0.3 to 0.4, per the official migration guide:

`ragas.metrics.collections` and are called with keyword arguments: `await metric.ascore(user_input=..., response=...)`.` MetricResult` (`.value` plus an optional `.reason`) instead of a bare float.` llm_factory("model-name", client=...)`. `LangchainLLMWrapper` and `LlamaIndexLLMWrapper` are deprecated.`evaluate()` still works but is deprecated in favour of the `@experiment` decorator. The old `SingleTurnSample` + `single_turn_ascore()` pattern is legacy.
The project also moved on GitHub: `explodinggradients/ragas` now points to `vibrantlabsai/ragas`. Docs are still at docs.ragas.io.

```
pip install ragas openai
```

The default examples use OpenAI as the judge model, so set `OPENAI_API_KEY`. `llm_factory` also supports other providers (Anthropic, Google, Azure, Bedrock and more) by passing that provider's client.

One thing worth knowing: Ragas collects minimal anonymized usage analytics. To opt out, set `RAGAS_DO_NOT_TRACK=true`.

Each metric looks at a different part of the pipeline, which is what makes them useful for debugging. All of them score from 0 to 1, higher is better, and all four use an LLM as the judge.

| Metric | Plain-words question | Needs | Points at | 
|---|---|---|---|
| **Context recall** | Did we retrieve everything needed to answer? | `user_input` ,`retrieved_contexts` ,`reference` | Retrieval missing info | 
| **Context precision** | Are the useful chunks ranked at the top? | `user_input` ,`retrieved_contexts` ,`reference` | Retrieval noise / ranking | 
| **Faithfulness** | Is every claim in the answer backed by the retrieved context? | `user_input` ,`response` ,`retrieved_contexts` | Generation making things up | 
| **Answer relevancy** | Does the answer address the question that was asked? | `user_input` ,`response` (+ embeddings) | Off-topic or padded answers | 

How they're computed, from the docs:

`strictness`) and averages their embedding cosine similarity to the real question. It deliberately does You don't need a giant benchmark. Start with 20 to 50 real questions your users actually ask, each with a short reference answer written by someone who knows the docs. That's the slow, valuable part.

*Example (hypothetical): an HR policy bot, built on the Docling chunks from this morning's post.*

```
{"id": "leave-01", "user_input": "How many unused leave days can I carry over?", "reference": "Up to 10 days of unused annual leave carry over to the next year."}
{"id": "leave-02", "user_input": "Do carried-over days expire?", "reference": "Carried-over days must be used by 31 March or they lapse."}
{"id": "exp-01", "user_input": "What's the limit for client dinners?", "reference": "Client dinners are reimbursed up to the per-person limit in the Expense Policy, with itemised receipts."}
```

Save that as `golden.jsonl`. Then run your *real* pipeline on each question and record what it retrieved and what it answered. Ragas evaluates outputs; it doesn't run your app for you.

``` python
# collect.py - run the pipeline under test and save its outputs
import json
from my_rag import answer_question  # your app: returns {"answer": str, "contexts": [str, ...]}

def collect(config: str, path: str = "golden.jsonl") -> list[dict]:
    rows = []
    for line in open(path):
        q = json.loads(line)
        out = answer_question(q["user_input"], config=config)
        rows.append({
            **q,
            "response": out["answer"],
            "retrieved_contexts": out["contexts"],  # the chunk texts, in rank order
        })
    return rows
```

Keep `retrieved_contexts` in the order your retriever returned them. Context precision depends on that order.

If you have no questions yet, Ragas also has a `TestsetGenerator` that builds synthetic questions from your documents. Treat those as a starting point and add real user questions as soon as you have them.

``` python
# eval_rag.py
import asyncio
import json
import statistics
import sys

from openai import AsyncOpenAI
from ragas.embeddings import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import (
    AnswerRelevancy,
    ContextPrecision,
    ContextRecall,
    Faithfulness,
)

from collect import collect

client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)  # the judge model
emb = embedding_factory("openai", model="text-embedding-3-small", client=client)

faithfulness = Faithfulness(llm=llm)
precision = ContextPrecision(llm=llm)
recall = ContextRecall(llm=llm)
relevancy = AnswerRelevancy(llm=llm, embeddings=emb)

METRICS = ["faithfulness", "context_precision", "context_recall", "answer_relevancy"]

async def score_row(row: dict) -> dict:
    q, ctx = row["user_input"], row["retrieved_contexts"]
    f, p, r, a = await asyncio.gather(
        faithfulness.ascore(user_input=q, response=row["response"], retrieved_contexts=ctx),
        precision.ascore(user_input=q, reference=row["reference"], retrieved_contexts=ctx),
        recall.ascore(user_input=q, retrieved_contexts=ctx, reference=row["reference"]),
        relevancy.ascore(user_input=q, response=row["response"]),
    )
    return {"id": row["id"], "faithfulness": f.value, "context_precision": p.value,
            "context_recall": r.value, "answer_relevancy": a.value}

async def main(config: str) -> None:
    rows = collect(config)
    scores = await asyncio.gather(*(score_row(r) for r in rows))
    summary = {m: round(statistics.mean(s[m] for s in scores), 3) for m in METRICS}
    json.dump({"config": config, "summary": summary, "rows": scores},
              open(f"results-{config}.json", "w"), indent=2)
    print(config, summary)

if __name__ == "__main__":
    asyncio.run(main(sys.argv[1]))
```

Run it: `python eval_rag.py baseline`. You get a summary per metric plus per-question scores in `results-baseline.json`. For a single spot check there's also a synchronous `metric.score(...)`, and `MetricResult.reason` is worth printing when a score surprises you.

Most existing tutorials and the "Evaluate a simple RAG system" guide use `evaluate()` with an `EvaluationDataset`. It still runs in 0.4 (with a deprecation warning) but expects the **legacy** metric classes from `ragas.metrics`, not the collections ones:

``` python
from ragas import EvaluationDataset, evaluate
from ragas.metrics import (
    Faithfulness,
    LLMContextPrecisionWithReference,
    LLMContextRecall,
    ResponseRelevancy,
)

dataset = EvaluationDataset.from_list(collect("baseline"))  # user_input, retrieved_contexts, response, reference
result = evaluate(
    dataset=dataset,
    metrics=[LLMContextRecall(), LLMContextPrecisionWithReference(), Faithfulness(), ResponseRelevancy()],
    llm=llm,
    embeddings=emb,
)
print(result)               # mean score per metric
df = result.to_pandas()     # one row per question
```

For new code, use the collections metrics above (or wrap them in `@experiment`, which also saves each run to CSV under `experiments/`). Keep `evaluate()` for existing code until you migrate.

The value of four metrics is that the *pattern* tells you where to look:

Don't stop at the averages. Sort `rows` by the lowest score and read those questions. That's where the fix usually is. The Ragas "Evaluate and Improve a RAG App" guide follows exactly this loop: its baseline BM25 RAG passed 43 of 66 questions (65.2%), the error analysis showed retrieval was the bottleneck, and an agentic retrieval version passed 58 of 66 (87.9%).

*Example workflow (hypothetical).* You want to compare your current 800-token chunks with Docling's heading-aware chunks. Same golden set, same judge model, one change:

```
python eval_rag.py chunks-800
python eval_rag.py docling-hybrid
python
# compare.py
import json
a = json.load(open("results-chunks-800.json"))
b = json.load(open("results-docling-hybrid.json"))

for m in a["summary"]:
    print(f"{m:18} {a['summary'][m]:.3f} -> {b['summary'][m]:.3f}")

# per-question regressions are more useful than the averages
by_id = {r["id"]: r for r in a["rows"]}
for r in b["rows"]:
    drop = by_id[r["id"]]["context_recall"] - r["context_recall"]
    if drop > 0.2:
        print("recall dropped:", r["id"], round(drop, 2))
```

Change one thing at a time (the Ragas experimentation docs say the same), otherwise you won't know which change moved the score.

Once you have a baseline, make the eval run on every pull request that touches retrieval, chunking or prompts:

``` python
# check_regression.py
import json, sys
TOLERANCE = 0.05  # my choice; tune it once you know how noisy your scores are

base = json.load(open("baseline.json"))["summary"]
new = json.load(open(sys.argv[1]))["summary"]
failed = [m for m in base if new[m] < base[m] - TOLERANCE]
for m in base:
    print(f"{m:18} base {base[m]:.3f}  new {new[m]:.3f}")
sys.exit(1 if failed else 0)
# .github/workflows/rag-eval.yml (excerpt)
on:
  pull_request:
    paths: ["rag/**", "prompts/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt   # pin the ragas version here
      - run: python eval_rag.py candidate
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          RAGAS_DO_NOT_TRACK: "true"
      - run: python check_regression.py results-candidate.json
```

Commit `baseline.json` and update it on purpose when you accept a new baseline. To save money on reruns, Ragas supports exact-match caching: `llm_factory("gpt-4o-mini", client=client, cache=DiskCacheBackend())` with `from ragas.cache import DiskCacheBackend`.

`ground_truths`, `instructor_llm_factory` or `single_turn_ascore`, you're reading pre-0.4 code (` reference`, a single string).
Change one thing. Rerun the same questions. Let the numbers argue with your gut.

*How do you check today whether a RAG change made things better or worse?*