Ragas: how to actually measure your RAG pipeline (faithfulness, context precision/recall, relevancy)

Thursday AI Tool/Workflow, part two. This morning was about parsing PDFs cleanly with Docling. This afternoon: Ragas, the open-source (Apache-2.0) library for evaluating RAG and LLM apps, so you can tell whether a change actually made your pipeline better.

Most RAG tuning goes like this: change the chunk size, ask the bot three questions, decide it "feels better", ship. Then a week later someone finds the question that broke. Ragas replaces the vibe check with numbers you can track: did retrieval find the right context, did the answer stick to that context, did it actually answer the question?

Here's exactly how to use it on a real pipeline.

Ragas has changed a lot, and many tutorials online use APIs that are now deprecated. Everything below targets Ragas 0.4.x (0.4.3 is the latest release on PyPI as I write). The main changes from 0.3 to 0.4, per the official migration guide:

ragas.metrics.collections and are called with keyword arguments: await metric.ascore(user_input=..., response=...). MetricResult (.value plus an optional .reason) instead of a bare float. llm_factory("model-name", client=...). LangchainLLMWrapper and LlamaIndexLLMWrapper are deprecated.evaluate() still works but is deprecated in favour of the @experiment decorator. The old SingleTurnSample + single_turn_ascore() pattern is legacy. The project also moved on GitHub: explodinggradients/ragas now points to vibrantlabsai/ragas. Docs are still at docs.ragas.io.

pip install ragas openai

The default examples use OpenAI as the judge model, so set OPENAI_API_KEY. llm_factory also supports other providers (Anthropic, Google, Azure, Bedrock and more) by passing that provider's client.

One thing worth knowing: Ragas collects minimal anonymized usage analytics. To opt out, set RAGAS_DO_NOT_TRACK=true.

Each metric looks at a different part of the pipeline, which is what makes them useful for debugging. All of them score from 0 to 1, higher is better, and all four use an LLM as the judge.

How they're computed, from the docs:

strictness) and averages their embedding cosine similarity to the real question. It deliberately does You don't need a giant benchmark. Start with 20 to 50 real questions your users actually ask, each with a short reference answer written by someone who knows the docs. That's the slow, valuable part.

Example (hypothetical): an HR policy bot, built on the Docling chunks from this morning's post.

{"id": "leave-01", "user_input": "How many unused leave days can I carry over?", "reference": "Up to 10 days of unused annual leave carry over to the next year."}
{"id": "leave-02", "user_input": "Do carried-over days expire?", "reference": "Carried-over days must be used by 31 March or they lapse."}
{"id": "exp-01", "user_input": "What's the limit for client dinners?", "reference": "Client dinners are reimbursed up to the per-person limit in the Expense Policy, with itemised receipts."}

Save that as golden.jsonl. Then run your real pipeline on each question and record what it retrieved and what it answered. Ragas evaluates outputs; it doesn't run your app for you.

import json
from my_rag import answer_question  # your app: returns {"answer": str, "contexts": [str, ...]}

def collect(config: str, path: str = "golden.jsonl") -> list[dict]:
    rows = []
    for line in open(path):
        q = json.loads(line)
        out = answer_question(q["user_input"], config=config)
        rows.append({
            **q,
            "response": out["answer"],
            "retrieved_contexts": out["contexts"],  # the chunk texts, in rank order
        })
    return rows

Keep retrieved_contexts in the order your retriever returned them. Context precision depends on that order.

If you have no questions yet, Ragas also has a TestsetGenerator that builds synthetic questions from your documents. Treat those as a starting point and add real user questions as soon as you have them.

import asyncio
import json
import statistics
import sys

from openai import AsyncOpenAI
from ragas.embeddings import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import (
    AnswerRelevancy,
    ContextPrecision,
    ContextRecall,
    Faithfulness,
)

from collect import collect

client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)  # the judge model
emb = embedding_factory("openai", model="text-embedding-3-small", client=client)

faithfulness = Faithfulness(llm=llm)
precision = ContextPrecision(llm=llm)
recall = ContextRecall(llm=llm)
relevancy = AnswerRelevancy(llm=llm, embeddings=emb)

METRICS = ["faithfulness", "context_precision", "context_recall", "answer_relevancy"]

async def score_row(row: dict) -> dict:
    q, ctx = row["user_input"], row["retrieved_contexts"]
    f, p, r, a = await asyncio.gather(
        faithfulness.ascore(user_input=q, response=row["response"], retrieved_contexts=ctx),
        precision.ascore(user_input=q, reference=row["reference"], retrieved_contexts=ctx),
        recall.ascore(user_input=q, retrieved_contexts=ctx, reference=row["reference"]),
        relevancy.ascore(user_input=q, response=row["response"]),
    )
    return {"id": row["id"], "faithfulness": f.value, "context_precision": p.value,
            "context_recall": r.value, "answer_relevancy": a.value}

async def main(config: str) -> None:
    rows = collect(config)
    scores = await asyncio.gather(*(score_row(r) for r in rows))
    summary = {m: round(statistics.mean(s[m] for s in scores), 3) for m in METRICS}
    json.dump({"config": config, "summary": summary, "rows": scores},
              open(f"results-{config}.json", "w"), indent=2)
    print(config, summary)

if __name__ == "__main__":
    asyncio.run(main(sys.argv[1]))

Run it: python eval_rag.py baseline. You get a summary per metric plus per-question scores in results-baseline.json. For a single spot check there's also a synchronous metric.score(...), and MetricResult.reason is worth printing when a score surprises you.

Most existing tutorials and the "Evaluate a simple RAG system" guide use evaluate() with an EvaluationDataset. It still runs in 0.4 (with a deprecation warning) but expects the legacy metric classes from ragas.metrics, not the collections ones:

from ragas import EvaluationDataset, evaluate
from ragas.metrics import (
    Faithfulness,
    LLMContextPrecisionWithReference,
    LLMContextRecall,
    ResponseRelevancy,
)

dataset = EvaluationDataset.from_list(collect("baseline"))  # user_input, retrieved_contexts, response, reference
result = evaluate(
    dataset=dataset,
    metrics=[LLMContextRecall(), LLMContextPrecisionWithReference(), Faithfulness(), ResponseRelevancy()],
    llm=llm,
    embeddings=emb,
)
print(result)               # mean score per metric
df = result.to_pandas()     # one row per question

For new code, use the collections metrics above (or wrap them in @experiment, which also saves each run to CSV under experiments/). Keep evaluate() for existing code until you migrate.

The value of four metrics is that the pattern tells you where to look:

Don't stop at the averages. Sort rows by the lowest score and read those questions. That's where the fix usually is. The Ragas "Evaluate and Improve a RAG App" guide follows exactly this loop: its baseline BM25 RAG passed 43 of 66 questions (65.2%), the error analysis showed retrieval was the bottleneck, and an agentic retrieval version passed 58 of 66 (87.9%).

Example workflow (hypothetical). You want to compare your current 800-token chunks with Docling's heading-aware chunks. Same golden set, same judge model, one change:

python eval_rag.py chunks-800
python eval_rag.py docling-hybrid
python
import json
a = json.load(open("results-chunks-800.json"))
b = json.load(open("results-docling-hybrid.json"))

for m in a["summary"]:
    print(f"{m:18} {a['summary'][m]:.3f} -> {b['summary'][m]:.3f}")

by_id = {r["id"]: r for r in a["rows"]}
for r in b["rows"]:
    drop = by_id[r["id"]]["context_recall"] - r["context_recall"]
    if drop > 0.2:
        print("recall dropped:", r["id"], round(drop, 2))

Change one thing at a time (the Ragas experimentation docs say the same), otherwise you won't know which change moved the score.

Once you have a baseline, make the eval run on every pull request that touches retrieval, chunking or prompts:

import json, sys
TOLERANCE = 0.05  # my choice; tune it once you know how noisy your scores are

base = json.load(open("baseline.json"))["summary"]
new = json.load(open(sys.argv[1]))["summary"]
failed = [m for m in base if new[m] < base[m] - TOLERANCE]
for m in base:
    print(f"{m:18} base {base[m]:.3f}  new {new[m]:.3f}")
sys.exit(1 if failed else 0)
on:
  pull_request:
    paths: ["rag/**", "prompts/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt   # pin the ragas version here
      - run: python eval_rag.py candidate
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          RAGAS_DO_NOT_TRACK: "true"
      - run: python check_regression.py results-candidate.json

Commit baseline.json and update it on purpose when you accept a new baseline. To save money on reruns, Ragas supports exact-match caching: llm_factory("gpt-4o-mini", client=client, cache=DiskCacheBackend()) with from ragas.cache import DiskCacheBackend.

ground_truths, instructor_llm_factory or single_turn_ascore, you're reading pre-0.4 code ( reference, a single string). Change one thing. Rerun the same questions. Let the numbers argue with your gut.

How do you check today whether a RAG change made things better or worse?