RAG evaluation metrics, frameworks and benchmarks compared, how far to trust an LLM judge, and how to test a retriever or model change in a RAG pipeline.
RAG evaluation measures a retrieval-augmented generation pipeline in two halves, then end to end. The retrieval half asks whether the right chunks came back: recall@k, precision@k, MRR and nDCG against labeled relevant documents, or LLM-judged context precision and context recall when you have no labels. The generation half asks whether the answer used those chunks well: faithfulness (every claim supported by the retrieved context), answer relevancy (it addresses the question) and correctness against a reference answer. Score the halves separately, because an end-to-end score of 70% does not tell you whether to fix the retriever or the prompt.
The tools that compute these metrics, Ragas, DeepEval, TruLens, MLflow, Phoenix and Promptfoo among them, agree on the names and disagree on the definitions. This post covers what each metric measures and needs, how the frameworks differ, how far to trust an LLM judge for this, how to build a test set, and how to test a change to one part of the pipeline without the rest moving. It includes a short script, tested on a small fixture set, that compares two retrievers query by query with a confidence interval, a step the top-ranking guides I read leave out.
A disclosure: I build EvalShift, a tool for testing model changes on agents. It has a short section at the end, which says what it does not do for RAG. Everything before it works without it.
Evaluate retrieval and generation separately #
A wrong RAG answer has one of three causes, and each has a different fix:
| What went wrong | What you see | Where the fix is |
|---|---|---|
| The relevant chunk was not retrieved | Low context recall; the answer is wrong or a refusal | Chunking, embedding model, hybrid search, query rewriting, top-k |
| It was retrieved, and the model ignored or misread it | High context recall, low faithfulness or correctness | Prompt, context order, the generator model |
| Nothing relevant exists in the corpus | No retriever can find it; the right answer is "I don't know" | The corpus, and the model's willingness to abstain |
The second row is more common than it sounds. In Lost in the Middle ↗ (Liu et al., TACL 2024), models used information at the start and end of a long context much better than information in the middle: GPT-3.5-Turbo's multi-document QA accuracy dropped by more than 20% when the answer moved to the middle, and with 20 or 30 documents and the answer in the middle it fell below its closed-book score of 56.1%. Retrieving the right chunk is necessary and not sufficient. Raising top-k improves recall and can lower answer quality at the same time.
The third row is the one test sets forget. If every question in your suite has an answer in the corpus, you never measure whether the system admits it does not know. RGB (Chen et al., AAAI 2024 ↗) calls this negative rejection, and found models struggled with it, as with integrating information across documents and with retrieved text that contradicts what they know.
RAG evaluation metrics #
| Metric | Measures | Needs | Model calls |
|---|---|---|---|
| Hit rate (success@k) | Share of queries with at least one relevant chunk in the top k | Relevant chunk IDs per query | None |
| Recall@k | Share of the relevant chunks that are in the top k | Relevant chunk IDs | None |
| Precision@k | Share of the top k that is relevant | Relevant chunk IDs | None |
| MRR | 1 / rank of the first relevant chunk, averaged | Relevant chunk IDs | None |
| nDCG@k | Ranking quality with graded relevance, discounted by position | Graded relevance labels | None |
| Context precision (LLM) | Whether relevant chunks are ranked above irrelevant ones | A reference answer | One judgment per chunk |
| Context recall (LLM) | Share of the reference answer's claims the retrieved context supports | A reference answer | Claim extraction and checks |
| Faithfulness, groundedness | Share of the answer's claims the retrieved context supports | The retrieved context | Claim extraction and checks |
| Answer relevancy | Whether the answer addresses the question | Only the question and answer | One or more calls, sometimes embeddings |
| Answer correctness | Whether the answer matches a reference | A reference answer | Claim-level comparison |
| Noise sensitivity | How often irrelevant retrieved text makes the answer wrong | A reference answer | Claim extraction and checks |
Retrieval metrics without a model
If you can label which chunks answer each question, the classic information-retrieval metrics are free, deterministic and fast. The definitions below follow the ranx documentation ↗:
- +Recall@k is the relevant chunks in the top k divided by all relevant chunks for the query. This is the metric to watch first, because a chunk the retriever never returned is one the generator cannot use.
- +Precision@k is the relevant chunks in the top k divided by k. It is capped by how many relevant chunks exist: with one relevant chunk and k = 5, the best possible score is 0.2.
- +Hit rate is the share of queries with at least one relevant chunk in the top k. It is the coarsest of these metrics and often the first one reported.
- +MRR averages 1 divided by the rank of the first relevant chunk, so it rewards putting one good chunk at the top and ignores the rest.
- +nDCG@k sums each retrieved chunk's relevance grade divided by log2 of its position plus one, then divides by the best possible ordering's score. It is the only one of these that uses graded labels ("answers the question" vs "helps").
BEIR (Thakur et al., NeurIPS 2021 ↗) reports nDCG@10 on every dataset because precision and recall ignore rank and MRR cannot use graded labels, and MTEB's retrieval tasks use the same main metric. If you report one retrieval number, report nDCG@10, and keep recall@k beside it for the question "did the chunk come back at all".
One caution on labels: eRAG (Salemi and Zamani, SIGIR 2024 ↗) found that query-document relevance labels have only a small correlation with how well the RAG system answers downstream. A chunk a person marks relevant is not always a chunk the model can use. Its fix is to run the generator on each retrieved document alone and use the result as the label, which costs a model call per document.
LLM-judged context metrics
Without labeled chunks, a judge model decides which retrieved chunks are relevant. Ragas Context Precision ↗ judges each chunk against a reference answer and computes a rank-weighted precision, the mean of precision@k over the positions that hold a relevant chunk. Ragas Context Recall ↗ splits the reference answer into claims and reports the share the retrieved context supports. Both need a reference answer but not chunk labels, which makes them cheaper to set up and more expensive to run.
Ragas also has NonLLMContextRecall and NonLLMContextPrecisionWithReference, which compare
retrieved chunks to reference chunks by string similarity, and IDBasedContextPrecision and
IDBasedContextRecall, which compare chunk IDs. Those are the labeled metrics above under other
names, and they make no model call.
Generation metrics
Faithfulness is the generation metric that matters most: the share of the answer's claims the retrieved context supports. It needs no reference answer, so it also works on production traffic. The hallucination detection post covers the cheaper ways to compute it, including small entailment models such as HHEM and MiniCheck that do the per-claim check without a large model.
Answer relevancy checks that the answer addresses the question, not that it is right. An answer can be relevant and wrong, or correct and padded with material nobody asked for.
Answer correctness compares the answer to a reference. Ragas FactualCorrectness splits both
into claims and reports precision, recall or F1 over them. It needs reference answers, which is
the expensive part of a test set.
Noise sensitivity (Ragas) is the share of the answer's claims that are incorrect, so lower is better, and a mode restricts it to errors caused by irrelevant retrieved chunks. It measures what happens when you raise top-k.
Same metric name, different metric #
The libraries reuse each other's names, and the definitions behind them differ in ways that move scores. Read the definition before comparing a number across tools:
| Name | Ragas | DeepEval | Promptfoo |
|---|---|---|---|
| Faithfulness | Claims supported by the context / all claims | Claims that do not contradict the context / all claims, so an unsupported claim the context never mentions passes | context-faithfulness : supported claims / all claims |
| Answer relevancy | Generates questions from the answer (3 by default) and averages their embedding similarity to the real question; the docs say the score is not guaranteed to fall between 0 and 1 | Share of the answer's statements an LLM classifies as relevant to the input; no embeddings | answer-relevance : generates questions from the output, compares them by embedding |
| Context recall | Reference claims the context supports / all reference claims | Statements in expected_output attributable to the context / all statements |
context-recall : attributable statements / all statements |
| Context relevance | Not in the RAG list (the Nvidia metric ContextRelevance rates 0, 1 or 2 twice and averages) |
ContextualRelevancyMetric : relevant statements in the context / all statements |
context-relevance : required sentences / all sentences |
The faithfulness difference is the one that bites. On a support bot that adds a plausible phone number the knowledge base never mentions, Ragas Faithfulness counts that claim against the answer and DeepEval's FaithfulnessMetric does not. Neither is wrong; they answer different questions. Decide which question you are asking before you pick the metric.
Can you trust an LLM judge for RAG metrics #
Partly, and it depends on the metric. The research gives these numbers:
- +The RAGAS paper ↗ (Es et al., EACL 2024 demos) measured agreement with human pairwise judgments on its WikiEval set: 0.95 for faithfulness, 0.78 for answer relevance and 0.70 for context relevance, which it calls "the hardest quality dimension to evaluate". The two human annotators agreed about 95% of the time on faithfulness and context relevance and 90% on answer relevance.
- +ARES ↗ (Saad-Falcon et al., NAACL 2024) fine-tunes small judges on synthetic data and uses prediction-powered inference with about 150 or more human-labeled examples to put confidence intervals on the scores. It reports evaluation accuracy 59.3 points above RAGAS on context relevance and 14.4 points above on answer relevance.
- +RAGChecker ↗ (Ru et al., 2024) checks answers claim by claim and correlated with human ratings at 0.50 to 0.62 (Pearson) across correctness, completeness and overall quality. Two humans correlated at 0.64 to 0.72, so the authors note "still a clear gap".
- +RAGBench ↗ (Friel et al., 2024) found that LLM-based RAG evaluation methods struggled to compete with a fine-tuned RoBERTa model.
The pattern holds across these papers: faithfulness judges are reliable, context relevance judges are not, and small fine-tuned classifiers compete with large judges. Before a judged metric gates a release, label 100 to 200 of your own cases by hand and measure the judge against them, as the LLM-as-a-judge post describes. For context relevance, prefer labeled recall@k and nDCG where you can get labels.
How to build a RAG test set #
A RAG test case needs a question and, depending on the metrics you want, some of: the relevant documents, a reference answer, and the context your pipeline retrieved for it. Five things make a test set useful:
- +Start from real questions. Pull them from logs, support tickets and search queries. They have the vocabulary, typos and vagueness your users actually bring.
- +Label relevance at the document level, with the answer's location. Chunk IDs change every time you re-chunk, which silently invalidates chunk-level labels. Store the source document and the quoted passage that answers the question, and map them to whatever chunks exist at evaluation time.
- +Include questions the corpus cannot answer. A share of the suite should have "I don't know" or a clarifying question as the correct output. Without them you cannot see a model that answers everything.
- +Include the queries retrievers find hard. Exact identifiers (error codes, SKUs, version numbers), questions that need two documents, and questions phrased nothing like the document that answers them.
- +Use synthetic questions to fill gaps, not to replace real ones. Ragas
TestsetGenerator↗ builds a knowledge graph from your documents and by default generates 50% single-hop and 50% multi-hop questions. DeepEval'sSynthesizer(generate_goldens_from_docs) and LlamaIndex'sgenerate_question_context_pairsdo similar jobs. A question written by reading a chunk tends to reuse that chunk's wording, which flatters retrieval, so read a sample and keep real questions in the mix.
How many cases you need depends on how small a difference you want to detect. The eval case sizing post has the arithmetic; the example below shows what 10 queries can and cannot tell you.
Comparing two retrievers query by query #
The most common RAG change is a retrieval change: a new embedding model, a different chunk size, a reranker. Averages hide what such a change does, because it usually helps some queries and breaks others. This script scores two retrieval runs against the same labels, prints the averages, then compares recall@k query by query with a bootstrap confidence interval and lists the queries the candidate lost. It needs only the Python standard library:
"""Score two retrieval runs against labeled relevant chunks, and compare them query by query.
Usage: python retrieval_eval.py labels.jsonl baseline.jsonl candidate.jsonl [k]
labels.jsonl: {"id": "q1", "relevant": {"doc-12": 2, "doc-40": 1}} grade 2 = answers it, 1 = helps
run files: {"id": "q1", "retrieved": ["doc-40", "doc-7", "doc-12", ...]} ranked, best first
"""
import json
import math
import random
import sys
def load(path):
return {row["id"]: row for row in map(json.loads, open(path))}
def score(retrieved, relevant, k):
top = retrieved[:k]
hits = [doc for doc in top if doc in relevant]
first = next((i for i, doc in enumerate(retrieved) if doc in relevant), None)
dcg = sum(relevant.get(doc, 0) / math.log2(i + 2) for i, doc in enumerate(top))
ideal = sorted(relevant.values(), reverse=True)[:k]
idcg = sum(grade / math.log2(i + 2) for i, grade in enumerate(ideal))
return {
f"hit@{k}": 1.0 if hits else 0.0,
f"recall@{k}": len(hits) / len(relevant),
f"precision@{k}": len(hits) / k,
"mrr": 0.0 if first is None else 1 / (first + 1),
f"ndcg@{k}": dcg / idcg,
}
def run_scores(labels, run, k):
return {qid: score(run[qid]["retrieved"], labels[qid]["relevant"], k) for qid in labels}
def bootstrap_ci(diffs, rounds=10_000, seed=0):
rng = random.Random(seed)
means = sorted(
sum(rng.choice(diffs) for _ in diffs) / len(diffs) for _ in range(rounds)
)
return means[int(0.025 * rounds)], means[int(0.975 * rounds) - 1]
if __name__ == "__main__":
labels, base, cand = (load(p) for p in sys.argv[1:4])
k = int(sys.argv[4]) if len(sys.argv) > 4 else 5
a, b = run_scores(labels, base, k), run_scores(labels, cand, k)
print(f"{'metric':<14}{'baseline':>10}{'candidate':>11}")
for metric in a[next(iter(a))]:
mean_a = sum(s[metric] for s in a.values()) / len(a)
mean_b = sum(s[metric] for s in b.values()) / len(b)
print(f"{metric:<14}{mean_a:>10.3f}{mean_b:>11.3f}")
key = f"recall@{k}"
diffs = [b[q][key] - a[q][key] for q in labels]
low, high = bootstrap_ci(diffs)
better = sum(d > 0 for d in diffs)
worse = [q for q in labels if b[q][key] < a[q][key]]
print(f"\n{key}: candidate better on {better}, worse on {len(worse)} of {len(diffs)} queries")
print(f"mean change {sum(diffs) / len(diffs):+.3f}, 95% bootstrap CI [{low:+.3f}, {high:+.3f}]")
for q in worse:
print(f" lost: {q}")
sys.exit(1 if high < 0 else 0)
I ran it on a small fixture set: 10 support questions with graded labels, and two made-up runs standing in for the current retriever and a candidate embedding model. These are fixtures written to show the output, not measurements of real models:
metric baseline candidate
hit@5 0.900 0.800
recall@5 0.767 0.800
precision@5 0.260 0.280
mrr 0.783 0.800
ndcg@5 0.672 0.800
recall@5: candidate better on 4, worse on 2 of 10 queries
mean change +0.033, 95% bootstrap CI [-0.350, +0.383]
lost: err-E1042
lost: sku-TX-900
Read it from the bottom. The candidate improves nDCG@5 from 0.672 to 0.800 and recall@5 on four
queries, and it loses both questions that hinge on an exact identifier, an error code and a SKU.
Four of the five averages say ship it. Only hit rate drops, from 0.900 to 0.800, because those two
queries now retrieve nothing relevant while one that retrieved nothing before now does. Check
exact-identifier queries separately whenever a change reduces the weight of keyword matching,
because an average lets a few wins hide them. The interval runs from
-0.35 to +0.38, so 10 queries cannot tell you whether recall improved. The script exits 1 only
when the whole interval is below zero, which makes it safe to run in CI on a suite of a few
hundred queries, and the lost: lines are the cases to read either way.
RAG evaluation frameworks compared #
Versions and dates as of November 17, 2026:
| Framework | Latest release | RAG metrics | Notes |
|---|---|---|---|
| Ragas ↗ | 0.4.4, Nov 14, 2026 | Context Precision, Context Recall, Context Entities Recall, Noise Sensitivity, Answer Relevancy, Faithfulness, Factual Correctness; ID-based and non-LLM variants | Test set generator; usage analytics on by default ( RAGAS_DO_NOT_TRACK=true ) |
| DeepEval ↗ | 4.3.4, Nov 11, 2026 | AnswerRelevancyMetric ,FaithfulnessMetric ,ContextualPrecisionMetric ,ContextualRecallMetric ,ContextualRelevancyMetric |
Runs in pytest; every metric returns a reason; Synthesizer for test data |
| TruLens ↗ | 2.16.0, Oct 19, 2026 | The RAG triad: context relevance, groundedness, answer relevance | Published by Snowflake; OpenTelemetry tracing |
| MLflow ↗ | 3.18.0, Nov 3, 2026 | RetrievalRelevance ,RetrievalGroundedness ,RetrievalSufficiency , plusCorrectness andRelevanceToQuery |
The retrieval judges need a traced span of type RETRIEVER ; Databricks uses the same judges |
| Arize Phoenix ↗ | phoenix-evals 3.10.0, Nov 10, 2026 | RetrievalRelevanceEvaluator ,FaithfulnessEvaluator ,CorrectnessEvaluator |
Its RAG cookbook computes nDCG, precision and hit rate from the relevance labels with pandas |
| Promptfoo ↗ | 0.125.1, Oct 25, 2026 | context-faithfulness ,context-recall ,context-relevance ,answer-relevance |
YAML assertions, default threshold 0.5 |
| Braintrust autoevals ↗ | 0.4.1, Oct 21, 2026 | ContextPrecision ,ContextRecall ,ContextRelevancy ,ContextEntityRecall ,Faithfulness ,AnswerRelevancy ,AnswerCorrectness |
Its source says they are "ported from the RAGAS project with some enhancements" |
| LlamaIndex ↗ | core 0.14.26, Nov 13, 2026 | RetrieverEvaluator (hit rate, MRR) against expected IDs;FaithfulnessEvaluator ,RelevancyEvaluator |
generate_question_context_pairs builds synthetic retrieval data |
A note on Ragas, since "ragas evaluation" is one of the most searched terms in this area: after
0.4.3 in January 2026 the repository went quiet from late February until November 13, 2026, and
0.4.4 followed on November 14. That release fixes import ragas failing against recent
langchain-community versions and a security issue (CVE-2026-6587) in its multimodal metrics, so
upgrade if you pinned 0.4.3. The metric API is also moving: ragas.metrics.collections is the
current one, and the docs say the older ragas.metrics classes will be removed in 1.0.
Which to pick: Ragas or DeepEval if you want the standard metrics as a Python library, DeepEval if your tests already live in pytest, MLflow or Phoenix if you already trace with them, Promptfoo if your evals are YAML, and plain code like the script above for labeled retrieval metrics, which need no framework. The open-source evaluation frameworks post compares these tools beyond RAG.
RAG benchmarks, and why they do not replace your test set #
Public benchmarks are useful for choosing a component, and weak at predicting how it will do on your corpus:
| Benchmark | What it contains | Use it for |
|---|---|---|
| BEIR ↗ (NeurIPS 2021) | 18 retrieval datasets, zero-shot | Comparing retrievers out of domain; BM25 is a strong baseline |
| MTEB ↗ ,MMTEB ↗ | Embedding tasks including retrieval; MMTEB has 500+ tasks in 250+ languages | Shortlisting embedding models |
| BRIGHT ↗ (ICLR 2025) | 1,384 real queries whose relevant documents take reasoning to find | Retrieval beyond keyword and topic match |
| CRAG ↗ (NeurIPS 2024) | 4,409 questions across five domains with mock web and knowledge-graph search | End-to-end RAG with a penalty for wrong answers |
| RGB ↗ (AAAI 2024) | English and Chinese questions testing noise robustness, negative rejection, integration and counterfactuals | How the generator handles bad context |
| MultiHop-RAG ↗ (COLM 2024) | 2,556 queries over news articles, some with no answer | Questions that need several documents |
| FRAMES ↗ (NAACL 2025) | 824 questions, each needing 2 to 15 Wikipedia articles | Multi-step retrieval |
| RAGTruth ↗ (ACL 2024) | Nearly 18,000 RAG responses labeled for hallucination at word level | Evaluating faithfulness detectors |
Three numbers from these papers make the case for testing on your own data:
- +On BRIGHT, SFR-Embedding-Mistral scores nDCG@10 of 59.0 on MTEB and 18.3 on BRIGHT. A leaderboard position measures the leaderboard's queries.
- +In CRAG, the most advanced LLMs reached at most 34% accuracy alone and 44% with straightforward RAG, and state-of-the-art industry RAG systems answered only 63% of questions without hallucinating.
- +In MultiHop-RAG, GPT-4 scored 0.56 with the chunks its retriever found and 0.89 with the ground-truth evidence. The gap between those two numbers is the retriever's share of the error. FRAMES shows the same from the other side: accuracy went from 0.408 with no retrieval to 0.66 with multi-step retrieval.
How to test a change to a RAG pipeline #
Most RAG regressions arrive with a change to one component. Hold the others still and measure the one that moved:
| What changed | Hold constant | Measure |
|---|---|---|
| Chunking, index rebuild | The labels (document plus quoted passage) | Recall@k, nDCG@10, query by query |
| Embedding model, hybrid weights | The chunks | Recall@k, nDCG@10, and the exact-identifier queries separately |
| Top-k, reranker | The retriever's candidates | Recall@k and faithfulness together, since more context can lower answer quality |
| Prompt | The retrieved context, recorded per case | Faithfulness, correctness, and abstention on the unanswerable cases |
| Generator model | The retrieved context, recorded per case | The same, plus cost and latency |
Holding the context constant is the step teams skip. If you rerun retrieval while testing a new generator model, a ranking change in the vector store shows up as a model difference. Record the retrieved chunks with each test case, feed the same chunks to both models, and score retrieval in a separate run.
Then compare against the current pipeline, not against an absolute threshold. A faithfulness score of 0.91 means little alone; 0.91 against 0.94 on the same cases, with an interval that excludes zero, is a regression. Gate on the metrics with explainable failures (labeled recall, correctness on cases with references, abstention on the unanswerable cases) and keep the judged metrics advisory until you have measured the judge.
When retrieval is a tool call
In an agent, retrieval is usually a tool such as search_kb, and the model decides whether to
search, what to search for, and how many times. That adds three things to evaluate: whether it
searched when the answer needed it, the query it wrote, and whether the answer used what came back.
BrowseComp-Plus (Chen et al., 2025 ↗) fixed the corpus for
deep-research agents to separate the two: its abstract reports GPT-5 at 55.9% accuracy, and 70.1%,
with fewer search calls, when paired with the Qwen3-Embedding-8B retriever. The
tool-call evaluation post covers grading the search calls
themselves.
Doing this with EvalShift #
Disclosure again: I'm the founder. EvalShift does not compute any of the RAG metrics in this post. It has no recall@k, nDCG, context precision or faithfulness score, and it does not compare embedding models or rerun your retriever. For those, use the script above or one of the frameworks.
What it does is the generator row of the table above: test whether a different model or a changed prompt behaves differently on recorded cases, with retrieval held constant.
- +Recorded context. A case promoted from acaptured run replays the
recorded model input, so if your pipeline put the retrieved chunks in the prompt, both models get
the same text. When retrieval is a tool,
--rounds allfeeds the recorded tool results back to the candidate, so it answers from the same documents the current model saw, whatever query it would have written. - +Search behavior. With retrieval as a tool,
tool_selectionchecks whether the candidate still searches where the current model did,tool_argumentscompares the query it writes, andtrace_invariantscan require a search before an answer or cap the number of calls. Theevaluators docs cover each. - +The verdict. A
migration_policyturns the differences into pass, conditional pass, fail or inconclusive, and inconclusive means the suite was too small to decide.
The pairwise LLM judge does not see the retrieved documents, so it cannot score groundedness. Run a faithfulness metric from one of the frameworks above on both models' answers for that.
FAQ #
What is RAG evaluation? It is measuring a retrieval-augmented generation pipeline in two halves and end to end: whether the retriever returns the chunks that answer the question, measured with recall@k, nDCG or LLM-judged context precision and recall, and whether the generator's answer is faithful to those chunks, relevant to the question and correct.
What are the main RAG evaluation metrics? For retrieval: hit rate, recall@k, precision@k, MRR and nDCG@k against labeled relevant chunks, or context precision and context recall judged by an LLM against a reference answer. For generation: faithfulness or groundedness, answer relevancy and answer correctness. Noise sensitivity measures how irrelevant retrieved text affects the answer.
What is the RAG triad? It is TruLens's name for three reference-free RAG checks: context relevance (are the retrieved chunks relevant to the query), groundedness (is each claim in the answer supported by the chunks) and answer relevance (does the answer address the query).
How do you evaluate RAG without ground truth? Use reference-free metrics: faithfulness or groundedness, which checks the answer against the retrieved context, LLM-judged context relevance and answer relevancy. They need no reference answers, so they also run on production traffic. They cannot tell you whether the answer is correct, or whether a better chunk existed that was not retrieved.
What is the difference between Ragas and DeepEval? Both are open-source Python libraries with LLM-judged RAG metrics. Ragas has more retrieval variants, including ID-based and non-LLM ones, and a knowledge-graph test set generator. DeepEval runs in pytest and returns a reason with every score. Their metrics with the same name differ: DeepEval's faithfulness counts an unsupported claim as faithful unless the context contradicts it, while Ragas counts it against the answer.
Is Ragas still maintained? Yes. After a gap from late February 2026, its repository became active again on November 13, 2026, and version 0.4.4 was released on November 14, 2026, with fixes for import errors and a security issue in its multimodal metrics.
Keep reading #
- +LLM hallucination detection : faithfulness checks in depth, including small grounding models.
- +LLM-as-a-judge : how to write and validate the judge behind a RAG metric.
- +How many eval cases do you need : sizing a test set to detect the difference you care about.
- +Open-source LLM evaluation frameworks : Ragas, DeepEval and the rest compared beyond RAG.