I read the metric libraries of five widely-used eval tools. The metric was never the hard part.

A developer surveyed the metric catalogs of five widely-used LLM evaluation tools—Arize Phoenix, DeepEval, Future AGI, Langfuse, and Ragas—and found that the built-in metrics are converging into a commodity, with faithfulness and tool-calling correctness appearing in most. The developer argues that the hard parts of evaluation, choosing the right metric for a failure mode and adding error bars, are not automated by any of these tools.

Every LLM eval tool sells you the same headline: a big bag of ready-made metrics. Fifty of them. Seventy. Pick one, call evaluate , get a number. The pitch works because it is true, and because it quietly relocates the hard part of evaluation to somewhere you cannot see it. I spent a week reading the actual metric catalogs of five widely-used eval libraries, not the landing pages. Arize Phoenix, DeepEval, Future AGI, Langfuse, and Ragas. I wanted to know two things. What do you get for free, and what does the free part let you stop thinking about. The answer to the second question is: almost nothing. The metric is the easy 20 percent. The two hard parts, choosing a metric that matches your failure mode and putting an error bar on the result, are still yours, and none of these tools does them for you. Here is the survey, then the argument. I ordered these alphabetically on purpose. I am not ranking them, and I built a comparison table precisely so I would not be tempted to. | Tool | License | Built-in metrics documented | LLM-as-judge | Aggregation | |---|---|---|---|---| | Arize Phoenix | Elastic License 2.0 source-available | ~16 pre-built evaluators: Faithfulness, Correctness, Q&A, Tool Selection, Hallucination, Toxicity | Yes, judge templates use function calling to extract structured verdicts | Datasets and Experiments: per-example scores, aggregate metrics, cross-experiment comparison | | DeepEval | Apache-2.0 | "50+" metrics: G-Eval, DAG, Answer Relevancy, Faithfulness, Tool Correctness, Hallucination, Bias | Yes, most metrics are LLM-judged QAG, DAG, G-Eval | Pytest-style pass/fail against thresholds; -r flag repeats a test case | | Future AGI | Apache-2.0 | 72 local metrics: BLEU, ROUGE, embedding similarity, JSON-schema, plus judged faithfulness and function-call accuracy | Yes, local heuristics augmentable with a judge via augment=True | Aggregated metric type; datasets in the platform | | Langfuse | MIT core; ee/ is commercial | Managed evaluator catalog: Hallucination, Context-Relevance, Toxicity, Helpfulness some maintained with Ragas | Yes, numeric, categorical, or boolean judge scores | Dataset runs scored at aggregate level, side-by-side experiment comparison | | Ragas | Apache-2.0 | "30+" metrics: Faithfulness, Context Precision, Context Recall, Response Relevancy, Tool Call Accuracy | Yes, LLM-based metrics issue one or more judge calls | evaluate returns mean scores across the dataset; to pandas for per-sample | A few honest notes before anyone quotes this table at me. The metric counts are each project's own stated approximation, not a hand count. "50+", "30+", and "~16" are marketing-adjacent numbers from their docs, and "72" is a figure from Future AGI's SDK README. I did not recount anyone's catalog line by line, so treat the column as order-of-magnitude, not a leaderboard. Phoenix is worth a license asterisk: Elastic License 2.0 is source-available, not OSI-approved open source, which matters if you are redistributing, less so if you are just running it. Langfuse is MIT at the core with a separately licensed enterprise directory. The other three are Apache-2.0. The overlap is the real story. Faithfulness shows up in four of the five. Tool-calling correctness shows up in four. Every one of them supports LLM-as-a-judge, and every one of them returns you a score. If you were choosing on metric coverage alone, you would have a hard time telling them apart, because for the common cases they are converging on the same list. That convergence is a signal. The catalog is becoming a commodity. Which means the catalog is not where your evaluation succeeds or fails. A built-in metric encodes a definition somebody else wrote. "Faithfulness" in a RAG library means the answer is grounded in the retrieved context. That is a real and useful thing to measure. It is also completely silent about whether your agent picked the right document to retrieve in the first place, whether it called the refund tool instead of the lookup tool, or whether it was confidently, groundedly wrong because the context itself was stale. I have watched teams adopt a library, wire up five of its metrics, get a dashboard full of 0.9s, and ship a regression anyway, because none of the five metrics measured the thing that actually broke. The tool did its job. The scores were accurate. They were just answers to questions nobody's users were asking. The move that works is unglamorous and the tools cannot do it for you. Write down your failure taxonomy first. Go read fifty real production traces, cluster the ways your system actually fails, and only then go shopping in the catalog for metrics that map onto those clusters. Most of the time you will find that two or three built-in metrics cover your top failure modes and the rest are noise you were about to pay judge-call money to compute. Occasionally you will find that your worst failure mode has no built-in metric at all, and you will write a custom judge. Every one of these five tools lets you write a custom metric, which tells you the authors already know the catalog does not cover you. So the first hard part is selection, and selection is a function of your traces, not their docs. Here is the finding that sent me down this whole path. I read the aggregation behavior of all five tools carefully, because aggregation is where statistics is supposed to happen. What I found is that all five will happily give you a mean. Ragas returns {'faithfulness': 0.892}. Phoenix and Langfuse show aggregate scores across an experiment. DeepEval gives you a pass rate against a threshold. Future AGI has an aggregated-metric type. Not one of them, as of July 2026, computes a confidence interval for you by default. I want to be precise, because this is the kind of claim that gets misquoted into "tool X can't do statistics." That is not what I am saying. I am saying I went looking in the documented, out-of-the-box behavior of each library and did not find confidence-interval computation in any of them, Future AGI included. DeepEval's -r repeat flag gets you closest to the raw material, since you can run a case multiple times, but the docs stop at repeating, not at turning the repeats into an interval. The others aggregate to a mean and stop. This matters more than the metric-count column everyone stares at. A faithfulness pass rate of 0.88 measured on 50 examples 44 of 50 above threshold is not 0.88. It is 0.88 with a 95 percent Wilson interval running from roughly 0.76 to 0.94 Wilson, 1927, the standard small-sample interval for a proportion . If your last release scored 0.82 41 of 50 on the same 50 and you are about to write "faithfulness improved," you are reading noise as signal. The tool told you 0.82 and 0.88. It did not tell you those two numbers are statistically indistinguishable, because computing that was never its job, and it did not warn you that you were about to assume it was. Here are the ten lines the catalog leaves to you. python from statsmodels.stats.proportion import proportion confint def pass rate with ci passes, n, alpha=0.05 : rate = passes / n lo, hi = proportion confint passes, n, alpha=alpha, method="wilson" return rate, lo, hi 44 of 50 examples passed the metric's threshold rate, lo, hi = pass rate with ci 44, 50 print f"{rate:.2f} 95% CI {lo:.2f}, {hi:.2f} " 0.88 95% CI 0.76, 0.94 That interval is the difference between "we improved" and "we cannot tell yet, run more examples." For continuous judge scores rather than pass/fail, bootstrap the mean instead of using Wilson, but the discipline is identical: never report a point estimate from an eval set without the interval around it, and never compare two releases whose intervals overlap as if one won. This is not an argument against any of these libraries. I use them. The catalog genuinely saves you from reimplementing BLEU, ROUGE, embedding similarity, and a dozen judge prompts, and standardizing on one of them is a good decision. Phoenix rides OpenTelemetry if that is your stack. DeepEval's pytest ergonomics are the cleanest if you live in CI. Future AGI if you want eval, tracing, and guardrails in one place. Langfuse if you want tracing and evals in one product. Ragas is still the sharpest on RAG. They are all fine, and they are all converging. The mistake is thinking the tool finished your evaluation. It started it. It computed the metric, which was the part a library can do, and it handed you back a number stripped of the two things that make the number mean anything: whether the metric matches your failure, and how much of the number is noise. Those two stay on your side of the API no matter which logo is on the dashboard. Which of these five should I pick? Whichever fits your stack and license constraints, because on metric coverage they are close to interchangeable for common cases. Decide on ergonomics pytest vs platform vs OTel , license Apache-2.0 vs Elastic vs MIT-core , and whether you need the surrounding tracing and datasets, not on the metric count. Is a bigger metric library better? No. A bigger library raises the chance that a metric matching your failure mode exists, but it does nothing to raise the chance that you picked it. Ten well-chosen metrics beat seventy defaults every time. Do I really need confidence intervals for eval? If you are making ship/no-ship decisions on the difference between two scores, yes, always. The interval is what tells you whether the difference is real. Ten extra lines of code, and it is the highest-leverage code in your eval suite. Does any tool do the interval for me? Not by default, in any of the five I read, as of July 2026. If that changes I will happily update this. Until then, wrap your own. If the metric catalog is commoditizing, the differentiation moves up a level, to selection and uncertainty. So why does no eval tool ship a "your last two runs are statistically indistinguishable" warning yet? It is a small feature. It would prevent a large fraction of the false "we improved" claims I see in eval writeups. My guess is that a tool that keeps telling you your result is inconclusive is a harder product to sell than one that always returns a confident number. I would love to be proven wrong by whichever project ships it first.