LLM Evaluation Tools Compared: How to Run Evals

LangChain's State of Agent Engineering survey of 1,340 practitioners found that 89 percent have observability on their agents but only 52.4 percent run offline evaluations against a test set, with a third naming quality as their main barrier to production ahead of latency at 20 percent. The guide recommends defining pass criteria before running anything, starting with 20 to 50 tasks drawn from real failures, grading with code where answers are checkable and a human-validated judge model where they are not, and gating merges on regressions. It cites Anthropic's January 2026 "Demystifying evals for AI agents" in noting that a 75 percent per-trial pass rate yields only a 42 percent chance of passing three trials in a row, and describes the method behind the open-source Featherbench harness.

← All posts https://reinvently.co.uk/blog/ The short answer: write down what a good answer is, test against 20 to 50 real tasks, and grade with code where you can and a checked judge model where you cannot. Then block releases that regress, and trace production so failures become new tests. For LLM evaluation, most teams need one framework in CI and one observability platform; if everything runs on one cloud, its built-in evaluators may cover both. LangChain's State of Agent Engineering https://www.langchain.com/state-of-agent-engineering survey of 1,340 practitioners found that 89 percent have observability on their agents, but only 52.4 percent run offline evaluations against a test set. A third named quality as their main barrier to production, ahead of latency at 20 percent. Most teams can see what their AI system did. Half cannot say whether a change made it better or worse. This guide sets out the loop that closes that gap and shows which tools cover each stage. It is the method behind our own open-source harness, Featherbench https://github.com/ed-is-ai/featherbench , which runs the Ed-o-meter leaderboard https://reinvently.co.uk/tools/ed-o-meter/ . How Do You Evaluate an AI Application? Anthropic's Demystifying evals for AI agents https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents January 2026 supplies the terms: a task is one test with success criteria, a trial one attempt at it, a grader scores the attempt, and the transcript records what the system did. 1. Define. Write the pass bar for each task before you run anything: the fact the answer must contain, the record that must exist, the tone it must not take. If two people cannot agree whether an output passes, the task is not ready. 2. Dataset. Start with 20 to 50 tasks from real failures, Anthropic's suggested size, plus the cases you must never get wrong. Keep the set private: public benchmarks leak into training data. 3. Grade. Use code wherever the answer is checkable, such as a schema check, unit test or database query. Use a judge model for tone, relevance and faithfulness, after checking it against human grades on a sample. Our Featherbench build found measurable judge bias https://reinvently.co.uk/blog/building-the-ed-o-meter-llm-eval-harness/ . 4. Gate. Run the suite on every prompt, model or retrieval change and block the merge on a regression. Run each task more than once: in Anthropic's example, a 75 percent per-trial pass rate gives only a 42 percent chance of passing three trials in a row. 5. Monitor. Trace every model and tool call, score a sample of live traffic with the same graders, and track cost and latency beside quality. 6. Learn. Read the transcripts of failed traces, label why each failed, and add the clearest to the dataset. Keep two suites. Capability evals should start with a low pass rate; regression evals should sit near 100 percent. When a capability task passes reliably, move it across. As Anthropic puts it, "An eval at 100% tracks regressions but provides no signal for improvement." LLM Evaluation Tools, Stage by Stage We selected the most-starred open-source eval frameworks and observability platforms, the commercial platforms most often shortlisted, including Datadog, and the evaluators built into the three main clouds. Three model benchmarks, including our own Featherbench, are included for comparison. Define is left out of the table because no tool does it for you. | Tool | GitHub stars | Dataset | Grade | Gate | Trace | Live scoring | Trace to test | |---|---|---|---|---|---|---|---| | Eval frameworks | | | | | | | | | promptfoo https://github.com/promptfoo/promptfoo | 26,339 | | | | | | | | DeepEval https://github.com/confident-ai/deepeval | 19,063 | | | | | | | | Ragas https://github.com/vibrantlabsai/ragas | 16,134 | | | | | | | | Inspect https://github.com/UKGovernmentBEIS/inspect ai | 3,071 | | | | | | | | Open-source observability platforms | | | | | | | | | Langfuse https://github.com/langfuse/langfuse | 36,256 | | | | | | | | MLflow https://github.com/mlflow/mlflow | 28,511 | | | | | | | | Opik https://github.com/comet-ml/opik | 22,849 | | | | | | | | Arize Phoenix https://github.com/Arize-ai/phoenix | 11,968 | | | | | | | | Commercial platforms | | | | | | | | | LangSmith https://www.langchain.com/langsmith | — | | | | | | | | Braintrust https://www.braintrust.dev/ | — | | | | | | | | Datadog https://docs.datadoghq.com/llm observability/ | — | | | | | | | | W&B Weave https://github.com/wandb/weave | 1,142 | | | | | | | | Cloud built-ins | | | | | | | | | Microsoft Foundry https://learn.microsoft.com/en-us/azure/foundry/concepts/observability | — | | | | | | | | Amazon Bedrock and AgentCore https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html | — | | | | | | | | Google Gen AI evaluation https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview | — | | | | | | | | Model benchmarks, for comparison | | | | | | | | | lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness | 14,305 | | | | | | | | HELM https://github.com/stanford-crfm/helm | 2,955 | | | | | | | | Featherbench ours https://github.com/ed-is-ai/featherbench | 4 | | | | | | | Built-in workflow for the stage · Partial, or through a paid add-on or your own scripting · Not covered. Ratings are our reading of each tool's documentation on 23 October 2026, not hands-on tests. LangSmith, Braintrust, Datadog and the cloud services are closed source, so stars do not apply. Eval frameworks: promptfoo, DeepEval, Ragas and Inspect Frameworks keep tests in your repository and run them from the command line or a test runner. promptfoo https://github.com/promptfoo/promptfoo defines tests in YAML, which makes it the quickest to put in CI and the easiest for non-Python teams; it also generates red-team attacks such as jailbreaks and prompt injection. promptfoo and Inspect record traces during eval runs, not in production. DeepEval https://github.com/confident-ai/deepeval writes evals as pytest-style Python tests, with ready-made metrics for hallucination, tool use and multi-turn conversation. Its tracing and live scoring come through its maker's paid platform, Confident AI. Ragas https://github.com/vibrantlabsai/ragas has the deepest retrieval metrics, separating retrieval failures from generation failures, and generates test sets from your own documents. It moved to a new GitHub organisation and has not released since 13 January, so pin the version. Inspect https://inspect.aisi.org.uk/ , from the UK AI Security Institute, runs agent evals in sandboxes where the end state can be checked directly, and its Inspect Evals collection has more than 200 ready-made evaluations. It is the most rigorous choice for agents and the most work to set up. Open-source platforms: Langfuse, MLflow, Opik and Phoenix Platforms add the production half: traces, judge scoring of live traffic and one-click promotion of a trace into a dataset. Langfuse https://langfuse.com/ is the most-starred, and ClickHouse acquired it in January; self-hosting it means running Postgres, ClickHouse, Redis and object storage. Arize Phoenix https://phoenix.arize.com/ is the lightest to self-host, as a single container built on OpenTelemetry, but its Elastic License 2.0 bars offering it as a hosted service. Opik https://github.com/comet-ml/opik is an Apache-licensed alternative, and MLflow https://mlflow.org/ is the natural fit if your data science team already runs it or you are on Databricks. Commercial platforms: LangSmith, Braintrust, Datadog and W&B Weave LangSmith https://www.langchain.com/langsmith is the default on LangChain or LangGraph but not limited to them. Braintrust https://www.braintrust.dev/ is built around experiments: compare versions of a prompt or agent against a dataset, then promote the winner. Both cover every stage in the table, and both offer self-hosted or hybrid deployment on enterprise plans. Datadog's LLM Observability https://docs.datadoghq.com/llm observability/ , which its documentation now also calls Agent Observability, suits enterprises that already run Datadog and traces model calls beside the rest of the application. It scores live traffic with judge-model templates https://docs.datadoghq.com/llm observability/investigate/evaluations/llm as a judge evaluations/template evaluations/ or custom judges, and its LLM Experiments https://www.datadoghq.com/blog/llm-experiments/ feature turns production traces into datasets in one click. We found no native CI gate, and it bills by LLM spans ingested. W&B Weave https://github.com/wandb/weave has an open-source SDK and fits teams already on Weights & Biases. Cloud built-ins: Microsoft Foundry, Amazon Bedrock and Google Cloud Microsoft Foundry https://learn.microsoft.com/en-us/azure/foundry/concepts/observability has evaluators for quality, retrieval, safety and agents, continuous evaluation of sampled production traffic, OpenTelemetry tracing and a red-teaming agent built on PyRIT; a preview feature https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/traces-to-dataset turns production traces into evaluation datasets. Playground evaluations are on by default and billed. Amazon Bedrock https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html evaluates models and knowledge bases with programmatic metrics, a judge model or human reviewers, and AgentCore Evaluations https://aws.amazon.com/about-aws/whats-new/2026/03/agentcore-evaluations-generally-available added 13 agent evaluators and live-trace scoring in March 2026. Google's Gen AI evaluation service https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview generates pass-or-fail rubrics for each prompt and scores agents on both the answer and the sequence of tool calls. Built-ins remove integration work but tie the suite to one cloud; our guide to AI vendor lock-in https://reinvently.co.uk/blog/ai-vendor-strategy-lock-in/ covers how to weigh that. The Foundry deep dive https://reinvently.co.uk/blog/microsoft-foundry-deep-dive/ covers the wider platform. For comparison: lm-evaluation-harness, HELM and Featherbench lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness and Stanford's HELM https://github.com/stanford-crfm/helm run public benchmarks against models. They help shortlist a model but cannot tell you whether your application works, because the tasks are not yours. Featherbench https://github.com/ed-is-ai/featherbench is our own open-source harness and sits between the two camps. It still benchmarks models rather than an application, but on tasks you write yourself, graded by code checks and a blind panel of judge models. It is a single Python file, MIT-licensed, and it produces the Ed-o-meter leaderboard https://reinvently.co.uk/tools/ed-o-meter/ . Use it to pick the smallest model that clears your bar, then evaluate the application with a framework above. How we built it https://reinvently.co.uk/blog/building-the-ed-o-meter-llm-eval-harness/ covers the design. What the Comparison Shows The offline half is well served. Every framework and platform can hold a dataset and run code and judge graders. The differences are in the production half, and in the step from a failed trace back to a test case, which is where suites improve. Frameworks leave that step to you; the platforms automate it. Stars also flatter the open-source tools, because closed products have none. Python SDK downloads over the 30 days to 23 October 2026, from PyPI Stats https://pypistats.org/ , tell a different story: langsmith at 77 million a month, Datadog's ddtrace at 38 million and langfuse at 24 million. Braintrust has 6.6 million, against 1.7 million for Opik and 0.6 million for Phoenix. Downloads overstate LangSmith, which installs with LangChain, and ddtrace covers all of Datadog's tracing, but commercial tools are at least as widely used as open-source ones. Langfuse is the clear open-source leader. Ownership is the other change. OpenAI agreed to acquire promptfoo https://openai.com/index/openai-to-acquire-promptfoo/ in March and is winding down its own Evals product https://developers.openai.com/cookbook/examples/evaluation/moving-from-openai-evals-to-promptfoo , with shutdown announced for 30 November 2026; if your suite lives there, export it now. ClickHouse bought Langfuse in January and says self-hosting and the open-source licence are unchanged. Neither changes what the tools do today, but both are reasons to keep tasks and grades in plain files you own. Which LLM Evaluation Tool Should You Choose? - Your team writes Python and wants evals beside unit tests. DeepEval in CI, with Langfuse or Phoenix for traces. - You need a CI gate quickly, or your team is not Python-first. promptfoo, which also gives you red-teaming. - Your application is retrieval-heavy. Ragas, pinned to a version, with DeepEval's retrieval metrics as the fallback. - You are evaluating agents that take actions. Inspect, because it checks the end state in a sandbox rather than trusting the agent's account. - Traces must stay in your estate. Phoenix for the lightest footprint, Langfuse for the fullest feature set. - You already run Datadog. Turn on its LLM Observability first, and add a framework for CI gates. - Everything runs on one cloud. Start with its built-in evaluators, and keep the test set in your repository so it can move. Traces hold users' prompts and your system's outputs, often including personal data. Where they are stored is a data protection decision, and it usually settles self-hosted versus SaaS before features do. Prefer tools that accept OpenTelemetry, but pin versions: its generative AI conventions https://opentelemetry.io/docs/specs/semconv/gen-ai/ are still marked as in development. Three mistakes to avoid Trusting a judge you have not checked. A judge model scores at scale, but it can favour some styles or model families. Compare it with human grades on a sample before its scores gate a release. Testing on public benchmarks. A leaderboard score says how a model does on someone else's tasks, which may also be in its training data. Your suite should be built from your own failures. Tracing without reading. Dashboards show that something failed, not why. Someone has to read the transcripts and label failures, or the loop never closes. What This Means for Your Organisation The tool is the easy part. Teams that evaluate well have a named owner for the test set, a pass bar that product and engineering agree on, and a rule that no model or prompt change ships without an eval run. Those are operating-model decisions. The guide for technology leaders https://reinvently.co.uk/leaders/ covers who should own them, and the board AI risk questions https://reinvently.co.uk/blog/what-boards-should-ask-about-ai-risk/ cover what to report upward. LLM Evaluation FAQ What is the difference between an eval and observability? An eval scores the system against a fixed set of tasks with a written pass bar, usually before release. Observability records what the system did in production, as traces of every model and tool call. You need both: evals catch regressions before users see them, and traces supply the real failures that become new eval tasks. How many test cases do I need to start evaluating an AI application? Anthropic's engineering guidance suggests starting with 20 to 50 tasks drawn from real failures. Early in development the differences between versions are large, so a small set shows them. Grow the set as the differences you need to detect get smaller. Can I trust an LLM as a judge? Only after you have checked it against people. A judge model is cheap and scales, but it is non-deterministic and can favour some styles or model families. Have a person grade a sample, compare the judge's scores with theirs, and recheck whenever you change the judge model or its prompt. Which LLM observability platforms can be self-hosted? Langfuse, Arize Phoenix, Opik and MLflow can be self-hosted from their open-source repositories. Phoenix runs as a single container; Langfuse needs Postgres, ClickHouse, Redis and object storage. LangSmith and Braintrust offer self-hosted or hybrid deployment on enterprise plans. Do I need a separate eval tool if I build on Microsoft Foundry, Amazon Bedrock or Google Cloud? Not necessarily. Each platform now ships built-in evaluators, judge-model scoring and production monitoring. A separate tool earns its place when you run models across more than one cloud, want evals to live in your code repository, or need red-teaming beyond what the platform provides. Sources. GitHub repositories for promptfoo https://github.com/promptfoo/promptfoo , DeepEval https://github.com/confident-ai/deepeval , Ragas https://github.com/vibrantlabsai/ragas , Inspect https://github.com/UKGovernmentBEIS/inspect ai , Langfuse https://github.com/langfuse/langfuse , MLflow https://github.com/mlflow/mlflow , Opik https://github.com/comet-ml/opik , Phoenix https://github.com/Arize-ai/phoenix , W&B Weave https://github.com/wandb/weave , lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness , HELM https://github.com/stanford-crfm/helm and Featherbench https://github.com/ed-is-ai/featherbench ; Anthropic, Demystifying evals for AI agents https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents 9 January 2026 ; LangChain, State of Agent Engineering https://www.langchain.com/state-of-agent-engineering 1,340 responses, 18 November to 2 December 2025 ; Microsoft Foundry observability https://learn.microsoft.com/en-us/azure/foundry/concepts/observability ; Amazon Bedrock evaluations https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html ; Google Gen AI evaluation service https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview ; ClickHouse https://clickhouse.com/blog/clickhouse-acquires-langfuse-open-source-llm-observability on Langfuse 16 January 2026 ; OpenAI https://openai.com/index/openai-to-acquire-promptfoo/ on promptfoo 9 March 2026 ; Datadog LLM Observability https://docs.datadoghq.com/llm observability/ ; download counts from PyPI Stats https://pypistats.org/ , last 30 days to 23 October 2026. Star counts come from the GitHub API on 23 October 2026 and will change. A companion piece for technology leaders, on who owns AI quality and what evals protect against, is next. Sign up for email updates https://reinvently.co.uk/contact/ to get it. ← All posts https://reinvently.co.uk/blog/