{"slug": "the-best-llm-evaluation-tools-in-2026-compared-by-the-job-they-do", "title": "The best LLM evaluation tools in 2026, compared by the job they do", "summary": "A November 2026 comparison of LLM evaluation tools recommends Promptfoo for prompt and model comparison and red teaming, DeepEval with Ragas for prebuilt quality metrics in pytest and RAG pipelines, Inspect AI for sandboxed agent benchmarks with error bars, and Langfuse, Arize Phoenix or Opik for self-hosted tracing with evals. The roundup, written by the developer of the agent regression-testing tool EvalShift, notes that OpenAI shut down its hosted Evals product on November 30, 2026, and that most teams end up running two tools: one watching production and one checking changes before they merge.", "body_md": "[← all posts](https://www.evalshift.dev/blog)\n\n# The best LLM evaluation tools in 2026, compared by the job they do\n\nThe best LLM evaluation tools and frameworks in 2026, compared: what each grades, agent and CI support, licenses, pricing, and which to pick.\n\nThe best LLM evaluation tool depends on the job. As of November 2026: for comparing prompts and models, and for red teaming, Promptfoo; for prebuilt quality metrics inside a pytest suite, DeepEval, with Ragas for RAG pipelines; for agent benchmarks run in sandboxes with error bars, Inspect AI; for tracing production and scoring it with evals in one place, Langfuse, Arize Phoenix or Opik if you want to self-host, and LangSmith or Braintrust if you want a commercial platform; for teams already on Databricks, MLflow. For one narrow job, checking that a production agent still behaves the same after a model swap, EvalShift.\n\nMost teams end up with two of these, not one: something that watches production, and something that checks a change before it merges. The rest of this post compares the tools on what they grade, how they compare two versions, how they plug into CI, where your data goes and what they cost, and lists what changed in the category this year, including OpenAI shutting down its hosted Evals product on November 30, 2026.\n\nA disclosure: I build EvalShift, which is on this list. It covers one narrow job, and its section says what it does not do. Every claim about another tool links to that tool's own docs, checked in November 2026.\n\n## LLM evaluation tools at a glance\n\n| Tool | What it is | License | Languages | Best for | \n|---|---|---|---|---|\n| [Promptfoo](https://www.evalshift.dev/blog/best-llm-evaluation-tools#promptfoo) | CLI eval and red-teaming framework | MIT | Node CLI, YAML config; tests in any language | Comparing prompts and models on your data; red teaming | \n| [DeepEval](https://www.evalshift.dev/blog/best-llm-evaluation-tools#deepeval) | Metric library that runs under pytest, plus the Confident AI platform | Apache-2.0 (platform proprietary) | Python; TypeScript in beta | Prebuilt quality metrics in your existing test suite | \n| [Inspect AI](https://www.evalshift.dev/blog/best-llm-evaluation-tools#inspect-ai) | Eval framework from the UK AI Security Institute | MIT | Python | Agent evals in sandboxes; results with standard errors | \n| [Ragas](https://www.evalshift.dev/blog/best-llm-evaluation-tools#ragas) | RAG metric library | Apache-2.0 | Python | Scoring retrieval and grounded answers | \n| [LangSmith](https://www.evalshift.dev/blog/best-llm-evaluation-tools#langsmith) | Tracing, evals and deployment platform | Proprietary (openevals, agentevals MIT) | Python, TypeScript | One commercial platform, especially on LangChain or LangGraph | \n| [Langfuse](https://www.evalshift.dev/blog/best-llm-evaluation-tools#langfuse) | Open-source observability platform with evals | MIT, except `ee` folders | Python, JS/TS | Self-hosted tracing with evals and prompt management | \n| [Braintrust](https://www.evalshift.dev/blog/best-llm-evaluation-tools#braintrust) | Eval and observability platform | Proprietary (autoevals MIT) | TypeScript, Python, plus Go, Java, Ruby, C# | Experiment comparison as a team workflow | \n| [Arize Phoenix](https://www.evalshift.dev/blog/best-llm-evaluation-tools#arize-phoenix) | OpenTelemetry-based observability with evals | Elastic License 2.0 | Python, TypeScript | Self-hosted tracing on OpenTelemetry | \n| [Opik](https://www.evalshift.dev/blog/best-llm-evaluation-tools#opik) | Open-source observability and eval platform | Apache-2.0 | Python, TypeScript | A fully open-source platform with agent metrics | \n| [MLflow](https://www.evalshift.dev/blog/best-llm-evaluation-tools#mlflow) | ML and GenAI platform with an evaluation API | Apache-2.0 | Python, TypeScript, Java | Teams already on MLflow or Databricks | \n| [EvalShift](https://www.evalshift.dev/blog/best-llm-evaluation-tools#evalshift) | Migration and regression testing for agents | CLI Apache-2.0, SDK and Action MIT | Python | Swapping the model under a production agent | \n\nThree more come up in searches and are covered briefly at the end: TruLens, W&B Weave and Galileo, along with OpenAI Evals and Humanloop, which are no longer options for new projects.\n\n## LLM evaluation framework vs LLM evaluation platform\n\nThe tools split into two shapes, and the split matters more than any feature list.\n\nAn **LLM evaluation framework** is a library or CLI you run in your own code: you write test cases,\npick or write scorers, and run the suite locally or in CI. Promptfoo, DeepEval, Inspect AI and\nRagas are frameworks. They are free, your data stays on your machine unless you opt into a hosted\ndashboard, and they fit naturally into a pull-request check. What they do not do is watch\nproduction.\n\nAn **LLM evaluation platform** starts from tracing. Your application sends every request, model\ncall and tool call to a server, and evals run over those traces: on datasets before release, and\non sampled live traffic after it. LangSmith, Langfuse, Braintrust, Phoenix, Opik and MLflow are\nplatforms. They are where you find out what went wrong for a specific user and turn that case into\na test. The cost is a server, either theirs or one you host, and an SDK in your app.\n\nThe line is blurring, since most frameworks now have a hosted dashboard and most platforms now have a CI integration. A useful rule still holds: pick a platform for what production is doing, and make sure something in your stack can fail a pull request before a change ships.\n\n## What to look for in an LLM evaluation tool\n\nSix questions separate the tools faster than a feature checklist:\n\n1. +**What does it grade?** The final text only, or the trace: which tools an agent called, with\nwhich arguments, in what order. For an agent, a correct-sounding reply after the wrong tool\ncall is the failure that matters most.\n2. +**Where do test cases come from?** Hand-written in config, generated synthetically, or\ncollected from production traces. Suites built from real traffic find the failures users will\nactually hit;[build a golden suite from production traffic](https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic) covers that loop.\n3. +**How does it compare two versions?** Almost every tool shows two runs side by side. Few tell\nyou whether the difference between them is larger than the noise. More on that[below](https://www.evalshift.dev/blog/best-llm-evaluation-tools#the-question-most-tools-leave-to-you-is-the-difference-real) .\n4. +**Can it gate a pull request?** A GitHub Action, a pytest exit code, or a threshold in a script\nthat fails the job.\n5. +**Where does your data go?** Locally only, to a vendor's cloud, or to a server you host.\n6. +**What can you run without paying?** Open-source license, free self-hosting, or a free cloud\ntier with limits.\n\n## Agent and CI support compared\n\n| Tool | Agent and tool-call checks | CI gate | Online evals on production | Red teaming | \n|---|---|---|---|---|\n| Promptfoo | Trajectory assertions: tool used, argument match, sequence, step count, goal success | Official GitHub Action | Enterprise only | Yes | \n| DeepEval | Tool Correctness, Argument Correctness, Task Completion, Step Efficiency, Plan Adherence | `deepeval test run` under pytest | Via Confident AI | Via DeepTeam | \n| Inspect AI | Built-in ReAct agent, tools, sandboxes (Docker, Kubernetes, Modal) | Not a documented feature | No | No | \n| Ragas | Tool Call Accuracy, Tool Call F1, Agent Goal Accuracy, Topic Adherence | No | No | No | \n| LangSmith | agentevals trajectory match (strict, unordered, subset, superset) and trajectory judges | pytest, Vitest, Jest | Yes | No | \n| Langfuse | Evaluators over tool calls; trajectory checks you write | `langfuse/experiment-action` | Yes | No | \n| Braintrust | Trace-level scorers over the full trace | `braintrustdata/eval-action` | Yes | No | \n| Arize Phoenix | Tool selection, invocation and response-handling evals | No documented action | On Arize AX | No | \n| Opik | Trajectory Accuracy, Agent Task Completion, Tool Correctness | pytest integration | Yes | No (has guardrails) | \n| MLflow | `ToolCallCorrectness` ,`ToolCallEfficiency` (experimental) | No documented action | Yes | No | \n| EvalShift | Tool selection, arguments, call structure and trace rules, on both models | Official GitHub Action | No | No | \n\n\"No\" in this table means the tool's documentation does not describe the feature as of November 2026, not that it cannot be built on top.\n\n## Promptfoo\n\n[Promptfoo ↗](https://www.promptfoo.dev/docs/intro/) is an open-source CLI for evaluating and red\nteaming LLM apps. You describe prompts, providers and test cases in a YAML file, run\n`promptfoo eval`, and get a matrix of every prompt against every model, which makes it the\nquickest way to line several candidates up on your own data. Its assertion catalog runs to dozens\nof types, from `contains` and JSON schema checks to `llm-rubric`, and its\n[trajectory assertions ↗](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/)\ncheck which tools an agent used, whether the arguments matched, the order and the step count.\n\nTwo things set it apart. Red teaming is a first-class part of the product: it generates\nadversarial inputs, runs them and reports the risk. And evals run locally against your own\nprovider keys. An [official GitHub Action ↗](https://www.promptfoo.dev/docs/integrations/github-action/)\nruns a before-and-after eval on a pull request and comments the result.\n\nPromptfoo is now part of OpenAI, which [announced the acquisition ↗](https://techcrunch.com/2026/03/09/openai-acquires-promptfoo-to-secure-its-ai-agents/)\non March 9, 2026. The [README ↗](https://github.com/promptfoo/promptfoo) says it remains open source\nunder MIT. Requires Node 22.22 or newer.\n\n**Choose it when** you are picking a model or a prompt, you need red teaming, or you want\nassertions off the shelf in a config file. [EvalShift vs Promptfoo](https://www.evalshift.dev/compare/evalshift-vs-promptfoo)\ngoes dimension by dimension.\n\n## DeepEval\n\n[DeepEval ↗](https://deepeval.com/docs/metrics-introduction) is an open-source Python library that\ntreats evals as unit tests: you write `assert_test` calls with metrics and run them with\n`deepeval test run`, which wraps pytest and fails when a metric misses its threshold. It ships 50+\nmetrics across RAG, agents, conversations and safety, plus G-Eval and DAG for writing a custom\njudged criterion. For agents it has Tool Correctness and Argument Correctness at the component\nlevel, and Task Completion, Step Efficiency and Plan Adherence over the whole trajectory. A\nSynthesizer generates test cases from documents when you have none yet.\n\nA [TypeScript version ↗](https://deepeval.com/blog/introducing-deepeval-typescript) runs under\nVitest and is in beta. Red teaming lives in a separate open-source project, DeepTeam. Results and\nproduction tracing go to Confident AI, the commercial platform from the same team, after\n`deepeval login`; its free plan covers 2 seats and 5 test runs a week, and paid plans start at\n$200 a month.\n\n**Choose it when** your evals belong in the pytest suite the team already runs, and you would\nrather pick a metric than write one. [EvalShift vs DeepEval](https://www.evalshift.dev/compare/evalshift-vs-deepeval) has\nthe full comparison.\n\n## Inspect AI\n\n[Inspect AI ↗](https://inspect.aisi.org.uk/) is an open-source Python framework from the UK AI\nSecurity Institute, built for rigorous evaluations of models and agents. An eval is a `@task` that\ncombines a dataset, a solver (the prompt chain or agent) and a scorer; you run it with\n`inspect eval` and browse the logs with `inspect view`. It has a built-in ReAct agent, bash and\neditor tools, MCP support, and sandboxes in Docker, Kubernetes or Modal, so an agent can act in a\nreal environment that is reset between samples. Inspect Evals collects more than 200 prebuilt\nbenchmark implementations.\n\nIt is the only tool on this list whose docs describe uncertainty as a built-in part of every\nresult: [metrics ↗](https://inspect.aisi.org.uk/metrics.html) include standard errors (clustered or\nbootstrapped), confidence intervals and Wilson intervals, and repeated epochs can be reduced to\npass@k.\n\n**Choose it when** you are running capability or safety evals in sandboxes, or reproducing public\nbenchmarks, and you want error bars on the numbers. It is not built for production tracing or a\npull-request gate.\n\n## Ragas\n\n[Ragas ↗](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/) is an open-source\nPython library of evaluation metrics, best known for RAG: faithfulness, context precision, context\nrecall, answer relevancy. It has about 30 metrics in all, including four for agents (Tool Call\nAccuracy, Tool Call F1, Agent Goal Accuracy and Topic Adherence), and can generate synthetic test\nsets from your documents. It is a library, not a platform: no dashboard, CI integration or\ntracing of its own, and its metrics are often run inside other tools, such as MLflow. The\nrepository now lives at [vibrantlabsai/ragas ↗](https://github.com/vibrantlabsai/ragas).\n\n**Choose it when** the question is whether your retrieval and grounding are good, and you want the\nmetrics without a platform around them.\n\n## LangSmith\n\n[LangSmith ↗](https://docs.langchain.com/langsmith/evaluation) is LangChain's platform for tracing,\nevaluating, deploying and monitoring agents. It is framework-agnostic: it traces the OpenAI,\nAnthropic and Vercel AI SDKs, plain code and OpenTelemetry spans, not only LangChain. Datasets run\nas experiments scored by code, LLM-as-a-judge, human or pairwise evaluators, and a comparison view\nmarks each example that improved or regressed against a baseline experiment. Evals can also run\nas pytest, Vitest or Jest tests. Online evaluators score live production runs, and annotation\nqueues route traces to human reviewers.\n\nFor agents, the open-source [agentevals ↗](https://github.com/langchain-ai/agentevals) package\nmatches a trajectory against a reference in strict, unordered, subset or superset mode, or has an\nLLM judge it. The Developer plan is free for one seat and 5,000 traces a month; Plus is $39 per\nseat per month. Self-hosting is an add-on to the Enterprise plan.\n\n**Choose it when** you want tracing, evals, prompts and deployment on one commercial platform,\nespecially if you build on LangChain or LangGraph.\n[EvalShift vs LangSmith](https://www.evalshift.dev/compare/evalshift-vs-langsmith) compares the two.\n\n## Langfuse\n\n[Langfuse ↗](https://langfuse.com/docs) is an open-source LLM engineering platform built around\ntracing. It records each request end to end, with model calls, retrievals and tool executions as\nnested observations, on OpenTelemetry. On top of the traces it runs LLM-as-a-judge and code\nevaluators, on datasets before release and on live traffic after it, alongside prompt management,\na playground and annotation queues. Evaluators can read structured tool calls, and trajectory\nchecks such as step counts and ordering are code you write. Live evaluators now run on\nobservations: on Langfuse Cloud, the older trace-level evaluators\n[stop producing results ↗](https://langfuse.com/faq/all/llm-as-a-judge-migration) at the v4\ncutover on November 16, 2026, so a team that set them up earlier has to recreate them.\n\nIts [experiment-action ↗](https://github.com/langfuse/experiment-action) for GitHub runs your\nexperiment script on a pull request, comments the result and fails the job when the script raises\na `RegressionError`. The code is MIT licensed except for its `ee` folders, and self-hosting is\nfree. The Cloud Hobby tier is free; paid plans start at $29 a month.\n[Langfuse joined ClickHouse ↗](https://langfuse.com/blog/joining-clickhouse) in January 2026, with\nno licensing changes planned.\n\n**Choose it when** you want production tracing and evals on infrastructure you control.\n[EvalShift vs Langfuse](https://www.evalshift.dev/compare/evalshift-vs-langfuse) covers how the two fit together.\n\n## Braintrust\n\n[Braintrust ↗](https://docs.braintrust.dev/evaluate/compare-experiments) is a commercial eval and\nobservability platform built around experiments. You define an `Eval()` in TypeScript or Python\nwith a dataset, a task and scorers, run it with the CLI, and compare experiments against a base row\nby row, with each experiment graded an improvement, a regression, a trade-off or a tie. Its\nopen-source\n[autoevals ↗](https://github.com/braintrustdata/autoevals) library supplies about 25 scorers\n(LLM-judged, RAG, heuristic, embedding), and trace-level scorers can read the whole trace of an\nagent run, tool calls included. Scoring rules evaluate production logs online, and\n[eval-action ↗](https://github.com/braintrustdata/eval-action) posts experiment results on a pull\nrequest.\n\nIt has the widest SDK coverage here: TypeScript and Python, plus Go, Java, Ruby and C#. The Starter plan is free and Pro is $249 a month. Running it in your own cloud is BYOC, on Enterprise, with Braintrust operating the data plane in your account.\n\n**Choose it when** your team iterates on prompts and models through experiments and wants the\ncomparison workflow, the playground and production scoring in one commercial product.\n\n## Arize Phoenix\n\n[Phoenix ↗](https://arize.com/docs/phoenix/evaluation) is Arize's open-source observability and\nevaluation tool, built on OpenTelemetry and the OpenInference conventions, so traces are portable\nto any OTel backend. It runs LLM-judged evals from templates, datasets and experiments, and has\nprebuilt evals for agents' tool selection, tool invocation and handling of tool responses. It is\nfree to self-host with pip, Docker or Helm.\n\nThe license is the [Elastic License 2.0 ↗](https://github.com/Arize-ai/phoenix), which is\nsource-available rather than OSI open source: fine for running it yourself, restrictive if you\nwant to offer it as a hosted service. Arize AX is the commercial product, with a free tier and Pro\nat $50 a month.\n\n**Choose it when** you are standardizing on OpenTelemetry and want tracing and evals you host.\n\n## Opik\n\n[Opik ↗](https://www.comet.com/docs/opik/evaluation/metrics/overview), from Comet, is an open-source\nobservability and evaluation platform under Apache-2.0, and the full platform is free to\nself-host. It has 43 built-in metrics, including agent metrics such as Trajectory Accuracy, Agent\nTask Completion and Tool Correctness, a pytest integration for running evals on every commit,\nonline evaluation rules for production traces, and guardrails for PII and prompt injection. Comet's\ncloud has a free tier, and Pro starts at $19 a month.\n\n**Choose it when** you want a platform like Langfuse or Phoenix under a permissive open-source\nlicense, with agent metrics built in.\n\n## MLflow\n\n[MLflow ↗](https://mlflow.org/docs/latest/genai/eval-monitor/) added GenAI tracing and evaluation to\nthe ML lifecycle platform many data teams already run. `mlflow.genai.evaluate()` takes a dataset,\na prediction function and scorers: built-in LLM judges for correctness, guidelines and\ngroundedness, multi-turn judges, and scorers from DeepEval, Ragas, Phoenix and TruLens. Tool-call\ncorrectness and efficiency scorers are experimental. Automatic evaluation runs judges on incoming\nproduction traces. MLflow is Apache-2.0 and free to self-host; Databricks runs a managed version,\nand a few judges are only available there.\n\n**Choose it when** your team already tracks models in MLflow or works in Databricks.\n\n## EvalShift\n\n[EvalShift](https://www.evalshift.dev/docs) is built for one decision: is it safe to move this agent from the model it runs\non today to a candidate? The capture SDK records real model and tool calls from your agent, and\n`evalshift capture sync` promotes them into a golden suite. One run sends every example through\nboth models, scores tool selection, arguments and call structure on both sides, holds both to\ntrace rules you write, and tests each difference for significance (paired t-test or Wilcoxon,\neffect size with a confidence interval, and a false-discovery-rate correction across the run). A\n[migration policy](https://www.evalshift.dev/docs/migration-policy) turns that into one verdict, and the GitHub Action fails\nthe pull request when the verdict is fail.\n\nWhat it does not do: red teaming, production monitoring, RAG retrieval metrics, or a broad library of quality metrics. It runs offline, in Python, and pairs naturally with one of the platforms above for production. The CLI and SDK are free and open source; the hosted gate with run history and baselines starts at $49 a month (Starter); Pro is $199.\n\n**Choose it when** you are switching or upgrading the model, or editing the prompt, under an agent\nthat is already in production.\n\n## The question most tools leave to you: is the difference real?\n\nEvery tool above will show you that a candidate scored 83% and the current version 86%. Almost\nnone will tell you whether that gap is a regression or noise. Of the tools here, only Inspect AI\ndocuments standard errors and confidence intervals as part of its results, and Confident AI,\nDeepEval's platform, [says ↗](https://www.confident-ai.com/docs/changelog/2026/7/3) its test-run\ncomparisons have included statistical significance since July 2026, without naming the test. The\ncomparison views in LangSmith, Braintrust and Langfuse count or color the examples that improved\nand regressed; their docs describe no significance test.\n\nThat matters because eval scores are noisy. On a suite of 200 cases, a pass rate of 86% has a 95% Wilson interval of roughly 80.5% to 90.1%, so a three-point drop sits comfortably inside it. Model outputs also vary between runs of the same input. Two ways to stop reading noise as signal, whatever tool you use:\n\n- +**Compare per example, not per run.** Run both versions on the same cases and look at the\nper-case differences. A paired comparison is far more sensitive than two independent pass rates.\n- +**Run each case more than once,** and size the suite so each slice you would act on has enough\ncases.[How many eval cases do you need?](https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need) works through\nthe numbers.\n\n## What changed in LLM evaluation tools in 2025 and 2026\n\nThe category consolidated fast. If you are reading an older \"best tools\" list, check these first:\n\n- +**OpenAI's hosted Evals is shutting down.** OpenAI[deprecated ↗](https://developers.openai.com/api/docs/deprecations) the Evals dashboard and API in June 2026. Existing evals became read-only on October 31, 2026,\nand the product shuts down on November 30, 2026. OpenAI's migration guide points to Promptfoo.\nThe open-source[openai/evals ↗](https://github.com/openai/evals) repository is a separate project\nand is not affected.\n- +**Promptfoo is part of OpenAI** since its March 2026 acquisition, and remains MIT licensed.\n- +**Langfuse joined ClickHouse** in January 2026; self-hosting and the license are unchanged.\n- +**Weights & Biases is part of CoreWeave.** The acquisition closed in May 2025, and W&B's products\nmoved into CoreWeave Forge in September 2026. CoreWeave's docs describe Agent Lens, in public\npreview, as the successor to W&B Weave.\n- +**Galileo is now Splunk Agent Observability,** after Cisco completed its acquisition in May 2026.\n- +**Humanloop shut down in September 2025.** Its founders and part of the team joined Anthropic,\nand customer data was deleted after September 8, 2025.\n\n## Other tools you will see on these lists\n\n- +**[TruLens ↗](https://www.trulens.org/)** : an MIT-licensed eval and tracing library, now\nmaintained by Snowflake, with RAG feedback functions and seven agent metrics, including tool\nselection and plan adherence.\n- +**[W&B Weave ↗](https://github.com/wandb/weave)** : tracing and`weave.Evaluation` with scorers,\nApache-2.0 SDK; see the CoreWeave note above before starting a new project on it.\n- +**[Galileo ↗](https://docs.galileo.ai/)** : agent metrics such as Tool Selection Quality and Action\nCompletion, plus small evaluator models for guardrails; now sold through Splunk.\n\n## How to choose an LLM evaluation tool\n\nStart from the job, not the feature list:\n\n| If your job is | Start with | \n|---|---|\n| Choosing between models or prompts before launch | Promptfoo | \n| Red teaming and security testing | Promptfoo, or DeepEval's DeepTeam | \n| Quality metrics in the pytest suite you already run | DeepEval | \n| Scoring a RAG pipeline | Ragas, or DeepEval's RAG metrics | \n| Agent benchmarks in sandboxes, with error bars | Inspect AI | \n| Tracing production and scoring it, self-hosted | Langfuse, Phoenix or Opik | \n| One commercial platform for traces, evals and prompts | LangSmith or Braintrust | \n| Evals inside an existing MLflow or Databricks setup | MLflow | \n| Gating a model swap under a production agent | EvalShift | \n\nThen check the six questions above against your shortlist, and run two or three tools on the same\nsmall suite of your own cases before committing. A tool that grades only final text will report\ngreen on an agent that skipped a tool call; [what breaks when you switch\nLLMs](https://www.evalshift.dev/blog/what-breaks-when-you-switch-llms) shows what that looks like in practice.\n\n## FAQ\n\n**What are the best LLM evaluation tools?** It depends on the job. Promptfoo for comparing prompts\nand models and for red teaming; DeepEval for metrics in pytest; Ragas for RAG; Inspect AI for agent\nevals in sandboxes; Langfuse, Phoenix or Opik for self-hosted tracing with evals; LangSmith or\nBraintrust for a commercial platform; MLflow on Databricks; and EvalShift for gating a model swap\nunder a production agent.\n\n**What is an LLM evaluation framework?** A library or CLI you run in your own code to test an LLM\napplication: you write test cases, choose scorers such as exact checks, metrics or an LLM judge,\nand run the suite locally or in CI. Promptfoo, DeepEval, Inspect AI and Ragas are frameworks;\nplatforms such as LangSmith and Langfuse add tracing and production monitoring around evals.\n\n**Which LLM evaluation tools are open source?** Promptfoo (MIT), DeepEval (Apache-2.0), Inspect AI\n(MIT), Ragas (Apache-2.0), Langfuse (MIT except its `ee` folders), Opik (Apache-2.0), MLflow\n(Apache-2.0) and TruLens (MIT). Arize Phoenix is source-available under the Elastic License 2.0.\nLangSmith and Braintrust are proprietary platforms with MIT-licensed evaluator libraries. The\nEvalShift CLI is Apache-2.0, and its SDK and GitHub Action are MIT.\n\n**Which LLM evaluation tool is best for AI agents?** One that grades the trace, not only the final\nreply. Promptfoo, DeepEval, LangSmith's agentevals, Opik, Ragas and TruLens ship tool-call or\ntrajectory checks; Inspect AI runs agents in sandboxes; EvalShift compares two models' tool calls\non the same recorded runs. Pick by whether you are testing one version or comparing two.\n\n**Is OpenAI Evals still available?** OpenAI's hosted Evals dashboard and API became read-only on\nOctober 31, 2026 and shut down on November 30, 2026, and OpenAI's migration guide points to\nPromptfoo. The open-source openai/evals repository is separate and still available.\n\n**Do I need more than one LLM evaluation tool?** Usually two: a platform that traces production and\nscores live traffic, and a framework or gate that tests a change before it merges. They answer\ndifferent questions, and most of the tools here run side by side without integration.\n\n## Keep reading\n\n- +[How to evaluate AI agents](https://www.evalshift.dev/blog/how-to-evaluate-ai-agents) : what to measure, at which level,\nand against which environment, whatever tool you pick.\n- +[AI evals for agents](https://www.evalshift.dev/blog/ai-evals-for-agents) : what an eval is made of, and how to write one.\n- +[LLM-as-a-judge](https://www.evalshift.dev/blog/llm-as-a-judge) : the scorer most of these tools rely on, and how to\nvalidate it.\n- +[LangSmith alternatives](https://www.evalshift.dev/blog/langsmith-alternatives) : Langfuse vs LangSmith on pricing,\nself-hosting and retention, and the other platforms to consider.\n- +[Braintrust alternatives](https://www.evalshift.dev/blog/braintrust-alternatives) : Braintrust vs Langfuse with a worked\nagent cost example, and Promptfoo alternatives.\n- +[EvalShift compared](https://www.evalshift.dev/compare) : side-by-side pages for Promptfoo, DeepEval, LangSmith and\nLangfuse.", "url": "https://wpnews.pro/news/the-best-llm-evaluation-tools-in-2026-compared-by-the-job-they-do", "canonical_source": "https://www.evalshift.dev/blog/best-llm-evaluation-tools", "published_at": "2026-11-03 00:00:00+00:00", "updated_at": "2026-11-17 20:11:25.264988+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "mlops", "ai-agents", "developer-tools"], "entities": ["Promptfoo", "DeepEval", "Ragas", "Inspect AI", "Langfuse", "Arize Phoenix", "Opik", "OpenAI"], "also_reported_by": [], "takeaway": "Most teams need two tools rather than one — a production observability platform such as Langfuse, Arize Phoenix or Opik plus a pre-merge check such as Promptfoo or DeepEval — and OpenAI's hosted Evals product is no longer an option after its November 30, 2026 shutdown.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/the-best-llm-evaluation-tools-in-2026-compared-by-the-job-they-do", "markdown": "https://wpnews.pro/news/the-best-llm-evaluation-tools-in-2026-compared-by-the-job-they-do.md", "text": "https://wpnews.pro/news/the-best-llm-evaluation-tools-in-2026-compared-by-the-job-they-do.txt", "jsonld": "https://wpnews.pro/news/the-best-llm-evaluation-tools-in-2026-compared-by-the-job-they-do.jsonld"}}