Promptfoo vs DeepEval compared on test format, judge costs, repeated runs, agent checks, red teaming, CI, pricing and telemetry, as of November 2026.
Promptfoo and DeepEval are the two most-used open-source LLM eval frameworks, and they are built for different jobs. Choose Promptfoo when you are comparing prompts or models side by side, when most of your checks can be written as code, when you need red teaming, or when your team is not on Python. Choose DeepEval when your tests live in pytest and you want a ready-made metric for RAG, agents or conversations, and you accept a judge model call for almost every metric on every case.
As of November 2026, Promptfoo is at 0.125.1 under the MIT license and has been part of OpenAI since March. DeepEval is at 4.3.4 under Apache-2.0 and is built by Confident AI, which sells the hosted platform around it. Both collect usage telemetry by default.
This post compares them on how a test is written, what a run costs, how each compares two models, what each does with repeated runs, agent checks, red teaming, CI, pricing and what leaves your machine.
A disclosure: I build EvalShift, an eval tool for a narrower job than either of these, and it appears only in the section on what both leave to you. Every claim about Promptfoo or DeepEval comes from that project's docs, repository or pricing page, read on November 13, 2026.
Promptfoo vs DeepEval at a glance #
| Promptfoo | DeepEval | |
|---|---|---|
| License | MIT | Apache-2.0 |
| Owner | OpenAI, since the acquisition announced March 9, 2026 | Confident AI |
| Runs on | Node 22.22+, through npm, npx or Homebrew | Python 3.10+, through pip; a TypeScript package on npm |
| Latest release | 0.125.1, October 25, 2026 | 4.3.4, November 11, 2026 |
| GitHub stars | 27.1k | 19.5k |
| Downloads | 1.2M a week on npm | 3.3M a month on PyPI |
| A test is | A YAML row of variables and assertions, run against every prompt and provider | A pytest function with a test case and metrics |
| Checks | About 40 deterministic assertion types and 15 model-graded | 50+ metrics, almost all graded by an LLM judge |
| Comparing two models | Built in: a matrix and a local web viewer | A judge picks a winner; run comparison is on Confident AI |
| Repeated runs | --repeat , with a Wilson 95% interval per repeated test |
--repeat , andflaky=True to stop a test blocking |
| Agent checks | Deterministic trajectory assertions on traces | Judged trajectory metrics, plus a deterministic tool check |
| Red teaming | Built in: 157 plugins, 34 strategies | A separate package, DeepTeam |
| CI | Exit code, pass-rate threshold, official GitHub Action | pytest exit code |
| Hosted side | Enterprise, custom price | Confident AI: free, $200 or $2,000 a month |
| Usage telemetry | On by default | On by default |
Sources: Promptfoo on GitHub ↗, Promptfoo on npm ↗, DeepEval on GitHub ↗, DeepEval on PyPI ↗, and the docs pages linked in each section below.
The same test in each #
One test, written twice: a support reply must repeat the order number and must not promise a delivery date.
In Promptfoo it is a config file. promptfoo eval runs every prompt against every provider and
every test case:
prompts:
- 'Reply to this support message in one sentence: {{message}}'
providers:
- openai:gpt-6-luna
- anthropic:messages:claude-sonnet-5-5
tests:
- vars:
message: Where is my order 4417?
assert:
- type: icontains
value: '4417'
- type: llm-rubric
value: Does not promise a delivery date
npx promptfoo@latest eval
In DeepEval it is a pytest file. You call your own code, build a test case from the result and attach metrics:
import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
from app import reply # your function that calls the model
no_promise = GEval(
name="No delivery promise",
criteria="The actual output does not promise a delivery date.",
evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
threshold=0.5,
)
@pytest.mark.parametrize("message, order_id", [("Where is my order 4417?", "4417")])
def test_reply(message, order_id):
output = reply(message)
assert order_id in output
assert_test(LLMTestCase(input=message, actual_output=output), [no_promise])
deepeval test run test_support.py
Three differences show up even in a test this small:
- +Who calls the model. Promptfoo calls it for you from the
providerslist, or calls your application through a script or HTTP provider. A DeepEval test calls your code directly, like any other pytest test. - +The matrix. Adding a second model in Promptfoo is one line, and every test runs against both. In DeepEval you parametrize the test over models yourself.
- +The deterministic check. In Promptfoo it is an assertion type. In DeepEval it is a plain
Python
assert, because the metric library is for the judged part.
If you copy DeepEval code from an older post, note one rename: LLMTestCaseParams is deprecated in
favour of SingleTurnParams. The old name still imports, with a warning.
Checks: assertions against metrics, and what a run costs #
This is the largest practical difference between the two.
Promptfoo's assertion list ↗ is
mostly code. About 40 types run without a model: contains, regex, is-json, is-sql,
javascript, python, text-overlap scores such as ROUGE and BLEU, latency, cost, and checks
on tool calls. About 15 more are model-graded, including llm-rubric, g-eval, factuality and
the RAG checks context-faithfulness, context-recall and answer-relevance. A suite can be
entirely deterministic, and then the only model you pay for is the one under test.
DeepEval ships more than 50 metrics ↗, and its own metrics page says almost all of them are LLM-as-a-judge. The deterministic ones are Exact Match, Pattern Match, JSON Correctness and Tool Correctness ↗, which compares the tools called with the tools expected and brings in a model only if you pass a list of available tools. Every metric has a default threshold of 0.5 and returns a score from 0 to 1 with a reason.
So the arithmetic for a DeepEval suite is simple. 200 cases with three judged metrics is at least
600 judge calls per run, on top of the 200 calls to your own application. Run each case three
times and it is 1,800. The same checks in Promptfoo cost the same if you write them as
llm-rubric, and nothing if they can be a regex, a JSON schema or a few lines of Python.
Two things about the judge are worth knowing:
- +The default judge. Both use OpenAI when you set nothing else. DeepEval defaults to an OpenAI model and can be pointed at another provider or at Ollama. Promptfoo picks its grader from whichever credentials are in the environment, and OpenAI wins when both an OpenAI and an Anthropic key are set.
- +Jev. In September 2026 DeepEval addedeval modes ↗ . In
system_onemode a metric is decided by Jev, a model from TypeSafe AI that the docs call System One, with no LLM call. It still needs a TypeSafe API key, so your test cases go to another vendor in place of the judge provider.
A ready-made metric saves you writing a rubric. It does not save you validating it: a judged metric with a 0.5 threshold is a classifier you have not measured yet. When to trust an LLM judge covers how to check one against hand labels.
Comparing two prompts or models #
Promptfoo was built for this. List two providers, or two prompts, and one eval runs the full
matrix. promptfoo view opens a local web viewer ↗
with the outputs side by side, a filter for rows where they differ, a comparison against an
earlier eval and charts of pass rate per column. The select-best and max-score assertions pick
a winner per test case. All of it is in the open-source CLI.
DeepEval's open-source answer is Arena G-Eval ↗: you give it the outputs of several contestants and a blinded LLM judge picks one. Test runs are saved locally as JSON, which the docs suggest diffing in Git. A side-by-side comparison of two runs, with regressions marked, is a Confident AI feature and needs an API key.
Neither tool tests whether one model is better than the other by more than chance. Promptfoo scores each column on its own. Confident AI's changelog ↗ says run comparisons have included statistical significance since July 2026, and its docs do not name the test.
Repeated runs and flaky tests #
LLM outputs vary between runs, so a case that passes once has not been shown to pass. The two tools handle this differently, and Promptfoo moved ahead here in October 2026.
Promptfoo. --repeat 3 runs every test three times. Since 0.125.0 the summary
flags tests that both passed and failed ↗
across repeats and reports a Wilson 95% confidence interval on each repeated test's pass rate. The
catch is the cache: responses are
cached on disk by default ↗ for 14 days,
and each repeat has its own cache entry. Run the same eval twice and the second run replays the
first, so use --no-cache when you are measuring variation. The interval is only reported for
uncached results.
DeepEval. deepeval test run -r 3 repeats each test case. There is no interval in its docs.
What it added, in July 2026, is flaky=True on a test case or a metric: the result is still
recorded, and a failure prints a warning and no longer fails the run.
Keep the size of the interval in mind before reading much into three repeats. Three passes out of three gives a Wilson interval of 44% to 100%. Ten out of ten gives 72% to 100%. A repeat count you can afford on every pull request tells you which cases flip often, and little more.
Agents and tool calls #
Both check more than the final reply, from opposite directions.
Promptfoo checks tool calls with code. On the output: is-valid-openai-tools-call and
tool-call-f1. On a trace: trajectory:tool-used, trajectory:tool-args-match,
trajectory:tool-sequence and trajectory:step-count, with trajectory:goal-success as the one
judged check. The trajectory assertions read
OpenTelemetry traces ↗, which Promptfoo receives on a
built-in OTLP endpoint, so your agent has to be instrumented. For multi-turn tests there is a
simulated-user provider. By default that user is generated by Promptfoo's hosted models.
DeepEval instruments your code with an @observe decorator and
scores the trace ↗ with Task
Completion, Step Efficiency, Plan Adherence and Plan Quality. All four are judged. Tool
Correctness is the deterministic one; Argument Correctness is judged and takes no reference. A
ConversationSimulator plays the user in multi-turn tests with your own judge model, and there
are metrics for MCP use.
If the question is "did the agent call refund after lookup_order, with this order id",
Promptfoo's deterministic assertions answer it without a judge. If the question is "was the plan
sensible", DeepEval has a metric ready. Evaluating agent tool
calls goes through which of these checks catch which failures.
Red teaming: built in, or a second package #
Promptfoo has red teaming in the same CLI. Its docs list 157 plugins ↗ that generate adversarial inputs and 34 strategies ↗ that deliver them. The free Community edition allows 10,000 probes a month.
Read the data-handling page ↗ before you run it against anything sensitive. Your target is always called from your machine. Generating and grading the attacks is another matter:
- +Without an OpenAI key, generation and grading go to Promptfoo's API, along with your application's stated purpose, the prompts sent to your target, its responses and your email address.
- +With your own key, most of it runs through your account. But plugins and strategies marked as remote in the docs always use Promptfoo's hosted inference. That includes 11 of the 34 strategies, among them the multi-turn GOAT and Hydra attacks.
- +
PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=trueturns remote generation off. Thedocs ↗ warn that local generation quality "is generally low for most models".
DeepEval's red teaming moved to a separate package, DeepTeam ↗: Apache-2.0, version 1.0.10 from October 16, 2026. Its README claims more than 50 vulnerabilities and more than 20 attack methods, and it runs the attacks with your own model, with no hosted generation step. It releases less often than DeepEval, four times since July 2026.
Promptfoo's red teaming is the larger and more developed of the two. DeepTeam is the choice when the attack traffic must not leave your own model accounts.
CI #
Promptfoo exits with
code 100 ↗ when a test fails or the pass rate
is below PROMPTFOO_PASS_RATE_THRESHOLD, and writes JUnit XML, JSON, CSV or HTML. The official
promptfoo-action ↗ posts a summary comment with
pass and fail counts on a pull request, and has inputs for a pass threshold, a repeat count and a
minimum number of passes per repeated test. Its README is explicit that it evaluates only the
current checkout. It does not run the base branch for you.
DeepEval in CI is pytest. A failed assert_test fails the test and the job, and the
docs ↗ show a plain workflow that runs
deepeval test run. There is no pass-rate threshold flag. Gating on a policy, with checks posted
to the pull request, runs through Confident AI.
Either way, a gate that requires every judged case to pass on every run will fail on noise. LLM regression testing in CI covers what to gate on instead.
What is open, what is paid, and who owns each #
Promptfoo. The CLI, assertions, red teaming and local viewer are MIT. OpenAI announced the acquisition ↗ on March 9, 2026, and Promptfoo's own post ↗ said it would remain open source and keep supporting a range of providers. Eight months later the README says the same and names the MIT license, the license file is unchanged, and releases continue, eight of them in the last three months. The paid side ↗ is Enterprise and On-Premise, both priced on request, adding SSO, role-based access, dashboards and continuous monitoring. The open-source server can be self-hosted, but the docs ↗ say that is not recommended for production: it stores results in SQLite and has no built-in authentication.
DeepEval. The metric library and test runner are Apache-2.0. Run history, comparison,
production tracing and dataset management are Confident AI ↗,
which is proprietary and bills per organization: a free plan with 2 seats, one project and 5 test
runs a week; Starter at $200 a month with unlimited seats; Team at $2,000 a month. Self-hosting
is on Enterprise. Uploads start only after deepeval login.
The practical difference: Promptfoo gives you a usable local viewer for free and puts team features behind a sales call. DeepEval gives you terminal output for free and a published price for the dashboard.
What each sends by default #
| Promptfoo | DeepEval | |
|---|---|---|
| Usage telemetry | On by default | On by default, to PostHog |
| What it records | Commands and assertion types, version, CI flag; user ID and email if you are logged in | Event and metric names, an anonymous ID, public IP address |
| Turn it off | PROMPTFOO_DISABLE_TELEMETRY=1 |
DEEPEVAL_TELEMETRY_OPT_OUT=1 |
| Judged checks | Output and rubric go to the grader provider | Test case fields go to the judge provider |
| Results upload | After promptfoo share |
After deepeval login |
Both projects say telemetry excludes prompts and outputs: Promptfoo telemetry ↗, DeepEval data privacy ↗. The larger flow is the judge. If your test cases come from production, every judged check sends customer data to a model provider, in both tools.
What users complain about #
These are reports from each project's issue tracker, not my findings, and some may be fixed by the time you read this.
Promptfoo:
- +An open issue ↗ reports one request still going out with telemetry disabled.
- +A red-team scan in CI ↗ stopped at an email verification prompt, and a strategy refused to run with remote generation off.
- +A request open since October 2025 ↗ asks for a consistency metric over repeats: the share of cases that pass on all N attempts.
DeepEval:
- +An issue open since March 2025 ↗ , with 16 upvotes, asks for a way to stop the console output promoting Confident AI.
- +Smaller and local judge models return JSON the metrics cannot parse ↗ .
- +The pytest plugin loads for every pytest run ↗ in the environment, which makes telemetry harder to disable from code.
Using both #
Plenty of teams do, and the split is natural: DeepEval metrics in the pytest suite that runs on every pull request, and Promptfoo for red teaming before a release and for lining up candidate models when one is being chosen. They share nothing but your provider keys.
The cost of running both is two config formats and two sets of results that do not know about each other. If you only want one, pick by where most of your checks fall: deterministic and comparative, Promptfoo; judged and Python-native, DeepEval.
What both leave to you #
Promptfoo and DeepEval answer "does this output pass my checks?" The question a model upgrade raises is different: on the same cases, does the new model behave differently from the old one, and is the difference real?
Promptfoo gets closest. Its matrix puts the two models side by side, and since October it tells you how stable each repeated test is. What it does not do is pair the two columns: take each case, compute the difference between the old model and the new one, and test whether those differences add up to more than noise. DeepEval scores one run at a time and leaves comparison to Confident AI.
That paired comparison is the job EvalShift does. It runs your recorded production cases through both models, scores tool selection, arguments and trace rules on both sides, and tests each per-example difference with a paired t-test or Wilcoxon, an effect size with a confidence interval and a false-discovery-rate correction across the run. A migration policy turns the result into a verdict that passes or fails the pull request.
In one published run, the candidate model was 58.8% cheaper and 75.2% faster with no failed calls, and the verdict was FAIL: it picked a different tool on 25% of comparisons against a 10% limit.
What EvalShift does not do: red teaming, a metric library the size of DeepEval's, or a matrix of five models. The CLI is free and open source, and it runs beside either tool. EvalShift vs Promptfoo and EvalShift vs DeepEval go dimension by dimension.
How to choose #
| If you | Use |
|---|---|
| Are picking between models or prompts | Promptfoo |
| Can write most checks as code | Promptfoo |
| Need red teaming with the widest attack coverage | Promptfoo |
| Need red teaming that stays inside your own model accounts | DeepTeam |
| Have no Python in the stack | Promptfoo, or DeepEval's TypeScript package if you want its metrics |
| Keep tests in pytest and want them to stay there | DeepEval |
| Want RAG, conversation or agent metrics without writing rubrics | DeepEval |
| Want a hosted dashboard with a published price | DeepEval with Confident AI |
| Need an eval tool not owned by a model provider | DeepEval |
| Are swapping the model under a production agent | EvalShift, beside either |
Then write the same ten test cases in both. It takes an afternoon, and the one whose failures you find easier to read is the one your team will keep running.
FAQ #
Is Promptfoo or DeepEval better? Neither is better overall. Promptfoo is better for comparing prompts and models side by side, for deterministic checks and for red teaming. DeepEval is better for teams that keep tests in pytest and want ready-made judged metrics for RAG, agents and conversations. Many teams run both.
What is the difference between Promptfoo and DeepEval? Promptfoo is a Node command-line tool configured in YAML that runs every prompt against every model and checks outputs with about 40 deterministic and 15 model-graded assertion types. DeepEval is a Python library that runs evals as pytest tests with more than 50 metrics, almost all of them graded by an LLM judge.
Can you use Promptfoo and DeepEval together? Yes. A common split is DeepEval metrics in the pytest suite on every pull request, and Promptfoo for red teaming and for comparing candidate models. They share nothing but provider keys and run as separate CI jobs.
Which is better for red teaming, Promptfoo or DeepEval? Promptfoo has the larger feature: 157 plugins and 34 strategies in the same CLI, with 10,000 probes a month free. Many of them use Promptfoo's hosted inference. DeepEval's red teaming is a separate Apache-2.0 package, DeepTeam, which runs attacks with your own model.
Are Promptfoo and DeepEval free? Both frameworks are free and open source: Promptfoo under MIT, DeepEval under Apache-2.0. You pay the model provider for every call, including judge calls. Promptfoo Enterprise is priced on request. Confident AI, the platform for DeepEval, has a free plan, then $200 and $2,000 a month per organization.
Does DeepEval support TypeScript? Yes. There is a deepeval package on npm, built from the
same repository, that runs on Vitest. It is younger than the Python library, at version 0.9 as of
November 2026. Promptfoo is a Node tool and works with any language through its config file.
Is Promptfoo still open source now that OpenAI owns it? Yes. OpenAI announced the acquisition on March 9, 2026. As of November 2026 the repository is MIT licensed, the license file is unchanged, and the project is still releasing, with 0.125.1 on October 25, 2026.
Do Promptfoo and DeepEval test statistical significance? Not between two models. Since version 0.125.0 Promptfoo reports a Wilson 95% confidence interval on the pass rate of each repeated test. DeepEval's docs describe no interval or test. Confident AI's changelog says its run comparisons include statistical significance, without naming the test.
Keep reading #
- +The best open-source LLM evaluation frameworks in 2026 : Promptfoo and DeepEval next to Inspect AI, Pydantic Evals, Ragas and five more.
- +Promptfoo alternatives : what to use for evals, red teaming and run history if Promptfoo is not the fit.
- +LLM testing : the six steps of testing an LLM application, whichever framework runs them.
- +When to trust an LLM judge : validating the judged metrics both tools rely on.