DeepEval's TypeScript SDK is out in beta. Every metric, model, and tracing integration you know from Python, running as a gate in your CI/CD pipeline.
Two months ago I wrote about why we put TypeScript inside DeepEval's Python monorepo instead of giving it its own repo. At the time it was still a client wrapper around Confident AI — it couldn't run a single metric.
Today it can run all of them. DeepEval for TypeScript is out in beta:
The headline isn't the metrics though. It's where they run: in CI/CD, as a gate on your pull requests.
Evals belong in your Typescript CI pipeline The reason I keep pushing this is that an eval you run by hand is a nice number, and an eval that runs on every PR is a decision. That's the whole point of an eval harness — a regression in your agent should block a merge exactly like a regression in your business logic does, without anyone remembering to check.
For that to happen, evals have to live where your pipeline already looks: your test suite. In Python that's Pytest. In TypeScript it's Vitest, and the surface is a single matcher — toPass()
— on top of a test file you'd recognize without ever having used DeepEval:
Then run it:
Plain vitest
works too, but you'd be leaving most of the value on the table. npx deepeval test run
captures a trace per test, caches metric results so a re-run doesn't re-bill you, and gives you the same flags Python users have — --official
, `-i/--identifier`
, `--max-concurrent`
, `-c/--use-cache`
, `--ignore-errors`
.
That one command is the whole integration story. It exits non-zero when a metric falls below its threshold, so any CI provider that runs a shell step already knows what to do with it:
Note what isn't in there: no CONFIDENT_API_KEY
, no hosted service, no vendor in the critical path of your merge queue. Give it a judge key and the whole thing runs on the runner. Add the key later if you want shared reports and regression tracking across commits, and add --official
on main
to mark the baseline that future runs get compared against. The full walkthrough is in [unit testing in CI/CD](/docs/evaluation-unit-testing-in-ci-cd).
[Every metric, open-source](#every-metric-open-source)
This is the part I care most about, because a second language that only ships the easy half of the metric library is worse than no second language at all.
47 of DeepEval's 49 metrics are ported and open-source in TypeScript. Not a curated subset — G-Eval with log-prob weighted scoring, DAG decision graphs, the multi-turn suite, the MCP metrics, the multimodal ones, arena comparisons. The only two that haven't landed are AgentLoopDetectionMetric
and ToolPermissionMetric
, and they'll be in shortly.
They agree with Python because they aren't rewritten from memory. Both SDKs compile the same language-neutral prompt templates — one JSON bundle of judge prompts, shared by the two implementations. Only the orchestration and the output schema (Pydantic in Python, Zod in TypeScript) are written per-language. So when we change a judge prompt, both languages change together or neither does.
The API is idiomatic TypeScript rather than transliterated Python — an options object instead of keyword arguments, camelCase
, and measure()
is always async
(there's no sync/a_measure
split, everything underneath is already async). Full list on the [metrics page](/docs/metrics-introduction).
[The models that judge them](#the-models-that-judge-them)
An LLM-as-a-judge metric is only as portable as the providers it can run on, so the model layer came over too. TypeScript ships 13 model integrations.
Two notes. LiteLLM is Python-only and always will be — AISDKModel
is the TypeScript answer to "route my judge through anything." And the default judge model is generated from Python's DEFAULT_MODELS
, so an unconfigured metric lands on the same model in both languages.
Set one once from the CLI and every metric picks it up:
Tracing, and evals on the trace Scoring a final string is fine for a RAG pipeline. It isn't enough for an agent, where the interesting failures are in the trajectory — the tool it shouldn't have called, the plan it abandoned, the four steps it took to do one thing.
So TypeScript gets tracing, with 6 framework integrations that turn your agent's execution into a span tree with no rewriting:
| Framework | Import | Setup |
|---|---|---|
deepeval/openai
`instrumentOpenAI(client)`
[LangChain](/integrations/frameworks/langchain)&[LangGraph](/integrations/frameworks/langgraph)`deepeval/integrations/langchain`
`new DeepEvalCallbackHandler({})`
[OpenAI Agents SDK](/integrations/frameworks/openai-agents)`deepeval/integrations/openai-agents`
`new DeepEvalTracingProcessor()`
[Mastra](/integrations/frameworks/mastra)`deepeval/integrations/mastra`
`new DeepEvalExporter()`
[Vercel AI SDK](/integrations/frameworks/ai-sdk)`deepeval/integrations/ai-sdk`
`configureAiSdkTracing({})`
deepeval/integrations/openinference
instrumentOpenInference({})
Mastra and the Vercel AI SDK are the fun ones — they have no Python counterpart at all, so for once TypeScript is ahead.
The reason this matters for CI/CD is that instrumenting your agent and gating on it are the same piece of work, not two. Once the app is traced, you hand toPass()
a golden and a task, and it runs your agent, captures the trace, and scores the whole trajectory:
You can also attach metrics to individual spans instead of the whole trace — component-level evals, where a retriever gets contextual precision and a tool call gets argument correctness, in the same test run.
The CLI came over too
Same binary name, same commands, one npx
in front:
| Command | What it does |
|---|---|
npx deepeval test run |
Run your eval suite as a gate, locally or in CI |
npx deepeval inspect |
Browse locally captured traces in a terminal UI |
npx deepeval view |
Open the latest test run on Confident AI |
npx deepeval login / logout |
Authenticate with Confident AI |
npx deepeval set-openai (and 11 more) |
Configure the judge model per provider |
npx deepeval gate |
Run a governance policy check |
npx deepeval diagnose |
Print the effective config when something's off |
npx deepeval inspect
is the one I'd try first. Traces are written to a local .json
file on your machine by default — nothing leaves your laptop — and inspect
renders them as a trace tree with per-span scores and metric reasons. When a coding agent is driving the loop, that's what stops it from overfitting to a number it can't see the reasoning behind.
What's missing, and why it's a beta I'd rather tell you than let you find out. Three things exist in Python and don't exist in TypeScript yet:
Synthesizer— generating goldens from your documents or knowledge base. Bring your own dataset for now: load goldens from a CSV, a JSON file, or Confident AI.Benchmarks— MMLU, HellaSwag, and the rest of the foundational-model benchmark suite.** Prompt optimization**— automatic prompt search against a metric.
There's also no score-parity guarantee. The prompts are shared and parity-checked, but we've spot-verified numeric scores for sanity rather than asserted them equal to Python's. Don't mix languages inside one longitudinal comparison yet.
That's what the beta label is for. Everything above it — metrics, models, tracing, the CI/CD gate, the CLI — is what we're asking you to actually use and break.
Getting started Or don't write the test file yourself. Install the DeepEval skill and let your coding agent drive the eval driven development loop:
Python still leads on behavior and TypeScript follows close behind — that hasn't changed, and one repo is what keeps "close behind" true. What changed is that "close behind" now means 47 metrics, 13 model providers, 6 tracing integrations, and a command that turns red on your pull requests, rather than a client that couldn't score anything.
DeepEval is free and 100% open-source on ⭐ GitHub. If TypeScript is your stack, this is the release I've been wanting to write for a year — go break it and open an issue.
FAQs
How do I install DeepEval for TypeScript? #
npm install -D deepeval
. It's the same package name as the Python one, published to npm, and it lives in the same open-source repoas Python.
How do I run evals in CI/CD? #
npx deepeval test run
as a step in your pipeline. It exits non-zero when a metric falls below its threshold, so a regression blocks the merge — any provider that runs a shell step works. Your evals live in your test suite (DeepEval registers a toPass()
matcher for Vitest), so there's no second harness to maintain.## Are all the metrics available in TypeScript?
AgentLoopDetectionMetric
and ToolPermissionMetric
are outstanding. Both SDKs compile the same language-neutral judge prompts, so behavior stays aligned.## Which LLM providers can I use as a judge?
Which agent frameworks can it trace? #
What's missing from the TypeScript SDK? #
Do I need a Confident AI account? #
OPENAI_API_KEY
and everything runs locally, with traces written to a local .json
file you can browse with npx deepeval inspect
. Adding CONFIDENT_API_KEY
is optional and only sends results to the cloud.