DeepEval Open-Sourced for TypeScript

DeepEval has open-sourced its TypeScript SDK in beta, enabling all 47 of its 49 metrics to run as a gate in CI/CD pipelines via a single Vitest matcher. The SDK, which compiles the same language-neutral prompt templates as the Python version, includes 13 model integrations and runs without a hosted service or API key. The two missing metrics, AgentLoopDetectionMetric and ToolPermissionMetric, are expected shortly.

DeepEval for TypeScript, now fully open-source DeepEval's TypeScript SDK is out in beta. Every metric, model, and tracing integration you know from Python, running as a gate in your CI/CD pipeline. Two months ago I wrote about why we put TypeScript inside DeepEval's Python monorepo /blog/typescript-in-deepeval-monorepo instead of giving it its own repo. At the time it was still a client wrapper around Confident AI — it couldn't run a single metric. Today it can run all of them. DeepEval for TypeScript is out in beta: The headline isn't the metrics though. It's where they run: in CI/CD, as a gate on your pull requests. Evals belong in your Typescript CI pipeline evals-belong-in-your-typescript-ci-pipeline The reason I keep pushing this is that an eval you run by hand is a nice number, and an eval that runs on every PR is a decision. That's the whole point of an eval harness /blog/what-is-an-eval-harness — a regression in your agent should block a merge exactly like a regression in your business logic does, without anyone remembering to check. For that to happen, evals have to live where your pipeline already looks: your test suite. In Python that's Pytest. In TypeScript it's Vitest, and the surface is a single matcher — toPass — on top of a test file you'd recognize without ever having used DeepEval: Then run it: Plain vitest works too, but you'd be leaving most of the value on the table. npx deepeval test run captures a trace per test, caches metric results so a re-run doesn't re-bill you, and gives you the same flags Python users have — --official , -i/--identifier , --max-concurrent , -c/--use-cache , --ignore-errors . That one command is the whole integration story. It exits non-zero when a metric falls below its threshold, so any CI provider that runs a shell step already knows what to do with it: Note what isn't in there: no CONFIDENT API KEY , no hosted service, no vendor in the critical path of your merge queue. Give it a judge key and the whole thing runs on the runner. Add the key later if you want shared reports and regression tracking across commits, and add --official on main to mark the baseline that future runs get compared against. The full walkthrough is in unit testing in CI/CD /docs/evaluation-unit-testing-in-ci-cd . Every metric, open-source every-metric-open-source This is the part I care most about, because a second language that only ships the easy half of the metric library is worse than no second language at all. 47 of DeepEval's 49 metrics are ported and open-source in TypeScript. Not a curated subset — G-Eval with log-prob weighted scoring, DAG decision graphs, the multi-turn suite, the MCP metrics, the multimodal ones, arena comparisons. The only two that haven't landed are AgentLoopDetectionMetric and ToolPermissionMetric , and they'll be in shortly. They agree with Python because they aren't rewritten from memory. Both SDKs compile the same language-neutral prompt templates — one JSON bundle of judge prompts, shared by the two implementations. Only the orchestration and the output schema Pydantic in Python, Zod in TypeScript are written per-language. So when we change a judge prompt, both languages change together or neither does. The API is idiomatic TypeScript rather than transliterated Python — an options object instead of keyword arguments, camelCase , and measure is always async there's no sync/ a measure split, everything underneath is already async . Full list on the metrics page /docs/metrics-introduction . The models that judge them the-models-that-judge-them An LLM-as-a-judge metric is only as portable as the providers it can run on, so the model layer came over too. TypeScript ships 13 model integrations . Two notes. LiteLLM is Python-only and always will be — AISDKModel is the TypeScript answer to "route my judge through anything." And the default judge model is generated from Python's DEFAULT MODELS , so an unconfigured metric lands on the same model in both languages. Set one once from the CLI and every metric picks it up: Tracing, and evals on the trace tracing-and-evals-on-the-trace Scoring a final string is fine for a RAG pipeline. It isn't enough for an agent, where the interesting failures are in the trajectory — the tool it shouldn't have called, the plan it abandoned, the four steps it took to do one thing. So TypeScript gets tracing, with 6 framework integrations that turn your agent's execution into a span tree with no rewriting: | Framework | Import | Setup | |---|---|---| | deepeval/openai instrumentOpenAI client LangChain /integrations/frameworks/langchain & LangGraph /integrations/frameworks/langgraph deepeval/integrations/langchain new DeepEvalCallbackHandler {} OpenAI Agents SDK /integrations/frameworks/openai-agents deepeval/integrations/openai-agents new DeepEvalTracingProcessor Mastra /integrations/frameworks/mastra deepeval/integrations/mastra new DeepEvalExporter Vercel AI SDK /integrations/frameworks/ai-sdk deepeval/integrations/ai-sdk configureAiSdkTracing {} deepeval/integrations/openinference instrumentOpenInference {} Mastra and the Vercel AI SDK are the fun ones — they have no Python counterpart at all, so for once TypeScript is ahead. The reason this matters for CI/CD is that instrumenting your agent and gating on it are the same piece of work, not two. Once the app is traced, you hand toPass a golden and a task, and it runs your agent, captures the trace, and scores the whole trajectory: You can also attach metrics to individual spans instead of the whole trace — component-level evals /docs/evaluation-component-level-llm-evals , where a retriever gets contextual precision and a tool call gets argument correctness, in the same test run. The CLI came over too the-cli-came-over-too Same binary name, same commands, one npx in front: | Command | What it does | |---|---| npx deepeval test run | Run your eval suite as a gate, locally or in CI | npx deepeval inspect | Browse locally captured traces in a terminal UI | npx deepeval view | Open the latest test run on Confident AI | npx deepeval login / logout | Authenticate with Confident AI | npx deepeval set-openai and 11 more | Configure the judge model per provider | npx deepeval gate | Run a governance policy check | npx deepeval diagnose | Print the effective config when something's off | npx deepeval inspect is the one I'd try first. Traces are written to a local .json file on your machine by default — nothing leaves your laptop — and inspect renders them as a trace tree with per-span scores and metric reasons. When a coding agent is driving the loop, that's what stops it from overfitting to a number it can't see the reasoning behind. What's missing, and why it's a beta whats-missing-and-why-its-a-beta I'd rather tell you than let you find out. Three things exist in Python and don't exist in TypeScript yet: Synthesizer — generating goldens from your documents or knowledge base. Bring your own dataset for now: load goldens from a CSV, a JSON file, or Confident AI. Benchmarks — MMLU, HellaSwag, and the rest of the foundational-model benchmark suite. Prompt optimization — automatic prompt search against a metric. There's also no score-parity guarantee. The prompts are shared and parity-checked, but we've spot-verified numeric scores for sanity rather than asserted them equal to Python's. Don't mix languages inside one longitudinal comparison yet. That's what the beta label is for. Everything above it — metrics, models, tracing, the CI/CD gate, the CLI — is what we're asking you to actually use and break. Getting started getting-started Or don't write the test file yourself. Install the DeepEval skill and let your coding agent drive the eval driven development /blog/eval-driven-development loop: Python still leads on behavior and TypeScript follows close behind — that hasn't changed, and one repo is what keeps "close behind" true. What changed is that "close behind" now means 47 metrics, 13 model providers, 6 tracing integrations, and a command that turns red on your pull requests, rather than a client that couldn't score anything. DeepEval is free and 100% open-source on ⭐ GitHub https://github.com/confident-ai/deepeval . If TypeScript is your stack, this is the release I've been wanting to write for a year — go break it and open an issue. FAQs faqs How do I install DeepEval for TypeScript? npm install -D deepeval . It's the same package name as the Python one, published to npm, and it lives in the same open-source repo https://github.com/confident-ai/deepeval as Python. How do I run evals in CI/CD? npx deepeval test run as a step in your pipeline. It exits non-zero when a metric falls below its threshold, so a regression blocks the merge — any provider that runs a shell step works. Your evals live in your test suite DeepEval registers a toPass matcher for Vitest , so there's no second harness to maintain. Are all the metrics available in TypeScript? AgentLoopDetectionMetric and ToolPermissionMetric are outstanding. Both SDKs compile the same language-neutral judge prompts, so behavior stays aligned. Which LLM providers can I use as a judge? Which agent frameworks can it trace? What's missing from the TypeScript SDK? Do I need a Confident AI account? OPENAI API KEY and everything runs locally, with traces written to a local .json file you can browse with npx deepeval inspect . Adding CONFIDENT API KEY is optional and only sends results to the cloud.