{"slug": "deepeval-open-sourced-for-typescript", "title": "DeepEval Open-Sourced for TypeScript", "summary": "DeepEval has open-sourced its TypeScript SDK in beta, enabling all 47 of its 49 metrics to run as a gate in CI/CD pipelines via a single Vitest matcher. The SDK, which compiles the same language-neutral prompt templates as the Python version, includes 13 model integrations and runs without a hosted service or API key. The two missing metrics, AgentLoopDetectionMetric and ToolPermissionMetric, are expected shortly.", "body_md": "# DeepEval for TypeScript, now fully open-source\n\nDeepEval's TypeScript SDK is out in beta. Every metric, model, and tracing integration you know from Python, running as a gate in your CI/CD pipeline.\n\nTwo months ago I wrote about [why we put TypeScript inside DeepEval's Python monorepo](/blog/typescript-in-deepeval-monorepo) instead of giving it its own repo. At the time it was still a client wrapper around Confident AI — it couldn't run a single metric.\n\nToday it can run all of them. DeepEval for TypeScript is out in beta:\n\nThe headline isn't the metrics though. It's where they run: **in CI/CD, as a gate on your pull requests.**\n\n[Evals belong in your Typescript CI pipeline](#evals-belong-in-your-typescript-ci-pipeline)\n\nThe reason I keep pushing this is that an eval you run by hand is a nice number, and an eval that runs on every PR is a decision. That's the whole point of an [eval harness](/blog/what-is-an-eval-harness) — a regression in your agent should block a merge exactly like a regression in your business logic does, without anyone remembering to check.\n\nFor that to happen, evals have to live where your pipeline already looks: your test suite. In Python that's Pytest. In TypeScript it's Vitest, and the surface is a single matcher — `toPass()`\n\n— on top of a test file you'd recognize without ever having used DeepEval:\n\nThen run it:\n\nPlain `vitest`\n\nworks too, but you'd be leaving most of the value on the table. `npx deepeval test run`\n\ncaptures a trace per test, caches metric results so a re-run doesn't re-bill you, and gives you the same flags Python users have — `--official`\n\n, `-i/--identifier`\n\n, `--max-concurrent`\n\n, `-c/--use-cache`\n\n, `--ignore-errors`\n\n.\n\nThat one command is the whole integration story. It exits non-zero when a metric falls below its threshold, so any CI provider that runs a shell step already knows what to do with it:\n\nNote what isn't in there: no `CONFIDENT_API_KEY`\n\n, no hosted service, no vendor in the critical path of your merge queue. Give it a judge key and the whole thing runs on the runner. Add the key later if you want shared reports and regression tracking across commits, and add `--official`\n\non `main`\n\nto mark the baseline that future runs get compared against. The full walkthrough is in [unit testing in CI/CD](/docs/evaluation-unit-testing-in-ci-cd).\n\n[Every metric, open-source](#every-metric-open-source)\n\nThis is the part I care most about, because a second language that only ships the easy half of the metric library is worse than no second language at all.\n\n**47 of DeepEval's 49 metrics are ported and open-source in TypeScript.** Not a curated subset — G-Eval with log-prob weighted scoring, DAG decision graphs, the multi-turn suite, the MCP metrics, the multimodal ones, arena comparisons. The only two that haven't landed are `AgentLoopDetectionMetric`\n\nand `ToolPermissionMetric`\n\n, and they'll be in shortly.\n\nThey agree with Python because they aren't rewritten from memory. Both SDKs compile the **same language-neutral prompt templates** — one JSON bundle of judge prompts, shared by the two implementations. Only the orchestration and the output schema (Pydantic in Python, Zod in TypeScript) are written per-language. So when we change a judge prompt, both languages change together or neither does.\n\nThe API is idiomatic TypeScript rather than transliterated Python — an options object instead of keyword arguments, `camelCase`\n\n, and `measure()`\n\nis always `async`\n\n(there's no sync/`a_measure`\n\nsplit, everything underneath is already async). Full list on the [metrics page](/docs/metrics-introduction).\n\n[The models that judge them](#the-models-that-judge-them)\n\nAn LLM-as-a-judge metric is only as portable as the providers it can run on, so the model layer came over too. TypeScript ships **13 model integrations**.\n\nTwo notes. LiteLLM is Python-only and always will be — `AISDKModel`\n\nis the TypeScript answer to \"route my judge through anything.\" And the default judge model is generated from Python's `DEFAULT_MODELS`\n\n, so an unconfigured metric lands on the same model in both languages.\n\nSet one once from the CLI and every metric picks it up:\n\n[Tracing, and evals on the trace](#tracing-and-evals-on-the-trace)\n\nScoring a final string is fine for a RAG pipeline. It isn't enough for an agent, where the interesting failures are in the trajectory — the tool it shouldn't have called, the plan it abandoned, the four steps it took to do one thing.\n\nSo TypeScript gets tracing, with **6 framework integrations** that turn your agent's execution into a span tree with no rewriting:\n\n| Framework | Import | Setup |\n|---|---|---|\n|\n\n`deepeval/openai`\n\n`instrumentOpenAI(client)`\n\n[LangChain](/integrations/frameworks/langchain)&[LangGraph](/integrations/frameworks/langgraph)`deepeval/integrations/langchain`\n\n`new DeepEvalCallbackHandler({})`\n\n[OpenAI Agents SDK](/integrations/frameworks/openai-agents)`deepeval/integrations/openai-agents`\n\n`new DeepEvalTracingProcessor()`\n\n[Mastra](/integrations/frameworks/mastra)`deepeval/integrations/mastra`\n\n`new DeepEvalExporter()`\n\n[Vercel AI SDK](/integrations/frameworks/ai-sdk)`deepeval/integrations/ai-sdk`\n\n`configureAiSdkTracing({})`\n\n`deepeval/integrations/openinference`\n\n`instrumentOpenInference({})`\n\nMastra and the Vercel AI SDK are the fun ones — they have no Python counterpart at all, so for once TypeScript is ahead.\n\nThe reason this matters for CI/CD is that instrumenting your agent and gating on it are the same piece of work, not two. Once the app is traced, you hand `toPass()`\n\na golden and a task, and it runs your agent, captures the trace, and scores the whole trajectory:\n\nYou can also attach metrics to individual spans instead of the whole trace — [component-level evals](/docs/evaluation-component-level-llm-evals), where a retriever gets contextual precision and a tool call gets argument correctness, in the same test run.\n\n[The CLI came over too](#the-cli-came-over-too)\n\nSame binary name, same commands, one `npx`\n\nin front:\n\n| Command | What it does |\n|---|---|\n`npx deepeval test run` | Run your eval suite as a gate, locally or in CI |\n`npx deepeval inspect` | Browse locally captured traces in a terminal UI |\n`npx deepeval view` | Open the latest test run on Confident AI |\n`npx deepeval login` / `logout` | Authenticate with Confident AI |\n`npx deepeval set-openai` (and 11 more) | Configure the judge model per provider |\n`npx deepeval gate` | Run a governance policy check |\n`npx deepeval diagnose` | Print the effective config when something's off |\n\n`npx deepeval inspect`\n\nis the one I'd try first. Traces are written to a local `.json`\n\nfile on your machine by default — nothing leaves your laptop — and `inspect`\n\nrenders them as a trace tree with per-span scores and metric reasons. When a coding agent is driving the loop, that's what stops it from overfitting to a number it can't see the reasoning behind.\n\n[What's missing, and why it's a beta](#whats-missing-and-why-its-a-beta)\n\nI'd rather tell you than let you find out. Three things exist in Python and don't exist in TypeScript yet:\n\n**Synthesizer**— generating goldens from your documents or knowledge base. Bring your own dataset for now: load goldens from a CSV, a JSON file, or Confident AI.**Benchmarks**— MMLU, HellaSwag, and the rest of the foundational-model benchmark suite.** Prompt optimization**— automatic prompt search against a metric.\n\nThere's also no score-parity guarantee. The prompts are shared and parity-checked, but we've spot-verified numeric scores for sanity rather than asserted them equal to Python's. Don't mix languages inside one longitudinal comparison yet.\n\nThat's what the beta label is for. Everything above it — metrics, models, tracing, the CI/CD gate, the CLI — is what we're asking you to actually use and break.\n\n[Getting started](#getting-started)\n\nOr don't write the test file yourself. Install the DeepEval skill and let your coding agent drive the [eval driven development](/blog/eval-driven-development) loop:\n\nPython still leads on behavior and TypeScript follows close behind — that hasn't changed, and one repo is what keeps \"close behind\" true. What changed is that \"close behind\" now means 47 metrics, 13 model providers, 6 tracing integrations, and a command that turns red on your pull requests, rather than a client that couldn't score anything.\n\nDeepEval is free and 100% [open-source on ⭐ GitHub](https://github.com/confident-ai/deepeval). If TypeScript is your stack, this is the release I've been wanting to write for a year — go break it and open an issue.\n\n[FAQs](#faqs)\n\n## How do I install DeepEval for TypeScript?\n\n`npm install -D deepeval`\n\n. It's the same package name as the Python one, published to npm, and it lives in the same [open-source repo](https://github.com/confident-ai/deepeval)as Python.\n\n## How do I run evals in CI/CD?\n\n`npx deepeval test run`\n\nas a step in your pipeline. It exits non-zero when a metric falls below its threshold, so a regression blocks the merge — any provider that runs a shell step works. Your evals live in your test suite (DeepEval registers a `toPass()`\n\nmatcher for Vitest), so there's no second harness to maintain.## Are all the metrics available in TypeScript?\n\n`AgentLoopDetectionMetric`\n\nand `ToolPermissionMetric`\n\nare outstanding. Both SDKs compile the same language-neutral judge prompts, so behavior stays aligned.## Which LLM providers can I use as a judge?\n\n## Which agent frameworks can it trace?\n\n## What's missing from the TypeScript SDK?\n\n## Do I need a Confident AI account?\n\n`OPENAI_API_KEY`\n\nand everything runs locally, with traces written to a local `.json`\n\nfile you can browse with `npx deepeval inspect`\n\n. Adding `CONFIDENT_API_KEY`\n\nis optional and only sends results to the cloud.", "url": "https://wpnews.pro/news/deepeval-open-sourced-for-typescript", "canonical_source": "https://deepeval.com/blog/introducing-deepeval-typescript", "published_at": "2026-08-12 18:58:16+00:00", "updated_at": "2026-08-12 19:12:33.532056+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "mlops", "artificial-intelligence"], "entities": ["DeepEval", "Confident AI", "TypeScript", "Python", "Vitest", "LiteLLM", "AISDKModel"], "alternates": {"html": "https://wpnews.pro/news/deepeval-open-sourced-for-typescript", "markdown": "https://wpnews.pro/news/deepeval-open-sourced-for-typescript.md", "text": "https://wpnews.pro/news/deepeval-open-sourced-for-typescript.txt", "jsonld": "https://wpnews.pro/news/deepeval-open-sourced-for-typescript.jsonld"}}