{"slug": "show-hn-jeveval-evals-using-jev-as-a-judge", "title": "Show HN: JevEval, evals using Jev-as-a-judge", "summary": "DeepEval released JevEval, a custom LLM evaluation metric that uses Jev as a judge to convert explicit questions and calibrated probabilities into deterministic scores. JevEval separates evaluation into three steps: the user defines the logic, Jev makes bounded decisions via three question types (Noul, Score, and Choice), and DeepEval calculates the score with fixed, inspectable math rather than a generated verdict. The release follows the company's prior introduction of Jev in DeepEval, which argued that language tasks should stay with language models while bounded decisions go to a model built to decide.", "body_md": "# Introducing JevEval: Jev-as-a-Judge for LLM Evaluation\n\nJevEval is a custom LLM evaluation metric powered by Jev that turns explicit questions and calibrated probabilities into deterministic scores.\n\nYesterday, we introduced [Jev in DeepEval](https://deepeval.com/blog/introducing-jev-in-deepeval) and the idea behind it: language tasks should stay with language models, while bounded decisions should go to a model built to decide.\n\nToday, we're taking that idea one step further with **JevEval**, a custom evaluation metric where Jev is not hidden inside someone else's metric. You write the decision points yourself, Jev answers them with calibrated probabilities, and DeepEval turns those answers into a score using fixed math.\n\nThis is Jev-as-a-Judge: no generated verdicts, no JSON recovery, and no prompt asking an LLM to invent a number between 1 and 10.\n\n## [The problem with custom LLM-as-a-Judge](#the-problem-with-custom-llm-as-a-judge)\n\nMost [custom LLM-as-a-Judge metrics](https://deepeval.com/blog/llm-as-a-judge) begin with a sentence:\n\nDetermine whether the response is helpful, accurate, and complete.\n\nThat sentence is passed to an LLM, which has to interpret the criteria, decide what matters, produce a score, and usually explain itself—all in one generative step.\n\nThis is convenient, but it hides too much. What did \"helpful\" mean for this test case? Did accuracy matter more than completeness? What does a score of `0.71` mean, and why did the same response receive `0.66` on the next run?\n\nThe problem is not that LLMs cannot read the response. It is that we ask generation to do three separate jobs:\n\n1. Define the evaluation logic.\n2. Make the decisions.\n3. Calculate the score.\n\nJevEval separates them. **You define the logic, Jev makes the decisions, and DeepEval calculates the score.**\n\n## [What is JevEval?](#what-is-jeveval)\n\n`JevEval` is a custom metric for [`LLMTestCase`](https://deepeval.com/docs/evaluation-test-cases#llm-test-case). Instead of giving it one open-ended criterion, you give it:\n\n- the test case fields Jev is allowed to see,\n- a list of bounded questions about those fields,\n- and, optionally, a weight for each question.\n\nJev receives the selected fields as structured state and answers every question in one request. It returns probability distributions rather than generated text. DeepEval then maps each answer onto a value from `0` to `1` and takes a weighted mean:\n\nHere, is the question's weight, is the value calculated from Jev's probabilities, and is the set of questions that apply to the test case.\n\nThe important part is where the score comes from. Jev does not generate it. An LLM does not interpret it after the fact. The equation is fixed, inspectable, and the same on every run.\n\n## [Three ways to ask a decision](#three-ways-to-ask-a-decision)\n\nJevEval is built from three question types, each matching one of [Jev's decision primitives](https://deepeval.com/blog/introducing-jev-in-deepeval#understanding-jevs-decision-primitives).\n\n### [Noul: is this proposition true?](#noul-is-this-proposition-true)\n\nA `Noul` is a binary proposition such as:\n\nEvery fact in `actual_output` appears in `tools_called`.\n\nJev returns , and that probability becomes the question's value:\n\nNouls are useful for requirements that read like a checklist: the answer cited its sources, the response followed the requested format, or the assistant verified the order before issuing a refund.\n\n### [Score: where does this land on an ordered scale?](#score-where-does-this-land-on-an-ordered-scale)\n\nA `Score` uses descriptive levels arranged from worst to best:\n\nFabricated → Mostly fabricated → Mostly grounded → Fully grounded\n\nJev returns a probability for every level. DeepEval calculates the expected position and normalizes it to `0–1`:\n\nThis preserves uncertainty between adjacent levels instead of forcing a brittle integer rating.\n\n### [Choice: which behavior occurred?](#choice-which-behavior-occurred)\n\nA `Choice` selects from an unordered set of behaviors. You assign each option a credit from `0` to `1`:\n\n- `left_it_out` :`1.0`\n- `flagged_it_as_unknown` :`1.0`\n- `hedged_it` :`0.5`\n- `stated_it_as_fact` :`0.0`\n\nUnlike an ordered score, two very different behaviors can earn the same credit. You can also assign `None` to an option such as `nothing_missing`. If Jev places at least half of the probability on not-applicable options, that question drops out of the score rather than rewarding or penalizing the response for a situation that never occurred.\n\nThe credits never leave DeepEval. Jev sees only the option names, so the value you assign cannot steer its decision.\n\n## [Building a Tool Faithfulness metric](#building-a-tool-faithfulness-metric)\n\nSuppose an agent calls a weather tool that returns `18°C` and `sunny`, but its answer also claims there is a light breeze and `40%` humidity. A useful metric should distinguish between reporting the returned values correctly and inventing additional ones.\n\nThis metric does not collapse tool faithfulness into one vague judgement. It asks four inspectable questions:\n\n1. Did every claim come from a tool?\n2. Were the values that did come from the tool reported accurately?\n3. How grounded was the response overall?\n4. How did the response handle information the tool did not provide?\n\nThe first question has twice the weight because hallucinating an unsupported fact is the failure we care about most. That importance is explicit in code rather than implied somewhere inside a prompt.\n\n## [Configuring JevEval](#configuring-jeveval)\n\nBeyond the questions themselves, three settings shape how a JevEval metric behaves: what it records, whether an LLM explains the result, and whether partial credit is allowed.\n\n### [Debug score breakdown](#debug-score-breakdown)\n\nJevEval sends the state and all four questions to Jev in a single `decide()` call. Jev evaluates the questions in parallel, and DeepEval stores every intermediate result in `score_breakdown`.\n\nFor each question, you can inspect:\n\n- its type and weight,\n- the complete probability distribution,\n- its mapped `0–1` value,\n- whether it applied,\n- and, in strict mode, whether it passed.\n\nThis makes the final score traceable. If the Tool Faithfulness metric fails, you can see whether the response invented information, copied a returned value incorrectly, or landed between two grounding levels. You are debugging an evaluation specification, not reverse-engineering a generated number.\n\n### [Optional LLM explainability](#optional-llm-explainability)\n\nBy default, JevEval can ask an evaluation LLM to write a human-readable reason grounded in the test case and the outcomes. The LLM explains the decision; it does not make it.\n\nSet `include_reason=False`, as in the example above, and JevEval makes no LLM call at all. The metric becomes one Jev request plus arithmetic, while still returning `score`, `success`, `confidence`, and `score_breakdown`.\n\nThis boundary matters. Explanations are language tasks, so an LLM is useful there. Passing or failing a regression test is a decision task, so the explanation cannot change the score.\n\n### [Enforce strict mode](#enforce-strict-mode)\n\nA weighted mean is useful when strengths can offset weaknesses. Sometimes they should not.\n\nWith `strict_mode=True`, JevEval returns `1` only when every applicable question passes in its best possible way; otherwise it returns `0`. The threshold is set to `1`, and the breakdown shows exactly which requirement failed.\n\nUse strict mode for hard constraints such as:\n\n- every citation must be supported,\n- no prohibited behavior may occur,\n- every required field must be present,\n- or every mandatory workflow step must be completed.\n\nIt turns the same question set from a quality score into an executable acceptance test you can use in an [eval-driven development loop](https://deepeval.com/blog/eval-driven-development).\n\n## [JevEval works for conversations too](#jeveval-works-for-conversations-too)\n\nMany important behaviors only make sense across multiple turns: whether an assistant remembered a constraint, verified identity before taking action, resolved the user's request, or stayed within its assigned role.\n\n[`ConversationalJevEval`](https://deepeval.com/docs/metrics-conversational-jev-eval) applies the same Noul, Score, and Choice model to an entire [`ConversationalTestCase`](https://deepeval.com/docs/evaluation-multiturn-test-cases). The conversation becomes a structured `turns` list, and you can add fields such as `scenario`, `expected_outcome`, and `chatbot_role` to the state.\n\nThe scoring equation stays the same. The only difference is what the questions can see.\n\nIf you do not have conversation test cases yet, start with the [conversation simulation workflow](https://deepeval.com/blog/how-to-evaluate-conversational-agent) and use `ConversationalJevEval` to score the generated conversations.\n\n## [Getting started](#getting-started)\n\nInstall DeepEval and the TypeSafe SDK, then set your API key:\n\nThen define the state, write the questions, and run JevEval like any other DeepEval metric.\n\nThe full [JevEval documentation](https://deepeval.com/docs/metrics-jev-eval) covers question design, strict mode, score calculation, and a worked example down to every probability. If you're evaluating multi-turn agents, start with the [Conversational JevEval documentation](https://deepeval.com/docs/metrics-conversational-jev-eval).\n\n## [Evals as specifications](#evals-as-specifications)\n\nCustom evals should tell you more than whether an LLM liked a response. They should encode what your system is required to do, show how each requirement was judged, and calculate the result in a way you can inspect.\n\nThat is what JevEval is for.\n\nYou write the evaluation specification as bounded questions. Jev returns calibrated decisions. DeepEval turns them into a traceable score with fixed math. And when you need an explanation, an LLM explains the result without getting a vote.\n\nJev-as-a-Judge is not about replacing language models everywhere. It is about giving language generation fewer decisions it was never designed to make.\n\nDeepEval Got a New Look\n\nAn announcement on DeepEval reaching 15,000 GitHub stars and the launch of a new docs and website experience for developers.\n\nHow the H*ll Do I Evaluate Conversational Agents?\n\nOver 90% of evals run on DeepEval are still run using single-turn metrics, even when the use case is a conversational agent. Here is what's stopping people from running multi-turn evals properly and how to solve it.", "url": "https://wpnews.pro/news/show-hn-jeveval-evals-using-jev-as-a-judge", "canonical_source": "https://deepeval.com/blog/introducing-jev-as-a-judge", "published_at": "2026-09-22 20:47:32+00:00", "updated_at": "2026-09-22 20:53:45.837699+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-tools", "developer-tools", "mlops"], "entities": ["JevEval", "Jev", "DeepEval", "LLMTestCase", "Noul"], "alternates": {"html": "https://wpnews.pro/news/show-hn-jeveval-evals-using-jev-as-a-judge", "markdown": "https://wpnews.pro/news/show-hn-jeveval-evals-using-jev-as-a-judge.md", "text": "https://wpnews.pro/news/show-hn-jeveval-evals-using-jev-as-a-judge.txt", "jsonld": "https://wpnews.pro/news/show-hn-jeveval-evals-using-jev-as-a-judge.jsonld"}}