Jev vs. LLM-as-a-Judge: Accuracy and cost benchmarks

Jev really is roughly 300x cheaper than an LLM judge, as long as you get one thing right

TL;DR: After tuning each judge’s decision threshold, Jev matched Claude Opus 5 at 87% accuracy on our held-out hallucination detection test, at roughly 1/300 of the cost and 23 times the speed. The catch is that the default 0.5 cutoff made Jev look substantially worse. Before you decide a judge isn’t good enough, tune its threshold against human-labeled examples and check the result on separate data.

When TypeSafe launched Jev, a model that answers typed questions with probabilities instead of generating any text, we wrote up the pitch: roughly frontier-model accuracy as a judge, at a tiny fraction of the cost and significantly better speed. That’s a testable claim, so I tested it, with 23,325 judgments across two human-labeled datasets, putting Jev up against Claude Opus 5 and GPT-5.6 Terra. The tasks were hallucination detection on RAGTruth and summary quality on SummEval.

The claim held up, but my first run made it look like it didn’t. At the 0.5 cutoff a lot of people would pick by default, Jev looked 7 points worse than Opus 5. The thing I’d missed was the threshold where a probability turns into a verdict. Once I tuned that against human labels and checked it on separate data, Jev matched Opus 5 on hallucination detection. The threshold turned out to be a bigger lever than which model I used, so most of this post is about how to tune it, because you should honestly be doing that for your LLM judges too.

Build better agents with Arize

Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.

Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.

How I benchmarked Jev against Claude Opus 5 and GPT-5.6 Terra #

Jev takes three kinds of questions. A Choice picks one option from a list, a Score rates something against ordered levels, and a Noul (which is a word they made up, a pretty bold move) takes a claim and returns the probability that it’s true. You don’t get prose back and you don’t get an explanation: you post your state and your questions to its own API and get typed answers with probability distributions attached.

I pointed it, as well as Opus and Terra, at two datasets where humans had already done the labeling. RAGTruth has trained annotators mark hallucinated spans in 2,675 RAG responses, which I collapsed to the question a judge actually gets asked: is anything in this response unsupported by the context? That shape of answer is what TypeSafe calls a Noul. SummEval has three expert annotators rate 1,700 news summaries from 1 to 5 on coherence, consistency and relevance, which is 5,100 graded ratings and what TypeSafe would call a Score. I used human annotations as the reference labels so the benchmark measures agreement with people rather than agreement with another model.

I tried pretty hard not to rig the baseline. The SummEval judges got the published G-Eval prompts verbatim rather than anything I wrote, Jev got the same criteria word for word, both LLM judges were asked for a confidence score through structured output so calibration could be compared fairly, and reasoning effort was pinned to high and logged on every call. The whole thing cost about $109 in API spend, with zero failed calls.

Jev benchmark results: Accuracy, cost, and latency #

On hallucination detection, Jev and Opus 5 both land at 87% accuracy. Jev gets there for $0.05 per thousand judgments, against $14.30 for Opus 5. It also answers in about 141 milliseconds where Opus 5 takes over 3 seconds, which is the difference between a judge you can run on every single response in the request path and one you run on a sample afterwards.

The graded task came out the same way. Jev’s rank correlation with the expert ratings is 0.74, slightly ahead of Opus 5 at 0.70, and remember: the three human experts only agree exactly with each other 53% of the time. If you treat one expert as the judge and the other two as ground truth, the human scores a correlation of 0.72. So all three of Jev, Opus and Terra in this test are performing at roughly human level on summary quality, but Jev is the cheapest by 216 times.

That’s the tuned result, though. The first time I ran it, Jev lost.

Why a 0.5 threshold hurt Jev’s hallucination detection accuracy #

At the default cutoff of 0.5, Opus 5 got 83% of RAGTruth right and Jev got 76%, with confidence intervals that don’t overlap.

The thing I’d missed is that Jev’s Noul is not the same as a decision. A Noul returns a probability, and one RAG response comes back as 0.76. The decision happened on a line in my own code that said label = “hallucinated” if p >= 0.5, and I picked the 0.5 because it seemed a reasonable default. And it is! 83% is pretty good! But you can do better if you tune it.

Split Jev’s output by what the humans said and the problem is obvious. Hallucinated responses pile up near 1.0, with a median of 0.96. Grounded responses spread across the whole range, with a long flat tail running right up to the top. 34% of perfectly grounded responses score above 0.5, so a cutoff there turns all of them into false alarms.

The best way to see that Jev’s judgment was fine and only my cutoff was wrong is a metric called ROC AUC. Take one response you know is hallucinated and one you know is grounded, and check whether the judge gave the bad one a higher number than the good one. Do that for every possible pair, and the fraction it gets right is the AUC (or “area under the curve”). Half means coin-flip ordering and 1.0 means it never got a pair backwards. Jev scores 0.94 and Opus 5 scores 0.95, so they order the cases almost equally well. Jev was losing on accuracy purely because of where I’d drawn the line, and that’s an easy, quick fix to my code, not to Jev.

How to tune hallucination detection thresholds #

To tune a hallucination detection threshold, compare the judge’s predicted probabilities with human labels on a validation set. Choose a cutoff based on the false positives and missed hallucinations your application can tolerate, then measure performance on a separate test set. Here are five cutoffs I compared for Jev, from 0.50 to 0.95:

False positive rate is the percentage of grounded responses incorrectly flagged as hallucinated. False negative rate is the percentage of hallucinated responses the judge missed.

The first thing to notice is that raw accuracy and balanced accuracy peak in different places. Balanced accuracy is the accuracy on each class worked out separately and then averaged, and the reason you want it is that RAGTruth is 65% grounded. Raw accuracy pays a judge for being good at the majority class, so pushing the cutoff up buys cheap wins on grounded responses while quietly letting hallucinations through. Going from 0.80 to 0.90 gains you 1 point of accuracy but raises the miss rate from 15% to 23%. For a hallucination detector that’s usually the wrong trade, because a made-up answer reaching a user costs more than a good answer getting queued for review.

The second thing is that you have to pick the threshold on data you’re not going to score it on, or you’re just fitting to your test set. I split RAGTruth in half, chose each judge’s cutoff on the first half by balanced accuracy, and scored it on the second half. Jev landed at 0.80. Opus 5 landed at 0.65, which is a reminder that LLM judges have exactly the same dial and nobody tunes it either.

Jev and Opus 5 come out identical to within a thousandth on every measure, including the error profile and the confidence intervals. Note: the intervals overlap, so my claim is that they’re indistinguishable on this data rather than that Jev wins, and if you optimize for raw accuracy instead of balanced accuracy, Opus 5 edges it by a point, because its ranking really is a hair better. Terra tunes to 0.95 and still flags 27% of grounded responses, which makes it a chronic over-flagger at any threshold.

So the recipe is this. Get a few hundred examples of your own traffic labeled by people. Run your judge over them, but keep the probability rather than the label. Sweep the cutoff and look at the false alarm and miss columns, not just accuracy, and pick the one that trades those off the way your application needs. Fit it on half, check it on the other half, and refit when the task or the model version changes. A judge that returns a probability gives you a dial, and an LLM judge that just says “yes, hallucinated” gives you prompt rewrites and hope.

Confidence calibration and escalation to an LLM judge #

My original post about Jev leaned on the idea that LLM judges are overconfident and Jev’s probabilities are calibrated, meaning that when it says 70% it’s right about 70% of the time. Terra behaves exactly the way that post predicted: it states an average confidence of 96%, is right 70% of the time, and puts 80% of its answers into a single confidence bucket, so its confidence carries almost no information.

Opus 5 at high effort doesn’t behave that way at all. It says 81% and is right 83%, which makes it the best-calibrated judge in the test, better than Jev, which is underconfident. So the overconfidence critique is true of some models and not others, and our own framing over-generalized from it. Jev’s confidence does spread across a much wider range than either LLM’s, which is what makes it usable for routing, but “LLM judges are overconfident” isn’t the law I thought it was.

I’d also expected confidence-gated escalation to be the payoff: run Jev everywhere, send its least-confident cases to Opus 5, get frontier accuracy for a fraction of frontier cost. Once Jev’s threshold is tuned, escalating any share of its judgments moves accuracy by less than a point in either direction, because Jev already matches Opus 5 on its own. So at least in my tests, escalation buys you nothing, just use Jev.

Using Jev for real-time evaluation in production #

My original post asked if we’re going to rearchitect our apps to use Jev, and I think we will. Here’s the shape of what I think we’ll end up building.

Put the typed judge in the request path, where the cost and latency mean you can score every response instead of sampling. Fit the threshold offline against a few hundred human-labeled examples, on whichever error you actually care about. Keep a generative judge around for the cases where somebody needs the reason in words, because Jev will never give you one, and any eval workflow whose job is to tell a team why something failed still needs prose.

I ran all of this in Arize AX, which is where the golden datasets, the per-judgment traces, and the cost and latency numbers live. One wrinkle: AX has no pricing for a model that isn’t an LLM, so Jev’s cost has to be set on the span by hand or it shows up as free.

The usual caveat applies, and it’s a single one. Both datasets are public and predate every model tested, so any of them may have seen the data in training, and Jev’s Score levels are ones I wrote because the primitive needs level descriptions and the G-Eval prompts don’t provide them. The fix for both is the same as the recipe above: a few hundred labeled examples of your own traffic.

Before you decide any judge is too dumb or too expensive, sweep its threshold against your own labels and look at which error you’re buying. In this benchmark, threshold tuning changed the comparison enough to change which judge I would choose.

Want to see how your judge holds up? Bring a few hundred human-labeled examples to Arize AX to compare accuracy, cost, and latency so you can test what happens when you move the threshold.