This is post 2 of a 3-part series on decision models. Part 1 introduced what they are, and part 3 will cover how they are built and trained.
One of the things that I hated the most about evaluating agents with LLM-as-a-judge is that it can be expensive and slow, forcing me in many cases to run those evals out-of-band instead of baking them into CI. So you can imagine that when I first read about decision models and all this idea of “Jev-as-a-judge”, I thought that this could be the perfect opportunity to finally introduce some evals in the critical path of my SDLC, as they may finally have become fast and cheap enough to be able to run them in CI. To test this, I built an eval pipeline to compare two code-review agents. Originally, it used an LLM-as-a-judge as the scorer, but I decided to replace it with a decision model as the scorer to compare their results and performance.
What I wanted to test with this experiment is the hypothesis that LLM judges are too slow, too expensive and too noisy to run on every commit, and that decision models make judging cheap and repeatable enough to live in CI. This only works if you can build scorers that rely on a rubric and return a choice, a score, or a yes or no, but it turns out that this is how scorers are (or at least should be) designed in the first place to make things objective and tractable.
How we judged our agent at Baselight
In May I wrote about the eval problem and how we tested the Baselight agent, an AI that allowed you to perform data-rich analysis in natural language. The suite ran real prompts through the real chat pipeline and scored each answer with a mix of static checks and LLM judges, including three of our own: one for whether the agent found the right datasets, one for whether its SQL answered the question, and one for whether the numbers in the answer came from the data.
Of course, the judges were the expensive layer of this suite, so we couldn’t run them on every commit. We built a mapping from the files a change touched to the evals that covered them, and ran the judged evals only when a PR touched the agent itself, plus a nightly run against main that ran the full suite. Initially, the first few agents that we developed were one big system prompt rather than a set of narrow agents, so a lot of changes touched the agent itself.
This produces the usual regression where there was a model update, and the new model version shifted how the agent behaved in ways no diff in our repo explained, and the evals that were supposed to tell us whether that was a regression weren’t stable either. The judges were models too, and the same answer could come back with a different verdict on a different run. This was a complete mess, because a red test could mean either that the agent got worse or that the judge’s behaviour changed. Debugging this was really time-consuming, and in many cases, the only way to find out what was happening was to rerun the suites again (and pay tokens again) to figure out if this was a regression in the agent or the scorer.
Moving Baselight AI to be a pipeline of narrow agents instead of a big system prompt made things significantly better on this front. Evaluating each of the narrow agents in an isolated way along with the end-to-end agent had clear benefits: model updates no longer cause that many regressions in the agent, because the narrow agents in the pipeline were narrow enough that new versions didn’t impact their reasoning path that much; regressions in specific stages of the pipeline due to some new feature, or even model updates, were easier to catch and fix; and all this led to having more narrow scorers for each agent, less vulnerable to their own model update regressions; finally, we could now track specific changes on specific parts of the agent pipeline, enabling us to activate the part of the eval that involved the new change introduced. In short, modularising the pipeline made things more tractable.
We did all of this through experiments and trial-and-error, but it turns out we weren’t being unlucky. A study called “Same Input, Different Scores” took five popular judge models, had them score the same real answers repeatedly, and found the scores moving between runs even with the temperature set to zero. The models also disagreed with each other on how strict to be, so swapping the judge can also have an impact on the evaluation.
Objective vs. Subjective Scorers
Unless explicitly told otherwise, an LLM judge writes its verdict. You give it the agent’s run and a rubric, it produces a few paragraphs of reasoning and a score, and your code pulls the score out of the text. Now that we can use structured outputs for LLMs, this has improved a bit, but the problem of using free-text LLM judges is that every token of that reasoning and the output costs time and money, and the more output tokens we use for the score the more “subjective” and non-deterministic it can feel
A decision model can’t write, so the rubric has to become questions it can answer by picking. And this is good! LangChain did exactly that with a small weather agent. They froze five of its runs and turned the rubric into three yes-or-no questions: did the answer stay grounded in what the search returned, did the agent search when it should have (and ask for clarification when the location was ambiguous), and was the answer useful. Then they asked each judge about the same five runs a hundred times.
If you recall our Baselight evaluations, we were using this same approach of asking specific questions about each of the stages in the agent pipeline, so replacing our LLM-as-a-judge scorers for decision models would have been straightforward. Jev matched the human reviewer on every pass or fail, and its quality scores barely moved from one repetition to the next. The LLM judges were 92 to 913 times more variable. The whole experiment cost $0.34 with Jev and $28.17 with Claude Sonnet 4.6. A radiology paper found something similar at a larger scale: Jev agreed reasonably well with expert error counts on medical reports for under three cents per hundred reports, though a metric built specifically for radiology still did better on the errors that matter clinically.
My recent experiments
Sadly, Baselight is no longer alive for me to test this migration for LLM scorers to decision-model scorers to test the difference. However, I still implement agents for fun and profit, so I decided to take the most recent evaluations I’ve made to one of the PR reviewers that we’ve written at work, and replace the LLM-as-a-judge with OpenAI-decisions-API-as-a-judge.
The benchmark was about twenty recently merged pull requests, picked at random, with two automated reviewers commenting on each. A judge reads each review and measures how many PRs it completed, if it identified valid bugs, if it missed any extra bugs, how accurate the suggestions were, and the quality of its output comments and overall analysis. I ran every judgement twice, once with a generative LLM judge, and once with OpenAI’s Decisions API.
The decision model gave the same verdict as the generative judge nine times out of ten, 8.6 times faster and for half the price. I’m reading the table in that order on purpose, because the speed is what changes how you’d use it, and where you can deploy your evals, as discussed above. A judge that answers in under 200 milliseconds can sit inside a test run. One that takes a second and a half per verdict, across a whole suite, is the thing you schedule for the night.
Agreement (meaning how many times the scorers returned the same verdict). If the generative judge is wrong about a review, the decision model copying it counts as a success. What I care about is the one verdict in eleven where they differ. For the dataset we used for testing, they generally were right when they agreed.
The cost gap, however, was far smaller than LangChain’s. (But I guess that can be an artefact of the model that I chose for the LLM judge, which was gpt-luna). Their Jev judge came out more than eighty times cheaper than Claude, and mine came out about twice as cheap. Another guess is that it can easily also be the diff. Judging a code review means reading a large change as well as the review, and some additional context about the surrounding code, and both judges pay for that reading, so the only thing the decision model saves is the few words the other one writes, and the underlying reasoning tokens. I need to make more experiments, but if that’s right, decision models save the most when the input is short and the verdict is long, which is the opposite of my use case.
What changes when judging is nearly free
The first thing that changes is that the judged evals can run on every commit. The file-to-eval mapping we built at Baselight was a workaround for a judge that cost too much to run everywhere, and a workaround that leaked whenever a change touched the big shared prompt. If a judgement costs a fraction of a cent and comes back in a fifth of a second, the cheapest option is to run the whole suite every time and stop maintaining the map.
The second is that a judged eval starts to behave like an ordinary test. If the judge gives the same score for the same run, a change in the score means the agent changed, and you can diff two commits the way you’d diff any other test result.
The third is one that needs more testing but that could easily give a bonus point to decision models over LLM judges. Decision models return the confidence of an output, and in theory we could use this confidence as a signal of how certain the judge is about this test passing. A judge that says yes with 0.98 and a judge that says yes with 0.51 are telling you different things. That 0.51 confidence may easily mean that the eval that we are running is not objective enough, or that there is something sketchy happening under the hood on your agent’s implementation. When an LLM judge is flaky, you have to run it several times to reduce the “noise”, with decision models you now have that “confidence” as the signal of how flaky or reliable your scorer and eval and agent implementations are.
Where I’d still pay for an LLM judge
Repeatable isn’t the same as right. A judge that gives the same wrong answer a hundred times has zero variance and is still wrong, and the audit I mentioned last week showed that with badly described options, Jev scored worse than guessing. Moving judging to decision models moves the hard work into writing the questions, which is the part of evals I said in May was most of the work anyway.
Some judgements also need a reason. When an eval fails, the first thing I want to know is why, and a decision model gives me a number. For open-ended quality (”was this analysis any good?”) and for factfulness that needs a chain of reasoning to check, I’d still use an LLM judge, and I’d read what it wrote.
And then there’s calibration on unfamiliar data, which worries me most. The audits found Jev well calibrated on public benchmarks and noticeably worse on support tickets it hadn’t seen, and long agent traces and code reviews are about as unfamiliar as inputs get. My 90.7% tells me the two judges mostly agree, and that when they agreed they were generally right. It tells me nothing about the one verdict in eleven where they split. I haven’t yet checked who was right in those, so I can’t say whether they’re blind spots of the decision model or mistakes of the LLM judge. That’s the first thing I’d look at before trusting a 0.98, and I don’t know yet which way it will go.
So the split I’d use is decision-model judges on every commit for everything you can phrase as a question, and LLM judges in the nightly and pre-release runs for the rest. I’d also start every new eval with an LLM judge. Being chatty is a feature in the first iterations, because its written reasoning shows you where your rubric is vague and where your agent goes wrong, and it gives you a baseline to compare against. Once the eval has stabilised and the questions are tight enough to answer by picking, that’s the moment to move it to a decision model.
Next week is the final round of the series: how decision models are built and trained, and how a model answers a question without writing a single word.
What’s the most expensive eval in your suite, and could you rewrite it as a yes or no?
Until next week!