Measure the evaluator's variance, then rebuild its prompt from zero.
Your LLM judge just scored 94%.
Great. Ship it.
Run it again. 82%. Again: 91%. Again: 97%. Again: 89%. That 15-point spread is not a rounding error. It is noise wearing a lab coat. You cannot tell whether the evaluator improved, regressed, or merely woke up in a different mood.
If a restaurant got your order wrong that often, you would check the bag in the parking lot every time. Teams put the same evaluator in CI and treat every verdict as a measurement. Most teams tune the model that produces the answer. Far fewer tune the model that grades it.
That is backwards. If the judge is noisy, every experiment downstream inherits the noise. The prompt change that “improved things by four points” may have improved nothing. A real regression may be hiding inside the spread.
So I started treating the evaluator as its own optimization problem.
Not “write a better judge prompt.” A measured loop:
- Run the same cases repeatedly and establish baseline variance.
- Hand the results to an agent such as Codex or Claude Code.
- Let it research likely causes and propose one prompt or configuration change.
- Run the evaluator again.
- Keep the change only if variance drops without accuracy, cost, or latency getting worse.
- Repeat.
That is the whole idea. The details matter, but the loop is not complicated.
Start with variance #
Accuracy tells you how often the evaluator agrees with a known answer. Variance tells you whether it gives the same answer twice.
You need both. Most teams only measure the first.
For a basic pass/fail evaluator, collect at least 50 representative cases and run each one five times. Ten is better when the evaluator is cheap. Include known-good outputs, known-bad outputs, borderline cases, malformed outputs, and a few examples that caused real problems. Record every result separately. Do not average away the thing you are trying to find.
| Metric | What it tells you |
|---|---|
| Human agreement | whether the evaluator is generally correct |
| Repeat agreement | whether it gives the same verdict across runs |
| Score spread | how far numeric scores move on identical inputs |
| False accepts | how often bad output passes |
| False rejects | how often good output fails |
| Invalid output rate | how often the evaluator itself breaks |
| Cost and latency | whether a proposed fix is practical |
If the evaluator returns a numeric score, track the mean and standard deviation per case. Then track the number that matters more than either: how often the score crosses your pass/fail threshold.
Scores of 78, 79, 81, 82, 80 look stable on a chart. If the pass threshold is 80, that evaluator changed its decision three times while looking perfectly well-behaved.
Variance is only harmless when it cannot change the decision.
Call that the decision flip rate: the share of repeated runs whose verdict differs from that case’s majority verdict. It is the number the rest of this post optimizes.
For pairwise evaluators, run both A/B and B/A. Research has repeatedly found position bias in LLM judges. Hide the model and provider names too. There is also evidence of self-preference bias, which is the polite academic way of saying models like answers that sound like them.
Build an eval for the evaluator #
The tuning loop needs known answers. Without them, it can reduce variance by becoming consistently wrong.
Use human-reviewed cases and split them before the agent touches anything:
| Split | Purpose |
|---|---|
| Development | lets the agent inspect failures and propose changes |
| Validation | decides whether a proposed change is actually better |
| Holdout | catches overfitting before you replace the production judge |
The holdout set must stay out of the agent’s reach. If the agent can read every answer, it will eventually optimize for those exact answers. This is not an LLM-specific problem. We have been overfitting test sets for decades.
Use expert reviewers when the task requires expert judgment. Keep the examples where reviewers disagree. If qualified humans cannot agree on the verdict, the evaluator may not be broken. The rubric may be unclear, and no prompt tweak fixes that. OpenAI’s grader guidance recommends comparing model graders against trusted ground-truth grades and adding edge cases as they are discovered. Anthropic recommends the same basic calibration: compare LLM judges with human experts and read the transcripts rather than trusting the aggregate score.
A small evaluator test case #
Tags earn their keep here. A change can improve the overall score while making one important failure class worse. If unsupported claims are the expensive failure, track them separately.
Give the loop to an agent #
This is exactly the kind of work Codex and Claude Code are good at.
The loop is: read code, run a command, diff JSONL results, research known evaluator failure modes, edit a prompt, run it again. It is tedious enough that humans stop after two attempts. Agents do not care.
Give the agent:
- the evaluator implementation
- the development and validation cases
- the command that runs the eval
- the current results, including raw outputs
- the configuration variables it is allowed to change
- the maximum experiment budget
- explicit instructions not to edit labels or inspect the holdout set
Then ask for one hypothesis per experiment.
Bad:
Better:
Both tools run non-interactively. codex exec supports JSONL output and JSON Schema-constrained final output. Claude Code supports claude -p with JSON output. Either is enough. You do not need a prompt-optimization platform.
What you do need is an experiment record from every iteration:
Save the failed experiments too. Otherwise the next tuning run will spend its budget rediscovering them.
Now delete the prompt #
Optimizing the current prompt is useful, but it carries a bias: every instruction already in the prompt looks important, because somebody added it for a reason.
Maybe it was a good reason. Maybe one output failed once and the fix became permanent. After enough of these, the evaluator prompt is long, repetitive, expensive, and strangely specific about things that stopped mattering two model versions ago.
So run a second track from an empty prompt.
Not literally empty. Give it only the irreducible contract:
Run that as-is.
The failures tell you what this specific model needs to be told. Add one instruction or example for the largest failure class, then run it again. Keep the addition only if the numbers improve.
This beats copying a “best LLM judge prompt” from another project. A small model may need explicit steps and examples. A stronger reasoning model may do better with a short rubric. One model stabilizes when it explains the evidence before choosing a verdict. Another gets more consistent when it returns only structured fields.
The optimal prompt belongs to the model and configuration being tested. That configuration includes:
- model and provider
- reasoning effort
- temperature and sampling controls, where supported
- pointwise versus pairwise grading
- output schema
- examples and their order
- maximum output tokens
- whether a rationale is required
Treat that whole set as the evaluator. Changing the prompt while ignoring the model settings is not a controlled test.
Fix the failure you measured #
The agent should not add generic “be accurate” instructions. It should respond to a specific, measured failure.
| Measured failure | Experiment to try |
|---|---|
| misses one required condition | split the rubric into atomic checks |
| accepts unsupported claims | require cited evidence before a pass |
| flips near a numeric threshold | use pass ,fail , anduncertain |
| favors the first answer | reverse order and require agreement |
| overweights writing style | score correctness separately from presentation |
| returns malformed JSON | use a strict schema and shorter output |
| costs too much | remove rationale, examples, or reasoning effort one at a time |
Some checks should not use an LLM at all.
JSON validity, citation existence, allowed tool names, answer-key positions, numeric bounds, and exact identifiers belong in code. A model adds cost and variance to questions your program can answer exactly.
Use the LLM where the check requires judgment: factual support, completeness, relevance, whether an explanation fits the intended audience. Even then, several small binary questions are usually more stable than one overall score from 1 to 10.
Optimize accuracy, stability, and cost together #
Zero variance is easy. An evaluator that always returns pass never flips.
It is also useless. So accuracy is a gate, not a term in the score.
Every candidate has to clear the accuracy requirements first. Only then do you compare stability, cost, and latency.
Hard requirements beat a combined score. A combined score lets a big cost improvement hide a meaningful accuracy regression, or lets a tiny accuracy gain justify doubling the bill.
If two configurations are statistically indistinguishable, take the cheaper and simpler one. Anthropic’s note on [statistical evaluation](https://www.anthropic.com/research/statistical-approach-to-model-evals) is a good reminder to report uncertainty instead of treating every decimal point as real.
For expensive or ambiguous cases, escalate:
Measure the whole route. A cheap first call followed by an expensive retry on 80% of cases is not a cheap evaluator. It is an expensive evaluator with a warm-up act.
Know when to stop #
The loop can run forever. Do not let it.
Stop when any of these happens:
- the variance target is reached
- validation improvements fall below a useful threshold
- the experiment budget is exhausted
- accuracy and stability stop improving together
- further improvements cost more than they are worth
Then run the winner against the untouched holdout set. Review every verdict that changed from the incumbent. If the result holds, promote the complete configuration, not just the prompt. Keep the old version around for regression testing.
Rerun the evaluator eval whenever the model, prompt, reasoning settings, schema, or provider behavior changes. Model upgrades change evaluator behavior even when your prompt does not.
The practical version #
If you already have an LLM evaluator, this is enough to start:
- Pull 50 to 100 labeled examples from real usage.
- Run each one five times with the current evaluator.
- Calculate accuracy, decision flip rate, cost, and latency.
- Split the cases into development, validation, and holdout sets.
- Give an agent the harness and ask for one measured experiment at a time.
- Tune the current prompt and build a second prompt from zero.
- Promote only changes that hold accuracy while reducing variance.
- Stop when the improvement is no longer worth the spend.
GEPA-style optimization can automate a more aggressive search over prompts. That may be useful later. Start with the simple loop, because it keeps every change easy to inspect and easy to revert.
The evaluator is part of the product. Test it like one. A stable liar is still a liar, so accuracy comes first. Then reduce the flips. Then reduce the bill.