In my experience rolling this out with my team, we realized that relying on a judge for everything creates a slow, expensive loop. The goal should be to promote recurring findings into faster, cheaper tiers of verification.
The Evidence Hierarchy #
To do this, you have to categorize your checks by how "corruptible" they are, rather than just by cost:
Tier 1 (Unforgeable Proof): These are binary facts. Did the code compile? Is the JSON valid? Did it timeout? These are free, instant, and deterministic.Tier 2 (Statistical Signal): These compare output against a baseline the agent didn't create, such as embedding similarity to a spec or repetition profiles.Tier 3 (Model-as-Judge): This is a subjective opinion. It's a signal, not a verdict, and it should stay offline because it's slow and inconsistent.
The rule is simple: Tier 1 and 2 are your real-time gates. Tier 3 is for discovery.
Implementing the Ratchet #
When a judge keeps flagging the same failure—for example, "the summary cites a section that doesn't exist"—you don't leave that check in Tier 3. You ask: what factual check would have caught this?
If you can verify that cited headers exist verbatim in the source, you've just promoted a subjective opinion to a Tier 1 deterministic gate.
type Finding = {
claim: string; // the judge's recurring complaint
sourceText: string; // artifact the agent did NOT author
citedSections: string[];
};
// Tier 1 promotion: the finding is now an unforgeable-proof check.
function citationsExist(f: Finding): { pass: boolean; missing: string[] } {
const missing = f.citedSections.filter(
(s) => !f.sourceText.includes(s)
);
return { pass: missing.length === 0, missing };
}
// Tier 2 promotion: when membership is too rigid, drop to a
// baseline-relative signal the agent didn't get to author.
function citationGroundedness(
f: Finding,
embed: (t: string) => number[],
cos: (a: number[], b: number[]) => number
): number {
const src = embed(f.sourceText);
const scores = f.citedSections.map((s) => cos(embed(s), src));
return scores.reduce((a, b) => a + b, 0) / (scores.length || 1);
}
If exact matching is too brittle, you drop to Tier 2 (like the citationGroundedness
function above) to check similarity against the source. It's still deterministic and essentially free compared to an LLM call.
By pushing 80% of your failures into Tier 1 and 2, you stop treating your eval pipeline like a guessing game and start treating it like a deployment gate. Only the truly subtle 20% should ever reach the judge.
Next MCP Server: Do You Actually Need One? →