# LLM Eval Strategy

> Source: <https://promptcube3.com/en/threads/2479/>
> Published: 2026-07-23 18:46:29+00:00

# LLM Eval Strategy

In my experience rolling this out with my team, we realized that relying on a judge for everything creates a slow, expensive loop. The goal should be to promote recurring findings into faster, cheaper tiers of verification.

## The Evidence Hierarchy

To do this, you have to categorize your checks by how "corruptible" they are, rather than just by cost:

**Tier 1 (Unforgeable Proof):** These are binary facts. Did the code compile? Is the JSON valid? Did it timeout? These are free, instant, and deterministic.**Tier 2 (Statistical Signal):** These compare output against a baseline the agent didn't create, such as embedding similarity to a spec or repetition profiles.**Tier 3 (Model-as-Judge):** This is a subjective opinion. It's a signal, not a verdict, and it should stay offline because it's slow and inconsistent.

The rule is simple: Tier 1 and 2 are your real-time gates. Tier 3 is for discovery.

## Implementing the Ratchet

When a judge keeps flagging the same failure—for example, "the summary cites a section that doesn't exist"—you don't leave that check in Tier 3. You ask: what factual check would have caught this?

If you can verify that cited headers exist verbatim in the source, you've just promoted a subjective opinion to a Tier 1 deterministic gate.

```
type Finding = {
 claim: string; // the judge's recurring complaint
 sourceText: string; // artifact the agent did NOT author
 citedSections: string[];
};

// Tier 1 promotion: the finding is now an unforgeable-proof check.
function citationsExist(f: Finding): { pass: boolean; missing: string[] } {
 const missing = f.citedSections.filter(
 (s) => !f.sourceText.includes(s)
 );
 return { pass: missing.length === 0, missing };
}

// Tier 2 promotion: when membership is too rigid, drop to a
// baseline-relative signal the agent didn't get to author.
function citationGroundedness(
 f: Finding,
 embed: (t: string) => number[],
 cos: (a: number[], b: number[]) => number
): number {
 const src = embed(f.sourceText);
 const scores = f.citedSections.map((s) => cos(embed(s), src));
 return scores.reduce((a, b) => a + b, 0) / (scores.length || 1);
}
```

If exact matching is too brittle, you drop to Tier 2 (like the `citationGroundedness`

function above) to check similarity against the source. It's still deterministic and essentially free compared to an LLM call.

By pushing 80% of your failures into Tier 1 and 2, you stop treating your eval pipeline like a guessing game and start treating it like a deployment gate. Only the truly subtle 20% should ever reach the judge.

[Next MCP Server: Do You Actually Need One? →](/en/threads/2468/)