{"slug": "llm-eval-strategy", "title": "LLM Eval Strategy", "summary": "A team's LLM evaluation strategy categorizes checks into three tiers—unforgeable proof, statistical signal, and model-as-judge—to reduce reliance on slow, expensive judge calls. The approach promotes recurring failures from Tier 3 to deterministic Tier 1 or 2 gates, aiming to handle 80% of failures with faster, cheaper verification.", "body_md": "# LLM Eval Strategy\n\nIn my experience rolling this out with my team, we realized that relying on a judge for everything creates a slow, expensive loop. The goal should be to promote recurring findings into faster, cheaper tiers of verification.\n\n## The Evidence Hierarchy\n\nTo do this, you have to categorize your checks by how \"corruptible\" they are, rather than just by cost:\n\n**Tier 1 (Unforgeable Proof):** These are binary facts. Did the code compile? Is the JSON valid? Did it timeout? These are free, instant, and deterministic.**Tier 2 (Statistical Signal):** These compare output against a baseline the agent didn't create, such as embedding similarity to a spec or repetition profiles.**Tier 3 (Model-as-Judge):** This is a subjective opinion. It's a signal, not a verdict, and it should stay offline because it's slow and inconsistent.\n\nThe rule is simple: Tier 1 and 2 are your real-time gates. Tier 3 is for discovery.\n\n## Implementing the Ratchet\n\nWhen a judge keeps flagging the same failure—for example, \"the summary cites a section that doesn't exist\"—you don't leave that check in Tier 3. You ask: what factual check would have caught this?\n\nIf you can verify that cited headers exist verbatim in the source, you've just promoted a subjective opinion to a Tier 1 deterministic gate.\n\n```\ntype Finding = {\n claim: string; // the judge's recurring complaint\n sourceText: string; // artifact the agent did NOT author\n citedSections: string[];\n};\n\n// Tier 1 promotion: the finding is now an unforgeable-proof check.\nfunction citationsExist(f: Finding): { pass: boolean; missing: string[] } {\n const missing = f.citedSections.filter(\n (s) => !f.sourceText.includes(s)\n );\n return { pass: missing.length === 0, missing };\n}\n\n// Tier 2 promotion: when membership is too rigid, drop to a\n// baseline-relative signal the agent didn't get to author.\nfunction citationGroundedness(\n f: Finding,\n embed: (t: string) => number[],\n cos: (a: number[], b: number[]) => number\n): number {\n const src = embed(f.sourceText);\n const scores = f.citedSections.map((s) => cos(embed(s), src));\n return scores.reduce((a, b) => a + b, 0) / (scores.length || 1);\n}\n```\n\nIf exact matching is too brittle, you drop to Tier 2 (like the `citationGroundedness`\n\nfunction above) to check similarity against the source. It's still deterministic and essentially free compared to an LLM call.\n\nBy pushing 80% of your failures into Tier 1 and 2, you stop treating your eval pipeline like a guessing game and start treating it like a deployment gate. Only the truly subtle 20% should ever reach the judge.\n\n[Next MCP Server: Do You Actually Need One? →](/en/threads/2468/)", "url": "https://wpnews.pro/news/llm-eval-strategy", "canonical_source": "https://promptcube3.com/en/threads/2479/", "published_at": "2026-07-23 18:46:29+00:00", "updated_at": "2026-07-24 03:05:37.159390+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/llm-eval-strategy", "markdown": "https://wpnews.pro/news/llm-eval-strategy.md", "text": "https://wpnews.pro/news/llm-eval-strategy.txt", "jsonld": "https://wpnews.pro/news/llm-eval-strategy.jsonld"}}