Last Updated on August 24, 2026 by Editorial Team Author(s): Armin Norouzi, Ph.D Originally published on Towards AI. The Evaluation Stack: Metrics That Predict Production Quality The hardest engineering problem in LLM deployment is not latency or cost — it is knowing whether your model got better. Metrics that seem rigorous (BLEU, ROUGE, BERTScore) correlate weakly with human judgment. Metrics that correlate well (GPT-4 judge, task success rate) cost hundreds of dollars per evaluation run and introduce their own biases. This article maps the evaluation landscape and proposes a layered stack that delivers high signal efficiently, without breaking the bank on every pull request. After explaining why surface metrics like BLEU and ROUGE fail (they reward overlap, are weakly correlated with human preference, and can be gamed), the article details how to use LLM judges responsibly by mitigating self-preference, position, length, rubric, and benchmark-saturation issues. It then argues for functional metrics (task success, spec compliance checklists) as an underused workhorse that is cheaper, more diagnostic, and harder to game. For safety, it highlights the base-rate/precision-recall tradeoff and emphasizes that safety thresholds are a product decision, not an F1 optimization. It covers benchmark saturation and why task-specific evaluation sets with headroom are necessary, then proposes ensemble evaluation across metric dimensions to improve reliability and reduce variance. Finally, it assembles all signals into a practical four-layer evaluation stack (continuous deterministic checks, nightly semantic/factual checks, release-candidate GPT-4 judging, and periodic human A/B evaluation), stressing evaluation-set hygiene and ongoing recalibration. Read the full blog for free on Medium. Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor. Published via Towards AI
The AI Feature That Works in the Demo and Breaks in Production