{"slug": "an-llm-judge-cannot-be-a-build-gate-and-it-is-not-about-the-cost", "title": "An LLM judge cannot be a build gate, and it is not about the cost", "summary": "A developer argues that LLM-based judges cannot serve as build gates in RAG evaluation due to cost and non-determinism, advocating for deterministic metrics like precision@k, recall@k, and lexical overlap proxies. The developer highlights that judged metrics are too expensive to run frequently and produce noisy results that make regression detection unreliable, whereas deterministic metrics provide consistent, cost-effective checks that can run on every commit.", "body_md": "Almost every RAG evaluation metric on offer needs a language model to produce it. Faithfulness, answer relevance, context precision: a model reads the answer and scores it.\n\nThose are good metrics. They measure things that are hard to measure otherwise, and for a research sweep or a quarterly quality review, use them.\n\nThey cannot gate a build, and the reason people usually give is only half of it.\n\nA judged evaluation costs money per run. A hundred cases across a few metrics is a few thousand model calls, which is real money on every commit and every branch.\n\nThe consequence is that you move the gate. It runs nightly instead of per commit, then weekly, then on a button someone remembers to press. A check that costs a dollar gets run less, and a check that runs less catches things later, which is the property you were buying.\n\nA judge is not deterministic.\n\nRun the same dataset against the same predictions twice and you get slightly different scores. Not wildly different, but different. That is workable for a report and disqualifying for a gate, because a gate exists to answer one question: did this change make things worse?\n\nAnswering it means comparing two numbers. If both carry noise of unknown size, you cannot separate a regression from the measurement. You get a check that fails sometimes for no reason, and the standard response to that is to disable it, usually within a week, usually by someone under deadline pressure.\n\nSo the judged metric fails twice: too expensive to run often, and untrustworthy on the difference when it does run.\n\nMore than you would think, and all of it deterministic.\n\nFrom labelled relevant documents: precision@k, recall@k, MRR, nDCG@k, hit rate. These need a golden dataset and nothing else.\n\nFrom expected answers: exact match after SQuAD style normalisation, token F1 for partial credit, required phrase presence for when an amount or a date must appear.\n\nRecall bounds everything downstream, so it is the one to be loudest about. The model cannot cite what retrieval never fetched. At recall@k of 0.6, forty percent of your questions were unanswerable before generation began, and no amount of prompt engineering touches that.\n\nPrecision measures how much noise is in the context window, which is what predicts hallucination.\n\nOne measure in my own tool is a proxy, and the documentation says so.\n\n`groundedness` is the share of answer content words that appear in the retrieved context:\n\n``` js\nconst answerTokens = normalise(predicted).split(' ').filter((t) => t.length > 2);\nconst contextTokens = new Set(normalise(contexts.join(' ')).split(' '));\n\nreturn answerTokens.filter((t) => contextTokens.has(t)).length / answerTokens.length;\n```\n\nThat is lexical overlap. It will miss a fluent misreading of a passage that was correctly retrieved, which a judge would catch. It does catch an answer invented wholesale, which is the failure that gets shipped, and it costs nothing and returns the same number every time.\n\nThe trade is a cheap proxy running on every commit against an accurate measure running monthly. Which one is the better metric and which one is still switched on in March are different questions.\n\nOne detail decides whether people trust the output.\n\nIf a case has no labelled relevant documents, the retrieval metrics for that case are omitted rather than scored zero. Aggregation skips missing values instead of averaging them in:\n\n```\nif (typeof value !== 'number' || Number.isNaN(value)) continue;\n```\n\nA partially labelled dataset should report what it can measure. Scoring the gaps as zero produces a column of failures that describes your labelling rather than your system, and nobody reads a report full of zeroes twice.\n\nThe platforms are priced for teams. Confident AI runs Free, Starter at $200 a month and Team at $2,000, with Enterprise above that. Braintrust Pro is $249. Galileo Pro is $100. Each has a free tier and each meters it: Confident AI's is two seats, one project and five test runs a week, which a per-commit gate exhausts by Tuesday. *(Checked on the vendors' own pages, 9 September 2026.)*\n\nThe open source libraries, RAGAS and DeepEval, compute good metrics and leave you to build the storage, the comparison and the CI gate yourself. Most solo projects end up with three tools wired together and no gate at all.\n\n```\nnpx ragbench gate --baseline main --threshold recall@k=0.8\n```\n\nLocal SQLite for history, zero dependencies, nothing leaves your machine. [ragbench](https://github.com/catidegla/ragbench).\n\nThe gate has a second half that matters more than the thresholds, and that is the next article.", "url": "https://wpnews.pro/news/an-llm-judge-cannot-be-a-build-gate-and-it-is-not-about-the-cost", "canonical_source": "https://dev.to/catidegla/an-llm-judge-cannot-be-a-build-gate-and-it-is-not-about-the-cost-314n", "published_at": "2026-09-09 16:50:07+00:00", "updated_at": "2026-09-09 17:19:15.254612+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "developer-tools", "ai-tools"], "entities": ["Confident AI", "Braintrust", "Galileo"], "alternates": {"html": "https://wpnews.pro/news/an-llm-judge-cannot-be-a-build-gate-and-it-is-not-about-the-cost", "markdown": "https://wpnews.pro/news/an-llm-judge-cannot-be-a-build-gate-and-it-is-not-about-the-cost.md", "text": "https://wpnews.pro/news/an-llm-judge-cannot-be-a-build-gate-and-it-is-not-about-the-cost.txt", "jsonld": "https://wpnews.pro/news/an-llm-judge-cannot-be-a-build-gate-and-it-is-not-about-the-cost.jsonld"}}