I gave the same fabricated answer to RAGAS and DeepEval. One scored it 0.0. The other scored it 1.0

A developer testing the two most popular LLM-as-judge faithfulness metrics, RAGAS and DeepEval, found that they gave opposite scores for the same fabricated answer: RAGAS scored it 0.0 while DeepEval scored it 1.0 in all five repeats. The developer built OpenGATE, a deterministic evaluation tool, and ran a controlled comparison showing that LLM judges excel at detecting meaning inversions but fail on omissions and fabricated numbers, while deterministic checks catch those failures reliably.

Here's an output from a RAG system asserting a pricing claim it was never given, for a question its context couldn't answer. I ran it past the two most popular LLM-as-judge faithfulness metrics, five times each, judges on gpt-4o at temperature 0 — the setting most favourable to judge stability. RAGAS scored it 0.000 . DeepEval scored it 1.000 — in all five repeats — and explained itself: "The score is 1.00 because there are no contradictions between the actual output and the retrieval context." One repeat added: "Great job maintaining accuracy and consistency " Both metrics are called faithfulness . Both are internally consistent. Only one of them notices the fabrication. I want to explain why that happens, what else I found when I measured it properly, and why my conclusion is use both — an LLM judge and deterministic checks — rather than the takedown of LLM-as-judge you might be expecting. I build AI tools for healthcare — claim verification for medical writers, a patient-facing text simplifier that's a registered medical device, clinical de-identification. In this domain, an answer is only useful if it can be justified from its source material, and the failure modes that matter are quiet: a medication dose silently dropped from a simplified discharge letter; a reference range the model helpfully invented; a system that answers confidently when its context holds no answer. I needed to gate releases on these properties — fail the build when a prompt change makes the system less trustworthy. And LLM-as-judge metrics are awkward to gate a build on: they cost money per run, they're non-deterministic, and as it turned out two implementations of the "same" metric don't measure the same thing. So I built OpenGATE https://github.com/nickjlamb/opengate , where every check is a pure function of the output and a hand-labelled gold case: required facts must be present with accepted paraphrases , every number in the answer must trace back to the context, and when the context can't answer, the system must abstain. No grader model anywhere. Then I ran a controlled comparison against RAGAS and DeepEval to find out what each approach actually sees. A frozen corpus of 27 outputs: six captured system outputs three from production plus mutants injecting exactly one known defect each — a dropped fact, a fabricated number, a failure to abstain, an appended contradiction, or an in-place meaning inversion. Five repeats per arm. Judges on gpt-4o , temperature 0. Three design choices matter for fairness. Defect classes are grouped by scope and never summed — deterministic checks cannot see meaning inversions, so lumping everything into one accuracy number would be rigging it in either direction . Detection is measured as a reaction to the injected defect relative to the same case's unmutated base output, so a pre-existing quirk can't be miscounted as a catch. And an output only counts as detected if it's flagged in every repeat — a gate that fails intermittently isn't a gate. All inputs, scripts, and per-repeat scores are committed in the paper artifact https://doi.org/10.5281/zenodo.21365095 . Where the judges win — decisively. On meaning inversions "take with food" becomes "take on an empty stomach" while every anchor and number survives verbatim , RAGAS caught 4/5 and DeepEval 5/5. The deterministic checks caught 0/5 — and always will . String checks have no model of meaning. This is a categorical advantage for judges, not a marginal one, and it's why this article isn't a takedown. Where the judges lose. Omission. A missing antibiotic dose asserts nothing unsupported — so a faithfulness metric is blind to it by construction. The judges caught 0/5 and 1/5 dropped facts. The deterministic anchor check caught 5/5. On the discharge summary whose 500 mg dose I deleted, DeepEval returned 1.00 with the rationale "there are no contradictions." A confident explanation of a passing score, for an output missing its antibiotic dose. Cost and speed. $11.49 RAGAS and $8.29 DeepEval per 1,000 evaluations, versus $0.00. 194 and 119 seconds per repeat, versus 3.8 milliseconds. That difference decides whether you can run the check on every commit — or on every answer. Localisation. When the deterministic check fails, it names the failure: missing fact "500 mg" . RAGAS returns a scalar. DeepEval returns a rationale that is sometimes good and sometimes the "no contradictions" sentence above. The finding I care most about is definitional. RAGAS asks "is every claim supported by the context?" DeepEval's faithfulness asks "does any claim contradict the context?" An invented figure contradicts nothing — the context is silent about it. So on the six outputs asserting things their source doesn't contain, RAGAS caught 5 and DeepEval caught 0 . On outright contradictions they agree closely 7/10 vs 9/10 . The divergence is systematic, not noise. The uncomfortable implication: a team that adopts "LLM-as-judge faithfulness" without reading the implementation hasn't chosen a level of rigour. It has unknowingly chosen which failure mode to be blind to . For a patient-facing medical tool, an invented dose is the failure that matters most — and it's precisely what a contradiction-based judge waves through. I designed this experiment expecting to demonstrate judge instability . On a strong judge model, that finding is weak, and I'm reporting it rather than burying it: at gpt-4o /temp 0, mean per-output score range was 0.026 RAGAS and 0.013 DeepEval across repeats. The same corpus on gpt-4o-mini was substantially noisier 11/27 outputs changed verdict between repeats — so instability is a property of the judge model , not of judging as such. The case for not gating a build on a judge rests on omission-blindness, the definitional divergence, and cost — none of which a better judge model repairs. The framework runs in CI across four production systems. First-run results, all previously unknown: a silent parse failure defaulting ~50% of multi-claim verdicts to "Not Supported"; two name-capture bugs in a de-identification engine apostrophe surnames like O'Brien fell out of the capture pattern ; and the simplifier dropping that antibiotic dose from a discharge summary. My favourite failure came months later. After the dose-dropping fix, the same system produced a fabricated number: "Normally, we like to see a haemoglobin level above 12 g/dL" — clinically correct, and absent from the source letter. The earlier prompt fix said never omit or change the numbers ; it forbade losing a figure and was silent on inventing one. A prompt hardened against the last defect is not hardened against the next — which is the entire argument for a gate that runs on every change rather than an audit that ran once. A green scorecard is a measurement, not a property. The mutants are synthetic, derived from defects I'd observed rather than sampled from the wild. The corpus is 27 outputs. One judge model. And I wrote both the defect taxonomy and one of the arms — which is why the scopes are reported separately, and why the inversion class, where my own approach scores zero, is in the corpus at all. The full paper https://doi.org/10.5281/zenodo.21365095 has the threats-to-validity section, including the experiment I've pre-registered but not yet run an independent second labeller for the gold sets . Use both, for what each can see. An LLM judge for semantics — meaning inversions, contradiction, open-ended quality — where it's categorically better. Deterministic checks as the gate : required facts present, numbers traceable, abstention honoured — reproducible, named failures, free, fast enough for every commit and every answer. If you want to try the deterministic side, it's MIT-licensed and runs in one line with no API key: npx @pharmatools/opengate Python pip install opengate-grounding : python from opengate grounding import check grounding result = check grounding answer, context, the retrieved evidence anchors= "500 mg", "twice daily" , facts the answer must contain result.grounded deterministic — same evidence, same verdict, every run There's also a pytest helper, a DeepEval GroundingMetric yes — the integration works fine alongside the judge metrics; that's the point , a GitHub Action that fails the build on regressions, and an MCP server so agents can check their own answers before replying. Repo: github.com/nickjlamb/opengate https://github.com/nickjlamb/opengate · Paper: doi.org/10.5281/zenodo.21365095 https://doi.org/10.5281/zenodo.21365095 I'm a medical writer who moved into building AI tools for healthcare. Everything in this post is reproducible from the committed artifact — if you find something that doesn't hold, open an issue; that's what the gate is for.