# Gate Agent Evals by Severity, Not a Flat Pass-Rate > Source: > Published: 2026-07-08 01:17:43+00:00 Gating an agent's eval run by severity means the deploy decision reads the per-severity distribution of the failures, not one flat pass-rate. `severity_gate.py` is an offline, stdlib-only tool that returns SHIP, REVIEW, or BLOCK. In this post's fixtures, flipping one `severity_class` field to critical turns SHIP into BLOCK while the pass-rate holds at 92.5%. AI disclosure:I wrote`severity_gate.py` with an AI assistant and ran it myself, offline, before publishing. Every output block below is pasted from a real local run on Python 3.13.5, standard library only, no network. I checked the exit codes (0 / 1 / 2), hashed the STDOUT of each scenario twice to confirm it is byte-for-byte deterministic, and edited every line. The 512 / 31 / 6 figures and the "rounding error" line belong to Ethan Walker's Dev.to post, not to me; I link the primary sources and keep their numbers in their own paragraphs, away from my fixture counts (40 / 3 / 2, 92.5%). **In short:** `severity_class` from `low` to `critical` on two of them, and the verdict flips SHIP to BLOCK. The pass-rate does not move. The diff is two lines.`json` , `sys` ). Offline, keyless, read-only, zero network, deterministic STDOUT. Exit 0 / 1 / 2 for a CI gate. The tool and every fixture are in this post.Here is the ritual I keep seeing at the end of an agent's CI pipeline. The eval suite runs, a number comes out, someone reads it, and if it clears the bar the change ships. The number is a mean: passes over total. It is a fine thing to *track*. It tells you, roughly, whether the run got better or worse than last week. It is a bad thing to *gate* on, and the reason is simple arithmetic. A mean gives every failure the same weight. So a run where forty tests failed on trailing whitespace and a run where one test leaked a customer's email into a log both cost the same amount of pass-rate. If your suite is big enough, the second run rounds off to nothing. The one failure that should stop the deploy is invisible at the resolution of the average. That is the whole gap this tool sits in. The pass-rate is *tracking*: a post-hoc measurement of what happened. The decision about what ships is *control*, and control has to see the distribution the mean flattens. `severity_gate.py` reads the finished eval results and makes the ship decision on the failures' severity, not their count. The tool keeps the pass-rate on screen, for reference, and then ignores it for the parts of the decision the mean can't reach. Three verdicts, one policy: `critical` . This fires no matter what the aggregate is. One critical fail out of ten thousand green cases is still a BLOCK.`high` ), or the flat pass-rate dropped below `ship_threshold` . A human looks. It is not an auto-ship and it is not a hard stop.The severity labels are yours. The tool does not invent them and does not guess them; it reads the `severity_class` field you put on each case. That is the load-bearing limit, and I come back to it at the end. What the tool contributes is that once you have committed to a label, the ship decision follows it deterministically instead of dissolving into an average. No keys. No network. No install beyond Python. Save the file, point it at a results JSON, run one command. Here is the whole thing, one file, standard library only: ``` bash #!/usr/bin/env python3 """ severity_gate.py -- an offline deploy gate that reads a FINISHED eval run and decides SHIP / REVIEW / BLOCK from the per-severity distribution of the failures, not from a single flat pass-rate. It reads a JSON file of eval RESULTS (it never runs the agent or the eval) plus an optional policy file. For every case it reads a verdict (pass/fail) and the severity_class the USER assigned (critical / high / medium / low). It reports the flat pass-rate for reference, then makes the ship decision on a rule the mean cannot express: BLOCK -- at least one FAIL whose severity_class is in the policy's block_on set (default: critical). Fires regardless of the aggregate. REVIEW -- no blocking fail, but either a FAIL in review_on (default: high), or the flat pass-rate is below ship_threshold. A human looks; it is not an auto-ship and not a hard block. SHIP -- no blocking fail, no review fail, and pass-rate >= ship_threshold. The point the tool exists to make: flip one field -- the severity_class on a handful of already-failing cases -- and the verdict moves SHIP -> BLOCK while the pass-rate does not move at all. The aggregate MEASURES the run; it was never the control over what ships. Offline. Keyless. Read-only. Zero network. Standard library only (json, sys). No subprocess, no exec, no eval, no import of the analyzed results, no model, no DB. The fixtures are read with json.load; they are DATA, never executed. Output is byte-for-byte deterministic across runs. It does NOT detect PII, does NOT judge whether a verdict is correct, and does NOT find new failures. It re-weights failures YOU already labeled. Garbage labels in, garbage gate out. It decouples the ship decision from the scalar mean; it does not discover anything. Exit codes (usable as a CI gate): 0 SHIP 1 REVIEW or BLOCK (the verdict text says which, and why) 2 bad input (missing / unreadable / unparseable file, missing field, unknown verdict or severity_class) -- fail-closed Usage: python3 severity_gate.py [policy.json] """ import json import sys DEFAULT_POLICY = { "ship_threshold": 0.90, "severity_order": ["low", "medium", "high", "critical"], "block_on": ["critical"], "review_on": ["high"], } VALID_VERDICTS = ("pass", "fail") def _bad(msg): print("ERROR: " + msg) raise SystemExit(2) def _load_json(path, what): try: with open(path, "r") as fh: return json.load(fh) except OSError as exc: _bad("cannot read %s %s: %s" % (what, path, exc)) except ValueError as exc: _bad("cannot parse %s %s: %s" % (what, path, exc)) def load_policy(path): if path is None: return dict(DEFAULT_POLICY) data = _load_json(path, "policy") if not isinstance(data, dict): _bad("policy %s is not a JSON object" % path) pol = dict(DEFAULT_POLICY) pol.update(data) order = pol["severity_order"] if not isinstance(order, list) or not order: _bad("policy severity_order must be a non-empty list") for key in ("block_on", "review_on"): for cls in pol[key]: if cls not in order: _bad("policy %s lists '%s', not in severity_order %s" % (key, cls, order)) try: pol["ship_threshold"] = float(pol["ship_threshold"]) except (TypeError, ValueError): _bad("policy ship_threshold must be a number") return pol def load_cases(path, order): data = _load_json(path, "results") cases = data.get("cases") if isinstance(data, dict) else data if not isinstance(cases, list) or not cases: _bad("results %s must hold a non-empty list of cases" % path) default_sev = order[0] out = [] for i, c in enumerate(cases): if not isinstance(c, dict): _bad("case %d is not an object" % i) verdict = c.get("verdict") if verdict not in VALID_VERDICTS: _bad("case %s has verdict %r, expected 'pass' or 'fail'" % (c.get("case_id", i), verdict)) sev = c.get("severity_class", default_sev) if sev not in order: _bad("case %s has severity_class %r, not in severity_order %s" % (c.get("case_id", i), sev, order)) out.append({ "case_id": str(c.get("case_id", "case-%d" % i)), "verdict": verdict, "severity_class": sev, "category": str(c.get("category", "")), }) return out def decide(cases, pol): block_on = set(pol["block_on"]) review_on = set(pol["review_on"]) total = len(cases) passed = sum(1 for c in cases if c["verdict"] == "pass") rate = passed / total block_fails = [c for c in cases if c["verdict"] == "fail" and c["severity_class"] in block_on] review_fails = [c for c in cases if c["verdict"] == "fail" and c["severity_class"] in review_on] if block_fails: verdict, code = "BLOCK", 1 reason = ("%d failing case(s) in a blocking severity class (%s)" % (len(block_fails), ", ".join(sorted(block_on)))) elif review_fails: verdict, code = "REVIEW", 1 reason = ("%d failing case(s) in a review severity class (%s)" % (len(review_fails), ", ".join(sorted(review_on)))) elif rate < pol["ship_threshold"]: verdict, code = "REVIEW", 1 reason = ("flat pass-rate %.1f%% is below ship_threshold %.1f%%" % (rate * 100, pol["ship_threshold"] * 100)) else: verdict, code = "SHIP", 0 reason = ("no blocking or review-class failure, pass-rate %.1f%% " ">= ship_threshold %.1f%%" % (rate * 100, pol["ship_threshold"] * 100)) return {"total": total, "passed": passed, "failed": total - passed, "rate": rate, "verdict": verdict, "reason": reason, "code": code} def render(cases, pol, d): order = pol["severity_order"] rank = {c: i for i, c in enumerate(order)} block_on, review_on = set(pol["block_on"]), set(pol["review_on"]) out = ["SEVERITY-GATE REPORT"] out.append("cases: %d pass: %d fail: %d" % (d["total"], d["passed"], d["failed"])) out.append("flat pass-rate: %.1f%% (%d/%d)" % (d["rate"] * 100, d["passed"], d["total"])) out.append("ship_threshold: %.1f%% block_on: [%s] review_on: [%s]" % (pol["ship_threshold"] * 100, ", ".join(pol["block_on"]), ", ".join(pol["review_on"]))) out.append("per-severity (fail / total):") for cls in reversed(order): # worst first tot = sum(1 for c in cases if c["severity_class"] == cls) fl = sum(1 for c in cases if c["severity_class"] == cls and c["verdict"] == "fail") mark = " <- blocking" if cls in block_on else ( " <- review" if cls in review_on else "") out.append(" %-8s %d / %d%s" % (cls, fl, tot, mark)) fails = [c for c in cases if c["verdict"] == "fail"] fails.sort(key=lambda c: (-rank[c["severity_class"]], c["case_id"])) if fails: out.append("failing cases (worst severity first):") for c in fails: cat = (" [%s]" % c["category"]) if c["category"] else "" out.append(" - %-8s %s%s" % (c["severity_class"], c["case_id"], cat)) out.append("decision: %s -- %s" % (d["verdict"], d["reason"])) return "\n".join(out) def main(argv): if len(argv) not in (2, 3): print("usage: severity_gate.py [policy.json]") raise SystemExit(2) pol = load_policy(argv[2] if len(argv) == 3 else None) cases = load_cases(argv[1], pol["severity_order"]) d = decide(cases, pol) print(render(cases, pol, d)) raise SystemExit(d["code"]) if __name__ == "__main__": main(sys.argv) ``` The inputs are eval results, one JSON object per run: a `cases` list where each case has a `verdict` and a `severity_class` . To keep the runs below reproducible byte for byte, I generate the sample files with a tiny builder that only writes data. The gate reads that data; it never runs it. ``` bash #!/usr/bin/env python3 # make_fixtures.py -- writes the sample eval-result files this post runs on. # It only emits DATA. severity_gate.py reads it with json.load, never executes it. import json import os os.makedirs("fixtures", exist_ok=True) def case(i, verdict="pass", sev="low", cat="general"): return {"case_id": "c%03d" % i, "verdict": verdict, "severity_class": sev, "category": cat} def write(name, cases): with open("fixtures/%s" % name, "w") as fh: json.dump({"cases": cases}, fh, indent=2, sort_keys=True) fh.write("\n") # 40 cases, 3 fail -> flat pass-rate 37/40 = 92.5%. Two fails are a PII cluster # left untriaged (severity low); one is an unrelated format miss (medium). base = [case(i) for i in range(1, 41)] base[9] = case(10, "fail", "medium", "output_format") base[19] = case(20, "fail", "low", "pii_leak") base[29] = case(30, "fail", "low", "pii_leak") write("ship_run.json", base) # Same 40 cases, same 37/40 = 92.5%. ONLY the two PII fields change: low->critical. killer = [dict(c) for c in base] killer[19]["severity_class"] = "critical" killer[29]["severity_class"] = "critical" write("block_run.json", killer) # Aggregate below the bar, nothing critical/high: 34/40 = 85.0%. review = [case(i) for i in range(1, 41)] for i in (5, 12, 18, 23, 31, 37): review[i - 1] = case(i, "fail", "medium", "verbosity") write("review_run.json", review) # A critical-severity case that PASSED must not block. 20 cases, one low fail. edge = [case(i) for i in range(1, 21)] edge[0] = case(1, "pass", "critical", "pii_leak") edge[14] = case(15, "fail", "low", "typo") write("edge_run.json", edge) # Unknown severity label -> fail-closed (exit 2). bad = [case(1), case(2, "fail", "catastrophic", "pii_leak")] write("bad_run.json", bad) ``` Start with `ship_run.json` : 40 cases, 37 pass, 3 fail. The pass-rate is 37/40 = 92.5%, comfortably over the 90% bar. Two of the three failures are a `pii_leak` cluster that nobody has triaged yet, so they carry the default `low` severity. The third is a formatting miss, `medium` . My fixture, my run: ``` bash $ python3 severity_gate.py fixtures/ship_run.json SEVERITY-GATE REPORT cases: 40 pass: 37 fail: 3 flat pass-rate: 92.5% (37/40) ship_threshold: 90.0% block_on: [critical] review_on: [high] per-severity (fail / total): critical 0 / 0 <- blocking high 0 / 0 <- review medium 1 / 1 low 2 / 39 failing cases (worst severity first): - medium c010 [output_format] - low c020 [pii_leak] - low c030 [pii_leak] decision: SHIP -- no blocking or review-class failure, pass-rate 92.5% >= ship_threshold 90.0% ``` Exit 0. SHIP. This is the run that ships today under a flat pass-rate: the number is green, nothing is labeled critical, out the door it goes. The two `pii_leak` cases are sitting right there in the report, but at severity `low` they are just two more failures the mean absorbed. This is the demo the post exists for. `block_run.json` is `ship_run.json` with exactly one thing changed: the two `pii_leak` cases, the ones already failing, are relabeled from `low` to `critical` . Same 40 cases. Same 37 passes. Same 92.5%. The diff: ``` bash $ diff fixtures/ship_run.json fixtures/block_run.json 120c120 < "severity_class": "low", --- > "severity_class": "critical", 180c180 < "severity_class": "low", --- > "severity_class": "critical", ``` Two lines. Nothing about the pass-rate changed, because nothing about the pass/fail counts changed. Now run the gate on it: ``` bash $ python3 severity_gate.py fixtures/block_run.json SEVERITY-GATE REPORT cases: 40 pass: 37 fail: 3 flat pass-rate: 92.5% (37/40) ship_threshold: 90.0% block_on: [critical] review_on: [high] per-severity (fail / total): critical 2 / 2 <- blocking high 0 / 0 <- review medium 1 / 1 low 0 / 37 failing cases (worst severity first): - critical c020 [pii_leak] - critical c030 [pii_leak] - medium c010 [output_format] decision: BLOCK -- 2 failing case(s) in a blocking severity class (critical) ``` Exit 1. BLOCK. The pass-rate line is identical to the SHIP run, character for character: `flat pass-rate: 92.5% (37/40)` . The deploy verdict inverted anyway. This is the thing to sit with. If the mean were a valid ship criterion, a change that leaves the mean untouched could not change the decision. It changed the decision. So the mean was not the criterion; it was a number we let stand in for one. And it would take one, not two. Relabel a single `pii_leak` case to `critical` and the gate blocks, because the policy tolerates zero critical failures. The cluster size is not the point. The point is that a severity edit with no effect on the aggregate has total effect on the verdict. If the tool blocked on everything, or ignored the pass-rate entirely, it would be a different kind of useless. So here is the counter-case that would break the thesis if it came out wrong. `review_run.json` drops the aggregate below the bar, 34/40 = 85.0%, but every failure is benign: six `medium` cases, nothing critical, nothing high. ``` bash $ python3 severity_gate.py fixtures/review_run.json SEVERITY-GATE REPORT cases: 40 pass: 34 fail: 6 flat pass-rate: 85.0% (34/40) ship_threshold: 90.0% block_on: [critical] review_on: [high] per-severity (fail / total): critical 0 / 0 <- blocking high 0 / 0 <- review medium 6 / 6 low 0 / 34 failing cases (worst severity first): - medium c005 [verbosity] - medium c012 [verbosity] - medium c018 [verbosity] - medium c023 [verbosity] - medium c031 [verbosity] - medium c037 [verbosity] decision: REVIEW -- flat pass-rate 85.0% is below ship_threshold 90.0% ``` Exit 1, but REVIEW, not BLOCK, and the reason names the aggregate, not a severity class. This is the third state doing real work. The gate did not throw the pass-rate away; when nothing dangerous failed, the aggregate is exactly what it falls back on. REVIEW is where a low mean lands. BLOCK is reserved for the failure class you said must never ship. Put this run next to the killer and the tool's shape is clear: it answers to the *distribution* of severity when severity is present, and to the *mean* when it is not. One boundary a careful reader will poke at. Does BLOCK fire on the presence of a critical case, or on a critical *failure*? It has to be the failure, or you could never ship a suite that tests a critical path at all. `edge_run.json` has a `critical` -severity case that *passed* (your PII redaction test ran and came back clean) and one unrelated `low` failure. ``` bash $ python3 severity_gate.py fixtures/edge_run.json SEVERITY-GATE REPORT cases: 20 pass: 19 fail: 1 flat pass-rate: 95.0% (19/20) ship_threshold: 90.0% block_on: [critical] review_on: [high] per-severity (fail / total): critical 0 / 1 <- blocking high 0 / 0 <- review medium 0 / 0 low 1 / 19 failing cases (worst severity first): - low c015 [typo] decision: SHIP -- no blocking or review-class failure, pass-rate 95.0% >= ship_threshold 90.0% ``` Exit 0. SHIP. Note the per-severity line: `critical 0 / 1` — one critical case exists, zero critical cases failed. A passing test on a critical path is exactly the signal you want; blocking on it would punish you for having good coverage. The gate blocks on `critical` *fail*, not on `critical` present. The default policy blocks on `critical` and reviews on `high` . That default is mine, and you should not keep it without thinking. The mapping from a failure category to a severity class, and the choice of which classes stop a deploy, is a decision your team owns, not something a linter should hand you. So the policy is a file you pass in. Here is `fixtures/strict_policy.json` — four lines that also block on `high` and send `medium` to review: ``` { "ship_threshold": 0.90, "block_on": ["critical", "high"], "review_on": ["medium"] } ``` Point the same `ship_run.json` at it: ``` bash $ python3 severity_gate.py fixtures/ship_run.json fixtures/strict_policy.json SEVERITY-GATE REPORT cases: 40 pass: 37 fail: 3 flat pass-rate: 92.5% (37/40) ship_threshold: 90.0% block_on: [critical, high] review_on: [medium] per-severity (fail / total): critical 0 / 0 <- blocking high 0 / 0 <- blocking medium 1 / 1 <- review low 2 / 39 failing cases (worst severity first): - medium c010 [output_format] - low c020 [pii_leak] - low c030 [pii_leak] decision: REVIEW -- 1 failing case(s) in a review severity class (medium) ``` Exit 1. Same run, same 92.5%, but under a policy that treats a `medium` format miss as review-worthy, the verdict is REVIEW. The tool did not decide that a format miss matters. You did, in the policy file. The gate just applied it the same way every time. I did not come up with this from nothing. On July 5, Ethan Walker published a Dev.to post whose title is the whole argument in one line: [a 94% pass rate hid a PII leak in 6 test cases](https://dev.to/ethanwritesai/-a-94-pass-rate-hid-a-pii-leak-in-6-test-cases-2ei5). His numbers, from his run, stated as his: 512 test cases, 31 failures, and 6 of those were a PII leak. His line, which I checked against the post before quoting: *"Six out of 512 is a rounding error against a flat pass-rate metric."* He also names the tools by hand — DeepEval, Promptfoo, LangSmith — and writes that they "all give you this by default; none of them force severity weighting on you." Those figures are his and none of the counts in my runs above are. I built fixtures of a different size on purpose so the two never blur: my numbers are 40 / 3 / 2 at 92.5%, computed by my tool on my synthetic files; his are 512 / 31 / 6 at 94%, from his own eval suite. The tool in this post is one narrow, runnable answer to the thing his title describes: the point where a small critical cluster hides inside a green aggregate. The general form of it was put well the same week by another writer, posting as aiexplore369zoho, under the heading [the mean is lying to you](https://dev.to/aiexplore369zoho/the-mean-is-lying-to-you-benchmarks-hide-the-variance-that-breaks-prod-1oil). Their framing: *"A benchmark score is a mean."* Reliability, they argue, is a tail statistic, not a central one; the mean can stay flat while the failure rate on a critical slice doubles. That is the concept. Naming the slice that must not fail and gating on it directly, instead of on the average that buries it, is one way to act on it. A fair objection: skip the labels, put a stronger model in the loop as an inspector, and let it catch the critical cases. Someone tested exactly that. Posting as zxpmail, they [ran three models as agent quality inspectors](https://dev.to/zxpmail/i-tested-3-models-as-ai-agent-quality-inspectors-the-stronger-the-model-the-more-valid-work-it-gl7) and reported the opposite of a free lunch: the stronger the model, the more valid work it rejected. Their strongest model reached 0% false positives on garbage but falsely rejected 3 of 4 perfectly valid outputs. Those are their measurements, not mine. The takeaway I draw for a *gate* is narrow. A probabilistic inspector trades one error for another and gives you a different answer on reruns. A gate over labels you already committed to is deterministic: same input, same verdict, every run. That determinism is not a nice-to-have here; it is the property the SHA hashes at the end are there to prove. This is a spoke on the [pre-execution gate for AI agents](https://finops.spinov.online/blog/pre-execution-gate-for-ai-agents/) cluster, and its object is the deploy *decision* over a finished eval run. The neighbors ask adjacent questions, and the differences matter: I would rather undersell this than have you deploy it as something it isn't. `fail` is really a fail or a `pass` is really a pass. It finds no new failures. Mislabel a critical failure as `low` and it ships; that is the GIGO limit, and it is real. The tool re-weights failures you already found and classified. It decouples the ship decision from the scalar mean. It does not discover anything.`critical / high / medium / low` and the choice of what blocks are a file you own. The tool ships a default so it runs out of the box; the default is a starting point to argue with, not a standard.One design choice I am still not certain about: REVIEW and BLOCK share exit 1, and only the verdict text tells them apart. I did that so a CI job has a clean "0 means auto-ship, non-zero means a human decides" contract, with exit 2 reserved for "I could not even read the input." If your pipeline needs to branch differently on REVIEW versus BLOCK, you would want three non-zero codes, and I could see arguing it either way. I went with the two-state contract because it matches the fail-closed shape of the other gates in this series, but I would not fight hard for it. A gate that crashes into a green is worse than no gate. Point it at a run with a severity label it does not recognize, and it refuses to decide rather than guess: ``` bash $ python3 severity_gate.py fixtures/bad_run.json ERROR: case c002 has severity_class 'catastrophic', not in severity_order ['low', 'medium', 'high', 'critical'] $ echo $? 2 ``` No path argument, a missing file, an unparseable JSON, a case missing `verdict` , an unknown severity value: all exit 2, distinct from the exit 1 a REVIEW or BLOCK returns, so your CI can tell "the gate says hold" apart from "the gate could not run." I ran each scenario twice and hashed the full STDOUT both times, on Python 3.13.5, offline: `ship_run` is `cca7a591...` , `block_run` is `346ee92c...` , `review_run` is `6a27e17f...` , `edge_run` is `99f4e8dc...` , and `ship_run` under the strict policy is `35f9f6bb...` . Identical across both runs, every time. Here is the one I do not have a good number for. For teams shipping an agent behind an eval suite: how many of your failing cases are labeled with a severity at all? Not "do you have an eval," which most people say yes to, but whether the `pii_leak` and the trailing-whitespace failure carry different weights when the deploy decision gets made, or whether they both just dent the same pass-rate. My guess is that most suites are still gating on the flat number and the severity field is empty, but that is a guess, and I would like to know if I am wrong. If this was useful, follow along here for the next runnable gate in the series, and tell me in the comments where a green aggregate has hidden a failure that should have blocked your deploy. I read every one.