Gating an agent's eval run by severity means the deploy decision reads the per-severity distribution of the failures, not one flat pass-rate. severity_gate.py
is an offline, stdlib-only tool that returns SHIP, REVIEW, or BLOCK. In this post's fixtures, flipping one severity_class
field to critical turns SHIP into BLOCK while the pass-rate holds at 92.5%.
AI disclosure:I wroteseverity_gate.py
with an AI assistant and ran it myself, offline, before publishing. Every output block below is pasted from a real local run on Python 3.13.5, standard library only, no network. I checked the exit codes (0 / 1 / 2), hashed the STDOUT of each scenario twice to confirm it is byte-for-byte deterministic, and edited every line. The 512 / 31 / 6 figures and the "rounding error" line belong to Ethan Walker's Dev.to post, not to me; I link the primary sources and keep their numbers in their own paragraphs, away from my fixture counts (40 / 3 / 2, 92.5%).
In short:
severity_class
from low
to critical
on two of them, and the verdict flips SHIP to BLOCK. The pass-rate does not move. The diff is two lines.json
, sys
). Offline, keyless, read-only, zero network, deterministic STDOUT. Exit 0 / 1 / 2 for a CI gate. The tool and every fixture are in this post.Here is the ritual I keep seeing at the end of an agent's CI pipeline. The eval suite runs, a number comes out, someone reads it, and if it clears the bar the change ships. The number is a mean: passes over total. It is a fine thing to track. It tells you, roughly, whether the run got better or worse than last week.
It is a bad thing to gate on, and the reason is simple arithmetic. A mean gives every failure the same weight. So a run where forty tests failed on trailing whitespace and a run where one test leaked a customer's email into a log both cost the same amount of pass-rate. If your suite is big enough, the second run rounds off to nothing. The one failure that should stop the deploy is invisible at the resolution of the average.
That is the whole gap this tool sits in. The pass-rate is tracking: a post-hoc measurement of what happened. The decision about what ships is control, and control has to see the distribution the mean flattens. severity_gate.py
reads the finished eval results and makes the ship decision on the failures' severity, not their count.
The tool keeps the pass-rate on screen, for reference, and then ignores it for the parts of the decision the mean can't reach. Three verdicts, one policy:
critical
. This fires no matter what the aggregate is. One critical fail out of ten thousand green cases is still a BLOCK.high
), or the flat pass-rate dropped below ship_threshold
. A human looks. It is not an auto-ship and it is not a hard stop.The severity labels are yours. The tool does not invent them and does not guess them; it reads the severity_class
field you put on each case. That is the load-bearing limit, and I come back to it at the end. What the tool contributes is that once you have committed to a label, the ship decision follows it deterministically instead of dissolving into an average.
No keys. No network. No install beyond Python. Save the file, point it at a results JSON, run one command. Here is the whole thing, one file, standard library only:
#!/usr/bin/env python3
"""
severity_gate.py -- an offline deploy gate that reads a FINISHED eval run and
decides SHIP / REVIEW / BLOCK from the per-severity distribution of the failures,
not from a single flat pass-rate.
It reads a JSON file of eval RESULTS (it never runs the agent or the eval) plus
an optional policy file. For every case it reads a verdict (pass/fail) and the
severity_class the USER assigned (critical / high / medium / low). It reports the
flat pass-rate for reference, then makes the ship decision on a rule the mean
cannot express:
BLOCK -- at least one FAIL whose severity_class is in the policy's block_on
set (default: critical). Fires regardless of the aggregate.
REVIEW -- no blocking fail, but either a FAIL in review_on (default: high),
or the flat pass-rate is below ship_threshold. A human looks; it is
not an auto-ship and not a hard block.
SHIP -- no blocking fail, no review fail, and pass-rate >= ship_threshold.
The point the tool exists to make: flip one field -- the severity_class on a
handful of already-failing cases -- and the verdict moves SHIP -> BLOCK while the
pass-rate does not move at all. The aggregate MEASURES the run; it was never the
control over what ships.
Offline. Keyless. Read-only. Zero network. Standard library only (json, sys).
No subprocess, no exec, no eval, no import of the analyzed results, no model, no
DB. The fixtures are read with json.load; they are DATA, never executed. Output
is byte-for-byte deterministic across runs.
It does NOT detect PII, does NOT judge whether a verdict is correct, and does NOT
find new failures. It re-weights failures YOU already labeled. Garbage labels in,
garbage gate out. It decouples the ship decision from the scalar mean; it does
not discover anything.
Exit codes (usable as a CI gate):
0 SHIP
1 REVIEW or BLOCK (the verdict text says which, and why)
2 bad input (missing / unreadable / unparseable file, missing field, unknown
verdict or severity_class) -- fail-closed
Usage:
python3 severity_gate.py <results.json> [policy.json]
"""
import json
import sys
DEFAULT_POLICY = {
"ship_threshold": 0.90,
"severity_order": ["low", "medium", "high", "critical"],
"block_on": ["critical"],
"review_on": ["high"],
}
VALID_VERDICTS = ("pass", "fail")
def _bad(msg):
print("ERROR: " + msg)
raise SystemExit(2)
def _load_json(path, what):
try:
with open(path, "r") as fh:
return json.load(fh)
except OSError as exc:
_bad("cannot read %s %s: %s" % (what, path, exc))
except ValueError as exc:
_bad("cannot parse %s %s: %s" % (what, path, exc))
def load_policy(path):
if path is None:
return dict(DEFAULT_POLICY)
data = _load_json(path, "policy")
if not isinstance(data, dict):
_bad("policy %s is not a JSON object" % path)
pol = dict(DEFAULT_POLICY)
pol.update(data)
order = pol["severity_order"]
if not isinstance(order, list) or not order:
_bad("policy severity_order must be a non-empty list")
for key in ("block_on", "review_on"):
for cls in pol[key]:
if cls not in order:
_bad("policy %s lists '%s', not in severity_order %s"
% (key, cls, order))
try:
pol["ship_threshold"] = float(pol["ship_threshold"])
except (TypeError, ValueError):
_bad("policy ship_threshold must be a number")
return pol
def load_cases(path, order):
data = _load_json(path, "results")
cases = data.get("cases") if isinstance(data, dict) else data
if not isinstance(cases, list) or not cases:
_bad("results %s must hold a non-empty list of cases" % path)
default_sev = order[0]
out = []
for i, c in enumerate(cases):
if not isinstance(c, dict):
_bad("case %d is not an object" % i)
verdict = c.get("verdict")
if verdict not in VALID_VERDICTS:
_bad("case %s has verdict %r, expected 'pass' or 'fail'"
% (c.get("case_id", i), verdict))
sev = c.get("severity_class", default_sev)
if sev not in order:
_bad("case %s has severity_class %r, not in severity_order %s"
% (c.get("case_id", i), sev, order))
out.append({
"case_id": str(c.get("case_id", "case-%d" % i)),
"verdict": verdict,
"severity_class": sev,
"category": str(c.get("category", "")),
})
return out
def decide(cases, pol):
block_on = set(pol["block_on"])
review_on = set(pol["review_on"])
total = len(cases)
passed = sum(1 for c in cases if c["verdict"] == "pass")
rate = passed / total
block_fails = [c for c in cases
if c["verdict"] == "fail" and c["severity_class"] in block_on]
review_fails = [c for c in cases
if c["verdict"] == "fail" and c["severity_class"] in review_on]
if block_fails:
verdict, code = "BLOCK", 1
reason = ("%d failing case(s) in a blocking severity class (%s)"
% (len(block_fails), ", ".join(sorted(block_on))))
elif review_fails:
verdict, code = "REVIEW", 1
reason = ("%d failing case(s) in a review severity class (%s)"
% (len(review_fails), ", ".join(sorted(review_on))))
elif rate < pol["ship_threshold"]:
verdict, code = "REVIEW", 1
reason = ("flat pass-rate %.1f%% is below ship_threshold %.1f%%"
% (rate * 100, pol["ship_threshold"] * 100))
else:
verdict, code = "SHIP", 0
reason = ("no blocking or review-class failure, pass-rate %.1f%% "
">= ship_threshold %.1f%%" % (rate * 100,
pol["ship_threshold"] * 100))
return {"total": total, "passed": passed, "failed": total - passed,
"rate": rate, "verdict": verdict, "reason": reason, "code": code}
def render(cases, pol, d):
order = pol["severity_order"]
rank = {c: i for i, c in enumerate(order)}
block_on, review_on = set(pol["block_on"]), set(pol["review_on"])
out = ["SEVERITY-GATE REPORT"]
out.append("cases: %d pass: %d fail: %d"
% (d["total"], d["passed"], d["failed"]))
out.append("flat pass-rate: %.1f%% (%d/%d)"
% (d["rate"] * 100, d["passed"], d["total"]))
out.append("ship_threshold: %.1f%% block_on: [%s] review_on: [%s]"
% (pol["ship_threshold"] * 100,
", ".join(pol["block_on"]), ", ".join(pol["review_on"])))
out.append("per-severity (fail / total):")
for cls in reversed(order): # worst first
tot = sum(1 for c in cases if c["severity_class"] == cls)
fl = sum(1 for c in cases
if c["severity_class"] == cls and c["verdict"] == "fail")
mark = " <- blocking" if cls in block_on else (
" <- review" if cls in review_on else "")
out.append(" %-8s %d / %d%s" % (cls, fl, tot, mark))
fails = [c for c in cases if c["verdict"] == "fail"]
fails.sort(key=lambda c: (-rank[c["severity_class"]], c["case_id"]))
if fails:
out.append("failing cases (worst severity first):")
for c in fails:
cat = (" [%s]" % c["category"]) if c["category"] else ""
out.append(" - %-8s %s%s" % (c["severity_class"], c["case_id"], cat))
out.append("decision: %s -- %s" % (d["verdict"], d["reason"]))
return "\n".join(out)
def main(argv):
if len(argv) not in (2, 3):
print("usage: severity_gate.py <results.json> [policy.json]")
raise SystemExit(2)
pol = load_policy(argv[2] if len(argv) == 3 else None)
cases = load_cases(argv[1], pol["severity_order"])
d = decide(cases, pol)
print(render(cases, pol, d))
raise SystemExit(d["code"])
if __name__ == "__main__":
main(sys.argv)
The inputs are eval results, one JSON object per run: a cases
list where each case has a verdict
and a severity_class
. To keep the runs below reproducible byte for byte, I generate the sample files with a tiny builder that only writes data. The gate reads that data; it never runs it.
#!/usr/bin/env python3
import json
import os
os.makedirs("fixtures", exist_ok=True)
def case(i, verdict="pass", sev="low", cat="general"):
return {"case_id": "c%03d" % i, "verdict": verdict,
"severity_class": sev, "category": cat}
def write(name, cases):
with open("fixtures/%s" % name, "w") as fh:
json.dump({"cases": cases}, fh, indent=2, sort_keys=True)
fh.write("\n")
base = [case(i) for i in range(1, 41)]
base[9] = case(10, "fail", "medium", "output_format")
base[19] = case(20, "fail", "low", "pii_leak")
base[29] = case(30, "fail", "low", "pii_leak")
write("ship_run.json", base)
killer = [dict(c) for c in base]
killer[19]["severity_class"] = "critical"
killer[29]["severity_class"] = "critical"
write("block_run.json", killer)
review = [case(i) for i in range(1, 41)]
for i in (5, 12, 18, 23, 31, 37):
review[i - 1] = case(i, "fail", "medium", "verbosity")
write("review_run.json", review)
edge = [case(i) for i in range(1, 21)]
edge[0] = case(1, "pass", "critical", "pii_leak")
edge[14] = case(15, "fail", "low", "typo")
write("edge_run.json", edge)
bad = [case(1), case(2, "fail", "catastrophic", "pii_leak")]
write("bad_run.json", bad)
Start with ship_run.json
: 40 cases, 37 pass, 3 fail. The pass-rate is 37/40 = 92.5%, comfortably over the 90% bar. Two of the three failures are a pii_leak
cluster that nobody has triaged yet, so they carry the default low
severity. The third is a formatting miss, medium
. My fixture, my run:
$ python3 severity_gate.py fixtures/ship_run.json
SEVERITY-GATE REPORT
cases: 40 pass: 37 fail: 3
flat pass-rate: 92.5% (37/40)
ship_threshold: 90.0% block_on: [critical] review_on: [high]
per-severity (fail / total):
critical 0 / 0 <- blocking
high 0 / 0 <- review
medium 1 / 1
low 2 / 39
failing cases (worst severity first):
- medium c010 [output_format]
- low c020 [pii_leak]
- low c030 [pii_leak]
decision: SHIP -- no blocking or review-class failure, pass-rate 92.5% >= ship_threshold 90.0%
Exit 0. SHIP. This is the run that ships today under a flat pass-rate: the number is green, nothing is labeled critical, out the door it goes. The two pii_leak
cases are sitting right there in the report, but at severity low
they are just two more failures the mean absorbed.
This is the demo the post exists for. block_run.json
is ship_run.json
with exactly one thing changed: the two pii_leak
cases, the ones already failing, are relabeled from low
to critical
. Same 40 cases. Same 37 passes. Same 92.5%. The diff:
$ diff fixtures/ship_run.json fixtures/block_run.json
120c120
< "severity_class": "low",
---
> "severity_class": "critical",
180c180
< "severity_class": "low",
---
> "severity_class": "critical",
Two lines. Nothing about the pass-rate changed, because nothing about the pass/fail counts changed. Now run the gate on it:
$ python3 severity_gate.py fixtures/block_run.json
SEVERITY-GATE REPORT
cases: 40 pass: 37 fail: 3
flat pass-rate: 92.5% (37/40)
ship_threshold: 90.0% block_on: [critical] review_on: [high]
per-severity (fail / total):
critical 2 / 2 <- blocking
high 0 / 0 <- review
medium 1 / 1
low 0 / 37
failing cases (worst severity first):
- critical c020 [pii_leak]
- critical c030 [pii_leak]
- medium c010 [output_format]
decision: BLOCK -- 2 failing case(s) in a blocking severity class (critical)
Exit 1. BLOCK. The pass-rate line is identical to the SHIP run, character for character: flat pass-rate: 92.5% (37/40)
. The deploy verdict inverted anyway. This is the thing to sit with. If the mean were a valid ship criterion, a change that leaves the mean untouched could not change the decision. It changed the decision. So the mean was not the criterion; it was a number we let stand in for one.
And it would take one, not two. Relabel a single pii_leak
case to critical
and the gate blocks, because the policy tolerates zero critical failures. The cluster size is not the point. The point is that a severity edit with no effect on the aggregate has total effect on the verdict.
If the tool blocked on everything, or ignored the pass-rate entirely, it would be a different kind of useless. So here is the counter-case that would break the thesis if it came out wrong. review_run.json
drops the aggregate below the bar, 34/40 = 85.0%, but every failure is benign: six medium
cases, nothing critical, nothing high.
$ python3 severity_gate.py fixtures/review_run.json
SEVERITY-GATE REPORT
cases: 40 pass: 34 fail: 6
flat pass-rate: 85.0% (34/40)
ship_threshold: 90.0% block_on: [critical] review_on: [high]
per-severity (fail / total):
critical 0 / 0 <- blocking
high 0 / 0 <- review
medium 6 / 6
low 0 / 34
failing cases (worst severity first):
- medium c005 [verbosity]
- medium c012 [verbosity]
- medium c018 [verbosity]
- medium c023 [verbosity]
- medium c031 [verbosity]
- medium c037 [verbosity]
decision: REVIEW -- flat pass-rate 85.0% is below ship_threshold 90.0%
Exit 1, but REVIEW, not BLOCK, and the reason names the aggregate, not a severity class. This is the third state doing real work. The gate did not throw the pass-rate away; when nothing dangerous failed, the aggregate is exactly what it falls back on. REVIEW is where a low mean lands. BLOCK is reserved for the failure class you said must never ship. Put this run next to the killer and the tool's shape is clear: it answers to the distribution of severity when severity is present, and to the mean when it is not.
One boundary a careful reader will poke at. Does BLOCK fire on the presence of a critical case, or on a critical failure? It has to be the failure, or you could never ship a suite that tests a critical path at all. edge_run.json
has a critical
-severity case that passed (your PII redaction test ran and came back clean) and one unrelated low
failure.
$ python3 severity_gate.py fixtures/edge_run.json
SEVERITY-GATE REPORT
cases: 20 pass: 19 fail: 1
flat pass-rate: 95.0% (19/20)
ship_threshold: 90.0% block_on: [critical] review_on: [high]
per-severity (fail / total):
critical 0 / 1 <- blocking
high 0 / 0 <- review
medium 0 / 0
low 1 / 19
failing cases (worst severity first):
- low c015 [typo]
decision: SHIP -- no blocking or review-class failure, pass-rate 95.0% >= ship_threshold 90.0%
Exit 0. SHIP. Note the per-severity line: critical 0 / 1
— one critical case exists, zero critical cases failed. A passing test on a critical path is exactly the signal you want; blocking on it would punish you for having good coverage. The gate blocks on critical
fail, not on critical
present.
The default policy blocks on critical
and reviews on high
. That default is mine, and you should not keep it without thinking. The mapping from a failure category to a severity class, and the choice of which classes stop a deploy, is a decision your team owns, not something a linter should hand you. So the policy is a file you pass in. Here is fixtures/strict_policy.json
— four lines that also block on high
and send medium
to review:
{
"ship_threshold": 0.90,
"block_on": ["critical", "high"],
"review_on": ["medium"]
}
Point the same ship_run.json
at it:
$ python3 severity_gate.py fixtures/ship_run.json fixtures/strict_policy.json
SEVERITY-GATE REPORT
cases: 40 pass: 37 fail: 3
flat pass-rate: 92.5% (37/40)
ship_threshold: 90.0% block_on: [critical, high] review_on: [medium]
per-severity (fail / total):
critical 0 / 0 <- blocking
high 0 / 0 <- blocking
medium 1 / 1 <- review
low 2 / 39
failing cases (worst severity first):
- medium c010 [output_format]
- low c020 [pii_leak]
- low c030 [pii_leak]
decision: REVIEW -- 1 failing case(s) in a review severity class (medium)
Exit 1. Same run, same 92.5%, but under a policy that treats a medium
format miss as review-worthy, the verdict is REVIEW. The tool did not decide that a format miss matters. You did, in the policy file. The gate just applied it the same way every time.
I did not come up with this from nothing. On July 5, Ethan Walker published a Dev.to post whose title is the whole argument in one line: a 94% pass rate hid a PII leak in 6 test cases. His numbers, from his run, stated as his: 512 test cases, 31 failures, and 6 of those were a PII leak. His line, which I checked against the post before quoting: "Six out of 512 is a rounding error against a flat pass-rate metric." He also names the tools by hand — DeepEval, Promptfoo, LangSmith — and writes that they "all give you this by default; none of them force severity weighting on you."
Those figures are his and none of the counts in my runs above are. I built fixtures of a different size on purpose so the two never blur: my numbers are 40 / 3 / 2 at 92.5%, computed by my tool on my synthetic files; his are 512 / 31 / 6 at 94%, from his own eval suite. The tool in this post is one narrow, runnable answer to the thing his title describes: the point where a small critical cluster hides inside a green aggregate.
The general form of it was put well the same week by another writer, posting as aiexplore369zoho, under the heading the mean is lying to you. Their framing: "A benchmark score is a mean." Reliability, they argue, is a tail statistic, not a central one; the mean can stay flat while the failure rate on a critical slice doubles. That is the concept. Naming the slice that must not fail and gating on it directly, instead of on the average that buries it, is one way to act on it.
A fair objection: skip the labels, put a stronger model in the loop as an inspector, and let it catch the critical cases. Someone tested exactly that. Posting as zxpmail, they ran three models as agent quality inspectors and reported the opposite of a free lunch: the stronger the model, the more valid work it rejected. Their strongest model reached 0% false positives on garbage but falsely rejected 3 of 4 perfectly valid outputs. Those are their measurements, not mine.
The takeaway I draw for a gate is narrow. A probabilistic inspector trades one error for another and gives you a different answer on reruns. A gate over labels you already committed to is deterministic: same input, same verdict, every run. That determinism is not a nice-to-have here; it is the property the SHA hashes at the end are there to prove.
This is a spoke on the pre-execution gate for AI agents cluster, and its object is the deploy decision over a finished eval run. The neighbors ask adjacent questions, and the differences matter:
I would rather undersell this than have you deploy it as something it isn't.
fail
is really a fail or a pass
is really a pass. It finds no new failures. Mislabel a critical failure as low
and it ships; that is the GIGO limit, and it is real. The tool re-weights failures you already found and classified. It decouples the ship decision from the scalar mean. It does not discover anything.critical / high / medium / low
and the choice of what blocks are a file you own. The tool ships a default so it runs out of the box; the default is a starting point to argue with, not a standard.One design choice I am still not certain about: REVIEW and BLOCK share exit 1, and only the verdict text tells them apart. I did that so a CI job has a clean "0 means auto-ship, non-zero means a human decides" contract, with exit 2 reserved for "I could not even read the input." If your pipeline needs to branch differently on REVIEW versus BLOCK, you would want three non-zero codes, and I could see arguing it either way. I went with the two-state contract because it matches the fail-closed shape of the other gates in this series, but I would not fight hard for it.
A gate that crashes into a green is worse than no gate. Point it at a run with a severity label it does not recognize, and it refuses to decide rather than guess:
$ python3 severity_gate.py fixtures/bad_run.json
ERROR: case c002 has severity_class 'catastrophic', not in severity_order ['low', 'medium', 'high', 'critical']
$ echo $?
2
No path argument, a missing file, an unparseable JSON, a case missing verdict
, an unknown severity value: all exit 2, distinct from the exit 1 a REVIEW or BLOCK returns, so your CI can tell "the gate says hold" apart from "the gate could not run." I ran each scenario twice and hashed the full STDOUT both times, on Python 3.13.5, offline: ship_run
is cca7a591...
, block_run
is 346ee92c...
, review_run
is 6a27e17f...
, edge_run
is 99f4e8dc...
, and ship_run
under the strict policy is 35f9f6bb...
. Identical across both runs, every time.
Here is the one I do not have a good number for. For teams shipping an agent behind an eval suite: how many of your failing cases are labeled with a severity at all? Not "do you have an eval," which most people say yes to, but whether the pii_leak
and the trailing-whitespace failure carry different weights when the deploy decision gets made, or whether they both just dent the same pass-rate. My guess is that most suites are still gating on the flat number and the severity field is empty, but that is a guess, and I would like to know if I am wrong.
If this was useful, follow along here for the next runnable gate in the series, and tell me in the comments where a green aggregate has hidden a failure that should have blocked your deploy. I read every one.