A CI job failed on every single push to my release branch. Including its very first run. For 255 commits.
Nobody filed it. Nobody bisected it. I merged to main with it red, because by then "detector-eval is red" was weather — a thing that was true, that had always been true, that didn't require an explanation.
It had never been green. Not once. And I didn't know that, because I'd never thought to ask.
This is from the agentwatch v0.2.1 release — the agent-security tool in agentic-suite. The receipts are in the v0.2.0 field test report and the release-readiness review.
The workflow is small. It was added as part of the detector-credibility milestone — build trust in the anomaly detectors by running them over a versioned corpus in CI:
- name: Detector eval corpus v0 (DET-1)
run: python scripts/detector_eval.py --json
Every push to the branch, and every PR, ran that. Every run ended like this:
FileNotFoundError: [Errno 2] No such file or directory:
'.../services/analytics/data/detector-corpus-v0.json'
Process completed with exit code 1.
The script has a default corpus path baked in: services/analytics/data/detector-corpus-v0.json. That file exists on my machine. It does not exist in a fresh clone.
The root .gitignore has this rule, added with the comment # Local dev data:
data/
Scratch files, local runs, throwaway output — a reasonable rule. And it matched services/analytics/data/ with the same two-line indifference it matched everything else. The corpus — the input to a CI gate — was excluded by a rule meant for junk.
The part that stings: the generator was sitting in the repo the entire time. scripts/generate_detector_corpus.py, one directory over from the workflow file. The corpus is a buildable artifact, not a hand-maintained one. Two fixes were available — generate it in the workflow, or commit it — and I had a third option, which was the one I actually took for 255 commits: nothing, because I never classified it as a problem.
Here's the property that made this survivable for so long, and it's the property I want to name because I think it's the general failure:
The workflow's first-ever run was red.
Not red after something regressed. Red from the first commit it existed on, on every push, forever. Which means nobody ever saw it turn green, which means nobody ever saw it turn meaningfully red. There was no delta. No before-and-after. The check had no authority because a check's authority comes entirely from "this was green, and now it isn't."
I went back and counted. The branch carried 255 commits at merge, and the workflow existed for essentially all of them:
$ git rev-list --count 31100ba..a673eef
255
Every push in that window — the full field test, the detector hardening, the release prep — shipped with that red X on it.
It wasn't alone. security.yml was failing too, in zero seconds, with GitHub's terse "This run likely failed because of a workflow file issue." Two red checks on every push, both meaningless, both ignored. And this is the part I keep turning over: while both were red, a real regression in either would have been invisible. The noise didn't just cost attention. It cost the signal the checks existed to carry.
I ran the release-readiness review with a simple rule: enumerate every red on the branch and demand an explanation for each. Not "is it red," but "why is it red, and is that reason acceptable."
The answers were:
| Check | Why red | Verdict |
|---|---|---|
detector-eval |
reads a gitignored corpus; never had the input | broken on arrival — fix it |
security.yml |
workflow file issue, fails in 0s | broken on arrival — fix it |
No regressions. Both broken checks. Which is the quiet hazard of CI noise: I'd been reading both as "flaky, probably fine" for weeks, and the truth was "structurally incapable of passing."
tip: "It's always been red" is the most expensive sentence in CI. It means no one has ever seen the check do its job, so no one can tell when it stops doing it.
Two lines of .gitignore and one commit:
!services/analytics/data/
!services/analytics/data/*.json
Whitelist the corpora, then commit them — detector-corpus-v0.json and the LLM corpus beside it. They're synthetic, shape-authored test fixtures; they're secret-scanned like everything else; there was no reason they shouldn't ship with the repo. The alternative — calling the generator inside the workflow — is equally fine. What mattered was that the input became reviewable, versioned state instead of ambient local files.
The receipt, from main after the fix:
completed success detector-eval main 52s
completed success detector-eval main 49s
completed success detector-eval main 51s
completed success detector-eval main 48s
Green. Deterministic. And now — for the first time — capable of going red for a reason, which is the entire value of the check.
security.yml is honestly carried with an owner and a known cause now — but to The security.yml one is still open on main as I write this. Still 0s, still failing. But it's not weather anymore — it's in the review log with a line number and a name, and that difference is the whole point.
All of this shipped with agentwatch v0.2.1 — on PyPI.
The question worth asking of your own repo tonight: which of your checks has the longest red streak, and does anyone know why it's red — or just that it is? gh run list will tell you in about a minute. That minute is the cheapest audit you'll run all week.
detector-eval.yml`` detector_eval.py``generate_detector_corpus.py``.gitignore corpus whitelist — the corpora now ship in services/analytics/data/