# My CI Was Red for 255 Commits and Nobody Looked > Source: > Published: 2026-10-21 07:00:00+00:00 A CI job failed on every single push to my release branch. Including its very first run. For 255 commits. Nobody filed it. Nobody bisected it. I merged to `main` with it red, because by then "detector-eval is red" was weather — a thing that was true, that had always been true, that didn't require an explanation. It had never been green. Not once. And I didn't know that, because I'd never thought to ask. This is from the [agentwatch](https://github.com/agentic-suite/agentsec-agentwatch) v0.2.1 release — the agent-security tool in [agentic-suite](https://github.com/agentic-suite). The receipts are in the [v0.2.0 field test report](https://github.com/agentic-suite/agentsec-agentwatch/blob/main/docs/field-test/v0.2.0/FIELD_TEST_REPORT.md) and the release-readiness review. The workflow is small. It was added as part of the detector-credibility milestone — build trust in the anomaly detectors by running them over a versioned corpus in CI: ``` - name: Detector eval corpus v0 (DET-1) run: python scripts/detector_eval.py --json ``` Every push to the branch, and every PR, ran that. Every run ended like this: ``` FileNotFoundError: [Errno 2] No such file or directory: '.../services/analytics/data/detector-corpus-v0.json' Process completed with exit code 1. ``` The script has a default corpus path baked in: `services/analytics/data/detector-corpus-v0.json`. That file exists on my machine. It does not exist in a fresh clone. The root `.gitignore` has this rule, added with the comment `# Local dev data`: ``` data/ ``` Scratch files, local runs, throwaway output — a reasonable rule. And it matched `services/analytics/data/` with the same two-line indifference it matched everything else. The corpus — the *input to a CI gate* — was excluded by a rule meant for junk. The part that stings: **the generator was sitting in the repo the entire time.** `scripts/generate_detector_corpus.py`, one directory over from the workflow file. The corpus is a buildable artifact, not a hand-maintained one. Two fixes were available — generate it in the workflow, or commit it — and I had a third option, which was the one I actually took for 255 commits: nothing, because I never classified it as a problem. Here's the property that made this survivable for so long, and it's the property I want to name because I think it's the general failure: **The workflow's first-ever run was red.** Not red *after* something regressed. Red from the first commit it existed on, on every push, forever. Which means nobody ever saw it turn green, which means nobody ever saw it turn *meaningfully* red. There was no delta. No before-and-after. The check had no authority because a check's authority comes entirely from "this was green, and now it isn't." I went back and counted. The branch carried **255 commits** at merge, and the workflow existed for essentially all of them: ``` bash $ git rev-list --count 31100ba..a673eef 255 ``` Every push in that window — the full field test, the detector hardening, the release prep — shipped with that red X on it. It wasn't alone. `security.yml` was failing too, in zero seconds, with GitHub's terse "This run likely failed because of a workflow file issue." Two red checks on every push, both meaningless, both ignored. And this is the part I keep turning over: **while both were red, a real regression in either would have been invisible.** The noise didn't just cost attention. It cost the *signal* the checks existed to carry. I ran the release-readiness review with a simple rule: enumerate every red on the branch and demand an explanation for each. Not "is it red," but "*why* is it red, and is that reason acceptable." The answers were: | Check | Why red | Verdict | |---|---|---| | `detector-eval` | reads a gitignored corpus; never had the input | broken on arrival — fix it | | `security.yml` | workflow file issue, fails in 0s | broken on arrival — fix it | No regressions. Both broken *checks*. Which is the quiet hazard of CI noise: I'd been reading both as "flaky, probably fine" for weeks, and the truth was "structurally incapable of passing." tip: "It's always been red" is the most expensive sentence in CI. It means no one has ever seen the check do its job, so no one can tell when it stops doing it. Two lines of `.gitignore` and one commit: ``` !services/analytics/data/ !services/analytics/data/*.json ``` Whitelist the corpora, then commit them — `detector-corpus-v0.json` and the LLM corpus beside it. They're synthetic, shape-authored test fixtures; they're secret-scanned like everything else; there was no reason they shouldn't ship with the repo. The alternative — calling the generator inside the workflow — is equally fine. What mattered was that the input became *reviewable, versioned state* instead of ambient local files. The receipt, from `main` after the fix: ``` completed success detector-eval main 52s completed success detector-eval main 49s completed success detector-eval main 51s completed success detector-eval main 48s ``` Green. Deterministic. And now — for the first time — *capable of going red for a reason*, which is the entire value of the check. `security.yml` is honestly carried with an owner and a known cause now — but to The `security.yml` one is still open on `main` as I write this. Still 0s, still failing. But it's not weather anymore — it's in the review log with a line number and a name, and that difference is the whole point. All of this shipped with [agentwatch v0.2.1](https://github.com/agentic-suite/agentsec-agentwatch/releases/tag/v0.2.1) — [on PyPI](https://pypi.org/project/agentsec-agentwatch/0.2.1/). The question worth asking of your own repo tonight: which of your checks has the longest red streak, and does anyone know *why* it's red — or just that it is? `gh run list` will tell you in about a minute. That minute is the cheapest audit you'll run all week. `detector-eval.yml`` detector_eval.py``generate_detector_corpus.py``.gitignore` corpus whitelist — the corpora now ship in `services/analytics/data/`