This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
This is the story of a bug with two layers: a crash we correctly decided not to fix — and the verdict-flipping second bug hiding behind it that we missed for a week, even though the fix for it already existed, sitting unread on disk with a note that literally said "trust this file over the exit status."
I build an AI agent harness — a control system around a language model where every claim needs receipts and every safety gate is enforced by code. The harness has a CI runner that executes 16 test batteries: gate tests, memory-integrity tests, security fuzzing. If a battery fails, the chain refuses to let changes land. The whole point of the system is that a verdict you can't trust is worse than no verdict — remember that, it's the punchline.
One battery — the harness security suite — had a ritual. Every single run:
=== 11 passed, 0 failed ===
Assertion failed: !(handle->flags & UV_HANDLE_CLOSING), file src\win\async.c, line 76
All eleven assertions green… then the Node process aborts in libuv teardown. Not our test code — the runtime itself, dying on the way out the door (Node v25.9.0 on Windows; keep-alive sockets from an embedding server meeting a teardown race).
Here's the part I'm actually proud of, because it's the unsexy discipline part: we tried to fix it and measured the attempts honestly. Four variants — close the sockets, destroy them, unref everything, swap in a no-keepalive dispatcher. Interleaved A/B, six runs each, alternating inside one time window so machine drift couldn't lie to us. Result: the targeted fix crashed 3/6 runs. Doing nothing crashed 0/6 that window — and the same untouched code had crashed 8/8 twenty minutes earlier. The environment dominated any code change we tested. Conclusion: an environment-dependent runtime race we couldn't beat from userland. So we did the honest thing: taught the CI to report CRASH
(assertions green, process died in teardown) as its own state, distinct from FAIL
. Documented, bounded, accepted.
That was the right call. And it concealed the real bug for a week.
Fast-forward a week: the same battery keeps "failing." Once it even blocked the commit chain with do NOT commit
. The very next run: green. Same code. Same machine. The operator instinct kicked in: "it keeps failing — figure out why, fix it, make sure it doesn't happen again."
The classifier's logic seemed sound: parse the battery's stdout for N passed, M failed
; if the process crashed but the counts look green, classify CRASH
(report it, don't block); if it crashed and there are no counts, assume it died mid-run and classify FAIL
(block everything).
See it yet? The abort and the final console.log
are in a race. When the log flushes first, the classifier sees "11 passed, 0 failed" → CRASH
→ the chain continues. When the abort wins, that last line never leaves the buffer → no counts → FAIL
→ "do NOT commit." Same crash, coin-flip verdict, decided by flush timing nobody controls. Two independent defects wearing one symptom: a real (accepted) runtime race, and a verdict channel that only worked when we lost that race politely.
While reading the battery's source to fix the classifier properly, I found this — written a week earlier, during the original investigation:
fs.writeFileSync(path.join(os.tmpdir(), 'harness-sec-result.json'), JSON.stringify({
ts: new Date().toISOString(), pass, fail,
verdict: fail ? 'FAIL' : 'PASS',
note: 'exit code may be from a libuv teardown race AFTER all assertions complete; trust this file over the exit status',
}));
The battery had been writing a crash-proof result file the entire time — a synchronous write that always survives the abort — with a note begging future readers to trust it over the exit code. And the CI runner never read it. We built the antidote and never wired it to the patient. The fix for the week-old mystery existed before the mystery did.
Make the sidecar a contract. The runner hands every battery a path; the battery writes its counts there synchronously before exiting; the sidecar is authoritative, stdout is just the fallback for batteries that don't have one:
const sumFile = path.join(os.tmpdir(), 'battery-summary-' + rel.replace(/[^a-z0-9]/gi, '_') + '.json');
const r = spawnSync(process.execPath, [abs], { ...opts, env: { ...process.env, BATTERY_SUMMARY_FILE: sumFile } });
let sidecar = null;
try { sidecar = JSON.parse(fs.readFileSync(sumFile, 'utf8')); } catch {}
// sidecar counts win; stdout parsing only when no sidecar exists.
// A verdict must never ride on a flush race.
And because a classifier is an instrument, we validated it like one — with a known-good and a known-bad control through the shipped code, not a copy: a fake battery that writes a green sidecar, prints nothing, and exits non-zero (the old code called this FAIL and blocked; the new code must classify CRASH and exit 0) — and a second whose sidecar reports real failures (must still FAIL and still block). Both behaved. Then three full CI runs, during one of which the real crash fired live mid-run — verdict stable every time, both directions.
Disclosure, proudly: I'm self-taught (April 2026 → now) and this hunt — like everything here — was me and my AI partner working the problem against our own production tooling. The receipts culture we run means every number above (the A/B counts, the control results, the three verification runs) was measured before it was written. The longer story of how we work is in my first post.