A bug report lands: one file with a very long expression crashes the whole scan, with no file name and no summary. You add a guard, write a test with a file that crashes, watch it pass, close the ticket.
How do you find out that the guard has a hole? Not from the test. The test is the exact file from the report, and the guard was shaped around it.
That was my situation last week with a CLI that reads a codebase and lists the analytics events it can send, and the small SDK next to it that sends them. Both are on npm, both had a few hundred tests, and I was fairly sure they were in decent shape. So I asked for a different kind of check: three reviewers who had never seen my code or my reasoning, each starting from the README and the published packages, installing them into empty folders the way a stranger would, and trying to make them misbehave. They were AI agents, which is the only way I could afford three of them for a whole afternoon, and I told them to distrust my tests and my changelog.
I did it three times. After each round I fixed what they found and sent a fresh set at the new build.
The first round found what you'd expect from a tool that grew by examples. A script on Node 18 threw on every track() because that version has no global crypto. An SDK flush() returned instantly when another flush was already running, so a serverless function could end before anything was delivered. An object I'd passed in, then changed later, could make a background send crash the process. A broken value in the browser's saved queue made the constructor throw on every page load. On the CLI side, a template-heavy Astro page killed the whole scan, and with no plugin installed a run quietly overwrote a catalog of 68 events with 2.
Boring, real, fixed in an evening. The count of findings the reviewers rated high was 6.
The second round is the reason for this post. Two of the three high findings were things I had added in round one.
The exit that wouldn't happen. A Node script that ends with events still queued should send them and go away. I'd noticed that Node fires beforeExit every time the event loop drains, and a script can track more events after the first drain, so I made the SDK listen again each time. It felt careful. Then a reviewer pointed an SDK at a server that always answers 500 and let the script end normally. With a one-second shutdown budget, the process stayed alive for 44 seconds. With the default ten seconds, 67. Every time the loop drained, the SDK tried again, failed again, and drained again. My careful change turned "give up quietly at exit" into "retry until the circuit breaker opens".
The fix was one condition: listen again only after a flush that delivered everything. The reviewers' next round found a second route into the same problem, a timer that started its own flush behind the exit flush and ran without the exit limit. I had fixed the symptom I was shown and not the property I cared about, which is that exit is bounded.
The stack overflow I "fixed". The original crash was a stack overflow in my own code walking a file with a very long "a" + "a" + ... chain. I rewrote the walk to use an explicit stack, added a guard that skips files nested too deeply, and exempted left-leaning binary chains from the depth count, because the TypeScript compiler handles + chains in a loop and I'd tested 12,000 terms fine. A reviewer wrote the same shape with &&: 800 operands, whole scan dies, no file named. TypeScript's binder recurses on logical operators. My exemption said "binary chains are flat", and that was true for the one operator I had tried.
The honest version of that lesson is small. I generalized from one case, and the test I wrote for the guard checked only the case that had already crashed.
High-severity findings in round two: 3.
By the third round the reviewers rated nothing high, and two of the interesting findings were again mine. One is worth telling because it's about a dependency.
The Astro plugin hands your page to Astro's own compiler, which is a Go program compiled to WebAssembly. On big pages it runs out of memory and dies hard, taking whatever else was running with it. After round one I'd added a check that looks at a page's markup first and refuses to feed it the pages likely to kill it. In that check I counted a missing end tag the way HTML does. In HTML, <li>a<li>b is two sibling items, because the end tag of li is optional.
A reviewer built a page of nested lists written that way and it ate 4.2 GB and four and a half seconds before the compiler gave up. My check had waved it through.
I stopped guessing and ran the compiler on small versions of the same markup, printing the depth of the tree it returned. 600 unclosed <li> in a row came back as a tree 602 levels deep. The compiler doesn't close them. It nests each item inside the previous one. 150 of the nested-list blocks came back 753 levels deep. My assumption about the spec was correct about HTML and wrong about the tool I was calling, and I had even written it as a comment.
Fixing it meant counting an unclosed tag as nesting, which makes the page look deeper than a browser would see it, which is what the compiler actually does. The reviewer's page is now refused up front, and the file is read without its template and reported as read only in part. The shape of that page is a test now.
| Round | Findings (bugs and docs that disagreed with the code) | Rated high | Came from a fix of mine |
|---|---|---|---|
| 1 | about 50 | 6 | not applicable |
| 2 | about 40 | 3 | 2 of the 3 |
| 3 | about 35 | 0 | at least 2 |
The totals are rough because I merged a few duplicates by hand, but the shape is right: the count fell slowly, and the severity fell fast. The CLI now has 1,359 unit tests and the SDK 245, roughly 200 and 75 more than before the first round. That is not why they got better. What changed things was that each round added a person who didn't share my picture of what was normal.
I'd keep the rule of pointing the next review at the fix, not at the original bug. A fix is new code written under pressure, with one example in mind, and nobody has read it yet. Mine produced bugs at a rate no worse than my first drafts.
I'd write a guard's test from the other side. A limit like "refuse anything nested more than N" gets tested with a file below N that works and a file above N that is refused, and nobody tests the file that crashes you at 0.8 N because nobody has met it yet. Now when I add a guard I try a few different shapes at about 60 percent of the limit before I trust it.
And I'd measure any third-party behavior I rely on, even when I'm sure. Two minutes with a probe script beat a day of reasoning from a spec that the tool didn't follow.
What I'd skip is trusting my own count of tests as a quality signal. Every one of these bugs lived in a codebase with a green suite, because every test came from the same head as the code.
If you want to look at what the tool says about your own repo:
npm install -D @eventra_dev/eventra-cli
npx eventra init
npx eventra sync
The changelog lists what changed in this release. If it misses something in your code, or crashes on a file shaped in a way I haven't thought of, I would like to see that file.