I don't write test suites for a living. My job is grading them, and grading the AI models that write and judge code in the first place. The rubric I work under does not allow ties. Every verdict needs a named file, a line number, a concrete before and after. "This one feels cleaner" is not a reason, it is a rejection.
Sitting in that seat long enough teaches you something the rest of the industry mostly avoids saying out loud: the green checkmark was never proof of anything. It was always a claim, and I have watched that claim fail the same way in three completely different settings.
Flaky tests are the most visible version of this. Google's own engineering team reported in 2016 that around 16 percent of their tests displayed some flaky behavior, and about 1.5 percent of all test runs across their corpus came back flaky on a given day, meaning roughly one in seven tests written by their engineers would occasionally fail for reasons unrelated to any real code change.
The standard fix, a retry count in the CI config, does not remove the underlying bug. It just changes the odds. A test with a genuine 25 percent failure rate will statistically pass on the order of 99.6 percent of the time once you allow three retries. The bug is still there. The pipeline just stopped telling you about it.
A 2014 study by Luo, Hariri, Eloussi and Marinov analyzed 201 real flaky test fixes across 51 open source projects and found the leading causes were concurrency, async waits, and test order dependency, not randomness. These are deterministic bugs that only manifest under specific timing. That is exactly why Carnegie Mellon's PASTA lab built Fray, a tool that deliberately controls thread scheduling to force the bad interleaving instead of hoping a rerun stumbles into it by chance.
Sources: https://www.googblogs.com/flaky-tests-at-google-and-how-we-mitigate-them/ and https://mir.cs.illinois.edu/lamyaa/publications/fse14.pdf You do not need an exotic system to see this one. You need a test method with five assertions and no messages. When it fails, you find out something broke, not what, and the rest of the method never even runs to tell you.
In 2015, researchers at UCL and Sheffield gave the broader version of this a name. Barr, Harman, McMinn, Shahbaz and Yoo published "The Oracle Problem in Software Testing: A Survey" in IEEE Transactions on Software Engineering, laying out why deciding whether an observed behavior is correct is often the harder half of testing, harder than generating the input that triggers it. When no specification or contract can make that call automatically, the job falls back to a human eyeballing the result. The paper has been cited well over a thousand times since and is still the reference point a decade later, which tells you how far the field has actually moved.
The everyday version of this has a name too. Assertion Roulette, a pile of asserts with no message telling you which one actually failed, is one of the original test smells catalogued in the early 2000s. An empirical study of eighteen open source Java projects found it present in roughly 62 percent of JUnit test classes, the most common smell in the codebases studied. A separate large scale study of open source Android apps put the figure above 50 percent as well.
Sources: https://www0.cs.ucl.ac.uk/staff/M.Harman/tse-oracle.pdf and https://www.cs.loyola.edu/~binkley/papers/icsm-12-smells.pdf Here is where my actual day job comes in. Every coding agent I evaluate starts each session with no memory of the last one. No recollection of the convention it was corrected on last week, the mistake it already made once, the file it already touched. So it re-derives from scratch, sometimes reintroduces a bug it already fixed, and still reports the run as a success.
This is not a training failure, it is an architectural fact. Recent survey work on agent memory, including a 2026 paper on codified context infrastructure for large codebases, makes the same point every practitioner in this space eventually rediscovers on their own: a stateless model, however capable in the moment, cannot carry forward what it learned last session without an explicit memory layer built for that purpose. That is the actual reason informal conventions like a persistent project context file have caught on across agentic coding tools. Nobody solved statelessness. Everyone independently hit the same wall and built the same workaround.
It gets more interesting once a second AI enters the loop as the judge. Recent evaluation reliability research shows that even when multiple LLM judges review the exact same output side by side, their agreement with each other, and with human raters, is inconsistent enough that researchers now argue standard inter rater metrics understate how shaky the judgment actually is, given the randomness baked into the judge itself. Which is exactly why the rubric I work under bans ties and demands a cited line number for every call. Not because the rule is elegant, but because without it, "verified" quietly turns into "I skimmed it and it looked fine."
Sources: https://arxiv.org/pdf/2602.20478 and https://arxiv.org/html/2412.12509v2 A CI pipeline, a JUnit test class, and an AI evaluation run look nothing alike on the surface. Underneath, they fail the same way. Each produces a single bit of output, pass or fail, good or bad, and each lets that bit stand in for a judgment nobody actually made carefully. Retries hide it in the first. Unlabeled asserts hide it in the second. Statelessness makes the third worse, because a system with no memory of what it already got wrong has no way to notice it is repeating itself.
Not a smarter checker bolted on after the fact. The thing that has actually moved the needle, in my experience grading this exact failure mode daily, is refusing to accept "it passed" as a complete answer to "how do you know." Require the file. Require the line. Require the reviewer, human or model, to show their work instead of returning a verdict. It is slower. It is also the only version of green that means anything.