A green test suite proves less than you think

A developer discovered that a green test suite for a routing agent was passing for the wrong reason, providing false confidence. The test reused the same task string, making the new capability structurally unreachable, yet the assertion passed due to a weak check. The developer argues that line coverage alone is insufficient for autonomous systems, which fail in seams between functions, and advocates for a multi-dimensional testing approach including integration, adversarial, concurrency, fault injection, and property-based testing.

The test that scared me was the one that passed. I had an integration test for a routing agent, the kind that takes a task and picks a capability to handle it. The test registered a new capability at runtime and then checked that the router would eventually route to it. Green run after run. Solid. I trusted it. Then I read it properly. It reused the same task string on every iteration of the loop. My scorer was deterministic by design, it hashed the task and indexed into the capability list, so a fixed string mapped to a fixed slot, and the newly registered capability lived at a different slot that the fixed string could never reach. The test asserted that the new capability got selected. The new capability was structurally unreachable. And the assertion passed anyway, because the loop happened to land on something registered every time, which was all the weak version of the check actually demanded. The test was not testing what it said it was testing. It was green for a reason that had nothing to do with the thing I cared about. The fix was almost insulting in its smallness, vary the task strings so the hash spreads across every slot including the new one, and suddenly the test could fail when the feature was broken, which is the entire point of a test. One line. I had been shipping false confidence behind a checkmark. That is the moment this whole piece is about. Not the bug. The checkmark. Here is the setup that produces this every time. You build an agent. You write unit tests. You watch line coverage climb to ninety-something percent. CI turns green. You deploy. And within a week the thing is making nonsensical decisions under load, falling over on inputs you never imagined a user would send, and getting stuck in loops your loop detector cannot see because two threads stepped on each other's state at the same instant. The unit tests were not lying to you. The functions genuinely worked in isolation. That is the trap. Line coverage measures whether your tests executed a line, not whether they cornered it. You can run every line in a file and assert nothing that matters about any of them, exactly like my integration test ran its loop and asserted the wrong thing. A green suite built on coverage tells you your tests touched the code. It tells you almost nothing about whether the code survives contact with production. And autonomous systems, agents that route, retry, fall back, remember, do not fail in isolated functions. They fail in the seams between functions. They fail where two modules meet and disagree about a type. They fail on the input the author never pictured. They fail when two requests arrive at once. They fail when a dependency dies and the system panics instead of limping. They fail on the edge case nobody wrote down. Coverage walks straight past all five, because every one of those failures lives in territory a unit test is structurally built to avoid. The shift that changed how I test agents was to stop asking "did my tests run the code" and start asking "what are the distinct ways this system actually breaks, and do I have a suite aimed at each one." Five answers came back, and they are genuinely distinct failure classes, not five flavors of the same check. None of these dimensions is mine to claim as an invention, they are long-standing testing practice, integration testing, adversarial and fuzz testing, concurrency testing, fault injection, and property-based testing each have decades of prior art behind them. The engineering distinctive is narrower and more honest, it is recognizing that an autonomous agent needs all five aimed at it at once, because it can fail in all five ways in a single week, and that a coverage number cannot stand in for any of them. The first seam is integration , where modules compose. The most common bug in a multi-module system is not "function X has wrong logic," it is "X works fine but Y expected a different type," or "A only works if B was set up first." Mocks paper over exactly this, they return what you told them to and never enforce the real interface, which is how my same-string test slept through a real defect. The second is adversarial input , the gap between the task you imagined and the task a real user sends, the hundred-thousand-character string, the embedded newline carrying a fake directive, the injection attempt, the empty string, the wall of emoji. The contract is not that nothing weird arrives. It is that weird input gets a safe answer or an honest error, never a crash and never a leak. The third is concurrency , the races that only appear when many requests hit shared state at once. A history list, a registry, a loop detector, anything two threads can write without a lock, will silently corrupt under load in a way no single-threaded test will ever reproduce. The fourth is failure cascade , what happens when the pieces an agent depends on, the registry, the scorer, the loop detector, start dying. A naive build lets any one failure crash the whole call. A real one degrades, and the failure you actually have to test is all of them dying at once, because real outages are correlated and take down several things together. The fifth is property-based testing , where instead of writing examples you state an invariant and let a generator hunt thousands of inputs for the one that breaks it. The invariants that look obvious, "routing always returns a real capability or a clean error, nothing in between," are exactly the ones a generated single-character task or a Unicode combining sequence quietly violates. No single one of these dimensions catches everything, and that is the whole argument. Integration finds the type-contract and setup-order bugs and is blind to races. Adversarial finds the injection and the boundary crash and never sees a component failure. Concurrency finds the race and ignores the malformed input. Failure-cascade finds the un-graceful crash and says nothing about invariants. Property-based finds the violated invariant and the boundary input and cannot see a thread race. The value is not in any one suite. It is in the combination, five independent nets under the trapeze, each catching what the others structurally drop. Each suite is its own short read, linked below, with the one specific failure it exists to catch and the cheapest test that catches it. I am not going to pretend this makes a system bulletproof. It does not. There are failures none of these five see, and there will be a sixth seam I learn about the hard way at 2am some night that is already coming. But the difference between the green checkmark before and the green checkmark after is the difference between a number and a sentence. Before, green meant "the tests ran the code." After, green means "this composes correctly, handles hostile input without leaking, holds together under concurrent load, degrades instead of crashing when its dependencies die, and keeps its invariants across inputs I never thought to write down." That is a checkmark worth trusting. The first one was just a string that happened to match. This is the hub for a five-part series, one short read per dimension: