An LLM agent has zero access to that folklore. It sees a red bar and a mandate to make it green. It doesn't know which failures are real and which are random because it can't ask the team lead which tests are acting up this week. For an agent, every red signal is a directive to fix something, and every green signal is permission to stop. When a test is flaky, the reward signal is corrupted.
The very thing that makes an LLM agent powerful—the fact that it doesn't get bored, doesn't skip steps, and doesn't shrug—becomes a liability here. A human's ability to "shrug off" a failure is actually a critical filter. Without that filter, the noise hits the optimizer directly.
The Folklore Gap #
Most engineering teams have a shared mental map of their "unreliable" tests. New hires learn during onboarding that OrderConfirmationTest
needs to run alone or that the suite only passes after the database warms up. This collective memory allows us to distinguish between a legitimate defect and a system glitch.
An agent cannot make this distinction from inside the test suite. There is nothing in the logs that says "this failure is noise." If you try to fix this by adding these tests to an "ignore" list in the prompt or config, you've just created a permanent blind spot. An agent told to ignore TimezoneTest
will never detect a real timezone bug.
While humans can be inconsistently attentive—ignoring a flake today but catching a real bug next month—an agent is relentlessly consistent. If the rule is to ignore, it ignores forever. If the rule is to fix, it will spend three hours rewriting a perfectly fine function to satisfy a random race condition.
For anyone building a real-world AI workflow or deploying LLM agents to handle TDD, the priority isn't the prompt—it's the determinism of the test suite. Your tests aren't just for verification anymore; they are the API the agent uses to understand your code. If the API is lying, the agent is useless.
[Next Gate.cat: Stopping AI Agents from Running rm -rf →](/en/threads/2489/)