AI coding agents have to a great extent changed who writes the tests, so much so that the engineer approving a pull request has often written neither the code nor the tests that came with it. But most teams still read a passing suite as though someone with a working model of the system decided what each test should check.
Consider an agent that writes a discount calculation and, in the same session, a set of tests for it. The tests call the function, capture what it returns and assert that value, so they pass on the first run and on every run after. If the discount is applied twice on orders above a threshold, the tests now encode that bug as expected behavior, because the only reference they had was the code they were written against.
That gap was always possible, but it stayed rare while the person writing a test had usually written or reviewed the code and carried its intent in their head. Vibe coding takes that person out of the loop. The knowledge that used to reach the suite through them, like the edge case someone hit two years ago or the reason behind an odd design choice, never reaches it. Coverage still counts the lines a test executed, so a suite can look complete on the dashboard while asserting almost nothing about intent.
Every test suite rests on an assumption so basic that nobody ever wrote it down, which is that the author of a test knows how the system is supposed to behave.
Sai Krishna, Director of Engineering at TestMu AI, an AI-native software testing platform formerly known as LambdaTest, sees that assumption failing across the enterprise teams he works with. “Vibe coding breaks a very basic assumption in testing,” he explains. “The person writing the test understands how the system is supposed to behave. When that assumption does not hold, the tests cannot just be about verifying the intent because the intent itself is not very clearly defined.” What replaces intent is observation. The agent tries something, the test records the result, and both move on without either knowing whether the result was correct. Krishna’s check for teams is to take a recently generated test and ask whether it would still pass if the underlying behavior were wrong, because a test that only records output will pass either way.
That question can be answered mechanically. Mutation testing tools such as PIT for Java or Stryker for JavaScript and .NET introduce small deliberate faults into the code, like a flipped comparison or a changed constant, and report which tests still pass afterward. A test that survives a mutation was never checking the behavior it covers, and running a mutation pass over a generated suite usually shows how much of it is description. I made a related case on DevOps.com about AI testing frameworks that pass tests they should be failing (https://devops.com/your-ai-testing-framework-might-be-passing-tests-it-should-be-failing/), and surviving mutants are the most direct evidence of that problem.
None of this shows up in the coverage report, because a test asserting the wrong value executes exactly the same lines as a test asserting the right one.
Krishna is blunt about what that does to the number most dashboards lead with. “Hundreds of test cases do not mean that you have good coverage,” he explains. “Numbers do not mean it is going to have good coverage. That is the reality in today’s world.” The pattern he sees is a team with thousands of tests and high coverage that keeps getting incidents from code the suite supposedly covers. His recommendation is to stop reporting test count as a quality metric, and to track where incidents originate against what the suite claims to cover.
Making that comparison is mostly a tagging exercise. If every production incident records the module or service where the fault was, the incident list can be laid next to the coverage report. Any module with high coverage and repeat incidents is where the suite is asserting behavior and not intent. Adding the mutation score for each module gives leadership a second number that actually moves when test quality changes, which test count never did. But the harder fix is upstream of both numbers, in where the intent is supposed to come from.
Every engineering organization runs on knowledge nobody wrote down, and for decades that knowledge reached the test suite through the people who wrote the tests.
Krishna made this point in a recent AWS Summit talk. “The knowledge can no longer be tribal,” he explains, and “it has to be more explicit in systems, through tests, data, and checks.” His advice is to inventory the assumptions the suite depends on but never states and encode each one as a check the agent has to satisfy, before an incident forces someone to rediscover it.
In a repository that usually takes two forms. Invariants become property-based tests, written with tools like Hypothesis or fast-check, that assert a rule holds across generated inputs, such as an order total never going negative. Business rules that cannot be expressed as properties go into a short, versioned file that the coding agent reads as context before it writes anything. I wrote earlier this year about why AI agents need a reliable context layer more than they need more instructions (https://techstrong.ai/contributed-content/ai-agents-need-reliable-context-layer-not-more-instructions/), and test intent is among the most valuable things that layer can carry.
Writing the intent down solves half the problem, and Srinivasan Sekar, also Director of Engineering at TestMu AI, describes the other half. “Even if you were magically perfect in terms of creating model-written tests, they can still fail at two layers,” Sekar explains. “The first is the absence of an independent entity that evaluates what the model has written based on the context provided. The second is the execution layer itself, where tests do not map cleanly to the messy realities of mobile environments, browser environments, or test data.” In his experience, even well-prompted models produce only about 20 percent tests worth keeping, with the rest generated to fill the count. His recommendation is to put an independent evaluator between generation and execution.
Independence is the property that matters here. An evaluator running on the same model with the same context will approve the same blind spots the generator introduced. So the check needs a different model, deterministic rules or a human reviewer who decides which generated tests get promoted into the suite. The execution layer needs the same honesty, which means running against real browsers and devices with realistic test data, never against mocks that make every environment look clean. With an evaluator in place, the remaining question is what the surviving tests should assert.
Once nobody has a reliable model of correct behavior, precise assertions become risky, because each one encodes a belief about the system that nobody has checked.
Krishna recommends moving toward invariants, critical user journeys and comparisons against known good behavior, so a test validates that behavior falls within a range defined by what is known to be correct. That requires a ground truth dataset, which most teams have not built. His advice is to start it from the few journeys that carry real business consequence, without trying to cover the whole system at once.
For a checkout or login flow, that can be as simple as recording a verified run as a golden baseline and diffing every future run against it, with a human signing off whenever the baseline changes. The baseline only has to be right once to be useful, and it does not depend on anyone’s memory of how the flow is supposed to work. Some teams build that ground truth from evidence they already have. Naga Chand Putta, a senior software engineering leader in a highly regulated industry who builds LLM agents alongside the services they connect to, starts from the incident record. “Every production incident we have already gives us a verified example of behavior the system must never repeat,” Putta explains. “As AI writes more of the code, I want each of those incidents turned into a regression test before the postmortem is closed, because that test does not depend on anyone remembering what the system was supposed to do.” In a regulated environment, he adds, the link between an incident and the test that now guards against it also gives auditors a concrete record showing that a failure the organization already paid for cannot return unnoticed. His recommendation for teams that cannot fund a ground truth dataset upfront is to start from the incidents they have already survived and make a linked test part of closing each one.
Incidents are the one source of truth that does not depend on anyone’s assumptions, which makes them the cheapest place to start. The same idea, applied before an incident happens, is behind the assurance workflow in TestMu AI’s Kane CLI. It ingests requirement documents, has an agent propose use-cases that cite the exact lines they came from, and gives every acceptance criterion an oracle, meaning an expected result a machine can check. Nothing the agent proposes becomes trusted until a person reviews it. And when a source document changes, everything derived from the old version is marked stale, so the intent behind a test cannot drift away from the requirement without the suite showing it.
The fastest way to see where a team stands is to take the last twenty generated tests, run a mutation pass over the code they cover and write down the intent behind every test that survives. Each survivor marks a place where the suite has been recording what the code does, and each written intent is a piece of tribal knowledge that an agent can now read.
Credits for Commentary
Vibe-Coded Test Suites Pass Because They Assert Observed Behavior was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.