{"slug": "why-green-tests-are-not-enough-for-ai-coding-agents", "title": "Why Green Tests Are Not Enough for AI Coding Agents#", "summary": "A developer built the AI Coding Agent Reliability Kit, a Linux-first local workflow for verifying AI-assisted bug fixes with stronger mechanical evidence. The tool enforces a declared modification scope so agents cannot silently touch unrelated files, and requires regression probes to fail on the known broken baseline and pass on the fixed version, flagging green-on-both results as weak evidence rather than proof. It also supports clean replay so reviewers can reproduce and inspect the evidence without reconstructing the change manually.", "body_md": "Over the last few months, I’ve been spending more time working with AI coding agents. They are fast, useful, and in many cases they can fix small bugs or implement focused changes much faster than doing everything manually.\n\nBut while using them, I kept coming back to the same problem: how do I know that the evidence around the fix is actually meaningful?\n\nIt is easy to ask an agent to change some code, run a test, and return a green result. The problem is that a green test only tells me that the final version passed. It does not necessarily tell me whether the test was capable of detecting the original bug.\n\nThat difference sounds small, but it matters a lot.\n\nImagine an AI agent fixes a bug and adds a regression test. The final test run passes, so the change looks good. Now imagine that we take the exact same test and run it against the broken version of the code, and it passes there too.\n\nIn that case, the test is green, but it did not really prove the bug was fixed.\n\nThat is the kind of problem I wanted to handle more systematically.\n\nThe principle is simple:\n\n**Broken baseline + regression probe = FAIL**\n\n**Fixed result + same regression probe = PASS**\n\nIf the same probe fails on the broken version and passes on the fixed version, the evidence is much stronger.\n\nIf it passes on both, then the test may not actually be testing the bug we care about.\n\nThis idea became the basis for a small tool I’ve been building called **AI Coding Agent Reliability Kit**.\n\nIt is a Linux-first local workflow for reviewing AI-assisted bug fixes with stronger mechanical evidence.\n\nThe goal is not to replace code review, security review, or engineering judgment. The goal is to make the evidence produced around an AI-generated fix easier to inspect and harder to misinterpret.\n\nWhen I started working on this, I did not want another wrapper that simply runs a command and prints success or failure.\n\nI wanted a workflow that would answer several questions around the change.\n\nWhat was the agent allowed to modify?\n\nDid it stay inside that scope?\n\nDid the final verification command really pass?\n\nDoes the regression probe fail on the broken baseline?\n\nCan the result be reproduced in a clean replay?\n\nCan another person review the evidence without having to reconstruct everything manually?\n\nThese are basic questions, but once coding agents start making larger or more frequent changes, they become important.\n\nOne of the things the kit does is define the task and the allowed paths before the coding work begins.\n\nThis is useful because an agent can sometimes fix the requested problem while also changing unrelated files. The final result may still work, but the change surface becomes larger than necessary.\n\nFor a focused bug fix, I usually want a clear boundary.\n\nFor example, if a task only needs a change in one source file and one regression test, then a modification to an unrelated README, configuration file, or another package should be visible immediately.\n\nThe kit tracks the declared scope and checks whether the final change stayed inside it.\n\nIf something changed outside the allowed boundary, that is reported instead of silently accepting the result.\n\nA normal workflow often ends after something like this:\n\n```\npytest tests/test_bug.py\n```\n\nThe command exits successfully, so the task is considered done.\n\nThat is useful, but for a regression fix I want one more piece of information.\n\nI want to know what happens when the same probe is run against the known broken baseline.\n\nA good regression probe should normally distinguish the two states.\n\nFor example:\n\n```\nBroken version: FAIL\nFixed version:  PASS\n```\n\nThat is a much stronger signal than:\n\n```\nBroken version: PASS\nFixed version:  PASS\n```\n\nThe second result does not necessarily mean the code is wrong. It means the evidence is weak.\n\nThat distinction is important.\n\nThe workflow should not pretend that a test proves something it does not prove.\n\nThis was one of the main reasons I built the kit.\n\nA test can look perfectly valid and still fail to discriminate between the broken and fixed versions.\n\nFor example, imagine a function that is supposed to multiply two numbers but currently adds them.\n\nThe broken code might look like this:\n\n``` python\ndef calculate(a, b):\n    return a + b\n```\n\nThe fix changes it to:\n\n``` python\ndef calculate(a, b):\n    return a * b\n```\n\nNow imagine the test only checks this:\n\n``` python\ndef test_calculate_exists():\n    assert callable(calculate)\n```\n\nThe test is green after the fix, but it was also green before the fix.\n\nIt proves that the function exists. It does not prove that the bug was fixed.\n\nThis type of result should not be treated the same as a real regression test.\n\nThe kit detects this condition and reports the evidence as non-discriminating instead of treating the final green result as sufficient.\n\nAnother case I wanted to handle is when the actual code change is correct, but something outside the allowed scope is modified.\n\nThis can happen accidentally, or because an automated workflow touches another file during execution.\n\nSuppose the agent is allowed to modify:\n\n```\nsrc/formatter.py\ntests/test_formatter.py\n```\n\nThe source fix and test may both be correct, but if the final change also modifies:\n\n```\nREADME.md\n```\n\nthen the workflow should report that separately.\n\nThe important point here is that correctness and scope are different things.\n\nA correct source fix does not automatically make an unrelated change acceptable.\n\nThe kit keeps those concerns separate so the reviewer can see exactly what happened.\n\nAnother feature I wanted was the ability to reproduce a result from a clean state when needed.\n\nSometimes a change passes because of something left behind in the working environment: generated files, caches, local state, or another uncommitted change.\n\nA clean replay helps answer a simple question:\n\nCan the same result be reproduced without depending on the current working directory state?\n\nNot every task needs a full clean replay, but for changes where reproducibility matters, it gives the reviewer another useful piece of evidence.\n\nOne problem with automation is that it can generate too much output.\n\nRaw logs are useful when investigating something, but they are not always the best way to review a change.\n\nThe kit produces a structured review summary that makes the main results easier to inspect.\n\nA reviewer can see things such as:\n\n```\nScope: PASS\nFinal verification: PASS\nBroken baseline probe: FAIL\nFixed probe: PASS\nDifferential evidence: PASS\nReplay: REPRODUCIBLE\n```\n\nIf something fails, the summary should show that clearly instead of hiding it behind a successful final command.\n\nThe idea is to reduce the amount of reconstruction the reviewer needs to do.\n\nThe workflow can also export proof material and verify its integrity.\n\nThis is useful when the evidence needs to be moved, archived, or reviewed later.\n\nInstead of only saying that a command passed at some point, the workflow can preserve the relevant result records and verify that the expected material has not changed.\n\nThis is not meant to turn the tool into a security certification system. It is simply a way to make the evidence around a run more inspectable and reproducible.\n\nOne thing I wanted to keep very clear is the meaning of the final status.\n\nWhen the kit reports:\n\n```\nREADY_FOR_REVIEW\n```\n\nit does not mean:\n\n```\nThe code is definitely correct.\n```\n\nIt does not mean:\n\n```\nThe code is secure.\n```\n\nIt also does not mean:\n\n```\nThis is automatically production-ready.\n```\n\nIt means that the configured mechanical evidence gates passed and the result is ready for a human reviewer to inspect.\n\nThat distinction is intentional.\n\nI do not think an automated workflow should silently upgrade mechanical evidence into a semantic engineering judgment.\n\nA human reviewer still needs to look at the actual change and decide whether it is correct for the system.\n\nA human developer usually has some context in their head while making a change. They know what they touched, what they were trying to fix, and what assumptions they made.\n\nWith an AI coding agent, more of that work can happen automatically and quickly.\n\nThat speed is useful, but it also means it becomes easier to accept a result simply because the agent says the task is complete and the final test is green.\n\nThe faster the implementation becomes, the more useful it is to make the verification process explicit.\n\nI do not see this as an argument against coding agents. I use them.\n\nThe point is that if agents are going to make more changes for us, then the evidence around those changes should become better as well.\n\nThe first version of the kit is intentionally simple in its environment assumptions.\n\nIt is designed for Linux and uses a local workflow with:\n\nI wanted the first version to stay focused instead of claiming support for platforms I had not properly validated.\n\nWindows and macOS are not part of the validated V1 environment.\n\nThat may change later, but I prefer to be explicit about what has actually been tested.\n\nThis tool does not try to prove that a program is correct.\n\nIt does not replace unit tests, integration tests, static analysis, security review, or a good human code review.\n\nIt also does not make a weak test strong automatically.\n\nIf the regression probe is poorly designed, the workflow can detect some types of weak evidence, such as a probe that passes on both the broken and fixed versions, but it cannot understand every semantic requirement of every project.\n\nWhat it does is provide a stricter process around the evidence that already exists.\n\nFor me, that is the useful part.\n\nA typical flow looks roughly like this:\n\nIt is intentionally not a fully autonomous approval system.\n\nThe last step is still review.\n\nSuppose the bug is in a slug function.\n\nThe broken implementation is:\n\n``` python\ndef slugify(text):\n    return text.strip().lower().replace(\" \", \"\")\n```\n\nFor this input:\n\n```\nHello World\n```\n\nthe broken result is:\n\n```\nhelloworld\n```\n\nThe required result is:\n\n```\nhello-world\n```\n\nThe fix might be:\n\n``` python\ndef slugify(text):\n    return \"-\".join(text.lower().split())\n```\n\nAnd the regression probe:\n\n``` python\ndef test_slugify():\n    assert slugify(\"Hello World\") == \"hello-world\"\n    assert slugify(\"Two Words\") == \"two-words\"\n```\n\nNow the useful part is not only that this test passes after the fix.\n\nThe stronger evidence is:\n\n```\nBroken baseline: FAIL\nFixed result:    PASS\n```\n\nThat tells the reviewer that the probe actually detects the behavior being corrected.\n\nThe main reason is that I want to use coding agents more, not less.\n\nBut if I am going to delegate more implementation work to them, I want a better boundary between “the agent says it is done” and “there is enough evidence for me to review the result seriously.”\n\nThat is the gap this project is trying to cover.\n\nIt is still a first version, and I expect the workflow to evolve as I use it on more projects and find more edge cases.\n\nFor now, the focus is narrow: bounded changes, stronger regression evidence, reproducibility, and human review.\n\nIf that sounds useful for the way you work with coding agents, I published the first version here:\n\n[https://sovereign-system.itch.io/ai-coding-agent-reliability-kit](https://sovereign-system.itch.io/ai-coding-agent-reliability-kit)", "url": "https://wpnews.pro/news/why-green-tests-are-not-enough-for-ai-coding-agents", "canonical_source": "https://dev.to/rabeh_arch/why-green-tests-are-not-enough-for-ai-coding-agents-app", "published_at": "2026-10-22 06:33:33+00:00", "updated_at": "2026-10-22 06:40:25.496904+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-safety"], "entities": ["AI Coding Agent Reliability Kit"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-green-tests-are-not-enough-for-ai-coding-agents", "markdown": "https://wpnews.pro/news/why-green-tests-are-not-enough-for-ai-coding-agents.md", "text": "https://wpnews.pro/news/why-green-tests-are-not-enough-for-ai-coding-agents.txt", "jsonld": "https://wpnews.pro/news/why-green-tests-are-not-enough-for-ai-coding-agents.jsonld"}}