I’m testing CodeGuard AI, my AI-powered GitHub PR reviewer, and today I caught an interesting failure.
Here’s the test scenario.
assertEquals(2, 1, "Discount calculation is incorrect");
The test expects 2, but the actual result is 1. So it fails, as intended.
Good news: CodeGuard correctly identified that the test was failing.
But then came the problem.
The AI suggested changing the assertion to:
assertEquals(1, 1, "Discount calculation is incorrect");
Now the test passes.
But did we fix the underlying problem?
No. We changed the expectation to match the actual result.
The test no longer checks whether the code produces the intended result. It simply confirms that 1 equals 1.
Detecting a failing test is not enough. An AI reviewer must also ensure that its suggested fix preserves the test’s original intent.
I’ve logged this as a bug and will address it after completing the current testing batches. For now, I’m continuing to test the other features and record the failures before fixing them together.
That’s the reality of building in public: sometimes the most useful update is what your own product gets wrong.
What would you want CodeGuard to validate before allowing an AI-suggested test fix?
I'm building CodeGuard AI for developers who want more trustworthy AI-assisted code reviews.
If you'd like to explore the source code, you can find it here: CodeGuard AI on Gumroad
I'm continuing to test it, document the failures, and improve it based on what I learn.