{"slug": "my-tests-were-describing-the-code-not-checking-it", "title": "My tests were describing the code, not checking it", "summary": "A developer who used Claude Code to build a .NET invoicing library discovered that the AI-generated tests passed while the code was still wrong, failing to catch a rounding bug that would have mispriced common UK invoices. By manually applying mutation testing, the developer found that the test suite did not pin the rounding policy or culture handling, and even the release pipeline silently failed to publish the correct version.", "body_md": "Built a .NET invoicing library with Claude Code over about a week. Zero dependencies, VAT and GST arithmetic, JSON and CSV export, the usual. By the third phase I had 202 tests passing on two target frameworks and green CI on three operating systems.\n\nIt would also have been wrong by 3p on any UK invoice with three items at £3.99.\n\nNot wrong in some pathological edge case. Wrong on a completely normal invoice, at the most common price point in British retail, in a way that ends up in someone's VAT return.\n\nI found it because I got suspicious of my own green checkmark.\n\nWhy agent-written tests lie to you\n\nAsk an agent to implement something and write tests for it. It writes both. They pass.\n\nYou have not tested the code. You have confirmed the code agrees with itself.\n\nI know how obvious that reads. It was not obvious to me at the time, because the suite looked legitimate. Property tests. Golden scenarios with hand-calculated values across USD, JPY and KWD. A file literally called CsvExportTests with a culture-swap test in it, which is the exact test you write if you know about the .NET decimal formatting trap.\n\nThat test could not catch the bug it existed to catch. I will get to it.\n\nBreaking my own code on purpose\n\nMutation testing, if you have not run into it: introduce a deliberate bug, see if any test fails. Nothing fails, that behaviour is not tested, regardless of what the test names imply.\n\nI tried Stryker.NET. It fell over on the multi-target setup and I burned fifteen minutes before giving up. Doing it by hand ended up better anyway, because I picked the mutations rather than getting a percentage at the end. I do not really want a coverage score. I want to know if the four rounding call sites are pinned.\n\nThe loop, which is as dumb as it sounds:\n\nApply one mutation. Run tests. Write down what failed. Revert. Check green. Next one.\n\nUnder three failures on an arithmetic mutation, I called it a hole.\n\nWhat fell out\n\nNothing pinned the rounding policy. The library rounds half-up, MidpointRounding.AwayFromZero, because that is what tax authorities expect. .NET defaults to banker's rounding, which will quietly hand you off-by-a-penny invoices. So I flipped it to ToEven and ran the suite.\n\nFive failures. All five were unit tests of the rounding helper itself. None of the twenty-five golden invoice scenarios blinked.\n\nNot one of them had a midpoint value in it. I had proved the helper rounds correctly on its own. I had not proved the calculator ever calls it. Two different things and I was only covering one.\n\nHere is the part that got me. The scenario I would have put money on catching this, ¥995 at 10% giving ¥99.5, does not discriminate at all. ToEven rounds 99.5 up to 100, because 100 is the even one. Same answer. My \"obvious\" midpoint test was not a midpoint test.\n\nDiscounts, same story. Pulled the rounding out of the invoice-level discount calculation. One test failed, and it was a generic property test about all decimals being rounded. Every discount scenario I had written used a clean percentage of a clean subtotal. Nothing landed on a half-penny, so nothing broke.\n\nThe culture test was decorative. This one still annoys me. The CSV test swapped CultureInfo.CurrentCulture to de-DE and tr-TR, exported, asserted. Correct in every way. When I swapped InvariantCulture for CurrentCulture in the actual export code, it passed.\n\nThe fixture invoice totalled 100, 20 and 120. Under de-DE, 100m.ToString() gives you \"100\". Same as invariant. The test was fine except for the single detail that mattered, which was having a number with a decimal point in it.\n\nChanged the unit price to 10.50. Suddenly the mutation kills three tests.\n\nAnd then the release pipeline just lied. Tagged v0.3.0 and forgot to bump the version in the project file. Workflow packed 0.2.0. NuGet returned a 409 because that version was already up there. --skip-duplicate ate the error. Every step green. Nothing published.\n\nGreen does not mean shipped. I did not know that until it happened to me.\n\nThe one I am glad I did not ship\n\nThose four are test-quality problems. Annoying, fixable. The fifth was a different animal.\n\nI was adding a per-line tax mode. HMRC and the ATO both let you round tax per line instead of on the subtotal, and if the accounting system on the other end does it per line, your totals disagree by a penny. Reasonable feature.\n\nSomewhere around phase three I had picked up the habit of making the agent answer design questions on paper first, with actual numbers, before touching code. So I asked four. One of them was what happens when per-line meets tax-inclusive pricing.\n\nThe numbers came back and they were not good.\n\nExtracting a tax-exclusive base out of an inclusive price leaves a residual on every line. Always zero or minus one minor unit. Never positive. One line, who cares. But the residual is periodic, and which prices trigger it is completely predictable:\n\n10% tax: line total in minor units where x mod 11 = 5\n\n20% VAT: x mod 6 = 3\n\n£3.99 is 399 pence. 399 mod 6 is 3.\n\nEvery £3.99 line loses a penny. Three lines, 3p. Seven lines, 7p. And the reconciliation rule I had written earlier only absorbs one minor unit, so it would have shaved the difference off the tax figure and produced a VAT breakdown implying 19.82% on a 20% invoice. That is not a rounding artefact, that is a wrong number on a filing report.\n\nSo I did not build it. Constructor throws on that combination, the analysis is written down, someone can pick it up later. Probably me.\n\nIf I had written the code first, a property test would have caught the broken invariant eventually. But the implementation would already exist by then, and at that point the tempting move is to loosen the invariant rather than admit the design is wrong. On paper it cost twenty minutes and no ego.\n\nWhat I would actually suggest\n\nNot \"adopt mutation testing.\" Everyone nods at that and nobody does it.\n\nSmaller: find the three or four places in your code where being wrong is expensive and quiet. Mine were four rounding call sites. Break each one by hand, run the tests, count. Half an hour, no tooling, and you will learn something uncomfortable.\n\nAnd when an agent is building something with non-obvious edge cases, make it do the hard design questions in writing before it writes anything. Not a plan document. Worked arithmetic. The £3.99 thing was invisible in prose and screamed the second someone actually did the sums.\n\nI am not turning this into a methodology. One project, a few questions I happened to get lucky with, and I would bet there is still stuff wrong in there that none of my mutations went near. The takeaway is smaller than a lesson: a passing test suite written by whatever wrote the code tells you those two things are consistent with each other. Not that either is right. Took me a few hundred green tests to properly feel that.\n\nLibrary is on NuGet as [InvoiceCore ](https://www.nuget.org/packages/InvoiceCore/)if you want it. Honestly it is more useful as a worked example of the above than as an invoicing package, and I would rather you took the protocol than the dependency.", "url": "https://wpnews.pro/news/my-tests-were-describing-the-code-not-checking-it", "canonical_source": "https://dev.to/aftabkh4n/my-tests-were-describing-the-code-not-checking-it-3m2p", "published_at": "2026-08-30 07:47:41+00:00", "updated_at": "2026-08-30 08:22:28.911046+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "artificial-intelligence"], "entities": ["Claude Code", "Stryker.NET", "NuGet", ".NET"], "alternates": {"html": "https://wpnews.pro/news/my-tests-were-describing-the-code-not-checking-it", "markdown": "https://wpnews.pro/news/my-tests-were-describing-the-code-not-checking-it.md", "text": "https://wpnews.pro/news/my-tests-were-describing-the-code-not-checking-it.txt", "jsonld": "https://wpnews.pro/news/my-tests-were-describing-the-code-not-checking-it.jsonld"}}