{"slug": "your-tests-pass-so-does-the-wrong-code", "title": "Your Tests Pass. So Does the Wrong Code", "summary": "A developer built TestSlop, a tool that takes a code diff and finds a nearby alternative implementation—an \"Evil Twin\"—that behaves differently but still passes the same test suite, then reports the input that distinguishes the two. In one replay of a real `ms` library commit that added month formatting, the original and a variant with `msAbs >= y` changed to `msAbs > y` both passed 163 of 163 tests, with the missing witness being exactly one year (31,557,600,000 ms), where output shifts from `1y` to `12mo`. The tool is limited to JavaScript and TypeScript boundary comparisons on changed lines and is available on GitHub as a local build, not published to npm.", "body_md": "A green test suite tells you one thing: the implementation you wrote satisfies the tests you wrote.\n\nIt doesn't tell you whether the version one small edit away would pass too. If swapping `<` for `<=` would have sailed through, the suite was never checking that boundary.\n\nCoding agents make this sharper. They write the code and the tests from the same assumption, the suite goes green, and nobody stops to ask what else it would accept.\n\nSo I built TestSlop. Point it at a diff and it finds one nearby alternative implementation your tests also accept, then shows the input that tells the two apart.\n\nOne sentence: TestSlop finds one nearby \"Evil Twin\" implementation that behaves differently but still passes the same tests.\n\nThat GIF runs a three-line quantity check. As text:\n\n```\nORIGINAL    quantity <= 0\nEVIL TWIN   quantity <= 1\n\nORIGINAL    3 / 3 tests passed\nEVIL TWIN   3 / 3 tests passed\n\nMissing witness: quantity = 1\n```\n\nBoth passed. The two versions only disagree at `quantity = 1`, and no test covers that value.\n\nAn Evil Twin is a nearby implementation that behaves differently but still passes the same tests.\n\nThe missing witness is the input that separates them. Here it is `quantity = 1`, the simplest input where the two versions give different answers.\n\nTestSlop hands you the pair and the witness. Which version matches your contract stays your call.\n\nMost of this repo's public face went through an agent workflow. This slice is the part that pushed the sanitized history to GitHub and checked the rendered README, then returned a release verdict:\n\nThe diagram is the whole loop: shift one boundary comparison, rerun your tests against the alternative in a scratch copy, and print both results with the witness. Your source tree stays untouched.\n\nOne fixture in the repo replays a real `ms` commit, the one that added month formatting, against its preserved test suite.\n\nThe original and the Evil Twin (`msAbs >= y` changed to `msAbs > y`) both passed 163 of 163 tests. The missing witness is exactly one year: at `31,557,600,000` ms the output moves from `1y` to `12mo`.\n\nThe formatter tests checked one millisecond past a year and never exactly one year. One historical example, replayable offline. It doesn't show that TestSlop catches shipped bugs in general.\n\nTestSlop is narrow on purpose. It looks at JavaScript and TypeScript boundary comparisons on changed lines, so plenty of diffs produce no Twin at all.\n\nThe code is on GitHub: [github.com/hyukvoid/TestSlop](https://github.com/hyukvoid/TestSlop)\n\nIt runs from a local build (`npm install`, then `npm run demo`). It isn't published to npm. The repo keeps the research notes from the three phases that narrowed the project down to this idea.\n\nIf you've used mutation testing or coding-agent workflows, I'd be curious whether one concrete counterexample feels more useful than a score.", "url": "https://wpnews.pro/news/your-tests-pass-so-does-the-wrong-code", "canonical_source": "https://dev.to/jay_stride/your-tests-pass-so-does-the-wrong-code-1eam", "published_at": "2026-09-25 13:37:00+00:00", "updated_at": "2026-09-25 14:00:49.968262+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools"], "entities": ["TestSlop", "GitHub", "ms", "npm"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-tests-pass-so-does-the-wrong-code", "markdown": "https://wpnews.pro/news/your-tests-pass-so-does-the-wrong-code.md", "text": "https://wpnews.pro/news/your-tests-pass-so-does-the-wrong-code.txt", "jsonld": "https://wpnews.pro/news/your-tests-pass-so-does-the-wrong-code.jsonld"}}