Your AI says "fixed". Would a test notice if it wasn't?

A developer released Bite, a free open-source tool that reverts each recent fix in a codebase and reruns the test suite to verify whether any test actually guards against the bug returning. On public projects it flagged a memory leak in prosemirror-py that a commit had claimed to fix, and a fix has since been merged upstream. The tool runs on copies without editing source code and is installable via pip, pipx, or uvx.

When an AI coding tool says "Fixed, and I added a test", how do you know the test would catch the bug coming back? Bite is a small free tool that checks. It takes each recent fix back out of your code and runs your tests. If they still pass, nothing guards that fix, and Bite names it. On real public projects, not a staged demo, it found a memory leak in prosemirror-py that a commit said it had fixed. The fix for it has since been merged there. Try it, no key needed. It works on copies and never edits your code. With plain Python, two lines in any terminal: python -m pip install bite-check python -m bite check --free It shows a demo on a made-up project first, then tells you how to check your own drag its folder into the window . Have pipx or uv? pipx run bite-check --free or uvx bite-check --free . Doing MLH Hack Week Nov 6-12 ? Bite has good first issues: small Python fixes, each with "where to look" and "done when". They get picked up fast, so the label always shows what's still open. Comment on one to claim it. https://github.com/connabyss/bite/labels/good%20first%20issue https://github.com/connabyss/bite/labels/good%20first%20issue So: which of your AI's "fixed" commits would you bet on? Run it and find out. The Bite cat says hi.