I've stopped treating AI bug fixes as conclusions and started treating them as claims that require evidence. If you're doing a deep dive into your own code, the rule is simple: a bug is only fixed when the failure is gone, the intended behavior works, and you haven't accidentally nuked a neighboring feature.
Documenting the Failure First #
Stop telling the AI "saving is broken." That's useless. Before you touch a single line of code, you need a reproducible failure path. I use this format to keep the LLM honest:
Starting state:
[what must already be true]
Actions:
1. [action]
2. [action]
3. [action]
Expected result:
[what a user should see or be able to do]
Actual result:
[what happens instead]
Evidence:
[error text, screenshot, console output, or saved record]
For instance, instead of "it doesn't save," use:
"Starting state: Signed in with one saved task. Actions: Edit title, press Save, refresh. Expected: New title stays. Actual: Says 'Saved' but old title returns. Evidence: Network request 200, but DB record is unchanged."
The Four-Point Verification Process #
Once the AI gives you a patch, don't just merge it. I run these four checks to make sure it's a real fix and not a mask.
1. The Original Repro Must Pass
Run the exact steps from your documentation. No paraphrasing. If you suddenly "can't reproduce it," that's a red flag. "Cannot reproduce" is not the same as "fixed."
2. Verify the Underlying State
UI is a liar. A "Saved!" toast notification doesn't mean the data hit the disk.
- Refresh the page.
- Hard restart the app.
- Check the DB dashboard.
3. Ensure "Honest" Failures
A common AI move is to make every input return a success message. Try an invalid input or a timeout. If the AI turned a specific validation error into a generic "Success" or a blank screen, it didn't fix the bug—it just hid the error message.
4. Check for Side-Effect Regressions
This is the classic "fixed the login, broke the password reset" scenario. Check the features immediately adjacent to the fix.
For anyone building an AI workflow from scratch, the goal isn't to make the AI sound confident; it's to make your proof requests specific.
Next OpenAI Usage Limit Glitch: Resetting Early →