Your Refactor Needs an Oracle: Characterization Tests vs. AI Diffs

A developer at MonkeyCode outlines a workflow that uses characterization tests as an oracle to validate AI-generated refactors of legacy code. The approach involves writing tests that lock in current behavior, running them against the AI's proposed changes, and using the test results to decide whether to accept the diff. The developer demonstrates the method with a JavaScript example and emphasizes that while the tests catch regressions, they do not replace human review.

AI-generated refactors fail silently. The code looks clean. The tests pass. Then a production edge case breaks. Characterization tests catch that break before it ships. This workflow locks current behavior first, then lets a free model propose changes, then uses tests as the oracle. Legacy code has undocumented quirks. Humans miss them. Models miss them too. A refactor that changes a single boundary condition is a regression waiting to happen. You need a way to say "this diff preserves behavior" with evidence. Characterization tests provide that evidence. They capture what the code does today, not what it should do. They freeze the behavior you are about to touch. Write tests that document the current output for known inputs. Include weird cases: zero, negative, undefined, boundary values. Here is a legacy function with nested conditions: // legacy.js export function applyDiscount total, user { if user.type === 'vip' { return total 0.9; } else { if total 100 { return total 0.95; } else { return total; } } } Now write tests that lock its actual behavior: js // applyDiscount.test.js import { describe, it, expect } from 'vitest'; import { applyDiscount } from './legacy.js'; describe 'applyDiscount current behavior', = { it 'applies 10% for vip regardless of total', = { expect applyDiscount 99, { type: 'vip' } .toBe 89.1 ; } ; it 'applies 5% for non-vip over 100', = { expect applyDiscount 101, { type: 'guest' } .toBe 95.95 ; } ; it 'no discount at 100 or below', = { expect applyDiscount 100, { type: 'guest' } .toBe 100 ; } ; it 'no discount for zero total', = { expect applyDiscount 0, { type: 'guest' } .toBe 0 ; } ; } ; Run it. Make sure it fails on an unmodified repo? No. It should pass and lock the behavior. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to request a refactor. The prompt was simple: "Refactor applyDiscount to reduce nesting. Keep the exact same observable behavior." The model returned guard clauses: export function applyDiscount total, user { if user.type === 'vip' return total 0.9; if total 100 return total 0.95; return total; } This looks fine. But you are not going to trust the look. You are going to run the tests. Run the suite on an isolated server instead of your laptop. MonkeyCode's free server option can execute the test command. That keeps your local environment clean and reproducible. Create a small verification script: bash /usr/bin/env bash set -euo pipefail git fetch origin CHANGED FILES=$ git diff --name-only origin/main echo "Changed files:" echo "$CHANGED FILES" npm install npm test Save it as verify refactor.sh and run it on the server. The script does three things: fetch latest, show the diff surface, and run the full test suite. The test result plus diff size drives the decision. | Test result | Diff size | Action | |---|---|---| | Pass | Small | Accept with confidence | | Pass | Large | Review hard; behavior may be untouched but risk is higher | | Fail | Any | Reject the diff | If tests fail, do not debug the model output. Instead, find which characterization test broke. That tells you exactly which behavior changed. Go back to Step 2 with a more specific prompt. Characterization tests are not a proof. They only cover inputs you thought about. The model can change behavior on untested inputs. You still need human review of the diff. This approach is not for safety-critical code. It is not for code requiring formal verification. It is also not for teams that cannot run a test suite in a clean environment. Do not use this workflow if you lack any test runner. Do not use it if your legacy module is too tangled to import. And do not use it to skip code review. The oracle saves you from silent regressions, not from thinking. AI accelerates refactoring. Acceleration is not a correctness guarantee. Characterization tests turn "the diff looks safe" into "the diff is safe for the inputs we know." Free model access and a free server make this workflow cost-effective. But nothing happens until you run the tests. Run them.