cancel_subscription
tool call. The prose is polished, but the business logic is broken.To stop these regressions from hitting production, I've implemented a strict 20-minute diffing protocol. Instead of guessing if a new frontier model is a "drop-in" replacement, I treat the swap as a breaking change until proven otherwise.
The Baseline Capture #
The biggest mistake is swapping the model before recording how the current version behaves. Once you change that string, your baseline is gone. I use a recording proxy to capture the exact tool calls and arguments being sent.
First, spin up the recorder:
npx whatbroke-cli record --out baseline.jsonl
Then, point the SDKs to the local proxy:
OPENAI_BASE_URL=http://127.0.0.1:4141/v1
ANTHROPIC_BASE_URL=http://127.0.0.1:4141
I run every critical scenario three times. Because LLMs are non-deterministic, a single run can be a fluke. If a tool call disappears 1/3 times, it's a "flap"; if it disappears 3/3 times, it's a regression. I focus exclusively on "action" scenarios—API hits and database writes—because pure chat rarely reveals the architectural drift that kills an agent.
The Isolated Swap #
I follow one golden rule: Change the model string and nothing else.
If you tweak the system prompt to "optimize" it for the new model at the same time you swap the version, you've introduced two variables. You won't know if the new behavior is due to the model's weights or your prompt edit. If the model requires a prompt change to function, that gets its own separate diff.
Executing the Diff #
After recording the same scenarios with the new model using npx whatbroke-cli record --out swapped.jsonl
, I run the comparison:
npx whatbroke-cli diff baseline.jsonl swapped.jsonl
I categorize the results into three buckets:
Breaking Changes: These are the red flags. A tool call that existed in the baseline is now missing, or a run that previously succeeded now errors out.Argument Drift: This is the sneakiest failure. The tool is still called, but the arguments have shifted (e.g., a date format changed fromYYYY-MM-DD
toMM/DD/YYYY
). This usually breaks the downstream API.Performance Ratios: I flag any latency increase over 1.5x or cost increase over 1.25x.
Integrating into the AI Workflow #
To prevent "vibe-based" deployments from creeping back into the team's habit, I've moved this into our CI pipeline. By using the --fail-on breaking
flag, the build fails if a model swap drops a tool call.
npx whatbroke-cli diff baseline.jsonl current.jsonl --fail-on breaking
For those of us already using observability tools like Langfuse or LangSmith, you don't necessarily need to record new runs. You can export traces from the previous week and the current week to perform a retrospective diff:
npx whatbroke-cli import last-week-export.json --run baseline
npx whatbroke-cli import this-week-export.json --run swapped
npx whatbroke-cli diff last-week-export.whatbroke.jsonl this-week-export.whatbroke.jsonl
This approach turns a subjective "it feels faster/smarter" conversation into a technical report with verifiable data.
https://github.com/arthi-arumugam-git/whatbroke
Next WeTask vs Redis: Performance Benchmarks →
All Replies （0） #
No replies yet — be the first!