Every time a new model ships, the same ritual: change the model string, run the agent, read a few replies, they look good, ship it.
The replies are the one part of an agent that almost never breaks visibly. What breaks is behavior. A tool call quietly disappears, an argument drifts, a refund amount loses its decimal point, and the agent keeps talking like everything is fine. I learned this the hard way when a swap made my agent stop calling cancel_subscription
while it kept telling users their subscription was cancelled.
With a new frontier model out this week, a lot of model strings are about to change. This is the check I now run before any swap. It takes about 20 minutes and produces a real diff instead of a vibe check.
This is the step you cannot recover later. Once you swap, the old behavior is gone.
Start a recording proxy and point your agent at it, no code changes:
npx whatbroke-cli record --out baseline.jsonl
OPENAI_BASE_URL=http://127.0.0.1:4141/v1 # openai sdk
ANTHROPIC_BASE_URL=http://127.0.0.1:4141 # anthropic sdk
Run your agent through its real scenarios. Agents are nondeterministic, so run each scenario three times and name the runs refund-flow#1
, refund-flow#2
, refund-flow#3
(an x-whatbroke-run
header per request, or --run
). Three samples per scenario is enough to tell flaps from real changes.
Pick scenarios where the agent has to do something: call tools, hit an API, write a record. Pure chat scenarios are where nothing ever visibly breaks, so they tell you the least.
Change the model string. Nothing else. If you also want to tweak the prompt for the new model, do that as a second swap with its own diff, otherwise you will never know which change caused what.
npx whatbroke-cli record --out swapped.jsonl
Same scenarios, same names, three runs each.
npx whatbroke-cli diff baseline.jsonl swapped.jsonl
Read it top down:
3/3
means it happens every time. 1/3
on something your baseline also flapped on is just your agent being itself, and the diff demotes those automatically.
npx whatbroke-cli diff baseline.jsonl current.jsonl --fail-on breaking
Exit code 1 on breaking changes, --md
for a report you can drop into a PR comment.
If your agent runs behind Langfuse, LangSmith, or anything emitting OTel GenAI spans, you already have the baseline, you just have not diffed it yet. Export last week's traces and this week's:
npx whatbroke-cli import last-week-export.json --run baseline
npx whatbroke-cli import this-week-export.json --run swapped
npx whatbroke-cli diff last-week-export.whatbroke.jsonl this-week-export.whatbroke.jsonl
The tool is deterministic, fully offline, MIT licensed, and your traces never leave your machine: https://github.com/arthi-arumugam-git/whatbroke
For a real example of what a swap changes while the replies all look fine, I ran the same agent on a 3x smaller model and wrote up what actually changed.
If you run this before your next swap and it catches something, I would genuinely love to hear what it was.