The 20-minute check I run before swapping an agent to a new model

A developer created WhatBroke, an open-source tool that diffs agent behavior before and after swapping to a new model, catching silent failures like dropped tool calls or drifted arguments. The tool records agent runs, compares them deterministically, and flags breaking changes in about 20 minutes.

Every time a new model ships, the same ritual: change the model string, run the agent, read a few replies, they look good, ship it. The replies are the one part of an agent that almost never breaks visibly. What breaks is behavior. A tool call quietly disappears, an argument drifts, a refund amount loses its decimal point, and the agent keeps talking like everything is fine. I learned this the hard way when a swap made my agent stop calling cancel subscription while it kept telling users their subscription was cancelled. With a new frontier model out this week, a lot of model strings are about to change. This is the check I now run before any swap. It takes about 20 minutes and produces a real diff instead of a vibe check. This is the step you cannot recover later. Once you swap, the old behavior is gone. Start a recording proxy and point your agent at it, no code changes: npx whatbroke-cli record --out baseline.jsonl OPENAI BASE URL=http://127.0.0.1:4141/v1 openai sdk ANTHROPIC BASE URL=http://127.0.0.1:4141 anthropic sdk Run your agent through its real scenarios. Agents are nondeterministic, so run each scenario three times and name the runs refund-flow 1 , refund-flow 2 , refund-flow 3 an x-whatbroke-run header per request, or --run . Three samples per scenario is enough to tell flaps from real changes. Pick scenarios where the agent has to do something: call tools, hit an API, write a record. Pure chat scenarios are where nothing ever visibly breaks, so they tell you the least. Change the model string. Nothing else. If you also want to tweak the prompt for the new model, do that as a second swap with its own diff, otherwise you will never know which change caused what. npx whatbroke-cli record --out swapped.jsonl Same scenarios, same names, three runs each. npx whatbroke-cli diff baseline.jsonl swapped.jsonl Read it top down: 3/3 means it happens every time. 1/3 on something your baseline also flapped on is just your agent being itself, and the diff demotes those automatically. npx whatbroke-cli diff baseline.jsonl current.jsonl --fail-on breaking Exit code 1 on breaking changes, --md for a report you can drop into a PR comment. If your agent runs behind Langfuse, LangSmith, or anything emitting OTel GenAI spans, you already have the baseline, you just have not diffed it yet. Export last week's traces and this week's: python npx whatbroke-cli import last-week-export.json --run baseline npx whatbroke-cli import this-week-export.json --run swapped npx whatbroke-cli diff last-week-export.whatbroke.jsonl this-week-export.whatbroke.jsonl The tool is deterministic, fully offline, MIT licensed, and your traces never leave your machine: https://github.com/arthi-arumugam-git/whatbroke https://github.com/arthi-arumugam-git/whatbroke For a real example of what a swap changes while the replies all look fine, I ran the same agent on a 3x smaller model and wrote up what actually changed https://dev.to/arthiarumugam/i-swapped-my-agent-to-a-3x-smaller-model-and-diffed-what-actually-changed-33j3 . If you run this before your next swap and it catches something, I would genuinely love to hear what it was.