{"slug": "the-20-minute-check-i-run-before-swapping-an-agent-to-a-new-model", "title": "The 20-minute check I run before swapping an agent to a new model", "summary": "A developer created WhatBroke, an open-source tool that diffs agent behavior before and after swapping to a new model, catching silent failures like dropped tool calls or drifted arguments. The tool records agent runs, compares them deterministically, and flags breaking changes in about 20 minutes.", "body_md": "Every time a new model ships, the same ritual: change the model string, run the agent, read a few replies, they look good, ship it.\n\nThe replies are the one part of an agent that almost never breaks visibly. What breaks is behavior. A tool call quietly disappears, an argument drifts, a refund amount loses its decimal point, and the agent keeps talking like everything is fine. I learned this the hard way when a swap made my agent stop calling `cancel_subscription`\n\nwhile it kept telling users their subscription was cancelled.\n\nWith a new frontier model out this week, a lot of model strings are about to change. This is the check I now run before any swap. It takes about 20 minutes and produces a real diff instead of a vibe check.\n\nThis is the step you cannot recover later. Once you swap, the old behavior is gone.\n\nStart a recording proxy and point your agent at it, no code changes:\n\n```\nnpx whatbroke-cli record --out baseline.jsonl\nOPENAI_BASE_URL=http://127.0.0.1:4141/v1     # openai sdk\nANTHROPIC_BASE_URL=http://127.0.0.1:4141     # anthropic sdk\n```\n\nRun your agent through its real scenarios. Agents are nondeterministic, so run each scenario three times and name the runs `refund-flow#1`\n\n, `refund-flow#2`\n\n, `refund-flow#3`\n\n(an `x-whatbroke-run`\n\nheader per request, or `--run`\n\n). Three samples per scenario is enough to tell flaps from real changes.\n\nPick scenarios where the agent has to *do* something: call tools, hit an API, write a record. Pure chat scenarios are where nothing ever visibly breaks, so they tell you the least.\n\nChange the model string. Nothing else. If you also want to tweak the prompt for the new model, do that as a second swap with its own diff, otherwise you will never know which change caused what.\n\n```\nnpx whatbroke-cli record --out swapped.jsonl\n```\n\nSame scenarios, same names, three runs each.\n\n```\nnpx whatbroke-cli diff baseline.jsonl swapped.jsonl\n```\n\nRead it top down:\n\n`3/3`\n\nmeans it happens every time. `1/3`\n\non something your baseline also flapped on is just your agent being itself, and the diff demotes those automatically.\n\n```\nnpx whatbroke-cli diff baseline.jsonl current.jsonl --fail-on breaking\n```\n\nExit code 1 on breaking changes, `--md`\n\nfor a report you can drop into a PR comment.\n\nIf your agent runs behind Langfuse, LangSmith, or anything emitting OTel GenAI spans, you already have the baseline, you just have not diffed it yet. Export last week's traces and this week's:\n\n``` python\nnpx whatbroke-cli import last-week-export.json --run baseline\nnpx whatbroke-cli import this-week-export.json --run swapped\nnpx whatbroke-cli diff last-week-export.whatbroke.jsonl this-week-export.whatbroke.jsonl\n```\n\nThe tool is deterministic, fully offline, MIT licensed, and your traces never leave your machine: [https://github.com/arthi-arumugam-git/whatbroke](https://github.com/arthi-arumugam-git/whatbroke)\n\nFor a real example of what a swap changes while the replies all look fine, I ran the same agent on a 3x smaller model and wrote up [what actually changed](https://dev.to/arthiarumugam/i-swapped-my-agent-to-a-3x-smaller-model-and-diffed-what-actually-changed-33j3).\n\nIf you run this before your next swap and it catches something, I would genuinely love to hear what it was.", "url": "https://wpnews.pro/news/the-20-minute-check-i-run-before-swapping-an-agent-to-a-new-model", "canonical_source": "https://dev.to/arthiarumugam/the-20-minute-check-i-run-before-swapping-an-agent-to-a-new-model-1kgi", "published_at": "2026-07-25 02:08:12+00:00", "updated_at": "2026-07-25 02:58:07.422202+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "machine-learning"], "entities": ["WhatBroke", "Langfuse", "LangSmith", "OpenAI", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/the-20-minute-check-i-run-before-swapping-an-agent-to-a-new-model", "markdown": "https://wpnews.pro/news/the-20-minute-check-i-run-before-swapping-an-agent-to-a-new-model.md", "text": "https://wpnews.pro/news/the-20-minute-check-i-run-before-swapping-an-agent-to-a-new-model.txt", "jsonld": "https://wpnews.pro/news/the-20-minute-check-i-run-before-swapping-an-agent-to-a-new-model.jsonld"}}