A friend's product summarizes customer support tickets using a fine-tuned LLM
prompt. It worked perfectly on GPT-4o for six months. Then OpenAI deprecated
4o, the team migrated to GPT-4.1, ran a smoke test in the playground, said
"looks fine," and shipped.
Two weeks later a customer escalated: "Your urgency tagging is wrong on
basically everything since last Wednesday."
The prompt asked for {"intent": "...", "urgency": "low|medium|high"}
. On
4o, the model returned exactly that. On 4.1, it started returning
{"intent": "...", "urgency_level": "..."}
— semantically identical, but
the downstream classifier was indexing on urgency
and silently fell
through to a default value of "low" on 100% of new tickets.
Nobody saw it because:
- The prompt didn't error. JSON parsed. Fields existed.
- The unit tests checked the prompt string, not theprompt output. - The integration tests mocked the LLM call.
- The output was indistinguishable from "everything's fine and quiet."
This is the silent regression problem. Code has tests; prompts have vibes.
Three categories of model-swap failure #
After looking at a dozen of these incidents, the failures cluster into three
groups. Knowing which kind you're looking at tells you what to test.
1. Format drift. The model decides to rename a field, drop a field, add
a field you didn't ask for, or change list ordering. JSON still parses. Your
downstream code breaks.
2. Reasoning regression. The model is "improved" but loses a hidden
constraint your prompt depended on. Classic example: GPT-4 reliably extracted
all requirements from a contract; GPT-4-Turbo extracted "the most important
ones," dropping 15-20% of clauses. The format was fine. The data was wrong.
3. Tone shift. Less common but expensive. The new model's outputs are
more verbose, less verbose, friendlier, blunter. If anything downstream
(another model, a regex, a fuzzy matcher) was tuned to the old tone, it
breaks.
What the team should have had #
A test suite of 30 representative tickets, each with an expected JSON shape.
On model swap day:
$ promptfork test summarize_ticket --baseline gpt-4o
→ running v12 across [gpt-4.1] vs baseline [gpt-4o]
✗ 30/30 ok, but 6 regressions detected
- urgency_field_renamed: 6 cases
- severity 2 (functional)
Six lines. Seven seconds. Two-week customer-facing bug avoided.
How to actually do this #
The setup for the team that got bitten took four minutes:
pip install promptfork
promptfork push summarize_ticket \
--file prompts/summarize.txt \
--message "current prod"
for t in tickets/*.json; do
name=$(basename "$t" .json)
promptfork add-test summarize_ticket "$name" \
--input ticket="$(cat "$t")" \
--rubric "must return urgency in {low,medium,high}"
done
promptfork test summarize_ticket --models gpt-4o
promptfork test summarize_ticket --baseline 1 --models gpt-4.1
That's it. The --baseline
flag is what catches drift — it pulls the
baseline output, runs the candidate, and asks Claude Haiku to compare them
under a strict "only flag strictly worse" rubric.
The CI version #
The same command in a GitHub Action means no prompt change ever ships
without running against a known-good baseline:
- uses: shaunvand/promptfork-cli@v0
with:
prompt: summarize_ticket
baseline: 1
api-key: ${{ secrets.PROMPTFORK_API_KEY }}
The action exits non-zero on regression. Branch protection blocks the merge.
If you ship LLM features, you need this. The first time it catches a silent
regression, it pays for itself a hundred times over. PromptFork has a free
tier (3 prompts, 50 runs/mo) at https://promptfork.online/diff — set it up
in five minutes, sleep better forever.