{"slug": "llm-testing-how-to-test-an-llm-app-and-what-to-rerun-when-the-model-changes", "title": "LLM testing: how to test an LLM app, and what to rerun when the model changes", "summary": "Setting temperature to 0 and asserting on exact output no longer works for testing LLM applications on models current as of November 2026, because Anthropic's newest models return an error when a request sets temperature, Google's models ignore it, and OpenAI's accept it only with reasoning turned off, according to a blog post by the developer of the EvalShift testing tool. The post argues LLM test suites must treat every case as a pass rate rather than a pass or fail, and lays out seven test types — unit, tool-call, functional, regression, performance, safety (red teaming) and production monitoring — plus a six-step process and guidance on what to rerun when the underlying model changes.", "body_md": "[← all posts](https://www.evalshift.dev/blog)\n\n# LLM testing: how to test an LLM app, and what to rerun when the model changes\n\nWhat LLM testing is, the types of LLM tests, six steps to test an LLM application, and what to rerun when the model underneath it changes.\n\nLLM testing is checking that an application built on a language model still does what you expect, before a change ships and after it. In practice that means a fixed set of inputs, a set of checks on what comes back, and a rule for when the result blocks a release. Three things make it different from testing ordinary code: the same input gives different outputs on different runs, there is rarely one correct string to compare against, and the model can change while your code stays the same.\n\nMost guides to LLM testing cover the first two with one piece of advice: set the temperature to 0\nand assert on the output. On the models current in November 2026 that advice no longer works.\nAnthropic's newest models return an error when a request sets `temperature`, Google's ignore it,\nand OpenAI's accept it only with reasoning turned off. A test suite for an LLM app now has to\ntreat every case as a pass rate, not a pass or a fail.\n\nThis post covers the types of LLM tests, why the habits from unit testing break, a six-step way to test an LLM application, and what to rerun when the model underneath it changes. A disclosure: I build EvalShift, a tool for one part of this, and it has its own section near the end. The rest works with any tool or none.\n\n## What LLM testing is\n\nAn LLM test has three parts: an input, the application's output for that input, and a check that\nturns the output into a result. The input can be one prompt, a whole conversation, or a recorded\nagent run with its tool results. The check can be code (does the reply parse, did the agent call\n`lookup_order` before `issue_refund`), another model acting as a judge, or a person.\n\n\"LLM testing\" and \"LLM evaluation\" describe the same activity from two angles. Evaluation is the measuring: how often is the answer grounded, how often is the right tool called. Testing is the decision built on it: this change may merge, this one may not. The tools are the same, and most teams use the words interchangeably.\n\n\"AI model testing\" is used for two different jobs. One is testing a model, which is what labs and\n[public benchmarks](https://www.evalshift.dev/blog/llm-benchmarks) do: fixed tasks, a harness, a leaderboard. The other is\ntesting an application built on a model: your prompt, your tools, your users' inputs. A model\nthat tops a benchmark can still fail your application's tests, because none of the benchmark's\ntasks are yours. This post is about the second job.\n\n## The types of LLM tests\n\n| Type | What it checks | Graded by | When it runs | \n|---|---|---|---|\n| Unit test | One model call or one step: the output parses, matches a schema, contains the required fields, picks a label from the allowed set | Code | Every pull request | \n| Tool-call test | Which tools an agent called, in what order, with which arguments | Code | Every pull request | \n| Functional test | A whole task from start to finish: the conversation ends in the right outcome | Code on the final state, plus an LLM judge for the text | Pull requests that touch the prompt, model or tools | \n| Regression test | The same suite before and after a change, compared case by case | A paired comparison of the two runs | Every change to the prompt, the model or a tool definition | \n| Performance test | Latency, tokens and cost per task | Measurement against a budget | Alongside the regression run | \n| Safety test (red teaming) | Prompt injection, jailbreaks, data leakage, off-policy answers | Generated adversarial inputs and a judge | Before launch, then on a schedule | \n| Production monitoring | A sample of live traffic | Code and judges running online | Continuously | \n\nMost lists name five of these: unit, functional, regression, performance and safety. I add two.\nTool-call tests get their own row because an agent can produce a correct-sounding reply after the\nwrong action, and a test that reads only the reply passes it.\n[Evaluating agent tool calls](https://www.evalshift.dev/blog/evaluating-agent-tool-calls) covers that failure in detail.\nMonitoring is on the list because it is where the test cases for everything above come from.\n\nRegression testing is the type that pays for the others. Unit and functional tests tell you whether one version works. A regression test tells you whether a change made it worse, and that is the question in front of you each time a prompt is edited or a model is swapped.\n\n## Why ordinary testing habits break on LLMs\n\n### The same input gives different outputs\n\nA unit test assumes that a function called twice with the same arguments returns the same value.\nA model does not. Atil and colleagues\n[measured this ↗](https://arxiv.org/abs/2408.04667) on five models configured to be deterministic,\nacross eight tasks and ten runs each: accuracy varied by up to 15% between runs, and the gap\nbetween the best and worst possible result reached 70%.\n\nThe providers say the same in their docs. Anthropic's\n[glossary ↗](https://platform.claude.com/docs/en/about-claude/glossary) states that results are not\nfully deterministic even at temperature 0. OpenAI's\n[API reference ↗](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create)\ndescribes `seed` as a beta feature that samples deterministically on a best-effort basis, with no\nguarantee.\n\n### You can no longer set the temperature to 0\n\nTemperature 0 never made outputs identical, but it narrowed them, and most testing guides still start there. Current models have taken the setting away:\n\n| Provider | What the docs say, November 2026 | \n|---|---|\n| Anthropic | On Claude Sonnet 5.5, requests that set `temperature` ,`top_p` or`top_k`[return an error ↗](https://platform.claude.com/docs/en/models/sonnet-5-5/migration-guide) . | \n| OpenAI | For GPT-6, [remove ↗](https://developers.openai.com/api/docs/guides/latest-model/gpt-6-astra)`temperature` and`top_p` unless reasoning effort is`none` . GPT-6 Astra and GPT-6.1 Sol do not support`none` . | \n|  | On Gemini 3.6 Flash, Gemini 3.5 Flash-Lite and later models, `temperature` ,`top_p` and`top_k` are[deprecated and ignored ↗](https://aistudio.google.com/docs/whats-new-gemini-3.6) , and future generations will reject them. | \n\nThere are two consequences for a test suite. The first is that a model upgrade can break the\nrequest before it changes any behavior: a client that still sends `temperature: 0` gets an error\nfrom Claude Sonnet 5.5. The second is that run-to-run variation is now a fixed property of the\nsystem under test. You cannot configure it away, so the suite has to measure it.\n\n### There is rarely one correct string\n\n`assertEqual(output, expected)` fails on a correct answer that is worded differently. The checks\nthat survive are checks on properties: the reply is valid JSON with these fields, it mentions the\norder number, it does not promise a refund the policy forbids, the agent called this tool with\nthese arguments. What is left over, such as tone or whether an explanation is actually helpful,\ngoes to an [LLM judge](https://www.evalshift.dev/blog/llm-as-a-judge), which is itself a model and needs checking against\nhuman labels before it is trusted.\n\n### The model changes while your code stays the same\n\nOrdinary dependencies change when you bump a version. A model can change in four ways, and only the first is your decision:\n\n- +**You change the model ID,** for an upgrade or a cheaper tier.\n- +**An alias moves.** Google's[model docs ↗](https://aistudio.google.com/docs/models) say a`latest` alias is swapped to each new release with two weeks' notice by email. Its[changelog ↗](https://aistudio.google.com/docs/changelog) records`gemini-pro-latest` and`gemini-flash-latest` switching to the Gemini 3 previews on January 21, 2026. OpenAI's[`chat-latest` ↗](https://developers.openai.com/api/docs/models/chat-latest) follows the model\ncurrently in ChatGPT and is refreshed regularly.\n- +**The model is retired,** and requests are rerouted or fail. Google's changelog says requests to`gemini-2.5-pro` will route to`gemini-3.8-flash` from December 1, 2026. The[deprecation schedule](https://www.evalshift.dev/blog/llm-model-deprecation-schedule) lists the dates for each provider.\n- +**The serving stack changes under a pinned ID.** Anthropic's[versioning page ↗](https://platform.claude.com/docs/en/about-claude/models/model-ids-and-versions) says the weights behind a model ID are fixed, and that the router, safety classifiers and\nsampling logic around them can change and occasionally produce minor differences in behavior.\n\nThe last one is rare, and it happens. In September 2025 Anthropic published a\n[postmortem ↗](https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues) of three\ninfrastructure bugs that degraded Claude's responses. One of them misrouted 0.8% of Sonnet 4\nrequests from August 5, and 16% in its worst hour on August 31. Anthropic wrote that the\nevaluations it ran did not capture the degradation users were reporting.\n\nThe oldest measurement of this kind is still the clearest. Chen, Zaharia and Zou\n[tracked ↗](https://arxiv.org/abs/2307.09009) the service called GPT-4 between March and June 2023\nand found its accuracy at telling prime numbers from composite ones had gone from 84% to 51%.\n\n## How to test an LLM application\n\nSix steps, in the order I would build them. Each one works without the next, so you can stop wherever the budget runs out and still have a suite worth running.\n\n## Step 1: collect test cases from real traffic\n\nStart from what users sent, not from what you can think of. The inputs you can invent are the ones you already handle. The cases that break are long conversations, odd phrasings and inputs in languages the prompt never anticipated, and those only exist in logs.\n\nTake 100 to 300 real inputs, de-duplicated and spread across the routes or intents your app\nhandles. For an agent, a case is the whole run: the conversation so far, the tool results it\nreceived and the tools it called. Remove personal data before the cases go into the repository.\n[Build a golden suite from production traffic](https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic)\nwalks through the collection, and\n[how many eval cases do you need](https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need) covers sizing.\n\n## Step 2: write deterministic checks first\n\nCode checks are free, instant and never disagree with themselves. Write one for every promise your application makes to the code downstream of it:\n\n- +The output parses, and validates against the schema your parser expects.\n- +Enumerated fields hold an allowed value.\n- +Required facts are present: the order ID from the input appears in the reply.\n- +Forbidden content is absent: no internal IDs, no promise outside policy.\n- +Length stays inside the bounds your UI can show.\n\nThese are the unit tests of an LLM app. They catch most of what a model change breaks, because most of what breaks is shape: a field renamed, a code fence around the JSON, a date in another format.\n\n## Step 3: check the tool calls, not only the reply\n\nFor an agent, grade the trace. A reply that says the refund is on its way reads the same whether the agent looked the order up first or skipped the check. Three checks cover most of it:\n\n``` python\ndef check_refund_run(run, expected):\n    \"\"\"`run` is one recorded agent run: its tool calls, in order, and the final reply.\"\"\"\n    names = [call[\"name\"] for call in run[\"tool_calls\"]]\n\n    # 1. Selection and order: verify before acting.\n    assert names == [\"lookup_order\", \"issue_refund\"], names\n\n    # 2. Arguments: exact for identifiers, bounded for amounts.\n    refund = run[\"tool_calls\"][1][\"arguments\"]\n    assert refund[\"order_id\"] == expected[\"order_id\"]\n    assert 0 < refund[\"amount\"] <= expected[\"order_total\"]\n\n    # 3. Rules that hold on every run, whatever the input.\n    assert names.count(\"issue_refund\") <= 1\n```\n\nSelection, arguments and rules that must hold on every run. The third kind is the cheapest to write and the most valuable: \"never issue a refund without a lookup\" needs no expected output per case, so it applies to every case you will ever add.\n\n## Step 4: use an LLM judge for what code can't check\n\nSome properties have no code check: whether the answer is grounded in the retrieved documents,\nwhether it addresses the question, whether the tone fits. An LLM judge can grade those, with three\nconditions. Give it one criterion at a time and a pass or fail answer, not a score out of ten.\nCheck its verdicts against 30 to 50 cases you labeled by hand before you rely on it. And keep its\nresults advisory, meaning reported but not blocking, until you have seen where it is wrong.\n[When to trust an LLM judge](https://www.evalshift.dev/blog/when-to-trust-an-llm-judge) has the audit procedure.\n\n## Step 5: run each case more than once\n\nOne run per case turns variation into flakiness. Take a suite of 50 cases where the application is fine and each case passes 98% of the time. The whole suite comes back green in only 36% of runs. The other 64% show a red check with nothing wrong, and a team that sees that twice learns to press re-run.\n\nSo record a pass rate per case, not a boolean:\n\n- +Run each case three to five times. Three catches the cases that flip often, and five gives a finer rate.\n- +Gate on the suite-level rate with its uncertainty, not on \"all green\". On 200 cases a pass rate of 86% has a 95% interval of roughly 80.5% to 90.1%, so a three-point drop on its own means nothing.\n- +Treat the cases that flip as findings. A case that passes two times out of three is telling you the prompt is ambiguous for that input.\n\nAnthropic's [statistical guide to model evals ↗](https://www.anthropic.com/research/statistical-approach-to-model-evals)\nrecommends the same things in more formal terms: report a standard error with every score, and\nresample each question several times and average, which reduces the variance within a question.\n\n## Step 6: put the suite in CI and gate on the comparison\n\nA suite that runs when someone remembers is a habit, not a test. Attach it to the pull request, and trigger it on the files that can move the result: the prompt, the model name, the tool definitions, the suite itself. Not on every commit, because each run costs real model calls.\n\nGate on the comparison with the base branch, not on an absolute score. \"Pass rate must be above 90%\" fails the day you add ten hard cases. \"This change must not regress more than the limit against main\" keeps working as the suite grows. Run both versions on the same cases and compare them case by case. Anthropic's guide makes the statistical argument for this: scores from two models on the same question are correlated, so a paired comparison detects a difference that two separate averages would hide.\n\n[LLM regression testing in CI](https://www.evalshift.dev/blog/llm-regression-testing-in-ci) shows the workflow file, and\nhow to roll the gate out in report-only mode first.\n\n## What to rerun when the model changes\n\nA model change is the regression test that matters most, because it alters every call at once and nothing in the diff shows it. The pull request is one line. Here is the order I run things in.\n\n**1. A smoke test of the request itself.** Send one request per code path to the new model before\nanything else. Removed parameters, a changed default for reasoning, a different maximum output:\nthese fail loudly and cheaply, and there is no point running 300 cases into a 400 error.\n\n**2. The full suite on both models, fresh.** Run the current model and the candidate on the same\ncases in the same session. Do not compare the candidate with outputs you recorded from the\ncurrent model three months ago: the current model varies between runs too, and you need to know\nby how much before you can read the difference.\n\n**3. The prompt and tools held fixed.** Change only the model. If the candidate needs a prompt\nchange to behave, that is a second change with its own run. Two variables in one comparison leave\nnobody able to say which one moved the result.\n\n**4. A per-example comparison, tool calls first.** Read the differences in this order: tool\nselection, tool arguments, output structure, then judged quality. The first three are where a\nmodel change does damage that nobody sees in the text.\n[What breaks when you switch LLMs](https://www.evalshift.dev/blog/what-breaks-when-you-switch-llms) has examples of each.\n\n**5. Cost and latency per task, not per token.** A cheaper model that takes one more tool call per\nturn is not cheaper. Measure the whole task.\n\n**6. A decision against limits you wrote down first.** How many regressions will you accept, on\nwhich routes, for what saving. Zero on the routes that move money.\n\nI have published two of these runs. In one, a candidate was 58.8% cheaper and 75.2% faster with no\nfailed calls, and it picked a different tool on 25% of comparisons against a 10% limit. That\n[report came back FAIL](https://www.evalshift.dev/blog/cheaper-faster-and-still-a-fail). In the other, a customer-support\nagent with six tools moved to a cheaper model over 120 production examples and\n[held every limit](https://www.evalshift.dev/blog/what-a-passing-migration-proves) at 71.2% lower cost. The price and the\nmodel card did not tell me which would pass. The run did.\n\nTwo longer walkthroughs cover this procedure end to end:\n[how to catch LLM regressions before a model upgrade ships](https://www.evalshift.dev/blog/llm-regression-testing-model-upgrade),\nwhich builds it from three small scripts, and\n[how to test an LLM model migration before you ship it](https://www.evalshift.dev/blog/test-llm-model-migration-before-you-ship),\nwhich covers the statistics.\n\nPin the model ID in production so that the change is yours to schedule. On Google, that means a\nstable model name and not a `latest` alias. Then put the provider's retirement dates in the same\ncalendar as your releases: the run above takes an afternoon when you choose the date and a week\nwhen the provider does.\n\n## LLM testing tools\n\n| Job | Tools to look at | \n|---|---|\n| Assertions and prompt or model comparison from a config file | Promptfoo | \n| Quality metrics inside a pytest suite | DeepEval | \n| RAG retrieval and grounding metrics | Ragas | \n| Agent evals in sandboxes | Inspect AI | \n| Red teaming | Promptfoo, DeepTeam | \n| Production tracing and online evals | Langfuse, LangSmith, Braintrust, Arize Phoenix, Opik | \n| Gating a model or prompt change with a paired comparison | EvalShift | \n\nTwo posts go through these one by one:\n[the best LLM evaluation tools, compared by the job they do](https://www.evalshift.dev/blog/best-llm-evaluation-tools), and\n[the open-source LLM evaluation frameworks](https://www.evalshift.dev/blog/open-source-llm-evaluation-frameworks), with\nwhat each license covers. Most teams end up with two tools: one that watches production and one\nthat can fail a pull request.\n\n## Doing this with EvalShift\n\nDisclosure again: I'm the founder. [EvalShift](https://www.evalshift.dev/docs) covers the regression half of this post,\nthe comparison of two versions of an agent on the same recorded cases. It does not do red teaming\nor production monitoring.\n\nThe capture SDK records real agent runs, and `evalshift capture sync` turns them into a golden\nsuite (step 1). The config names the checks from steps 2 and 3, and the number of samples from\nstep 5:\n\n```\ndefaults:\n  samples_per_example: 3   # every case runs three times on each model\n\nevaluators:\n  structural:\n    - type: json_schema\n      schema_path: schemas/reply.json\n      applies_to: [\"*\"]\n  tool_selection:\n    - name: routing\n  tool_arguments:\n    - name: routing_args\n      against: expected\n```\n\nOne command runs both models over the suite, scores each side, and tests whether the differences are larger than the noise:\n\n```\nevalshift compare --from claude-sonnet-5 --to claude-sonnet-5-5\n```\n\nThe result is a report with the per-example differences and one verdict from a\n[migration policy](https://www.evalshift.dev/docs/migration-policy) you write in the same file: pass, conditional pass,\nfail, or inconclusive when the suite is too small to tell. The\n[GitHub Action](https://www.evalshift.dev/docs/action) runs the same comparison on a pull request (step 6). The CLI and SDK\nare open source and run locally with your own provider keys.\n\n## FAQ\n\n**What is LLM testing?** LLM testing is checking that an application built on a language model\ndoes what you expect, before and after a change. A test is an input, the application's output and\na check on that output, which can be code, an LLM judge or a person. Because outputs vary between\nruns, each test produces a pass rate, not a single pass or fail.\n\n**How do you test an LLM application?** Collect 100 to 300 real inputs from production, write\ndeterministic checks for structure and required content, check the tool calls an agent makes and\nnot only its reply, add an LLM judge for qualities code can't check, run each case three to five\ntimes, and run the suite in CI so a change is compared with the base branch before it merges.\n\n**What are the types of LLM testing?** Unit tests on a single call, tool-call tests on an agent's\nactions, functional tests on a whole task, regression tests that compare two versions, performance\ntests for latency and cost, safety tests such as red teaming, and monitoring of live traffic.\nRegression testing is the one to build first, because it answers whether a change made things\nworse.\n\n**How is LLM testing different from software testing?** In three ways. The same input produces\ndifferent outputs on different runs, so a test is a rate. There is rarely one correct output, so\nchecks target properties such as structure, facts and actions. And the model can change without\nyour code changing, through a moved alias, a retirement or a serving change.\n\n**How do you test non-deterministic LLM outputs?** Run each case several times and record the\nfraction that pass. Setting the temperature to 0 is no longer an option on current models: Claude\nSonnet 5.5 returns an error when it is set, Gemini 3.6 Flash ignores it, and GPT-6 accepts it only\nwith reasoning effort set to none. Gate on the suite-level rate with its interval, and treat cases\nthat flip between runs as findings.\n\n**What is LLM regression testing?** Running the same fixed set of cases before and after a change\nto the prompt, the model or the tools, and comparing the two runs case by case. The comparison is\npaired: both versions see identical inputs, so differences between cases cancel and a smaller\nchange becomes detectable.\n\n**How do you test an LLM model change?** Send one request per code path to the new model first, to\ncatch removed parameters. Then run the current model and the candidate on the same cases in the\nsame session, with the prompt and tools unchanged. Compare per example, starting with tool\nselection and arguments, measure cost and latency per task, and decide against limits written\nbefore the run.\n\n**What are the best LLM testing tools?** It depends on the job: Promptfoo for assertions and\ncomparing prompts or models, DeepEval for metrics inside pytest, Ragas for RAG, Inspect AI for\nsandboxed agent evals, Langfuse, LangSmith or Braintrust for production tracing with online evals,\nand EvalShift for gating a model or prompt change with a paired comparison.\n\n## Keep reading\n\n- +[How to catch LLM regressions before a model upgrade ships](https://www.evalshift.dev/blog/llm-regression-testing-model-upgrade) :\nthe regression test built from three scripts, on a real upgrade.\n- +[LLM regression testing in CI](https://www.evalshift.dev/blog/llm-regression-testing-in-ci) : the suite as a pull-request\ncheck.\n- +[Evaluating agent tool calls](https://www.evalshift.dev/blog/evaluating-agent-tool-calls) : what a test that reads only\nthe reply cannot see.\n- +[LLM benchmarks](https://www.evalshift.dev/blog/llm-benchmarks) : why a model's leaderboard score does not replace a test\non your own application.", "url": "https://wpnews.pro/news/llm-testing-how-to-test-an-llm-app-and-what-to-rerun-when-the-model-changes", "canonical_source": "https://www.evalshift.dev/blog/llm-testing", "published_at": "2026-11-10 22:52:45.482060+00:00", "updated_at": "2026-11-10 22:52:47.760310+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "mlops", "ai-agents", "developer-tools"], "entities": ["Anthropic", "Google", "OpenAI", "EvalShift"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llm-testing-how-to-test-an-llm-app-and-what-to-rerun-when-the-model-changes", "markdown": "https://wpnews.pro/news/llm-testing-how-to-test-an-llm-app-and-what-to-rerun-when-the-model-changes.md", "text": "https://wpnews.pro/news/llm-testing-how-to-test-an-llm-app-and-what-to-rerun-when-the-model-changes.txt", "jsonld": "https://wpnews.pro/news/llm-testing-how-to-test-an-llm-app-and-what-to-rerun-when-the-model-changes.jsonld"}}