LLM testing: how to test an LLM app, and what to rerun when the model changes

Setting temperature to 0 and asserting on exact output no longer works for testing LLM applications on models current as of November 2026, because Anthropic's newest models return an error when a request sets temperature, Google's models ignore it, and OpenAI's accept it only with reasoning turned off, according to a blog post by the developer of the EvalShift testing tool. The post argues LLM test suites must treat every case as a pass rate rather than a pass or fail, and lays out seven test types — unit, tool-call, functional, regression, performance, safety (red teaming) and production monitoring — plus a six-step process and guidance on what to rerun when the underlying model changes.

← all posts https://www.evalshift.dev/blog LLM testing: how to test an LLM app, and what to rerun when the model changes What LLM testing is, the types of LLM tests, six steps to test an LLM application, and what to rerun when the model underneath it changes. LLM testing is checking that an application built on a language model still does what you expect, before a change ships and after it. In practice that means a fixed set of inputs, a set of checks on what comes back, and a rule for when the result blocks a release. Three things make it different from testing ordinary code: the same input gives different outputs on different runs, there is rarely one correct string to compare against, and the model can change while your code stays the same. Most guides to LLM testing cover the first two with one piece of advice: set the temperature to 0 and assert on the output. On the models current in November 2026 that advice no longer works. Anthropic's newest models return an error when a request sets temperature , Google's ignore it, and OpenAI's accept it only with reasoning turned off. A test suite for an LLM app now has to treat every case as a pass rate, not a pass or a fail. This post covers the types of LLM tests, why the habits from unit testing break, a six-step way to test an LLM application, and what to rerun when the model underneath it changes. A disclosure: I build EvalShift, a tool for one part of this, and it has its own section near the end. The rest works with any tool or none. What LLM testing is An LLM test has three parts: an input, the application's output for that input, and a check that turns the output into a result. The input can be one prompt, a whole conversation, or a recorded agent run with its tool results. The check can be code does the reply parse, did the agent call lookup order before issue refund , another model acting as a judge, or a person. "LLM testing" and "LLM evaluation" describe the same activity from two angles. Evaluation is the measuring: how often is the answer grounded, how often is the right tool called. Testing is the decision built on it: this change may merge, this one may not. The tools are the same, and most teams use the words interchangeably. "AI model testing" is used for two different jobs. One is testing a model, which is what labs and public benchmarks https://www.evalshift.dev/blog/llm-benchmarks do: fixed tasks, a harness, a leaderboard. The other is testing an application built on a model: your prompt, your tools, your users' inputs. A model that tops a benchmark can still fail your application's tests, because none of the benchmark's tasks are yours. This post is about the second job. The types of LLM tests | Type | What it checks | Graded by | When it runs | |---|---|---|---| | Unit test | One model call or one step: the output parses, matches a schema, contains the required fields, picks a label from the allowed set | Code | Every pull request | | Tool-call test | Which tools an agent called, in what order, with which arguments | Code | Every pull request | | Functional test | A whole task from start to finish: the conversation ends in the right outcome | Code on the final state, plus an LLM judge for the text | Pull requests that touch the prompt, model or tools | | Regression test | The same suite before and after a change, compared case by case | A paired comparison of the two runs | Every change to the prompt, the model or a tool definition | | Performance test | Latency, tokens and cost per task | Measurement against a budget | Alongside the regression run | | Safety test red teaming | Prompt injection, jailbreaks, data leakage, off-policy answers | Generated adversarial inputs and a judge | Before launch, then on a schedule | | Production monitoring | A sample of live traffic | Code and judges running online | Continuously | Most lists name five of these: unit, functional, regression, performance and safety. I add two. Tool-call tests get their own row because an agent can produce a correct-sounding reply after the wrong action, and a test that reads only the reply passes it. Evaluating agent tool calls https://www.evalshift.dev/blog/evaluating-agent-tool-calls covers that failure in detail. Monitoring is on the list because it is where the test cases for everything above come from. Regression testing is the type that pays for the others. Unit and functional tests tell you whether one version works. A regression test tells you whether a change made it worse, and that is the question in front of you each time a prompt is edited or a model is swapped. Why ordinary testing habits break on LLMs The same input gives different outputs A unit test assumes that a function called twice with the same arguments returns the same value. A model does not. Atil and colleagues measured this ↗ https://arxiv.org/abs/2408.04667 on five models configured to be deterministic, across eight tasks and ten runs each: accuracy varied by up to 15% between runs, and the gap between the best and worst possible result reached 70%. The providers say the same in their docs. Anthropic's glossary ↗ https://platform.claude.com/docs/en/about-claude/glossary states that results are not fully deterministic even at temperature 0. OpenAI's API reference ↗ https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create describes seed as a beta feature that samples deterministically on a best-effort basis, with no guarantee. You can no longer set the temperature to 0 Temperature 0 never made outputs identical, but it narrowed them, and most testing guides still start there. Current models have taken the setting away: | Provider | What the docs say, November 2026 | |---|---| | Anthropic | On Claude Sonnet 5.5, requests that set temperature , top p or top k return an error ↗ https://platform.claude.com/docs/en/models/sonnet-5-5/migration-guide . | | OpenAI | For GPT-6, remove ↗ https://developers.openai.com/api/docs/guides/latest-model/gpt-6-astra temperature and top p unless reasoning effort is none . GPT-6 Astra and GPT-6.1 Sol do not support none . | | | On Gemini 3.6 Flash, Gemini 3.5 Flash-Lite and later models, temperature , top p and top k are deprecated and ignored ↗ https://aistudio.google.com/docs/whats-new-gemini-3.6 , and future generations will reject them. | There are two consequences for a test suite. The first is that a model upgrade can break the request before it changes any behavior: a client that still sends temperature: 0 gets an error from Claude Sonnet 5.5. The second is that run-to-run variation is now a fixed property of the system under test. You cannot configure it away, so the suite has to measure it. There is rarely one correct string assertEqual output, expected fails on a correct answer that is worded differently. The checks that survive are checks on properties: the reply is valid JSON with these fields, it mentions the order number, it does not promise a refund the policy forbids, the agent called this tool with these arguments. What is left over, such as tone or whether an explanation is actually helpful, goes to an LLM judge https://www.evalshift.dev/blog/llm-as-a-judge , which is itself a model and needs checking against human labels before it is trusted. The model changes while your code stays the same Ordinary dependencies change when you bump a version. A model can change in four ways, and only the first is your decision: - + You change the model ID, for an upgrade or a cheaper tier. - + An alias moves. Google's model docs ↗ https://aistudio.google.com/docs/models say a latest alias is swapped to each new release with two weeks' notice by email. Its changelog ↗ https://aistudio.google.com/docs/changelog records gemini-pro-latest and gemini-flash-latest switching to the Gemini 3 previews on January 21, 2026. OpenAI's chat-latest ↗ https://developers.openai.com/api/docs/models/chat-latest follows the model currently in ChatGPT and is refreshed regularly. - + The model is retired, and requests are rerouted or fail. Google's changelog says requests to gemini-2.5-pro will route to gemini-3.8-flash from December 1, 2026. The deprecation schedule https://www.evalshift.dev/blog/llm-model-deprecation-schedule lists the dates for each provider. - + The serving stack changes under a pinned ID. Anthropic's versioning page ↗ https://platform.claude.com/docs/en/about-claude/models/model-ids-and-versions says the weights behind a model ID are fixed, and that the router, safety classifiers and sampling logic around them can change and occasionally produce minor differences in behavior. The last one is rare, and it happens. In September 2025 Anthropic published a postmortem ↗ https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues of three infrastructure bugs that degraded Claude's responses. One of them misrouted 0.8% of Sonnet 4 requests from August 5, and 16% in its worst hour on August 31. Anthropic wrote that the evaluations it ran did not capture the degradation users were reporting. The oldest measurement of this kind is still the clearest. Chen, Zaharia and Zou tracked ↗ https://arxiv.org/abs/2307.09009 the service called GPT-4 between March and June 2023 and found its accuracy at telling prime numbers from composite ones had gone from 84% to 51%. How to test an LLM application Six steps, in the order I would build them. Each one works without the next, so you can stop wherever the budget runs out and still have a suite worth running. Step 1: collect test cases from real traffic Start from what users sent, not from what you can think of. The inputs you can invent are the ones you already handle. The cases that break are long conversations, odd phrasings and inputs in languages the prompt never anticipated, and those only exist in logs. Take 100 to 300 real inputs, de-duplicated and spread across the routes or intents your app handles. For an agent, a case is the whole run: the conversation so far, the tool results it received and the tools it called. Remove personal data before the cases go into the repository. Build a golden suite from production traffic https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic walks through the collection, and how many eval cases do you need https://www.evalshift.dev/blog/how-many-eval-cases-do-you-need covers sizing. Step 2: write deterministic checks first Code checks are free, instant and never disagree with themselves. Write one for every promise your application makes to the code downstream of it: - +The output parses, and validates against the schema your parser expects. - +Enumerated fields hold an allowed value. - +Required facts are present: the order ID from the input appears in the reply. - +Forbidden content is absent: no internal IDs, no promise outside policy. - +Length stays inside the bounds your UI can show. These are the unit tests of an LLM app. They catch most of what a model change breaks, because most of what breaks is shape: a field renamed, a code fence around the JSON, a date in another format. Step 3: check the tool calls, not only the reply For an agent, grade the trace. A reply that says the refund is on its way reads the same whether the agent looked the order up first or skipped the check. Three checks cover most of it: python def check refund run run, expected : """ run is one recorded agent run: its tool calls, in order, and the final reply.""" names = call "name" for call in run "tool calls" 1. Selection and order: verify before acting. assert names == "lookup order", "issue refund" , names 2. Arguments: exact for identifiers, bounded for amounts. refund = run "tool calls" 1 "arguments" assert refund "order id" == expected "order id" assert 0 < refund "amount" <= expected "order total" 3. Rules that hold on every run, whatever the input. assert names.count "issue refund" <= 1 Selection, arguments and rules that must hold on every run. The third kind is the cheapest to write and the most valuable: "never issue a refund without a lookup" needs no expected output per case, so it applies to every case you will ever add. Step 4: use an LLM judge for what code can't check Some properties have no code check: whether the answer is grounded in the retrieved documents, whether it addresses the question, whether the tone fits. An LLM judge can grade those, with three conditions. Give it one criterion at a time and a pass or fail answer, not a score out of ten. Check its verdicts against 30 to 50 cases you labeled by hand before you rely on it. And keep its results advisory, meaning reported but not blocking, until you have seen where it is wrong. When to trust an LLM judge https://www.evalshift.dev/blog/when-to-trust-an-llm-judge has the audit procedure. Step 5: run each case more than once One run per case turns variation into flakiness. Take a suite of 50 cases where the application is fine and each case passes 98% of the time. The whole suite comes back green in only 36% of runs. The other 64% show a red check with nothing wrong, and a team that sees that twice learns to press re-run. So record a pass rate per case, not a boolean: - +Run each case three to five times. Three catches the cases that flip often, and five gives a finer rate. - +Gate on the suite-level rate with its uncertainty, not on "all green". On 200 cases a pass rate of 86% has a 95% interval of roughly 80.5% to 90.1%, so a three-point drop on its own means nothing. - +Treat the cases that flip as findings. A case that passes two times out of three is telling you the prompt is ambiguous for that input. Anthropic's statistical guide to model evals ↗ https://www.anthropic.com/research/statistical-approach-to-model-evals recommends the same things in more formal terms: report a standard error with every score, and resample each question several times and average, which reduces the variance within a question. Step 6: put the suite in CI and gate on the comparison A suite that runs when someone remembers is a habit, not a test. Attach it to the pull request, and trigger it on the files that can move the result: the prompt, the model name, the tool definitions, the suite itself. Not on every commit, because each run costs real model calls. Gate on the comparison with the base branch, not on an absolute score. "Pass rate must be above 90%" fails the day you add ten hard cases. "This change must not regress more than the limit against main" keeps working as the suite grows. Run both versions on the same cases and compare them case by case. Anthropic's guide makes the statistical argument for this: scores from two models on the same question are correlated, so a paired comparison detects a difference that two separate averages would hide. LLM regression testing in CI https://www.evalshift.dev/blog/llm-regression-testing-in-ci shows the workflow file, and how to roll the gate out in report-only mode first. What to rerun when the model changes A model change is the regression test that matters most, because it alters every call at once and nothing in the diff shows it. The pull request is one line. Here is the order I run things in. 1. A smoke test of the request itself. Send one request per code path to the new model before anything else. Removed parameters, a changed default for reasoning, a different maximum output: these fail loudly and cheaply, and there is no point running 300 cases into a 400 error. 2. The full suite on both models, fresh. Run the current model and the candidate on the same cases in the same session. Do not compare the candidate with outputs you recorded from the current model three months ago: the current model varies between runs too, and you need to know by how much before you can read the difference. 3. The prompt and tools held fixed. Change only the model. If the candidate needs a prompt change to behave, that is a second change with its own run. Two variables in one comparison leave nobody able to say which one moved the result. 4. A per-example comparison, tool calls first. Read the differences in this order: tool selection, tool arguments, output structure, then judged quality. The first three are where a model change does damage that nobody sees in the text. What breaks when you switch LLMs https://www.evalshift.dev/blog/what-breaks-when-you-switch-llms has examples of each. 5. Cost and latency per task, not per token. A cheaper model that takes one more tool call per turn is not cheaper. Measure the whole task. 6. A decision against limits you wrote down first. How many regressions will you accept, on which routes, for what saving. Zero on the routes that move money. I have published two of these runs. In one, a candidate was 58.8% cheaper and 75.2% faster with no failed calls, and it picked a different tool on 25% of comparisons against a 10% limit. That report came back FAIL https://www.evalshift.dev/blog/cheaper-faster-and-still-a-fail . In the other, a customer-support agent with six tools moved to a cheaper model over 120 production examples and held every limit https://www.evalshift.dev/blog/what-a-passing-migration-proves at 71.2% lower cost. The price and the model card did not tell me which would pass. The run did. Two longer walkthroughs cover this procedure end to end: how to catch LLM regressions before a model upgrade ships https://www.evalshift.dev/blog/llm-regression-testing-model-upgrade , which builds it from three small scripts, and how to test an LLM model migration before you ship it https://www.evalshift.dev/blog/test-llm-model-migration-before-you-ship , which covers the statistics. Pin the model ID in production so that the change is yours to schedule. On Google, that means a stable model name and not a latest alias. Then put the provider's retirement dates in the same calendar as your releases: the run above takes an afternoon when you choose the date and a week when the provider does. LLM testing tools | Job | Tools to look at | |---|---| | Assertions and prompt or model comparison from a config file | Promptfoo | | Quality metrics inside a pytest suite | DeepEval | | RAG retrieval and grounding metrics | Ragas | | Agent evals in sandboxes | Inspect AI | | Red teaming | Promptfoo, DeepTeam | | Production tracing and online evals | Langfuse, LangSmith, Braintrust, Arize Phoenix, Opik | | Gating a model or prompt change with a paired comparison | EvalShift | Two posts go through these one by one: the best LLM evaluation tools, compared by the job they do https://www.evalshift.dev/blog/best-llm-evaluation-tools , and the open-source LLM evaluation frameworks https://www.evalshift.dev/blog/open-source-llm-evaluation-frameworks , with what each license covers. Most teams end up with two tools: one that watches production and one that can fail a pull request. Doing this with EvalShift Disclosure again: I'm the founder. EvalShift https://www.evalshift.dev/docs covers the regression half of this post, the comparison of two versions of an agent on the same recorded cases. It does not do red teaming or production monitoring. The capture SDK records real agent runs, and evalshift capture sync turns them into a golden suite step 1 . The config names the checks from steps 2 and 3, and the number of samples from step 5: defaults: samples per example: 3 every case runs three times on each model evaluators: structural: - type: json schema schema path: schemas/reply.json applies to: " " tool selection: - name: routing tool arguments: - name: routing args against: expected One command runs both models over the suite, scores each side, and tests whether the differences are larger than the noise: evalshift compare --from claude-sonnet-5 --to claude-sonnet-5-5 The result is a report with the per-example differences and one verdict from a migration policy https://www.evalshift.dev/docs/migration-policy you write in the same file: pass, conditional pass, fail, or inconclusive when the suite is too small to tell. The GitHub Action https://www.evalshift.dev/docs/action runs the same comparison on a pull request step 6 . The CLI and SDK are open source and run locally with your own provider keys. FAQ What is LLM testing? LLM testing is checking that an application built on a language model does what you expect, before and after a change. A test is an input, the application's output and a check on that output, which can be code, an LLM judge or a person. Because outputs vary between runs, each test produces a pass rate, not a single pass or fail. How do you test an LLM application? Collect 100 to 300 real inputs from production, write deterministic checks for structure and required content, check the tool calls an agent makes and not only its reply, add an LLM judge for qualities code can't check, run each case three to five times, and run the suite in CI so a change is compared with the base branch before it merges. What are the types of LLM testing? Unit tests on a single call, tool-call tests on an agent's actions, functional tests on a whole task, regression tests that compare two versions, performance tests for latency and cost, safety tests such as red teaming, and monitoring of live traffic. Regression testing is the one to build first, because it answers whether a change made things worse. How is LLM testing different from software testing? In three ways. The same input produces different outputs on different runs, so a test is a rate. There is rarely one correct output, so checks target properties such as structure, facts and actions. And the model can change without your code changing, through a moved alias, a retirement or a serving change. How do you test non-deterministic LLM outputs? Run each case several times and record the fraction that pass. Setting the temperature to 0 is no longer an option on current models: Claude Sonnet 5.5 returns an error when it is set, Gemini 3.6 Flash ignores it, and GPT-6 accepts it only with reasoning effort set to none. Gate on the suite-level rate with its interval, and treat cases that flip between runs as findings. What is LLM regression testing? Running the same fixed set of cases before and after a change to the prompt, the model or the tools, and comparing the two runs case by case. The comparison is paired: both versions see identical inputs, so differences between cases cancel and a smaller change becomes detectable. How do you test an LLM model change? Send one request per code path to the new model first, to catch removed parameters. Then run the current model and the candidate on the same cases in the same session, with the prompt and tools unchanged. Compare per example, starting with tool selection and arguments, measure cost and latency per task, and decide against limits written before the run. What are the best LLM testing tools? It depends on the job: Promptfoo for assertions and comparing prompts or models, DeepEval for metrics inside pytest, Ragas for RAG, Inspect AI for sandboxed agent evals, Langfuse, LangSmith or Braintrust for production tracing with online evals, and EvalShift for gating a model or prompt change with a paired comparison. Keep reading - + How to catch LLM regressions before a model upgrade ships https://www.evalshift.dev/blog/llm-regression-testing-model-upgrade : the regression test built from three scripts, on a real upgrade. - + LLM regression testing in CI https://www.evalshift.dev/blog/llm-regression-testing-in-ci : the suite as a pull-request check. - + Evaluating agent tool calls https://www.evalshift.dev/blog/evaluating-agent-tool-calls : what a test that reads only the reply cannot see. - + LLM benchmarks https://www.evalshift.dev/blog/llm-benchmarks : why a model's leaderboard score does not replace a test on your own application.