Can Provider Routing Change LLM Outputs?
Provider routing can change an LLM's output when a request reaches a different model version, fallback model, parameter configuration, precision level, inference…
Test an LLM application as a workflow that must meet behavioral requirements across repeated runs, not as a function that returns one exact string. The application should produce acceptable answers, make authorized tool calls, preserve required data, and reach the correct external state even when wording or execution paths vary.
Nondeterminism means that the same input can produce different outputs or actions. Sampling is one cause, but provider implementation details, model updates, retrieval results, tool responses, concurrency, retries, and multi-step agent decisions also introduce variation. Temperature zero and fixed seeds reduce some variation, but they do not guarantee identical execution. Research has found residual nondeterminism under those settings for some models, so testing must measure behavior rather than assume reproducibility from configuration alone (Towards Reproducible LLM Evaluation).
The practical approach is to control the inputs and runtime where possible, run important cases repeatedly, evaluate outcomes and side effects, and combine deterministic checks with model-based and human review.
Reproducibility asks whether a test produces the same or equivalent result when repeated under stated conditions. Reliability asks whether the application succeeds often enough and safely enough for its intended use.
An application can be non-reproducible but reliable. A support assistant might use different wording on every run and still give the correct policy-based answer each time. It can also be reproducible but unreliable if it consistently returns the wrong answer or performs an unauthorized action.
Define reliability using observable outcomes such as:
Do not make identical wording the default success condition.
Write a behavioral contract for each important workflow. The contract states what the application must do, must not do, and is allowed to vary on.
Separate hard requirements from soft qualities.
Hard requirements are non-negotiable:
Soft requirements can receive a score rather than a binary result:
Parse structured output before grading the response. Check required fields, data types, allowed values, numeric ranges, cross-field constraints, and business rules. Two JSON responses with different whitespace or field order should pass if they represent the same valid result.
List every component that can change a trial’s result:
Temperature is only one contributor. A fixed prompt can still produce a different result when the retrieved documents change, an API returns a different record, or a retry takes a different path.
The unit under test should usually be the complete workflow, not only the model call. That workflow includes prompt construction, conversation state, retrieval, tools, business logic, safeguards, storage, retries, and the user-facing response.
Anthropic describes an agent harness as code surrounding the agent that processes inputs, orchestrates tools, records interactions, and returns the final result. Testing only the model response misses failures in that harness (Demystifying evals for AI agents).
Use conventional software tests for components that should be deterministic:
These tests should run quickly on every code change. They prevent the model from receiving malformed context and prevent model output from bypassing application safeguards.
Then test model behavior and orchestration statistically. For example, a structured extraction workflow might use the following layers:
This separation makes failures easier to diagnose. A malformed database write is an application defect even if the model produced correct text. A wrong classification after valid parsing belongs to the model or orchestration evaluation.
For agents and state-changing workflows, inspect the entire execution:
Never treat the model’s claim that it completed a task as proof that it completed the task. An agent can say that it canceled an order while the cancellation API failed, or claim that it updated a record while a validation rule rejected the write.
For a refund workflow, a passing trial should require all of the following:
A transcript alone cannot verify the last two conditions.
Create a test corpus that reflects real usage, important business workflows, known failures, and unsafe conditions. Each case should include an input, expected behavioral requirements, and a grading method.
Combine several sources:
Production-derived examples expose assumptions that curated tests miss. Google’s evaluation documentation describes using production logs and synthetic examples to build evaluation datasets (Google Cloud evaluation overview). Redact personal data, restrict access, encrypt stored traces, and define a retention period before using production records.
Synthetic cases expand coverage but do not replace real examples. A generator can reproduce the same assumptions and blind spots as the application under test.
Test both when an action should occur and when it should not.
For a customer-service agent, include cases where it should:
Also include cases where it should not:
A system that always refuses can pass a refusal-heavy test set while failing its actual purpose. Balance successful-action cases with refusal, abstention, and escalation cases.
Test direct and indirect attacks, including instructions embedded in retrieved documents, email, webpages, uploaded files, tool results, and database fields.
Cover:
The OWASP LLM risk list identifies prompt injection, sensitive information disclosure, improper output handling, excessive agency, system-prompt leakage, and vector or embedding weaknesses as application risks.
Test the boundary between model output and executable behavior. If generated text reaches SQL, shell commands, HTML, email, file systems, or privileged APIs, validate and authorize it with ordinary code before execution. A refusal from the model is not a substitute for access control.
For a retrieval-augmented generation application, test retrieval and generation as separate failure surfaces.
Measure retrieval quality:
Then measure the generated answer:
A factually correct answer can still fail if it cites the wrong document or uses evidence the user is not allowed to access. Grade access control, grounding, and citation accuracy separately.
A single successful run hides the distribution of possible outcomes. Run important cases multiple times with the same controlled inputs, then record how frequently each behavior occurs.
Use more trials when:
A 2024 study found that three repeats were often enough for a particular prediction-interval target under its tested temperature-zero, fixed-seed conditions. That result does not establish a universal repeat count for production applications because variability depends on the model, benchmark, provider, and workflow (Towards Reproducible LLM Evaluation).
Reduce avoidable variation during regression testing with:
Treat these controls as measurement aids, not guarantees of determinism. The reproducibility research found that some models remain variable even with temperature zero and a fixed seed.
Run a second suite with realistic variation when that variation exists in production. A fully frozen test can show regression behavior while missing failures caused by changing documents, external APIs, time, or concurrent requests.
For each test case, record:
A basic estimate is:
success rate = successful trials / total trials
Report an uncertainty interval with the rate. A result of 9 successful trials out of 10 does not provide the same evidence as 90 out of 100, even though both have a nominal 90% success rate.
Track failure categories separately. For example, “answer incorrect,” “citation unsupported,” “tool unauthorized,” and “database write failed” should not collapse into one aggregate score. Hard failures should remain visible even when average quality improves.
Pass@1 measures whether the first attempt succeeds.
Pass@k is useful when the application safely generates multiple candidates in an isolated setting. It is not a substitute for first-attempt reliability when each attempt can send an email, charge a card, modify a record, or delete data.
For state-changing workflows, prioritize:
The probability of safe, correct completion on the first authorized attempt.
Retries require separate testing. A retry after a timeout can duplicate a payment or create two records unless the operation is idempotent, meaning repeated requests produce the same final effect. Test timeout, partial failure, and retry scenarios against a sandbox or a transaction-safe test environment.
Match each evaluator to the requirement. No single grader should judge schema validity, tool authorization, groundedness, tone, and safety in one opaque score.
Use code-based checks for requirements with an objective answer:
These checks are fast and reproducible. They become brittle when several answers are valid, so normalize structured data and compare semantic fields rather than raw text.
Use a model-based grader for qualities that depend on semantic judgment, including relevance, completeness, groundedness, instruction following, and helpfulness. Give it a rubric with explicit pass and fail conditions, and provide examples of borderline cases.
Test the grader itself by:
Model graders are inference systems, so their judgments also vary. OpenAI’s grader documentation exposes sampling controls for grader configurations, which reinforces the need to evaluate grader repeatability rather than treating its score as ground truth.
Use human reviewers to calibrate rubrics, resolve disagreements, and audit high-impact outcomes. Reviewers should examine safety, privacy, fairness, and ambiguous cases, not only average-quality examples.
A practical pattern is:
A failure is actionable only when the recorded evidence shows what happened. Capture the context needed to distinguish a model change from a prompt, retrieval, tool, infrastructure, or application change.
Record, where applicable:
test_case_id
user_input
conversation_state
prompt_versions
model_identifier
model_version_or_snapshot
sampling_settings
seed_if_supported
retrieved_documents
tool_definitions
tool_inputs_and_outputs
full_transcript
retry_and_timeout_events
final_output
external_state_changes
grader_versions
latency
token_usage
cost
timestamp
deployment_and_runtime_details
For an agent, preserve the tool sequence, intermediate messages, and final response. For a retrieval workflow, preserve document identifiers and the retrieved text or a privacy-safe representation that permits later inspection.
Evaluation traces often contain user messages, private documents, credentials accidentally included in tool output, or sensitive business data. Apply redaction, access controls, encryption, retention limits, and privacy review to both test and production logs.
Use synthetic substitutes for secrets and personal data whenever they preserve the behavior under test. Reproducibility does not justify indefinite retention of sensitive user content.
Run evaluations after changing code, prompts, models, retrieval data, tool definitions, safety policies, or provider configuration. A prompt-only change can alter tool behavior, and a retrieval-index update can change answers without any model change.
A release gate should include:
Compare results with a pinned baseline. Set explicit thresholds for hard failures, first-attempt success, quality scores, latency, and cost. Do not ship a release that improves a soft quality score by introducing an unacceptable safety or authorization failure.
Document residual risks and properties that remain unmeasured. NIST’s AI Risk Management Framework recommends testing before deployment and regularly during operation, with documented metrics, uncertainty, benchmarks, and independent review.
Production monitoring should track:
Sample transcripts for manual review under strict privacy controls. When a production failure reveals a missing case, add a minimized and de-identified version to the regression corpus. Maintain incident-response and rollback procedures for model, prompt, retrieval, and tool changes.
No. Temperature zero reduces sampling variation, and a fixed seed can reduce it further, but some models still produce different results under those settings. Test repeatability under the actual provider, model, prompt, retrieval, and tool conditions you deploy (Towards Reproducible LLM Evaluation).
There is no universal repeat count. Use more trials for high-risk cases, rare serious failures, unstable graders, and results near a release threshold. A small smoke test can use fewer trials, but its result should not be treated as strong evidence of production reliability.
Use exact matching only when exact text is a real requirement, such as a fixed command or a narrow classification label. For open-ended responses, check required facts, structured meaning, citations, policy compliance, tool behavior, and final state instead.
Yes, model-based graders can scale semantic evaluation, but they are themselves variable and can contain bias. Calibrate them against expert judgments, repeat borderline grading tasks, and keep deterministic checks and human review for hard or high-impact requirements (Demystifying evals for AI agents).
Test the full agent harness, including input handling, tool selection, arguments, intermediate actions, retries, final responses, authorization, and environmental side effects. Verify the resulting database or external state instead of trusting the agent’s statement that it completed the task.
Maintain continuous evaluation, monitoring, transcript review, incident response, and regression-case creation. A one-time benchmark cannot detect failures caused by new model versions, changing retrieval data, provider behavior, tool responses, or real user inputs.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com