cd /news/large-language-models/how-to-differential-test-llm-routing… · home › topics › large-language-models › article
[ARTICLE · art-172466] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=· neutral

How to Differential-Test LLM Routing Changes

A developer outlined a differential-testing approach for LLM gateway routers, comparing a reference router against a candidate across a contract table of dimensions—caller status, selected target, attempt sequence, stream state, upstream request shape, policy results and side effects—rather than relying on status codes alone. The method, drawn from LiteLLM's account of moving its routing to a Rust core, requires pinning reference and candidate builds and covering branch points like retryable 5xx, upstream 429s and connect failures. The author warns that latency and cost should stay out of the equality oracle unless the experiment controls provider, model, cache, concurrency, geography and pricing.

recommendation

When refactoring or upgrading an LLM router, run the same cases against pinned reference and candidate builds and compare normalized attempt sequences, targets and side effects—not just final status codes.

more in playbook: Evals and testing for LLM apps
by read8 min views1 publishedNov 19, 2026

An LLM router can return the same final 200 response after making a very different decision. It may choose another deployment, add a retry, cross a provider boundary, consume a fallback, or start a stream before a later failure. A status-code comparison will miss those regressions.

Differential testing runs the same case against a reference router and a candidate router, normalizes their observable results, and compares the behaviors that must remain stable. It is especially useful for a rewrite, version upgrade, configuration migration, or policy refactor whose stated goal is “change the implementation without changing the caller contract.”

LiteLLM described this approach in its November 11, 2026 account of moving routing to a Rust core: each behavior-preserving step received a differential test, and the comparison included more than the selected deployment. This article turns that engineering idea into a gateway-operator acceptance pack that works with any router whose decisions and upstream attempts can be observed.

The old and new routers do not need identical internal data structures or log text. They need equivalent outcomes in the dimensions your applications and operators depend on.

Start with a contract table:

Dimension Compare when Example normalized value
Caller status Every case 200 ,429 ,502
Caller-visible model or route The API exposes one chat-prod
Selected target Routing choice matters primary-us
Attempt sequence Retry or fallback is enabled primary-us, primary-us, backup-eu
Terminal stream state The response streams complete ,upstream_break ,client_cancel
Upstream request shape Translation or passthrough may change normalized method, path, headers, and body fields
Policy result Identity, budget, or guardrail applies allowed ,rate_limited ,blocked
Side effects The router writes cooldown, spend, or session state normalized state delta
Latency and cost Only for a separately designed benchmark distribution and method, not one exact value

Do not put latency or cost into an equality oracle unless the experiment controls provider, model, cache, concurrency, geography, and pricing. A differential contract test is not automatically a performance benchmark.

A useful test harness has four explicit pieces:

This vocabulary helps prevent a common mistake: comparing only the channel that is easiest to collect. If the final client response matches but the candidate sends three upstream attempts instead of one, the caller channel passes while the attempt and cost channels fail.

Record exact artifacts before collecting a result:

For AISIX, also record whether the alias is a direct model, multi-target model, or semantic router, and whether the deployment is standalone or managed. Pin the reference and candidate environments to explicit releases or build identifiers, because retry, fallback, session-affinity, and telemetry behavior can change between versions.

Do not compare “old production” with “whatever main does today.” A difference without pinned inputs is not reproducible.

Cover the branch points that can change a decision, not merely a list of successful prompts.

Case Upstream behavior What the oracle should inspect
Primary success First target returns a valid success Selected target, one attempt, caller contract
Retryable 5xx First attempt returns 503 , next succeeds Same-target retry count, fallback boundary, final status
Upstream 429 Target returns a rate limit Behavior with retry_on_429 off and on
Non-retryable 4xx Target returns 400 or401 Error returned without spraying invalid traffic to more targets
Connect failure Connection fails before response Retry and fallback sequence
First-byte timeout Stream has not committed bytes Whether another eligible target can be attempted
Mid-stream break Bytes have already reached the caller Terminal stream state; no claim of transparent failover after delivery
All targets unavailable Health or policy removes every target Configured 503 or documented try-anyway behavior
Caller not authorized Caller cannot use the alias or target Denial before upstream dispatch
Cooldown transition Repeated failure crosses a health threshold State delta and next-request selection
Alias or wildcard edge Requested name resolves indirectly Resolved target and caller-visible name
Empty or unusual input Empty string, long value, Unicode, or missing field Validation and stable error contract

Add cases for every custom policy: region tags, team overrides, price-based ranking, sticky assignment, semantic routes, provider-specific headers, or session state.

AISIX documents streaming retry boundaries that should become separate cases rather than one generic “streaming” row. In current routing documentation, a streaming request can move to another target before response bytes are committed but does not fail over after a stream has started. Verify the exact endpoint and version you operate.

Raw observations contain noise: request IDs, timestamps, generated response IDs, host ports, log order, and floating-point durations. Strip only fields that the contract declares irrelevant.

Keep values that reveal a semantic change:

Never normalize away a difference merely because it makes the test fail. If a generated ID is irrelevant, write that rule before viewing the candidate result.

The script below compares normalized JSON Lines produced by two test runs. Each line must contain a stable case_id. It ignores no fields by default; pass --ignore only for fields already classified as noise.


import argparse

import json

import sys

from pathlib import Path

def load(path: Path):

    rows = {}

    for number, line in enumerate(path.read_text().splitlines(), 1):

        if not line.strip():

            continue

        row = json.loads(line)

        case_id = row.pop("case_id", None)

        if not case_id:

            raise ValueError(f"{path}:{number}: missing case_id")

        if case_id in rows:

            raise ValueError(f"{path}:{number}: duplicate case_id {case_id}")

        rows[case_id] = row

    return rows

def remove_path(value, dotted_path):

    parts = dotted_path.split(".")

    cursor = value

    for part in parts[:-1]:

        if not isinstance(cursor, dict) or part not in cursor:

            return

        cursor = cursor[part]

    if isinstance(cursor, dict):

        cursor.pop(parts[-1], None)

def main():

    parser = argparse.ArgumentParser()

    parser.add_argument("reference", type=Path)

    parser.add_argument("candidate", type=Path)

    parser.add_argument("--ignore", action="append", default=[])

    args = parser.parse_args()

reference = load(args.reference) candidate = load(args.candidate) failed = False

for case_id in sorted(reference.keys() | candidate.keys()): if case_id not in reference: print(f"EXTRA candidate case: {case_id}") failed = True continue if case_id not in candidate: print(f"MISSING candidate case: {case_id}") failed = True continue

left = reference[case_id]
right = candidate[case_id]
for path in args.ignore:
    remove_path(left, path)
    remove_path(right, path)

if left != right:
    failed = True
    print(f"DIFF {case_id}")
    print("  reference:", json.dumps(left, sort_keys=True))
    print("  candidate:", json.dumps(right, sort_keys=True))

if failed: return 1 print(f"PASS: {len(reference)} cases are equivalent") return 0


if **name** == "**main**":

    sys.exit(main())

Create a passing reference:

{"case_id":"primary-200","status":200,"attempts":["primary"],"target":"primary","terminal":"complete"}
{"case_id":"primary-503-fallback","status":200,"attempts":["primary","backup"],"target":"backup","terminal":"complete"}

Then introduce a candidate regression:


{"case_id":"primary-200","status":200,"attempts":["primary"],"target":"primary","terminal":"complete"}

{"case_id":"primary-503-fallback","status":200,"attempts":["primary","primary","backup"],"target":"backup","terminal":"complete"}

Run the comparison:

python3 compare_routes.py reference.jsonl candidate.jsonl

The command exits 1 and reports the extra primary attempt. That difference matters even though the final status and target match. After correcting the candidate fixture to use the valid JSON array ["primary", "backup"], the command exits 0 and prints PASS: 2 cases are equivalent.

The comparator has intentionally narrow scope. It verifies the normalized artifacts you provide; it does not collect gateway traffic, decide which fields matter, or prove model quality.

For each case, collect evidence at the point where it is authoritative:

In AISIX 1.10.0, x-aisix-served-by is a debug header that requires proxy.expose_debug_headers: true, which is off by default and read at startup. Even when enabled, it appears only on qualifying successful /v1/chat/completions routing responses, not as a universal response field. Use x-aisix-request-id to correlate the request with the access log's served_by_model, or use the mock upstream capture, as the default serving-target oracle.

An upgrade may contain both behavior-preserving work and deliberate product changes. Label each case:

must_match: any difference is a regression; expected_change: reference and candidate should differ in a predeclared way;informational: record the result but do not gate the rollout yet. For an expected change, write a directional oracle. If the candidate is supposed to stop retrying authentication errors, the test should require one 401 attempt and no fallback, not merely allow any difference.

Keep the old implementation as a reference only while it represents the desired contract. Differential testing can preserve an existing bug perfectly. Pair equivalence with specification tests for behavior that is meant to be correct.

Run the reference against itself twice before trusting a reference-versus-candidate difference. If two identical builds disagree, the case is nondeterministic or the normalizer is incomplete.

Common sources of noise include:

Use isolated state, deterministic mock upstreams, fixed clocks or bounded time assertions, and one owner for retries. If randomness is the contract, compare distributions in a separate statistical test instead of forcing one request to match.

After deterministic cases pass, replay a permitted, sanitized sample of representative request shapes through the candidate in a non-serving path. Do not duplicate sensitive prompts or side-effecting tool calls without explicit authorization and controls.

Compare normalized routing decisions and upstream request shapes. Keep candidate responses away from users until the mismatch classes are understood. A shadow run is evidence about the sampled workload, not proof that every production edge case is covered.

Before switching routers or configurations, require all of the following:

Differential testing is valuable because it makes “behavior-preserving” falsifiable. For an LLM gateway, the observable behavior is larger than the final response: target choice, attempts, policy, streaming terminal state, and side effects are all part of the contract. Capture them explicitly before a router rewrite, upgrade, or configuration migration reaches production.

For an AISIX rollout, start with the current Routing and Failover documentation and adapt the matrix to the exact endpoint, strategy, and version you operate.

── more in #large-language-models 4 stories · sorted by recency
── more on @litellm 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-to-differential-…] indexed:0 read:8min 2026-11-19 · —