cd /news/large-language-models/llm-call-is-doing-more-than-you-thin… · home › topics › large-language-models › article
[ARTICLE · art-172553] src=seldon-ai.com ↗ pub= topic=large-language-models verified=true sentiment=· neutral

LLM call is doing more than you think

Audiala replaced an LLM router with a decision model and rolled it back 23 minutes later because the replacement classified intent but dropped city resolution, conversational reference handling and tool-control flags, according to a blog post surveying decision-model adoption. The post argues the unit of replacement should be the behavior an application depends on, illustrated with an invoice-processing function that must return Matched, NeedsReview or NoMatch outcomes rather than a bare vendor_id. It separates three standards — matching the output schema, agreeing with the incumbent, and meeting the application's requirements — and warns that agreement can preserve an incumbent's mistake.

recommendation

Trace every field, tool argument and downstream message the LLM call feeds before replacing it, and require the replacement to return explicit Matched, NeedsReview and NoMatch outcomes so lost behaviors surface in tests rather than production.

more in playbook: Designing agent systems
by read11 min views7 publishedNov 19, 2026
LLM call is doing more than you think
Image: source

← Blog

Audiala replaced an LLM router with a decision model and rolled it back 23 minutes later. The replacement could classify intent. The incumbent also resolved cities, conversational references, and tool-control flags. The team had evaluated one responsibility and replaced several.

The migration account was one of the clearest examples in Part 1, our survey of decision-model adoption. Other integrations exposed related problems: candidate construction constrained action selection, missing handoff instructions weakened downstream agents, and faster decisions sometimes produced more recovery work. Those reports describe particular implementations, not a universal penalty for specialization.

They do suggest a practical starting point: the unit of replacement is the behavior the application depends on. This article follows that idea into the implementation. We will recover a call's contract, construct alternatives, evaluate the complete replacement, and account for keeping it reliable.

Consider an illustrative invoice-processing application with this interface:

resolve_vendor(invoice, catalog) → vendor_id

It appears to select a vendor. But which entity counts as the vendor: the issuing legal entity, a trading name, or the parent company? What happens when two catalog entries share a name? Does a tax identifier override the letterhead? Can the function return an entity absent from the catalog? Does a successful result merely populate a review screen, or does downstream code use it to choose a payment destination?

The original prompt may have absorbed several of those responsibilities. It might interpret the invoice, reconcile conflicting evidence, identify an entity, and explain uncertainty in a field the next step reads. A replacement that returns a valid identifier can still lose behavior the application needs.

Start with the consumers. Inspect field access, database lookups, tool arguments, intermediate messages, and recovery paths. Read the prompt alongside those dependencies. A field ignored by the immediate caller may still become evidence in a later model call. Conversely, a requested explanation nobody consumes may not belong in the required output.

Part 1's LAS case makes this concrete: planners supplied handoff instructions, and reviewers supplied repair guidance. A finite choice could replace one judgment while removing information needed downstream. If that information remains necessary, preserve it or assign its production to another step.

Recover the contract before choosing a model #

For our example, suppose the application owner confirms that the function must resolve the issuing legal entity within a supplied catalog snapshot, preserve supporting evidence, and refer unresolved ambiguity for review. A conceptual result type could be:

Resolution =
    Matched(vendor_id, catalog_version, evidence)
  | NeedsReview(reason, candidate_ids)
  | NoMatch(reason)

Each outcome needs semantics. Matched requires a member of the supplied catalog and evidence for choosing it. NeedsReview includes conflicting identifiers, ambiguous entities, and insufficient evidence. Here, NoMatch means the available evidence establishes that no catalog entry qualifies under the matching policy. An empty shortlist alone cannot establish that; retrieval may have missed the answer.

This changes the caller too. It must handle review and no-match outcomes explicitly instead of treating every syntactically valid identifier as success. Existing authorization checks still decide what a match permits the application to do.

Keep three standards separate: matching the output schema, agreeing with the incumbent, and meeting the application's requirements. Agreement can preserve an incumbent's mistakes. Historical traces help identify common inputs and dependencies, but cannot establish every required behavior. A previously accepted wrong vendor is evidence of a failure, not a new specification.

Record which requirements came from application policy or adjudicated examples and which remain inferred. Resolve consequential ambiguity before measuring candidates. Calling this work “refactoring” has a limit: learned replacements can change answers and error distributions. The goal is to meet declared requirements with evaluated tolerances; universal equivalence to the incumbent is usually not established.

Construct alternatives around the responsibilities #

Begin with the cheapest experiment to implement: send the existing request to a cheaper general model. A useful result may require no decomposition. Compare it with alternatives that make the recovered responsibilities explicit:

Implementation What changes Main assumption to test
Same prompt, cheaper model Model choice The replacement handles the full contract
Extraction, retrieval, specialist matching Evidence and candidate selection become separate stages Extraction and retrieval preserve the information needed to match
Guarded exact lookup, then semantic matching for unresolved inputs Some requests avoid learned matching The identifier is authoritative, unique in scope, and acceptable under the matching policy

An exact lookup is justified only under those preconditions. Text that resembles a tax identifier is not automatically authoritative. A catalog alias can also be shared by several legal entities. The implementation needs an unresolved path when its assumptions fail.

The candidate set is part of the program

Part 1's Stagehand example showed why action selection depends on candidate construction. The same issue appears here: a specialist cannot select the correct vendor if retrieval excludes it. For cases with a correct catalog match, on a path that must choose from its retrieved candidates:

P(correct match)
  = P(correct vendor retrieved)
    × P(correct selection | correct vendor retrieved)

Illustratively, 95% retrieval coverage and 98% conditional selection accuracy yield 93.1% correctness on that population. These are hypothetical numbers, but the implication is operational: evaluate retrieval separately and together with selection. A strong classifier score measured only on lists containing the answer hides an upstream failure mode.

Preserve candidate identities through the stages. If the model evaluates pairs, attach each judgment to a pair ID rather than assuming responses retain their order. Similarly, extracting a list of companies and a list of addresses does not establish which address belongs to which company. Those relationships need evidence and explicit representation.

Separate judgment from mechanical work

Some responsibilities can become ordinary computation once their inputs are explicit. A model can identify relevant line items while a decimal-arithmetic kernel computes their total, or quote supporting text while a grounding kernel locates it in the source. In vendor resolution, deterministic operations can construct candidate pairs, preserve their IDs, and assemble match decisions without guessing relationships from list order. The model interprets evidence; code performs the declared transformations. These operations also make intermediate results available for inspection and testing.

Determinism does not establish that the interpretation was correct. An accurately located quotation can carry the wrong label, an exact name match can identify the wrong legal entity, and a correct sum can use incorrectly extracted operands. Failure policy changes the result too: rejecting an ungrounded quotation may trigger expensive fallback, while dropping it may omit something important. For redaction, that omission could expose sensitive information. Evaluate those choices as part of the replacement, including completeness and downstream errors; deterministic operations remove particular failure modes, not the need for semantic evaluation.

Evaluate the complete replacement #

Define the acceptance criteria before tuning the candidates. In invoice matching, wrong automatic matches, unnecessary reviews, and unresolved invoices have different consequences. Count them separately. A higher aggregate score does not decide whether extra incorrect matches are acceptable in exchange for fewer reviews.

Use three layers of checks:

Layer What it establishes
Contract checks Outputs have the required fields, identities refer to the correct catalog, and unresolved outcomes are handled
Paired function evaluation Complete implementations are compared on the same inputs for correctness, consequential errors, unresolved results, cost, and latency
Workflow execution The changed output works through subsequent state transitions, recovery, and task completion

Build examples around the failure mechanisms, not only common successful invoices. Include a correct vendor omitted from retrieval, similar parent and subsidiary names, conflicting identifiers, a vendor absent from the catalog, and insufficient evidence. Reorder candidates and vary equivalent wording to test whether incidental presentation changes the answer. Include untrusted document text that attempts to direct the model; schema validation does not establish resistance to such instructions.

Use development data to construct candidates and tune prompts or thresholds. Freeze the chosen implementation before final held-out evaluation. Group related invoices or templates when necessary to prevent near duplicates crossing the split. If the intended claim is generalization to unseen vendors, the split must test that too. Repeated runs measure variability on the same examples; they do not create additional task coverage.

Historical answers can support disagreement analysis, but meaningful correctness claims need an appropriate authority: adjudicated matches, trusted identifiers, or verified outcomes. Report the source of the labels and uncertainty in consequential error estimates, especially when important failures are rare. Zero observed errors on a small set does not establish a zero error rate.

Replay does not cover every trajectory

For a bounded resolver, replay against captured invoices and catalog snapshots is useful. For an agent controller, replaying incumbent traces evaluates states the incumbent reached. A changed action can lead to different states, arguments, and recovery paths that never appear in those traces.

Part 1's complete-task comparisons illustrate why this matters: improved component speed or median episode time did not consistently preserve success and tail latency. Exercise the affected workflow in a controlled environment with external effects contained. For the invoice application, follow ambiguous results into review and failed matches into recovery; for an agent, execute the subsequent trajectory. Measure completed tasks and retain failed attempts in the accounting.

Acceptance, fallback, and permission are separate decisions #

A confidence threshold answers only part of the deployment question. Distinguish three decisions:

  • Applicability: are this implementation's assumptions satisfied, including input format and catalog scope?
  • Acceptance: does this particular result meet the policy for automatic use?
  • Authorization: may the application perform the resulting action?

A model-generated assertion that a customer granted permission is not a substitute for trusted authorization state. Part 1's uncertainty and trust cases show why high confidence cannot supply missing instructions or protect the evidence channel.

Evaluate an acceptance policy using both coverage, the fraction of requests it accepts, and error among accepted answers. Accepting very few cases can make the specialist look accurate while sending almost everything to an expensive fallback. Also report errors after fallback, unresolved outcomes, total cost, and latency. The policy's complete behavior determines its usefulness.

Fallback cannot repair an error the system accepts. It helps when a detected failure or rejected result sends the request elsewhere. A fallback model can still share the original misunderstanding. Where practical, let it inspect the original evidence; sending only the incomplete shortlist can preserve the cause of the error.

Specify boundary behavior: a score exactly at the threshold, missing fields, invalid identities, timeouts, and an exhausted fallback budget. Part 1's Fastbrowse threshold issue is a reminder that a small branching mistake can defeat an otherwise careful evaluation. These cases deserve direct tests, independent of average model accuracy.

Count the cost of keeping the result valid #

Compare economics among implementations that satisfy the required outcome constraints. For a defined workload and period:

Net benefit over the period
  = execution savings
    − migration and evaluation cost
    − maintenance and revalidation cost

Execution savings must include both specialist attempts and fallback, retries, serving overhead, and any changed review or recovery work. Distinguish cost per attempt from total spending divided by successful tasks; the latter keeps failed attempts in the numerator. A high-volume stable function may recover its setup cost quickly. An occasional or rapidly changing task may not.

Treat a measured revision as more than a model name. Record its prompt, schemas, candidate builder, catalog or index version, acceptance policy, evaluation data, and evaluator. Changing retrieval can change candidate difficulty and score distributions while leaving the model untouched. The earlier threshold evidence may no longer support the new implementation. Check which assumptions changed and rerun the relevant evaluation before relying on the old result.

Instrument the revision and branch that actually execute, including fallback and subsequent recovery. Part 1's integration-path reports show why this matters: a host change can bypass the measured implementation without changing its model. Retain a usable previous revision and an explicit rollback path. New measurements can inform a deployment decision without automatically replacing the implementation currently serving requests.

From a measured replacement to a maintained function #

These requirements motivate Seldon's view of Semantic Compute: an application should define a semantic function whose implementation can evolve under an explicit contract and measured tradeoffs. Today, Seldon Functions starts with a developer-declared request or supported short workflow, searches supported alternatives—including model calls combined with predefined deterministic operations—and hosts the revision the developer chooses to activate. Its measurements concern that bounded function; the surrounding application still needs its own validation.

In the next article, we'll follow one Function through that process: freezing its contract, constructing executable candidates, separating search from held-out evaluation, and serving a selected revision with bounded fallback. We'll examine what the evidence establishes, where it remains incomplete, and when keeping the original implementation is the right result. The practical test is whether this reduces the work required to reach and maintain an acceptable replacement.

Route your traffic through Seldon and investigate recurring work that could benefit from a specialized program.

Sign up for open beta →

── more in #large-language-models 4 stories · sorted by recency
── more on @audiala 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/llm-call-is-doing-mo…] indexed:0 read:11min 2026-11-19 · —