# When Validators Matter More Than Better LLMs > Source: > Published: 2026-04-27 06:00:00+00:00 # When Validators Matter More Than Better LLMs In a multi-step AI pipeline, the dangerous failure is not just a bad model output. It is a bad intermediate result becoming trusted state. Validators matter because they stop bad state at the boundary. Most agent reliability work starts with the generator: better prompts, larger models, [longer context windows](https://arizenai.com/context-window-fallacy/), maybe fine-tuning. That is often useful, but it is not always the highest-return place to spend engineering effort. In a multi-step system, the dangerous failure is not just that one model call is wrong. It is that a wrong intermediate result becomes trusted state for the next step. A classifier picks the wrong intent, an extractor returns a malformed field, a router calls the wrong tool, and the final answer looks coherent because every later step is now reasoning from corrupted input. The **Validator Asymmetry Principle** is narrow: when correctness can be expressed as a predicate, a deterministic validator at the boundary can buy more reliability than another round of generator optimization. The validator does not make the model smarter. It prevents bad state from quietly moving forward. **TL;DR - Key Takeaways:** - In sequential agent pipelines, errors multiply because every downstream step inherits upstream state. - A validator matters when it catches invalid output before that output becomes input to the next step. - The common 95% / 99% framing is illustrative, not a benchmark: the real question is whether bad state is detected and contained. - Useful validators are deterministic, independently testable, and placed at step boundaries. - If you cannot write a check for bad output, you probably do not yet have a specification. ## A Small Reliability Model Here is the toy model that makes the asymmetry visible. Assume three independent generator steps, each with a 95% chance of producing acceptable output for its local task. Ungated end-to-end success is: **Ungated chain:** 0.95 x 0.95 x 0.95 = **0.857** **If all three generators improve to 0.99:** 0.99 x 0.99 x 0.99 = **0.970** **If a validator catches 95% of bad outputs:** silent bad-state rate per boundary falls from 0.05 to 0.0025 **caught failures can retry, route, or stop instead of propagating** Those numbers are not a production benchmark. They assume independent steps, a clear definition of "bad output", and a validator whose error-detection recall is known. The point is more basic: a better generator improves the probability of producing good output; a boundary validator changes whether bad output is allowed to become downstream state. That distinction matters because "caught failure" and "silent corruption" are not the same failure mode. A caught failure can retry, route to a human, return a safe error, or dead-letter the job. Silent corruption keeps running. ## Why The Gate Matters A generator and a validator have different engineering properties. A generator is probabilistic. You can improve it with examples, prompts, fine-tuning, model choice, and tool design, but you do not usually get a simple invariant like "this output is now always valid." Even at temperature zero, model behavior can vary across provider changes, model revisions, context changes, and tool output. A validator can be a normal program. It can check whether JSON matches a schema, whether an enum is closed, whether an amount is non-negative, whether a date falls inside an allowed interval, whether a cited document ID exists, or whether a tool call is permitted in the current state. Those checks are boring. That is why they are valuable. The gate also has local feedback. If the validator fails, the generator receives a precise error: missing field, invalid enum, date out of range, unauthorized tool, confidence below threshold. That is a better retry signal than "try again, but be more careful." This is the same control instinct behind [the supervisor pattern](https://arizenai.com/the-supervisor-pattern/): put a narrower layer where the system can stop, retry, or route before damage becomes state. **A validator does not make the model deterministic. It makes the boundary deterministic.** ## What Makes A Validator Real Not every check deserves to be called a validator. A useful boundary validator has three properties. **Determinism.** Given the same input, it returns the same verdict. A [second LLM judging the first LLM](https://arizenai.com/seven-models-judged-each-other/) may be useful as a critic, but it has not broken the stochastic chain. It is another probabilistic step. **Independent testability.** The validator can be tested against fixed examples without running the generator. That makes it a small engineering artifact rather than an intuition about model behavior. **Boundary placement.** It runs before state crosses into the next step. End-of-pipeline validation is still useful, but it catches problems after corrupted state has already consumed tokens, triggered tools, or shaped downstream reasoning. A minimal validator for a structured routing step looks like this: ``` python from typing import Literal from pydantic import BaseModel, field_validator Intent = Literal["book_flight", "cancel_reservation", "check_status"] class ClassifiedIntent(BaseModel): intent: Intent confidence: float extracted_entity: str @field_validator("confidence") @classmethod def confidence_must_be_usable(cls, value: float) -> float: if not 0.0 <= value <= 1.0: raise ValueError("confidence must be between 0 and 1") if value < 0.7: raise ValueError("confidence below routing threshold") return value ``` This validator is not checking whether the model is globally intelligent. It is enforcing a contract between one state and the next. The generator can change. The boundary should still say what is allowed to pass. ## Prior Art: This Is Old Software Engineering This is not a new idea. Type systems, parsers, assertions, database constraints, schema validators, circuit breakers, workflow engines, and test-driven development all make versions of the same argument: do not let invalid state move freely through a system. The useful update for LLM systems is where the old idea gets placed. A model call looks like a function call, but its output is sampled text. That makes every model boundary a place where traditional software invariants need to re-enter the system, especially once the workflow starts to behave like a [probabilistic state machine](https://arizenai.com/probabilistic-state-machine/). ## Where This Breaks Validators work best when correctness can be specified. They work poorly when the desired output is subjective, ambiguous, or impossible to reduce to a computable predicate. A schema can prove that a field exists. It cannot prove that a summary is fair. A regex can prove that a date has the right shape. It cannot prove that the date is the one the user meant. A rubric can help evaluate tone, but once the rubric requires judgment, the validator becomes a scoring model or a reviewer, not a deterministic gate. Validators can also create false confidence. A system with a 99% pass rate on the wrong validator is not reliable. It is reliably enforcing the wrong specification. **Validator design is specification design: the check only protects the system when it encodes the failure mode that matters.** Distribution shift is another failure mode. When prompts, model versions, tool outputs, or user populations change, validators need regression tests too. A stale validator can reject newly valid output or pass a failure mode it was never designed to see. ## The Validator Is The Specification The uncomfortable part is that validators force a team to say what correct means. That is why they are often missing. It is easier to ask the model to "extract the relevant information" than to define which fields are required, which values are legal, which uncertainties should escalate, and which downstream actions are forbidden. But that definition is the engineering work. Once the validator exists, the model has a target, the retry loop has useful feedback, and the workflow has a boundary where bad state can be stopped. The generator remains probabilistic. The system does not have to be.