Our production chain is deterministic end to end: same configuration, same build, same app. That's what a platform promises. And a few months ago, we plugged into it the least deterministic component in existence: a language model, tasked with producing code that will be installed in a customer's app.
What happens to that code — how it fits in, what it's allowed to touch — is a story I've told elsewhere. This one is about the machinery around the model: what you have to build so that a component that never answers the same way twice can live inside a chain that isn't allowed to vary.
That's the first mental shift. When the model hands back its work, nothing treats it as a result: it's a proposal — and it's about to be inspected.
The rules are non-negotiable: one root container; no script sneaked into the HTML — external dependencies are declared in the open; HTTPS everywhere; only approved domains are reachable. None of this is politely requested in the prompt in the hope the model remembers. The prompt educates; the validator decides. Everything gets re-checked mechanically, after the fact, on every generation.
That's the difference between trusting a component and putting it under contract: you don't ask it to be reliable — you make its unreliability harmless.
When validation rejects a proposal, we don't call the model back right away. The response is tiered, cheapest first — because in production, every model call costs time and money, and a deterministic program that knows how to repair beats a regeneration that might.
At the bottom: mechanical repairs. Malformed JSON gets fixed without a model. A missing field in a revision gets compared against the parent: if everything else matches byte for byte, the missing field is inherited — the model had simply judged it unchanged, and it was right. And when one block is genuinely missing, we make a micro-call for that block, not for the whole section: far cheaper than a full regeneration.
Only then, correction by the model: the exact validation error goes back to it as feedback, and it gets two attempts. With one detail that looks bureaucratic and isn't: in correction mode, the model isn't allowed to answer with anything but a section. Without that lock, "I failed to build this" can dress up as "actually, that was a conversational question" — the failure changes costume, and a real regression in the generator becomes invisible in the metrics.
And when everything fails, we fail cleanly: the generation is marked failed with the original error — the one from the first check, which says what actually happened, not the last error on the retry pile — and a counter ticks up in telemetry. The user gets a simple message and what to do next; the technical detail is for us, not for them. A clean, countable failure beats a suspicious success.
Let's say it upfront: testing properties instead of outputs, writing evals, having one model grade another — none of that is original. It's standard equipment by now for anyone running an LLM in production. What's worth telling is where those tools apply — because everything hinges on an asymmetry of budgets.
In real time, with a user watching, every second and every call counts: you can afford neither best-of-N nor a judge. The budget goes to the loop above — mechanical repairs first, targeted corrections second. Offline, the equation flips: for pre-generated content the cost amortizes, so we can produce several candidates and let a judge model keep the best one — not the first one that clears the checks. Same tools, two economies.
And one habit that has genuinely paid off: when a failure mode keeps coming back, it gets a name and a harness — a set of replayable cases, not a ticket to close. The most puzzling example: prompts that fail, then succeed, under identical conditions. You don't "fix" that — it isn't a bug, it's the nature of the component. Named and replayable, it stops being team superstition and becomes engineering data.
One more floor on the same logic: there isn't a model, there are roles. Generating, judging, translating don't run on the same model; each role pins its own, and changing one is a decision you make — not an event that happens to you. Each role's prompt lives in the code, versioned and reviewed like everything else: it's an API surface, in the fullest sense.
Put end to end, this adds up to a decidedly unromantic position: we don't treat the model as a brain — we treat it as a vendor. A talented, unpredictable vendor that never gets to deliver straight to the shelf: goods-in inspection, quality control, non-conformities sent back with the delivery slip, and a fallback route for when the shipment doesn't arrive.
It's less thrilling than "AI writes code." But it's what lets a component that never answers the same way twice live inside a chain that isn't allowed to vary — and it's the loop, not the generation, that makes it possible.