{"slug": "shipping-a-component-that-never-answers-the-same-way-twice", "title": "Shipping a component that never answers the same way twice", "summary": "A developer at GoodBarber describes how they integrated a language model into a deterministic production chain, treating its output as a proposal subject to mechanical validation and tiered repairs rather than as a final result. The system uses cheap mechanical fixes first, then targeted model corrections, and fails cleanly with countable errors to avoid masking regressions.", "body_md": "Our production chain is deterministic end to end: same configuration, same build, same app. That's what a platform promises. And a few months ago, we plugged into it the least deterministic component in existence: a language model, tasked with producing code that will be installed in a customer's app.\n\nWhat happens to that code — how it fits in, what it's allowed to touch — is [a story I've told elsewhere](https://dev.to/goodbarber/we-let-an-ai-write-code-inside-our-no-code-platform-generating-it-was-the-easy-part-1p65). This one is about the machinery around the model: what you have to build so that a component that never answers the same way twice can live inside a chain that isn't allowed to vary.\n\nThat's the first mental shift. When the model hands back its work, nothing treats it as a result: it's a proposal — and it's about to be inspected.\n\nThe rules are non-negotiable: one root container; no script sneaked into the HTML — external dependencies are declared in the open; HTTPS everywhere; only approved domains are reachable. None of this is politely requested in the prompt in the hope the model remembers. The prompt educates; the validator decides. Everything gets re-checked mechanically, after the fact, on every generation.\n\nThat's the difference between trusting a component and putting it under contract: you don't ask it to be reliable — you make its unreliability harmless.\n\nWhen validation rejects a proposal, we don't call the model back right away. The response is tiered, cheapest first — because in production, every model call costs time and money, and a deterministic program that knows how to repair beats a regeneration that might.\n\nAt the bottom: mechanical repairs. Malformed JSON gets fixed without a model. A missing field in a revision gets compared against the parent: if everything else matches byte for byte, the missing field is inherited — the model had simply judged it unchanged, and it was right. And when one block is genuinely missing, we make a micro-call for that block, not for the whole section: far cheaper than a full regeneration.\n\nOnly then, correction by the model: the exact validation error goes back to it as feedback, and it gets two attempts. With one detail that looks bureaucratic and isn't: in correction mode, the model isn't allowed to answer with anything but a section. Without that lock, \"I failed to build this\" can dress up as \"actually, that was a conversational question\" — the failure changes costume, and a real regression in the generator becomes invisible in the metrics.\n\nAnd when everything fails, we fail cleanly: the generation is marked failed with the original error — the one from the first check, which says what actually happened, not the last error on the retry pile — and a counter ticks up in telemetry. The user gets a simple message and what to do next; the technical detail is for us, not for them. A clean, countable failure beats a suspicious success.\n\nLet's say it upfront: testing properties instead of outputs, writing evals, having one model grade another — none of that is original. It's standard equipment by now for anyone running an LLM in production. What's worth telling is where those tools apply — because everything hinges on an asymmetry of budgets.\n\nIn real time, with a user watching, every second and every call counts: you can afford neither best-of-N nor a judge. The budget goes to the loop above — mechanical repairs first, targeted corrections second. Offline, the equation flips: for pre-generated content the cost amortizes, so we can produce several candidates and let a judge model keep the best one — not the first one that clears the checks. Same tools, two economies.\n\nAnd one habit that has genuinely paid off: when a failure mode keeps coming back, it gets a name and a harness — a set of replayable cases, not a ticket to close. The most puzzling example: prompts that fail, then succeed, under identical conditions. You don't \"fix\" that — it isn't a bug, it's the nature of the component. Named and replayable, it stops being team superstition and becomes engineering data.\n\nOne more floor on the same logic: there isn't a model, there are roles. Generating, judging, translating don't run on the same model; each role pins its own, and changing one is a decision you make — not an event that happens to you. Each role's prompt lives in the code, versioned and reviewed like everything else: it's an API surface, in the fullest sense.\n\nPut end to end, this adds up to a decidedly unromantic position: we don't treat the model as a brain — we treat it as a vendor. A talented, unpredictable vendor that never gets to deliver straight to the shelf: goods-in inspection, quality control, non-conformities sent back with the delivery slip, and a fallback route for when the shipment doesn't arrive.\n\nIt's less thrilling than \"AI writes code.\" But it's what lets a component that never answers the same way twice live inside a chain that isn't allowed to vary — and it's the loop, not the generation, that makes it possible.", "url": "https://wpnews.pro/news/shipping-a-component-that-never-answers-the-same-way-twice", "canonical_source": "https://dev.to/goodbarber/shipping-a-component-that-never-answers-the-same-way-twice-5eo8", "published_at": "2026-07-28 12:41:58+00:00", "updated_at": "2026-07-28 13:04:41.220108+00:00", "lang": "en", "topics": ["large-language-models", "ai-products", "developer-tools", "ai-infrastructure"], "entities": ["GoodBarber"], "alternates": {"html": "https://wpnews.pro/news/shipping-a-component-that-never-answers-the-same-way-twice", "markdown": "https://wpnews.pro/news/shipping-a-component-that-never-answers-the-same-way-twice.md", "text": "https://wpnews.pro/news/shipping-a-component-that-never-answers-the-same-way-twice.txt", "jsonld": "https://wpnews.pro/news/shipping-a-component-that-never-answers-the-same-way-twice.jsonld"}}