{"slug": "lesson-4-validation-a-must-have-cornerstone", "title": "Lesson 4 - Validation: A must-have cornerstone", "summary": "A developer building Slooster, an AI-powered vendor matching tool, discovered that LLM outputs can degrade unpredictably in production, returning placeholder data like 'Vendor A' despite weeks of clean local results. This led to implementing an inline validation layer with retries and a model-as-judge gate to catch semantic errors, alongside intelligent caching and per-prompt model selection to manage costs and latency. The developer emphasizes that validation must be built into AI systems from day one, not added after the MVP.", "body_md": "In a traditional SDLC, validation was someone else's job and it came later. QA teams checked that the software did what it was supposed to. The behavior was repeatable, so eventually you automated the checks and moved on.\n\nSo when building Slooster, I did the hard parts first:\n\nEverything worked great locally, so **I told myself the validation layer could wait**.\n\nUnfortunately, the traditional SDLC model breaks the moment an LLM enters the equation. Remember, LLMs are producing output based on guessing off the context. They're getting better at it, no doubt, but they are still guessing. The same prompt will give you a different answer next time, or even an incorrect one, with nothing changed, so validation must be part of the software itself.\n\n**When a model is in the loop, you have to, from day one, build a system that can absorb the unpredictability without a hiccup.**\n\nAfter deploying Slooster Guide to staging, I gave it a once-over before demoing to my co-founders.\n\nThe exact same prompts (and model, and configs) that had been returning me real, criteria-matched vendors for weeks came back with what looked like placeholder data, literally outputting the words **Vendor A, Vendor B, Vendor C.** Regenerating then brought **Vendor D** into the mix.\n\n**A model that gives you clean output locally for weeks can still hand you garbage on a real call, and no amount of testing beforehand would be able to prevent it.** That is the whole case for validating inline, on every actual output, instead of once before you ship.\n\nI added the validation layer, with retries built in: bad output gets caught and runs the call again. But retries cost time, and the app needs to stay responsive while a user is waiting on it.\n\nSo validation isn't free, and it isn't one-size-fits-all. Where it runs, how hard it retries, and whether a human sees the result all depend on your use case.\n\nThe deterministic checks come first: a schema validator rejects anything structurally off-spec. But a placeholder like \"Vendor A\" is structurally valid. It passes a schema check, yet it's still not a valid output. That is a *semantic* miss, not a *structural* one, and you need a separate type of check for it.\n\n**So the last gate is a model checking a model: a validation prompt judging the output against the same criteria the first prompt was given.**\n\nOne AI-gate prompt, roughly:\n\n```\nYou are checking another model's answer before a user sees it.\nThe task it was given: [the original request and selection criteria].\nIts answer: [the response to check].\nReply as JSON: 'pass' (true/false), 'reason' (one line), 'confidence' (0 to 1).\nFail it if the answer has placeholder names, generic filler, items that don't meet the stated criteria, or anything that reads like example data instead of a real result.\n```\n\nThe gate can run on the same model that produced the answer, or on a different provider and model entirely for an independent second opinion. Make it a config choice, so that you don't need a code change to tune.\n\nThe three important takeaways:\n\n**A model shouldn't get the last word on its own output. Keeping a human in the loop is a feature, not a gap you'll close later.**\n\nA model won't give you the same response on the next call, which makes naive caching a trap, and no caching expensive.\n\nThe answer is caching intelligently:\n\nIntelligent caching also pays for validations. Structuring prompts so the stable part is cached offsets a good chunk of the extra cost the validation layer incurs.\n\nBy treating all prompts the same, you'll end up with increased costs and an underwhelming performance.\n\nInstead, match each prompt type to the proper provider and model:\n\nPer-prompt selection isn't a premature optimization. It's how you keep the whole system responsive while still validating the parts that count.\n\nI had planned to add validation after the MVP. That staging shock pulled it all the way forward, and I'm glad it did, as I otherwise very likely would not have accounted for getting placeholder data in the response.\n\nThe validation layer is an integral part of the system, along with transformers, schemas and specs, and it's the part I'd start with for the next system I build.\n\n**When a model is doing the work, validation can't simply be a layer you choose to add. It needs to be the cornerstone you build on.**\n\nAgain, there is no one-size-fits-all solution here. AI is giving you leverage by doing a lot of the work, but you still need to be in charge, and [organizing your thoughts up front](https://dev.to/smukker/lesson-3-architecture-learn-to-organize-your-thoughts-2h) matters more than ever.\n\nIf you're building software with AI workflows as part of the solution and wrestling with the same unpredictability, I'd be happy to share my experiences and learn about yours.", "url": "https://wpnews.pro/news/lesson-4-validation-a-must-have-cornerstone", "canonical_source": "https://dev.to/smukker/lesson-4-validation-a-must-have-cornerstone-3bd2", "published_at": "2026-08-02 22:59:53+00:00", "updated_at": "2026-08-02 23:23:22.762808+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-infrastructure", "mlops"], "entities": ["Slooster", "Slooster Guide"], "alternates": {"html": "https://wpnews.pro/news/lesson-4-validation-a-must-have-cornerstone", "markdown": "https://wpnews.pro/news/lesson-4-validation-a-must-have-cornerstone.md", "text": "https://wpnews.pro/news/lesson-4-validation-a-must-have-cornerstone.txt", "jsonld": "https://wpnews.pro/news/lesson-4-validation-a-must-have-cornerstone.jsonld"}}