Member-only story
Most failures in production are independent of the model’s architecture or output. If we blame the model for all the issues, it will create a false sense of security regarding the rest of the technical stack, causing teams to waste valuable time optimizing algorithms when they should actually be fixing infrastructure, pipeline dependencies, or data quality issues elsewhere.
There is a chance that most production issues begin with the documents, metadata, permissions and retrieval logic. This can happen before an LLM writes a sentence.
The Launch-day Failure Nobody Expects #
Imagine a support guy who has excelled in all demos. A customer inquires if he or she can be reimbursed and the system checks the company knowledge base and presents a polished response with a confident reference. The language is comprehensible. It takes a short response time. All in the room are amazed.
Next take the same setup and put it into production. The customer is in a different country, the policy cited was superseded 3 months ago and the exception currently is in a table that was not picked up by the parser at all. The answer is fluent and based on a real document but wrong.
This is where many teams start losing weeks. They improve the language model, modify the system prompt, increase the context window and experiment with a different temperature. The output is smoother, but the same factual errors are repeated since the failure happened prior to the model seeing the question.