By Piotr Ryszewski, AI Engineering Manager at deepsense.ai I spent years in QA and business analysis before moving into AI delivery. One question kept coming back: how do the principles of traditional Quality Engineering apply to systems built on top of LLMs? In this article, I look at what changes when we move from deterministic testing to probabilistic evaluation.
What changes when software becomes probabilistic #
Classical software testing usually assumes that the same input produces the same output. You define what correct behaviour means, verify it, and assess how often the system meets that standard. A language model can produce different answers for the same input, even when the surrounding application has not changed. I therefore need to measure how often a system behaves as expected, under which conditions it does so, and how I will notice when that rate changes.
The model may also change outside the team’s codebase, because a provider can update it without a new commit in the application. Prompts, data, user questions and connected tools can change the result as well. Traditional QA practices still matter, because I still want to test early, repeat tests after changes, and use conditions that resemble the intended use. The acceptance criteria and the way I measure them need more care.
I cover this in the full article.
The testing pyramid still helps #
The familiar testing pyramid gives me a useful structure, although each layer contains different checks when AI is part of the system. At the bottom, unit tests cover deterministic code such as data processing, chunking, response parsing and error handling. A small set of model evaluations can sit alongside them, using a curated golden set of inputs and expected outputs.
Integration tests then show whether the parts work together. In a retrieval-augmented system, I need to evaluate retrieval and generation as one pipeline, while an agent needs tests that cover the model, its tools and a sequence of tasks. At the acceptance level, the question becomes whether the system handles the scenarios that the client agreed to treat as successful. A client-approved golden set can provide concrete acceptance criteria instead of relying only on broad statements in a requirements document.
Production monitoring sits outside the pyramid because it continues between releases. Output distributions can change, user questions can take an unexpected form, and the underlying model can evolve. Monitoring helps me see when the assumptions behind the test suite stop matching the system that users are actually using.
The failure modes I need to measure #
Hallucinations are easy to name, but they are only one part of the problem. A prompt change or a provider update can quietly reduce the quality of answers that worked the previous week. This prompt regression may produce no error, which means that a versioned prompt and an automated regression suite are important.
Metrics can also create a false sense of safety when they measure format and latency while ignoring whether the answer is useful. A missing baseline creates a similar problem, because without a reference set it is difficult to say whether a new model or prompt has improved the system. Hallucinations still require deliberate checks, especially when the team needs to find plausible but false answers at scale.
For that reason, I would start with a golden set agreed with the business owner, then run automated evaluations when prompts, models or other important parts of the system change. The checks should combine deterministic rules, model-graded criteria and human review, while grounding checks should confirm that answers are supported by retrieved sources. I also need release thresholds that lead to a clear action. When results fall below the agreed level, the team should know whether to block the release, roll back the change or escalate. A threshold without an owner or response plan does not protect the system.
The architecture changes the testing strategy #
Different AI systems have different failure modes, so I cannot use the same evaluation approach for a plain language model, a RAG pipeline, an agent and an external service integration.
For RAG, I first measure whether retrieval returns the right documents, because generation cannot repair missing or irrelevant context. For agents, I examine the trace, including decisions, tool calls and intermediate states, because a correct final answer can hide an unreliable path. For external integrations, I test both the interface and the model’s use of it, since a tool can work correctly while its description leaves the model unsure when it should be called.
Client validation is part of testing #
In my experience, many problems appear when a team and a client use the same words while imagining different system behaviour. A requirements document can describe what an AI system should do, but language alone leaves room for different interpretations.
I therefore prefer to build alignment around concrete examples and review them throughout the project. The technical review should measure the system against clear criteria, while the business review should ask whether it supports the client’s actual workflow. When the client changes direction, the team needs a simple process for classifying, documenting and deciding on that change.
A different meaning of “done” #
An AI system that performs well today may behave differently later because the underlying model changes, the inputs shift or the retrieval data becomes outdated. This means that deployment starts a new phase of quality work, with monitoring, ownership and a shared understanding of what acceptable behaviour means over time.
The useful question is therefore: how often does the system behave correctly, under what conditions, and how will I know when that changes? When I can answer those questions, I have a much stronger basis for deciding whether an AI system is ready to remain in use.
Read the full article on deepsense.ai for a deeper discussion.