Traditional automated testing is usually straightforward: provide an input, define the expected output, and compare the two. If I am testing a function that calculates a discount, the same input should consistently produce the same result. That makes assertions simple.
LLM applications behave differently. Imagine we’re testing an employee assistant against a policy that says employees can carry over a maximum of 5 vacation days. We ask:
How many unused vacation days can I carry over?
The model could answer “You can carry over up to five unused vacation days” or “The maximum vacation carryover is five days.” Both are correct, even though the strings are completely different. This makes an assertion like the following too restrictive:
expected = "You can carry over up to five unused vacation days."
response = ask_llm(
"How many unused vacation days can I carry over?"
)
assert response == expected
The test may fail simply because the model chose different words. Instead, we can start by validating the things that are deterministic and then evaluate the semantic quality separately.
def test_vacation_policy():
question = "How many unused vacation days can I carry over?"
context = """
Employees may carry over a maximum of
5 unused vacation days into the next year.
"""
response = ask_llm(question, context)
assert response is not None
assert len(response.strip()) > 0
assert "5" in response or "five" in response.lower()
assert evaluate_relevance(question, response) >= 0.8
assert evaluate_groundedness(context, response) >= 0.8
This is where LLM testing starts to look different. The first few assertions behave like normal automated tests. But relevance asks whether the response actually answers the user’s question, while groundedness asks whether the answer is supported by the supplied policy.
Suppose our test suddenly receives “Employees can carry over 10 vacation days.” The API might still return 200, the response might be perfectly formatted, and latency might be excellent—but the application is wrong. We now need to determine whether retrieval supplied an outdated policy or whether the correct five-day policy was retrieved and the LLM hallucinated the number ten.
That’s why I don’t think LLM testing replaces traditional testing. It adds another layer to it. We still test APIs, schemas, permissions, latency, tool calls, and integrations deterministically. But for generated responses, we also need to evaluate qualities such as relevance, groundedness, hallucination, and consistency.
A useful way to think about the difference is:
Traditional testing asks:
“Did I get the expected output?”
LLM testing asks:
“Is this output correct and trustworthy, even if it isn’t exactly what I expected?”
That small change in the definition of “correct” is what makes testing LLM applications fundamentally different.