AI agents are often presented as applications that simply connect an LLM to a collection of tools.
In production, the reality is considerably more complicated.
A serious agent may contain planners, routers, state management, retrieval systems, database tools, validation logic, retry mechanisms, and multiple execution paths. The LLM introduces another layer of uncertainty because the same input does not necessarily produce exactly the same reasoning or tool usage.
As a result, testing an agent requires a different mindset from testing a conventional backend service.
The central idea of this write-up is to separate testing into two categories:
deterministic software testing and probabilistic agent evaluation.
Deterministic components should be tested directly. LLM-driven behaviour should be evaluated through realistic scenarios.
A traditional function might behave like:
input → function → output
An agent can behave like:
user question → planner → graph routing → tool selection → retrieval → validation → retry → synthesis → response
The number of possible execution paths can grow quickly.
A single user question may result in:
This makes exhaustive unit testing impractical.
The difficulty increases further when business requirements are highly contextual.
A useful way to understand testing requirements is to decompose the agent into layers.
Examples:
These components can usually be tested using conventional unit tests.
Examples:
Tools usually have deterministic contracts even when the LLM decides when to call them.
Examples:
This layer is inherently probabilistic.
This is the overall combination of the previous layers.
The most important question becomes:
Given a realistic user request, does the agent eventually produce the correct behaviour?
Suppose an agent has ten graph nodes and each node can potentially branch into several paths.
Testing every possible combination can quickly become unmanageable.
It is also possible to have a test pass even though the overall agent is broken.
For example:
Tool A
works correctly.
Tool B
works correctly.
Planner
works correctly.
Router
works correctly.
But the graph may still send the planner output to the wrong branch.
This is why testing only individual components is insufficient.
A practical architecture uses multiple testing layers rather than one large test suite.
Test ordinary software directly.
Examples:
These tests should be fast and inexpensive.
Test tools without involving an LLM.
For example:
project_name → search_project() → database → expected project
This allows database and retrieval behaviour to be validated independently.
Test whether the graph transitions correctly.
For example:
Planner → ProjectQueryDecision → GetProjectName → ProjectQuery → Reply
The LLM output can be replaced with a deterministic fixture so that graph behaviour can be verified independently.
Provide realistic user questions to the complete agent.
The test evaluates the final behaviour rather than an individual function.
One of the most useful design principles is:
Do not use an LLM to test functionality that can be tested deterministically.
Suppose a project search tool costs an LLM call every time it is tested indirectly.
That means thousands of tokens might be spent validating a function that is ultimately just querying a database.
Instead:
Input → Tool → Expected Result
can be tested directly.
Then:
User Question → LLM → Tool Selection
can be tested separately.
This gives each component the appropriate testing strategy.
Before exposing a tool to an LLM, validate the tool itself.
For example, a project search tool can have tests for:
The output should have a deterministic expectation.
Once those tests pass, the tool becomes a trusted component of the agent.
This significantly reduces debugging complexity.
If a production test fails later, the investigation can focus on the LLM's decision-making rather than immediately suspecting the underlying tool.
For the complete application, a scenario-based dataset is more useful than hundreds of artificial unit cases.
The dataset should represent real user behaviour.
For a property-information agent, scenarios might include:
The expected result does not necessarily have to be an exact text match.
It can instead define expected behaviour.
For example:
Expected:
- identify project
- retrieve project information
- retrieve attachments
- answer user
Not acceptable:
- choose unrelated project
- skip required retrieval
- invent project information
Business requirements are often more difficult than technical requirements.
A user might ask:
"Tell me about the project."
That simple sentence could require very different behaviour depending on conversation history.
The agent may need to determine:
Therefore, test cases should include conversational context rather than isolated questions.
This is where scenario-based evaluation becomes particularly valuable.
Graph-based agents introduce another category of failures.
The nodes themselves may work correctly while the transitions are incorrect.
Testing should therefore verify:
state + node output → expected next node
For example:
Intent = project_search
↓
ProjectDecision
↓
GetProjectName
↓
ProjectQuery
Tests should verify both:
This makes graph errors much easier to isolate.
It is not enough for the agent to call the correct tool.
The tool arguments must also be correct.
For example:
Tool: get_project
Expected:
{
"project_name": "ABC Residence"
}
The agent could fail by:
Therefore, an evaluation framework should record tool calls as part of the agent trace.
Production agents frequently contain feedback loops.
For example:
retrieve → validate → re-query → validate
This creates another testing dimension.
You should test:
The goal is not only to verify that the happy path works.
The recovery path must also be predictable.
Reliable agents need explicit failure testing.
Examples include:
A mature agent should fail in controlled ways rather than simply producing an incorrect answer.
Whenever possible, intermediate LLM outputs should use a schema.
For example:
{
"intent": "project_search",
"requires_tool": true,
"tool": "get_project",
"reason": "Project identity must be resolved first."
}
Testing becomes easier because the evaluator can validate:
This is considerably easier to evaluate than unconstrained natural language.
Agent evaluation should go beyond final-answer accuracy.
Useful metrics include:
Did the agent provide the correct answer?
Did the agent call the correct tools?
Did it traverse the correct graph path?
Did it retrieve relevant information?
Does it behave consistently across repeated runs?
How many LLM calls and tokens were required?
How long did the full execution take?
This allows an agent to be evaluated as a software system rather than merely as a chatbot.
Every production failure can become a future test case.
For example:
Production incident
↓
Identify failure mode
↓
Create regression scenario
↓
Add to evaluation dataset
↓
Prevent recurrence
Over time, the scenario dataset becomes a practical representation of the application's business requirements.
This is particularly valuable because agent behaviour can change significantly when:
A useful evaluation dataset should contain:
| Field | Purpose |
|---|---|
| User question | Original scenario |
| Conversation context | Relevant history |
| Expected intent | Required interpretation |
| Expected tools | Valid tool usage |
| Expected entities | Project/property/etc. |
| Expected outcome | Required behaviour |
| Failure conditions | Unacceptable behaviour |
| Evaluation result | Pass/fail or score |
This makes agent testing repeatable rather than dependent on manually inspecting conversations.
Different tests require different levels of realism.
For unit tests, mock external dependencies aggressively.
For integration tests, use real databases and real tools where practical.
For end-to-end evaluation, test the complete production-like pipeline.
A useful principle is:
The lower the test level, the more deterministic it should be.
The higher the test level, the more realistic it should be.
LLM behaviour introduces variance.
For stable evaluation, control variables such as:
Even then, exact output matching is usually inappropriate.
Evaluation should focus on semantic correctness and required behaviour.
Several approaches tend to fail in production.
This is expensive and makes failures difficult to diagnose.
This misses orchestration and graph-level failures.
A correct answer can be expressed in many valid ways.
Production failures frequently occur in ambiguous and incomplete requests.
The final answer may look reasonable even though the agent reached it through an invalid process.
A practical architecture can therefore look like:
┌──────────────────────┐
│ Unit Tests │
│ Deterministic Code │
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ Tool Tests │
│ DB / Retrieval │
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ Graph Tests │
│ Routing / State │
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ Agent Evaluation │
│ Scenario Dataset │
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ Production Feedback │
│ Regression Cases │
└──────────────────────┘
The important point is that no single testing strategy is sufficient.
The main lesson is that testing an agent is fundamentally an exercise in decomposition.
Trying to test the entire system through end-to-end LLM calls is expensive.
Trying to test the entire system through traditional unit tests is insufficient.
The more practical approach is to divide the system into deterministic and probabilistic boundaries.
Deterministic logic should be tested directly.
Tools should be tested independently.
Graph execution should be tested with controlled inputs.
LLM reasoning should be evaluated through realistic scenarios.
End-to-end tests should verify whether the complete system satisfies business requirements.
Production AI agents sit somewhere between software engineering and probabilistic systems engineering.
The challenge is not simply making an LLM produce a good answer.
The challenge is making the entire system predictable enough to operate reliably:
LLM + tools + graph + retrieval + business logic + state
That changes how testing needs to be designed.
Rather than asking:
"Can I unit test this agent?"
A better question is:
"Which parts of this agent are deterministic, which parts are probabilistic, and what is the correct testing strategy for each?"
That distinction has become one of the most useful principles in building stable production agents.