Originally published on AIdeazz — cross-posted here with canonical link.
My first production AI agent shipped with 131 tests in its evaluation harness. This wasn't a luxury; it was a non-negotiable step after a $0.03/run silent failure nearly cost me a client. I built the harness before writing new features, a decision driven by the fundamental limitations of traditional unit tests in multi-agent systems. Without it, I would have pushed a "working" agent that consistently failed to meet user intent under specific, common conditions, burning compute and trust.
The agent's job was simple: process a user's natural language request for a specific report, query a database, and return the data. My initial unit tests covered the LLM's parsing of intent, the database query generation, and the final data formatting. All green. The agent worked perfectly in my dev environment with my curated test cases.
Then, a user asked for "last month's sales figures for product X." The agent returned "no data found." My logs showed the LLM correctly identified "product X" and "last month." The database query looked correct. But the user was seeing an empty result.
The problem wasn't in the individual components. The LLM correctly extracted the date range as "last month." My database query generator correctly translated "last month" into BETWEEN '2023-10-01' AND '2023-10-31'
. The silent failure was in the interaction between the LLM's interpretation of "last month" and the database's actual data schema. My sales data was stored with a report_date
field that represented the end of the reporting period. So "last month's sales" meant data with report_date = '2023-10-31'
. The agent was querying for a range when it should have been querying for a specific date.
This wasn't a bug in my code, nor in the LLM's logic, nor in the database. It was a mismatch in semantic interpretation across system boundaries, a class of error traditional unit tests are blind to. Each component passed its individual test, but the system as a whole failed to deliver the correct outcome for a common user request. Each failed run cost $0.03 in Groq inference and Oracle Cloud database queries. Small individually, but compounding rapidly with user interaction.
Unit tests validate isolated functions or methods. They are excellent for ensuring add(a, b)
returns a + b
. They can even validate that an LLM call with a specific prompt returns a JSON object matching a schema.
What unit tests cannot do for AI agents:
My current harness for a production agent on Oracle Cloud, routing between Groq and Claude, consists of 131 tests across four distinct layers. This structure allows me to catch failures at different levels of abstraction and interaction.
These are the closest to traditional unit tests, but specifically for LLM outputs. They ensure that individual LLM calls adhere to expected formats and basic content constraints.
assert pydantic_model.parse_raw(llm_output)
)assert "SELECT" in sql_query_output
)assert re.search(r'\d{4}-\d{2}-\d{2}', date_string)
)assert tool_call.name == "get_report" and "product_id" in tool_call.args
)These tests run quickly, often in milliseconds, and are crucial for catching regressions in prompt engineering or LLM model updates. I run them with Groq's Llama 3 8B for speed and cost-efficiency ($0.00000027/token for output).
This layer tests the internal decision-making and state transitions of the agent. It ensures that given a specific input, the agent progresses through the correct sequence of actions.
assert agent_state.current_intent == "REPORT_GENERATION"
)assert "previous_product_id" in agent_state.context
)assert "error_message" in agent_response and "try again" in agent_response.text
)assert "missing_parameter" in agent_response.text
)These tests often involve mocking external services (database, APIs) to isolate the agent's internal logic. They are still relatively fast, typically under 100ms per test.
This is where the eval harness truly shines. These tests simulate real user interactions from start to finish, including all external integrations (database, APIs, other agents). This is where my "last month" bug would have been caught.
These tests are more expensive, as they involve full LLM calls and database interactions. I typically run them with Claude 3 Haiku or Opus depending on complexity, costing $0.00025/token for Haiku input and $0.00125/token for output. A full run of this layer can cost $0.01-$0.03 per test. This is why I keep the number of tests here focused on critical paths and high-impact scenarios.
This layer is less about correctness and more about the qualitative aspects of the agent.
assert response_time < 5.0
seconds for a complex query). This is critical for Telegram/WhatsApp agents.assert len(response_text.split()) < 200
for a simple query).These tests are often the most expensive due to the LLM-as-a-judge component, but they are crucial for ensuring the agent is not just correct, but usable. Running all 131 tests costs approximately $0.03 per full harness execution. This breaks down roughly as:
This cost is negligible compared to the potential loss of client trust or wasted development time debugging production issues. I run the full harness on every significant code commit and before every deployment. It's integrated into my CI/CD pipeline on Oracle Cloud Infrastructure.
Building this harness took time – about two weeks initially. But it has paid for itself multiple times over by catching subtle interaction bugs and semantic mismatches that no amount of manual testing or traditional unit tests would have found. It's the only way I can confidently ship AI agents into production without constant fear of silent, costly failures.
Q: How do you generate the test cases for Layer 3 (End-to-End Functional Scenarios)? A: I start with real user queries from logs, then use an LLM (Claude 3 Opus) to generate variations and edge cases based on those. I also manually craft critical path scenarios.
Q: What's your strategy for mocking external services in Layer 2?
A: I use Python's unittest.mock
library extensively. For database interactions, I mock the ORM layer or the database client directly to return predefined datasets, ensuring the agent's logic is tested independently of actual DB state.
Q: How do you handle non-deterministic LLM outputs in your tests?
A: For Layer 1, I use schema validation and keyword presence, which are robust to minor phrasing variations. For Layer 3, I often use fuzzy string matching or LLM-as-a-judge (another LLM) to evaluate the correctness of the final output, rather than exact string matching.
Q: What's the biggest challenge in maintaining such a large test suite?
A: Keeping the test data and expected outputs up-to-date as the agent's capabilities evolve. I automate this where possible by regenerating expected outputs for certain tests, but it still requires periodic manual review.
Q: Why Oracle Cloud for infrastructure?
A: Oracle Cloud offers competitive pricing for compute and database services, especially for smaller operations like mine without VC funding. Their free tier and always-free services are also a significant advantage for bootstrapping.