You have built an AI agent harness. It calls tools, routes requests, and returns results. Your team trusts its telemetry to tell you which tool was selected and why.
That trust is a liability.
An agent's own logs are self-reported. They tell you what the agent thinks it did, not what actually happened. A hallucinated tool name, a misrouted parameter, a silent fallback to a different function — none of these surface in the agent's own trace. You need an external witness.
Here is how to build one.
Most teams validate agent behavior by reading the agent's own output. They check the tool_calls
field in the response, match it against an expected schema, and call it done.
This works until it doesn't.
Consider a common failure mode: the agent decides to call search_knowledge_base
but the LLM formats the tool name as searchKnowledgeBase
. The routing layer silently normalizes it, the call succeeds, and the agent logs search_knowledge_base
. Your test passes. The actual execution path was different from what you verified.
Another pattern: the agent selects the correct tool but passes a parameter that the tool silently coerces. A date string gets parsed into a different timezone. A user ID gets truncated. The tool returns a result, the agent logs success, and your test never catches the drift.
The root cause is the same. You are testing the agent's intent, not its execution. Intent is cheap to fake. Execution leaves fingerprints.
You need a layer that sits between the agent and the tools it calls. This observer records every invocation — tool name, parameters, response, latency — without the agent knowing it is being watched.
The observer does not trust the agent's logs. It trusts what it sees on the wire.
Here is the architecture at a high level:
This is not middleware. It is a test harness that wraps the tool-calling layer.
I will show you a minimal implementation using Python and a mock tool server. The same pattern works in TypeScript with Playwright's route interception or a custom fetch wrapper.
Start with a simple tool registry. Each tool has a name, a handler, and a schema.
from dataclasses import dataclass, field
from typing import Any, Callable, Dict
import json
import time
@dataclass
class Tool:
name: str
handler: Callable
schema: Dict[str, Any]
class ToolRegistry:
def __init__(self):
self._tools: Dict[str, Tool] = {}
self._invocations: list = []
def register(self, tool: Tool):
self._tools[tool.name] = tool
def call(self, name: str, params: Dict[str, Any]) -> Any:
invocation = {
"tool_name": name,
"params": params,
"timestamp": time.time(),
"raw_name": name # This is what the agent actually sent
}
tool = self._tools.get(name)
if tool is None:
invocation["error"] = f"Tool '{name}' not found"
self._invocations.append(invocation)
raise ValueError(f"Tool '{name}' not found")
start = time.time()
try:
result = tool.handler(**params)
invocation["result"] = result
invocation["latency"] = time.time() - start
except Exception as e:
invocation["error"] = str(e)
invocation["latency"] = time.time() - start
raise
finally:
self._invocations.append(invocation)
return result
def get_invocations(self) -> list:
return self._invocations
def clear(self):
self._invocations.clear()
The key detail: invocation["raw_name"]
captures exactly what the agent sent. No normalization. No aliasing. If the agent sends searchKnowledgeBase
, you record searchKnowledgeBase
. Your test can then assert that the agent sent the canonical name, not a variant.
Now register a tool and simulate an agent call.
def search_kb(query: str, max_results: int = 5) -> list:
return [{"id": 1, "title": f"Result for {query}"}]
registry = ToolRegistry()
registry.register(Tool(
name="search_knowledge_base",
handler=search_kb,
schema={"query": "string", "max_results": "integer"}
))
try:
registry.call("searchKnowledgeBase", {"query": "AI testing", "max_results": 3})
except ValueError:
pass
invocations = registry.get_invocations()
print(invocations[0]["raw_name"]) # "searchKnowledgeBase"
Your test can now assert on raw_name
directly.
def test_agent_uses_canonical_tool_name():
registry.clear()
agent_decides_to_call("search_knowledge_base", {"query": "testing"})
invocations = registry.get_invocations()
assert len(invocations) == 1
assert invocations[0]["raw_name"] == "search_knowledge_base"
This catches the normalization failure. If the agent sends a variant, the test fails.
The same pattern catches parameter drift. Record the raw parameters before the tool handler processes them. If the agent sends a string where an integer is expected, your test sees the raw string.
def test_agent_passes_correct_param_types():
registry.clear()
agent_decides_to_call("search_knowledge_base", {"query": "testing", "max_results": "3"})
invocations = registry.get_invocations()
params = invocations[0]["params"]
assert isinstance(params["max_results"], int), "max_results should be int"
The agent's log might show max_results: 3
as an integer because the routing layer coerced it. Your observer shows the raw string. That difference matters when the tool's behavior depends on type.
The principle is simple: test the boundary, not the summary.
An agent's internal logs are a summary of what it intended. The actual execution happens at the boundary between the agent and the tool. That boundary is where failures live. Normalization, coercion, fallback routing, silent retries — none of these appear in the agent's own trace.
By placing an observer at that boundary, you shift your testing from intent to execution. You stop asking "did the agent think it called the right tool?" and start asking "did the agent actually call the right tool with the right parameters?"
This is not a new idea. It is the same principle that makes contract testing valuable in microservices. You test the API contract, not the service's internal state. The agent is just another service with a particularly unreliable internal narrator.
An external observer adds latency and storage. Every invocation gets recorded, serialized, and stored for the duration of the test. In production, you might sample or aggregate. In test, you record everything.
The trade-off is worth it. A single undetected tool misrouting can cascade into hours of debugging. The observer pays for itself the first time it catches a failure the agent's logs missed.
Your team is probably testing the agent's intent right now. The logs look clean, the traces are green, and the demos work. But the real failures live in the gap between what the agent says it did and what actually happened.
Build an observer. Record the raw invocation. Assert on what you see, not what you are told.
Which of your agent's tool calls have you never actually witnessed?