How to Test an AI Agent's Tool Selection Without Trusting Its Own Logs

A developer warns that an AI agent's own logs are unreliable for verifying tool selection, as they report intent rather than execution. They propose an external observer layer that records raw invocations—tool name, parameters, and latency—without normalization, catching failures like hallucinated names or silent parameter coercion.

You have built an AI agent harness. It calls tools, routes requests, and returns results. Your team trusts its telemetry to tell you which tool was selected and why. That trust is a liability. An agent's own logs are self-reported. They tell you what the agent thinks it did, not what actually happened. A hallucinated tool name, a misrouted parameter, a silent fallback to a different function — none of these surface in the agent's own trace. You need an external witness. Here is how to build one. Most teams validate agent behavior by reading the agent's own output. They check the tool calls field in the response, match it against an expected schema, and call it done. This works until it doesn't. Consider a common failure mode: the agent decides to call search knowledge base but the LLM formats the tool name as searchKnowledgeBase . The routing layer silently normalizes it, the call succeeds, and the agent logs search knowledge base . Your test passes. The actual execution path was different from what you verified. Another pattern: the agent selects the correct tool but passes a parameter that the tool silently coerces. A date string gets parsed into a different timezone. A user ID gets truncated. The tool returns a result, the agent logs success, and your test never catches the drift. The root cause is the same. You are testing the agent's intent , not its execution . Intent is cheap to fake. Execution leaves fingerprints. You need a layer that sits between the agent and the tools it calls. This observer records every invocation — tool name, parameters, response, latency — without the agent knowing it is being watched. The observer does not trust the agent's logs. It trusts what it sees on the wire. Here is the architecture at a high level: This is not middleware. It is a test harness that wraps the tool-calling layer. I will show you a minimal implementation using Python and a mock tool server. The same pattern works in TypeScript with Playwright's route interception or a custom fetch wrapper. Start with a simple tool registry. Each tool has a name, a handler, and a schema. python from dataclasses import dataclass, field from typing import Any, Callable, Dict import json import time @dataclass class Tool: name: str handler: Callable schema: Dict str, Any class ToolRegistry: def init self : self. tools: Dict str, Tool = {} self. invocations: list = def register self, tool: Tool : self. tools tool.name = tool def call self, name: str, params: Dict str, Any - Any: Record the raw invocation before any processing invocation = { "tool name": name, "params": params, "timestamp": time.time , "raw name": name This is what the agent actually sent } tool = self. tools.get name if tool is None: invocation "error" = f"Tool '{name}' not found" self. invocations.append invocation raise ValueError f"Tool '{name}' not found" start = time.time try: result = tool.handler params invocation "result" = result invocation "latency" = time.time - start except Exception as e: invocation "error" = str e invocation "latency" = time.time - start raise finally: self. invocations.append invocation return result def get invocations self - list: return self. invocations def clear self : self. invocations.clear The key detail: invocation "raw name" captures exactly what the agent sent. No normalization. No aliasing. If the agent sends searchKnowledgeBase , you record searchKnowledgeBase . Your test can then assert that the agent sent the canonical name, not a variant. Now register a tool and simulate an agent call. php def search kb query: str, max results: int = 5 - list: Simulated search return {"id": 1, "title": f"Result for {query}"} registry = ToolRegistry registry.register Tool name="search knowledge base", handler=search kb, schema={"query": "string", "max results": "integer"} Simulate an agent call with a non-canonical name try: registry.call "searchKnowledgeBase", {"query": "AI testing", "max results": 3} except ValueError: pass invocations = registry.get invocations print invocations 0 "raw name" "searchKnowledgeBase" Your test can now assert on raw name directly. python def test agent uses canonical tool name : registry.clear Simulate the agent's decision loop agent decides to call "search knowledge base", {"query": "testing"} invocations = registry.get invocations assert len invocations == 1 assert invocations 0 "raw name" == "search knowledge base" This catches the normalization failure. If the agent sends a variant, the test fails. The same pattern catches parameter drift. Record the raw parameters before the tool handler processes them. If the agent sends a string where an integer is expected, your test sees the raw string. python def test agent passes correct param types : registry.clear agent decides to call "search knowledge base", {"query": "testing", "max results": "3"} invocations = registry.get invocations params = invocations 0 "params" assert isinstance params "max results" , int , "max results should be int" The agent's log might show max results: 3 as an integer because the routing layer coerced it. Your observer shows the raw string. That difference matters when the tool's behavior depends on type. The principle is simple: test the boundary, not the summary. An agent's internal logs are a summary of what it intended. The actual execution happens at the boundary between the agent and the tool. That boundary is where failures live. Normalization, coercion, fallback routing, silent retries — none of these appear in the agent's own trace. By placing an observer at that boundary, you shift your testing from intent to execution. You stop asking "did the agent think it called the right tool?" and start asking "did the agent actually call the right tool with the right parameters?" This is not a new idea. It is the same principle that makes contract testing valuable in microservices. You test the API contract, not the service's internal state. The agent is just another service with a particularly unreliable internal narrator. An external observer adds latency and storage. Every invocation gets recorded, serialized, and stored for the duration of the test. In production, you might sample or aggregate. In test, you record everything. The trade-off is worth it. A single undetected tool misrouting can cascade into hours of debugging. The observer pays for itself the first time it catches a failure the agent's logs missed. Your team is probably testing the agent's intent right now. The logs look clean, the traces are green, and the demos work. But the real failures live in the gap between what the agent says it did and what actually happened. Build an observer. Record the raw invocation. Assert on what you see, not what you are told. Which of your agent's tool calls have you never actually witnessed?