{"slug": "how-to-test-an-ai-agent-s-tool-selection-without-trusting-its-own-logs", "title": "How to Test an AI Agent's Tool Selection Without Trusting Its Own Logs", "summary": "A developer warns that an AI agent's own logs are unreliable for verifying tool selection, as they report intent rather than execution. They propose an external observer layer that records raw invocations—tool name, parameters, and latency—without normalization, catching failures like hallucinated names or silent parameter coercion.", "body_md": "You have built an AI agent harness. It calls tools, routes requests, and returns results. Your team trusts its telemetry to tell you which tool was selected and why.\n\nThat trust is a liability.\n\nAn agent's own logs are self-reported. They tell you what the agent *thinks* it did, not what actually happened. A hallucinated tool name, a misrouted parameter, a silent fallback to a different function — none of these surface in the agent's own trace. You need an external witness.\n\nHere is how to build one.\n\nMost teams validate agent behavior by reading the agent's own output. They check the `tool_calls`\n\nfield in the response, match it against an expected schema, and call it done.\n\nThis works until it doesn't.\n\nConsider a common failure mode: the agent decides to call `search_knowledge_base`\n\nbut the LLM formats the tool name as `searchKnowledgeBase`\n\n. The routing layer silently normalizes it, the call succeeds, and the agent logs `search_knowledge_base`\n\n. Your test passes. The actual execution path was different from what you verified.\n\nAnother pattern: the agent selects the correct tool but passes a parameter that the tool silently coerces. A date string gets parsed into a different timezone. A user ID gets truncated. The tool returns a result, the agent logs success, and your test never catches the drift.\n\nThe root cause is the same. You are testing the agent's *intent*, not its *execution*. Intent is cheap to fake. Execution leaves fingerprints.\n\nYou need a layer that sits between the agent and the tools it calls. This observer records every invocation — tool name, parameters, response, latency — without the agent knowing it is being watched.\n\nThe observer does not trust the agent's logs. It trusts what it sees on the wire.\n\nHere is the architecture at a high level:\n\nThis is not middleware. It is a test harness that wraps the tool-calling layer.\n\nI will show you a minimal implementation using Python and a mock tool server. The same pattern works in TypeScript with Playwright's route interception or a custom fetch wrapper.\n\nStart with a simple tool registry. Each tool has a name, a handler, and a schema.\n\n``` python\nfrom dataclasses import dataclass, field\nfrom typing import Any, Callable, Dict\nimport json\nimport time\n\n@dataclass\nclass Tool:\n    name: str\n    handler: Callable\n    schema: Dict[str, Any]\n\nclass ToolRegistry:\n    def __init__(self):\n        self._tools: Dict[str, Tool] = {}\n        self._invocations: list = []\n\n    def register(self, tool: Tool):\n        self._tools[tool.name] = tool\n\n    def call(self, name: str, params: Dict[str, Any]) -> Any:\n        # Record the raw invocation before any processing\n        invocation = {\n            \"tool_name\": name,\n            \"params\": params,\n            \"timestamp\": time.time(),\n            \"raw_name\": name  # This is what the agent actually sent\n        }\n\n        tool = self._tools.get(name)\n        if tool is None:\n            invocation[\"error\"] = f\"Tool '{name}' not found\"\n            self._invocations.append(invocation)\n            raise ValueError(f\"Tool '{name}' not found\")\n\n        start = time.time()\n        try:\n            result = tool.handler(**params)\n            invocation[\"result\"] = result\n            invocation[\"latency\"] = time.time() - start\n        except Exception as e:\n            invocation[\"error\"] = str(e)\n            invocation[\"latency\"] = time.time() - start\n            raise\n        finally:\n            self._invocations.append(invocation)\n\n        return result\n\n    def get_invocations(self) -> list:\n        return self._invocations\n\n    def clear(self):\n        self._invocations.clear()\n```\n\nThe key detail: `invocation[\"raw_name\"]`\n\ncaptures exactly what the agent sent. No normalization. No aliasing. If the agent sends `searchKnowledgeBase`\n\n, you record `searchKnowledgeBase`\n\n. Your test can then assert that the agent sent the canonical name, not a variant.\n\nNow register a tool and simulate an agent call.\n\n``` php\ndef search_kb(query: str, max_results: int = 5) -> list:\n    # Simulated search\n    return [{\"id\": 1, \"title\": f\"Result for {query}\"}]\n\nregistry = ToolRegistry()\nregistry.register(Tool(\n    name=\"search_knowledge_base\",\n    handler=search_kb,\n    schema={\"query\": \"string\", \"max_results\": \"integer\"}\n))\n\n# Simulate an agent call with a non-canonical name\ntry:\n    registry.call(\"searchKnowledgeBase\", {\"query\": \"AI testing\", \"max_results\": 3})\nexcept ValueError:\n    pass\n\ninvocations = registry.get_invocations()\nprint(invocations[0][\"raw_name\"])  # \"searchKnowledgeBase\"\n```\n\nYour test can now assert on `raw_name`\n\ndirectly.\n\n``` python\ndef test_agent_uses_canonical_tool_name():\n    registry.clear()\n    # Simulate the agent's decision loop\n    agent_decides_to_call(\"search_knowledge_base\", {\"query\": \"testing\"})\n    invocations = registry.get_invocations()\n    assert len(invocations) == 1\n    assert invocations[0][\"raw_name\"] == \"search_knowledge_base\"\n```\n\nThis catches the normalization failure. If the agent sends a variant, the test fails.\n\nThe same pattern catches parameter drift. Record the raw parameters before the tool handler processes them. If the agent sends a string where an integer is expected, your test sees the raw string.\n\n``` python\ndef test_agent_passes_correct_param_types():\n    registry.clear()\n    agent_decides_to_call(\"search_knowledge_base\", {\"query\": \"testing\", \"max_results\": \"3\"})\n    invocations = registry.get_invocations()\n    params = invocations[0][\"params\"]\n    assert isinstance(params[\"max_results\"], int), \"max_results should be int\"\n```\n\nThe agent's log might show `max_results: 3`\n\nas an integer because the routing layer coerced it. Your observer shows the raw string. That difference matters when the tool's behavior depends on type.\n\nThe principle is simple: test the boundary, not the summary.\n\nAn agent's internal logs are a summary of what it intended. The actual execution happens at the boundary between the agent and the tool. That boundary is where failures live. Normalization, coercion, fallback routing, silent retries — none of these appear in the agent's own trace.\n\nBy placing an observer at that boundary, you shift your testing from intent to execution. You stop asking \"did the agent think it called the right tool?\" and start asking \"did the agent actually call the right tool with the right parameters?\"\n\nThis is not a new idea. It is the same principle that makes contract testing valuable in microservices. You test the API contract, not the service's internal state. The agent is just another service with a particularly unreliable internal narrator.\n\nAn external observer adds latency and storage. Every invocation gets recorded, serialized, and stored for the duration of the test. In production, you might sample or aggregate. In test, you record everything.\n\nThe trade-off is worth it. A single undetected tool misrouting can cascade into hours of debugging. The observer pays for itself the first time it catches a failure the agent's logs missed.\n\nYour team is probably testing the agent's intent right now. The logs look clean, the traces are green, and the demos work. But the real failures live in the gap between what the agent says it did and what actually happened.\n\nBuild an observer. Record the raw invocation. Assert on what you see, not what you are told.\n\nWhich of your agent's tool calls have you never actually witnessed?", "url": "https://wpnews.pro/news/how-to-test-an-ai-agent-s-tool-selection-without-trusting-its-own-logs", "canonical_source": "https://dev.to/qawalah/how-to-test-an-ai-agents-tool-selection-without-trusting-its-own-logs-1pl3", "published_at": "2026-07-21 09:51:28+00:00", "updated_at": "2026-07-21 10:01:00.722541+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/how-to-test-an-ai-agent-s-tool-selection-without-trusting-its-own-logs", "markdown": "https://wpnews.pro/news/how-to-test-an-ai-agent-s-tool-selection-without-trusting-its-own-logs.md", "text": "https://wpnews.pro/news/how-to-test-an-ai-agent-s-tool-selection-without-trusting-its-own-logs.txt", "jsonld": "https://wpnews.pro/news/how-to-test-an-ai-agent-s-tool-selection-without-trusting-its-own-logs.jsonld"}}