Tool Schema Drift: The Silent Failure Mode in Production Agentic Systems

A developer warns that tool schema drift—when a tool's implementation changes without updating its registration—is a silent failure mode in production agentic systems. This mismatch can cause weeks of debugging because the model receives unexpected responses without error signals, and evals often miss the issue. The developer recommends validating tool responses against expected schemas to make mismatches loud.

The most common agentic system failure I encounter in production is not a bad prompt. It is not a context overflow. It is a tool that changed without its registration changing. I have seen this cause weeks of debugging in systems that were working fine until they weren't — and because the failure is silent, teams often spend time looking at the model first. Most agentic frameworks let you register tools with a name, a description, and a JSON schema for parameters. The model reads the description to decide when to call the tool. It reads the schema to know what parameters to send. Initial registration — looks fine tools = { "name": "search entities", "description": "Search the entity registry by name. Returns a list of matching entities.", "input schema": { "type": "object", "properties": { "query": {"type": "string", "description": "Search term"} }, "required": "query" } } Six months later, the underlying function has changed. The team added a required entity type filter after learning that unfiltered searches returned too many results. They updated the implementation. They forgot to update the registration. python Current implementation — mismatched with registration def search entities query: str, entity type: str - list dict : entity type is now required — "company", "person", "location" return registry.search query=query, type=entity type Now the model calls search entities with {"query": "Samsung"} . The function raises a TypeError because entity type is missing. Or worse — if you have a default fallback somewhere upstream — it silently runs with the wrong parameters and returns results the model was not expecting. The model sees a response. It generates text from whatever came back. No exception fires. The output just drifts from what it should have been. The failure mode is subtle for two reasons. First, evals usually test output quality, not the call-response cycle. If your eval checks whether the final answer is correct, a tool that silently misbehaves can still produce plausible-looking output — especially if the task has any ambiguity in what "correct" means. Second, the description is what the model acts on, not the schema. A schema mismatch causes a runtime error. A description mismatch causes behavioral drift — the model calls the tool in situations where it should not, or does not call it when it should, and there is no error signal. Both failure modes happen in production. Neither shows up in type checking or unit tests. Most frameworks validate that the model produced a well-formed tool call correct JSON, required fields present . Fewer validate that the tool's response matched what the model was told to expect. python from pydantic import BaseModel, ValidationError from typing import Any class EntitySearchResult BaseModel : entity id: str name: str entity type: str confidence: float def call tool with validation tool name: str, args: dict - Any: raw response = dispatch tool tool name, args Validate response matches what the model expects if tool name == "search entities": try: validated = EntitySearchResult item for item in raw response return validated except ValidationError as e: Response shape changed — surface this immediately raise ToolResponseSchemaError f"Tool '{tool name}' returned unexpected shape. " f"Registration is out of sync with implementation.\n{e}" return raw response This makes the mismatch loud. A ToolResponseSchemaError is unmistakable. A silently wrong answer is not. Treat the description as part of the implementation contract, not as documentation. The practical pattern: keep tool schemas in a registry file under version control, and make updating the registry a required step when any tool function signature or return shape changes. Code review that accepts a function change without a registry update is accepting a potential drift. tools/registry/v2.py — versioned alongside implementation TOOL REGISTRY = { "search entities": { "version": "2.0.0", "description": "Search the entity registry by name and type. " "Returns matching entities with confidence scores. " "entity type must be one of: company, person, location." , "input schema": { "type": "object", "properties": { "query": {"type": "string"}, "entity type": { "type": "string", "enum": "company", "person", "location" } }, "required": "query", "entity type" }, "response schema": { "type": "array", "items": { "type": "object", "required": "entity id", "name", "entity type", "confidence" } } } } If the interface changes in a backward-incompatible way — new required parameter, different return shape — give it a new name search entities v2 rather than silently updating the existing entry. Agents that depended on the old interface continue to work until they are explicitly migrated. A single eval prompt per tool is enough to catch schema drift in CI. python tests/evals/test tool contracts.py import pytest def test search entities contract : """ Canary eval: triggers the tool and validates the full cycle. If search entities changes its interface, this breaks before production does. """ agent response = run agent prompt="Find all company entities named Samsung in the registry.", tools=TOOL REGISTRY, Check the tool was called tool calls = extract tool calls agent response search calls = c for c in tool calls if c "name" == "search entities" assert len search calls 0, "Agent did not call search entities" Check the call used the current schema for call in search calls: assert "entity type" in call "args" , "Agent called search entities without entity type — " "tool description may be out of sync with schema" Check the response matched expected shape assert agent response.tool errors == , f"Tool returned unexpected shape: {agent response.tool errors}" This eval does not test whether the final answer is good. It tests whether the tool call cycle completed without a schema mismatch. That is exactly what breaks first when a tool drifts. The reason this happens repeatedly is that tool descriptions live outside the usual code review discipline. They are strings in a config or a dict literal. Nobody has a linter that flags "the function signature changed but the description did not." The fix is not complicated. It is a process discipline problem more than a technical one. The registration is the contract between the model and the implementation. Versioning it like a contract — with change control, backward-compatibility rules, and automated validation — closes the gap that produces silent drift. The agent is not wrong when it mishandles an unexpected tool response. It was not told the tool changed. If you treat the registration as the contract, you start maintaining it like one. I work on entity resolution and agentic infrastructure at er-api.hannune.ai. If you have seen this failure mode in your own systems, curious what the trigger was.