Tool Schema Drift: The Silent Failure Mode in Production Agentic Systems

The most common agentic system failure I encounter in production is not a bad prompt. It is not a context overflow. It is a tool that changed without its registration changing.

I have seen this cause weeks of debugging in systems that were working fine until they weren't — and because the failure is silent, teams often spend time looking at the model first.

Most agentic frameworks let you register tools with a name, a description, and a JSON schema for parameters. The model reads the description to decide when to call the tool. It reads the schema to know what parameters to send.

tools = [
    {
        "name": "search_entities",
        "description": "Search the entity registry by name. Returns a list of matching entities.",
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "Search term"}
            },
            "required": ["query"]
        }
    }
]

Six months later, the underlying function has changed. The team added a required entity_type

filter after learning that unfiltered searches returned too many results. They updated the implementation. They forgot to update the registration.

def search_entities(query: str, entity_type: str) -> list[dict]:
    return registry.search(query=query, type=entity_type)

Now the model calls search_entities

with {"query": "Samsung"}

. The function raises a TypeError

because entity_type

is missing. Or worse — if you have a default fallback somewhere upstream — it silently runs with the wrong parameters and returns results the model was not expecting.

The model sees a response. It generates text from whatever came back. No exception fires. The output just drifts from what it should have been.

The failure mode is subtle for two reasons.

First, evals usually test output quality, not the call-response cycle. If your eval checks whether the final answer is correct, a tool that silently misbehaves can still produce plausible-looking output — especially if the task has any ambiguity in what "correct" means.

Second, the description is what the model acts on, not the schema. A schema mismatch causes a runtime error. A description mismatch causes behavioral drift — the model calls the tool in situations where it should not, or does not call it when it should, and there is no error signal.

Both failure modes happen in production. Neither shows up in type checking or unit tests.

Most frameworks validate that the model produced a well-formed tool call (correct JSON, required fields present). Fewer validate that the tool's response matched what the model was told to expect.

from pydantic import BaseModel, ValidationError
from typing import Any

class EntitySearchResult(BaseModel):
    entity_id: str
    name: str
    entity_type: str
    confidence: float

def call_tool_with_validation(tool_name: str, args: dict) -> Any:
    raw_response = dispatch_tool(tool_name, args)

    if tool_name == "search_entities":
        try:
            validated = [EntitySearchResult(**item) for item in raw_response]
            return validated
        except ValidationError as e:
            raise ToolResponseSchemaError(
                f"Tool '{tool_name}' returned unexpected shape. "
                f"Registration is out of sync with implementation.\n{e}"
            )

    return raw_response

This makes the mismatch loud. A ToolResponseSchemaError

is unmistakable. A silently wrong answer is not.

Treat the description as part of the implementation contract, not as documentation.

The practical pattern: keep tool schemas in a registry file under version control, and make updating the registry a required step when any tool function signature or return shape changes. Code review that accepts a function change without a registry update is accepting a potential drift.

TOOL_REGISTRY = {
    "search_entities": {
        "version": "2.0.0",
        "description": (
            "Search the entity registry by name and type. "
            "Returns matching entities with confidence scores. "
            "entity_type must be one of: company, person, location."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {"type": "string"},
                "entity_type": {
                    "type": "string",
                    "enum": ["company", "person", "location"]
                }
            },
            "required": ["query", "entity_type"]
        },
        "response_schema": {
            "type": "array",
            "items": {
                "type": "object",
                "required": ["entity_id", "name", "entity_type", "confidence"]
            }
        }
    }
}

If the interface changes in a backward-incompatible way — new required parameter, different return shape — give it a new name (search_entities_v2

) rather than silently updating the existing entry. Agents that depended on the old interface continue to work until they are explicitly migrated.

A single eval prompt per tool is enough to catch schema drift in CI.

import pytest

def test_search_entities_contract():
    """
    Canary eval: triggers the tool and validates the full cycle.
    If search_entities changes its interface, this breaks before production does.
    """
    agent_response = run_agent(
        prompt="Find all company entities named Samsung in the registry.",
        tools=TOOL_REGISTRY,
    )

    tool_calls = extract_tool_calls(agent_response)
    search_calls = [c for c in tool_calls if c["name"] == "search_entities"]
    assert len(search_calls) > 0, "Agent did not call search_entities"

    for call in search_calls:
        assert "entity_type" in call["args"], (
            "Agent called search_entities without entity_type — "
            "tool description may be out of sync with schema"
        )

    assert agent_response.tool_errors == [], (
        f"Tool returned unexpected shape: {agent_response.tool_errors}"
    )

This eval does not test whether the final answer is good. It tests whether the tool call cycle completed without a schema mismatch. That is exactly what breaks first when a tool drifts.

The reason this happens repeatedly is that tool descriptions live outside the usual code review discipline. They are strings in a config or a dict literal. Nobody has a linter that flags "the function signature changed but the description did not."

The fix is not complicated. It is a process discipline problem more than a technical one. The registration is the contract between the model and the implementation. Versioning it like a contract — with change control, backward-compatibility rules, and automated validation — closes the gap that produces silent drift.

The agent is not wrong when it mishandles an unexpected tool response. It was not told the tool changed. If you treat the registration as the contract, you start maintaining it like one.

I work on entity resolution and agentic infrastructure at er-api.hannune.ai. If you have seen this failure mode in your own systems, curious what the trigger was.