An LLM chooses a search tool and sends this:
{
"query": "What is our refund policy?",
"limit": "a few"
}
The request is understandable to a person. The backend still needs an integer.
Should it guess what “a few” means? Use a default? Return an error?
This is where tool contracts become useful. While exploring this topic, I looked at function-calling research, JSON Schema, and the Model Context Protocol specification.
One practical lesson stood out: choosing a tool, producing valid arguments, and executing an allowed action are separate concerns.
The Berkeley Function Calling Leaderboard (BFCL), published at ICML 2025, evaluates whether models can make correct function calls across different settings. It treats function calling as a capability that needs explicit evaluation. A fluent response alone doesn’t demonstrate correct tool use.
My engineering takeaway is to evaluate the actual call: which tool was selected, which arguments were supplied, and whether execution produced the intended result.
A schema helps with part of that problem.
For a document search tool, an input contract could require:
JSON Schema provides keywords for declaring properties, required fields, and whether additional properties are allowed. Defining a property does not automatically make it required; that needs to be specified separately.
Under our proposed contract:
| Payload | Expected outcome |
|---|---|
{"query": "refund policy", "limit": 3} |
Accept |
{"query": "refund policy"} |
Reject: missing limit |
{"query": "refund policy", "limit": "a few"} |
Reject: incorrect type |
{"query": "refund policy", "limit": 100} |
Reject: outside allowed range |
{"query": " ", "limit": 3} |
Reject: blank query |
These are illustrative cases, not measured experiment results. They make the intended behavior concrete before implementation.
There’s a subtler case:
{
"query": "refund policy",
"limit": "3"
}
Should "3" become 3, or should validation fail?
That depends on the validation policy.
Pydantic’s documentation distinguishes its usual coercion behavior from strict mode, which restricts conversions and rejects incorrect types more aggressively. Strict behavior also depends on the type and whether validation receives Python objects or JSON.
For this tool’s integer limit, I would reject the string and return a specific error. The key is to choose that behavior deliberately and test it.
The application needs predictable results after execution.
A successful search might return:
{
"status": "success",
"results": [
{
"document_id": "policy_014",
"text": "Refund requests must be submitted within 30 days."
}
]
}
MCP supports an optional outputSchema for structured tool results. When one is declared, the specification requires servers to return conforming structured results and recommends that clients validate them.
That gives downstream code an explicit structure to work with.
It still doesn’t prove that the retrieved text is accurate or relevant. Structural validity and evidence quality need separate checks.
I would represent these outcomes differently:
{"status": "success", "results": []}
{
"status": "error",
"error": {
"code": "SEARCH_UNAVAILABLE"
}
}
The first means the search completed without matches. The second means the search could not be completed.
This is an application design choice. It helps the next step decide whether to report missing evidence, retry, or explain a temporary failure.
A perfectly typed request can still ask for a document the user cannot access.
The MCP specification lists input validation and access controls as separate server responsibilities. Passing schema validation does not establish permission.
For a retrieval tool, I would therefore check both the argument contract and the caller’s access to the requested data.
My takeaway: a tool schema defines what can enter and leave a tool. Runtime validation enforces that contract. Authorization and backend checks determine which actions are allowed.
For a small experiment, I’d start with accepted and rejected payloads, malformed outputs, empty results, and tool failures—and record the observed behavior for each.
What has caused more trouble in your AI applications: unexpected tool arguments or inconsistent tool responses?