Originally published on Loop & Retry — field notes on building LLM agents that survive production.
Most agent bugs I've chased weren't in the model. They were in the tools — specifically, in the gap between what a tool's schema implied it wanted and what it actually did with what it got. The model is a caller that reads your parameter names and descriptions, forms a plausible theory of how the tool works, and acts on that theory under uncertainty. When it misuses a tool, the usual cause is that the tool let it.
You can't make the caller deterministic. You can make the tool hard to misuse. Four properties do most of the work: a legible schema, a validating boundary, recoverable errors, and idempotency. Here's each, with the failing version and the fix.
The schema is the entire spec the model gets. It can't read your code, your docstrings elsewhere, or the ticket that explains the edge case. If the contract isn't in the name, the type, and the description, it doesn't exist. Here's a tool that leaks its contract:
{
"name": "search",
"description": "Search for items.",
"input_schema": {
"type": "object",
"properties": {
"query": {"type": "string"},
"filters": {"type": "string"}, # a string of... what?
"options": {"type": "object"}, # anything goes
"limit": {"type": "integer"},
},
"required": ["query"],
},
}
Every field here invites a guess. filters
is a string, so the model will invent a syntax — "status:open"
, or "status=open,priority=high"
, or JSON, depending on its mood — and you'll parse whichever it picked. options
is a free object, which means the model can pass anything and you handle nothing reliably. limit
has no bounds, so you'll eventually get limit: 10000
. The name search
doesn't say search what.
The fix is to make illegal states unrepresentable in the schema itself, and to spend words on the description where the type can't carry the meaning:
{
"name": "search_support_tickets",
"description": (
"Search the customer support ticket database. Returns tickets ordered "
"by last-updated, newest first. Use `status` and `assignee_email` to "
"narrow results; omit them to search all tickets. Does NOT search "
"archived tickets older than 90 days — use `search_ticket_archive` for those."
),
"input_schema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Free-text search over ticket subject and body.",
},
"status": {
"type": "string",
"enum": ["open", "pending", "resolved", "closed"],
"description": "Filter to one status. Omit to include all statuses.",
},
"assignee_email": {
"type": "string",
"description": "Filter to tickets assigned to this exact email address.",
},
"limit": {
"type": "integer",
"minimum": 1,
"maximum": 50,
"description": "Max results to return (1-50). Default 20.",
},
},
"required": ["query"],
},
}
What changed, and why each matters:
search_support_tickets
, not search
. When an agent has fifteen tools, a bare search
competes with search_docs
and search_users
for the same intent, and the model picks wrong. Name the noun.enum
replaces a free string.minimum
/maximum
bound the blast radius.Keep the surface small, too. Every optional parameter is another axis the model can get wrong. If you have a tool with twelve optional knobs, you probably have three or four tools wearing a trench coat — split them by intent so each call has an obvious shape.
A legible schema constrains what the model can send. It doesn't guarantee what the model should send — semantics the schema can't express (this email must exist, this date range must be non-empty, this ID must belong to the current user). Validate those at the top of the tool, before any side effect, and when you reject, say why in a way the model can act on.
def create_calendar_event(title, start, end, attendee_emails):
event = calendar.insert( # raises deep in the client on bad input
title=title, start=start, end=end, attendees=attendee_emails,
)
return {"event_id": event.id}
If end
is before start
, or attendee_emails
contains a typo'd address, this fails somewhere inside the calendar client with an exception the model never sees cleanly — or worse, it half-succeeds. Compare:
from datetime import datetime
def create_calendar_event(title, start, end, attendee_emails):
errors = []
try:
t0, t1 = datetime.fromisoformat(start), datetime.fromisoformat(end)
if t1 <= t0:
errors.append(
f"`end` ({end}) must be after `start` ({start}). "
"Both must be ISO-8601, e.g. 2026-07-10T14:00:00-04:00."
)
except ValueError:
errors.append(
"`start`/` end` must be ISO-8601 datetimes, "
"e.g. 2026-07-10T14:00:00-04:00."
)
unknown = [e for e in attendee_emails if not directory.exists(e)]
if unknown:
errors.append(
f"These attendees are not in the directory: {unknown}. "
"Check spelling, or call `search_people` to find the correct address."
)
if errors:
return {"ok": False, "errors": errors} # returned, not raised
event = calendar.insert(title=title, start=start, end=end,
attendees=attendee_emails)
return {"ok": True, "event_id": event.id}
The point isn't defensive coding for its own sake. It's that a validating boundary turns "the tool exploded" into "the tool told the model what to fix," and a model can act on the second. Which brings up the property people skip.
When a tool fails, its output goes straight back into the model's context as the next thing it reads. That means your error message is a prompt — it's instructions the model will try to follow. Most tools return errors written for a human tailing logs, and the model does its best with them, which is usually badly.
return {"error": "HTTP 429"}
return {"error": "psycopg2.errors.UniqueViolation: duplicate key value ..."}
return {"error": "null"}
HTTP 429
will make the model retry immediately — exactly the wrong move, and now you're paying the retry tax from the last post for nothing. The stack trace leaks implementation and buries the actionable part. null
tells it nothing. Write errors that say what happened, whether to retry, and what to do instead:
return {
"ok": False,
"error": "rate_limited",
"retry_after_seconds": 30,
"message": "The search API is rate-limited. Wait 30s before retrying, "
"or narrow the query with a `status` filter to reduce load.",
}
return {
"ok": False,
"error": "duplicate",
"message": "A ticket with this external_id already exists (id: T-4821). "
"Use `get_ticket` to read it, or `update_ticket` to modify it. "
"Do not create a new one.",
"existing_id": "T-4821",
}
A good error does three things: names the condition (so the model can branch on it), says whether and when to retry (so it doesn't hammer a rate limit), and offers the recovery path (so it isn't left guessing). The duplicate
case is the sharpest example — instead of the model retrying the create and failing again, the error hands it the existing ID and the two tools that resolve the situation. You've written the recovery into the failure.
Assume every mutating tool gets called more than once with the same arguments. The model retries after a timeout it can't distinguish from a real failure; the harness replays a step; a network blip drops the response after the write landed. If "create" isn't safe to repeat, you get duplicate orders, double charges, and two calendar invites to the same meeting.
def charge_customer(customer_id, amount_cents):
return payments.charge(customer_id, amount_cents)
Make repeated calls converge on the same result. The standard move is a client-supplied idempotency key that the model passes and you deduplicate on:
def charge_customer(customer_id, amount_cents, idempotency_key):
existing = charges.find_by_key(idempotency_key)
if existing:
return {"ok": True, "charge_id": existing.id, "deduplicated": True}
charge = payments.charge(customer_id, amount_cents, key=idempotency_key)
charges.record(idempotency_key, charge.id)
return {"ok": True, "charge_id": charge.id, "deduplicated": False}
with the key in the schema and a description that tells the model how to choose it:
"idempotency_key": {
"type": "string",
"description": (
"A stable unique ID for THIS logical charge, e.g. the order ID. "
"Reusing a key returns the original charge instead of charging again. "
"Use the same key when retrying; use a new key for a genuinely new charge."
),
}
If a stable natural key isn't available, generate the key on the server for the logical operation and dedupe within a time window — the important part is that the tool, not the model's discipline, is what guarantees a retry is safe. Idempotency is what makes the retry budgets from the last post survivable: retries are going to happen; idempotency decides whether they're free or catastrophic.
A short checklist I run through for every tool an agent can call:
search
/get
/run
when the agent has more than a handful of tools.enum
. No unbounded numbers — use minimum
/maximum
.None of this makes the caller deterministic. It makes the tool forgiving of a caller that isn't — which is the only kind of caller you have. The model will still occasionally reach for the wrong tool or pass a strange argument. A well-designed tool turns that from a silent corruption into a legible, recoverable event, and most of the reliability of an agent lives in that difference.