The demo went perfectly. Stakeholders nodded, someone called it magic, and we released the agent to a small group of real users the following Monday. Within an hour, it had issued a refund against the wrong order, called the same tool eleven times in a row, and replied to a customer in a language they never wrote in.
Nothing was wrong with the model. What failed was everything around it. This post covers what broke, why the demo hid it, and the specific changes that made the agent survive contact with real people.
The agent handled support tickets for an e-commerce back office. It read an incoming message, looked up the order through an internal API, drafted a reply, and could issue small refunds through a tool call. The stack was simple: a hosted LLM with function calling, a Python orchestration loop, and a few REST tools.
In the demo, it was flawless. I typed “Where is my order ORD-10293847?” and it fetched the status, wrote a polite answer, and finished in two steps.
A demo is a controlled experiment where you are the only variable. I had made four unconscious choices that guaranteed success:
This gap between a convincing prototype and a dependable system is exactly what custom AI agent development services exist to close. The model call is the easy part. The engineering around it (validation, limits, observability, and testing) is where the real work lives.
Real users violate all four of my demo assumptions within minutes. Here is what that looked like, one failure at a time.
Real tickets arrived as pasted email threads with signatures, quoted replies, and three order numbers. One was a forwarded message in Portuguese. Another contained only “same problem as last time.”
The agent treated all of this as equally trustworthy instructions and picked the first ID it saw, which was in the quoted history rather than the actual complaint.
The fix was a normalization step before the model sees anything: strip quoted replies, detect language, extract candidate IDs, and flag ambiguity. If a message contains more than one candidate order, the agent asks a clarifying question instead of guessing. This single change removed most of our wrong-order incidents.
Function calling gives you structured output, but structured does not mean correct. The model sometimes produced an order ID that matched the format but did not exist, or a refund amount larger than the order total.
I had been trusting arguments because they parsed as JSON. That is not validation. Now every tool call passes through a strict schema, and failures go back to the model as readable errors:
from pydantic import BaseModel, Field, ValidationError
class RefundArgs(BaseModel):
order_id: str = Field(pattern=r"^ORD-\d{8}$")
amount_cents: int = Field(gt=0, le=50_000)
reason: str = Field(min_length=5)
def run_tool(name: str, raw_args: str) -> dict:
if name == "issue_refund":
try:
args = RefundArgs.model_validate_json(raw_args)
except ValidationError as e:
return {"error": "invalid_arguments", "details": e.errors()}
order = orders_api.get(args.order_id)
if order is None:
return {"error": "order_not_found"}
if args.amount_cents > order.total_cents:
return {"error": "amount_exceeds_order_total"}
return refunds_api.create(args)
Returning the error as a tool result lets the model correct itself on the next step, which it usually does. The backend checks matter just as much as the schema, because the schema cannot know whether an order exists.
When the orders API timed out, the agent did what any persistent system does: it tried again. And again. Eleven identical calls later, we had burned tokens and hit a rate limit.
A raw agent loop has no concept of “this is not working.” You have to give it one. I added a step budget and a repeated-call detector:
MAX_STEPS = 8
def run_agent(messages):
seen_calls = set()
for step in range(MAX_STEPS):
resp = llm.chat(messages, tools=TOOLS)
if not resp.tool_calls:
return resp.content
for call in resp.tool_calls:
signature = (call.name, call.arguments)
if signature in seen_calls:
return escalate("repeated_tool_call", messages)
seen_calls.add(signature)
messages.append(run_tool_message(call))
return escalate("step_budget_exceeded", messages)
Escalation matters here. When the agent gives up, it should hand the case to a human with the full context attached, not return a vague apology.
When the first bad refund landed, my debugging tool was print. I had final answers but not the path that produced them: which tool ran, with what arguments, how long it took, and what came back.
Agents are non-deterministic pipelines, so you need traces, not just logs. Every step now emits a structured event:
trace.emit({
"run_id": run_id,
"step": step,
"tool": call.name,
"args_hash": hash_args(call.arguments),
"latency_ms": elapsed_ms,
"tokens_in": usage.input,
"tokens_out": usage.output,
"outcome": "ok" if "error" not in result else result["error"],
})
This turned a bad run into a five-minute investigation instead of a guessing game. We could also see aggregate patterns, such as which tool failed most often and which step consumed the most tokens.
The most expensive bug was behavioral. When uncertain, the agent acted anyway, because language models are fluent and fluency looks like confidence.
I split tools into two classes. Read-only tools run freely. Anything that changes state, like refunds or cancellations, goes through an approval gate:
RISKY_TOOLS = {"issue_refund", "cancel_order", "update_address"}
def run_tool_message(call):
if call.name in RISKY_TOOLS and not is_approved(call):
ticket_id = review_queue.add(call, context=current_messages())
return tool_msg(call, {"status": "pending_human_review", "ticket": ticket_id})
return tool_msg(call, run_tool(call.name, call.arguments))
For low-risk, low-value refunds, we later relaxed the gate with explicit limits. Starting strict and loosening with evidence is far safer than starting open and tightening after an incident.
The best change wasn’t in the agent at all. I started saving every failed conversation as a test case and replaying them after each change:
def test_regressions():
for case in load_jsonl("evals/real_failures.jsonl"):
result = agent.run(case["input"])
assert case["check"](result), f"Regression: {case['id']}"
My demo had never been a test. It was a performance. A regression set built from real failures is the first thing that tells you whether a prompt tweak helped or quietly broke three other behaviors. It also turns every production incident into permanent protection.
If I were starting over, I would change the order of work:
Outside perspective pays off early, too. A friend who does AI consulting asked me one question before launch that I couldn’t answer: “What happens when the agent is wrong?” Had I answered it during design instead of after the incident, most of this post wouldn’t exist.
The agent did not fail because the model was weak. It failed because I had built a demo and called it a product. Production brings messy inputs, flaky dependencies, ambiguous requests, and consequences, and an agent needs engineering for each.
The good news is that these problems are tractable. Normalize inputs, validate tool calls, cap loops, trace everything, gate risky actions, and test against real failures. None of it is glamorous, and all of it is the difference between a demo that impresses and a system people can trust.
What was the first thing that broke when your agent met a real user? Share it in the comments. I’d love to compare failure modes.
A demo uses clean inputs, a cooperative user, and a healthy backend, while production brings messy messages, ambiguous requests, slow APIs, and real consequences. The model usually isn’t the problem. What fails is the missing engineering around it: input normalization, argument validation, loop limits, and testing.
Set a hard step budget and track repeated tool calls. If the agent makes the same call with the same arguments twice, or hits the step limit, stop the run and escalate to a human with the full context attached. It also helps to cap tokens, time, and cost per run.
Start with read-only tools running freely and put anything that changes state, such as refunds, cancellations, or address updates, behind a human approval gate. Once you have real data showing the agent is reliable, relax the gate for low-risk, low-value actions with explicit limits.
Don’t treat the demo as a test. Save every real failed conversation as a regression case and replay the full set after each prompt or code change. This shows whether a tweak helped or quietly broke other behaviors, and it turns each production incident into permanent protection.
Log traces, not just final answers. For every step, capture the run ID, which tool was called, hashed arguments, latency, token usage, and the outcome (success or error type). This turns a bad run into a quick investigation instead of a guessing game, and helps you spot patterns like which tool fails most often.