Originally published on tamiz.pro.
The demo works beautifully. The agent reads the inbox, writes a draft, calls the API, and updates the database — all in one fluid, 90-second recording where the LLM "just knows" what to do. Then someone asks you to ship it. That's when the real work begins.
Every engineering team watching the AI agent wave is now under pressure to deliver. But the path from a polished demo to a production system is not a linear scaling exercise. It's a series of failure modes that no prompt can save you from. Below are the patterns I've seen break production deployments — and the engineering disciplines required to fix them.
LLMs are non-deterministic by design. A demo succeeds because the recording was made on a happy path. Production doesn't care about happy paths — it only cares about edge cases, which are exponentially more numerous than the scenarios you tested in the demo.
The first lesson: stop treating LLM output as a stable API. Every agent call must be validated against a schema. Use structured output (JSON mode, function calling, Pydantic validators) and treat every malformed response as a system error, not a prompt-tuning problem.
from pydantic import BaseModel, Field
from typing import Literal
class ActionRequest(BaseModel):
intent: Literal["query", "write", "execute", "escalate"]
target_resource: str
parameters: dict[str, str]
confidence: float = Field(ge=0.0, le=1.0)
def parse_agent_response(raw: str) -> ActionRequest:
return ActionRequest.model_validate_json(raw)
Demos showcase agents with three or four tools. Production agents need dozens, often interacting with systems that have inconsistent APIs, auth gateways, rate limits, and partial failures.
The key architectural shift: the agent is a scheduler, not a reasoner. Your orchestration layer must enforce timeouts, circuit breakers, idempotency keys, and retry logic around every tool call. An agent that can loop forever because a tool timed out is an agent that will burn $4,000 in a single night.
agent_framework:
max_steps: 15
step_timeout_seconds: 30
total_timeout_seconds: 120
circuit_breaker:
threshold: 5
cooldown_seconds: 60
idempotency:
key_generator: "hash(input + tool_name + attempt)"
retry_policy: "exponential_backoff, max 3"
"Just throw more context at it," the demo engineer says. Production engineers learn that a 128K context window with 90% retrieval noise performs worse than a 8K window with precise, curated context.
The real engineering challenge is context management at scale. This means:
This is the single biggest gap between demo and production. Demos are evaluated by humans watching a video. Production systems must be evaluated by automated pipelines that measure correctness, latency, cost, and safety across thousands of scenarios.
Good production evaluation requires:
This is where most startups hit the wall. Your agent touches customer data, makes API calls on their behalf, and generates content that may be regulated. The demo didn't have a compliance officer. Production does.
Requirements that will reshape your architecture:
orders.all()
should not be the same agent that writes to users.payments
.A demo runs once. Production runs continuously, and the token bill is real. Consider these production-scale costs for a typical agent handling 1,000 requests/day:
| Component | Approximate cost/month | What drives it |
|---|---|---|
| LLM inference (reasoning + tool calls) | $2,400–$8,000 | Token volume, model tier |
| Embedding + retrieval store | $400–$1,200 | Vector dimension, query frequency |
| Tool orchestration (API calls) | $200–$600 | External API rates, retries |
| Logging + observability | $150–$400 | Audit log retention |
| Human review queue | $500–$2,000 | SLA-bound triage |
The question isn't whether your agent is accurate. It's whether the economics work at scale. Many demos fail here because the token budget assumed 10-step reasoning paths. Production reveals that the average path is 47 steps with 3 retries.
After years of watching this pattern repeat, the agents that make it to production share a few traits:
The gap between demo and production isn't a prompt engineering problem. It's an engineering problem — the kind your team already knows how to solve: testing, observability, cost control, failure modes, and incremental rollout.
The agents that ship aren't the ones with the smartest prompts. They're the ones with the tightest feedback loops, the harshest evals, and the humility to admit when a rule-based system does the job better. The hype will fade. The systems you build to handle failure will remain.
Q: Should I use agentic frameworks like LangChain or AutoGen for production?
A: They are useful for prototyping, but most production teams strip them down to their primitives within six months. The abstractions that hide tool-calling complexity in a demo add invisible complexity in production. Use them to prototype, then rebuild the critical paths with minimal, observable code.
Q: How do I know when my agent is ready for production?
A: When your automated evals show >95% task success on your golden test set, p99 latency is under your SLA, cost per task is sustainable, and you have a rollback plan for every model update. No amount of human praise during a demo satisfies this bar.
Q: Is it worth building AI agents when rule-based systems solve 80% of the problem?
A: Yes — if the remaining 20% involves ambiguity, natural language understanding, or judgment calls that rules can't capture. The architecture should reflect this: rules for the deterministic 80%, agents for the rest, with clear handoff boundaries between them.