Originally published on tamiz.pro.
You spent weeks building an agentic workflow that works flawlessly on your local machine. It handles edge cases, calls APIs correctly, and follows the chain of thought precisely. Then you deploy it. Within hours, users report hallucinated tool calls, lost context after five turns, and infinite loops that drain your budget. You stare at the logs and realize the agent isn't broken—it’s just not engineered for production reality.
The gap between a prototype agent and a production-grade system is not complexity; it’s discipline. Most agents fail in production due to three specific engineering gaps: Memory Leakage (context drift and state management), Evaluation Blindness (lack of deterministic testing), and Tooling Fragility (unhandled error states and race conditions). This deep-dive dissects these failure modes and provides the architectural patterns to bridge them.
LLMs are stateless functions. Every token generated is conditioned entirely on the input history provided in the prompt. In production, this simplicity becomes a liability when the conversation exceeds the model’s context window or when “memory” is required across sessions.
The most common failure point is naive prompt accumulation. Developers often push the entire conversation history into every subsequent call:
messages = [
{"role": "system", "content": "You are a helpful assistant..."}
]
for turn in conversation_history: # Grows indefinitely
messages.append(turn)
response = client.chat.completions.create(
model="gpt-4",
messages=messages # Context window blows up
)
messages.append(response)
By turn 10, you’re sending 8,000 tokens of historical noise. Latency spikes, costs explode, and the signal-to-noise ratio degrades the LLM’s reasoning quality—a phenomenon known as lost in the middle.
Production agents require a Hybrid Memory System comprising three layers:
Here’s how to implement a robust memory abstraction layer:
// Core Memory Interface
interface AgentMemory {
// Short-term: Active conversation window
getConversationWindow(userId: string): Promise<Message[]>;
// Medium-term: Semantic recall via embeddings
recallRelevantContext(query: string, userId: string): Promise<ContextChunk[]>;
// Long-term: Persistent fact storage
saveFact(userId: string, fact: string): Promise<void>;
getPersistentProfile(userId: string): Promise<UserProfile>;
}
// Implementation Strategy: Sliding Window + Summary
async function getConversationWindow(userId: string): Promise<Message[]> {
const fullHistory = await db.getMessages(userId);
if (fullHistory.length <= MAX_WINDOW_SIZE) {
return fullHistory;
}
// Keep last N turns raw, compress older history
const recent = fullHistory.slice(-MAX_WINDOW_SIZE);
const older = fullHistory.slice(0, -MAX_WINDOW_SIZE);
// Generate summary of older context
const summary = await llm.summarize(older);
return [summary, ...recent];
}
Key Insight: Never treat the LLM as the database. Use the LLM only for reasoning; use databases for storage. The separation of concerns is what keeps production agents stable.
You can’t unit test an LLM like you test a Java service. Non-determinism, prompt sensitivity, and semantic correctness make traditional assertions impossible. Yet most teams skip evaluation entirely, assuming “it works on my prompt” is sufficient.
When you send the same prompt twice to an LLM, you get different outputs. This isn’t a bug—it’s temperature. But production systems often require determinism for debugging and consistency. The solution isn’t to disable randomness but to control the evaluation surface.
For production, you need a test suite that evaluates semantic correctness, not exact string matching. Use LLM-as-a-judge patterns where a secondary LLM scores the primary agent’s output against a rubric.
from typing import List, Dict
import asyncio
async def evaluate_agent_response(
user_input: str,
agent_response: str,
expected_fact: str,
model: str = "gpt-4-turbo"
) -> Dict[str, float]:
evaluation_prompt = f"""
Evaluate the following agent response for factual correctness and tool usage.
User Input: {user_input}
Agent Response: {agent_response}
Expected Fact: {expected_fact}
Score from 0-10 based on:
1. Did the agent call the correct tool?
2. Is the response factually aligned with the expected fact?
3. Was the tone appropriate?
Return JSON only: {{"tool_call_correct": bool, "factual_score": int, "overall_score": int}}
"""
result = await llm.complete(evaluation_prompt)
return parse_json(result)
Build a Golden Dataset—a curated set of 50–100 representative user queries with expected tool calls and responses. Run this dataset weekly against your agent. If the score drops, you have a regression.
| Test Case Type | Purpose | Metric |
|---|---|---|
| Syntax | ||
| Does the agent call tools with valid JSON? | % Valid Tool Calls |
|
| Semantic | ||
| Does the response answer the user’s intent? | LLM-as-a-Judge Score |
|
| Safety | ||
| Does the agent refuse harmful requests? | % Blocked Attacks |
|
| Cost | ||
| How many tokens per successful task? | Tokens per Turn |
Without this baseline, you are flying blind. A 5% drop in accuracy might be invisible to manual QA but catastrophic at scale.
Tools are the hands of your agent. In prototypes, tools are simple HTTP calls. In production, they are complex integrations subject to network timeouts, API schema changes, rate limits, and authentication failures.
Consider an agent that needs to:
If step 2 fails, what happens? Most naive implementations halt or retry infinitely. Production agents need circuit breakers and graceful degradation.
Never retry indefinitely. Implement bounded retries with exponential backoff and jitter.
async function callToolWithResilience(
toolName: string,
args: any,
maxRetries: number = 3
): Promise<any> {
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
return await executeTool(toolName, args);
} catch (error) {
if (attempt === maxRetries - 1) throw error;
// Exponential backoff with jitter
const delay = Math.pow(2, attempt) * 1000 + Math.random() * 1000;
console.warn(`Tool ${toolName} failed, retrying in ${delay}ms`);
await sleep(delay);
}
}
}
If a downstream API (e.g., Slack, Salesforce) is down, don’t waste tokens asking the LLM to “try again.” Use a circuit breaker pattern to fail fast.
class CircuitBreaker:
def __init__(self, service_name: str, threshold: int = 5):
self.service_name = service_name
self.failure_count = 0
self.threshold = threshold
self.state = "CLOSED" # CLOSED, OPEN, HALF_OPEN
async def execute(self, func, *args):
if self.state == "OPEN":
raise Exception(f"Service {self.service_name} is circuit-broken")
try:
result = await func(*args)
self.failure_count = 0
self.state = "CLOSED"
return result
except Exception as e:
self.failure_count += 1
if self.failure_count >= self.threshold:
self.state = "OPEN"
raise
LLMs often hallucinate tool parameters. Always validate inputs before executing tools. This prevents 400 errors from downstream APIs and keeps the agent on track.
// VALIDATION BEFORE EXECUTION
const validatedArgs = z
.object({
query: z.string().min(1),
date_range: z.object({ start: z.string(), end: z.string() })
})
.safeParse(agentOutput.toolInputs);
if (!validatedArgs.success) {
// Return structured error to LLM so it can self-correct
return {
isError: true,
message: "Invalid tool parameters: " + validatedArgs.error.message
};
}
Most agents fail because teams lack observability. They see a user complaint but can’t trace why. Production requires three pillars of observability:
Every LLM call, tool invocation, and memory read must be logged with unique trace IDs. Use OpenTelemetry or LangSmith to instrument your agent.
import logging
import uuid
logger = logging.getLogger("agent.tracer")
async def run_agent_step(user_id: str, step: str, input_data: dict):
trace_id = str(uuid.uuid4())
logger.info(
"Agent Step Start",
extra={
"trace_id": trace_id,
"user_id": user_id,
"step": step,
"input_tokens": len(input_data),
"timestamp": datetime.utcnow().isoformat()
}
)
try:
result = await execute_step(step, input_data)
logger.info(
"Agent Step Success",
extra={
"trace_id": trace_id,
"output_tokens": len(result),
"latency_ms": calculate_latency(),
"cost_usd": estimate_cost(result)
}
)
return result
except Exception as e:
logger.error(
"Agent Step Failed",
extra={
"trace_id": trace_id,
"error": str(e),
"stack_trace": traceback.format_exc()
}
)
raise
Add a cost-per-request middleware. If a single agent turn costs $0.50 instead of $0.05, you need to know immediately.
Design your agent to recognize uncertainty. If the confidence score drops below a threshold, hand off to a human operator. Log the handoff reason for future training.
if confidence_score < 0.7:
await escalate_to_human(
user_query=user_query,
agent_thought_process=agent_thoughts,
suggested_action="Manual review required"
)
return {"status": "escalated", "trace_id": trace_id}
Bridging these gaps requires a cultural shift from “prompt engineering” to “agent systems engineering.” Here’s the checklist for production readiness:
Q: How do I test an agent without a large labeled dataset?
A: Start with a small set of 20–30 high-confidence cases (your “happy path”). Use synthetic data generation to expand this over time. An LLM can generate plausible edge cases by mutating your golden dataset.
Q: Should I use RAG for all memory types?
A: No. Use RAG (vector search) for unstructured, semantic recall (e.g., “what did we discuss last week?”). Use structured databases for factual data (e.g., user preferences, order history). Mixing these approaches leads to expensive, slow, and inaccurate retrieval.
Q: How do I handle rate limits from upstream APIs?
A: Implement a token bucket or leaky bucket rate limiter in your tool orchestration layer. If you hit the limit, return a structured error to the LLM asking it to retry later or provide partial information, rather than crashing the entire conversation.
For more insights on production AI engineering patterns, explore advanced guides on agentic workflows and systematic evaluation frameworks.