Stop Blaming the LLM: Why Your AI Agent's Bugs Are Actually Distributed Systems Failures

A developer argues that production AI agent failures commonly blamed on the LLM are actually distributed systems failures, since an agent spans an orchestrator, LLM API, tool executors, vector stores and state stores, each a separate failure domain with its own latency and error modes. The writeup maps symptoms such as repeated output, truncated responses, hallucinated facts and ignored instructions to causes like tool timeouts and retries, gateway cutoffs, stale vector results and truncated system prompts, and recommends treating agents with distributed systems operational practices.

Originally published on tamiz.pro https://tamiz.pro/insights/ai-agent-bugs-distributed-systems-failures . You ship an AI agent. It works in staging. Production hits and suddenly it's "hallucinating," "ignoring instructions," or "getting stuck in loops." The team blames the model. They swap providers, tweak prompts, add few-shot examples. The bugs persist. Here's the uncomfortable truth: your AI agent is not a model. It's a distributed system. Every agent—whether a simple ReAct loop or a multi-agent swarm—decomposes into multiple independent processes communicating over unreliable networks: the LLM API, tool executors code interpreters, browsers, API clients , vector databases, memory stores, orchestration logic, and often human-in-the-loop checkpoints. Each component fails independently. Each introduces latency variance. Each has its own failure modes that look exactly like "model stupidity" when observed from the outside. This article maps the anatomy of an AI agent to distributed systems primitives, shows why classic LLM debugging fails, and gives you the operational toolkit to treat agents like the distributed systems they are. Draw the architecture of any production agent and you'll find these components: ┌─────────────────────────────────────────────────────────────┐ │ ORCHESTRATOR Your Code │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Planner │→ │ Executor │→ │ Critic │→ │ Memory │ │ │ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ │ └───────┼────────────┼────────────┼────────────┼──────────────┘ │ │ │ │ ▼ ▼ ▼ ▼ ┌───────────────┐ ┌─────────┐ ┌──────────┐ ┌────────────┐ │ LLM Provider │ │ Tools │ │ Vector │ │ State │ │ API Gateway │ │ Code, │ │ Store │ │ Store │ │ │ │ Browse,│ │ Pinecone│ │ Redis, │ │ - Rate limits│ │ API │ │ , etc. │ │ Postgres │ │ - Latency │ │ │ │ │ │ │ │ - Errors │ │ - Sand- │ │ - Index │ │ - TTL │ │ - Quotas │ │ boxes │ │ lag │ │ - Locks │ └───────────────┘ └─────────┘ └──────────┘ └────────────┘ Each box is a separate failure domain with its own: The orchestrator is the only component you fully control. Everything else is a remote service you don't own . A typical agent turn involves 3–10 LLM calls plus tool invocations. At 500ms–3s per LLM call and 100ms–5s per tool, a single user request spans 5–60 seconds of wall-clock time across 10–50 network hops. ponytail: naive sequential execution, parallelize independent calls if latency matters async def agent turn user input: str, ctx: AgentContext - AgentResponse: plan = await llm.plan user input, ctx.history 1.2s tool results = await execute tools plan.tools 2.8s parallel critique = await llm.critique plan, tool results 0.9s memory.update ctx.session id, critique 0.1s return await llm.synthesize plan, tool results, critique 1.1s Total: ~6s sequential, ~3s with parallelization Distributed systems rule : latency adds up, and the tail latency dominates. The 99th percentile of your agent's response time is determined by the slowest component's 99th percentile, compounded across hops. | Observed Symptom | Actual Distributed Systems Cause | Why It Looks Like an LLM Bug | |---|---|---| | Agent repeats itself | Tool call timed out, orchestrator retries with same prompt | "Model is stuck in a loop" | | Incomplete output | LLM stream cut off by gateway timeout | "Model stopped reasoning" | | Hallucinated facts | Vector store returned stale/empty results | "Model doesn't know" | | Ignores instructions | System prompt truncated by context window management | "Model doesn't follow prompts" | | Nonsensical tool args | Tool schema validation failed silently, fell back to string parsing | "Model can't use tools" | | Inconsistent behavior | Race condition between memory reads/writes across replicas | "Model is unpredictable" | | Silent degradation | Provider silently routes to smaller/faster model variant | "Model got dumber" | Most LLM APIs stream tokens. Your gateway Cloudflare, AWS ALB, nginx has a default idle timeout of 60s . A complex reasoning trace with tool calls can exceed this. What the model actually produced streaming : "I need to check the database... tool: query db ... The result shows 42 users. Now I'll calculate... tool: python ... The answer is 42 1.5 = 63. So the final answer is—" What your orchestrator receives gateway timeout at 60s : "I need to check the database... tool: query db ... The result shows 42 users. Now I'll calculate... tool: python ... The answer is 42 1.5 = 63. So the final answer is" Your code sees: incomplete response → treats as model error → retries The model did reason correctly. The infrastructure ate the tail. Major providers route traffic across model variants e.g., gpt-4o → gpt-4o-mini for cost optimization without changing the model identifier. You observe quality drift. You blame "prompt sensitivity." The reality: you're calling a different model. Detect this by logging model fingerprint headers OpenAI returns: x-request-id, x-ratelimit- , and sometimes model fingerprint curl -v https://api.openai.com/v1/chat/completions ... 2 &1 | grep -i fingerprint x-model-fingerprint: fp abc123 ← log this every request Agents retry. A lot. Network blips, 429s, 5xx errors, timeouts—all trigger retries. But most agent operations are not idempotent . python DANGEROUS: Non-idempotent operations in a retry loop async def run agent task: Task - Result: for attempt in range 3 : try: plan = await llm.plan task Idempotent read-only result = await execute plan plan NOT idempotent return result except TransientError: continue raise MaxRetriesExceeded execute plan might: - Charge a credit card twice - Send an email twice - Mutate database state twice - Create a Jira ticket twice Every tool invocation that mutates state must carry an idempotency key derived from the agent's execution context. python import hashlib import json from dataclasses import dataclass from typing import Any @dataclass class ToolInvocation: tool name: str args: dict str, Any idempotency key: str attempt: int = 0 @classmethod def create cls, tool name: str, args: dict, execution id: str, step: int - "ToolInvocation": Deterministic key: execution id + step + tool + args hash content = f"{execution id}:{step}:{tool name}:{json.dumps args, sort keys=True }" key = hashlib.sha256 content.encode .hexdigest :32 return cls tool name=tool name, args=args, idempotency key=key Tool executor checks idempotency store before executing async def execute tool invocation: ToolInvocation - ToolResult: existing = await idempotency store.get invocation.idempotency key if existing: return ToolResult.from cached existing Return previous result result = await actual tool call invocation.tool name, invocation.args await idempotency store.set invocation.idempotency key, result, ttl=86400 return result Retrying the entire agent loop because one tool failed is wasteful and dangerous. It re-does reasoning, re-calls other tools, and burns tokens. python BETTER: Granular retry at tool level, not agent level async def execute plan plan: Plan, ctx: ExecutionContext - PlanResult: results = {} for step in plan.steps: if step.depends on and not all results d .success for d in step.depends on : results step.id = StepResult skipped=True, reason="dependency failed" continue Retry ONLY this step, with its own budget results step.id = await execute step with retry step, max attempts=3, timeout=step.timeout or 30, idempotency key=ctx.make key step.id return PlanResult results An agent turn is a long-running transaction across multiple services. When step 3 of 5 fails, steps 1–2 may have already mutated state. You need compensating actions —the Saga pattern. @dataclass class SagaStep: name: str execute: Callable , Awaitable StepResult compensate: Callable StepResult , Awaitable None Undo action idempotency key: str class AgentSaga: def init self, execution id: str : self.execution id = execution id self.completed steps: list SagaStep = self.step results: dict str, StepResult = {} async def run self, steps: list SagaStep - SagaResult: for step in steps: try: result = await step.execute self.step results step.name = result self.completed steps.append step except Exception as e: Compensate in reverse order await self. compensate return SagaResult success=False, failed step=step.name, error=e return SagaResult success=True, results=self.step results async def compensate self : for step in reversed self.completed steps : try: await step.compensate self.step results step.name except Exception as comp err: Log but continue compensating other steps logger.error f"Compensation failed for {step.name}: {comp err}" await alert oncall f"Saga {self.execution id} compensation failed at {step.name}" | Tool Action | Compensation | |---|---| | send email | Send follow-up "Please disregard previous email" | | create jira ticket | Transition ticket to "Cancelled" | | charge customer | Initiate refund via payment provider | | write database | Delete inserted rows / revert update | | call external api | Call cancellation endpoint if available | | create file | Delete file | Critical : Compensations themselves can fail. They must be idempotent and retried independently. Store saga state in durable storage PostgreSQL, not memory so crashes don't leave dangling partial state. Most agent observability stops at "input → output." That's like debugging a microservice by only reading the API gateway logs. You need distributed tracing across every hop . Every agent execution gets a trace id . Every sub-operation gets a span id with parent linkage. python from contextvars import ContextVar import uuid current trace: ContextVar str = ContextVar "current trace", default="" current span: ContextVar str = ContextVar "current span", default="" class TracedAgent: async def run self, input: str - AgentOutput: trace id = str uuid.uuid4 current trace.set trace id with span trace id, "agent.run", {"input": input} as root span: plan = await self. plan with trace input, root span tool results = await self. execute with trace plan, root span output = await self. synthesize with trace plan, tool results, root span root span.set attribute "output", output return output async def plan with trace self, input: str, parent span - Plan: with span parent span.trace id, "llm.plan", {"model": self.model}, parent span.span id as span: span.set attribute "prompt tokens", estimate tokens input result = await self.llm.plan input span.set attribute "completion tokens", estimate tokens result span.set attribute "latency ms", span.duration ms return result | Signal | Where to Capture | Alert Threshold | |---|---|---| | Latency p50, p95, p99 | Per LLM call, per tool, per agent turn | p99 30s | | Error rate | Per provider, per tool, per error type | 1% 5xx | | Token usage | Input/output per call, per session | Budget exceeded | | Retry rate | Per operation type | 10% retries | | Partial failure rate | Saga compensation triggered | 0.1% | | Context window pressure | Tokens used / max tokens per call | 80% | | Tool success rate | Per tool, per error category | < 99% | { "timestamp": "2024-01-15T10:23:45.123Z", "level": "INFO", "trace id": "abc-123", "span id": "span-456", "component": "tool executor", "tool": "web search", "args": {"query": "quarterly revenue", "max results": 5}, "idempotency key": "abc123...", "attempt": 2, "latency ms": 2340, "status": "success", "result summary": {"results count": 5, "tokens": 1200} } Query this in production : "Show me all traces where web search took 5s and the subsequent LLM call produced <100 tokens." You cannot unit-test an LLM call like a pure function. But you can test the system around it . Record production traces inputs, outputs, timings, errors . Replay them against code changes. python test replay.py import pytest from agent.tracing import TraceRecord from agent.orchestrator import AgentOrchestrator @pytest.mark.parametrize "trace", load production traces "2024-01", sample=1000 async def test agent replay trace: TraceRecord : """Replay a real production trace through current code.""" orchestrator = AgentOrchestrator llm=FakeLLM responses=trace.llm responses , Deterministic replay tools=FakeTools responses=trace.tool responses , memory=FakeMemory initial state=trace.initial memory result = await orchestrator.run trace.input Assert structural equivalence, not exact string match assert result.tool calls == trace.expected tool calls assert result.final answer.semantic equiv trace.expected answer assert result.latency ms < trace.original latency ms 1.5 No regression Inject failures at the orchestration layer to verify compensations, retries, and fallbacks. class ChaosAgentWrapper: """Wraps an agent and injects failures per configuration.""" def init self, agent, chaos config: ChaosConfig : self.agent = agent self.chaos = chaos config async def run self, input: str - AgentOutput: if self.chaos.should fail "llm timeout" : raise TimeoutError "Injected LLM timeout" if self.chaos.should fail "tool error" : raise ToolError "Injected tool failure" if self.chaos.should delay "network latency" : await asyncio.sleep self.chaos.latency ms / 1000 return await self.agent.run input Run in CI: 1000 iterations with 5% random failure injection Assert: no uncompensated mutations, no lost user data, graceful degradation Treat your eval set as a regression test suite , not a one-time benchmark. evals/regression suite.yaml cases: - id: "finance calc 001" input: "What's 15% of $2,340?" expected tools: "calculator" expected answer pattern: "351" max latency ms: 5000 tags: "math", "single tool" - id: "multi hop research 001" input: "Compare Q3 revenue of Apple vs Microsoft" expected tools: "web search", "calculator", "web search" expected answer contains: "Apple", "Microsoft", "revenue", "billion" max latency ms: 30000 tags: "multi hop", "research" php async def run regression suite agent, suite path: str - EvaluationReport: results = for case in load suite suite path : trace = await agent.run with trace case.input results.append EvalResult case id=case.id, passed=check case case, trace , latency ms=trace.total latency ms, tokens used=trace.total tokens, trace id=trace.trace id return EvaluationReport results Distributed systems survive partial degradation through explicit boundaries . Your agent needs the same. python from enum import Enum import time class CircuitState Enum : CLOSED = "closed" Normal operation OPEN = "open" Failing fast HALF OPEN = "half open" Testing recovery class ProviderCircuitBreaker: def init self, failure threshold: int = 5, recovery timeout: float = 30.0, half open max calls: int = 3 : self.state = CircuitState.CLOSED self.failure count = 0 self.last failure time = 0 self.half open calls = 0 self.failure threshold = failure threshold self.recovery timeout = recovery timeout self.half open max calls = half open max calls async def call self, func: Callable, args, kwargs : if self.state == CircuitState.OPEN: if time.time - self.last failure time self.recovery timeout: self.state = CircuitState.HALF OPEN self.half open calls = 0 else: raise CircuitOpenError "Circuit breaker open" if self.state == CircuitState.HALF OPEN: if self.half open calls = self.half open max calls: raise CircuitOpenError "Half-open limit reached" self.half open calls += 1 try: result = await func args, kwargs self. on success return result except Exception as e: self. on failure raise def on success self : self.failure count = 0 if self.state == CircuitState.HALF OPEN: self.state = CircuitState.CLOSED def on failure self : self.failure count += 1 self.last failure time = time.time if self.failure count = self.failure threshold: self.state = CircuitState.OPEN Wire it per-provider: llm clients = { "openai": CircuitBreakerClient OpenAIClient , ProviderCircuitBreaker , "anthropic": CircuitBreakerClient AnthropicClient , ProviderCircuitBreaker , "local": CircuitBreakerClient LocalLLMClient , ProviderCircuitBreaker failure threshold=10 , } async def call llm with fallback messages, preferred="openai" : for provider in preferred, "anthropic", "local" : try: return await llm clients provider .complete messages except CircuitOpenError, RateLimitError, TimeoutError : continue raise AllProvidersUnavailable Fixed timeouts are wrong. Set timeouts at p99 + buffer per operation type. python class AdaptiveTimeout: def init self, percentile: float = 0.99, buffer multiplier: float = 1.5 : self.percentile = percentile self.buffer = buffer multiplier self.latencies: list float = Rolling window self.max samples = 1000 def record self, latency ms: float : self.latencies.append latency ms if len self.latencies self.max samples: self.latencies.pop 0 def get timeout self - float: if len self.latencies < 10: return 30000 Default 30s sorted lat = sorted self.latencies idx = int len sorted lat self.percentile return sorted lat idx self.buffer Per-operation adaptive timeouts timeouts = { "llm.plan": AdaptiveTimeout 0.99, 1.5 , "llm.critique": AdaptiveTimeout 0.99, 1.5 , "tool.web search": AdaptiveTimeout 0.95, 2.0 , More variable "tool.code exec": AdaptiveTimeout 0.99, 1.2 , "memory.read": AdaptiveTimeout 0.999, 1.1 , } async def execute with timeout op name: str, coro : timeout = timeouts op name .get timeout try: start = time.time result = await asyncio.wait for coro, timeout=timeout / 1000 timeouts op name .record time.time - start 1000 return result except asyncio.TimeoutError: timeouts op name .record timeout Record timeout as latency raise When things go wrong, degrade functionality , not correctness . class DegradedAgent: """Agent that progressively sheds capabilities under stress.""" async def run self, input: str, context: AgentContext - AgentOutput: Level 0: Full capability if not self.degraded: return await self. full run input, context Level 1: No web search slow, unreliable if self.degraded level = 1: context.available tools.discard "web search" Level 2: No code execution sandbox overhead if self.degraded level = 2: context.available tools.discard "python" Level 3: Single-turn only, no planning loop if self.degraded level = 3: return await self. single turn fallback input, context Level 4: Canned responses only return await self. canned fallback input def update health self, metrics: HealthMetrics : """Called periodically by health monitor.""" if metrics.error rate 0.2 or metrics.p99 latency 60000: self.degraded level = min self.degraded level + 1, 4 elif metrics.error rate < 0.01 and metrics.p99 latency < 15000: self.degraded level = max self.degraded level - 1, 0 "Agent gets stuck repeating 'Let me search for that...' three times then gives generic answer. Model quality issue?" Trace analysis revealed: Trace: agent 7x9k2 ├─ llm.plan 1.2s ✓ ├─ tool.web search "quarterly revenue" 28.3s → TIMEOUT 30s gateway ├─ llm.plan RETRY 1.1s → "Let me search for that..." ├─ tool.web search "quarterly revenue" 29.1s → TIMEOUT ├─ llm.plan RETRY 1.3s → "Let me search for that..." ├─ tool.web search "quarterly revenue" 31.2s → TIMEOUT ├─ llm.synthesize 0.8s → Generic answer no tool results Root cause : The search API's p99 latency was 28s. The gateway timeout was 30s. The orchestrator retried the entire plan on any tool failure. Three retries = 90s of user-facing latency. 1. Increase gateway timeout to 60s match search p99 + buffer 2. Make web search idempotent cache results for 5min 3. Retry ONLY the failed tool, not the whole plan Before broken : async def run agent : for attempt in range 3 : try: return await full agent loop except ToolTimeout: continue After fixed : async def run agent : plan = await llm.plan No retry needed, idempotent results = {} for step in plan.steps: results step.id = await execute with retry step, max attempts=2, timeout=60 Per-step config return await llm.synthesize plan, results Result : Loop eliminated. Latency p99 dropped from 90s → 35s. User complaints dropped to zero. A: If it's truly one call with no tools, no memory, no retries—then yes, it's just an LLM call. But the moment you add: a second LLM call, a tool, a retry loop, or session memory—you have a distributed system. The complexity threshold is lower than most teams realize. A: Start with one trace that proves the point. Find a "model bug" ticket, add distributed tracing, show the actual failure was a timeout/cascade/partial failure. The evidence speaks louder than architecture diagrams. A: Structured JSON logs with trace id / span id + a log aggregator Datadog, Loki, CloudWatch + basic dashboards for the 7 golden signals in section 5.2. You don't need Jaeger/Tempo to start—correlation IDs in logs get you 80% of the value. A: No. Compensation is only for mutating operations writes, charges, sends, creates . Search, calculation, read-only API calls need no compensation—but they do need idempotency keys if you retry them, to avoid duplicate costs/latency. This article is adapted from patterns used in production agent systems at scale. The code snippets are simplified for clarity—production versions include metrics emission, structured error types, and integration with your observability stack. The second pillar is idempotency . LLMs are inherently non-deterministic; the same prompt can yield different completions across calls. When an agent retries a failed action, duplicate side-effects become a critical problem. The standard pattern is to require an idempotency key from the caller, which the service stores and returns unchanged on repeat executions. Consider a checkout flow where an agent initiates a payment, the LLM times out, and retries. Without idempotency, the customer is charged twice. With it, the payment service recognizes the key and returns the original transaction status. Agent side: generate and attach a key payment id = "pay 12345" idempotency key = f"checkout-{payment id}-{int time.time }" result = await agent.call tool "payments process", {"amount": 999, "currency": "USD", "idempotency key": idempotency key}, python Service side: check cache before executing async def process payment data: dict : key = data.pop "idempotency key", None cached = idempotency store.get key if cached: return cached return original result unchanged ... actual payment processing ... idempotency store.set key, result, ttl=86400 return result ponytail: in-process cache only; production uses distributed ID generators Redis, DynamoDB with global TTL coordination. The third pillar is liveness enforcement . Distributed systems fail—network partitions, GC pauses, thread starvation. An agent without timeouts will hang indefinitely, consuming resources and cascading failures downstream. Every external call must have a hard timeout, and the agent must have a circuit-breaker pattern to stop hammering unhealthy services. async with asyncio.timeout 5 : 5-second hard limit result = await agent.call tool "some remote service", payload Circuit breaker state persisted across restarts class CircuitBreaker: def init self, failure threshold=5, recovery timeout=30 : self.failure threshold = failure threshold self.recovery timeout = recovery timeout self.failures = 0 self.last failure = None self.state = "closed" closed/open/half-open async def call self, fn, args, kwargs : if self.state == "open": if time.time - self.last failure self.recovery timeout: self.state = "half-open" else: raise CircuitOpenError "service temporarily unavailable" try: result = await fn args, kwargs self. on success return result except Exception as e: self. on failure raise def on success self : self.failures = 0 self.state = "closed" def on failure self : self.failures += 1 self.last failure = time.time if self.failures = self.failure threshold: self.state = "open" ponytail: simple in-memory breaker; production persists state and uses exponential backoff with jitter. The fourth pillar is system architecture choice . In a choreographed system, agents react to events without central coordination—message queues, topics, eventual consistency. In an orchestrated system, a conductor or state machine drives the flow. For AI agents, pure choreography often leads to undetected stuck states; a lightweight orchestrator Temporal, LangGraph, or even a DB-driven state table makes liveness and idempotency guarantees explicit. Consider the difference: in a choreographed flow, an agent receives a "payment completed" event but the UI never renders because the agent crashed before updating state. An orchestrated flow tracks each step's completion explicitly and can replay or compensate on failure. All of this hinges on observability. You cannot fix what you cannot measure. Distributed systems already require tracing, metrics, and logs; AI agents add non-determinism on top. Structure your agent outputs so they integrate with your existing stack. Metrics to emit per call: agent call duration seconds agent call success total increment on success, decrement on failure agent call error type labels: timeout, validation, external service, llm token limit agent retry count histogram or counter Structured logs should include: request id correlates across all service calls llm model and temperature tokens used and cost estimate external service latency idempotency key for duplicate detection debugging Traces should span the entire agent workflow: user prompt → tool selection → tool execution → response generation. If a tool fails, the trace shows exactly which step and what input caused it. Below is a minimal, runnable skeleton that demonstrates the patterns discussed. It is not a production-grade framework but illustrates the core ideas: timeout, idempotency key, circuit breaker, and structured metrics. python import asyncio import time import random from typing import Optional, Dict, Any --- Mock infrastructure --- class InMemoryIdempotencyStore: def init self : self. store: Dict str, Any = {} def get self, key: str - Optional Any : return self. store.get key def set self, key: str, value: Any, ttl: int = 3600 : self. store key = value class SimpleCircuitBreaker: def init self, failure threshold: int = 5, recovery timeout: float = 30.0 : self.failure threshold = failure threshold self.recovery timeout = recovery timeout self.failures = 0 self.last failure: Optional float = None self.state = "closed" closed / open / half-open async def call self, fn, args, kwargs : if self.state == "open": if time.time - self.last failure self.recovery timeout: self.state = "half-open" else: raise RuntimeError "CircuitOpen: service unavailable" try: result = await fn args, kwargs self. on success return result except Exception as e: self. on failure raise def on success self : self.failures = 0 self.state = "closed" def on failure self : self.failures += 1 self.last failure = time.time if self.failures = self.failure threshold: self.state = "open" --- Agent core --- class DistributedAgent: def init self : self.idempotency = InMemoryIdempotencyStore self.circuit = SimpleCircuitBreaker self.metrics = {"calls": 0, "errors": 0, "retries": 0} async def call external self, payload: dict - dict: """Wrap external calls with timeout, idempotency, and circuit breaking.""" Attach idempotency key if present key = payload.pop "idempotency key", None if key: cached = self.idempotency.get key if cached: self.metrics "calls" += 1 return cached Enforce timeout try: async with asyncio.timeout 5 : result = await self.circuit.call self. do call, payload except asyncio.TimeoutError: self.metrics "errors" += 1 raise RuntimeError "Call timed out after 5s" from None Cache result for idempotency if key: self.idempotency.set key, result, ttl=3600 self.metrics "calls" += 1 return result async def do call self, payload: dict - dict: """Actual external call—here simulated with random latency.""" Simulate flaky external service await asyncio.sleep random.uniform 0.1, 2.0 if random.random < 0.3: raise RuntimeError "Simulated external failure" return {"status": "ok", "data": payload.get "value" } async def run workflow self, input value: str - dict: """Example workflow that uses all four pillars.""" Step 1: idempotent lookup lookup key = f"workflow-{input value}" cached = self.idempotency.get lookup key if cached: self.metrics "retries" += 1 return cached Step 2: perform external action with timeout + circuit breaker payload = {"value": input value, "idempotency key": lookup key} result = await self.call external payload Step 3: store result for future idempotent repeats self.idempotency.set lookup key, result, ttl=3600 Emit metrics in real code, push to Prometheus/Datadog self. emit metrics return result def emit metrics self : Placeholder for metric emission pass --- Example usage --- async def main : agent = DistributedAgent First call—may succeed or fail print "Attempt 1:", await agent.run workflow "hello-world" Second call with same input—idempotent cache hit print "Attempt 2 cached :", await agent.run workflow "hello-world" if name == " main ": asyncio.run main ponytail: in-memory store and breaker for demo only; replace with Redis/DynamoDB and persistent breaker state for production. AI agent bugs are almost never purely an LLM quality problem. They are distributed systems problems: network calls that fail, services that become unavailable, retries that create duplicates, and state that gets out of sync. By applying standard distributed-systems patterns—idempotency, timeouts, circuit breakers, and explicit orchestration—you turn unpredictable LLM behavior into deterministic, observable, and recoverable workflows. The LLM remains the source of non-determinism in content generation, but the surrounding infrastructure can absorb, retry, and compensate for that non-determinism without bringing the whole system down. Treat your agent like any other distributed service: instrument it, guard its calls, and design for failure from the start.