{"slug": "stop-blaming-the-llm-why-your-ai-agent-s-bugs-are-actually-distributed-systems", "title": "Stop Blaming the LLM: Why Your AI Agent's Bugs Are Actually Distributed Systems Failures", "summary": "A developer argues that production AI agent failures commonly blamed on the LLM are actually distributed systems failures, since an agent spans an orchestrator, LLM API, tool executors, vector stores and state stores, each a separate failure domain with its own latency and error modes. The writeup maps symptoms such as repeated output, truncated responses, hallucinated facts and ignored instructions to causes like tool timeouts and retries, gateway cutoffs, stale vector results and truncated system prompts, and recommends treating agents with distributed systems operational practices.", "body_md": "*Originally published on [tamiz.pro](https://tamiz.pro/insights/ai-agent-bugs-distributed-systems-failures).*\n\nYou ship an AI agent. It works in staging. Production hits and suddenly it's \"hallucinating,\" \"ignoring instructions,\" or \"getting stuck in loops.\" The team blames the model. They swap providers, tweak prompts, add few-shot examples. The bugs persist.\n\nHere's the uncomfortable truth: **your AI agent is not a model. It's a distributed system.**\n\nEvery agent—whether a simple ReAct loop or a multi-agent swarm—decomposes into multiple independent processes communicating over unreliable networks: the LLM API, tool executors (code interpreters, browsers, API clients), vector databases, memory stores, orchestration logic, and often human-in-the-loop checkpoints. Each component fails independently. Each introduces latency variance. Each has its own failure modes that look exactly like \"model stupidity\" when observed from the outside.\n\nThis article maps the anatomy of an AI agent to distributed systems primitives, shows why classic LLM debugging fails, and gives you the operational toolkit to treat agents like the distributed systems they are.\n\nDraw the architecture of any production agent and you'll find these components:\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                    ORCHESTRATOR (Your Code)                  │\n│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐        │\n│  │ Planner │→ │ Executor │→ │ Critic  │→ │ Memory  │        │\n│  └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘        │\n└───────┼────────────┼────────────┼────────────┼──────────────┘\n        │            │            │            │\n        ▼            ▼            ▼            ▼\n┌───────────────┐ ┌─────────┐ ┌──────────┐ ┌────────────┐\n│  LLM Provider │ │  Tools  │ │  Vector  │ │  State     │\n│  (API Gateway)│ │ (Code,  │ │  Store   │ │  Store     │\n│               │ │  Browse,│ │ (Pinecone│ │ (Redis,    │\n│  - Rate limits│ │  API)   │ │  , etc.) │ │  Postgres) │\n│  - Latency    │ │         │ │          │ │            │\n│  - Errors     │ │ - Sand- │ │ - Index  │ │ - TTL      │\n│  - Quotas     │ │   boxes │ │   lag    │ │ - Locks    │\n└───────────────┘ └─────────┘ └──────────┘ └────────────┘\n```\n\nEach box is a **separate failure domain** with its own:\n\nThe orchestrator is the only component you fully control. Everything else is a **remote service you don't own**.\n\nA typical agent turn involves 3–10 LLM calls plus tool invocations. At 500ms–3s per LLM call and 100ms–5s per tool, a single user request spans **5–60 seconds of wall-clock time** across 10–50 network hops.\n\n```\n# ponytail: naive sequential execution, parallelize independent calls if latency matters\nasync def agent_turn(user_input: str, ctx: AgentContext) -> AgentResponse:\n    plan = await llm.plan(user_input, ctx.history)           # 1.2s\n    tool_results = await execute_tools(plan.tools)           # 2.8s (parallel)\n    critique = await llm.critique(plan, tool_results)        # 0.9s\n    memory.update(ctx.session_id, critique)                  # 0.1s\n    return await llm.synthesize(plan, tool_results, critique) # 1.1s\n# Total: ~6s sequential, ~3s with parallelization\n```\n\n**Distributed systems rule**: *latency adds up, and the tail latency dominates.* The 99th percentile of your agent's response time is determined by the slowest component's 99th percentile, compounded across hops.\n\n| Observed Symptom | Actual Distributed Systems Cause | Why It Looks Like an LLM Bug | \n|---|---|---|\n| Agent repeats itself | Tool call timed out, orchestrator retries with same prompt | \"Model is stuck in a loop\" | \n| Incomplete output | LLM stream cut off by gateway timeout | \"Model stopped reasoning\" | \n| Hallucinated facts | Vector store returned stale/empty results | \"Model doesn't know\" | \n| Ignores instructions | System prompt truncated by context window management | \"Model doesn't follow prompts\" | \n| Nonsensical tool args | Tool schema validation failed silently, fell back to string parsing | \"Model can't use tools\" | \n| Inconsistent behavior | Race condition between memory reads/writes across replicas | \"Model is unpredictable\" | \n| Silent degradation | Provider silently routes to smaller/faster model variant | \"Model got dumber\" | \n\nMost LLM APIs stream tokens. Your gateway (Cloudflare, AWS ALB, nginx) has a default **idle timeout of 60s**. A complex reasoning trace with tool calls can exceed this.\n\n```\n# What the model actually produced (streaming):\n\"I need to check the database... [tool: query_db] ... \n The result shows 42 users. Now I'll calculate... [tool: python] ...\n The answer is 42 * 1.5 = 63. So the final answer is—\"\n\n# What your orchestrator receives (gateway timeout at 60s):\n\"I need to check the database... [tool: query_db] ... \n The result shows 42 users. Now I'll calculate... [tool: python] ...\n The answer is 42 * 1.5 = 63. So the final answer is\"\n\n# Your code sees: incomplete response → treats as model error → retries\n```\n\nThe model *did* reason correctly. The infrastructure ate the tail.\n\nMajor providers route traffic across model variants (e.g., `gpt-4o` → `gpt-4o-mini` for cost optimization) without changing the model identifier. You observe quality drift. You blame \"prompt sensitivity.\" The reality: **you're calling a different model.**\n\n```\n# Detect this by logging model fingerprint headers\n# OpenAI returns: x-request-id, x-ratelimit-*, and sometimes model fingerprint\ncurl -v https://api.openai.com/v1/chat/completions ... 2>&1 | grep -i fingerprint\n# x-model-fingerprint: fp_abc123  ← log this every request\n```\n\nAgents retry. A lot. Network blips, 429s, 5xx errors, timeouts—all trigger retries. But **most agent operations are not idempotent**.\n\n``` python\n# DANGEROUS: Non-idempotent operations in a retry loop\nasync def run_agent(task: Task) -> Result:\n    for attempt in range(3):\n        try:\n            plan = await llm.plan(task)           # Idempotent (read-only)\n            result = await execute_plan(plan)     # NOT idempotent!\n            return result\n        except TransientError:\n            continue\n    raise MaxRetriesExceeded()\n\n# execute_plan might:\n# - Charge a credit card (twice)\n# - Send an email (twice)\n# - Mutate database state (twice)\n# - Create a Jira ticket (twice)\n```\n\nEvery tool invocation that mutates state **must** carry an idempotency key derived from the agent's execution context.\n\n``` python\nimport hashlib\nimport json\nfrom dataclasses import dataclass\nfrom typing import Any\n\n@dataclass\nclass ToolInvocation:\n    tool_name: str\n    args: dict[str, Any]\n    idempotency_key: str\n    attempt: int = 0\n\n    @classmethod\ndef create(cls, tool_name: str, args: dict, execution_id: str, step: int) -> \"ToolInvocation\":\n        # Deterministic key: execution_id + step + tool + args hash\n        content = f\"{execution_id}:{step}:{tool_name}:{json.dumps(args, sort_keys=True)}\"\n        key = hashlib.sha256(content.encode()).hexdigest()[:32]\n        return cls(tool_name=tool_name, args=args, idempotency_key=key)\n\n# Tool executor checks idempotency store before executing\nasync def execute_tool(invocation: ToolInvocation) -> ToolResult:\n    existing = await idempotency_store.get(invocation.idempotency_key)\n    if existing:\n        return ToolResult.from_cached(existing)  # Return previous result\n\n    result = await actual_tool_call(invocation.tool_name, invocation.args)\n    await idempotency_store.set(invocation.idempotency_key, result, ttl=86400)\n    return result\n```\n\nRetrying the *entire agent loop* because one tool failed is wasteful and dangerous. It re-does reasoning, re-calls other tools, and burns tokens.\n\n``` python\n# BETTER: Granular retry at tool level, not agent level\nasync def execute_plan(plan: Plan, ctx: ExecutionContext) -> PlanResult:\n    results = {}\n    for step in plan.steps:\n        if step.depends_on and not all(results[d].success for d in step.depends_on):\n            results[step.id] = StepResult(skipped=True, reason=\"dependency_failed\")\n            continue\n\n        # Retry ONLY this step, with its own budget\n        results[step.id] = await execute_step_with_retry(\n            step, \n            max_attempts=3,\n            timeout=step.timeout or 30,\n            idempotency_key=ctx.make_key(step.id)\n        )\n    return PlanResult(results)\n```\n\nAn agent turn is a **long-running transaction** across multiple services. When step 3 of 5 fails, steps 1–2 may have already mutated state. You need **compensating actions**—the Saga pattern.\n\n```\n@dataclass\nclass SagaStep:\n    name: str\n    execute: Callable[[], Awaitable[StepResult]]\n    compensate: Callable[[StepResult], Awaitable[None]]  # Undo action\n    idempotency_key: str\n\nclass AgentSaga:\n    def __init__(self, execution_id: str):\n        self.execution_id = execution_id\n        self.completed_steps: list[SagaStep] = []\n        self.step_results: dict[str, StepResult] = {}\n\n    async def run(self, steps: list[SagaStep]) -> SagaResult:\n        for step in steps:\n            try:\n                result = await step.execute()\n                self.step_results[step.name] = result\n                self.completed_steps.append(step)\n            except Exception as e:\n                # Compensate in reverse order\n                await self._compensate()\n                return SagaResult(success=False, failed_step=step.name, error=e)\n        return SagaResult(success=True, results=self.step_results)\n\n    async def _compensate(self):\n        for step in reversed(self.completed_steps):\n            try:\n                await step.compensate(self.step_results[step.name])\n            except Exception as comp_err:\n                # Log but continue compensating other steps\n                logger.error(f\"Compensation failed for {step.name}: {comp_err}\")\n                await alert_oncall(f\"Saga {self.execution_id} compensation failed at {step.name}\")\n```\n\n| Tool Action | Compensation | \n|---|---|\n| `send_email` | Send follow-up \"Please disregard previous email\" | \n| `create_jira_ticket` | Transition ticket to \"Cancelled\" | \n| `charge_customer` | Initiate refund via payment provider | \n| `write_database` | Delete inserted rows / revert update | \n| `call_external_api` | Call cancellation endpoint if available | \n| `create_file` | Delete file | \n\n**Critical**: Compensations themselves can fail. They must be idempotent and retried independently. Store saga state in durable storage (PostgreSQL, not memory) so crashes don't leave dangling partial state.\n\nMost agent observability stops at \"input → output.\" That's like debugging a microservice by only reading the API gateway logs. You need **distributed tracing across every hop**.\n\nEvery agent execution gets a `trace_id`. Every sub-operation gets a `span_id` with parent linkage.\n\n``` python\nfrom contextvars import ContextVar\nimport uuid\n\ncurrent_trace: ContextVar[str] = ContextVar(\"current_trace\", default=\"\")\ncurrent_span: ContextVar[str] = ContextVar(\"current_span\", default=\"\")\n\nclass TracedAgent:\n    async def run(self, input: str) -> AgentOutput:\n        trace_id = str(uuid.uuid4())\n        current_trace.set(trace_id)\n\n        with span(trace_id, \"agent.run\", {\"input\": input}) as root_span:\n            plan = await self._plan_with_trace(input, root_span)\n            tool_results = await self._execute_with_trace(plan, root_span)\n            output = await self._synthesize_with_trace(plan, tool_results, root_span)\n            root_span.set_attribute(\"output\", output)\n            return output\n\n    async def _plan_with_trace(self, input: str, parent_span) -> Plan:\n        with span(parent_span.trace_id, \"llm.plan\", {\"model\": self.model}, parent_span.span_id) as span:\n            span.set_attribute(\"prompt_tokens\", estimate_tokens(input))\n            result = await self.llm.plan(input)\n            span.set_attribute(\"completion_tokens\", estimate_tokens(result))\n            span.set_attribute(\"latency_ms\", span.duration_ms)\n            return result\n```\n\n| Signal | Where to Capture | Alert Threshold | \n|---|---|---|\n| **Latency** (p50, p95, p99) | Per LLM call, per tool, per agent turn | p99 > 30s | \n| **Error rate** | Per provider, per tool, per error type | > 1% 5xx | \n| **Token usage** | Input/output per call, per session | Budget exceeded | \n| **Retry rate** | Per operation type | > 10% retries | \n| **Partial failure rate** | Saga compensation triggered | > 0.1% | \n| **Context window pressure** | Tokens used / max tokens per call | > 80% | \n| **Tool success rate** | Per tool, per error category | < 99% | \n\n```\n{\n  \"timestamp\": \"2024-01-15T10:23:45.123Z\",\n  \"level\": \"INFO\",\n  \"trace_id\": \"abc-123\",\n  \"span_id\": \"span-456\",\n  \"component\": \"tool_executor\",\n  \"tool\": \"web_search\",\n  \"args\": {\"query\": \"quarterly revenue\", \"max_results\": 5},\n  \"idempotency_key\": \"abc123...\",\n  \"attempt\": 2,\n  \"latency_ms\": 2340,\n  \"status\": \"success\",\n  \"result_summary\": {\"results_count\": 5, \"tokens\": 1200}\n}\n```\n\n**Query this in production**: \"Show me all traces where `web_search` took >5s and the subsequent LLM call produced <100 tokens.\"\n\nYou cannot unit-test an LLM call like a pure function. But you **can** test the *system around it*.\n\nRecord production traces (inputs, outputs, timings, errors). Replay them against code changes.\n\n``` python\n# test_replay.py\nimport pytest\nfrom agent.tracing import TraceRecord\nfrom agent.orchestrator import AgentOrchestrator\n\n@pytest.mark.parametrize(\"trace\", load_production_traces(\"2024-01\", sample=1000))\nasync def test_agent_replay(trace: TraceRecord):\n    \"\"\"Replay a real production trace through current code.\"\"\"\n    orchestrator = AgentOrchestrator(\n        llm=FakeLLM(responses=trace.llm_responses),  # Deterministic replay\n        tools=FakeTools(responses=trace.tool_responses),\n        memory=FakeMemory(initial_state=trace.initial_memory)\n    )\n\n    result = await orchestrator.run(trace.input)\n\n    # Assert structural equivalence, not exact string match\n    assert result.tool_calls == trace.expected_tool_calls\n    assert result.final_answer.semantic_equiv(trace.expected_answer)\n    assert result.latency_ms < trace.original_latency_ms * 1.5  # No regression\n```\n\nInject failures at the *orchestration layer* to verify compensations, retries, and fallbacks.\n\n```\nclass ChaosAgentWrapper:\n    \"\"\"Wraps an agent and injects failures per configuration.\"\"\"\n    def __init__(self, agent, chaos_config: ChaosConfig):\n        self.agent = agent\n        self.chaos = chaos_config\n\n    async def run(self, input: str) -> AgentOutput:\n        if self.chaos.should_fail(\"llm_timeout\"):\n            raise TimeoutError(\"Injected LLM timeout\")\n        if self.chaos.should_fail(\"tool_error\"):\n            raise ToolError(\"Injected tool failure\")\n        if self.chaos.should_delay(\"network_latency\"):\n            await asyncio.sleep(self.chaos.latency_ms / 1000)\n\n        return await self.agent.run(input)\n\n# Run in CI: 1000 iterations with 5% random failure injection\n# Assert: no uncompensated mutations, no lost user data, graceful degradation\n```\n\nTreat your eval set as a **regression test suite**, not a one-time benchmark.\n\n```\n# evals/regression_suite.yaml\ncases:\n  - id: \"finance_calc_001\"\n    input: \"What's 15% of $2,340?\"\n    expected_tools: [\"calculator\"]\n    expected_answer_pattern: \"351\"\n    max_latency_ms: 5000\n    tags: [\"math\", \"single_tool\"]\n\n  - id: \"multi_hop_research_001\"\n    input: \"Compare Q3 revenue of Apple vs Microsoft\"\n    expected_tools: [\"web_search\", \"calculator\", \"web_search\"]\n    expected_answer_contains: [\"Apple\", \"Microsoft\", \"revenue\", \"billion\"]\n    max_latency_ms: 30000\n    tags: [\"multi_hop\", \"research\"]\nphp\nasync def run_regression_suite(agent, suite_path: str) -> EvaluationReport:\n    results = []\n    for case in load_suite(suite_path):\n        trace = await agent.run_with_trace(case.input)\n        results.append(EvalResult(\n            case_id=case.id,\n            passed=check_case(case, trace),\n            latency_ms=trace.total_latency_ms,\n            tokens_used=trace.total_tokens,\n            trace_id=trace.trace_id\n        ))\n    return EvaluationReport(results)\n```\n\nDistributed systems survive partial degradation through **explicit boundaries**. Your agent needs the same.\n\n``` python\nfrom enum import Enum\nimport time\n\nclass CircuitState(Enum):\n    CLOSED = \"closed\"      # Normal operation\n    OPEN = \"open\"          # Failing fast\n    HALF_OPEN = \"half_open\" # Testing recovery\n\nclass ProviderCircuitBreaker:\n    def __init__(\n        self,\n        failure_threshold: int = 5,\n        recovery_timeout: float = 30.0,\n        half_open_max_calls: int = 3\n    ):\n        self.state = CircuitState.CLOSED\n        self.failure_count = 0\n        self.last_failure_time = 0\n        self.half_open_calls = 0\n        self.failure_threshold = failure_threshold\n        self.recovery_timeout = recovery_timeout\n        self.half_open_max_calls = half_open_max_calls\n\n    async def call(self, func: Callable, *args, **kwargs):\n        if self.state == CircuitState.OPEN:\n            if time.time() - self.last_failure_time > self.recovery_timeout:\n                self.state = CircuitState.HALF_OPEN\n                self.half_open_calls = 0\n            else:\n                raise CircuitOpenError(\"Circuit breaker open\")\n\n        if self.state == CircuitState.HALF_OPEN:\n            if self.half_open_calls >= self.half_open_max_calls:\n                raise CircuitOpenError(\"Half-open limit reached\")\n            self.half_open_calls += 1\n\n        try:\n            result = await func(*args, **kwargs)\n            self._on_success()\n            return result\n        except Exception as e:\n            self._on_failure()\n            raise\n\n    def _on_success(self):\n        self.failure_count = 0\n        if self.state == CircuitState.HALF_OPEN:\n            self.state = CircuitState.CLOSED\n\n    def _on_failure(self):\n        self.failure_count += 1\n        self.last_failure_time = time.time()\n        if self.failure_count >= self.failure_threshold:\n            self.state = CircuitState.OPEN\n```\n\nWire it per-provider:\n\n```\nllm_clients = {\n    \"openai\": CircuitBreakerClient(OpenAIClient(), ProviderCircuitBreaker()),\n    \"anthropic\": CircuitBreakerClient(AnthropicClient(), ProviderCircuitBreaker()),\n    \"local\": CircuitBreakerClient(LocalLLMClient(), ProviderCircuitBreaker(failure_threshold=10)),\n}\n\nasync def call_llm_with_fallback(messages, preferred=\"openai\"):\n    for provider in [preferred, \"anthropic\", \"local\"]:\n        try:\n            return await llm_clients[provider].complete(messages)\n        except (CircuitOpenError, RateLimitError, TimeoutError):\n            continue\n    raise AllProvidersUnavailable()\n```\n\nFixed timeouts are wrong. Set timeouts at **p99 + buffer** per operation type.\n\n``` python\nclass AdaptiveTimeout:\n    def __init__(self, percentile: float = 0.99, buffer_multiplier: float = 1.5):\n        self.percentile = percentile\n        self.buffer = buffer_multiplier\n        self.latencies: list[float] = []  # Rolling window\n        self.max_samples = 1000\n\n    def record(self, latency_ms: float):\n        self.latencies.append(latency_ms)\n        if len(self.latencies) > self.max_samples:\n            self.latencies.pop(0)\n\n    def get_timeout(self) -> float:\n        if len(self.latencies) < 10:\n            return 30000  # Default 30s\n        sorted_lat = sorted(self.latencies)\n        idx = int(len(sorted_lat) * self.percentile)\n        return sorted_lat[idx] * self.buffer\n\n# Per-operation adaptive timeouts\ntimeouts = {\n    \"llm.plan\": AdaptiveTimeout(0.99, 1.5),\n    \"llm.critique\": AdaptiveTimeout(0.99, 1.5),\n    \"tool.web_search\": AdaptiveTimeout(0.95, 2.0),  # More variable\n    \"tool.code_exec\": AdaptiveTimeout(0.99, 1.2),\n    \"memory.read\": AdaptiveTimeout(0.999, 1.1),\n}\n\nasync def execute_with_timeout(op_name: str, coro):\n    timeout = timeouts[op_name].get_timeout()\n    try:\n        start = time.time()\n        result = await asyncio.wait_for(coro, timeout=timeout / 1000)\n        timeouts[op_name].record((time.time() - start) * 1000)\n        return result\n    except asyncio.TimeoutError:\n        timeouts[op_name].record(timeout)  # Record timeout as latency\n        raise\n```\n\nWhen things go wrong, degrade *functionality*, not *correctness*.\n\n```\nclass DegradedAgent:\n    \"\"\"Agent that progressively sheds capabilities under stress.\"\"\"\n\n    async def run(self, input: str, context: AgentContext) -> AgentOutput:\n        # Level 0: Full capability\n        if not self.degraded:\n            return await self._full_run(input, context)\n\n        # Level 1: No web search (slow, unreliable)\n        if self.degraded_level >= 1:\n            context.available_tools.discard(\"web_search\")\n\n        # Level 2: No code execution (sandbox overhead)\n        if self.degraded_level >= 2:\n            context.available_tools.discard(\"python\")\n\n        # Level 3: Single-turn only, no planning loop\n        if self.degraded_level >= 3:\n            return await self._single_turn_fallback(input, context)\n\n        # Level 4: Canned responses only\n        return await self._canned_fallback(input)\n\n    def update_health(self, metrics: HealthMetrics):\n        \"\"\"Called periodically by health monitor.\"\"\"\n        if metrics.error_rate > 0.2 or metrics.p99_latency > 60000:\n            self.degraded_level = min(self.degraded_level + 1, 4)\n        elif metrics.error_rate < 0.01 and metrics.p99_latency < 15000:\n            self.degraded_level = max(self.degraded_level - 1, 0)\n```\n\n\"Agent gets stuck repeating 'Let me search for that...' three times then gives generic answer. Model quality issue?\"\n\n**Trace analysis** revealed:\n\n```\nTrace: agent_7x9k2\n├─ llm.plan (1.2s) ✓\n├─ tool.web_search \"quarterly revenue\" (28.3s) → TIMEOUT (30s gateway)\n├─ llm.plan RETRY (1.1s) → \"Let me search for that...\"\n├─ tool.web_search \"quarterly revenue\" (29.1s) → TIMEOUT\n├─ llm.plan RETRY (1.3s) → \"Let me search for that...\"\n├─ tool.web_search \"quarterly revenue\" (31.2s) → TIMEOUT\n├─ llm.synthesize (0.8s) → Generic answer (no tool results)\n```\n\n**Root cause**: The search API's p99 latency was 28s. The gateway timeout was 30s. The orchestrator retried the *entire plan* on any tool failure. Three retries = 90s of user-facing latency.\n\n```\n# 1. Increase gateway timeout to 60s (match search p99 + buffer)\n# 2. Make web_search idempotent (cache results for 5min)\n# 3. Retry ONLY the failed tool, not the whole plan\n\n# Before (broken):\nasync def run_agent():\n    for attempt in range(3):\n        try:\n            return await full_agent_loop()\n        except ToolTimeout:\n            continue\n\n# After (fixed):\nasync def run_agent():\n    plan = await llm.plan()  # No retry needed, idempotent\n    results = {}\n    for step in plan.steps:\n        results[step.id] = await execute_with_retry(\n            step, max_attempts=2, timeout=60  # Per-step config\n        )\n    return await llm.synthesize(plan, results)\n```\n\n**Result**: Loop eliminated. Latency p99 dropped from 90s → 35s. User complaints dropped to zero.\n\n**A:** If it's truly one call with no tools, no memory, no retries—then yes, it's just an LLM call. But the moment you add: a second LLM call, a tool, a retry loop, or session memory—you have a distributed system. The complexity threshold is lower than most teams realize.\n\n**A:** Start with **one** trace that proves the point. Find a \"model bug\" ticket, add distributed tracing, show the actual failure was a timeout/cascade/partial failure. The evidence speaks louder than architecture diagrams.\n\n**A:** Structured JSON logs with `trace_id`/` span_id` + a log aggregator (Datadog, Loki, CloudWatch) + basic dashboards for the 7 golden signals in section 5.2. You don't need Jaeger/Tempo to start—correlation IDs in logs get you 80% of the value.\n\n**A:** No. Compensation is only for **mutating operations** (writes, charges, sends, creates). Search, calculation, read-only API calls need no compensation—but they *do* need idempotency keys if you retry them, to avoid duplicate costs/latency.\n\n*This article is adapted from patterns used in production agent systems at scale. The code snippets are simplified for clarity—production versions include metrics emission, structured error types, and integration with your observability stack.*\n\nThe second pillar is **idempotency**. LLMs are inherently non-deterministic; the same prompt can yield different completions across calls. When an agent retries a failed action, duplicate side-effects become a critical problem. The standard pattern is to require an idempotency key from the caller, which the service stores and returns unchanged on repeat executions.\n\nConsider a checkout flow where an agent initiates a payment, the LLM times out, and retries. Without idempotency, the customer is charged twice. With it, the payment service recognizes the key and returns the original transaction status.\n\n```\n# Agent side: generate and attach a key\npayment_id = \"pay_12345\"\nidempotency_key = f\"checkout-{payment_id}-{int(time.time())}\"\nresult = await agent.call_tool(\n    \"payments_process\",\n    {\"amount\": 999, \"currency\": \"USD\", \"idempotency_key\": idempotency_key},\n)\npython\n# Service side: check cache before executing\nasync def process_payment(data: dict):\n    key = data.pop(\"idempotency_key\", None)\n    cached = idempotency_store.get(key)\n    if cached:\n        return cached  # return original result unchanged\n    # ... actual payment processing ...\n    idempotency_store.set(key, result, ttl=86400)\n    return result\n```\n\n*ponytail: in-process cache only; production uses distributed ID generators (Redis, DynamoDB) with global TTL coordination.*\n\nThe third pillar is **liveness enforcement**. Distributed systems fail—network partitions, GC pauses, thread starvation. An agent without timeouts will hang indefinitely, consuming resources and cascading failures downstream. Every external call must have a hard timeout, and the agent must have a circuit-breaker pattern to stop hammering unhealthy services.\n\n```\nasync with asyncio.timeout(5):  # 5-second hard limit\n    result = await agent.call_tool(\"some_remote_service\", payload)\n# Circuit breaker state (persisted across restarts)\nclass CircuitBreaker:\n    def __init__(self, failure_threshold=5, recovery_timeout=30):\n        self.failure_threshold = failure_threshold\n        self.recovery_timeout = recovery_timeout\n        self.failures = 0\n        self.last_failure = None\n        self.state = \"closed\"  # closed/open/half-open\n\n    async def call(self, fn, *args, **kwargs):\n        if self.state == \"open\":\n            if time.time() - self.last_failure > self.recovery_timeout:\n                self.state = \"half-open\"\n            else:\n                raise CircuitOpenError(\"service temporarily unavailable\")\n\n        try:\n            result = await fn(*args, **kwargs)\n            self._on_success()\n            return result\n        except Exception as e:\n            self._on_failure()\n            raise\n\n    def _on_success(self):\n        self.failures = 0\n        self.state = \"closed\"\n\n    def _on_failure(self):\n        self.failures += 1\n        self.last_failure = time.time()\n        if self.failures >= self.failure_threshold:\n            self.state = \"open\"\n```\n\n*ponytail: simple in-memory breaker; production persists state and uses exponential backoff with jitter.*\n\nThe fourth pillar is **system architecture choice**. In a choreographed system, agents react to events without central coordination—message queues, topics, eventual consistency. In an orchestrated system, a conductor or state machine drives the flow. For AI agents, pure choreography often leads to undetected stuck states; a lightweight orchestrator (Temporal, LangGraph, or even a DB-driven state table) makes liveness and idempotency guarantees explicit.\n\nConsider the difference: in a choreographed flow, an agent receives a \"payment_completed\" event but the UI never renders because the agent crashed before updating state. An orchestrated flow tracks each step's completion explicitly and can replay or compensate on failure.\n\nAll of this hinges on observability. You cannot fix what you cannot measure. Distributed systems already require tracing, metrics, and logs; AI agents add non-determinism on top. Structure your agent outputs so they integrate with your existing stack.\n\n**Metrics to emit per call:**\n\n`agent_call_duration_seconds`` agent_call_success_total` (increment on success, decrement on failure)`agent_call_error_type` (labels: timeout, validation, external_service, llm_token_limit)`agent_retry_count` (histogram or counter)\n**Structured logs** should include:\n\n`request_id` (correlates across all service calls)`llm_model` and `temperature`\n`tokens_used` and `cost_estimate`\n`external_service_latency`` idempotency_key` (for duplicate detection debugging)\n**Traces** should span the entire agent workflow: user prompt → tool selection → tool execution → response generation. If a tool fails, the trace shows exactly which step and what input caused it.\n\nBelow is a minimal, runnable skeleton that demonstrates the patterns discussed. It is not a production-grade framework but illustrates the core ideas: timeout, idempotency key, circuit breaker, and structured metrics.\n\n``` python\nimport asyncio\nimport time\nimport random\nfrom typing import Optional, Dict, Any\n\n# --- Mock infrastructure ---\n\nclass InMemoryIdempotencyStore:\n    def __init__(self):\n        self._store: Dict[str, Any] = {}\n\n    def get(self, key: str) -> Optional[Any]:\n        return self._store.get(key)\n\n    def set(self, key: str, value: Any, ttl: int = 3600):\n        self._store[key] = value\n\nclass SimpleCircuitBreaker:\n    def __init__(self, failure_threshold: int = 5, recovery_timeout: float = 30.0):\n        self.failure_threshold = failure_threshold\n        self.recovery_timeout = recovery_timeout\n        self.failures = 0\n        self.last_failure: Optional[float] = None\n        self.state = \"closed\"  # closed / open / half-open\n\n    async def call(self, fn, *args, **kwargs):\n        if self.state == \"open\":\n            if time.time() - self.last_failure > self.recovery_timeout:\n                self.state = \"half-open\"\n            else:\n                raise RuntimeError(\"CircuitOpen: service unavailable\")\n\n        try:\n            result = await fn(*args, **kwargs)\n            self._on_success()\n            return result\n        except Exception as e:\n            self._on_failure()\n            raise\n\n    def _on_success(self):\n        self.failures = 0\n        self.state = \"closed\"\n\n    def _on_failure(self):\n        self.failures += 1\n        self.last_failure = time.time()\n        if self.failures >= self.failure_threshold:\n            self.state = \"open\"\n\n# --- Agent core ---\n\nclass DistributedAgent:\n    def __init__(self):\n        self.idempotency = InMemoryIdempotencyStore()\n        self.circuit = SimpleCircuitBreaker()\n        self.metrics = {\"calls\": 0, \"errors\": 0, \"retries\": 0}\n\n    async def call_external(self, payload: dict) -> dict:\n        \"\"\"Wrap external calls with timeout, idempotency, and circuit breaking.\"\"\"\n        # Attach idempotency key if present\n        key = payload.pop(\"idempotency_key\", None)\n        if key:\n            cached = self.idempotency.get(key)\n            if cached:\n                self.metrics[\"calls\"] += 1\n                return cached\n\n        # Enforce timeout\n        try:\n            async with asyncio.timeout(5):\n                result = await self.circuit.call(self._do_call, payload)\n        except asyncio.TimeoutError:\n            self.metrics[\"errors\"] += 1\n            raise RuntimeError(\"Call timed out after 5s\") from None\n\n        # Cache result for idempotency\n        if key:\n            self.idempotency.set(key, result, ttl=3600)\n\n        self.metrics[\"calls\"] += 1\n        return result\n\n    async def _do_call(self, payload: dict) -> dict:\n        \"\"\"Actual external call—here simulated with random latency.\"\"\"\n        # Simulate flaky external service\n        await asyncio.sleep(random.uniform(0.1, 2.0))\n        if random.random() < 0.3:\n            raise RuntimeError(\"Simulated external failure\")\n        return {\"status\": \"ok\", \"data\": payload.get(\"value\")}\n\n    async def run_workflow(self, input_value: str) -> dict:\n        \"\"\"Example workflow that uses all four pillars.\"\"\"\n        # Step 1: idempotent lookup\n        lookup_key = f\"workflow-{input_value}\"\n        cached = self.idempotency.get(lookup_key)\n        if cached:\n            self.metrics[\"retries\"] += 1\n            return cached\n\n        # Step 2: perform external action with timeout + circuit breaker\n        payload = {\"value\": input_value, \"idempotency_key\": lookup_key}\n        result = await self.call_external(payload)\n\n        # Step 3: store result for future idempotent repeats\n        self.idempotency.set(lookup_key, result, ttl=3600)\n\n        # Emit metrics (in real code, push to Prometheus/Datadog)\n        self._emit_metrics()\n\n        return result\n\n    def _emit_metrics(self):\n        # Placeholder for metric emission\n        pass\n\n# --- Example usage ---\n\nasync def main():\n    agent = DistributedAgent()\n\n    # First call—may succeed or fail\n    print(\"Attempt 1:\", await agent.run_workflow(\"hello-world\"))\n\n    # Second call with same input—idempotent cache hit\n    print(\"Attempt 2 (cached):\", await agent.run_workflow(\"hello-world\"))\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n*ponytail: in-memory store and breaker for demo only; replace with Redis/DynamoDB and persistent breaker state for production.*\n\nAI agent bugs are almost never purely an LLM quality problem. They are distributed systems problems: network calls that fail, services that become unavailable, retries that create duplicates, and state that gets out of sync. By applying standard distributed-systems patterns—idempotency, timeouts, circuit breakers, and explicit orchestration—you turn unpredictable LLM behavior into deterministic, observable, and recoverable workflows.\n\nThe LLM remains the source of non-determinism in content generation, but the surrounding infrastructure can absorb, retry, and compensate for that non-determinism without bringing the whole system down. Treat your agent like any other distributed service: instrument it, guard its calls, and design for failure from the start.", "url": "https://wpnews.pro/news/stop-blaming-the-llm-why-your-ai-agent-s-bugs-are-actually-distributed-systems", "canonical_source": "https://dev.to/tamizuddin/stop-blaming-the-llm-why-your-ai-agents-bugs-are-actually-distributed-systems-failures-15j9", "published_at": "2026-10-20 12:05:40+00:00", "updated_at": "2026-10-20 12:18:49.823661+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "mlops", "large-language-models", "ai-tools"], "entities": ["Pinecone", "Redis", "Postgres"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/stop-blaming-the-llm-why-your-ai-agent-s-bugs-are-actually-distributed-systems", "markdown": "https://wpnews.pro/news/stop-blaming-the-llm-why-your-ai-agent-s-bugs-are-actually-distributed-systems.md", "text": "https://wpnews.pro/news/stop-blaming-the-llm-why-your-ai-agent-s-bugs-are-actually-distributed-systems.txt", "jsonld": "https://wpnews.pro/news/stop-blaming-the-llm-why-your-ai-agent-s-bugs-are-actually-distributed-systems.jsonld"}}