{"slug": "why-agent-infrastructure-is-its-own-discipline", "title": "Why Agent Infrastructure Is Its Own Discipline", "summary": "A developer argues that running AI agents in production requires its own infrastructure discipline distinct from standard microservice practices, because agent requests break four core assumptions: bounded latency, constant per-request compute, statelessness, and clear HTTP-level success signals. The writeup details how naive Kubernetes liveness probes kill healthy pods mid-reasoning and how CPU-based autoscaling misjudges load that actually tracks reasoning depth and tool fan-out, recommending that liveness checks confirm only process aliveness while readiness probes reflect capacity.", "body_md": "Ask a platform team how they're going to run their first production agent and you'll get a confident answer within ten seconds: containerize it, put it behind an ingress, wire up a Horizontal Pod Autoscaler, done. It's the same playbook that's shipped every stateless service for the last decade, and there's no obvious reason an agent should be different. It accepts a request. It returns a response. It's just a container.\n\nThat answer is wrong — and it's wrong in a way that doesn't show up in a demo. It shows up three weeks into production, when a pod gets killed mid-reasoning because a liveness probe decided a 40-second tool call was a hang, or when the autoscaler adds five replicas because CPU spiked on a single request calling six tools in sequence, or when two \"sessions\" turn out to share a K8s Service without anyone having reasoned about what that means for isolation.\n\nA typical microservice request has a shape platform engineers have spent fifteen years optimizing around: bounded latency, a single unit of compute per request, statelessness between requests, and a clear success/failure signal at the HTTP layer. An agent request breaks all four at once.\n\n``` php\nStandard microservice request:\n  client -> service -> [DB/cache lookup] -> response\n  Duration: milliseconds to low seconds\n  Compute: roughly constant per request\n  State: none carried between requests\n  Outcome signal: HTTP status code\n\nAgent request:\n  client -> agent -> [reason] -> tool call 1 -> [reason] -> tool call 2\n        -> [reason] -> tool call N -> [reason] -> response\n  Duration: seconds to minutes, highly variable\n  Compute: proportional to reasoning depth and tool fan-out\n  State: conversation/task context carried across the entire chain\n  Outcome signal: HTTP 200 with a semantically wrong answer is common\n```\n\nThat last line is the one platform teams underestimate most. An agent that loops on a tool, calls the wrong one, or returns a plausible-but-incorrect result will still return a healthy status code. The infrastructure layer has no way to distinguish a correct run from a confidently wrong one.\n\n| Microservice assumption | Agent reality | Practical implication | \n|---|---|---|\n| Liveness = \"process is alive and responsive\" | A live agent can be legitimately unresponsive for 30–90 seconds mid-tool-call | Naive liveness probes kill healthy pods mid-reasoning | \n| CPU/memory tracks load | Load tracks reasoning depth and tool fan-out, not CPU | HPA on CPU/memory over- or under-scales unpredictably | \n| Requests are independent | A single task often spans multiple round trips carrying shared context | Session/state handling needs an explicit design, not an assumption | \n| One replica serves any request | Tool credentials and context may be tenant-specific | Routing and isolation boundaries need to be architectural, not incidental | \n| Timeout = failure | Timeout at 30s might just mean the agent is still reasoning | Timeout budgets have to be set per tool-call chain, not per request | \n\nWith `periodSeconds: 10` and `failureThreshold: 3`, a standard probe kills a pod if `/health` doesn't respond within roughly 30 seconds — which is an entirely normal duration for a reasoning step that includes a tool call to a slow downstream API.\n\nThe fix isn't a bigger `failureThreshold`. It's separating \"the process is alive\" from \"the process is making forward progress\":\n\n```\nlivenessProbe:\n  httpGet:\n    path: /health          # answers: is the process itself alive?\n    port: 8080\n  periodSeconds: 15\n  timeoutSeconds: 5\n  failureThreshold: 3      # ~45s of true unresponsiveness before restart\n\nreadinessProbe:\n  httpGet:\n    path: /ready            # answers: can this pod accept new work right now?\n    port: 8080\n  periodSeconds: 5\n  failureThreshold: 2\n```\n\n`/health` should do nothing more than confirm the process's event loop is running — never block on the status of an in-progress tool call. `/ready` reflects capacity: a pod mid-reasoning can report itself not-ready for new work without being treated as dead. For a request/response service these two questions have the same answer. For an agent, they routinely don't.\n\nA large part of the awkwardness in running MCP-based agents on Kubernetes came from the original protocol design, which required persistent, pinned sessions between a client and a specific server instance — the opposite of what horizontally scaled infrastructure wants. That constraint forced teams into sticky routing and shared session stores just to keep a conversation coherent across requests.\n\nThe July 2026 MCP specification revision removed the protocol-level session entirely. Any request can now land on any server instance, and applications that need to carry state across calls do it the way HTTP APIs always have — by minting an explicit handle passed back as an ordinary argument, rather than relying on the transport to remember. That single change removes an entire category of infrastructure workaround (sticky routing, pinned sessions, shared session stores) that used to be treated as unavoidable.\n\nThe second shift is A2A (Agent-to-Agent protocol), which reached v1.0 in early 2026. Where MCP governs how an agent talks to tools and data sources, A2A governs how agents talk to each other — and that distinction matters for infrastructure design. A multi-agent system where agents call each other over A2A has different routing, identity, and isolation requirements than one where a single orchestrator calls tools over MCP. Both patterns are in production today.\n\nA liveness probe that kills healthy pods doesn't fail loudly — it shows up as an elevated error rate attributed to \"model flakiness\" or \"the tool API being unreliable,\" and teams spend weeks tuning retry logic against a problem that's actually a probe misconfiguration one layer down. An autoscaler tuned on the wrong signal doesn't fail either — it just runs 30% more replicas than the workload needs, indefinitely, because nobody has a reason to suspect the scaling metric itself.\n\nGetting the infrastructure layer right doesn't guarantee correct agent behavior, but getting it wrong guarantees you can't tell the difference between an agent that's actually failing and one that's simply being run on infrastructure that wasn't built for it.\n\nThe summary covers the core argument. The full article goes deeper on:\n\n**👉 [Why Agent Infrastructure Is Its Own Discipline — Full Article](https://aloknecessary.in/blogs/why-agent-infrastructure-is-its-own-discipline/?utm_source=devto&utm_medium=referral&utm_campaign=blog_syndication&utm_content=why-agent-infrastructure-is-its-own-discipline)**", "url": "https://wpnews.pro/news/why-agent-infrastructure-is-its-own-discipline", "canonical_source": "https://dev.to/aloknecessary/why-agent-infrastructure-is-its-own-discipline-4778", "published_at": "2026-09-18 05:39:20+00:00", "updated_at": "2026-09-18 05:52:48.577950+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "mlops", "developer-tools"], "entities": ["Kubernetes"], "alternates": {"html": "https://wpnews.pro/news/why-agent-infrastructure-is-its-own-discipline", "markdown": "https://wpnews.pro/news/why-agent-infrastructure-is-its-own-discipline.md", "text": "https://wpnews.pro/news/why-agent-infrastructure-is-its-own-discipline.txt", "jsonld": "https://wpnews.pro/news/why-agent-infrastructure-is-its-own-discipline.jsonld"}}