{"slug": "ai-agents-are-being-given-way-too-much-power-in-production", "title": "AI agents are being given way too much power in production", "summary": "A developer proposes NEXUS (Mesh Intelligence Hub), an architecture combining an LLM agent with an Istio service mesh on Amazon EKS to enforce read-only access and explicit deny policies, preventing AI agents from modifying production workloads. In a Chaos Mesh test, the agent detected a 56.5% error rate and correctly isolated the fault, while its JSON output stated it could not execute kubectl commands. The approach aims to reduce the blast radius of compromised or hallucinating agents to zero.", "body_md": "# AI agents are being given way too much power in production\n\nI've been looking into how to solve this using a Zero Trust approach, specifically by combining an LLM agent with an Istio service mesh. I call this architecture NEXUS (Mesh Intelligence Hub). The core philosophy is simple: the agent should be able to observe everything and propose solutions, but it should be physically impossible for it to actually touch or modify the workloads it monitors.\n\n## The Architecture: Security via Service Mesh\n\nInstead of relying on the agent \"promising\" to be well-behaved, we use the platform to enforce boundaries. In my setup on Amazon EKS, the agent runs in its own isolated Kubernetes namespace with a dedicated ServiceAccount and a SPIFFE cryptographic identity:\n\n`spiffe://cluster.local/ns/ai-agent/sa/ai-agent`\n\nBy using Istio in STRICT mTLS mode, we can move away from traditional IP-based security to identity-based security. Here is how the enforcement works:\n\n**Read-Only Access:** An`AuthorizationPolicy`\n\nis configured to allow the agent to pull data from Prometheus, Jaeger, and Kiali.**Explicit Deny:** A separate policy explicitly blocks the agent from reaching any actual application workloads (like payment APIs or frontend services).\n\nEven if the LLM hallucinates a command to delete a namespace or if the agent itself is compromised, the service mesh acts as a hard physical barrier. The blast radius is effectively zero.\n\n## The AI Workflow: From Telemetry to Structured Diagnosis\n\nThe agent doesn't just \"chat\" with the system. It follows a rigorous, automated loop. Every 30 seconds, it polls Prometheus for two specific metrics per service:\n\n1. Error rates (derived from `istio_requests_total`\n\n)\n\n2. p99 latency (from histogram buckets)\n\nWhen a threshold is crossed, the agent pulls a full telemetry snapshot and sends it to [Claude](/en/tags/claude/) 3.5 Sonnet. The key here is the prompt engineering. To make this useful for a real-world SRE dashboard, the agent must not return conversational text. It must return structured JSON.\n\nHere is the prompt template I used to ensure the output remains machine-readable and strictly bounded:\n\n```\nYou are an expert Site Reliability Engineer. Analyze the provided telemetry snapshot and identify the root cause of the anomaly.\n\nYour output must be a valid JSON object ONLY. Do not include markdown formatting, preamble, or conversational text.\n\nThe JSON schema must follow this structure:\n{\n  \"severity\": \"critical|warning|info\",\n  \"summary\": \"A concise one-sentence description of the issue\",\n  \"root_cause\": \"Detailed technical explanation of the failure\",\n  \"remediation_steps\": [\n    \"Step 1...\",\n    \"Step 2...\",\n    \"Step 3...\"\n  ],\n  \"scope_boundary\": \"Explicitly state what actions you are physically unable to perform\",\n  \"required_approval_role\": \"The specific IAM or RBAC role needed to execute these steps\"\n}\n\nTelemetry Data:\n{{telemetry_snapshot}}\n```\n\n## Real-World Test: Chaos Engineering\n\nTo see if this actually worked, I used Chaos Mesh to inject a `NetworkChaos`\n\nfault into a backend service.\n\nThe results were impressive. Within one polling cycle, the agent detected a 56.5% error rate. Because it had access to the mesh telemetry, it didn't just say \"the network is down.\" It correctly isolated the fault to the specific backend service, ruled out a mesh-wide issue, and identified that the `NaN`\n\np99 latency values were actually a sign of a broken metrics pipeline rather than a slow application.\n\nMost importantly, the `scope_boundary`\n\nfield in the JSON output correctly stated: \"I cannot execute kubectl commands or modify cluster state.\" This is the kind of predictable, constrained AI behavior we need for production deployment.\n\n[Next Hands lets your LLM agent control your actual Windows desktop →](/en/threads/7428/)\n\n[these AI tool field notes](https://tanyan888.com/), with plenty of directly applicable cases.", "url": "https://wpnews.pro/news/ai-agents-are-being-given-way-too-much-power-in-production", "canonical_source": "https://promptcube3.com/en/threads/7528/", "published_at": "2026-08-24 16:33:22+00:00", "updated_at": "2026-08-24 16:44:25.534575+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-infrastructure"], "entities": ["NEXUS", "Istio", "Amazon EKS", "Prometheus", "Jaeger", "Kiali", "Chaos Mesh", "Claude 3.5 Sonnet"], "alternates": {"html": "https://wpnews.pro/news/ai-agents-are-being-given-way-too-much-power-in-production", "markdown": "https://wpnews.pro/news/ai-agents-are-being-given-way-too-much-power-in-production.md", "text": "https://wpnews.pro/news/ai-agents-are-being-given-way-too-much-power-in-production.txt", "jsonld": "https://wpnews.pro/news/ai-agents-are-being-given-way-too-much-power-in-production.jsonld"}}