{"slug": "i-recorded-my-kubernetes-ai-agent-failing-on-purpose", "title": "I recorded my Kubernetes AI agent failing, on purpose", "summary": "A developer released kubeintellect, an open-source Kubernetes AI agent that gates every mutating operation behind a human approval chokepoint rather than a system-prompt guardrail, runs compiled-predicate detectors on every observation before invoking an LLM, and appends each decision to a hash-chained log for replay. The tool, distributed via pip under AGPL-3.0 and self-hosted with a bring-your-own LLM provider, answers cluster questions in plain English using kubectl, PromQL, and LogQL while quoting the evidence it read. The developer argues that a confident wrong diagnosis is the worst failure mode for incident-response tooling, and demonstrated the agent correctly identifying that a successful restart never fixed a missing DATABASE_URL environment variable.", "body_md": "Most demos of AI-for-infrastructure tools show a clean run. I think that is exactly backwards, and I want to explain why by walking through the thirty seconds of my own demo that a normal product video would have cut.\n\nA deployment is crash-looping. The agent has already diagnosed it. I ask it to restart the deployment.\n\nIt stops and asks for approval. I approve. The restart runs, and the command succeeds.\n\nThen I ask the follow-up question: did the restart change anything?\n\nNo. The pods fail with the same error. A restart was never going to supply a missing environment variable. The agent re-reads the pods, re-reads the events, and names the same root cause it found at the very beginning: `DATABASE_URL` is not set, so the container exits 1.\n\nThe restart was a reasonable thing to try and the wrong thing to fix it, and the system is the one saying so.\n\nThe worst failure for an incident-response tool is not being unhelpful. It is emitting something that *looks* like a real diagnosis and is not. At 3am a confident wrong answer costs more than no answer, because it sends a tired human down a wrong path with false authority.\n\nSo the design follows from that.\n\n**The approval gate is at the tool boundary, not in the prompt.** A guardrail written into a system prompt is a suggestion, and models can be argued out of suggestions. Instead every mutating operation passes through one chokepoint that returns exactly three answers: do it, ask a human, or refuse. The model does not get a vote, so there is nothing to talk around.\n\n**Detectors are compiled predicates, not model calls.** They are always on, they run on every observation, and they cost zero tokens. The LLM is only invoked once a detector actually fires. Watching your cluster is free; thinking about it is the expensive part, so it happens last.\n\n**Every decision is appended to a hash-chained log.** A run can be replayed afterwards instead of remembered. If you are going to let software touch production, \"what exactly did it do, and why\" has to be answerable after the fact.\n\n**Roles are real.** readonly, operator, admin and superadmin, enforced per key.\n\nkubectl for cluster state, Prometheus for metrics via PromQL, and Loki for logs via LogQL. It answers in plain English and quotes the evidence it actually read, so you can check it rather than trust it.\n\nEight minutes, every terminal scene a verbatim recording against a live cluster. Nothing typed by hand, nothing reconstructed. The approval-gate segment starts at 2:27.\n\n`pip install kubeintellect` - AGPL-3.0, self-hosted, bring your own LLM provider\nThe v1 architecture is written up in the Journal of Grid Computing ([https://doi.org/10.1007/s10723-026-09837-6](https://doi.org/10.1007/s10723-026-09837-6)), with a preprint at [https://arxiv.org/abs/2509.02449](https://arxiv.org/abs/2509.02449). The repo is several generations past that now.\n\nIf you run Kubernetes in production, the thing I most want to hear is which failure you would throw at it that I have not handled.", "url": "https://wpnews.pro/news/i-recorded-my-kubernetes-ai-agent-failing-on-purpose", "canonical_source": "https://dev.to/mskazemi/i-recorded-my-kubernetes-ai-agent-failing-on-purpose-1ib2", "published_at": "2026-09-13 00:22:41+00:00", "updated_at": "2026-09-13 01:26:46.977686+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure", "mlops", "developer-tools"], "entities": ["kubeintellect", "Kubernetes", "Prometheus", "Loki", "Journal of Grid Computing", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/i-recorded-my-kubernetes-ai-agent-failing-on-purpose", "markdown": "https://wpnews.pro/news/i-recorded-my-kubernetes-ai-agent-failing-on-purpose.md", "text": "https://wpnews.pro/news/i-recorded-my-kubernetes-ai-agent-failing-on-purpose.txt", "jsonld": "https://wpnews.pro/news/i-recorded-my-kubernetes-ai-agent-failing-on-purpose.jsonld"}}