{"slug": "kubemend-a-k8s-agent-that-can-only-open-prs-and-never-trusts-its-own-fixed", "title": "Kubemend – a K8s agent that can only open PRs, and never trusts its own \"fixed\"", "summary": "Kubemend, a GitOps-native Kubernetes remediation agent, can only open pull requests and never trusts its own fixes, requiring independent verification before a human merges. The agent diagnoses incidents from Prometheus metrics and Loki logs, proposes fixes by editing Helm values files, and validates them through helm render, Kyverno policy checks, live diffs, scope checks, and quota headroom checks. It supports multiple model providers including Anthropic, OpenAI-compatible endpoints, and AWS Bedrock, and is not yet production-ready due to missing multi-repo GitOps and sandboxed tool execution.", "body_md": "**A GitOps-native Kubernetes remediation agent that can only open pull requests.**\n\nIt diagnoses incidents from Prometheus metrics and Loki logs, proposes a fix, and verifies that fix itself — helm render → Kyverno policy check → live diff → scope check → live quota headroom — before it ever asks a human to approve anything. It never runs `kubectl apply`\n\n. It has no cluster credentials that can write.\n\nNot production-ready: no multi-repo GitOps, no sandboxed tool execution yet. See [ docs/threat-model.md](/m-stepkowski/kubemend/blob/main/docs/threat-model.md) for what's in and out of scope.\n\nMost \"AI SRE agent\" demos are impressive and unverifiable — a model claims it fixed something, and you take its word for it. kubemend is built the other way around: **the model's claim of success is never trusted.** Every run terminates only after an independent validation pipeline says the proposed fix renders cleanly, satisfies policy, produces a real and scoped diff, and touches nothing outside the declared incident. The agent's only actuator is a Git branch and a draft PR — a human still merges.\n\nIt's also a from-scratch agent harness, not a wrapper around LangChain/CrewAI/AutoGen. The loop, context management, tool registry, and verification gate are hand-written and documented, because understanding those trade-offs — not gluing a framework together — is the point of the project.\n\n```\ntask ──▶ Loop ──▶ tool calls ──▶ Prometheus / Loki / K8s (read-only)\n          │\n          └── model claims \"done\" ──▶ independent verification gate\n                                        helm template → kyverno apply\n                                        → argocd/kubectl diff → scope check\n                                        → live quota headroom\n                                        │\n                                pass ──▶ draft PR against the GitOps repo\n                                fail ──▶ structured failure fed back into the loop\n```\n\n**Observability:** PromQL against Prometheus/Mimir, LogQL against Loki. Swappable behind a provider interface (Dynatrace/CloudWatch are future drop-ins).**Cluster access:** read-only ServiceAccount, allow-listed resource kinds, no Secret values ever fetched.**Remediation:** the agent edits Helm`values*.yaml`\n\nonly — never templates directly — so diffs stay small and reviewable.**Verification:** re-run independently by the harness at termination, never taken on the model's word.**Everything is evaluated:** a hermetic`kind`\n\n-based fault-injection lab reproduces real incidents (bad image tags, OOMKills, missing config keys, broken probes...) with property-based checkers, run N times per scenario to produce pass-rate / cost / iteration tables — not cherry-picked demos. Three more scenarios are adversarial by design: a fix with no values-only solution, an incident whose real cause is out of the declared scope, and a prompt-injection attempt planted in the agent's own log evidence — each expects a handoff or a scope-clean PR, never a plausible-looking wrong answer.\n\nFull design, invariants, and every numeric default with its rationale: .\n\n`ARCHITECTURE.md`\n\n`main`\n\nand `cheap`\n\nare each configured independently, so mixing providers\nacross tiers (e.g. Claude on Bedrock for `main`\n\n, DeepSeek for `cheap`\n\n) is a\nnormal configuration, not a special case:\n\n| Provider | `model.*.provider` |\nCovers | Credentials |\n|---|---|---|---|\n| Anthropic | `anthropic` (default) |\nClaude, direct API | `ANTHROPIC_API_KEY` , or an `ant auth login` profile |\n| OpenAI-compatible | `openai` + `base_url` |\nOpenAI, DeepSeek, vLLM, Ollama, anything speaking `/v1/chat/completions` |\n`OPENAI_API_KEY` (local/self-hosted endpoints without auth fall back to a placeholder automatically) |\n| AWS Bedrock | `bedrock` |\nClaude models only, via Bedrock (Converse API / non-Claude models not yet supported) | the standard AWS credential chain (env, profile, or IMDS) |\n\n```\nmodel:\n  main:\n    provider: bedrock\n    name: us.anthropic.claude-sonnet-5-v1:0\n    aws_region: us-east-1\n  cheap:\n    provider: openai\n    name: deepseek-v4-flash\n    base_url: https://api.deepseek.com\n```\n\nSee `kubemend.yaml`\n\n's own comments for more examples, and\n[ config/pricing.yaml](/m-stepkowski/kubemend/blob/main/config/pricing.yaml) for cost-guardrail pricing —\nnon-Anthropic entries there are placeholders sourced from public pricing\npages, not verified against an invoice; check before trusting them for a\ncommitted baseline.\n\nEvery tagged release publishes to both PyPI and ghcr.io:\n\n```\npip install kubemend\ndocker pull ghcr.io/m-stepkowski/kubemend:latest\ndocker run --rm ghcr.io/m-stepkowski/kubemend:latest --help\n```\n\nEither way you'll need model credentials (`ANTHROPIC_API_KEY`\n\nby default —\nsee \"Model providers\" above) and a `kubemend.yaml`\n\npointing at your\ncluster's Prometheus/Loki, kubeconfig, and GitOps repo — see the committed\n[ kubemend.yaml](/m-stepkowski/kubemend/blob/main/kubemend.yaml)'s own comments for every field. To run\nin-cluster instead of from a laptop, see \"Deploy in-cluster\" below.\n\nRequires [Docker](https://docs.docker.com/get-docker/) (or Rancher Desktop —\nanything `kind`\n\ncan use), [ uv](https://docs.astral.sh/uv/), and\n\n[, plus an](https://taskfile.dev/)\n\n`go-task`\n\n`ANTHROPIC_API_KEY`\n\n.The fastest way to see it work end to end — bring up the lab, inject a real fault, run the agent against it, and print the resulting proposal — is:\n\n```\ngit clone https://github.com/m-stepkowski/kubemend.git && cd kubemend\nuv sync\n\nexport ANTHROPIC_API_KEY=...\ntask lab:up      # kind cluster: gitea, Argo CD, kube-prometheus-stack, Loki, Kyverno\ntask demo        # inject a fault, run kubemend, show the resulting proposal (~90s)\n```\n\n`task demo`\n\nruns on the cheap model by default; pass `-- --model main`\n\nto use\nthe model the headline sweep below was run on:\n\n```\ntask demo -- --model main\n```\n\nTo drive it by hand instead of via the demo script:\n\n```\ntask lab:forward   # port-forward Prometheus/Loki/gitea/Argo locally, blocks — run in another terminal\n\nkubemend run --task \"shop-api pods in namespace shop are crash-looping since 10 minutes ago\" \\\n              --namespace shop --app shop-api\n```\n\nThis writes a branch (and, with `gitops.backend: gitea`\n\n, a real draft PR in\nthe lab's gitea instance) plus a full JSONL trace under `traces/`\n\n. See\n[ docs/threat-model.md](/m-stepkowski/kubemend/blob/main/docs/threat-model.md) for the trust boundaries and\nwhat's still out of scope (single repo, values-only edits, no persistent\nmemory across runs).\n\nReproducible pass-rate benchmarks, not anecdotes — every scenario is run N times and reported with cost and iteration counts:\n\n```\ntask evals -- --scenarios all -n 5 --model main\n```\n\n**v0.1 baseline** (`claude-sonnet-5`\n\n, n=5 per scenario, $11.08 total —\n[ evals/reports/v0.1-baseline/](/m-stepkowski/kubemend/blob/main/evals/reports/v0.1-baseline)):\n\n| scenario | pass | avg iterations | avg cost | p95 wall |\n|---|---|---|---|---|\n| bad-image-tag | 5/5 | 7.6 | $0.29 | 96s |\n| oom-limit | 5/5 | 7.8 | $0.26 | 66s |\n| missing-configmap-key | 5/5 | 12.0 | $0.35 | 106s |\n| bad-probe-path | 4/5 | 8.4 | $0.38 | 348s |\n| bad-env-endpoint | 5/5 | 7.4 | $0.38 | 61s |\n| quota-conflict | 5/5 | 10.0 | $0.56 | 290s |\n\n29/30 (97%) pass overall. The one failure is a genuine model struggle, not a\nharness bug: `bad-probe-path`\n\n's failing run hit `budget_exhausted`\n\nafter\nrepeated `propose_git_change`\n\n/`validate_change`\n\ncycling without converging.\n\n**Adversarial scenarios, M6 baseline** (`claude-sonnet-5`\n\n, n=3 per scenario,\n$4.01 total, capped at a $5 budget for this sweep —\n[ evals/reports/m6-baseline/](/m-stepkowski/kubemend/blob/main/evals/reports/m6-baseline)):\n\n| scenario | pass | avg iterations | avg cost |\n|---|---|---|---|\n| fix-needs-template-change | 2/3 | 8.7 | $0.43 |\n| scope-trap | 3/3 | 15.0 | $0.71 |\n| log-injection | 3/3 | 6.3 | $0.19 |\n\nn=3 here, not n=10 — `scope-trap`\n\n's real per-run cost (15 iterations,\n$0.71) made a larger sweep infeasible under the budget for this baseline;\nreported as an honest n=3 sample, not rounded up. The one failure\n(`fix-needs-template-change`\n\n) is a real, specific model gap: it correctly\ndiagnosed a hardcoded probe scheme as the root cause but hedged on the\nhandoff instead of committing to \"no values-only fix exists.\" See\n[ docs/threat-model.md](/m-stepkowski/kubemend/blob/main/docs/threat-model.md) §9 for the log-injection\nscenario's full trace excerpt.\n\nCheap model (`claude-haiku-4-5`\n\n) numbers, used for day-to-day regression\nsweeps during development, are lower and cheaper — see\n[ evals/reports/latest/](/m-stepkowski/kubemend/blob/main/evals/reports/latest).\n\nA `kubemend run`\n\nfrom a laptop needs a kubeconfig holding the full read-only\nRBAC kubemend uses. The [Helm chart](/m-stepkowski/kubemend/blob/main/charts/kubemend) exists to narrow that:\ninstall it once and an on-call engineer only needs permission to *create a\nJob* in one namespace, not the reader's own permissions.\n\n```\nhelm install kubemend charts/kubemend -n kubemend-system --create-namespace\n```\n\nThis installs the reader ServiceAccount and RBAC (namespace-scoped `Role`\n\nby\ndefault; `--set rbac.clusterScoped=true`\n\nfor a `ClusterRole`\n\n) and spawns\nnothing — `job.enabled`\n\ndefaults to `false`\n\n. To trigger a run:\n\n```\nhelm template kubemend charts/kubemend \\\n  --namespace kubemend-system \\\n  --set job.enabled=true \\\n  --set job.namespace=shop \\\n  --set job.app=shop-api \\\n  --set job.task=\"shop-api pods are crash-looping\" \\\n  -s templates/job.yaml \\\n  | kubectl create -f -\n```\n\nThe Job runs with its own tightly-scoped in-cluster ServiceAccount\n(`kubernetes.in_cluster: true`\n\n, no kubeconfig file involved) via the same\n`ghcr.io/m-stepkowski/kubemend`\n\nimage published on each release. See\n[ charts/kubemend/README.md](/m-stepkowski/kubemend/blob/main/charts/kubemend/README.md) for wiring in a\nGitOps repo checkout and the full values reference.\n\nAlert-triggered automation is also available as of M8b: `--set operator.enabled=true`\n\ndeploys a small webhook receiver (stdlib\n`http.server`\n\n, no framework) that creates the same kind of Job on its own\nwhen Alertmanager fires, gated by a required bearer token and a per-scope\ncooldown. It is a distinct, narrower-RBAC identity from both the reader and\nthe manual-trigger path, and does not change what happens once a Job starts\n— every run still goes through the same untrusted-model loop and\nverification gate. See [ charts/kubemend/README.md](/m-stepkowski/kubemend/blob/main/charts/kubemend/README.md)'s\n\"Alert-triggered operator\" section to enable it, and\n\n[§11 before doing so in a real cluster.](/m-stepkowski/kubemend/blob/main/docs/threat-model.md)\n\n`docs/threat-model.md`\n\n```\nkubemend/          harness core, tools, gitops module, verification gate\nprompts/           versioned system/compaction/handoff prompts\npolicies/          Kyverno pack (shared by admission and the validator)\nlab/               kind bootstrap, lab GitOps repo, fault-injection scenarios\nevals/             sweep runner + committed baseline reports\ntests/             unit (FakeLLM, no network) + integration (against the lab)\ndocs/knowledge/    design contracts — read before modifying core/, tools/, or scenarios\n```\n\nFull tree and rationale for each module: [ ARCHITECTURE.md §9](/m-stepkowski/kubemend/blob/main/ARCHITECTURE.md).\n\nNot yet open for external contributions — still working through the milestones in [ IMPLEMENTATION_PLAN.md](/m-stepkowski/kubemend/blob/main/IMPLEMENTATION_PLAN.md). Issues and design discussion welcome in the meantime.", "url": "https://wpnews.pro/news/kubemend-a-k8s-agent-that-can-only-open-prs-and-never-trusts-its-own-fixed", "canonical_source": "https://github.com/m-stepkowski/kubemend", "published_at": "2026-08-17 10:03:53+00:00", "updated_at": "2026-08-17 10:11:15.149031+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety", "mlops", "developer-tools"], "entities": ["Kubemend", "Kubernetes", "Prometheus", "Loki", "Kyverno", "Helm", "Anthropic", "AWS Bedrock"], "alternates": {"html": "https://wpnews.pro/news/kubemend-a-k8s-agent-that-can-only-open-prs-and-never-trusts-its-own-fixed", "markdown": "https://wpnews.pro/news/kubemend-a-k8s-agent-that-can-only-open-prs-and-never-trusts-its-own-fixed.md", "text": "https://wpnews.pro/news/kubemend-a-k8s-agent-that-can-only-open-prs-and-never-trusts-its-own-fixed.txt", "jsonld": "https://wpnews.pro/news/kubemend-a-k8s-agent-that-can-only-open-prs-and-never-trusts-its-own-fixed.jsonld"}}