Kubemend – a K8s agent that can only open PRs, and never trusts its own "fixed"

Kubemend, a GitOps-native Kubernetes remediation agent, can only open pull requests and never trusts its own fixes, requiring independent verification before a human merges. The agent diagnoses incidents from Prometheus metrics and Loki logs, proposes fixes by editing Helm values files, and validates them through helm render, Kyverno policy checks, live diffs, scope checks, and quota headroom checks. It supports multiple model providers including Anthropic, OpenAI-compatible endpoints, and AWS Bedrock, and is not yet production-ready due to missing multi-repo GitOps and sandboxed tool execution.

A GitOps-native Kubernetes remediation agent that can only open pull requests. It diagnoses incidents from Prometheus metrics and Loki logs, proposes a fix, and verifies that fix itself — helm render → Kyverno policy check → live diff → scope check → live quota headroom — before it ever asks a human to approve anything. It never runs kubectl apply . It has no cluster credentials that can write. Not production-ready: no multi-repo GitOps, no sandboxed tool execution yet. See docs/threat-model.md /m-stepkowski/kubemend/blob/main/docs/threat-model.md for what's in and out of scope. Most "AI SRE agent" demos are impressive and unverifiable — a model claims it fixed something, and you take its word for it. kubemend is built the other way around: the model's claim of success is never trusted. Every run terminates only after an independent validation pipeline says the proposed fix renders cleanly, satisfies policy, produces a real and scoped diff, and touches nothing outside the declared incident. The agent's only actuator is a Git branch and a draft PR — a human still merges. It's also a from-scratch agent harness, not a wrapper around LangChain/CrewAI/AutoGen. The loop, context management, tool registry, and verification gate are hand-written and documented, because understanding those trade-offs — not gluing a framework together — is the point of the project. task ──▶ Loop ──▶ tool calls ──▶ Prometheus / Loki / K8s read-only │ └── model claims "done" ──▶ independent verification gate helm template → kyverno apply → argocd/kubectl diff → scope check → live quota headroom │ pass ──▶ draft PR against the GitOps repo fail ──▶ structured failure fed back into the loop Observability: PromQL against Prometheus/Mimir, LogQL against Loki. Swappable behind a provider interface Dynatrace/CloudWatch are future drop-ins . Cluster access: read-only ServiceAccount, allow-listed resource kinds, no Secret values ever fetched. Remediation: the agent edits Helm values .yaml only — never templates directly — so diffs stay small and reviewable. Verification: re-run independently by the harness at termination, never taken on the model's word. Everything is evaluated: a hermetic kind -based fault-injection lab reproduces real incidents bad image tags, OOMKills, missing config keys, broken probes... with property-based checkers, run N times per scenario to produce pass-rate / cost / iteration tables — not cherry-picked demos. Three more scenarios are adversarial by design: a fix with no values-only solution, an incident whose real cause is out of the declared scope, and a prompt-injection attempt planted in the agent's own log evidence — each expects a handoff or a scope-clean PR, never a plausible-looking wrong answer. Full design, invariants, and every numeric default with its rationale: . ARCHITECTURE.md main and cheap are each configured independently, so mixing providers across tiers e.g. Claude on Bedrock for main , DeepSeek for cheap is a normal configuration, not a special case: | Provider | model. .provider | Covers | Credentials | |---|---|---|---| | Anthropic | anthropic default | Claude, direct API | ANTHROPIC API KEY , or an ant auth login profile | | OpenAI-compatible | openai + base url | OpenAI, DeepSeek, vLLM, Ollama, anything speaking /v1/chat/completions | OPENAI API KEY local/self-hosted endpoints without auth fall back to a placeholder automatically | | AWS Bedrock | bedrock | Claude models only, via Bedrock Converse API / non-Claude models not yet supported | the standard AWS credential chain env, profile, or IMDS | model: main: provider: bedrock name: us.anthropic.claude-sonnet-5-v1:0 aws region: us-east-1 cheap: provider: openai name: deepseek-v4-flash base url: https://api.deepseek.com See kubemend.yaml 's own comments for more examples, and config/pricing.yaml /m-stepkowski/kubemend/blob/main/config/pricing.yaml for cost-guardrail pricing — non-Anthropic entries there are placeholders sourced from public pricing pages, not verified against an invoice; check before trusting them for a committed baseline. Every tagged release publishes to both PyPI and ghcr.io: pip install kubemend docker pull ghcr.io/m-stepkowski/kubemend:latest docker run --rm ghcr.io/m-stepkowski/kubemend:latest --help Either way you'll need model credentials ANTHROPIC API KEY by default — see "Model providers" above and a kubemend.yaml pointing at your cluster's Prometheus/Loki, kubeconfig, and GitOps repo — see the committed kubemend.yaml /m-stepkowski/kubemend/blob/main/kubemend.yaml 's own comments for every field. To run in-cluster instead of from a laptop, see "Deploy in-cluster" below. Requires Docker https://docs.docker.com/get-docker/ or Rancher Desktop — anything kind can use , uv https://docs.astral.sh/uv/ , and , plus an https://taskfile.dev/ go-task ANTHROPIC API KEY .The fastest way to see it work end to end — bring up the lab, inject a real fault, run the agent against it, and print the resulting proposal — is: git clone https://github.com/m-stepkowski/kubemend.git && cd kubemend uv sync export ANTHROPIC API KEY=... task lab:up kind cluster: gitea, Argo CD, kube-prometheus-stack, Loki, Kyverno task demo inject a fault, run kubemend, show the resulting proposal ~90s task demo runs on the cheap model by default; pass -- --model main to use the model the headline sweep below was run on: task demo -- --model main To drive it by hand instead of via the demo script: task lab:forward port-forward Prometheus/Loki/gitea/Argo locally, blocks — run in another terminal kubemend run --task "shop-api pods in namespace shop are crash-looping since 10 minutes ago" \ --namespace shop --app shop-api This writes a branch and, with gitops.backend: gitea , a real draft PR in the lab's gitea instance plus a full JSONL trace under traces/ . See docs/threat-model.md /m-stepkowski/kubemend/blob/main/docs/threat-model.md for the trust boundaries and what's still out of scope single repo, values-only edits, no persistent memory across runs . Reproducible pass-rate benchmarks, not anecdotes — every scenario is run N times and reported with cost and iteration counts: task evals -- --scenarios all -n 5 --model main v0.1 baseline claude-sonnet-5 , n=5 per scenario, $11.08 total — evals/reports/v0.1-baseline/ /m-stepkowski/kubemend/blob/main/evals/reports/v0.1-baseline : | scenario | pass | avg iterations | avg cost | p95 wall | |---|---|---|---|---| | bad-image-tag | 5/5 | 7.6 | $0.29 | 96s | | oom-limit | 5/5 | 7.8 | $0.26 | 66s | | missing-configmap-key | 5/5 | 12.0 | $0.35 | 106s | | bad-probe-path | 4/5 | 8.4 | $0.38 | 348s | | bad-env-endpoint | 5/5 | 7.4 | $0.38 | 61s | | quota-conflict | 5/5 | 10.0 | $0.56 | 290s | 29/30 97% pass overall. The one failure is a genuine model struggle, not a harness bug: bad-probe-path 's failing run hit budget exhausted after repeated propose git change / validate change cycling without converging. Adversarial scenarios, M6 baseline claude-sonnet-5 , n=3 per scenario, $4.01 total, capped at a $5 budget for this sweep — evals/reports/m6-baseline/ /m-stepkowski/kubemend/blob/main/evals/reports/m6-baseline : | scenario | pass | avg iterations | avg cost | |---|---|---|---| | fix-needs-template-change | 2/3 | 8.7 | $0.43 | | scope-trap | 3/3 | 15.0 | $0.71 | | log-injection | 3/3 | 6.3 | $0.19 | n=3 here, not n=10 — scope-trap 's real per-run cost 15 iterations, $0.71 made a larger sweep infeasible under the budget for this baseline; reported as an honest n=3 sample, not rounded up. The one failure fix-needs-template-change is a real, specific model gap: it correctly diagnosed a hardcoded probe scheme as the root cause but hedged on the handoff instead of committing to "no values-only fix exists." See docs/threat-model.md /m-stepkowski/kubemend/blob/main/docs/threat-model.md §9 for the log-injection scenario's full trace excerpt. Cheap model claude-haiku-4-5 numbers, used for day-to-day regression sweeps during development, are lower and cheaper — see evals/reports/latest/ /m-stepkowski/kubemend/blob/main/evals/reports/latest . A kubemend run from a laptop needs a kubeconfig holding the full read-only RBAC kubemend uses. The Helm chart /m-stepkowski/kubemend/blob/main/charts/kubemend exists to narrow that: install it once and an on-call engineer only needs permission to create a Job in one namespace, not the reader's own permissions. helm install kubemend charts/kubemend -n kubemend-system --create-namespace This installs the reader ServiceAccount and RBAC namespace-scoped Role by default; --set rbac.clusterScoped=true for a ClusterRole and spawns nothing — job.enabled defaults to false . To trigger a run: helm template kubemend charts/kubemend \ --namespace kubemend-system \ --set job.enabled=true \ --set job.namespace=shop \ --set job.app=shop-api \ --set job.task="shop-api pods are crash-looping" \ -s templates/job.yaml \ | kubectl create -f - The Job runs with its own tightly-scoped in-cluster ServiceAccount kubernetes.in cluster: true , no kubeconfig file involved via the same ghcr.io/m-stepkowski/kubemend image published on each release. See charts/kubemend/README.md /m-stepkowski/kubemend/blob/main/charts/kubemend/README.md for wiring in a GitOps repo checkout and the full values reference. Alert-triggered automation is also available as of M8b: --set operator.enabled=true deploys a small webhook receiver stdlib http.server , no framework that creates the same kind of Job on its own when Alertmanager fires, gated by a required bearer token and a per-scope cooldown. It is a distinct, narrower-RBAC identity from both the reader and the manual-trigger path, and does not change what happens once a Job starts — every run still goes through the same untrusted-model loop and verification gate. See charts/kubemend/README.md /m-stepkowski/kubemend/blob/main/charts/kubemend/README.md 's "Alert-triggered operator" section to enable it, and §11 before doing so in a real cluster. /m-stepkowski/kubemend/blob/main/docs/threat-model.md docs/threat-model.md kubemend/ harness core, tools, gitops module, verification gate prompts/ versioned system/compaction/handoff prompts policies/ Kyverno pack shared by admission and the validator lab/ kind bootstrap, lab GitOps repo, fault-injection scenarios evals/ sweep runner + committed baseline reports tests/ unit FakeLLM, no network + integration against the lab docs/knowledge/ design contracts — read before modifying core/, tools/, or scenarios Full tree and rationale for each module: ARCHITECTURE.md §9 /m-stepkowski/kubemend/blob/main/ARCHITECTURE.md . Not yet open for external contributions — still working through the milestones in IMPLEMENTATION PLAN.md /m-stepkowski/kubemend/blob/main/IMPLEMENTATION PLAN.md . Issues and design discussion welcome in the meantime.