A GitOps-native Kubernetes remediation agent that can only open pull requests.
It diagnoses incidents from Prometheus metrics and Loki logs, proposes a fix, and verifies that fix itself — helm render → Kyverno policy check → live diff → scope check → live quota headroom — before it ever asks a human to approve anything. It never runs kubectl apply
. It has no cluster credentials that can write.
Not production-ready: no multi-repo GitOps, no sandboxed tool execution yet. See docs/threat-model.md for what's in and out of scope.
Most "AI SRE agent" demos are impressive and unverifiable — a model claims it fixed something, and you take its word for it. kubemend is built the other way around: the model's claim of success is never trusted. Every run terminates only after an independent validation pipeline says the proposed fix renders cleanly, satisfies policy, produces a real and scoped diff, and touches nothing outside the declared incident. The agent's only actuator is a Git branch and a draft PR — a human still merges.
It's also a from-scratch agent harness, not a wrapper around LangChain/CrewAI/AutoGen. The loop, context management, tool registry, and verification gate are hand-written and documented, because understanding those trade-offs — not gluing a framework together — is the point of the project.
task ──▶ Loop ──▶ tool calls ──▶ Prometheus / Loki / K8s (read-only)
│
└── model claims "done" ──▶ independent verification gate
helm template → kyverno apply
→ argocd/kubectl diff → scope check
→ live quota headroom
│
pass ──▶ draft PR against the GitOps repo
fail ──▶ structured failure fed back into the loop
Observability: PromQL against Prometheus/Mimir, LogQL against Loki. Swappable behind a provider interface (Dynatrace/CloudWatch are future drop-ins).Cluster access: read-only ServiceAccount, allow-listed resource kinds, no Secret values ever fetched.Remediation: the agent edits Helmvalues*.yaml
only — never templates directly — so diffs stay small and reviewable.Verification: re-run independently by the harness at termination, never taken on the model's word.Everything is evaluated: a hermetickind
-based fault-injection lab reproduces real incidents (bad image tags, OOMKills, missing config keys, broken probes...) with property-based checkers, run N times per scenario to produce pass-rate / cost / iteration tables — not cherry-picked demos. Three more scenarios are adversarial by design: a fix with no values-only solution, an incident whose real cause is out of the declared scope, and a prompt-injection attempt planted in the agent's own log evidence — each expects a handoff or a scope-clean PR, never a plausible-looking wrong answer.
Full design, invariants, and every numeric default with its rationale: .
ARCHITECTURE.md
main
and cheap
are each configured independently, so mixing providers
across tiers (e.g. Claude on Bedrock for main
, DeepSeek for cheap
) is a normal configuration, not a special case:
| Provider | model.*.provider |
Covers | Credentials |
|---|---|---|---|
| Anthropic | anthropic (default) |
Claude, direct API | ANTHROPIC_API_KEY , or an ant auth login profile |
| OpenAI-compatible | openai + base_url |
OpenAI, DeepSeek, vLLM, Ollama, anything speaking /v1/chat/completions |
OPENAI_API_KEY (local/self-hosted endpoints without auth fall back to a placeholder automatically) |
| AWS Bedrock | bedrock |
Claude models only, via Bedrock (Converse API / non-Claude models not yet supported) | the standard AWS credential chain (env, profile, or IMDS) |
model:
main:
provider: bedrock
name: us.anthropic.claude-sonnet-5-v1:0
aws_region: us-east-1
cheap:
provider: openai
name: deepseek-v4-flash
base_url: https://api.deepseek.com
See kubemend.yaml
's own comments for more examples, and config/pricing.yaml for cost-guardrail pricing — non-Anthropic entries there are placeholders sourced from public pricing pages, not verified against an invoice; check before trusting them for a committed baseline.
Every tagged release publishes to both PyPI and ghcr.io:
pip install kubemend
docker pull ghcr.io/m-stepkowski/kubemend:latest
docker run --rm ghcr.io/m-stepkowski/kubemend:latest --help
Either way you'll need model credentials (ANTHROPIC_API_KEY
by default —
see "Model providers" above) and a kubemend.yaml
pointing at your cluster's Prometheus/Loki, kubeconfig, and GitOps repo — see the committed kubemend.yaml's own comments for every field. To run in-cluster instead of from a laptop, see "Deploy in-cluster" below.
Requires Docker (or Rancher Desktop —
anything kind
can use), uv, and
go-task
ANTHROPIC_API_KEY
.The fastest way to see it work end to end — bring up the lab, inject a real fault, run the agent against it, and print the resulting proposal — is:
git clone https://github.com/m-stepkowski/kubemend.git && cd kubemend
uv sync
export ANTHROPIC_API_KEY=...
task lab:up # kind cluster: gitea, Argo CD, kube-prometheus-stack, Loki, Kyverno
task demo # inject a fault, run kubemend, show the resulting proposal (~90s)
task demo
runs on the cheap model by default; pass -- --model main
to use the model the headline sweep below was run on:
task demo -- --model main
To drive it by hand instead of via the demo script:
task lab:forward # port-forward Prometheus/Loki/gitea/Argo locally, blocks — run in another terminal
kubemend run --task "shop-api pods in namespace shop are crash-looping since 10 minutes ago" \
--namespace shop --app shop-api
This writes a branch (and, with gitops.backend: gitea
, a real draft PR in
the lab's gitea instance) plus a full JSONL trace under traces/
. See docs/threat-model.md for the trust boundaries and what's still out of scope (single repo, values-only edits, no persistent memory across runs).
Reproducible pass-rate benchmarks, not anecdotes — every scenario is run N times and reported with cost and iteration counts:
task evals -- --scenarios all -n 5 --model main
v0.1 baseline (claude-sonnet-5
, n=5 per scenario, $11.08 total — evals/reports/v0.1-baseline/):
| scenario | pass | avg iterations | avg cost | p95 wall |
|---|---|---|---|---|
| bad-image-tag | 5/5 | 7.6 | $0.29 | 96s |
| oom-limit | 5/5 | 7.8 | $0.26 | 66s |
| missing-configmap-key | 5/5 | 12.0 | $0.35 | 106s |
| bad-probe-path | 4/5 | 8.4 | $0.38 | 348s |
| bad-env-endpoint | 5/5 | 7.4 | $0.38 | 61s |
| quota-conflict | 5/5 | 10.0 | $0.56 | 290s |
29/30 (97%) pass overall. The one failure is a genuine model struggle, not a
harness bug: bad-probe-path
's failing run hit budget_exhausted
after
repeated propose_git_change
/validate_change
cycling without converging.
Adversarial scenarios, M6 baseline (claude-sonnet-5
, n=3 per scenario, $4.01 total, capped at a $5 budget for this sweep — evals/reports/m6-baseline/):
| scenario | pass | avg iterations | avg cost |
|---|---|---|---|
| fix-needs-template-change | 2/3 | 8.7 | $0.43 |
| scope-trap | 3/3 | 15.0 | $0.71 |
| log-injection | 3/3 | 6.3 | $0.19 |
n=3 here, not n=10 — scope-trap
's real per-run cost (15 iterations,
$0.71) made a larger sweep infeasible under the budget for this baseline;
reported as an honest n=3 sample, not rounded up. The one failure
(fix-needs-template-change
) is a real, specific model gap: it correctly diagnosed a hardcoded probe scheme as the root cause but hedged on the handoff instead of committing to "no values-only fix exists." See docs/threat-model.md §9 for the log-injection scenario's full trace excerpt.
Cheap model (claude-haiku-4-5
) numbers, used for day-to-day regression sweeps during development, are lower and cheaper — see evals/reports/latest/.
A kubemend run
from a laptop needs a kubeconfig holding the full read-only RBAC kubemend uses. The Helm chart exists to narrow that: install it once and an on-call engineer only needs permission to create a Job in one namespace, not the reader's own permissions.
helm install kubemend charts/kubemend -n kubemend-system --create-namespace
This installs the reader ServiceAccount and RBAC (namespace-scoped Role
by
default; --set rbac.clusterScoped=true
for a ClusterRole
) and spawns
nothing — job.enabled
defaults to false
. To trigger a run:
helm template kubemend charts/kubemend \
--namespace kubemend-system \
--set job.enabled=true \
--set job.namespace=shop \
--set job.app=shop-api \
--set job.task="shop-api pods are crash-looping" \
-s templates/job.yaml \
| kubectl create -f -
The Job runs with its own tightly-scoped in-cluster ServiceAccount
(kubernetes.in_cluster: true
, no kubeconfig file involved) via the same
ghcr.io/m-stepkowski/kubemend
image published on each release. See charts/kubemend/README.md for wiring in a GitOps repo checkout and the full values reference.
Alert-triggered automation is also available as of M8b: --set operator.enabled=true
deploys a small webhook receiver (stdlib
http.server
, no framework) that creates the same kind of Job on its own when Alertmanager fires, gated by a required bearer token and a per-scope cooldown. It is a distinct, narrower-RBAC identity from both the reader and the manual-trigger path, and does not change what happens once a Job starts — every run still goes through the same untrusted-model loop and verification gate. See charts/kubemend/README.md's "Alert-triggered operator" section to enable it, and
§11 before doing so in a real cluster.
docs/threat-model.md
kubemend/ harness core, tools, gitops module, verification gate
prompts/ versioned system/compaction/handoff prompts
policies/ Kyverno pack (shared by admission and the validator)
lab/ kind bootstrap, lab GitOps repo, fault-injection scenarios
evals/ sweep runner + committed baseline reports
tests/ unit (FakeLLM, no network) + integration (against the lab)
docs/knowledge/ design contracts — read before modifying core/, tools/, or scenarios
Full tree and rationale for each module: ARCHITECTURE.md §9.
Not yet open for external contributions — still working through the milestones in IMPLEMENTATION_PLAN.md. Issues and design discussion welcome in the meantime.