Kubernetes OOMKilled — diagnose with a plan, not a wall of kubectl

Kubernetes OOMKilled errors are a common day-2 operational issue that can be misdiagnosed, leading to risky fixes. kprompt, an AI-powered tool, turns the investigation into a reviewable plan, detecting OOMKilled and proposing memory limit patches that require human approval before applying. The tool reads live cluster signals and suggests concrete mutations, ensuring changes are not silently applied to production.

OOMKilled is a classic day-2 rabbit hole: events, limits, restarts, then a risky scale or edit. kprompt turns the investigation prompt into a reviewable plan before apply. Originally published at https://kprompt.ai/blog/kubernetes-oomkilled. OOMKilled is one of the most common “the app is broken” signals in Kubernetes — and one of the easiest to misread. The Pod may still show Running. Restarts climb. Logs look fine until they stop mid-request. Someone raises the memory limit “a bit,” the Deployment rolls, and two hours later it happens again. Or worse: they remove the limit entirely and the node starts evicting neighbors. This guide is the operator ladder for memory kills: how to confirm OOMKilled, how requests and limits differ, what kubectl shows, and how to apply a bounded fix with a reviewable plan. kprompt's explain path detects OOM findings and can propose a memory patch — still behind approval, because raising limits is a real cluster change. When a container exceeds its memory limit, the Linux OOM killer via cgroup enforcement terminates the process. Kubernetes records the termination reason as OOMKilled. Exit code is often 137 128 + SIGKILL . That is not an application “bug code” — it is the kernel saying the cgroup ran out of memory. Do not raise memory because “it feels like OOM.” Read the Pod status. The smoking gun is usually Last State / Last Termination State on the container: Reason OOMKilled, Exit Code 137. Classic kubectl confirmation kubectl get pods -n staging kubectl describe pod -l app=api -n staging Look under Containers → Last State: Reason: OOMKilled Exit Code: 137 kubectl get pod -n staging -o jsonpath='{range .items }{.metadata.name}{"\t"}{range .status.containerStatuses }{.name}{"="}{.lastState.terminated.reason}{" "}{end}{"\n"}{end}' Also check current limits on the Deployment template — describe Pod shows what ran; the Deployment owns what will run next: See memory requests and limits kubectl get deploy api -n staging -o jsonpath='{range .spec.template.spec.containers }{.name}{" limits="}{.resources.limits.memory}{" requests="}{.resources.requests.memory}{"\n"}{end}' | Field | What it does | OOM relevance | |---|---|---| | requests.memory | Scheduler places the Pod on a node with enough capacity | Too low → noisy neighbor risk; does not by itself OOMKill | | limits.memory | Hard cgroup cap for the container | Exceed this → OOMKilled | | No limit | Container can use free node memory | May avoid OOMKilled on that Pod; can hurt the node | A healthy fix usually raises the limit and often the request toward a sensible fraction of that limit based on observed usage — not deleting limits to “make it stop.” If you have Prometheus, compare working set / RSS to the current limit before you double everything. Investigation sequence kubectl describe deploy api -n staging kubectl describe pod -l app=api -n staging kubectl logs deploy/api -n staging --previous --tail=100 kubectl get events -n staging --field-selector reason=OOMKilling --sort-by='.lastTimestamp' kprompt's explain path walks live Deployment → Pod → Events → Logs style signals. When it finds OOMKilled on a container, it can propose a follow-up: raise the Deployment memory limit typically doubling a known limit in the suggested plan and show the plan for approval. Reads run immediately; the patch does not apply until you confirm — or you pass --approve in a context you trust. Detect and review a memory fix bash $ kprompt "explain why api is crashing" -n staging … findings include OOMKilled on container app … Suggested fix requires approval : Plan 1. patch Deployment/api memory limit e.g. 64Mi → 128Mi Risk: medium Apply? y/N That is the intent-compiler shape: evidence from the apiserver, a concrete mutation plan, human gate. It is not “the model silently edited production.” If you reject the plan, nothing changes — dig into leaks, heap dumps, or a bad release instead. Sometimes you already know the target 512Mi limit, 256Mi request . Use kubectl or a reviewed kprompt plan with an explicit change — do not approve a suggested bump you have not sanity-checked against metrics. Explicit memory patch kubectl set resources deploy/api -n staging \ --limits=memory=512Mi --requests=memory=256Mi or edit the template kubectl edit deploy api -n staging Spin a tiny limit on kind or staging, force an OOM, then run explain and decide whether to approve the suggested patch. Pair with the ImagePullBackOff guide https://kprompt.ai/blog/kubernetes-imagepullbackoff when the Pod never starts, and with the CrashLoopBackOff guide https://kprompt.ai/blog/kubernetes-crashloopbackoff when the memory kill is what keeps the container looping. Quick start curl -fsSL https://kprompt.ai/install | bash export KPROMPT GEMINI API KEY="..." kprompt "explain why api is crashing" -n staging review Suggested fix → y or n Try: kprompt.ai https://kprompt.ai · GitHub https://github.com/kprompt/kprompt · brew install kprompt/tap/kprompt