OOMKilled is a classic day-2 rabbit hole: events, limits, restarts, then a risky scale or edit. kprompt turns the investigation prompt into a reviewable plan before apply.
Originally published at https://kprompt.ai/blog/kubernetes-oomkilled.
OOMKilled is one of the most common “the app is broken” signals in Kubernetes — and one of the easiest to misread. The Pod may still show Running. Restarts climb. Logs look fine until they stop mid-request. Someone raises the memory limit “a bit,” the Deployment rolls, and two hours later it happens again. Or worse: they remove the limit entirely and the node starts evicting neighbors.
This guide is the operator ladder for memory kills: how to confirm OOMKilled, how requests and limits differ, what kubectl shows, and how to apply a bounded fix with a reviewable plan. kprompt's explain path detects OOM findings and can propose a memory patch — still behind approval, because raising limits is a real cluster change.
When a container exceeds its memory limit, the Linux OOM killer (via cgroup enforcement) terminates the process. Kubernetes records the termination reason as OOMKilled. Exit code is often 137 (128 + SIGKILL). That is not an application “bug code” — it is the kernel saying the cgroup ran out of memory.
Do not raise memory because “it feels like OOM.” Read the Pod status. The smoking gun is usually Last State / Last Termination State on the container: Reason OOMKilled, Exit Code 137.
Classic kubectl confirmation
kubectl get pods -n staging
kubectl describe pod -l app=api -n staging
kubectl get pod -n staging -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .status.containerStatuses[*]}{.name}{"="}{.lastState.terminated.reason}{" "}{end}{"\n"}{end}'
Also check current limits on the Deployment template — describe Pod shows what ran; the Deployment owns what will run next:
See memory requests and limits
kubectl get deploy api -n staging -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" limits="}{.resources.limits.memory}{" requests="}{.resources.requests.memory}{"\n"}{end}'
| Field | What it does | OOM relevance |
|---|---|---|
| requests.memory | Scheduler places the Pod on a node with enough capacity | Too low → noisy neighbor risk; does not by itself OOMKill |
| limits.memory | Hard cgroup cap for the container | Exceed this → OOMKilled |
| No limit | Container can use free node memory | May avoid OOMKilled on that Pod; can hurt the node |
A healthy fix usually raises the limit (and often the request toward a sensible fraction of that limit) based on observed usage — not deleting limits to “make it stop.” If you have Prometheus, compare working set / RSS to the current limit before you double everything.
Investigation sequence
kubectl describe deploy api -n staging
kubectl describe pod -l app=api -n staging
kubectl logs deploy/api -n staging --previous --tail=100
kubectl get events -n staging --field-selector reason=OOMKilling --sort-by='.lastTimestamp'
kprompt's explain path walks live Deployment → Pod → Events → Logs style signals. When it finds OOMKilled on a container, it can propose a follow-up: raise the Deployment memory limit (typically doubling a known limit in the suggested plan) and show the plan for approval. Reads run immediately; the patch does not apply until you confirm — or you pass --approve in a context you trust.
Detect and review a memory fix
$ kprompt "explain why api is crashing" -n staging
Suggested fix (requires approval):
Plan
1. patch Deployment/api memory limit (e.g. 64Mi → 128Mi)
Risk: medium
Apply? [y/N]
That is the intent-compiler shape: evidence from the apiserver, a concrete mutation plan, human gate. It is not “the model silently edited production.” If you reject the plan, nothing changes — dig into leaks, heap dumps, or a bad release instead.
Sometimes you already know the target (512Mi limit, 256Mi request). Use kubectl or a reviewed kprompt plan with an explicit change — do not approve a suggested bump you have not sanity-checked against metrics.
Explicit memory patch
kubectl set resources deploy/api -n staging \
--limits=memory=512Mi --requests=memory=256Mi
kubectl edit deploy api -n staging
Spin a tiny limit on kind or staging, force an OOM, then run explain and decide whether to approve the suggested patch. Pair with the ImagePullBackOff guide when the Pod never starts, and with the CrashLoopBackOff guide when the memory kill is what keeps the container looping.
Quick start
curl -fsSL https://kprompt.ai/install | bash
export KPROMPT_GEMINI_API_KEY="..."
kprompt "explain why api is crashing" -n staging
Try: kprompt.ai · GitHub · brew install kprompt/tap/kprompt