# Kubernetes OOMKilled — diagnose with a plan, not a wall of kubectl

> Source: <https://dev.to/muhtalipdede/kubernetes-oomkilled-diagnose-with-a-plan-not-a-wall-of-kubectl-4997>
> Published: 2026-08-14 20:29:14+00:00

OOMKilled is a classic day-2 rabbit hole: events, limits, restarts, then a risky scale or edit. kprompt turns the investigation prompt into a reviewable plan before apply.

*Originally published at https://kprompt.ai/blog/kubernetes-oomkilled.*

OOMKilled is one of the most common “the app is broken” signals in Kubernetes — and one of the easiest to misread. The Pod may still show Running. Restarts climb. Logs look fine until they stop mid-request. Someone raises the memory limit “a bit,” the Deployment rolls, and two hours later it happens again. Or worse: they remove the limit entirely and the node starts evicting neighbors.

This guide is the operator ladder for memory kills: how to confirm OOMKilled, how requests and limits differ, what kubectl shows, and how to apply a bounded fix with a reviewable plan. kprompt's explain path detects OOM findings and can propose a memory patch — still behind approval, because raising limits is a real cluster change.

When a container exceeds its memory limit, the Linux OOM killer (via cgroup enforcement) terminates the process. Kubernetes records the termination reason as OOMKilled. Exit code is often 137 (128 + SIGKILL). That is not an application “bug code” — it is the kernel saying the cgroup ran out of memory.

Do not raise memory because “it feels like OOM.” Read the Pod status. The smoking gun is usually Last State / Last Termination State on the container: Reason OOMKilled, Exit Code 137.

*Classic kubectl confirmation*

```
kubectl get pods -n staging
kubectl describe pod -l app=api -n staging
# Look under Containers → Last State:
#   Reason: OOMKilled
#   Exit Code: 137

kubectl get pod -n staging -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .status.containerStatuses[*]}{.name}{"="}{.lastState.terminated.reason}{" "}{end}{"\n"}{end}'
```

Also check current limits on the Deployment template — describe Pod shows what ran; the Deployment owns what will run next:

*See memory requests and limits*

```
kubectl get deploy api -n staging -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" limits="}{.resources.limits.memory}{" requests="}{.resources.requests.memory}{"\n"}{end}'
```

| Field | What it does | OOM relevance |
|---|---|---|
| requests.memory | Scheduler places the Pod on a node with enough capacity | Too low → noisy neighbor risk; does not by itself OOMKill |
| limits.memory | Hard cgroup cap for the container | Exceed this → OOMKilled |
| No limit | Container can use free node memory | May avoid OOMKilled on that Pod; can hurt the node |

A healthy fix usually raises the limit (and often the request toward a sensible fraction of that limit) based on observed usage — not deleting limits to “make it stop.” If you have Prometheus, compare working set / RSS to the current limit before you double everything.

*Investigation sequence*

```
kubectl describe deploy api -n staging
kubectl describe pod -l app=api -n staging
kubectl logs deploy/api -n staging --previous --tail=100
kubectl get events -n staging --field-selector reason=OOMKilling --sort-by='.lastTimestamp'
```

kprompt's explain path walks live Deployment → Pod → Events → Logs style signals. When it finds OOMKilled on a container, it can propose a follow-up: raise the Deployment memory limit (typically doubling a known limit in the suggested plan) and show the plan for approval. Reads run immediately; the patch does not apply until you confirm — or you pass --approve in a context you trust.

*Detect and review a memory fix*

``` bash
$ kprompt "explain why api is crashing" -n staging

# … findings include OOMKilled on container app …

Suggested fix (requires approval):
Plan
  1. patch Deployment/api memory limit (e.g. 64Mi → 128Mi)

Risk: medium
Apply? [y/N]
```

That is the intent-compiler shape: evidence from the apiserver, a concrete mutation plan, human gate. It is not “the model silently edited production.” If you reject the plan, nothing changes — dig into leaks, heap dumps, or a bad release instead.

Sometimes you already know the target (512Mi limit, 256Mi request). Use kubectl or a reviewed kprompt plan with an explicit change — do not approve a suggested bump you have not sanity-checked against metrics.

*Explicit memory patch*

```
kubectl set resources deploy/api -n staging \
  --limits=memory=512Mi --requests=memory=256Mi

# or edit the template
kubectl edit deploy api -n staging
```

Spin a tiny limit on kind or staging, force an OOM, then run explain and decide whether to approve the suggested patch. Pair with the [ImagePullBackOff guide](https://kprompt.ai/blog/kubernetes-imagepullbackoff) when the Pod never starts, and with the [CrashLoopBackOff guide](https://kprompt.ai/blog/kubernetes-crashloopbackoff) when the memory kill is what keeps the container looping.

*Quick start*

```
curl -fsSL https://kprompt.ai/install | bash
export KPROMPT_GEMINI_API_KEY="..."

kprompt "explain why api is crashing" -n staging
# review Suggested fix → y or n
```

**Try:** [kprompt.ai](https://kprompt.ai) · [GitHub](https://github.com/kprompt/kprompt) · `brew install kprompt/tap/kprompt`