{"slug": "kubernetes-oomkilled-diagnose-with-a-plan-not-a-wall-of-kubectl", "title": "Kubernetes OOMKilled — diagnose with a plan, not a wall of kubectl", "summary": "Kubernetes OOMKilled errors are a common day-2 operational issue that can be misdiagnosed, leading to risky fixes. kprompt, an AI-powered tool, turns the investigation into a reviewable plan, detecting OOMKilled and proposing memory limit patches that require human approval before applying. The tool reads live cluster signals and suggests concrete mutations, ensuring changes are not silently applied to production.", "body_md": "OOMKilled is a classic day-2 rabbit hole: events, limits, restarts, then a risky scale or edit. kprompt turns the investigation prompt into a reviewable plan before apply.\n\n*Originally published at https://kprompt.ai/blog/kubernetes-oomkilled.*\n\nOOMKilled is one of the most common “the app is broken” signals in Kubernetes — and one of the easiest to misread. The Pod may still show Running. Restarts climb. Logs look fine until they stop mid-request. Someone raises the memory limit “a bit,” the Deployment rolls, and two hours later it happens again. Or worse: they remove the limit entirely and the node starts evicting neighbors.\n\nThis guide is the operator ladder for memory kills: how to confirm OOMKilled, how requests and limits differ, what kubectl shows, and how to apply a bounded fix with a reviewable plan. kprompt's explain path detects OOM findings and can propose a memory patch — still behind approval, because raising limits is a real cluster change.\n\nWhen a container exceeds its memory limit, the Linux OOM killer (via cgroup enforcement) terminates the process. Kubernetes records the termination reason as OOMKilled. Exit code is often 137 (128 + SIGKILL). That is not an application “bug code” — it is the kernel saying the cgroup ran out of memory.\n\nDo not raise memory because “it feels like OOM.” Read the Pod status. The smoking gun is usually Last State / Last Termination State on the container: Reason OOMKilled, Exit Code 137.\n\n*Classic kubectl confirmation*\n\n```\nkubectl get pods -n staging\nkubectl describe pod -l app=api -n staging\n# Look under Containers → Last State:\n#   Reason: OOMKilled\n#   Exit Code: 137\n\nkubectl get pod -n staging -o jsonpath='{range .items[*]}{.metadata.name}{\"\\t\"}{range .status.containerStatuses[*]}{.name}{\"=\"}{.lastState.terminated.reason}{\" \"}{end}{\"\\n\"}{end}'\n```\n\nAlso check current limits on the Deployment template — describe Pod shows what ran; the Deployment owns what will run next:\n\n*See memory requests and limits*\n\n```\nkubectl get deploy api -n staging -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{\" limits=\"}{.resources.limits.memory}{\" requests=\"}{.resources.requests.memory}{\"\\n\"}{end}'\n```\n\n| Field | What it does | OOM relevance |\n|---|---|---|\n| requests.memory | Scheduler places the Pod on a node with enough capacity | Too low → noisy neighbor risk; does not by itself OOMKill |\n| limits.memory | Hard cgroup cap for the container | Exceed this → OOMKilled |\n| No limit | Container can use free node memory | May avoid OOMKilled on that Pod; can hurt the node |\n\nA healthy fix usually raises the limit (and often the request toward a sensible fraction of that limit) based on observed usage — not deleting limits to “make it stop.” If you have Prometheus, compare working set / RSS to the current limit before you double everything.\n\n*Investigation sequence*\n\n```\nkubectl describe deploy api -n staging\nkubectl describe pod -l app=api -n staging\nkubectl logs deploy/api -n staging --previous --tail=100\nkubectl get events -n staging --field-selector reason=OOMKilling --sort-by='.lastTimestamp'\n```\n\nkprompt's explain path walks live Deployment → Pod → Events → Logs style signals. When it finds OOMKilled on a container, it can propose a follow-up: raise the Deployment memory limit (typically doubling a known limit in the suggested plan) and show the plan for approval. Reads run immediately; the patch does not apply until you confirm — or you pass --approve in a context you trust.\n\n*Detect and review a memory fix*\n\n``` bash\n$ kprompt \"explain why api is crashing\" -n staging\n\n# … findings include OOMKilled on container app …\n\nSuggested fix (requires approval):\nPlan\n  1. patch Deployment/api memory limit (e.g. 64Mi → 128Mi)\n\nRisk: medium\nApply? [y/N]\n```\n\nThat is the intent-compiler shape: evidence from the apiserver, a concrete mutation plan, human gate. It is not “the model silently edited production.” If you reject the plan, nothing changes — dig into leaks, heap dumps, or a bad release instead.\n\nSometimes you already know the target (512Mi limit, 256Mi request). Use kubectl or a reviewed kprompt plan with an explicit change — do not approve a suggested bump you have not sanity-checked against metrics.\n\n*Explicit memory patch*\n\n```\nkubectl set resources deploy/api -n staging \\\n  --limits=memory=512Mi --requests=memory=256Mi\n\n# or edit the template\nkubectl edit deploy api -n staging\n```\n\nSpin a tiny limit on kind or staging, force an OOM, then run explain and decide whether to approve the suggested patch. Pair with the [ImagePullBackOff guide](https://kprompt.ai/blog/kubernetes-imagepullbackoff) when the Pod never starts, and with the [CrashLoopBackOff guide](https://kprompt.ai/blog/kubernetes-crashloopbackoff) when the memory kill is what keeps the container looping.\n\n*Quick start*\n\n```\ncurl -fsSL https://kprompt.ai/install | bash\nexport KPROMPT_GEMINI_API_KEY=\"...\"\n\nkprompt \"explain why api is crashing\" -n staging\n# review Suggested fix → y or n\n```\n\n**Try:** [kprompt.ai](https://kprompt.ai) · [GitHub](https://github.com/kprompt/kprompt) · `brew install kprompt/tap/kprompt`", "url": "https://wpnews.pro/news/kubernetes-oomkilled-diagnose-with-a-plan-not-a-wall-of-kubectl", "canonical_source": "https://dev.to/muhtalipdede/kubernetes-oomkilled-diagnose-with-a-plan-not-a-wall-of-kubectl-4997", "published_at": "2026-08-14 20:29:14+00:00", "updated_at": "2026-08-14 20:36:19.420466+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops"], "entities": ["Kubernetes", "kprompt"], "alternates": {"html": "https://wpnews.pro/news/kubernetes-oomkilled-diagnose-with-a-plan-not-a-wall-of-kubectl", "markdown": "https://wpnews.pro/news/kubernetes-oomkilled-diagnose-with-a-plan-not-a-wall-of-kubectl.md", "text": "https://wpnews.pro/news/kubernetes-oomkilled-diagnose-with-a-plan-not-a-wall-of-kubectl.txt", "jsonld": "https://wpnews.pro/news/kubernetes-oomkilled-diagnose-with-a-plan-not-a-wall-of-kubectl.jsonld"}}