{"slug": "debugging-stuck-node-js-processes", "title": "Debugging stuck Node.js processes", "summary": "HOAi engineers debugged a Node.js application that intermittently failed health checks and required automatic restarts, with CPU pegged at 100% on one core and no network traffic or log activity. After initial tracing-based detection blamed short-lived blocks under 5 seconds, they used Node's built-in debugger to capture CPU profiles of the process in its final moments, identifying a long-running single segment that never yielded to the event loop.", "body_md": "When production software fails, it’s critically important to both understand why it failed and prevent it from happening again. From the largest incidents to the “wish” priority bugs, root cause analysis remains a tested strategy for fixing software defects. This usually involves checking logs, metrics, reading the code, and running local simulations. But what do you do when you have a failure that evades all attempts to understand it? We were facing just that scenario at HOAi.\n\nOne of our critical applications was failing its [health checks](https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes) intermittently, requiring an automatic restart. We sent in engineers and AI agents to investigate the issue. The hypotheses generally amounted to claims that the app was running some rare resource intensive activity, pointing to its final log messages as the culprits. We applied performance fixes, tweaks, the whole bucket. It kept failing. We turned to performing a complete root cause analysis - what *exactly* was this process doing in its final moments? We dove into Node.js internals to find out.\n\n## Node event loop\n\n[Node.js](https://nodejs.org/en) runs programs in a continuous loop of “events” to implement asynchronous execution. Its behaviour is [well documented](https://nodejs.org/learn/asynchronous-work/event-loop-timers-and-nexttick), so I will be brief. Programs are broken into segments of continuous execution, also known as “callbacks” or “events”. Segments start by being created by another segment, and stop by handing off their result to a segment that is waiting on it. Node programs start with a single entry segment and terminate gracefully when there are no more to run.\n\nWhen we investigated our failing containers, we saw plain indicators of event loop issues: CPU was pegged at 100% on only one core, no network traffic, and no log activity. There was likely a part of our code that was hogging the event loop with a long-running single segment, or a large number of “[microtask](https://nodejs.org/api/globals.html#queuemicrotaskcallback)” events that get processed on the same phase. We set out to locate the code responsible.\n\n## Detection by tracing\n\nTo determine where the bug was, we needed a tool that automatically captured what was running in our application. We were unable to reproduce the issue locally, and we had no way to predict when it would happen on a production container. We needed an approach that was hands-off and production ready.\n\nOur first approach was an event loop block detector in the code of the application itself. We were inspired by Ashby’s blog post [Detecting Node Event Loop Blockers](https://www.ashbyhq.com/blog/engineering/detecting-event-loop-blockers). It tackles this challenge by combining the event loop with generally-available software tracing products, emitting visible spans whenever the loop was blocked. We re-implemented their strategy and deployed it, and eagerly waited for results to come in.\n\nHowever, our reports blamed only a small number of inefficient functions that could not possibly account for the instability we saw. These blocks would last less than 5 seconds on average, not the minutes needed to trigger a container restart. We fixed them despite our doubts, but our doubts were confirmed when that did not fix our problem.\n\nWe then had an aha moment - what if the problematic code never yielded back to the event loop? With tracing-based detection, code would need to complete its segment so the tracer could finish its span and report it. If it was a single segment that looped infinitely, it would never self-report. We decided to approach the problem from an entirely different angle.\n\n## Detection by crash reports\n\nOur new idea was to utilize [Node’s built-in debugger](https://nodejs.org/api/debugger.html) tool to create a CPU profile of the Node process in its final moments. The debugger can attach to any running Node process, even one that is stuck in an infinite loop. We were interested in its `profile`\n\ncommand, which samples the program’s running stack and records which functions were in the stack at each sample.\n\nWe added a new container [entrypoint](https://docs.docker.com/reference/dockerfile/#entrypoint) to manage process signals and call a profiler script. The entrypoint watched for termination or interrupt [signals](https://man7.org/linux/man-pages/man7/signal.7.html) from our container platform. At that point in each container’s lifecycle, its health checks were already failing, and our load balancer already removed it from the target pool. It was at the right moment when the looping code was sure to be running, and everything else was quiet.\n\n```\n# entrypoint.sh\nnode main.mjs &\nNODE_PID=\"$!\"\nterminate() {\n  /crash-report.sh \"$NODE_PID\"\n  kill -TERM \"$NODE_PID\" 2>/dev/null\n}\ntrap terminate TERM INT\n```\n\nThe profiler script was an [expect](https://linux.die.net/man/1/expect)-based wrapper around the Node debugger to start and stop the profiler. The profile only had to run for 5 seconds. Once done, the script uploaded the results to an AWS S3 bucket and the entrypoint killed the main process.\n\n```\n# crash-report.sh\nexpect <<EOF\n  set timeout 30\n  spawn node inspect -p $NODE_PID\n  expect \"debug>\"\n  send \"profile\\r\"\n  expect \"debug>\"\n  sleep 5\n  send \"profileEnd\\r\"\n  expect \"debug>\"\n  send \"profiles\\[0\\].save()\\r\"\n  expect \"debug>\"\n  send \".exit\\r\"\n  expect eof\nEOF\naws s3 cp node.cpuprofile \"s3://${CRASH_REPORTS_BUCKET}/$(date -u +%Y-%m-%dT%H-%M-%SZ)-$(hostname)/node.cpuprofile\"\n```\n\nOnce we deployed our new strategy, we saw results within the hour from newly crashed containers. We viewed the crash reports in [freely available flame graph visualization tools](https://www.speedscope.app/), and quickly located the problem code. The final root cause after all of this investigation? Calling `indexOf(str, N)`\n\non an empty string.\n\n## Future resilience\n\nWe kept the crash-reporting entrypoint in production since it caused no performance loss. Rather than running it 24/7 with [Node’s —prof option](https://nodejs.org/learn/getting-started/profiling), we ran the profiler only when a container was about to exit anyway due to being unhealthy. The only functions risked were the guilty code still blocking the event loop.\n\nOur detection tool has since found two more unique bugs that would have cost us days to solve. Instead, we spent one hour each checking the crash reports and fixing the reported functions.\n\nWe have open-sourced a demonstration version of our tool to Github that you can [try out today](https://github.com/Vantaca/sample-project-event-loop-blocks) with just Docker and curl.", "url": "https://wpnews.pro/news/debugging-stuck-node-js-processes", "canonical_source": "https://engineering.myhoai.com/posts/debugging-stuck-node-js-processes/", "published_at": "2026-07-29 00:00:00+00:00", "updated_at": "2026-07-30 23:08:20.568315+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["HOAi", "Node.js", "Ashby"], "alternates": {"html": "https://wpnews.pro/news/debugging-stuck-node-js-processes", "markdown": "https://wpnews.pro/news/debugging-stuck-node-js-processes.md", "text": "https://wpnews.pro/news/debugging-stuck-node-js-processes.txt", "jsonld": "https://wpnews.pro/news/debugging-stuck-node-js-processes.jsonld"}}