Keep Distributed Training Running with Automated Workload Recovery

NVIDIA released NVIDIA Recovery, an open-source, workload-level service within its NVIDIA DSX OS software portfolio that automates recovery of distributed AI training jobs across a cluster around the clock. NVIDIA Recovery detects interruptions using application logs, GPU metrics and scheduler signals, attributes failures to implicated resources, and coordinates isolation and checkpoint restarts on healthy capacity, integrating with the Slurm workload manager. NVIDIA says the automation addresses the manual burden of log analysis, fault isolation and checkpoint restarts that leaves affected GPUs idle and erodes training goodput as job counts and sizes grow.

As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery essential. A stalled training rank can trigger an NVIDIA Collective Communications Library NCCL timeout, a GPU node can stop responding to the scheduler, or a straggler can slow the entire job. Each disruption is manageable on its own, but the cumulative impact and recovery overhead determine how efficiently the system converts wall-clock time into productive training progress or goodput https://cloud.google.com/blog/products/ai-machine-learning/goodput-metric-as-measure-of-ml-productivity . Manual recovery becomes unsustainable when it must cover every training job across a cluster, around the clock, at scale. Each interruption demands log analysis, fault isolation, and a restart from a saved checkpoint, while affected GPUs remain idle during recovery. As the number and size of training jobs grow, overlapping failures increase the engineering burden and wasted GPU-hours. Without continuous coverage, those losses can accumulate, increasing overall compute costs and extending time to train. NVIDIA Recovery https://github.com/dsx-ai-factory/NVIDIA-Recovery , part of NVIDIA DSX OS https://www.nvidia.com/en-us/data-center/products/dsx/os/ , the modular software portfolio with the NVIDIA DSX AI Factory Platform https://www.nvidia.com/en-us/data-center/products/dsx/ , is an open-source, workload-level service that automates recovery across a cluster’s training jobs, around the clock. It detects interruptions using application logs, GPU metrics, and scheduler signals, uses failure attribution to identify implicated resources, and coordinates isolation and checkpoint restarts from checkpoints on healthy capacity. By automating this recovery loop across workloads, it handles concurrent interruptions, reduces manual intervention, and helps sustain training goodput. This post explains how the recovery loop works, how NVIDIA Recovery integrates with Slurm https://slurm.schedmd.com/overview.html , and how to deploy it on your own cluster. Workload recovery is a distinct AI factory concern Our previous NVIDIA DGX Cloud resilience post https://developer.nvidia.com/blog/ensuring-reliable-model-training-on-nvidia-dgx-cloud/ , GTC talk https://www.nvidia.com/en-us/on-demand/session/gtc25-s73410/ and NVIDIA Mission Control 2.5 post https://developer.nvidia.com/blog/how-nvidia-mission-control-2-5-brings-autonomous-recovery-to-kubernetes/ describe a layered approach to keeping training productive. NVIDIA Recovery builds on that architecture by connecting infrastructure remediation to workload recovery. Break-fix automation identifies faulty nodes, removes them from service, and routes hardware for remediation. Framework-level libraries such as NVRx https://github.com/NVIDIA/nvidia-resiliency-ext provide recovery, hang detection, and straggler mitigation within individual training jobs, but each user must configure and manage these capabilities for their own workloads. Cluster workload managers such as Slurm manage resource allocation, queuing, and container execution. Workload recovery is a distinct coordination problem: restoring productive training requires attributing failures to their causes, deciding whether and where affected jobs should restart, and acting consistently across workloads that share infrastructure. Addressing this problem requires a cluster-level control loop that integrates infrastructure remediation, framework-level recovery, and cluster workload management, with operational metrics that track recovery latency, recovery success rate, and time to restore productive training. Multi-level NVIDIA health and resilience stack As shown in Figure 1, below, NVIDIA Recovery provides the dedicated workload recovery layer within the DSX OS Stack https://docs.nvidia.com/dsx . Node health monitoring and isolation depend on the cluster’s orchestration and workload management stack. Kubernetes deployments use NVSentinel https://docs.nvidia.com/nvsentinel/getting-started/overview/ to monitor GPU health metrics and XID https://docs.nvidia.com/deploy/xid-errors/latest/index.html events and automatically cordon and drain faulty nodes. Many Slurm sites check node health with custom prolog and epilog scripts. The slurm-operator from the Slinky https://www.nvidia.com/en-us/software/slinky/ project natively supports and injects these same scripts when running Slurm on Kubernetes. Within individual training jobs, NVRx provides a library for in-job recovery and hang detection. NVIDIA Recovery coordinates recovery across workloads at the cluster level , integrating with these capabilities to manage job restarts, placement, and node exclusion. These layers operate independently without hard dependencies, allowing you to adopt each modular component or plug in your own platform component to integrate. For example, if your cluster already relies on NVSentinel or NVIDIA Health Validation, you can integrate NVIDIA Recovery with little or no changes to your existing health workflows. For environments already running NVRx, NVIDIA Recovery fills the gap that in-job restart cannot reach by managing the cross-workload lifecycle and node exclusion. When paired with NVIDIA Attribution, it receives rich cross-layer diagnostic verdicts that enrich the persistent audit trail. Cost of the gap At cluster scale, rare per-GPU failures compound. Even if each GPU operates without failure on a given day with 99.9% probability, a job spanning 10,000 GPUs completes that day uninterrupted with a probability of only 0.999 <sup 10,000</sup ≈ 0.005%. This explains why, in an 11-month study of 24,000 A100 GPUs at Meta https://arxiv.org/abs/2410.21680 , hardware failures interrupted fewer than 1% of jobs while those jobs accounted for 19% of GPU runtime: the largest, longest-running jobs are the most exposed. NVIDIA Recovery automates recovery across a cluster’s training workloads, and is useful for both small and large distributed runs. Across many smaller jobs, interruptions accumulate, increasing the operational burden. In large jobs, each interruption affects more GPUs, amplifying the compute cost of recovery. Automated recovery addresses both challenges. Our production observations underscore the need for such efficient, automated recovery that scales across cluster workloads: from July 9 to Oct. 9, NVIDIA Recovery monitoring on one training cluster covered nearly 200 jobs per day and recorded approximately 8,550 anomaly detections, with mean detection and recovery-action latencies of 69 and 94 seconds, respectively. This unattended, cluster-wide response reduces dependence on individual engineers and per-user recovery arrangements as interruptions accumulate. Recovery-action latency ends at the automated action; scheduling and checkpoint restoration determine when productive training resumes. The automated recovery loop NVIDIA Recovery coordinates recovery across the cluster’s training workloads. Four stages organize the loop, with actions determined by the interruption and configured policy: 1. Detect : The Anomaly Detector combines application logs and scheduler signals to identify crashes, hangs, and stragglers. It recognizes PyTorch and NCCL failures, CUDA errors, and interconnect failure patterns. Missing progress signals or recognized communication-hang patterns indicate potential hangs; sustained throughput degradation relative to a recent baseline indicates potential stragglers. 2. Attribute : NVIDIA Recovery uses the scheduler signals and application logs gathered during detection to classify the interruption and guide recovery actions. Optional integration with NVIDIA Attribution adds deeper diagnosis to distinguish hardware faults, system-software issues, and application bugs. This evidence informs node exclusion and enriches the audit trail. Operators can also integrate their own attribution service and adopt NVIDIA Recovery independently. 3. Isolate: The Anomaly Remediator applies the recovery recommendation, excluding implicated nodes from subsequent restart attempts or stopping the workload when further retries would be unproductive. It tracks implicated nodes across attempts to avoid recurring failures. In addition, a safety net watchdog service allows normal scheduling to proceed if a recovery decision takes too long or outright fails, preventing workloads from waiting indefinitely. 4. Restart or Stop : The scheduler integration acts on NVIDIA Recovery’s recommendation through the Advisor API to restart the workload or stop unproductive retries. On restart, the application resumes from its last saved checkpoint using its existing restart logic. If the job fails to save a new checkpoint across a configurable number of attempts, further retries can be stopped. The audit trail records restart and stop decisions, with allocation restart, restored training progress, and training completion tracked as distinct events. NVIDIA Recovery integration with workload schedulers NVIDIA Recovery runs as a control-plane service and communicates with the cluster’s scheduler integration through an HTTP or gRPC API. It supports Slurm, with Kubernetes support on the roadmap. Recovery recommendations follow a common API contract, which a cluster-side controller translates into actions for the workload manager. This keeps NVIDIA Recovery independent of scheduler internals and avoids changes to application code. For Slurm, the reference Advisor Controller is available for easy deployment as a Helm chart. It supports two modes: 1 In-cluster mode where the controller runs within the cluster; 2 SSH mode where the control plane connects to Slurm remotely, without adding an in-cluster service. This provides an option when deploying a new cluster component requires a change window or security review. NVIDIA Recovery records each recovery recommendation before the controller acts, creating an audit trail of intended actions. If the controller restarts during recovery, it checks the current scheduler state before proceeding, a safeguard against repeating actions already completed. Running it: Monitor mode and the audit trail New deployments start in monitor mode by default. In this mode, NVIDIA Recovery publishes advice and records what it would have done, and it takes no scheduler actions. Review that advice, compare the decisions with what your own engineers would have done, and switch to action mode per cluster once you are satisfied with the signal-to-noise ratio. controller: environment: prod acknowledgeSchedulerActions: false watchdog: enabled: true mode: observe acknowledgeSchedulerActions: false environmentPattern: prod IBM Research took this path on their Slurm cluster. They ran in monitor mode first, compared the advice with their own manual recovery decisions, and enabled autonomous action after validating the signal. Verda, a neocloud provider, deployed on Victoria Logs rather than Loki and collaborated on a VictoriaLogs connector for the repository, jump starting the OSS partner program. NVIDIA Recovery writes each action to a persistent, queryable audit trail. Each entry captures the matched detection pattern, the action taken kill, requeue, release, or node exclusion , the timestamp, the workload identifier, and the NVIDIA Attribution verdict when one was returned. Policy-engine actions and watchdog actions land in the same trail, so the record has no gaps. The Recovery Console is the operator dashboard. It shows live workload state across scheduler types, surfaces anomaly flags inline, and gives you a per-workload timeline of every detection event and recovery action. NVIDIA Recovery is available in the dsx-ai-factory/NVIDIA-Recovery https://github.com/dsx-ai-factory/NVIDIA-Recovery GitHub repo under the Apache 2.0 license. Getting a first workload under recovery takes three steps: 1. Install the chart. NVIDIA Recovery deploys from a single Helm umbrella chart. A bundled MySQL sub-chart ships with it so you can run the quickstart locally without provisioning a database or a physical GPU. For production, point the chart at a managed MySQL service. The operator deployment guide in the repository covers both paths and the current version requirements for Kubernetes, Helm, and Slurm. 2. Validate your signals. Point the chart at your existing Loki or Victoria Logs endpoint and your Prometheus scrape target, then run the log ingestion configurator before your first workload. It tests your signals against the NVIDIA Recovery contract, maps label names that differ from the defaults, and surfaces mismatches before they become silent failures. 3. Opt a workload in. Add one directive to your sbatch submission script: SBATCH --comment='{"nvidia-recovery":{"environment":"prod","mode":"observe"}}' From there, NVIDIA Recovery monitors the workload through application logs and scheduler signals. In monitor mode, it publishes recovery recommendations without taking action. After validating those recommendations in the Recovery Console, enable autonomous recovery by setting mode to action and acknowledgeSchedulerActions to true in the Helm values. Additional resources cover the rest: - The dsx-ai-factory/nvidia-recovery GitHub repo, for the source and the issue tracker. - The quickstart guide, which runs without a physical GPU. - The operator deployment guide, for in-cluster and SSH-based Slurm deployments. - The architecture reference, for how the control plane fits together. NVIDIA Attribution https://github.com/dsx-ai-factory/attribution is the recommended companion deployment. It gives NVIDIA Recovery a deeper diagnostic verdict and a richer audit trail; it is optional, and it ships alongside NVIDIA Recovery. What Comes Next This Version recovers Slurm jobs, including Slurm on Kubernetes through slurm-operator. Two capabilities come next. Recovery for Slurm job arrays gives the hyperparameter sweeps many teams run as arrays the same protection as single jobs: each failed task is recovered on its own, and Recovery stops when failures look systemic. Native Kubernetes recovery starts with JobSet and Kubeflow PyTorchJob, using the open-source Workload Map https://github.com/dsx-ai-factory/workload-map project to support each workload type. Beyond those, the roadmap https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/ROADMAP.md covers sharper failure classification, a clearer record of every recovery decision in the dashboard and the API so you can feed it into your own observability tools, and the work toward a stable release. Upvote and comment on roadmap issues to shape what comes first. IBM Research has deployed NVIDIA Recovery for training workloads and is working with the team on refining failure patterns from them. Verda, the first NVIDIA Cloud Partner to deploy it, runs NVIDIA Recovery with VictoriaLogs and has submitted its first pull requests. As more adopters contribute, their changes go through the same review as NVIDIA’s own. NVIDIA Recovery is part of DSX OS, NVIDIA’s open-source software layer for AI factory operations. It is licensed under Apache 2.0. Get started NVIDIA Recovery automates the recovery cycle that manual processes leave to an engineer at 3 a.m. Clone the repository, run the configurator against your telemetry stack, and opt a workload in. The first interruption will show you what the loop looks like end to end. - The dsx-ai-factory/nvidia-recovery https://github.com/dsx-ai-factory/NVIDIA-Recovery GitHub repo, for the source and the issue tracker - The quickstart guide, which runs without a physical GPU - The operator deployment guide, for in-cluster and SSH-based Slurm deployments - The architecture reference, for how the control plane fits together Acknowledgments Abhishek Jha, Alex Kondratenko, Amy You, Andrew McHarg, Ashwin Poojary, Ashwini Awatiger, Bob Schatz, Collin Neale, Dave Yao, Deepak Narayanan, Dilpreet Singh, Divyansh Jain, Dong H. Ahn, Dorrin Poorkay, Evan Briones, Gargi Prasad, Haim Elisha, Hamzeh Doostie, Harmandeep Singh, Harri Lin, Himanshu Soni, Ian Gibbs, Jason Sewall, Kumar Anik, Maer Rodrigues de Melo, Megan Chao, Mike Houston, Orlando Gentil, Oscar Bejarano Ardila, Pavan Kumar Gondhi, Prateek Gera, Rajan Singh, Rakshit Sanadhya, Rajasi Purandare, Ripple Patel, Rick Izzo, Ruoxi Zhang, Satish Pasumarthi, Shelby Thomas, Stefania Alborghetti, Tony Wang, Vincent Pun, and Yugal Jain contributed to this post.