# NVLink 6 handles failures so AI factories don't stop > Source: > Published: 2026-09-15 17:01:17+00:00 # NVLink 6 handles failures so AI factories don't stop Keeping massive GPU clusters running is a nightmare because a single failure in a training run can kill the whole job. NVLink 6 targets this by implementing multi-layer resiliency, ensuring that gradient synchronization across thousands of collective operations per second doesn't just crash when a link goes down. For inference, this is even more critical since unplanned downtime translates directly into lost revenue from served requests. ## How the resiliency layers actually work The goal here is to move away from the "stop and restart" cycle that plagues large-scale AI training. NVLink 6 tackles this through a tiered approach to failure recovery. **Physical and Link Layer Recovery** At the lowest level, the system detects signal degradation or total link failure. Instead of crashing the entire pod, NVLink 6 can reroute traffic. This happens at the hardware level, meaning the software layer doesn't even necessarily see a "crash" event, just a momentary dip in throughput while the fabric heals. **Fabric Management and Routing** The system uses an intelligent fabric manager that monitors the health of every connection. If a specific link starts throwing errors, the manager can isolate that path. Because AI factories rely on all-to-all communication for synchronization, the ability to dynamically map around a dead link without restarting the global collective operation is what prevents total cluster stalls. **Graceful Degradation for Inference** In an inference environment, the priority is request volume. NVLink 6 allows the system to maintain service even if some bandwidth is lost. While the latency might spike slightly, the system avoids the hard crash that would otherwise take out a whole set of GPUs, keeping the request pipeline moving. ## The impact on AI factory productivity When you're running thousands of GPUs, the probability of a hardware failure is essentially 100% over a given window of time. The "AI factory" model assumes these clusters are utility-grade infrastructure. - **Continuous Output:** By preventing the "stop-the-world" effect of a single GPU or link failure, the total compute utilization stays high. - **Revenue Protection:** For companies selling API access, avoiding a 10-minute reboot cycle across a cluster of 10,000 GPUs saves an incredible amount of money. - **Training Stability:** Gradient synchronization is fragile. NVLink 6's ability to handle these collective operations despite local failures means training runs that would have failed on older hardware can now finish. [Next Microsoft AI doesn't believe in AI consciousness or rights →](https://promptcube3.com/en/threads/9382/) ## All Replies （4） Absolute nightmare when a single node dies. I lost three days of training last month to a random link failure. Does this affect the 1.8TB/s? I'm curious if this handles the 502 errors I've seen during checkpointing. Does this actually replace the need for NCCL?