# PyTorch 2.15: Rewritten NCCL Backend and Flight Recorder

> Source: <https://byteiota.com/pytorch-2-15-nccl-flight-recorder/>
> Published: 2026-11-18 23:07:13+00:00

**Key takeaway:** PyTorch 2.15 makes nccl2 the default NCCL backend with one communicator per process group and adds a flight recorder that dumps per-rank diagnostics on hang, while requiring Python 3.11+ and dropping macOS 14 for MPS.

PyTorch 2.15 ships a rewritten NCCL backend as the new default and adds a flight recorder that dumps a full post-mortem when your distributed job hangs — not a generic timeout, an actual file that tells you which rank stalled and why. Released October 28 with 3,117 commits from 583 contributors, the release is primarily about making multi-GPU training debuggable at scale. If you have ever watched a 512-node job freeze and had nothing to show for it but a wall of “timeout expired,” this one is for you.

## The NCCL Backend Is Rewritten — and Now Default

PyTorch 2.14 introduced a rewritten NCCL backend called `nccl2` as an opt-in preview, ported from torchcomms. In 2.15 it becomes the default for every `nccl` process group. The central rule is strict: one process group gets exactly one communicator during initialization. This eliminates a class of hard-to-diagnose hangs caused by leaked communicator state crossing process group boundaries.

The legacy backend ships as a rollback — set `TORCH_DIST_USE_NCCL2=0` or use the `nccl-legacy` backend name — but treat it as temporary. The plan is to remove it in 2.16. If you have a workflow that depends on the old behavior, now is the time to identify it.

## The Flight Recorder: Finally, a Diagnostic You Can Read

The more practically useful addition is the flight recorder. When any collective hangs, it writes a post-mortem to disk. A watchdog publishes health data to a debug server, and a timeout on any single rank triggers diagnostic dumps from every rank in the job simultaneously. The result is a snapshot of what each rank was doing at the moment things stopped — which rank was stuck in AllReduce, how far ahead the others were, what calls preceded the stall.

This matters because the failure mode it addresses is expensive. Meta’s LLaMA3 pre-training run on 16,384 H100 GPUs saw 466 job interruptions over 54 days. At that scale, mean time to failure drops to around 1.8 hours. At $2–6 per GPU-hour, every hour of idle cluster time on a 16,000-GPU job costs tens of thousands of dollars. The flight recorder does not prevent failures, but it cuts the diagnostic cycle from “restart and hope” to “read the dump and fix it.” The [Register’s coverage](https://www.theregister.com/devops/2026/11/06/pytorch-215-tackles-the-one-bad-gpu-problem/5308227) of the release focused on exactly this problem.

## FSDP2 Gets Precision Overrides and Full CUDA Graph Capture

FSDP2 picks up two notable improvements. First, sharded and unsharded gradient dtypes are now separate, so precision-sensitive parameters — MoE router gates, norm weights — can stay in fp32 while the rest of the model trains in bf16. Second, a full gradient-accumulation step is now capturable in a single CUDA graph, which reduces kernel launch overhead for long accumulation runs. See the [full release notes on GitHub](https://github.com/pytorch/pytorch/releases/tag/v2.15.0) for the complete FSDP2 changelog.

FSDP2 is the trajectory for large-model training in PyTorch. If you are still on FSDP v1, 2.15 adds more reasons to migrate.

## Hardware Updates Across the Board

The release adds FP8 support on Apple Silicon’s MPS backend: `float8_e4m3fn` tensors with scaled matrix multiplication (tensorwise and rowwise). A new `grouped_mm` decode fast path delivers 1.32–1.34x MoE decode throughput on an M3 Ultra. If you do local fine-tuning on Apple hardware, this is the release that makes FP8 worth trying.

On NVIDIA Blackwell (H200, B100, B200), NVFP4 LLM decoding gets faster through improved kernel overlap in NVGEMM. AMD’s gfx950 gains FlyDSL, a new kernel DSL that sits alongside Triton and CuTeDSL for GEMM and scaled matrix operations. Hugging Face has a [detailed MPS guide](https://huggingface.co/blog/mps-pytorch-2-15) if you are on Apple Silicon.

## Breaking Changes Before You Upgrade

Three breaking changes to know:

- **Python 3.10 is dropped.** PyTorch 2.15 requires Python 3.11 or newer.
- **macOS 14 is dropped for MPS.** Apple Silicon acceleration requires macOS 15.
- **Distributed checkpoints are not backward-compatible.** Checkpoints saved with 2.15 record tensor strides in metadata and cannot be loaded by 2.14 or earlier.

TorchScript is also one step closer to gone. Loading a TorchScript archive through `torch.load(..., weights_only=False)` now emits a `FutureWarning`. The API is not removed yet — 2.15 still patches `torch.jit` bugs — but the migration path is `torch.export` and the warnings are not going away.

## Upgrade Now If You Run Multi-GPU Training

The NCCL rewrite and flight recorder together address a pain point that every distributed-training team has accepted as an unavoidable cost of doing business. The upgrade path is straightforward:

```
pip install --upgrade torch
```

Read the [PyTorch 2.15 release blog](https://pytorch.org/blog/pytorch-2-15-release-blog/) for the full deprecations list before deploying to production. If you use TorchScript archives, start the `torch.export` migration now — do not wait for 2.16 to force it.
