{"slug": "pytorch-2-15-rewritten-nccl-backend-and-flight-recorder", "title": "PyTorch 2.15: Rewritten NCCL Backend and Flight Recorder", "summary": "PyTorch 2.15, released October 28 with 3,117 commits from 583 contributors, makes the rewritten NCCL backend (nccl2) the default for every nccl process group and adds a flight recorder that writes a post-mortem to disk when a distributed job hangs, identifying which rank stalled and why. The release also gives FSDP2 separate sharded and unsharded gradient dtypes and single-CUDA-graph capture of a full gradient-accumulation step, adds FP8 support on Apple Silicon's MPS backend with a grouped_mm decode fast path delivering 1.32–1.34x MoE decode throughput on an M3 Ultra, and drops Python 3.10 (requiring 3.11 or newer) and macOS 14 for MPS. The legacy NCCL backend remains as a rollback via TORCH_DIST_USE_NCCL2=0 or the nccl-legacy name but is slated for removal in 2.16, a change aimed at the failure economics Meta reported in its LLaMA3 pre-training run on 16,384 H100 GPUs, which saw 466 job interruptions over 54 days.", "body_md": "PyTorch 2.15 ships a rewritten NCCL backend as the new default and adds a flight recorder that dumps a full post-mortem when your distributed job hangs — not a generic timeout, an actual file that tells you which rank stalled and why. Released October 28 with 3,117 commits from 583 contributors, the release is primarily about making multi-GPU training debuggable at scale. If you have ever watched a 512-node job freeze and had nothing to show for it but a wall of “timeout expired,” this one is for you.\n\n## The NCCL Backend Is Rewritten — and Now Default\n\nPyTorch 2.14 introduced a rewritten NCCL backend called `nccl2` as an opt-in preview, ported from torchcomms. In 2.15 it becomes the default for every `nccl` process group. The central rule is strict: one process group gets exactly one communicator during initialization. This eliminates a class of hard-to-diagnose hangs caused by leaked communicator state crossing process group boundaries.\n\nThe legacy backend ships as a rollback — set `TORCH_DIST_USE_NCCL2=0` or use the `nccl-legacy` backend name — but treat it as temporary. The plan is to remove it in 2.16. If you have a workflow that depends on the old behavior, now is the time to identify it.\n\n## The Flight Recorder: Finally, a Diagnostic You Can Read\n\nThe more practically useful addition is the flight recorder. When any collective hangs, it writes a post-mortem to disk. A watchdog publishes health data to a debug server, and a timeout on any single rank triggers diagnostic dumps from every rank in the job simultaneously. The result is a snapshot of what each rank was doing at the moment things stopped — which rank was stuck in AllReduce, how far ahead the others were, what calls preceded the stall.\n\nThis matters because the failure mode it addresses is expensive. Meta’s LLaMA3 pre-training run on 16,384 H100 GPUs saw 466 job interruptions over 54 days. At that scale, mean time to failure drops to around 1.8 hours. At $2–6 per GPU-hour, every hour of idle cluster time on a 16,000-GPU job costs tens of thousands of dollars. The flight recorder does not prevent failures, but it cuts the diagnostic cycle from “restart and hope” to “read the dump and fix it.” The [Register’s coverage](https://www.theregister.com/devops/2026/11/06/pytorch-215-tackles-the-one-bad-gpu-problem/5308227) of the release focused on exactly this problem.\n\n## FSDP2 Gets Precision Overrides and Full CUDA Graph Capture\n\nFSDP2 picks up two notable improvements. First, sharded and unsharded gradient dtypes are now separate, so precision-sensitive parameters — MoE router gates, norm weights — can stay in fp32 while the rest of the model trains in bf16. Second, a full gradient-accumulation step is now capturable in a single CUDA graph, which reduces kernel launch overhead for long accumulation runs. See the [full release notes on GitHub](https://github.com/pytorch/pytorch/releases/tag/v2.15.0) for the complete FSDP2 changelog.\n\nFSDP2 is the trajectory for large-model training in PyTorch. If you are still on FSDP v1, 2.15 adds more reasons to migrate.\n\n## Hardware Updates Across the Board\n\nThe release adds FP8 support on Apple Silicon’s MPS backend: `float8_e4m3fn` tensors with scaled matrix multiplication (tensorwise and rowwise). A new `grouped_mm` decode fast path delivers 1.32–1.34x MoE decode throughput on an M3 Ultra. If you do local fine-tuning on Apple hardware, this is the release that makes FP8 worth trying.\n\nOn NVIDIA Blackwell (H200, B100, B200), NVFP4 LLM decoding gets faster through improved kernel overlap in NVGEMM. AMD’s gfx950 gains FlyDSL, a new kernel DSL that sits alongside Triton and CuTeDSL for GEMM and scaled matrix operations. Hugging Face has a [detailed MPS guide](https://huggingface.co/blog/mps-pytorch-2-15) if you are on Apple Silicon.\n\n## Breaking Changes Before You Upgrade\n\nThree breaking changes to know:\n\n- **Python 3.10 is dropped.** PyTorch 2.15 requires Python 3.11 or newer.\n- **macOS 14 is dropped for MPS.** Apple Silicon acceleration requires macOS 15.\n- **Distributed checkpoints are not backward-compatible.** Checkpoints saved with 2.15 record tensor strides in metadata and cannot be loaded by 2.14 or earlier.\n\nTorchScript is also one step closer to gone. Loading a TorchScript archive through `torch.load(..., weights_only=False)` now emits a `FutureWarning`. The API is not removed yet — 2.15 still patches `torch.jit` bugs — but the migration path is `torch.export` and the warnings are not going away.\n\n## Upgrade Now If You Run Multi-GPU Training\n\nThe NCCL rewrite and flight recorder together address a pain point that every distributed-training team has accepted as an unavoidable cost of doing business. The upgrade path is straightforward:\n\n```\npip install --upgrade torch\n```\n\nRead the [PyTorch 2.15 release blog](https://pytorch.org/blog/pytorch-2-15-release-blog/) for the full deprecations list before deploying to production. If you use TorchScript archives, start the `torch.export` migration now — do not wait for 2.16 to force it.", "url": "https://wpnews.pro/news/pytorch-2-15-rewritten-nccl-backend-and-flight-recorder", "canonical_source": "https://byteiota.com/pytorch-2-15-nccl-flight-recorder/", "published_at": "2026-11-18 23:07:13+00:00", "updated_at": "2026-11-18 23:14:38.518552+00:00", "lang": "en", "topics": ["machine-learning", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["PyTorch", "NCCL", "FSDP2", "Meta", "LLaMA3", "NVIDIA", "AMD", "Hugging Face"], "also_reported_by": [], "takeaway": "PyTorch 2.15 makes nccl2 the default NCCL backend with one communicator per process group and adds a flight recorder that dumps per-rank diagnostics on hang, while requiring Python 3.11+ and dropping macOS 14 for MPS.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/pytorch-2-15-rewritten-nccl-backend-and-flight-recorder", "markdown": "https://wpnews.pro/news/pytorch-2-15-rewritten-nccl-backend-and-flight-recorder.md", "text": "https://wpnews.pro/news/pytorch-2-15-rewritten-nccl-backend-and-flight-recorder.txt", "jsonld": "https://wpnews.pro/news/pytorch-2-15-rewritten-nccl-backend-and-flight-recorder.jsonld"}}