# PyTorch 2.14: Fault-Tolerant Training and Apple Silicon Fixes > Source: > Published: 2026-09-17 16:06:50+00:00 PyTorch 2.14 landed on September 15 with 2,995 commits from 487 contributors, and the clearest signal yet that Meta is done treating production reliability as an afterthought. The marquee change — fault-tolerant distributed training as a first-class framework concept — addresses a problem that has cost teams millions in wasted GPU hours. That alone earns this release serious attention. ## Distributed Training Finally Gets Fault Tolerance If you’ve run large distributed training jobs on real hardware, you’ve hit this: one node flakes, the entire process group dies, and you restart from your last checkpoint. PyTorch 2.14 changes that with a rewritten NCCL backend (`nccl2`), ported from the torchcomms project that shipped in 2.13. The key capability is in-place process-group reconfiguration. When a rank fails, the group can now rebuild without a full teardown. Alongside that, nonblocking communicators and one-sided RMA windows give teams finer control over inter-rank communication during recovery. The Flight Recorder — previously NCCL-only — now works across any backend, which means you get distributed training observability regardless of your communication layer. Gloo gets fault tolerance support too. This is overdue. Production ML teams have been patching around this limitation for years with external orchestrators and brittle checkpoint-restart loops. Having it as a first-class `c10d` concept simplifies deployments considerably. ## Apple Silicon: The 8.5x Bug Is Fixed There was a routing bug in the MPS backend that sent single-token decode calls — the `[B, 1, K]` activation shape that dominates autoregressive inference — through a path 8.5 times slower than it needed to be on bf16 and fp16. PyTorch 2.14 fixes that routing and adds new GEMV kernels optimized for the vector-matrix products that autoregressive decoding constantly produces. Beyond the bug fix, Apple Silicon gets native linear algebra in this release. SVD, `eigh`, QR, and Cholesky now run on hand-written Metal kernels instead of MPSGraph, eliminating the per-op compilation overhead MPSGraph carried. The prefill attention kernel delivers 2–4x gains across head dimensions and sequence lengths. A five-part reduction rewrite and migration of common ops — conv3d, argmin, argmax, median, linspace, and more — off MPSGraph add up to a meaningfully better inference experience on Mac. PyTorch on Apple Silicon is not at CUDA parity yet. But this release closes one of the largest remaining gaps for LLM inference workloads, which is the audience that matters most for local Mac deployment. ## NVGEMM: Fused Kernels, No More Memory Re-reads NVGEMM is a new GPU math backend inside Inductor — CuTeDSL-generated CUTLASS kernels that fuse epilogues directly into matrix operations. Operations like bias additions, activations, and rescales that previously required reading the result back from memory now happen inside the same kernel launch. On Blackwell hardware (GB200), NVGEMM extends fusion to NVFP4 paths, so low-precision workloads get the same benefit. Fused kernels are cached to disk, which matters for production deployments that previously paid recompilation costs on every restart. Inductor’s `simple_overlap` reordering — which interleaves collective communication with independent computation — is now enabled by default. Distributed training jobs compiled through Inductor automatically get better GPU utilization without any configuration change. ## Compiler: Cleaner Dynamic Shapes and Better MoE Support Two compiler additions are worth flagging for teams doing heavy `torch.compile` work. The `@dynamic_spec` decorator provides a single, clean way to declare which tensor dimensions are dynamic at runtime. Previously, you needed separate mechanisms depending on whether you were going through `torch.compile`, `torch.export`, or `make_fx`. Now one spec covers all three — a meaningful quality-of-life improvement for teams fighting dynamic shape guard explosions. `torch.switch` is a new higher-order op for multi-way branching. Mixture-of-experts architectures previously required nested `torch.cond` chains that grew traced graphs and obscured intent. `torch.switch` replaces that with a clean index-based dispatch. `torch.while_loop` is also now CUDA-graph-capturable, removing a previous limitation for compiled loop-heavy models. ## Platform, Python 3.15, and TorchVision ABI Stability ROCm 7.14 wheels are now produced via the TheRock pip SDK. Intel XPU gains native graph capture. Inductor adds Rubin (sm_107) targeting ahead of broad NVIDIA Rubin availability. Python 3.15 and the free-threaded 3.15t build get wheels across all supported platforms — but `torch.compile` is not yet supported under Python 3.15 and will raise a RuntimeError if invoked. Note that 3.15 wheels are not on PyPI; use `download.pytorch.org` directly. TorchVision 0.29 ships ABI-stable against future PyTorch versions. That means torchvision 0.29 works with PyTorch 2.15, 2.16, and beyond — no more synchronized reinstalls every time you upgrade the core framework. ## How to Upgrade ``` # Standard upgrade pip install --upgrade torch torchvision # CUDA 13.0 specific pip install torch --index-url https://download.pytorch.org/whl/cu130 ``` Most features in this release carry an “API Unstable” designation in the release notes. That is not a reason to skip the upgrade — it is a reason to read the [official release blog](https://pytorch.org/blog/pytorch-2-14-release-blog/) and [full release notes on GitHub](https://github.com/pytorch/pytorch/releases/tag/v2.14.0) before rolling out to production. The fault tolerance and NVGEMM interfaces in particular are likely to evolve in 2.15. A [live Q&A webinar](https://pytorch.org/event/pytorch-2-14-release-live-qa/) with engineers from Meta and Reflection AI is happening today, September 17 — worth attending if you have questions about migrating specific workloads. The [official installation guide](https://pytorch.org/get-started/locally/) covers all platform-specific paths.