{"slug": "pytorch-2-14-release-blog", "title": "PyTorch 2.14 Release Blog", "summary": "PyTorch 2.14, released by the PyTorch team, introduces NVGEMM for CuTeDSL-generated CUTLASS kernels, a new nccl2 backend for distributed training, fault tolerance as a first-class concept, native linear algebra on Apple Silicon, and experimental torch.compile support for complex-valued tensors. The release includes 2,995 commits from 487 contributors since PyTorch 2.13, with a Q&A webinar scheduled for September 17, 2026.", "body_md": "### Featured projects\n\nWe are excited to announce the release of PyTorch® 2.14 ([release notes](https://github.com/pytorch/pytorch/releases/tag/v2.14.0))!\n\nThe PyTorch 2.14 release features the following changes:\n\n**NVGEMM brings CuTeDSL-generated CUTLASS kernels to Inductor,** with epilogue fusion, scaled and NVFP4 GEMM, and grouped-reduction epilogues autotuned alongside Triton and ATen**A new nccl2 backend for PyTorch Distributed,** ported from torchcomms, implementing the full collective contract with nonblocking communicators and eager communicator splitting**Fault tolerance becomes a first-class c10d concept,** with in-place process-group reconfiguration, one-sided RMA windows, and a Flight Recorder that works for any backend rather than only NCCL**Apple Silicon gains native linear algebra,** including Jacobi-kernel SVD, eigh, QR and Cholesky, alongside a five-part reduction rewrite and a further MPSGraph to Metal kernel migration**torch.switch generalizes torch.cond to multi-way branching,** and torch.while_loop can now be captured in a CUDA graph**Declarative dynamic shapes via @dynamic_spec,** shared across torch.compile, torch.export and make_fx**Broader platform support**: ROCm 7.14 wheels are produced from the TheRock pip SDK, Intel XPU adds native graph capture, and Inductor targets Rubin (sm_107)**Experimental torch.compile support for complex-valued tensors:** Opt-in support decomposes supported complex operations into real and imaginary computations, enabling compiler backends to optimize more complex-number workloads.\n\nThis release is composed of 2,995 commits from 487 contributors since PyTorch 2.13. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.14. More information about how to get started with the PyTorch 2-series can be found at our [Getting Started](https://pytorch.org/get-started/locally/) page.\n\nBring any questions you may have about this release to our Q&A Webinar. It will be on Thursday, September 17, 2026 and will feature Andrey Talman (Meta), Natalia Gimelshein (Meta), Joe Spisak (Reflection AI), and Chris Gottbrath (Gottbrath Tech, moderator) who will share an overview of the 2.14 release and provide answers to community questions about PyTorch and the new capabilities in this release. [Register today](https://pytorch.org/event/pytorch-2-14-release-live-qa/).\n\nConnect with the global PyTorch community at the upcoming [PyTorch Conference North America](https://hubs.la/Q04vXWRN0) October 20-21, 2026, in San Jose, CA. Explore what’s new with PyTorch framework with sessions spanning compiler and runtime work, distributed communication, device portability, release engineering, CI, observability, accelerator integration, contributor infrastructure, and much more. PyTorch Conference is the place for engineers, researchers and maintainers solving real problems in training, inference, kernels, applications and responsible AI to convene.\n\nThroughout the 2.x series, PyTorch has been evolving from a research-first framework into a unified, hardware-agnostic platform for production training and inference at scale. [PyTorch 2.12](https://pytorch.org/blog/pytorch-2-12-release-blog/) added a device-agnostic `torch.accelerator.Graph`\n\nAPI and Microscaling quantization export support. [PyTorch 2.13](https://pytorch.org/blog/pytorch-2-13-release-blog/) landed FlexAttention on Apple Silicon, gave Inductor a CuTeDSL code path, and introduced torchcomms for large-cluster training.\n\nPyTorch 2.14 builds directly on those threads. The CuTeDSL path matures into NVGEMM, a full GEMM backend with epilogue fusion and low-precision support. torchcomms lands in-tree as the nccl2 backend, with fault-tolerant reconfiguration and one-sided RMA windows raising fault tolerance from a backend detail to a c10d concept. Apple Silicon moves from attention kernels to native linear algebra, and dynamic shapes become declarative through a spec that travels across compile, export and tracing.\n\nPyTorch 2.14 delivers meaningful improvements across performance, reliability, and hardware support. The release introduces NVGEMM, a new GPU math backend that automatically selects the fastest kernel for matrix operations — including support for lower-precision formats that reduce memory use during training and inference. For teams training across many machines, a redesigned distributed communication backend (nccl2) offers better scalability, while new fault-tolerance features allow training jobs to recover from node failures without restarting from scratch.\n\nApple Silicon users benefit from native linear-algebra routines (SVD, QR, Cholesky, and more) and a broad migration to hand-tuned Metal kernels that cut overhead on Mac GPUs. On the compiler side, new control-flow primitives (torch.switch, torch.while_loop) give model authors more flexibility when writing dynamic logic that still compiles efficiently, and a new @dynamic_spec decorator provides a single, clean way to declare which tensor dimensions can change at runtime — simplifying workflows across compilation, export, and tracing.\n\nPlatform support expands to AMD ROCm 7.14, Intel XPU native graph capture, and NVIDIA’s next-generation Rubin architecture. Under the hood, the compiler now overlaps communication with computation by default, batches small GPU kernels more intelligently, and reduces per-call overhead — all of which add up to faster end-to-end model execution without any code changes from users.\n\n## Performance Improvements\n\n### Large MPS Op Migration to Native Metal\n\nContinuing the migration begun in 2.13, a further set of MPS operators moved off Apple’s MPSGraph framework onto hand-written Metal compute kernels, including `index_add`\n\n, `index_select`\n\n, `argmin`\n\n, `argmax`\n\n, `conv3d`\n\n, `median`\n\n, `nanmedian`\n\n, `linspace`\n\n, `arange`\n\n, `nan_to_num`\n\n, `log_sigmoid`\n\n, `sigmoid_backward`\n\n, `mish`\n\nand GLU.\n\nReductions received a dedicated five-part rewrite covering full reductions, inner-dimension reductions, strided and batched outer reductions, small-dimension and narrow kernels, and the `argmax`\n\nand `argmin`\n\nsplit-K paths. The final part migrates `min`\n\nand `max`\n\noff MPSGraph. Skipping input up-casts and using vec4 loads removes work the MPSGraph path could not avoid.\n\nThe native Metal path eliminates MPSGraph’s per-op compilation cost and gives PyTorch direct control over thread dispatch and memory access patterns, reducing kernel launch latency across common training and inference workloads on Apple Silicon.\n\nAPI Unstable\n\n(PR [#191101](https://github.com/pytorch/pytorch/pull/191101), [#191097](https://github.com/pytorch/pytorch/pull/191097), [#191098](https://github.com/pytorch/pytorch/pull/191098), [#191099](https://github.com/pytorch/pytorch/pull/191099) and [#191100](https://github.com/pytorch/pytorch/pull/191100) by Irakli Salia, Hugging Face, [#187109](https://github.com/pytorch/pytorch/pull/187109) and [#188802](https://github.com/pytorch/pytorch/pull/188802) by Nikita Shulga, Thinking Machines Lab)\n\n### MPS Memory and Copy Paths\n\nLong-running decode workloads grew the MPS caching allocator’s reserved footprint faster than necessary. The allocator now buckets large allocations to bound reserved memory and uses placement heaps to reduce fragmentation.\n\nHost and device transfers also got shorter paths. CPU to MPS copies blit directly from pinned buffers with event-deferred reclaim, contiguous same-dtype copies use a compute kernel instead of a graph, elementwise ops are vectorized on inner-contiguous sliced views, and `cat`\n\ngains a vectorized contiguous fast path for any dimension.\n\nAPI Unstable\n\n(PR [#187441](https://github.com/pytorch/pytorch/pull/187441) and [#190438](https://github.com/pytorch/pytorch/pull/190438) by Irakli Salia, Hugging Face, [#189512](https://github.com/pytorch/pytorch/pull/189512) and [#188613](https://github.com/pytorch/pytorch/pull/188613) by Nikita Shulga, Thinking Machines Lab, [#188483](https://github.com/pytorch/pytorch/pull/188483) and [#188200](https://github.com/pytorch/pytorch/pull/188200) by Joona Havukainen, Apple)\n\n### F.linear Decode Path on MPS\n\nSingle-token decode passes a `[B, 1, K]`\n\nactivation to `F.linear`\n\n, a shape that was falling off the fast path on MPS and costing an 8.5x slowdown on bf16 and fp16 according to the fix. The sequence-length-1 case is now routed correctly, and new GEMV kernels back the vector-matrix shapes that dominate autoregressive decoding. Together, the routing fix and new GEMV kernels close one of the largest remaining performance gaps between MPS and CUDA for autoregressive workloads.\n\nAPI Unstable\n\n(PR [#189855](https://github.com/pytorch/pytorch/pull/189855) by Giovanni Versiglioni, Apple, [#186927](https://github.com/pytorch/pytorch/pull/186927) by Irakli Salia, Hugging Face)\n\n### Compute and Communication Overlap On By Default in Inductor\n\nInductor’s `simple_overlap`\n\nreordering, which interleaves collectives with independent compute so communication is not left on the critical path, is now enabled by default rather than opt-in. By enabling overlap by default, distributed training workloads compiled through Inductor automatically benefit from better GPU utilization without any configuration change.\n\nAPI Unstable\n\n(PR [#184240](https://github.com/pytorch/pytorch/pull/184240), #[184235](https://github.com/pytorch/pytorch/pull/184235)by Ivan Kobzarev, Meta)\n\n### reorder_for_locality Opt-In for Training Graphs\n\n`reorder_for_locality`\n\n, Inductor’s post-grad locality reordering pass, can now be opted into on training graphs via the new reorder_for_locality_in_training config (default off), where before it only ran on inference. This extends locality optimization to training workloads, giving users a tuning knob to improve cache behavior in training graphs without affecting default behavior.\n\nAPI Unstable\n\n(PR [#186643](https://github.com/pytorch/pytorch/pull/186643) by @reger-men)\n\n### Combo Kernels and Reductions in Inductor\n\nCombo kernels batch many small kernels into one launch, but a single very large reduction in the batch would previously shape the whole kernel. Large reductions are now split out of combo partitioning, combo reductions get dynamic `RBLOCK`\n\nscaling, and sub-kernel bodies are emitted as non-inlined device functions to keep register pressure down. For each combo kernel bodies are shared, and split-reduction heuristics are tuned for GB200. The net effect is fewer kernel launches with tighter resource usage, closing a gap where one oversized reduction could penalize an entire fused batch.\n\nAPI Unstable\n\n(PR [#186668](https://github.com/pytorch/pytorch/pull/186668), [#186957](https://github.com/pytorch/pytorch/pull/186957) and [#190689](https://github.com/pytorch/pytorch/pull/190689) by Karthick Panner Selvam, Meta, [#184323](https://github.com/pytorch/pytorch/pull/184323) by Jason Ansel, Meta, [#188579](https://github.com/pytorch/pytorch/pull/188579) by Liqiang Lu, Nvidia)\n\n### Dynamo Per-Call Overhead\n\nFor models with many small compiled regions, fixed per-call cost matters more than graph quality. This release trims that cost in several places. `compile_wrapper`\n\navoids `DispatchKeySet`\n\npybind churn on every call, `torch._dynamo.disable`\n\ngets a cheaper path, and the pregraph profiler marker is gated on an active profiler instead of always being emitted. Guard creation is skipped for unused function inputs, and `invoke_subgraph`\n\nreuse lookup is faster for pytree arguments such as dataclasses and namedtuples. These micro-optimizations collectively lower the tax of entering compiled code, making `torch.compile`\n\nmore practical for real-world models that mix many small compiled regions with eager execution.\n\nAPI Unstable\n\n(PR [#190390](https://github.com/pytorch/pytorch/pull/190390), [#190392](https://github.com/pytorch/pytorch/pull/190392) and [#190623](https://github.com/pytorch/pytorch/pull/190623) by William Wen, Meta, [#187782](https://github.com/pytorch/pytorch/pull/187782) and [#191817](https://github.com/pytorch/pytorch/pull/191817) by Aditya Sanjeev)\n\n### Eager Dispatch and CPU Kernels\n\nSeveral eager-mode hot paths got cheaper. PyObject dispatch is optimized, AOTAutograd avoids an expensive `Tensor.detach()`\n\nwhen saving graph-input views for backward, autograd stops copying `at::Tensor`\n\nwhen the profiler is off, `addmm`\n\navoids a device-to-device copy when C and D are distinct, and CPU `quantile`\n\nand `nanquantile`\n\nuse partial selection instead of a full sort.\n\nThese targeted fixes reduce the per-operation tax in eager mode, tightening the performance floor so that common operations like linear layers, autograd bookkeeping, and statistical aggregations don’t carry unnecessary overhead. They keep PyTorch’s default development experience fast without requiring users to reach for `torch.compile`\n\n.\n\nAPI Unstable\n\n(PR [#187949](https://github.com/pytorch/pytorch/pull/187949) [#189759](https://github.com/pytorch/pytorch/pull/189759) and [#189582](https://github.com/pytorch/pytorch/pull/189582) by Richard Zou, Meta, [#191706](https://github.com/pytorch/pytorch/pull/191706) by Animesh Jain, Meta, and [#188394](https://github.com/pytorch/pytorch/pull/188394) by Kimon N.)\n\n## Core Features\n\n### torch.linalg.polar and torch.linalg.matrix_sqrth\n\nTwo additions to `torch.linalg`\n\n. `torch.linalg.polar`\n\ncomputes the polar decomposition using cuSOLVER’s QDWH algorithm, with a backward formula on CPU, CUDA and MPS, which makes it usable inside training loops rather than only for analysis. `torch.linalg.matrix_sqrth`\n\ncomputes the matrix square root for symmetric and Hermitian positive-definite matrices, a case that previously required composing an eigendecomposition by hand.\n\nAPI Unstable\n\n(PR [#185837](https://github.com/pytorch/pytorch/pull/185837) by Simon Layton, Meta, [#189732](https://github.com/pytorch/pytorch/pull/189732) by Irakli Salia, Hugging Face, [#187987](https://github.com/pytorch/pytorch/pull/187987) by Colin Alberts, Cisco)\n\n### Autograd Extension Points\n\nThree additions give more control over how the autograd graph is built and inspected. `torch.autograd.graph.node_creation_hook`\n\nfires as each autograd node is created, letting tools attach metadata or register hooks at graph-construction time instead of reconstructing that context afterward — the motivating case is attributing backward-pass memory usage back to the forward region that produced it. `ctx.set_output_grad_dtype`\n\nlets a custom `autograd.Function`\n\ndeclare the dtype its output’s incoming gradient should be, independent of the output’s own storage dtype, for mixed-precision functions where the two don’t match. Double backward is now implemented for `cdist`\n\nand `pdist`\n\n, unblocking `create_graph=True`\n\nuses that previously failed outright — Hessians, gradient penalties, and Hessian-vector products through pairwise-distance computations.\n\nAPI Unstable\n\n(PR[ #189284](https://github.com/pytorch/pytorch/pull/189284) by Edward Yang, Meta,[ #189634](https://github.com/pytorch/pytorch/pull/189634) by @SongyuanZhao,[ #188901](https://github.com/pytorch/pytorch/pull/188901) by Colin Alberts, Cisco)\n\n### torch.switch Higher-Order Op\n\n`torch.cond`\n\nexpresses a two-way branch, so an n-way dispatch had to be written as nested conditionals, which grows the traced graph and obscures intent. `torch.switch`\n\nis a new higher-order op for multi-way branching on an index, with lifted-argument deduplication in Dynamo so shared operands are not re-lifted per branch. The result is a more expressive and efficient way to trace models with multi-way branching, particularly mixture-of-experts architectures where torch.cond nesting was a practical barrier.\n\nAPI Unstable\n\n(PR [#182902](https://github.com/pytorch/pytorch/pull/182902) and [#188374](https://github.com/pytorch/pytorch/pull/188374) by Thomas Ortner, IBM)\n\n### SDPA Fused Backends for Rank-3 Inputs\n\nScaled dot-product attention now dispatches to the fused CUDA backends for rank-3 inputs instead of falling back to the math path, so callers who pass unbatched or already-flattened tensors get the fused kernels without reshaping. The fix closes a common performance trap where missing or flattened batch dimensions silently bypassed the fast fused kernels.\n\nAPI Unstable\n\n(PR [#192271](https://github.com/pytorch/pytorch/pull/192271) by Driss Guessous, Meta)\n\n### Experimental torch.compile support for complex-valued tensors\n\ntorch.compile support for programs using complex-valued tensors. Supported complex operations are decomposed into real-valued computations that compiler backends can optimize. This enables more complex-number workloads including signal processing, scientific computing, and complex-valued neural networks to benefit from compiled execution. Not all complex operations are supported yet. See the [feature tracking issue](https://github.com/pytorch/pytorch/issues/194061), implementation\n\n(PRs [#167621](https://github.com/pytorch/pytorch/pull/167621) and [#169832](https://github.com/pytorch/pytorch/pull/169832), [#172813](https://github.com/pytorch/pytorch/pull/172813) by Hameer Abbasi, OpenTeams)\n\n### Smaller API Additions\n\nA number of smaller public additions landed this release.\n\n`torch.utils.checkpoint.checkpoint`\n\naccepts a decorator and curried calling convention in eager ([#189411](https://github.com/pytorch/pytorch/pull/189411)by Edward Yang, Meta).- Read-only DLPack export and\n`ReadOnlyTensorWrapper`\n\n, so consumers can be handed a tensor they must not mutate ([#188554](https://github.com/pytorch/pytorch/pull/188554)by Edward Yang, Meta). `Generator.philox_state`\n\nexposes Philox RNG state reservation to Python ([#191019](https://github.com/pytorch/pytorch/pull/191019)by Simon Layton, Meta).`torch.accelerator`\n\ngains`initial_seed`\n\n,`get_rng_state`\n\nand`get_rng_state_all`\n\n, closing part of the gap with the CUDA-specific RNG APIs ([#186597](https://github.com/pytorch/pytorch/pull/186597)by Guangye Yu, Intel).`LBFGS`\n\ngains`maximize`\n\nand is a no-op on an empty parameter group ([#187309](https://github.com/pytorch/pytorch/pull/187309)by Raj Vijay Firke, Red Hat).`linear_cross_entropy`\n\n, introduced in 2.13, supports probability targets on the chunked path ([#187053](https://github.com/pytorch/pytorch/pull/187053)by Pearu Peterson, Quansight).`c10::utils::get_env`\n\nand`set_env`\n\nare exposed to Python ([#191015](https://github.com/pytorch/pytorch/pull/191015)by Nikita Shulga, Thinking Machines Lab).\n\nAPI Unstable\n\n### Python 3.15 Support and Torchvision ABI Stability – Release Engineering\n\nPyTorch 2.14 adds binary support for Python 3.15, including the free-threaded (no-GIL) build, 3.15t across all platforms. Wheels are published for Linux on x86_64 and aarch64, Windows, and macOS on Apple silicon, spanning the CPU, CUDA, ROCm, and XPU builds. Also torchvision 0.29.0 ships matching 3.15 and 3.15t wheels for the same set of platforms.\n\nTorchVision is now ABI stable w.r.t. torch 2.14! This means that torchvision 0.29 will be compatible with future versions of torch: 2.15, 2.16, etc. You won’t need to install a new version of TorchVision when you upgrade torch. As a result, we might stop releasing TorchVision in sync with pytorch. But TorchVision is still actively maintained and developed: we’ll still be pushing releases, just not with the same cadence.\n\n### Installation\n\nPython 3.15 and 3.15t wheels are not published to PyPI — they are available to download only via download.pytorch.org, using any of the following commands:\n\n# CPU\n\npip3 install torch –index-url https://download.pytorch.org/whl/cpu\n\n# CUDA (substitute the CUDA version, e.g. cu126 / cu130)\n\npip3 install torch –index-url https://download.pytorch.org/whl/cu130\n\n# ROCm (substitute the ROCm version)\n\npip3 install torch –index-url https://download.pytorch.org/whl/rocm7.14\n\n# XPU\n\npip3 install torch –index-url https://download.pytorch.org/whl/xpu\n\nThe same commands install the free-threaded 3.15t build when run under a free-threaded interpreter.\n\nThe same applies to the free-threaded build. Install into a 3.15t interpreter and pip will resolve the cp315t wheels automatically.\n\n### torch.compile is not yet supported on Python 3.15\n\nPython 3.15 support in 2.14 is eager-only. Calling torch.compile under Python 3.15 raises a RuntimeError rather than falling back silently, so the limitation surfaces immediately rather than as a silent performance loss. If your workload depends on torch.compile, stay on Python 3.14 or earlier for now.\n\nDynamo support for 3.15 is in active development, with bytecode and symbolic-conversion handling already landed for the new interpreter. Progress is tracked in [pytorch/pytorch#184352](https://github.com/pytorch/pytorch/issues/184352).\n\n## Distributed Training\n\n### nccl2 Backend\n\ntorchcomms arrived in 2.13 as a communications backend integrated into PyTorch Distributed’s CI and device-mesh paths. In 2.14 the APIs landed in-tree with a new `nccl2`\n\nc10d backend, gated behind `USE_C10D_NCCL`\n\n, implementing the full `Work`\n\ncontract on top of a reusable `NcclApi`\n\nabstraction. The backend is eager only with new features such as one-sided windows, fault-tolerance, suspend and resume memory offload, and a greatly cleaned up implementation. A compatibility `nccl-lazy`\n\nwrapper builds per-peer P2P communicators on demand for workloads that require the old lazy initialization behavior.\n\nAPI Unstable\n\n(PR [#188582](https://github.com/pytorch/pytorch/pull/188582), [#189359](https://github.com/pytorch/pytorch/pull/189359), [#190943](https://github.com/pytorch/pytorch/pull/190943) and [#191272](https://github.com/pytorch/pytorch/pull/191272) by Tristan Rice, Meta, [#191528](https://github.com/pytorch/pytorch/pull/191528) and [#192105](https://github.com/pytorch/pytorch/pull/192105) by Tushar Jain, Meta)\n\n### Fault-Tolerant Collectives in c10d\n\nWhen a rank fails in a large job, the usual recovery is to tear down the process group and restart, which discards warm state across the whole cluster. `Backend`\n\nand `ProcessGroup`\n\nnow expose reconfiguration interfaces so a group can be rebuilt in place, with abort hooks and pre and post collective hooks wired through the same path. Gloo gains fault-tolerance support alongside nccl2, and the reconfigure APIs are documented.\n\nAPI Unstable\n\n(PR [#186298](https://github.com/pytorch/pytorch/pull/186298), [#186300](https://github.com/pytorch/pytorch/pull/186300), [#187381](https://github.com/pytorch/pytorch/pull/187381) and [#191384](https://github.com/pytorch/pytorch/pull/191384) by Tristan Rice, Meta)\n\n### One-Sided (RMA) Window APIs\n\n`Backend`\n\nand `ProcessGroup`\n\ngain one-sided window interfaces, giving remote-memory-access semantics alongside the existing two-sided collectives. One-sided operations let a rank read or write peer memory without the peer posting a matching call, which suits irregular access patterns such as embedding lookups, weight transfer and expert routing. This exposes the new ncclGet and ncclPut APIs via the nccl2 backend.\n\nAPI Unstable\n\n(PR [#186299](https://github.com/pytorch/pytorch/pull/186299) and [#189360](https://github.com/pytorch/pytorch/pull/189360) by Tristan Rice, Meta)\n\n### Backend-Agnostic Flight Recorder\n\nFlight Recorder, the collective trace buffer used to diagnose hangs and mismatched collectives, was tied to NCCL. `FlightRecorderHook`\n\nrecords through `ProcessGroup`\n\nhooks instead, so it works for any backend, and log serialization is portable through `DebugMode`\n\n. Debugging a Gloo or custom-backend job no longer means giving up the trace.\n\nAPI Unstable\n\n(PR [#189363](https://github.com/pytorch/pytorch/pull/189363) by Tristan Rice, Meta, [#185010](https://github.com/pytorch/pytorch/pull/185010) by Jason Ansel, Meta)\n\n### Pluggable Distributed Backends\n\nAdding a communications backend previously meant patching c10d. Backends can now register through Python entry points, backend strings are auto-qualified, and implementation accessors are exposed. We’ve brought the `PyProcessGroup`\n\ntrampoline to parity with C++, so an out-of-tree backend can implement the full collective surface from either C++ or Python, including `batch_isend_irecv`\n\n, the coalescing manager and the cleaned up *_single variants.\n\nAPI Unstable\n\n(PR [#187388](https://github.com/pytorch/pytorch/pull/187388), [#186853](https://github.com/pytorch/pytorch/pull/186853) and [#188570](https://github.com/pytorch/pytorch/pull/188570) by Tristan Rice, Meta, [#187494](https://github.com/pytorch/pytorch/pull/187494) by Kapil Sharma, Meta*)*\n\n### torch.distributed API improvements: set_timeout, per-op timeouts, get_backend_impl, hooks, weights_only=True, *_single\n\nWe’ve made a whole host of improvements to the torch.distributed API which allow for more control as well as cleaning up some inconsistencies. You can change a process group’s collective timeout after initialization — extending it around a slow checkpoint load, or shortening it so a wedged rank fails fast instead of hanging for the full default window — via the stable `torch.distributed.set_timeout`\n\nmethod as well as we now support per-operation collectives across all timeouts. We’ve made it easier to access advanced backend specific features via `torch.distributed.get_backend_impl`\n\nas well as add programmatic hooks to them to customize behavior and for observability. Object collectives now support the same `weights_only=True`\n\nmode as torch.load which can improve your training cluster security. We’ve also updated the names for all single tensor variants to share the `_single`\n\nsuffix such as in `all_to_all_single`\n\n.\n\nAPI Unstable\n\n(PR [#187387](https://github.com/pytorch/pytorch/pull/187387) and [#187693](https://github.com/pytorch/pytorch/pull/187693) by Tristan Rice, Meta)\n\n### DTensor Single-Dim Sharding Strategies\n\nDTensor’s sharding rules were historically written per operator against the whole device mesh, so each rule had to enumerate every combination of placements across all mesh dimensions — long to write and easy to get subtly wrong once the mesh had more than one dimension. This release continues moving op coverage to single-dim strategy functions, which describe how one mesh dimension shards an operator and leave the framework to expand that across the full mesh; matrix, math, and tensor ops are converted here, cutting direct `register_op_strategy`\n\nregistrations from 158 to 114. Convolution also gains sharding on the last spatial dimension when the windows tile it exactly — zero padding, dilation 1, stride equal to kernel width, and a dimension divisible by kernel width times mesh size — so those convolutions run locally in forward and backward instead of allgathering to replicate. The new rules are stricter than the ones they replace, so annotations that previously matched a base strategy by accident will now be reported as needing redistribution, and `Partial(\"product\")`\n\nis no longer produced by the migrated ops. This brings the number of total ops with registered sharding rules to 1239, up from 585 in Jan 2026.\n\nAPI Unstable\n\n(PR[ #186667](https://github.com/pytorch/pytorch/pull/186667),[ #179203](https://github.com/pytorch/pytorch/pull/179203),[ #186754](https://github.com/pytorch/pytorch/pull/186754), and[ #192147](https://github.com/pytorch/pytorch/pull/192147) by Anshul Sinha, Meta)\n\n### Symmetric Memory: NCCL backend fixes and allocation layout\n\nSymmetric memory’s NCCL backend had gaps that only surfaced at runtime: `barrier()`\n\nraised a not-implemented error, and the signal pad was never zeroed after allocation, so the signaling protocol had nothing reliable to build on. Both are fixed — barrier now reuses the existing CUDA barrier kernel, and the pad is zeroed at allocation time. The signal pad also moves to the front of every symmetric allocation across all three backends (CUDA, NCCL, NVSHMEM), so a recycled or resized allocation can’t inherit a polluted pad, and only the pad gets zeroed rather than the whole block — large allocations no longer pay a full-buffer memset on every `alloc()`\n\n. The timing of when the signal pad is zeroed can be tricky in multiple stream scenarios and users may want to do that in user space code to be sure that they have the sequence right. CUDA allocations now set the GPUDirect RDMA capable flag when the driver reports support. Signal-pad slots are still shared across process groups on the same allocation, so concurrent barriers from overlapping groups can interfere.\n\nAPI Unstable\n\n(PRs [#188051](https://github.com/pytorch/pytorch/pull/188051) by Kapil Sharma, Meta, [#189088](https://github.com/pytorch/pytorch/pull/189088) by Junjie Wang, NVIDIA, #[189941](https://github.com/pytorch/pytorch/pull/189941) by Natalia Gimelshein, Meta)\n\nSymmetric Memory: reaching NCCL symmetric kernels from ordinary collectives\n\nSymmetric memory based kernels became available on the NVLink domain in NCCL 2.27. These have been implemented in PyTorch’s symmetric memory in nightly builds since Jan 2026 but weren’t documented in a way that made it easy for users. Responding to feedback we improved the documentation in this release. The docs now cover both symmetric memory and ring or tree collectives — registering a `torch.cuda.MemPool`\n\nwith `register_mem_pool(pool, symm=True)`\n\n, or `set_backend(\"NCCL\")`\n\nplus rendezvous — along with the eligibility rules (`all_gather`\n\non any dtype; `all_reduce`\n\nand `reduce_scatter`\n\nonly for `SUM`\n\n/`AVG`\n\non float dtypes excluding `float64`\n\n) and the requirements: NCCL 2.27+, a single direct-NVLink domain, and `NCCL_WIN_ENABLE`\n\n. Users can verify via the `[Symmetric]`\n\ntag under `NCCL_DEBUG_SUBSYS=TUNING`\n\nor `ncclSymkDevKernel_*`\n\nnames in a profile.\n\nAPI Unstable*\n*(PR\n\n[#192515](https://github.com/pytorch/pytorch/pull/192515)by Kapil Sharma, Meta)\n\n**Symmetric Memory: one-sided** `get`\n\n**Symmetric Memory: one-sided**\n\n`get`\n\nReading data that lives on another rank has generally meant a collective: every rank participates and synchronizes, even when only one rank actually needs the data. Symmetric memory now exposes `get`\n\n, a one-sided copy that reads a peer’s symmetric allocation directly into a local destination tensor, with no participation from the peer and no group-wide synchronization. It works on the NVSHMEM, NCCL symmetric-memory, and CUDA backends — XPU and rocSHMEM aren’t supported yet — and requires the source to be a rendezvoused symmetric allocation matching the destination in dtype and element count. It’s the primitive underneath the in-progress one-sided DTensor work, and it’s directly usable for any algorithm where pulling from one peer beats an all-gather across all of them.\n\nAPI Unstable*\n*(PR\n\n[#182378](https://github.com/pytorch/pytorch/pull/182378)by Benjamin Brock, Intel)\n\n### TokenSwitch\n\nMixture-of-experts training spends much of its step time sending each token to the ranks holding its chosen experts and bringing the expert outputs back, and teams generally wire that up themselves against a vendor kernel library, backward pass included. `TokenSwitch`\n\nputs an interface around it — `create_routing()`\n\n, `dispatch()`\n\n, `combine()`\n\n— with `TokenSwitchNCCL`\n\nas the first backend, built on NCCL’s expert-parallel kernels. Called without an `out=`\n\nargument, dispatch and combine return differentiable tensors, so an MoE layer can be written as ordinary autograd-tracked Python; passing `out=`\n\nkeeps the buffer-reuse path but gives up autograd. It is early code: the module is private and needs a build with `USE_NCCL_EP=1`\n\nagainst the NCCL 2.30 pin, so today it is NVIDIA-only and out of reach of a stock wheel.\n\nAPI Unstable\n\n(PR [#178712](https://github.com/pytorch/pytorch/pull/178712) and [#181314](https://github.com/pytorch/pytorch/pull/181314) by Ke Wen, NVIDIA)\n\n### Compile-on-One-Rank\n\nEvery rank in a distributed job compiles the same model independently, so a single multi-minute compile is paid N times over before training starts. Compile-on-one-rank makes one compiled artifact reusable everywhere: `make_fx`\n\nno longer bakes the tracing rank’s device into factory and cast ops, and Inductor’s codegen and Triton launcher resolve the device at load time, so the generated source is identical across ranks and a kernel compiled on `cuda:0`\n\nloads and runs on `cuda:3`\n\n. `DeviceMesh.get_group()`\n\nlikewise fetches the group from the mesh in-graph instead of baking in a torchbind `ProcessGroup`\n\n, which used to leave the graph unserializable, and under the flag legacy collectives like `dist.all_reduce`\n\ntrace as their functional forms. torchtitan’s experimental [ graph_trainer](https://github.com/pytorch/torchtitan/tree/main/torchtitan/experiments/graph_trainer) shows the intended shape: a single process compiles ahead of time and writes one artifact that every rank loads at startup, no N-GPU job required to produce it — though the mode assumes a single accelerator device per program and refuses a graph that touches a second, and stays opt-in behind\n\n`torch.compiler.config.compile_on_one_rank`\n\n.API Unstable\n\n(PR[ #187869](https://github.com/pytorch/pytorch/pull/187869),[ #186892](https://github.com/pytorch/pytorch/pull/186892),[ #187870](https://github.com/pytorch/pytorch/pull/187870) and[ #188215](https://github.com/pytorch/pytorch/pull/188215) by Aaron Orenstein, University of Alberta)\n\n## Compilation and Export\n\n### Declarative Dynamic Shapes with @dynamic_spec\n\nTelling PyTorch which input dimensions vary has meant a different mechanism per entry point — a `dynamic_shapes`\n\ndict for `torch.export`\n\n, a coarse `dynamic=`\n\nflag for `torch.compile`\n\n, a global tracing mode for `make_fx`\n\n— and in each case the declaration sits at the call site, far from the model it describes. This release adds a `ShapesSpec`\n\nAPI under `torch.fx.experimental.dynamic_spec`\n\n: you name a dimension once (`ShapeVar(\"batch\", min=2, max=128)`\n\n), reuse it across inputs, build derived dims like `batch * 2`\n\n, and attach assumptions such as `batch % 2 == 0`\n\n, with all three entry points now taking it under the same `dynamic_shapes=`\n\nkeyword. A `@dynamic_spec`\n\ndecorator attaches that spec directly to a function or a module’s `forward`\n\n, so `torch.compile`\n\n, strict or non-strict `torch.export.export`\n\n, and `make_fx(tracing_mode=\"fake\")`\n\nall pick it up with nothing passed at the call site. Dimensions declared this way become unbacked symbols, so the compiler cannot quietly specialize on the batch size it happened to trace — the trade-off is that shape-dependent branching now surfaces as a data-dependent error rather than a guard and a recompile. The API is experimental and still moving: `make_fx`\n\nsupport is limited to `tracing_mode=\"fake\"`\n\n, and combining a spec with `prefer_deferred_runtime_asserts_over_guards=True`\n\n, or a decorator with a call-site `dynamic_shapes=`\n\nargument, raises an error.\n\nAPI Unstable**\n** (PR\n\n[#187639](https://github.com/pytorch/pytorch/pull/187639),\n\n[#185982](https://github.com/pytorch/pytorch/pull/185982),\n\n[#187602](https://github.com/pytorch/pytorch/pull/187602),\n\n[#186751](https://github.com/pytorch/pytorch/pull/186751)and\n\n[#187010](https://github.com/pytorch/pytorch/pull/187010)by Laith Sakka, Meta)\n\n### AOTInductor External Constants and Zero-Copy Weight Sharing\n\nServing several AOTInductor models that share the same weights used to mean every model container allocating and loading its own copy on the GPU. A new C API, `AOTInductorModelContainerCreateWithExternalConstants`\n\n, lets the caller hand in weight tensors at container creation; AOTI skips constant loading entirely and uses the caller’s memory instead, so one copy can back several models or be shared across processes via CUDA IPC. The caller retains ownership, which means those tensors have to outlive the container, and the API is available through the C ABI only, with no Python entry point yet. Existing code paths are untouched — the new constructor only engages when external constants are explicitly supplied. For a fleet serving many variants of the same base model, this turns per-model weight memory into a single shared allocation.\n\nAPI Unstable\n\n(PR[ #188643](https://github.com/pytorch/pytorch/pull/188643) by @iuliur-meta)\n\n### AOTInductor Compilation\n\nPackaging a model with `triton.autotune_at_compile_time=False`\n\nused to run the whole codegen twice: compile, run to collect kernel metadata, reset state, then recompile for packaging. That path now emits the JIT and AOTI wrapper bodies in a single codegen pass, running the JIT body once with real inputs to capture the Triton kernel configuration that the packaged source then embeds. `torch.cond`\n\nand `torch.while_loop`\n\nare supported on the new path. Separately, `cpp_wrapper`\n\ncan now emit explicit user streams and events, currently CUDA-only and not usable alongside CUDA graphs. The second codegen pass is gone from the lazy-autotune flow, and traced multi-stream code now survives into an AOTI package.\n\nAPI Unstable\n\n(PR[ #184735](https://github.com/pytorch/pytorch/pull/184735) and[ #184736](https://github.com/pytorch/pytorch/pull/184736) by @desertfire,[ #182971](https://github.com/pytorch/pytorch/pull/182971) by Brian Bustamante)\n\n### AOTInductor Constant Loading\n\nLoading a model’s weights copies them from host memory to the GPU, and a synchronous copy out of pageable memory forces a device-wide synchronization that stalls inference already running on other streams. `AOTInductorSetUsePinnedAsyncConstantsCopy`\n\nroutes constant loading and updates through pinned staging buffers instead, overlapping host copies with device transfers, with a companion call to size the buffers and the `AOTI_COPY_USE_PINNED_ASYNC`\n\nenvironment variable as a fallback. It’s off by default and has to be enabled before the model or container is created. The stretch between loading the `.so`\n\nand having a model ready to serve was previously silent, so setting `AOTI_LOG_LOADING`\n\nnow emits `[AOTI_LOAD]`\n\nmarkers with copy timing and pinned-pool diagnostics. For a server swapping models in and out under live traffic, that combination keeps the rest of the GPU busy during a load and makes a slow one diagnosable without a rebuild.\n\nAPI Unstable\n\n(PR[ #186258](https://github.com/pytorch/pytorch/pull/186258) and[ #186309](https://github.com/pytorch/pytorch/pull/186309) by @joshuuuasu)\n\n### Helion Backend Integration\n\nWriting a fast GPU kernel by hand means choosing tile sizes, loop order, and memory access patterns, then re-tuning all of it for every new shape and every new GPU. Helion raises that a level: you write the algorithm in Python, and Helion searches the schedule space and emits Triton for you. PyTorch 2.14 registers Helion as a third entry in the native DSL registry introduced in 2.13, so Helion-authored kernels can override ATen operations the same way Triton and CuTeDSL ones already can, controlled through `torch.backends.python_native.helion`\n\n. Registration requires the `helion`\n\npackage plus its lowering backend, and is unavailable on ROCm builds. No operators are routed through Helion in this release; this is the foundation for Helion-backed kernel overrides landing in subsequent ones.\n\nAPI Unstable\n\n(PR [#190636](https://github.com/pytorch/pytorch/pull/190636) by Karthick Panner Selvam, Meta)\n\n## Platform Features and Updates\n\n### CUDA\n\n#### NVGEMM, a CuTeDSL GEMM Backend for Inductor\n\nPyTorch 2.13 introduced the NVGEMM CuTeDSL backend for TorchInductor and in this release we are excited to expand support to epilogue fusion — the previous version could emit a standalone kernel, and whatever followed it (a bias add, an activation, a rescale) stayed in a separate kernel that re-read the result from memory. This release now utilizes NVGEMM, NVIDIA’s official `cutlass.operators`\n\nAPI, generating candidates that compete with Triton and ATen for `mm`\n\n, `addmm`\n\n, and `scaled_mm`\n\n. Its kernels fuse epilogues the way Triton templates do: addmm’s bias add, chained pointwise ops, and reductions over the GEMM result, including cases where the kernel returns both the reduced value and the full output matrix. Fusion now reaches the low-precision paths as well, so pointwise work after a scaled GEMM folds into the kernel, and NVFP4’s runtime global scale is applied inside the epilogue rather than as a separate multiply. Fused kernels are also cached to disk, so a process that recompiles them from scratch today reuses them instead. Enable it by adding `NVGEMM`\n\nto `max_autotune_gemm_backends`\n\nunder `max_autotune`\n\n; it needs `nvidia-cutlass-dsl`\n\n4.6.0, the NVFP4 paths require Blackwell, and epilogues the backend can’t express fall back to Triton so those cases keep their existing fusion. We are continuing to invest in this backend, with further improvements to autotuning time and performance currently being implemented.\n\nAPI Unstable\n\n(PRs[ #186183](https://github.com/pytorch/pytorch/pull/186183),[ #187013](https://github.com/pytorch/pytorch/pull/187013),[ #189772](https://github.com/pytorch/pytorch/pull/189772),[ #189774](https://github.com/pytorch/pytorch/pull/189774),[ #189805](https://github.com/pytorch/pytorch/pull/189805),[ #190808](https://github.com/pytorch/pytorch/pull/190808) and[ #190823](https://github.com/pytorch/pytorch/pull/190823) by Michael Lazos, Meta)\n\n#### CUDA Graph Lifecycle Hooks\n\nTools that want to watch CUDA graphs from the outside, like a profiler or a memory tracker, could only register per-graph hooks, which is no help when the graph was constructed by Inductor or NCCL rather than by the tool. This release adds module-level hooks that fire for every graph in the process (capture start and end, replay start and end, instantiate, destroy), along with per-graph replay start/end hooks and a capture-start hook that previously existed in neither form. `CUDAGraph`\n\nalso gains `register_destroy_callback`\n\nand `retain_object`\n\nfor tying cleanup or object lifetime to the graph’s own; pass `synchronize_before_release=True`\n\nif the callback frees memory the graph still references, since teardown is asynchronous and releasing under an in-flight replay is a use-after-free. Observability tooling can now follow a graph’s full lifecycle without the graph code carrying any knowledge of the tool, and registering nothing costs nothing.\n\nAPI Unstable\n\n(PR[ #190582](https://github.com/pytorch/pytorch/pull/190582) and[ #190602](https://github.com/pytorch/pytorch/pull/190602) by Natalia Gimelshein,[ #191299](https://github.com/pytorch/pytorch/pull/191299) and[ #192162](https://github.com/pytorch/pytorch/pull/192162) by @dolpm)\n\n#### Multiple Memory Pools in a Single CUDA Graph\n\nA `CUDAGraph`\n\ncapture previously bound to exactly one memory pool, which meant allocations that need to come from a separate pool — symmetric memory being the case that forced the issue — couldn’t participate in a captured region. A capture can now enter side pools with `torch.cuda.use_mem_pool()`\n\nand the graph retains all of them: `g.pool()`\n\nreturns the primary pool passed to `torch.cuda.graph()`\n\n, and `g.pools()`\n\nreturns the full set including any side pools entered during capture. This lets symmetric-memory buffers survive inside a CUDA graph, which unblocks graph capture for distributed workloads that allocate through a dedicated pool. One existing limitation carries over: `use_mem_pool`\n\nroutes allocations by thread ID, so calling `.backward()`\n\ninside it with multithreaded autograd won’t send the backward allocations to the pool — run autograd on the same thread if you need that.\n\nAPI Unstable\n\n(PR[ #187929](https://github.com/pytorch/pytorch/pull/187929) by @Aidyn-A)\n\n#### CUDA Graph Capture for torch.while_loop\n\nData-dependent loop counts have been one of the standard reasons a workload can’t be fully CUDA-graph captured, forcing a device-to-host copy to decide how many iterations to run and breaking the capture at exactly the point you’d rather keep it intact. `torch.while_loop`\n\ncan now be captured into a CUDA graph using CUDA’s `while`\n\nconditional nodes: the condition is evaluated on the parent stream before the node is added and re-evaluated at the end of each body execution, so a single captured graph runs a runtime-determined number of iterations on replay. This isn’t a throughput win on its own — the point is that the loop no longer forces you out of graph capture, so cases like a reduction over a variable-length index tensor, or a loss applied to a variable number of packed sequences, stay inside one graph. The usual `while_loop`\n\nconstraints still apply, including a fixed maximum trip count and tensor-only carried inputs.\n\nAPI Unstable\n\n(PR[ #186055](https://github.com/pytorch/pytorch/pull/186055) by Daniel Galvez, NVIDIA)\n\n#### Kernel Annotations for CUDA Graphs\n\n`torch.cuda.graph_annotations`\n\nmakes the kernel-annotation API used with CUDA graph capture public: `mark_kernels`\n\nlets you tag GPU work with a name so it shows up labeled when you later export a profiler trace, rather than as an anonymous kernel launch. Previously this only worked for forward-pass kernels captured lexically inside the `mark_kernels`\n\nscope — backward kernels, captured later when autograd actually runs, were never tagged. Backward kernels are now annotated automatically, using the `node_creation_hook`\n\nmechanism from above to attribute them back to whichever forward scope created them, including through double-backward and checkpoint recomputation. This is opt-out via `backward=False`\n\nfor callers that want to do their own backward attribution.\n\nAPI Unstable\n\n(PR[ #189417](https://github.com/pytorch/pytorch/pull/189417) and[ #191563](https://github.com/pytorch/pytorch/pull/191563) by Edward Yang, Meta)\n\n#### Post-Facto Memory Snapshot Annotations\n\nMemory snapshots already let you attach metadata to an allocation, but only at the moment it’s created — some information, like whether a tensor ended up retained by the autograd graph, is only known later, after it’s been packed into the backward tape. `torch.cuda.memory._annotate_tensor(tensor, metadata)`\n\nlets you attach metadata to a live allocation after the fact, recorded as a separate timestamped event so it doesn’t clobber whatever was recorded at alloc time. Views and offset tensors resolve to the allocation’s base address automatically, and the memory snapshot visualizer now shows these annotations alongside the allocation in the timeline.\n\nAPI Unstable\n\n(PR[ #190575](https://github.com/pytorch/pytorch/pull/190575) by Edward Yang, Meta)\n\n#### TunableOp on CUDA\n\nTunableOp profiles the available GEMM implementations for each input shape at runtime and caches the fastest, but on CUDA builds it previously had only one candidate to pick from — the cuBLAS default. It now registers cuBLASLt heuristic candidates too, with the candidate count set by `PYTORCH_TUNABLEOP_CUBLASLT_REQUESTED_ALGO_COUNT`\n\nor `torch.cuda.tunable.set_cublaslt_requested_algo_count()`\n\n. It’s a tool for rescuing shapes the stock heuristic handles badly rather than a general speedup: on H100 the mean is roughly flat, with individual shapes ranging from 0.66x to 1.57x. Offline tuning is also fixed for the case where a padded leading dimension equals one of m/n/k, which previously made `tune_gemm_in_file`\n\nsilently tune the wrong shape.\n\nAPI Unstable\n\n(PR[ #186270](https://github.com/pytorch/pytorch/pull/186270) by Grayson Derossi, NVIDIA and[ #189355](https://github.com/pytorch/pytorch/pull/189355) by Aditya Srichandan, AMD)\n\n#### cuBLASLt as a grouped GEMM backend\n\nGrouped GEMM drives MoE layers, where many differently-shaped matmuls are issued together. cuBLASLt joins CUTLASS and the fallback as a backend: default for fp16 on Blackwell with CUDA 13.2+ and Hopper with CUDA 13.3+, opt-in for bf16 on the same combinations via `torch.backends.cuda.matmul.prefer_cublaslt_grouped_gemm = True`\n\n. That split reflects the measurements — it wins on ragged MoE-style groups but generally trails the CUTLASS bf16 kernel on uniform ones. It works with `torch.compile`\n\nand CUDA Graphs; the one thing you must handle yourself is 16-byte alignment on matrices and leading dimensions.\n\nAPI Unstable\n\n(PR[ #177037](https://github.com/pytorch/pytorch/pull/177037) by Grayson Derossi, NVIDIA)\n\n### ROCm\n\n#### Grouped GEMM, CK Templates and Origami (ROCm GEMM)\n\nMixture-of-experts models on AMD GPUs used to miss out on Inductor’s Triton-compiled grouped GEMM, which was previously limited to NVIDIA SM90+ hardware — ROCm fell back to a slower for-loop over hipBLASLt/rocBLAS calls instead. This release brings that Triton lowering to ROCm, including the scaled (FP8) variant, and also lets Composable Kernel GEMM templates work under JIT cpp_wrapper compilation, not just ahead-of-time compilation. On top of that, Origami — AMD’s analytical tile-size selector — is now on by default for ROCm max-autotune, so Inductor can pick near-optimal GEMM configs from a latency model instead of paying for a full autotuning sweep. All of this is ROCm-specific and only affects max-autotune paths; NVIDIA users won’t see any change.\n\nAPI Unstable\n\n(PR[ #188600](https://github.com/pytorch/pytorch/pull/188600) and[ #188742](https://github.com/pytorch/pytorch/pull/188742) by Nichols A. Romero, AMD,[ #185505](https://github.com/pytorch/pytorch/pull/185505) by Bin Bao, Meta,[ #186644](https://github.com/pytorch/pytorch/pull/186644) by Umesh Chand, AMD)\n\n#### FlexAttention Tile Configs for RDNA3\n\nFlexAttention on AMD’s RDNA3 GPUs (Radeon workstation and consumer cards, not the MI-series datacenter line) was using tile sizes that weren’t tuned for the architecture, leaving performance on the table for short-to-medium sequence lengths. This release adds sequence-length-aware tile configs specifically profiled for RDNA3, so Inductor picks a tile size matched to the actual sequence length instead of one fixed default. The result is roughly 2-8x lower latency in the low-hundreds-of-tokens range, with no regression on very short sequences. If you’re running FlexAttention on other AMD or NVIDIA hardware, this change doesn’t apply to you.\n\nAPI Unstable\n\n(PR[ #177840](https://github.com/pytorch/pytorch/pull/177840) by Robert Esclapez, AMD)\n\n### MPS (Apple Silicon)\n\n#### Native Linear Algebra\n\nMPS linear algebra has historically leaned on Apple’s MPSGraph primitives or fallen back to CPU entirely for anything beyond the basics, which made mixed CPU/MPS round-trips a common source of slowdown in numerical code. This release replaces several of those gaps with native Metal kernels. SVD, `eigh`\n\n, and `lstsq`\n\nnow run natively via Jacobi-style kernels for float32 and complex64 (falling back to CPU for float64, since Metal has no double type, and for small matrices/batches where the GPU launch overhead isn’t worth it) — and this also lights up `matrix_rank`\n\n, `pinv`\n\n, `cond`\n\n, and norm computations that depend on them, though non-Hermitian `eig`\n\n/`eigvals`\n\nare left for later. Cholesky gets a faster panel-factorization algorithm with a `matmul2d`\n\n-based trailing update (roughly 1.2–2.8x faster depending on size) plus a correctness fix for complex dtypes, which previously had no dtype guard and could silently produce wrong results. `lu_factor`\n\nand `lu_solve`\n\nmove from Apple’s `MPSMatrixDecompositionLU`\n\nto hand-written Metal kernels, and the op level speedups can be substantial — the submitter measured >100x on small batched matrices and 2–9x on larger single matrices. Rounding things out, `geqrf`\n\nis added and `linalg_qr`\n\nis refactored to share the same device-agnostic code path CPU and CUDA use, and `matrix_exp`\n\nand `linalg.polar`\n\n(including its backward pass) are now available on MPS as well — though `matrix_exp`\n\nonly overtakes CPU above roughly 512×512.\n\nAPI Unstable\n\n(PR[ #185954](https://github.com/pytorch/pytorch/pull/185954) by Darko SImonovski,[ #187022](https://github.com/pytorch/pytorch/pull/187022) and[ #191836](https://github.com/pytorch/pytorch/pull/191836) by Irakli Salia, Hugging Face,[ #189192](https://github.com/pytorch/pytorch/pull/189192) by Kurt Mohler, OpenTeams, [ #187038](https://github.com/pytorch/pytorch/pull/187038),[ #189200](https://github.com/pytorch/pytorch/pull/189200),[ #188954](https://github.com/pytorch/pytorch/pull/188954),[ #189701](https://github.com/pytorch/pytorch/pull/189701), and[ #189732](https://github.com/pytorch/pytorch/pull/189732) by Irakli Salia, Hugging Face)\n\n#### FlexAttention Improvements\n\nBuilding on FlexAttention’s arrival on MPS in 2.13, this release rounds out several gaps that showed up once people started using it for real models. KV batch broadcasting lets key/value tensors be shared across the query batch instead of requiring an exact match — a prerequisite for paged attention, which needs this to serve multiple sequences against a shared KV cache. `flex_attention`\n\ncan now return the log-sum-exp and max-score auxiliary outputs alongside the main result, needed by anything that consumes attention weights beyond the output itself (custom losses, analysis, and eventually MPS-native backward support). And `score_mod`\n\n/`mask_mod`\n\nfunctions can now capture dynamic shape values (SymInts) directly, so masks built from runtime sizes — like a sequence-length-dependent cutoff — no longer force a recompile every time the shape changes under `torch.compile(dynamic=True)`\n\n. A follow-up optimization narrows those captured SymInts to 32-bit integers when the value fits, since 64-bit arithmetic in the kernel is measurably slower.\n\nAPI Unstable\n\n(PR[ #187722](https://github.com/pytorch/pytorch/pull/187722),[ #187768](https://github.com/pytorch/pytorch/pull/187768),[ #188362](https://github.com/pytorch/pytorch/pull/188362), and[ #188403](https://github.com/pytorch/pytorch/pull/188403) , [#188663](https://github.com/pytorch/pytorch/pull/188663) by Irakli Salia, Hugging Face)\n\n#### MPS Prefill Attention Acceleration\n\nApple’s Metal Performance Primitives (MPP), new in macOS 26.2, expose lower-level building blocks for attention-style workloads that weren’t previously available to Metal kernels. This release takes advantage of them with a second prefill attention kernel for MPS, porting the approach MLX uses on M5 chips but extending it to older Apple silicon generations as well, for fp16/bf16 inputs with head dims of 64, 96, 128, or 256 and query length greater than 8 (macOS 26.2+; other shapes and dtypes keep using the existing simdgroup-matrix kernel). The payoff is substantial — the author’s own benchmarks show roughly 2–4x speedups over the previous kernel across head dims and sequence lengths, with the largest gains at smaller head dims and longer sequences. For anyone running attention-heavy models on Apple silicon, this is a meaningful prefill speedup with no code changes required — MPS just picks the faster kernel automatically when the shape and dtype qualify. This kernel’s performance benefits are best confirmed on Apple M5 hardware, which is what the underlying per-lane data layout was validated against; behavior on earlier Apple silicon generations has not been independently verified for this release and may not reflect the numbers above.\n\nAPI Unstable\n\n(PR[ #182256](https://github.com/pytorch/pytorch/pull/182256) by Irakli Salia, Hugging Face)\n\n#### MPS acceleration for CTC Loss\n\n`ctc_loss`\n\n— the loss behind alignment-free sequence models like speech recognition and OCR — gets both forward and backward passes on MPS for the first time, closing a gap that previously forced Mac users to fall back to CPU for this one op. The implementation follows the same log-domain approach as the CUDA kernel, including correct handling of variable-length (padded) batches. Training CTC-based models end-to-end on Apple silicon no longer requires dropping into CPU for the loss computation.\n\nAPI Unstable\n\n(PR[ #187716](https://github.com/pytorch/pytorch/pull/187716) and[ #188187](https://github.com/pytorch/pytorch/pull/188187) by Kurt Mohler, OpenTeams)\n\n### XPU (Intel GPUs)\n\n#### Enhanced XPU Graph Performance\n\nReduced graph capture and replay overhead in XPU Graph, improving execution efficiency and delivering better performance for graph-based training and inference workloads on Intel® Arc™ B-Series and newer Intel GPUs.\n\nAPI Unstable\n\n(PR [#188874](https://github.com/pytorch/pytorch/pull/188874) by Jing Ma, Intel)\n\n#### MXFP8 and MXFP4 Support for scaled_mm\n\nAdded MXFP8 and MXFP4 support for scaled_mm, enabling early software readiness for next-generation Intel GPUs and helping developers prepare AI workloads for emerging low-precision computation formats.\n\nAPI Unstable\n\n(PR [#181726](https://github.com/pytorch/pytorch/pull/181726), [#181727](https://github.com/pytorch/pytorch/pull/181727), and [#187315](https://github.com/pytorch/pytorch/pull/187315) by Carson Wang, Intel)\n\n#### Symmetric Memory for Distributed AI Workloads\n\nEnabled the XPU Symmetric Memory backend for scale-up deployments, unlocking Async Tensor Parallelism (Async TP) on Intel GPUs and providing the foundation for more scalable distributed AI workloads.\n\nAPI Unstable\n\n(PR[#185102](https://github.com/pytorch/pytorch/pull/185102) by Cherry Zhang, Intel)\n\n#### Fine-grained Per-Process Intel GPU Memory Tracking\n\nAdded torch.xpu.list_gpu_processes(), enabling detailed tracking and reporting of Intel GPU memory usage on a per-process basis.\n\nAPI Unstable\n\n(PR [#185192](https://github.com/pytorch/pytorch/pull/185192) by Guangye Yu, Intel)\n\n#### Expanded WSL2 Support\n\nAdded support for Ubuntu 24.04 and Ubuntu 26.04 running under Windows Subsystem for Linux 2 (WSL2), making it easier for developers to build and run AI workloads on Intel GPUs from Windows environments.\n\n### C++ ABI\n\n#### Expanded torch::stable Surface\n\nC++ applications that use the subset of APIs defined in `torch::stable`\n\ncan rely on ABI compatibility across releases, and this release expands that surface. (C++ applications can also use the full API surface of `libtorch.so`\n\nif they’re willing to rebuild regularly or pin their libtorch version.) Following `torch::stable::Generator`\n\nin 2.13, this release adds `PyObject`\n\n-to-`torch::stable::Tensor`\n\nconversion and `Tensor::has_storage`\n\n, plus stable overloads for `bitwise_and`\n\n, `bitwise_or`\n\n, `left_shift`\n\n, `right_shift`\n\n, `permute`\n\n, `view_dtype`\n\n, `index_select`\n\n, `floor_divide`\n\n, and `is_pinned`\n\n. More utilities have also moved into the header-only `torch::headeronly`\n\n(including `fastAtomicAdd`\n\nand `isinf`\n\n/`isnan`\n\n), so extension authors can use them without linking against `libtorch`\n\nat all.\n\nC++ interface API Unstable while C interface is API Stable and ABI Stable.\n\n(PR[ #183323](https://github.com/pytorch/pytorch/pull/183323) by Paweł Gadziński, NVIDIA,[ #189877](https://github.com/pytorch/pytorch/pull/189877),[ #191973](https://github.com/pytorch/pytorch/pull/191973) and [#193604](https://github.com/pytorch/pytorch/pull/193604) by Jane Xu, Meta,[ #192083](https://github.com/pytorch/pytorch/pull/192083) and [#192097](https://github.com/pytorch/pytorch/pull/192097) by Chris Leonard, Red Hat)\n\n## Profiling and Debugging\n\n#### Memory Snapshots for Pinned CPU Memory\n\nMemory snapshots have covered device allocations for a while, but not the pinned (page-locked) host memory used to stage host-to-device transfers — so if that memory grew unexpectedly, there was no way to see it in the same tool you’d already reach for. Pinned buffers are also easy to lose track of: they’re typically allocated once and held for the life of a process, since CUDA graphs require a fixed host address for any captured copy. Passing `record_host=True`\n\nto `torch.cuda.memory._record_memory_history()`\n\nnow captures pinned allocations too, surfaced as new `host_segments`\n\nand `host_traces`\n\nkeys alongside the existing device data. This gives host and device memory a single, consistent view for tracking down leaks or unexpected growth. The `memory_viz`\n\nvisualizer does not yet render host data, and allocations made via a raw `cudaHostRegister`\n\ncall outside PyTorch’s allocator are not captured.\n\nAPI Unstable\n\n(PR[ #182407](https://github.com/pytorch/pytorch/pull/182407) by Edward Yang, Meta)\n\n## Deprecations and Backwards-Incompatible Changes\n\n**TorchScript deprecation warnings are now visible** and TorchScript is kept out of import paths. Deprecated`isIntegral`\n\noverloads are removed. See[#189914](https://github.com/pytorch/pytorch/pull/189914)and[#187115](https://github.com/pytorch/pytorch/pull/187115).- Python function events are excluded from profiler\n`key_averages()`\n\nby default, a visible change in profiler output. See[#188631](https://github.com/pytorch/pytorch/pull/188631). - In the profiler, the deprecated\n`use_cuda`\n\noption is removed and`with_modules`\n\nis deprecated; the pattern matcher,`BasicEvaluation`\n\n,`profiler_metrics`\n\nand`profiler_measure_per_kernel`\n\nare removed. See[#192543](https://github.com/pytorch/pytorch/pull/192543),[#192808](https://github.com/pytorch/pytorch/pull/192808),[#187362](https://github.com/pytorch/pytorch/pull/187362),[#187439](https://github.com/pytorch/pytorch/pull/187439)and[#187204](https://github.com/pytorch/pytorch/pull/187204). - The Dynamo TVM backend’s Relay path is removed after a\n`FutureWarning`\n\ndeprecation; use the relax frontend. See[#189639](https://github.com/pytorch/pytorch/pull/189639)and[#190766](https://github.com/pytorch/pytorch/pull/190766). - In distributed,\n`_set_pg_timeout`\n\ngives way to`torch.distributed.set_timeout`\n\n,`setSequenceNumberForGroup`\n\nbecomes a deprecated no-op, the control collectives implementation is removed, and the compile-on-one-rank`torch.distributed`\n\nalias gives way to`torch.compiler.config`\n\n. See[#187387](https://github.com/pytorch/pytorch/pull/187387),[#188611](https://github.com/pytorch/pytorch/pull/188611),[#188617](https://github.com/pytorch/pytorch/pull/188617)and[#187869](https://github.com/pytorch/pytorch/pull/187869). - CUDA green context\n`set`\n\nand`pop`\n\nare deprecated, and green contexts moved to the CUDA Python bindings. See[#188419](https://github.com/pytorch/pytorch/pull/188419)and[#185527](https://github.com/pytorch/pytorch/pull/185527). - Sparse tensors are validated for consistency when loaded with\n`weights_only`\n\n. See[#184750](https://github.com/pytorch/pytorch/pull/184750). - The\n`balanced`\n\naccuracy policy is removed from`linear_cross_entropy`\n\n. See[#188283](https://github.com/pytorch/pytorch/pull/188283).\n\n## Non-Feature Updates\n\nComponent |\n2.13 |\n2.14 |\n|---|---|---|\n| CUDA | 12.6, 13.0, 13.2 | 12.6, 13.0, 13.2 |\n| Default wheel | CUDA 13.0 | CUDA 13.0, unchanged |\n| ROCm | 7.1, 7.2 | 7.2, 7.14. 7.1 dropped; 7.14 via TheRock |\n| Python | 3.10 to 3.15, incl. 3.14t, 3.15t | unchanged |\n| C++ standard | C++20 | unchanged |\n\n- The build system migrated from setuptools to scikit-build-core, and Windows and macOS wheel builds are refactored into Python pipelines. See\n[#180247](https://github.com/pytorch/pytorch/pull/180247),[#184407](https://github.com/pytorch/pytorch/pull/184407)and[#187944](https://github.com/pytorch/pytorch/pull/187944). - ROCm 7.14 wheels are built from the TheRock pip SDK with RPATH-based library resolution, manywheels are repackaged with auditwheel to fix invalid ZIP64 on wheels over 4 GB, and tooling migrates from\n`rocm_smi`\n\nto`amd_smi`\n\n. See[#190276](https://github.com/pytorch/pytorch/pull/190276),[#189903](https://github.com/pytorch/pytorch/pull/189903)and[#190014](https://github.com/pytorch/pytorch/pull/190014). - cuDNN advances to 9.24 with conv engine 5 re-enabled, oneDNN to 3.12.3, and the XPU support package to 2026.1. See\n[#189483](https://github.com/pytorch/pytorch/pull/189483),[#188785](https://github.com/pytorch/pytorch/pull/188785)and[#189593](https://github.com/pytorch/pytorch/pull/189593). - C++20 remains the minimum standard, and header-guard enforcement completes. See\n[#178150](https://github.com/pytorch/pytorch/pull/178150). - New CI and platform coverage includes a native linux-riscv64 build image, a B200 benchmark workflow, a dedicated H100 fabric runner for P2P IPC tests, and Intel BMG client smoke tests. See\n[#190887](https://github.com/pytorch/pytorch/pull/190887),[#192659](https://github.com/pytorch/pytorch/pull/192659),[#191280](https://github.com/pytorch/pytorch/pull/191280)and[#187421](https://github.com/pytorch/pytorch/pull/187421). - Inductor targets Rubin (sm_107) with tuned vectorized elementwise kernels. See\n[#190654](https://github.com/pytorch/pytorch/pull/190654)and[#190546](https://github.com/pytorch/pytorch/pull/190546).", "url": "https://wpnews.pro/news/pytorch-2-14-release-blog", "canonical_source": "https://pytorch.org/blog/pytorch-2-14-release-blog/", "published_at": "2026-09-02 18:41:48+00:00", "updated_at": "2026-09-02 18:54:37.264305+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools"], "entities": ["PyTorch", "Meta", "Reflection AI", "Gottbrath Tech", "Andrey Talman", "Natalia Gimelshein", "Joe Spisak", "Chris Gottbrath"], "alternates": {"html": "https://wpnews.pro/news/pytorch-2-14-release-blog", "markdown": "https://wpnews.pro/news/pytorch-2-14-release-blog.md", "text": "https://wpnews.pro/news/pytorch-2-14-release-blog.txt", "jsonld": "https://wpnews.pro/news/pytorch-2-14-release-blog.jsonld"}}