PyTorch 2.13 Release Blog

PyTorch 2.13 has been released with performance improvements including FlexAttention on Apple Silicon with up to 12x speedups, a CuTeDSL backend for Inductor, and fused nn.LinearCrossEntropyLoss reducing peak memory by up to 4x. The release also introduces a new torchcomms backend for distributed training and FSDP2 communication overlap, along with broader platform support and ExecuTorch integration into PyTorch Core.

Featured projects We are excited to announce the release of PyTorch® 2.13 release notes https://github.com/pytorch/pytorch/releases/tag/v2.13.0 The PyTorch 2.13 release features the following changes: - FlexAttention lands on Apple Silicon MPS , with up to ~12x speedup over SDPA on sparse patterns, and gains a deterministic backward path on CUDA for reproducible gradient computation - CuTeDSL “Native DSL” backend gives Inductor a second high-performance code path alongside Triton for key GPU operations, with faster compilation Prototype - nn.LinearCrossEntropyLoss combines final prediction and loss computation operations to cut peak GPU memory by up to 4x for large-vocabulary language model training - torchcomms, a new communications backend for PyTorch Distributed, improves fault tolerance, scalability, and debuggability for large-cluster training - FSDP2 now overlaps reduce-scatter and all-gather communications via a dedicated process group opt-in , increasing distributed training throughput - Python 3.15 wheel support for Torch on Linux via the pytorch repository index. This support includes builds compatible with free-threaded 3.15t. - Broader platform support : ROCm gains AOTriton 0.12b with native HIP CMake, Arm adds Armv9-A torch.compile targeting, and Intel XPU exposes new device telemetry APIs This release is composed of 3,328 commits from 526 contributors since PyTorch 2.12. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.13. More information about how to get started with the PyTorch 2-series can be found at our Getting Started https://pytorch.org/get-started/locally/ page. Have questions? Join us on July 22, 2026, at 11 a.m. PT for a live Q&A with panelists Alban Desmaison, Andrey Talman, Piotr Bialecki and moderator Chris Gottbrath. We will provide a brief overview of the release and answer your questions live. Register today. https://pytorch.org/event/pytorch-2-13-release-live-qa/ Throughout the 2.x series, PyTorch has been evolving from a research-first framework into a unified, hardware-agnostic platform for production training and inference at scale. PyTorch 2.11 https://pytorch.org/blog/pytorch-2-11-release-blog/ introduced differentiable collectives for distributed training and FlashAttention-4 on next-generation GPUs. PyTorch 2.12 https://pytorch.org/blog/pytorch-2-12-release-blog/ added a device-agnostic torch.accelerator.Graph API, up to 100× faster batched eigendecomposition, and Microscaling quantization export support. PyTorch 2.13 pushes performance further across platforms and scales: FlexAttention lands on Apple Silicon with up to 12× speedups, CuTeDSL brings CUTLASS-grade GEMM kernels to Inductor, and fused nn.LinearCrossEntropyLoss cuts peak memory up to 4x for large-vocabulary models. On the distributed side, a new torchcomms backend improves fault tolerance and debuggability at cluster scale, and FSDP2 unlocks communication overlap between all-gather and reduce-scatter for higher training throughput. This release also marks ExecuTorch’s integration https://pytorch.org/blog/executorch-becomes-part-of-pytorch-core/ into PyTorch Core, making on-device inference a first-class capability of the framework. Performance Improvements Deterministic Backward for FlexAttention Flash Backend This improvement focuses on correctness and debugging for the existing CUDA implementation of FlexAttention by making gradient computation reproducible. By default, the FlexAttention flash backend uses atomic operations in the backward pass for dQ accumulation, which introduces non-determinism — repeated runs on the same input can produce slightly different gradients. This makes debugging, regression testing, and reproducible research difficult. The new deterministic backward path compute dq write order replaces atomics with a pre-computed write ordering that guarantees bit-for-bit reproducible gradients without meaningful performance penalty. The measured end-to-end overhead on create block mask is well under 1% at longer sequence lengths e.g., +0.2% at S=32768 , making determinism effectively free for most production workloads. Users can opt in via the existing torch.use deterministic algorithms True setting with no additional code changes. API Unstable PR 174813 https://github.com/pytorch/pytorch/pull/174813 by Driss Guessous, Meta Large MPS Op Migration to Native Metal PyTorch’s MPS backend previously delegated most operations to Apple’s MPSGraph framework, which adds compilation and scheduling overhead per dispatch. For high-frequency, latency-sensitive operations this overhead can dominate execution time. This release migrates a broad set of operations to hand-written Metal compute kernels — copy/cast, uniform/normal/randint, comparisons, reductions sum/mean , cumsum/cumprod, sort multi-block and stable , embedding backward, and scatter/gather with bounds checking. The native Metal path eliminates MPSGraph’s per-op compilation cost and gives PyTorch direct control over thread dispatch and memory access patterns, reducing kernel launch latency across common training and inference workloads on Apple Silicon. API Unstable PR 184740 https://github.com/pytorch/pytorch/pull/184740 by Nikita Shulga, Meta, 185609 https://github.com/pytorch/pytorch/pull/185609 and 185119 https://github.com/pytorch/pytorch/pull/185119 by Irakli Salia, EPAM. FlexAttention on Apple Silicon MPS FlexAttention, PyTorch’s unified API for expressing custom attention patterns as plain Python functions compiled into fused kernels, is now available on Metal/MPS. The MPS implementation provides hand-written Metal kernels for both the sparse prefill and decode paths including GQA and captured buffers . So instead of writing a custom CUDA kernel for every attention variant you need, FlexAttention will write a 2-line Python function, and the compiler builds a fast kernel for you automatically. The benchmark numbers are impressive for sparse masks. On long, sparse attention patterns the speedups over SDPA are substantial — e.g. on a 1×8×32768×64 shape with a 256-element sliding window 0.8% density , FlexAttention runs in ~35 ms vs ~431 ms for SDPA ~12.3x ; a smaller 8192-length / 64-window case achieves ~4.15x . Dense patterns continue to favor SDPA, as expected. API Unstable PR 182552 https://github.com/pytorch/pytorch/pull/182552 , 186215 https://github.com/pytorch/pytorch/pull/186215 , and 181575 https://github.com/pytorch/pytorch/pull/181575 by Irakli Salia, EPAM Core Features nn.LinearCrossEntropyLoss In standard large-vocabulary training e.g., language models with 100K+ token vocabularies , computing the cross-entropy loss requires materializing the entire logits matrix over the vocabulary, which can consume tens of gigabytes of GPU memory. nn.LinearCrossEntropyLoss and the corresponding linear cross entropy functional fuses the final linear projection and cross-entropy computation into a single module that processes the vocabulary dimension in chunks, never materializing the full logits matrix. This reduces peak memory by up to ~4x for large-vocabulary workloads while maintaining numerical equivalence with the unfused path. The implementation supports label smoothing, weight tying, and z-loss regularization out of the box, and integrates with torch.compile for further optimization. As a drop-in replacement for separate nn.Linear + nn.CrossEntropyLoss, adoption requires no other code changes. API Unstable See 172446 https://github.com/pytorch/pytorch/pull/172446 , 172286 https://github.com/pytorch/pytorch/pull/172286 , and 185852 https://github.com/pytorch/pytorch/pull/185852 by Pearu Peterson, OpenTeams. Load Safetensors Directly with torch.load Safetensors has become a widely adopted format for distributing model weights used by Hugging Face, Stability AI, and others due to its memory-mapped loading and lack of arbitrary code execution risk. Previously, loading safetensors files required installing and importing a separate library. torch.load “foo.safetensors” now works natively, detecting the format automatically and returning tensors directly. This removes a dependency for common workflows and makes PyTorch a seamless drop-in for loading models distributed in safetensors format. API Unstable PR 170592 https://github.com/pytorch/pytorch/pull/170592 by Nikita Shulga, Meta Python 3.15 Binary Support – Release Engineering PyTorch wheels now support Python 3.15, including the experimental free-threaded 3.15t build. Important Note: Python 3.15 is currently in pre-release beta , with the final stable release scheduled for October 2026. Support is limited to torch wheels only torchvision is not built for 3.15 yet and to Linux builds on x86 64 and aarch64, covering the CPU, CUDA, ROCm, and XPU variants. torch.compile is not yet supported on Python 3.15. There are no Windows or macOS 3.15 wheels for this python version in this release. Python 3.15 and 3.15t wheels are not published to PyPI — they are available to download only via download.pytorch.org, using any of the following commands: CPU pip3 install torch –index-url https://download.pytorch.org/whl/cpu CUDA substitute the CUDA version, e.g. cu126 / cu130 pip3 install torch –index-url https://download.pytorch.org/whl/cu130 ROCm substitute the ROCm version pip3 install torch –index-url https://download.pytorch.org/whl/rocm7.2 XPU pip3 install torch –index-url https://download.pytorch.org/whl/xpu The same commands install the free-threaded 3.15t build when run under a free-threaded interpreter. API Unstable. For updates See tracker issue: 184352 https://github.com/pytorch/pytorch/issues/184352 PR 182954 https://github.com/pytorch/pytorch/pull/182954 by Nikita Shulga, PR 184600 https://github.com/pytorch/pytorch/pull/184600 and 186244 https://github.com/pytorch/pytorch/pull/186244 by Andrey Talman, Meta, 186017 https://github.com/pytorch/pytorch/pull/186017 by Rob Timpe, OpenTeams Distributed Training torchcomms Backend PyTorch Distributed has historically relied on c10d’s ProcessGroup abstraction for collective communications all-reduce, all-gather, etc. , which was designed primarily around NCCL. As distributed training grows more complex multi-dimensional parallelism, elastic scaling, heterogeneous interconnects , the original backend’s error handling and observability limitations become bottlenecks for operations teams. torchcomms is a new communications backend integrated into PyTorch Distributed’s CI and device-mesh paths, providing improved fault tolerance graceful timeout and partial-group recovery , better scalability across large clusters, and richer debuggability through structured logging and collective tracing. It serves as a modern alternative to the existing c10d backends while maintaining API compatibility. API Unstable PR 181662 https://github.com/pytorch/pytorch/pull/181662 by Tristan Rice, Meta, 178533 https://github.com/pytorch/pytorch/pull/178533 Pangiotis Kourdis, Intel, and 182057 https://github.com/pytorch/pytorch/pull/182057 by Kapil Sharma, Meta FSDP2 Separate Reduce-Scatter Group In fully-sharded data-parallel FSDP training, all-gather and reduce-scatter share a single NCCL communicator by default. Because NCCL serializes operations on the same communicator, these two collectives cannot overlap, leaving communication bandwidth underutilized. FSDPModule.set separate reduce scatter group enable=True gives reduce-scatter its own dedicated NCCL communicator, allowing it to progress concurrently with all-gather operations. This enables AG/RS overlap, improving training throughput for fully-sharded workloads without any changes to model code. API Unstable PR 186335 https://github.com/pytorch/pytorch/pull/186335 by Wei Feng, Meta Compilation and Export torch.compiler.set default backend When using custom or out-of-tree compiler backends e.g., for specialized hardware , users previously had to pass backend= explicitly to every torch.compile call, making code verbose and error-prone. torch.compiler.set default backend mirrors the pattern established by torch.set default dtype and torch.set default device, letting backend authors or infrastructure teams set a process-wide default once. All subsequent torch.compile calls automatically use that backend unless an explicit backend= argument overrides it. This simplifies adoption of custom backends across large codebases. API Unstable PR 178944 https://github.com/pytorch/pytorch/pull/178944 by Angela Yi, Meta Platform Features and Updates CUDA CuTeDSL “Native DSL” Backend for Inductor CuTeDSL is a Python-native domain-specific language built on NVIDIA’s CuTe CUDA Templates library, giving developers direct control over GPU tensor layouts, tiling strategies, and memory access patterns. PyTorch’s Inductor compiler can now use CuTeDSL as an alternative code-generation backend alongside Triton — specifically for matrix multiplication GEMM and normalization RMSNorm , two of the most performance-critical operations in transformer training. These Quack-derived kernel overrides produce higher-quality matrix-multiply code without requiring Triton for these workloads. Kernel compilation has also moved from the thread pool to a subprocess pool, eliminating Python’s GIL bottleneck and improving compile-time parallelism. API Unstable PR 181267 https://github.com/pytorch/pytorch/pull/181267 by Michael Lazos, 182108 https://github.com/pytorch/pytorch/pull/182108 by Simon Layton, and 186310 https://github.com/pytorch/pytorch/pull/186310 by Driss Guessous, Meta ROCm AOTriton 0.12b, Origami GEMM Selection, Native HIP CMake The ROCm stack gains three improvements: AOTriton advances to 0.12b with support for asymmetric head dimensions, deterministic algorithms, and new GPU targets gfx1100/gfx1151 promoted to stable, partial FlashAttention v3 on gfx950 . The Origami tool replaces brute-force autotuning with analytical GEMM-configuration selection, cutting tune time. The build system now uses CMake’s native HIP language support. API Unstable PR 184288 https://github.com/pytorch/pytorch/pull/184288 and 172512 https://github.com/pytorch/pytorch/pull/172512 by Xinya Zhang and Umesh Chand, AMD Arm Armv9-A Target Support for torch.compile torch.compile on AArch64 now recognizes Armv9-A CPUs e.g., Neoverse V2 used in AWS Graviton4 , propagating the correct target triple and feature set 128-bit and 256-bit SVE through Inductor codegen. No behavior change for x86. API Unstable PR 184555 https://github.com/pytorch/pytorch/pull/184555 by Zhibo Li, Arm XPU Intel GPUs Device Telemetry APIs New query APIs for Intel GPUs expose runtime device state: memory usage torch.xpu.device memory used , utilization, power draw, clock rate, and temperature, plus device-wide synchronization and hardware properties last-level cache size, integrated-GPU detection . API Unstable PR 183431 https://github.com/pytorch/pytorch/pull/183431 , 183429 https://github.com/pytorch/pytorch/pull/183429 , 183428 https://github.com/pytorch/pytorch/pull/183428 , and 183427 https://github.com/pytorch/pytorch/pull/183427 by Guangye Yu, Intel C++ ABI torch::stable::Generator Custom kernel authors can now retrieve an at::Generator through the stable C++ ABI previously always passed as null , enabling RNG support in ABI-stable extensions without recompilation against PyTorch internals. API Unstable PR 186423 https://github.com/pytorch/pytorch/pull/186423 and 183930 https://github.com/pytorch/pytorch/pull/183930 by Jane Xu, Meta and Chris Leonard, Red Hat Profiling and Debugging Experimental CUPTI Monitor Profiler PyTorch’s existing profiler collects GPU activity through CUPTI’s activity API, which requires synchronization points that can distort timing and contend with the GIL in multi-threaded workloads. The new experimental CUPTI monitor backend collects GPU metrics asynchronously completely off the GIL , eliminating profiling-induced overhead while reusing the existing CPU profiler path. The result is merged Chrome traces that accurately reflect real-world execution timing, with GPU-side user annotations automatically recreated for record function scopes. This makes it practical to profile production training loops without measurably perturbing their performance. API Unstable PR 186037 https://github.com/pytorch/pytorch/pull/186037 and 186295 https://github.com/pytorch/pytorch/pull/186295 by Natalia Gimelshein, Meta CUDAGraph.get graph data When debugging performance issues in CUDA graph captures, understanding the internal structure which kernels run, their dependencies, execution order previously required external tools or manual inspection. CUDAGraph.get graph data exposes the full graph topology programmatically: node types, kernel names, dependency edges, and IDs remapped to match CUPTI profiler output. This lets developers correlate captured graph structure with profiled kernel launches, making it straightforward to identify bottlenecks, unnecessary serialization, or suboptimal kernel fusion within a captured graph. API Unstable PR 183165 https://github.com/pytorch/pytorch/pull/183165 by Natalia Gimelshein, Meta Deprecations and Backwards-Incompatible Changes Named tensors removed. The deprecated named-tensor feature Tensor.names and associated APIs has been hard-removed to cut overhead and code bloat. See 173895 https://github.com/pytorch/pytorch/pull/173895 . Distributed collectives renamed to a single scheme. all gather into tensor → all gather single and reduce scatter tensor → reduce scatter single, aligning with torchcomms. The old names remain as thin wrappers marked deprecated via FutureWarning. See 186123 https://github.com/pytorch/pytorch/pull/186123 . Bazel build removed. The Bazel build was never broadly adopted and depended on an antiquated Bazel 6; it has been removed. See 180883 https://github.com/pytorch/pytorch/pull/180883 . Non-Feature Updates Python support : CPython 3.13t dropped from the Linux binary matrix. Python 3.15 binary support is listed above as a Prototype feature. See 182951 https://github.com/pytorch/pytorch/pull/182951 . CUDA : CUDA 13.0 remains the default build; small wheels are now always built for CUDA Linux and the CUDA 12.8/12.9 builds were removed; ptxas is no longer bundled in the cu13 binary. See 180612 https://github.com/pytorch/pytorch/pull/180612 , 174716 https://github.com/pytorch/pytorch/pull/174716 . Triton : pin advanced to 3.7.1. See 186792 https://github.com/pytorch/pytorch/pull/186792 . oneDNN : submodule upgraded to v3.12. See 181222 https://github.com/pytorch/pytorch/pull/181222 .