{"slug": "pytorch-2-13-release-blog", "title": "PyTorch 2.13 Release Blog", "summary": "PyTorch 2.13 has been released with performance improvements including FlexAttention on Apple Silicon with up to 12x speedups, a CuTeDSL backend for Inductor, and fused nn.LinearCrossEntropyLoss reducing peak memory by up to 4x. The release also introduces a new torchcomms backend for distributed training and FSDP2 communication overlap, along with broader platform support and ExecuTorch integration into PyTorch Core.", "body_md": "### Featured projects\n\nWe are excited to announce the release of PyTorch® 2.13 ([release notes](https://github.com/pytorch/pytorch/releases/tag/v2.13.0))!\n\nThe PyTorch 2.13 release features the following changes:\n\n-\n**FlexAttention lands on Apple Silicon (MPS),** with up to ~12x speedup over SDPA on sparse patterns, and gains a deterministic backward path on CUDA for reproducible gradient computation -\n**CuTeDSL “Native DSL” backend** gives Inductor a second high-performance code path (alongside Triton) for key GPU operations, with faster compilation [Prototype] -\n**nn.LinearCrossEntropyLoss** combines final prediction and loss computation operations to cut peak GPU memory by up to 4x for large-vocabulary language model training -\n**torchcomms,** a new communications backend for PyTorch Distributed, improves fault tolerance, scalability, and debuggability for large-cluster training -\n**FSDP2 now overlaps reduce-scatter and all-gather communications** via a dedicated process group (opt-in), increasing distributed training throughput -\n**Python 3.15 wheel support for Torch on Linux** via the pytorch repository index. This support includes builds compatible with free-threaded 3.15t. -\n**Broader platform support**: ROCm gains AOTriton 0.12b with native HIP CMake, Arm adds Armv9-A torch.compile targeting, and Intel XPU exposes new device telemetry APIs\n\nThis release is composed of 3,328 commits from 526 contributors since PyTorch 2.12. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.13. More information about how to get started with the PyTorch 2-series can be found at our[ Getting Started](https://pytorch.org/get-started/locally/) page.\n\nHave questions? Join us on July 22, 2026, at 11 a.m. PT for a live Q&A with panelists Alban Desmaison, Andrey Talman, Piotr Bialecki and moderator Chris Gottbrath. We will provide a brief overview of the release and answer your questions live. [Register today.](https://pytorch.org/event/pytorch-2-13-release-live-qa/)\n\nThroughout the 2.x series, PyTorch has been evolving from a research-first framework into a unified, hardware-agnostic platform for production training and inference at scale.[ PyTorch 2.11](https://pytorch.org/blog/pytorch-2-11-release-blog/) introduced differentiable collectives for distributed training and FlashAttention-4 on next-generation GPUs.[ PyTorch 2.12](https://pytorch.org/blog/pytorch-2-12-release-blog/) added a device-agnostic torch.accelerator.Graph API, up to 100× faster batched eigendecomposition, and Microscaling quantization export support.\n\nPyTorch 2.13 pushes performance further across platforms and scales: FlexAttention lands on Apple Silicon with up to 12× speedups, CuTeDSL brings CUTLASS-grade GEMM kernels to Inductor, and fused nn.LinearCrossEntropyLoss cuts peak memory up to 4x for large-vocabulary models. On the distributed side, a new torchcomms backend improves fault tolerance and debuggability at cluster scale, and FSDP2 unlocks communication overlap between all-gather and reduce-scatter for higher training throughput. This release also marks[ ExecuTorch’s integration](https://pytorch.org/blog/executorch-becomes-part-of-pytorch-core/) into PyTorch Core, making on-device inference a first-class capability of the framework.\n\n**Performance Improvements**\n\n**Deterministic Backward for FlexAttention Flash Backend**\n\nThis improvement focuses on correctness and debugging for the existing CUDA implementation of FlexAttention by making gradient computation reproducible. By default, the FlexAttention flash backend uses atomic operations in the backward pass for dQ accumulation, which introduces non-determinism — repeated runs on the same input can produce slightly different gradients. This makes debugging, regression testing, and reproducible research difficult.\n\nThe new deterministic backward path (compute_dq_write_order) replaces atomics with a pre-computed write ordering that guarantees bit-for-bit reproducible gradients without meaningful performance penalty. The measured end-to-end overhead on create_block_mask is well under 1% at longer sequence lengths (e.g., +0.2% at S=32768), making determinism effectively free for most production workloads. Users can opt in via the existing torch.use_deterministic_algorithms(True) setting with no additional code changes.\n\nAPI Unstable\n\n(PR[ #174813](https://github.com/pytorch/pytorch/pull/174813) by Driss Guessous, Meta)\n\n**Large MPS Op Migration to Native Metal**\n\nPyTorch’s MPS backend previously delegated most operations to Apple’s MPSGraph framework, which adds compilation and scheduling overhead per dispatch. For high-frequency, latency-sensitive operations this overhead can dominate execution time. This release migrates a broad set of operations to hand-written Metal compute kernels — copy/cast, uniform/normal/randint, comparisons, reductions (sum/mean), cumsum/cumprod, sort (multi-block and stable), embedding backward, and scatter/gather with bounds checking.\n\nThe native Metal path eliminates MPSGraph’s per-op compilation cost and gives PyTorch direct control over thread dispatch and memory access patterns, reducing kernel launch latency across common training and inference workloads on Apple Silicon.\n\nAPI Unstable\n\n(PR[ #184740](https://github.com/pytorch/pytorch/pull/184740) by Nikita Shulga, Meta,[ #185609](https://github.com/pytorch/pytorch/pull/185609) and[ #185119](https://github.com/pytorch/pytorch/pull/185119) by Irakli Salia, EPAM.)\n\n**FlexAttention on Apple Silicon (MPS)**\n\nFlexAttention, PyTorch’s unified API for expressing custom attention patterns as plain Python functions compiled into fused kernels, is now available on Metal/MPS. The MPS implementation provides hand-written Metal kernels for both the sparse prefill and decode paths (including GQA and captured buffers). So instead of writing a custom CUDA kernel for every attention variant you need, FlexAttention will write a 2-line Python function, and the compiler builds a fast kernel for you automatically.\n\nThe benchmark numbers are impressive for sparse masks. On long, sparse attention patterns the speedups over SDPA are substantial — e.g. on a 1×8×32768×64 shape with a 256-element sliding window (0.8% density), FlexAttention runs in ~35 ms vs ~431 ms for SDPA (**~12.3x**); a smaller 8192-length / 64-window case achieves **~4.15x**. Dense patterns continue to favor SDPA, as expected. API Unstable\n\n(PR[ #182552](https://github.com/pytorch/pytorch/pull/182552),[ #186215](https://github.com/pytorch/pytorch/pull/186215), and[ #181575](https://github.com/pytorch/pytorch/pull/181575) by Irakli Salia, EPAM)\n\n**Core Features**\n\n**nn.LinearCrossEntropyLoss**\n\nIn standard large-vocabulary training (e.g., language models with 100K+ token vocabularies), computing the cross-entropy loss requires materializing the entire logits matrix over the vocabulary, which can consume tens of gigabytes of GPU memory. nn.LinearCrossEntropyLoss (and the corresponding linear_cross_entropy functional) fuses the final linear projection and cross-entropy computation into a single module that processes the vocabulary dimension in chunks, never materializing the full logits matrix. This reduces peak memory by up to ~4x for large-vocabulary workloads while maintaining numerical equivalence with the unfused path. The implementation supports label smoothing, weight tying, and z-loss regularization out of the box, and integrates with torch.compile for further optimization. As a drop-in replacement for separate nn.Linear + nn.CrossEntropyLoss, adoption requires no other code changes.\n\nAPI Unstable\n\n(See[ #172446](https://github.com/pytorch/pytorch/pull/172446),[ #172286](https://github.com/pytorch/pytorch/pull/172286), and[ #185852](https://github.com/pytorch/pytorch/pull/185852) by Pearu Peterson, OpenTeams.)\n\n**Load Safetensors Directly with torch.load**\n\nSafetensors has become a widely adopted format for distributing model weights (used by Hugging Face, Stability AI, and others) due to its memory-mapped loading and lack of arbitrary code execution risk. Previously, loading safetensors files required installing and importing a separate library. torch.load(“foo.safetensors”) now works natively, detecting the format automatically and returning tensors directly. This removes a dependency for common workflows and makes PyTorch a seamless drop-in for loading models distributed in safetensors format.\n\nAPI Unstable\n\n(PR[ #170592](https://github.com/pytorch/pytorch/pull/170592) by Nikita Shulga, Meta)\n\n**Python 3.15 Binary Support – Release Engineering**\n\nPyTorch wheels now support Python 3.15, including the experimental free-threaded 3.15t build. Important Note: Python 3.15 is currently in pre-release (beta), with the final stable release scheduled for October 2026. Support is limited to torch wheels only (torchvision is not built for 3.15 yet) and to Linux builds on x86_64 and aarch64, covering the CPU, CUDA, ROCm, and XPU variants. torch.compile is not yet supported on Python 3.15. There are no Windows or macOS 3.15 wheels for this python version in this release.\n\nPython 3.15 and 3.15t wheels are not published to PyPI — they are available to download only via download.pytorch.org, using any of the following commands:\n\n# CPU\n\npip3 install torch –index-url https://download.pytorch.org/whl/cpu\n\n# CUDA (substitute the CUDA version, e.g. cu126 / cu130)\n\npip3 install torch –index-url https://download.pytorch.org/whl/cu130\n\n# ROCm (substitute the ROCm version)\n\npip3 install torch –index-url https://download.pytorch.org/whl/rocm7.2\n\n# XPU\n\npip3 install torch –index-url https://download.pytorch.org/whl/xpu\n\nThe same commands install the free-threaded 3.15t build when run under a free-threaded interpreter.\n\nAPI Unstable. For updates See tracker issue: #[184352\n](https://github.com/pytorch/pytorch/issues/184352)(PR\n\n[#182954](https://github.com/pytorch/pytorch/pull/182954)by Nikita Shulga, PR\n\n[#184600](https://github.com/pytorch/pytorch/pull/184600)and\n\n[#186244](https://github.com/pytorch/pytorch/pull/186244)by Andrey Talman, Meta,\n\n[#186017](https://github.com/pytorch/pytorch/pull/186017)by Rob Timpe, OpenTeams)\n\n**Distributed Training**\n\n**torchcomms Backend**\n\nPyTorch Distributed has historically relied on c10d’s ProcessGroup abstraction for collective communications (all-reduce, all-gather, etc.), which was designed primarily around NCCL. As distributed training grows more complex (multi-dimensional parallelism, elastic scaling, heterogeneous interconnects), the original backend’s error handling and observability limitations become bottlenecks for operations teams.\n\ntorchcomms is a new communications backend integrated into PyTorch Distributed’s CI and device-mesh paths, providing improved fault tolerance (graceful timeout and partial-group recovery), better scalability across large clusters, and richer debuggability through structured logging and collective tracing. It serves as a modern alternative to the existing c10d backends while maintaining API compatibility.\n\nAPI Unstable\n\n(PR[ #181662](https://github.com/pytorch/pytorch/pull/181662) by Tristan Rice, Meta,[ #178533](https://github.com/pytorch/pytorch/pull/178533) Pangiotis Kourdis, Intel, and[ #182057](https://github.com/pytorch/pytorch/pull/182057) by Kapil Sharma, Meta)\n\n**FSDP2 Separate Reduce-Scatter Group**\n\nIn fully-sharded data-parallel (FSDP) training, all-gather and reduce-scatter share a single NCCL communicator by default. Because NCCL serializes operations on the same communicator, these two collectives cannot overlap, leaving communication bandwidth underutilized. FSDPModule.set_separate_reduce_scatter_group(enable=True) gives reduce-scatter its own dedicated NCCL communicator, allowing it to progress concurrently with all-gather operations. This enables AG/RS overlap, improving training throughput for fully-sharded workloads without any changes to model code.\n\nAPI Unstable\n\n(PR[ #186335](https://github.com/pytorch/pytorch/pull/186335) by Wei Feng, Meta)\n\n**Compilation and Export**\n\n**torch.compiler.set_default_backend**\n\nWhen using custom or out-of-tree compiler backends (e.g., for specialized hardware), users previously had to pass backend= explicitly to every torch.compile() call, making code verbose and error-prone. torch.compiler.set_default_backend mirrors the pattern established by torch.set_default_dtype and torch.set_default_device, letting backend authors or infrastructure teams set a process-wide default once. All subsequent torch.compile() calls automatically use that backend unless an explicit backend= argument overrides it. This simplifies adoption of custom backends across large codebases.\n\nAPI Unstable\n\n(PR[ #178944](https://github.com/pytorch/pytorch/pull/178944) by Angela Yi, Meta)\n\n**Platform Features and Updates**\n\n**CUDA**\n\n**CuTeDSL “Native DSL” Backend for Inductor**\n\nCuTeDSL is a Python-native domain-specific language built on NVIDIA’s CuTe (CUDA Templates) library, giving developers direct control over GPU tensor layouts, tiling strategies, and memory access patterns. PyTorch’s Inductor compiler can now use CuTeDSL as an alternative code-generation backend alongside Triton — specifically for matrix multiplication (GEMM) and normalization (RMSNorm), two of the most performance-critical operations in transformer training. These Quack-derived kernel overrides produce higher-quality matrix-multiply code without requiring Triton for these workloads. Kernel compilation has also moved from the thread pool to a subprocess pool, eliminating Python’s GIL bottleneck and improving compile-time parallelism.\n\nAPI Unstable\n\n(PR[ #181267](https://github.com/pytorch/pytorch/pull/181267) by Michael Lazos,[ #182108](https://github.com/pytorch/pytorch/pull/182108) by Simon Layton, and[ #186310](https://github.com/pytorch/pytorch/pull/186310) by Driss Guessous, Meta)\n\n**ROCm**\n\n**AOTriton 0.12b, Origami GEMM Selection, Native HIP CMake**\n\nThe ROCm stack gains three improvements: AOTriton advances to 0.12b with support for asymmetric head dimensions, deterministic algorithms, and new GPU targets (gfx1100/gfx1151 promoted to stable, partial FlashAttention v3 on gfx950). The Origami tool replaces brute-force autotuning with analytical GEMM-configuration selection, cutting tune time. The build system now uses CMake’s native HIP language support.\n\nAPI Unstable\n\n(PR[ #184288](https://github.com/pytorch/pytorch/pull/184288) and[ #172512](https://github.com/pytorch/pytorch/pull/172512) by Xinya Zhang and Umesh Chand, AMD)\n\n**Arm**\n\n**Armv9-A Target Support for ****torch.compile**\n\ntorch.compile on AArch64 now recognizes Armv9-A CPUs (e.g., Neoverse V2 used in AWS Graviton4), propagating the correct target triple and feature set (128-bit and 256-bit SVE) through Inductor codegen. No behavior change for x86.\n\nAPI Unstable\n\n(PR[ #184555](https://github.com/pytorch/pytorch/pull/184555) by Zhibo Li, Arm)\n\n**XPU (Intel GPUs)**\n\n**Device Telemetry APIs**\n\nNew query APIs for Intel GPUs expose runtime device state: memory usage (torch.xpu.device_memory_used), utilization, power draw, clock rate, and temperature, plus device-wide synchronization and hardware properties (last-level cache size, integrated-GPU detection).\n\nAPI Unstable\n\n(PR[ #183431](https://github.com/pytorch/pytorch/pull/183431),[ #183429](https://github.com/pytorch/pytorch/pull/183429),[ #183428](https://github.com/pytorch/pytorch/pull/183428), and[ #183427](https://github.com/pytorch/pytorch/pull/183427) by Guangye Yu, Intel)\n\n**C++ ABI**\n\n**torch::stable::Generator**\n\nCustom kernel authors can now retrieve an at::Generator through the stable C++ ABI (previously always passed as null), enabling RNG support in ABI-stable extensions without recompilation against PyTorch internals.\n\nAPI Unstable\n\n(PR[ #186423](https://github.com/pytorch/pytorch/pull/186423) and[ #183930](https://github.com/pytorch/pytorch/pull/183930) by Jane Xu, Meta and Chris Leonard, Red Hat)\n\n**Profiling and Debugging**\n\n**Experimental CUPTI Monitor Profiler**\n\nPyTorch’s existing profiler collects GPU activity through CUPTI’s activity API, which requires synchronization points that can distort timing and contend with the GIL in multi-threaded workloads. The new experimental CUPTI monitor backend collects GPU metrics asynchronously (completely off the GIL), eliminating profiling-induced overhead while reusing the existing CPU profiler path. The result is merged Chrome traces that accurately reflect real-world execution timing, with GPU-side user annotations automatically recreated for record_function scopes. This makes it practical to profile production training loops without measurably perturbing their performance.\n\nAPI Unstable\n\n(PR[ #186037](https://github.com/pytorch/pytorch/pull/186037) and[ #186295](https://github.com/pytorch/pytorch/pull/186295) by Natalia Gimelshein, Meta)\n\n**CUDAGraph.get_graph_data()**\n\nWhen debugging performance issues in CUDA graph captures, understanding the internal structure (which kernels run, their dependencies, execution order) previously required external tools or manual inspection. CUDAGraph.get_graph_data() exposes the full graph topology programmatically: node types, kernel names, dependency edges, and IDs remapped to match CUPTI profiler output. This lets developers correlate captured graph structure with profiled kernel launches, making it straightforward to identify bottlenecks, unnecessary serialization, or suboptimal kernel fusion within a captured graph.\n\nAPI Unstable\n\n(PR[ #183165](https://github.com/pytorch/pytorch/pull/183165) by Natalia Gimelshein, Meta)\n\n**Deprecations and Backwards-Incompatible Changes**\n\n**Named tensors removed.** The deprecated named-tensor feature (Tensor.names and associated APIs) has been hard-removed to cut overhead and code bloat. See[#173895](https://github.com/pytorch/pytorch/pull/173895).**Distributed collectives renamed to a****_single****scheme.** all_gather_into_tensor → all_gather_single and reduce_scatter_tensor → reduce_scatter_single, aligning with torchcomms. The old names remain as thin wrappers marked deprecated via FutureWarning. See[#186123](https://github.com/pytorch/pytorch/pull/186123).**Bazel build removed.** The Bazel build was never broadly adopted and depended on an antiquated Bazel 6; it has been removed. See[#180883](https://github.com/pytorch/pytorch/pull/180883).\n\n**Non-Feature Updates**\n\n**Python support**: CPython 3.13t dropped from the Linux binary matrix. (Python 3.15 binary support is listed above as a Prototype feature.) See[#182951](https://github.com/pytorch/pytorch/pull/182951).**CUDA**: CUDA 13.0 remains the default build; small wheels are now always built for CUDA Linux and the CUDA 12.8/12.9 builds were removed; ptxas is no longer bundled in the cu13 binary. See[#180612](https://github.com/pytorch/pytorch/pull/180612),[#174716](https://github.com/pytorch/pytorch/pull/174716).**Triton**: pin advanced to 3.7.1. See[#186792](https://github.com/pytorch/pytorch/pull/186792).** oneDNN**: submodule upgraded to v3.12. See[#181222](https://github.com/pytorch/pytorch/pull/181222).", "url": "https://wpnews.pro/news/pytorch-2-13-release-blog", "canonical_source": "https://pytorch.org/blog/pytorch-2-13-release-blog/", "published_at": "2026-07-08 17:42:23+00:00", "updated_at": "2026-07-08 17:58:15.407086+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools", "ai-infrastructure"], "entities": ["PyTorch", "Apple Silicon", "CUDA", "ROCm", "Intel", "Arm", "ExecuTorch", "Meta"], "alternates": {"html": "https://wpnews.pro/news/pytorch-2-13-release-blog", "markdown": "https://wpnews.pro/news/pytorch-2-13-release-blog.md", "text": "https://wpnews.pro/news/pytorch-2-13-release-blog.txt", "jsonld": "https://wpnews.pro/news/pytorch-2-13-release-blog.jsonld"}}