PyTorch 2.13: FlexAttention on Apple Silicon, 4x Memory Savings, Upgrade Guide

PyTorch 2.13, released July 8, introduces FlexAttention with native Metal support on Apple Silicon, delivering up to 12x speedup over SDPA on sparse patterns, and a new fused loss function that reduces peak GPU memory by 4x for large-vocabulary LLM training. The release also includes API renames and a hard removal of one feature set, prompting users to review changes before upgrading.

PyTorch 2.13 shipped July 8 with changes worth acting on before your next training run. FlexAttention now has native Metal support on Apple Silicon — up to 12x faster than SDPA on sparse patterns. A new fused loss function cuts peak GPU memory 4x for large-vocabulary LLM training. And two APIs got renamed, with one feature set hard-removed. Here is what to check before upgrading. FlexAttention on Apple Silicon: Finally a Reason to Train on MPS If you have been running inference on an M-series Mac but sending training jobs to a cloud GPU, PyTorch 2.13 gives you a reason … The post