FSDP

Researchers propose Gefen, a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x while maintaining performance, enabling larger microbatches and improved throughput in deep learnin…

DTensor, PyTorch's distributed tensor abstraction, attaches placement metadata to every tensor to automatically propagate layouts and insert correct collective operations during distributed training. …

Researchers at Microsoft have introduced AutoSP, a compiler-based solution that automatically converts standard training code into multi-GPU sequence parallel code for long-context language model trai…