DDP

Stanford NLP's CS336 course, audited by a student who built an LLM from scratch for $353, delivers on its premise of deepening understanding of modern LLMs through hands-on assignments that include bu…

Shrijith Venkatramana, a developer building git-lrc, explains that NVIDIA Collective Communications Library (NCCL) is the critical infrastructure enabling multi-GPU training of large language models. …

Researchers propose Gefen, a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x while maintaining performance, enabling larger microbatches and improved throughput in deep learnin…