{"slug": "understanding-pytorch-distributeddataparallel-ddp-from-scratch-why-every-gpu-the", "title": "Understanding PyTorch DistributedDataParallel (DDP) From Scratch: Why Every GPU Trains the Entire…", "summary": "PyTorch's DistributedDataParallel (DDP) enables multi-GPU training by synchronizing gradients across all GPUs via AllReduce, ensuring each GPU maintains identical model weights. The article explains the mechanics of DDP, including the necessity of one Python process per GPU, the use of torchrun for launching, and the requirement of shuffle=False with DistributedSampler. It provides a first-principles guide to help developers troubleshoot common distributed training issues.", "body_md": "*Why does every GPU need its own copy of the model? How do gradients get averaged without a central master worker? Why is **shuffle=False mandatory when using a **DistributedSampler? Here is the first-principles guide to PyTorch DDP that explains what actually happens under the hood before walking line-by-line through a complete training script.*\n\nIf you have ever attempted to scale a PyTorch training pipeline from a single GPU to a cluster or a multi-GPU workstation, you have likely encountered **DistributedDataParallel (DDP)**.\n\nMost tutorials start by telling you:\n\n“Just wrap your model in DDP(model) and launch with torchrun!\"\n\nWhile that sounds delightfully simple, it often leaves developers stranded when subtle bugs appear. Why did your training freeze? Why are all GPUs receiving identical data batches? Why is loss diverging? Why does PyTorch insist on setting shuffle=False in the data loader?\n\nTo troubleshoot or build robust distributed deep learning systems, you need a solid mental model of **what every GPU is doing at every microsecond of the training loop**.\n\nIn this article, we will unpack DDP from first principles: starting from single-GPU mechanics, exposing the intuition behind gradient synchronization, clarifying distributed terminology, and finally walking line-by-line through a clean, working PyTorch DDP script.\n\nDeep learning models and datasets have grown exponentially. Consider a standard computer vision task training **ResNet-50** on ImageNet:\n\nAt 300 ms per batch, 5,000 batches per epoch over 90 epochs takes **days** on a single card.\n\nTo accelerate training, we need multiple GPUs working concurrently. But *how* should GPUs divide the work?\n\nBroadly, distributed training falls into two paradigms:\n\nPyTorch’s DistributedDataParallel (DDP) relies on **Data Parallelism**.\n\nBefore adding multiple GPUs, let’s revisit the canonical single-GPU PyTorch training loop:\n\nIn a standard training loop:\n\nEvery single parameter update is strictly driven by the gradients calculated from that local batch.\n\nSuppose we have 2 GPUs (GPU 0 and GPU 1).\n\n**Naive Idea:** What if we simply split the batch of 256 images into two chunks of 128 images, load them on GPU 0 and GPU 1, and run independent training loops on both?\n\nBecause GPU 0 and GPU 1 processed completely different images, their loss outputs differed, producing **different gradients**.\n\nApplying those gradients independently causes Model Copy A and Model Copy B to diverge immediately. Within a few steps, you no longer have one trained model—you have two competing models with completely different weights!\n\nThe Core Challenge:How do we let GPUs train on different data batches while ensuring all GPU copies remain100% mathematically identicalat every step?\n\nDDP resolves this challenge through **Gradient Synchronization via AllReduce**.\n\nInstead of letting each GPU update its weights using its own local gradients, **DDP averages the gradients across all GPUs before the optimizer step occurs.**\n\nHere is the master blueprint of DDP:\n\nIf:\n\nThen GPU 0 and GPU 1 will arrive at **identical weights **W₁.\n\nThis mathematical invariant holds true for step 1, step 2, and step 100,000.\n\nBefore writing code, let me clarify the core terminology:\n\nNCCL (NVIDIA Collective Communication Library):Optimized for GPU-to-GPU transfers via NVLink or PCIe.Always use NCCL for CUDA GPUs.\n\nGLOO:A cross-platform fallback library (used for CPU distributed training or Windows environments).\n\nWhen you type python train.py, your operating system launches a **single Python process**.\n\nHowever, DDP requires **N independent Python processes running simultaneously** — one for each GPU.\n\ntorchrun is PyTorch’s official process launcher. It automates:\n\nInstead of executing:\n\n```\npython train.py\n```\n\nYou execute:\n\n```\ntorchrun --nproc_per_node=2 train.py\n```\n\nLet me trace a single epoch inside a 2-GPU DDP setup:\n\nNow let me walk through a complete, runnable DDP script:\n\n[pytorch-primer/scripts/ddp_train.py at main · sourangshupal/pytorch-primer](https://github.com/sourangshupal/pytorch-primer/blob/main/scripts/ddp_train.py)\n\n``` python\nimport osimport platformimport torchfrom torch.distributed import init_process_group, destroy_process_groupdef ddp_setup(rank: int, world_size: int) -> None:    # Only set MASTER_ADDR / MASTER_PORT if torchrun hasn't already set them.    if \"MASTER_ADDR\" not in os.environ:        os.environ[\"MASTER_ADDR\"] = \"localhost\"    if \"MASTER_PORT\" not in os.environ:        os.environ[\"MASTER_PORT\"] = \"12345\"    if platform.system() == \"Windows\":        # Windows fallback (gloo)        os.environ[\"USE_LIBUV\"] = \"0\"        init_process_group(backend=\"gloo\", rank=rank, world_size=world_size)    else:        # Standard Linux GPU setup (nccl)        init_process_group(backend=\"nccl\", rank=rank, world_size=world_size)    if torch.cuda.is_available():        torch.cuda.set_device(rank)\npython\nfrom torch.utils.data import Dataset, DataLoaderfrom torch.utils.data.distributed import DistributedSamplerdef prepare_dataset():    train_ds = ToyDataset(X_train, y_train)    test_ds = ToyDataset(X_test, y_test)    train_loader = DataLoader(        dataset=train_ds,        batch_size=2,        shuffle=False,          # CRITICAL: shuffle MUST be False!        pin_memory=True,        drop_last=True,        sampler=DistributedSampler(train_ds),  # Handles chunking across GPUs    )    test_loader = DataLoader(        dataset=test_ds,        batch_size=2,        shuffle=False,    )    return train_loader, test_loader\n```\n\n[IMPORTANT]Whyshuffle=False is mandatory:When usingDistributedSampler, the sampler itself handles data shuffling across ranks. If you passshuffle=True intoDataLoader, it conflicts withDistributedSampler and raises a runtime error!\n\nIf your dataset has 10 samples ([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) and world_size = 2:\n\nNo data point is duplicated across GPUs within the same batch.\n\n``` python\nfrom torch.nn.parallel import DistributedDataParallel as DDP# 1. Instantiate local modelmodel = NeuralNetwork(num_inputs=2, num_outputs=2)# 2. Move model to process GPU FIRSTmodel.to(rank)# 3. Wrap model with DDPmodel = DDP(model, device_ids=[rank] if torch.cuda.is_available() else None)\n```\n\nWhen you invoke loss.backward() in your training loop:\n\n```\nfor epoch in range(num_epochs):    # CRITICAL: Reshuffle sampler seed for each epoch    train_loader.sampler.set_epoch(epoch)    model.train()    for features, labels in train_loader:        features, labels = features.to(rank), labels.to(rank)        logits = model(features)        loss = F.cross_entropy(logits, labels)        optimizer.zero_grad()        loss.backward()  # <--- HERE IS WHERE DDP ALLREDUCE HAPPENS        optimizer.step()\n```\n\nNotice line 3 in the loop above:\n\n```\ntrain_loader.sampler.set_epoch(epoch)\n```\n\nWithout this line, DistributedSampler will generate the **exact same dataset ordering for every single epoch**. set_epoch(epoch) seeds the sampler deterministically per epoch so that:\n\n```\n    destroy_process_group()\n```\n\nAt the end of training, destroy_process_group() cleanly closes the socket connection between ranks. Forgetting this can lead to orphaned Python processes holding onto VRAM.\n\n```\nif __name__ == \"__main__\":    # Environment variables populated by torchrun    world_size = int(os.environ.get(\"WORLD_SIZE\", 1))    rank = int(os.environ.get(\"LOCAL_RANK\", os.environ.get(\"RANK\", 0)))    if rank == 0:        print(\"PyTorch version:\", torch.__version__)        print(\"CUDA available:\", torch.cuda.is_available())        print(\"Number of GPUs available:\", torch.cuda.device_count())    torch.manual_seed(123)    main(rank, world_size, num_epochs=3)\n```\n\nHere is a summary of what you write vs. what DDP handles under the hood:\n\nHere are 5 common mistakes when implementing DDP:\n\nRunning python script.py only starts 1 process on 1 GPU. You **must** launch distributed jobs with torchrun:\n\n```\ntorchrun --nproc_per_node=2 scripts/ddp_train.py\n```\n\nSetting shuffle=True inside DataLoader conflicts with DistributedSampler. Always set shuffle=False in DataLoader and let DistributedSampler manage shuffling.\n\nIf omitted, your model trains on identical batch ordering every epoch, harming generalization.\n\nIf you log output or save model checkpoints inside the training loop without checking if rank == 0:, all N processes will attempt to write to disk simultaneously, causing corrupt files or cluttered terminal output.\n\n```\n# CORRECT WAY TO SAVE CHECKPOINTif rank == 0:    torch.save(model.module.state_dict(), \"model.pt\")\n# Access original model methods or parameters:raw_model = model.module\n```\n\nPyTorch legacy codebase tutorials sometimes mention torch.nn.DataParallel (DP). **Do not use DP for training.**\n\nHere is how they compare:\n\nDataParallel uses a single master thread to scatter input data and gather outputs across GPUs. The single Python Global Interpreter Lock (GIL) creates a massive bottleneck. DDP avoids this entirely by giving every GPU its own independent Python interpreter process.\n\nIf you retain only one paragraph from this article, let it be this:\n\nThe Golden Rule of DDP:Every GPU holds a complete, independent copy of the model. Every GPU processes a unique batch of data. Duringloss.backward(), DDP uses Ring-AllReduce to average gradients across all GPUs. Because every GPU starts from identical weights and applies identical averaged gradients, every model copy remains in perfect mathematical synchronization after every optimization step.\n\nHappy distributed training!\n\n[Understanding PyTorch DistributedDataParallel (DDP) From Scratch: Why Every GPU Trains the Entire…](https://pub.towardsai.net/understanding-pytorch-distributeddataparallel-ddp-from-scratch-why-every-gpu-trains-the-entire-2b80f89bed3d) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/understanding-pytorch-distributeddataparallel-ddp-from-scratch-why-every-gpu-the", "canonical_source": "https://pub.towardsai.net/understanding-pytorch-distributeddataparallel-ddp-from-scratch-why-every-gpu-trains-the-entire-2b80f89bed3d?source=rss----98111c9905da---4", "published_at": "2026-08-02 13:01:02+00:00", "updated_at": "2026-08-02 13:22:06.799936+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools"], "entities": ["PyTorch", "DistributedDataParallel", "NCCL", "GLOO", "torchrun", "ResNet-50", "ImageNet"], "alternates": {"html": "https://wpnews.pro/news/understanding-pytorch-distributeddataparallel-ddp-from-scratch-why-every-gpu-the", "markdown": "https://wpnews.pro/news/understanding-pytorch-distributeddataparallel-ddp-from-scratch-why-every-gpu-the.md", "text": "https://wpnews.pro/news/understanding-pytorch-distributeddataparallel-ddp-from-scratch-why-every-gpu-the.txt", "jsonld": "https://wpnews.pro/news/understanding-pytorch-distributeddataparallel-ddp-from-scratch-why-every-gpu-the.jsonld"}}