# How do you train a PyTorch model on multiple GPUs?

> Source: <https://dev.to/viswaretas_kotra/how-do-you-train-a-pytorch-model-on-multiple-gpus-493o>
> Published: 2026-10-28 01:38:50+00:00

Wrap your model in DistributedDataParallel (DDP) if it fits on one GPU, or shard it with FSDP if it doesn't, and start one process per GPU with `torchrun`. On Nodus, `nodus run --gpu H100:8 -- torchrun train.py` runs that script on all eight GPUs of one machine with no launcher flags, and `--nodes 2 --launcher torchrun` spreads it across two machines. Fill one machine before you add a second: the second machine buys wall-clock time, not a cheaper run.

This post walks through the three decisions in order: which wrapper, how to launch it, and when more than one machine is worth paying for. The code follows the PyTorch tutorials and the Nodus docs; the memory and cost figures are our own arithmetic, shown in full so you can redo it for your model.

Use DDP by default and skip `nn.DataParallel`. The [PyTorch DDP tutorial](https://docs.pytorch.org/tutorials/intermediate/ddp_tutorial.html) explains why: DataParallel is a single process with many threads, so it fights the Python GIL and only works on one machine, while DDP runs one process per GPU and works across machines too.

DDP gives every GPU a full copy of the model. Each one trains on a different slice of the batch, then an all-reduce averages the gradients so every copy stays identical. That is simple and fast, but it means the whole training state has to fit on each GPU.

FSDP removes that limit. The [PyTorch FSDP tutorial](https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html) describes it as sharding model parameters, gradients and optimizer states across GPUs, which "makes it feasible to train models that cannot fit on a single GPU." Each GPU holds only its shard and gathers a layer's full weights just before it needs them.

The decision rule we use: if the model, its gradients and the optimizer state fit on one GPU with room for activations, use DDP. If they don't, use FSDP. If even FSDP is tight, look at LoRA before you add hardware.

Full training with Adam in mixed precision costs about 16 bytes per parameter before activations. That figure comes from the [ZeRO paper](https://arxiv.org/abs/1910.02054): 2 bytes for 16-bit weights, 2 for gradients and 12 for the FP32 master weights plus Adam's two moments. DDP keeps all 16 bytes on every GPU. FSDP with full sharding divides them by the number of GPUs.

| Model | Model state per GPU with DDP | FSDP on 8 GPUs | FSDP on 16 GPUs | 
|---|---|---|---|
| 1B parameters | 16 GB | 2 GB | 1 GB | 
| 8B parameters | 128 GB | 16 GB | 8 GB | 
| 70B parameters | 1,120 GB | 140 GB | 70 GB | 

The math is parameters x 16 bytes, divided by the GPU count for FSDP. Activations, temporary all-gather buffers and the CUDA context come on top, so treat these as floors.

*On 80 GB GPUs, an 8B model needs FSDP and a 70B model needs more than one 8-GPU machine of them.*

Read it against an 80 GB NVIDIA H100. A 1B model fits comfortably with DDP. An 8B model does not fit at 128 GB, but drops to 16 GB per GPU with FSDP on one 8-GPU machine. A full 70B run needs 140 GB per GPU even when sharded over 8 GPUs, so on H100s it needs two machines.

Even then, 70 GB on each of 16 GPUs leaves little room for activations. That is why our [VRAM guide for 70B models](https://www.nodus-compute.ai/blog/vram-needed-for-70b-model/) points full 70B fine-tuning at one machine with 8 NVIDIA B200s instead: a bigger single machine beats a second small one.

Four changes turn a single-GPU script into a DDP script: start a process group, pin each process to its GPU, wrap the model, and give each process its own slice of the data. This sketch follows the DDP tutorial's pattern:

``` python
import os

import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader
from torch.utils.data.distributed import DistributedSampler

dist.init_process_group("nccl")
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)

model = DDP(build_model().cuda(local_rank), device_ids=[local_rank])
sampler = DistributedSampler(dataset)
loader = DataLoader(dataset, batch_size=32, sampler=sampler)
opt = torch.optim.AdamW(model.parameters(), lr=1e-4)

for epoch in range(epochs):
    sampler.set_epoch(epoch)
    for x, y in loader:
        loss = loss_fn(model(x.cuda(local_rank)), y.cuda(local_rank))
        opt.zero_grad()
        loss.backward()
        opt.step()

dist.destroy_process_group()
```

`build_model`, `dataset`, `epochs` and `loss_fn` are yours. Two lines are easy to miss. [`DistributedSampler`](https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler) stops every GPU from training on the same examples, and `sampler.set_epoch(epoch)` reshuffles each epoch; without it, PyTorch's own docstring warns, the same ordering is used every time.

The batch size above is per GPU. On 8 GPUs the effective batch is 8 x 32 = 256, so revisit your learning rate when you scale out.

Replace the DDP wrapper with `fully_shard` from FSDP2, applied to each transformer block first and then to the whole model. The FSDP tutorial uses exactly this order:

``` python
from torch.distributed.fsdp import fully_shard

model = build_model()
for layer in model.layers:
    fully_shard(layer)
fully_shard(model)
opt = torch.optim.AdamW(model.parameters(), lr=1e-4)
```

Sharding per block matters. It lets FSDP gather one block's weights, run it and free them again, so only one block is ever whole on a GPU. Create the optimizer after sharding so it sees the sharded parameters. The data loading and training loop stay the same as the DDP version.

Run the script under `torchrun`, which starts one process per GPU and sets `RANK`, `LOCAL_RANK` and `WORLD_SIZE` for each. On your own machine you pass `--nproc-per-node`. On Nodus you ask for the GPUs and leave the flags off:

```
nodus run --gpu H100:8 --max-cost 250 -d -- torchrun train.py
```

The Nodus GPU guide explains why no flags are needed: Nodus sets `PET_NPROC_PER_NODE` to the GPU count, so `torchrun` starts one process per GPU, and `/dev/shm` is already sized for NCCL. `--max-cost` caps the spend in dollars, here just above the $208.00 estimate worked out below, and the same guide shows a `--dry-run` that prints the estimate before anything starts.

Add a machine when one machine cannot hold the job or cannot finish it in time. More GPUs rarely make a run cheaper, because the gradient traffic between machines is slower than inside one, so each GPU does a little less useful work.

Here is a worked example. Say a run takes 10 hours on one machine with 8 H100s. H100s are listed at $2.60/hr on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/), so that is 80 GPU-hours, or $208.00.

Now run it on two machines. The cost depends on scaling efficiency, the share of perfect speed-up you actually get, which you have to measure for your own model and network.

| Setup | Wall-clock | GPU-hours | Cost at $2.60/GPU-hour | 
|---|---|---|---|
| 1 machine, 8 GPUs | 10.0 h | 80.0 | $208.00 | 
| 2 machines, 16 GPUs, 90% efficiency | 5.6 h | 88.9 | $231.11 | 
| 2 machines, 16 GPUs, 75% efficiency | 6.7 h | 106.7 | $277.33 | 

Wall-clock is 10 hours divided by (2 x efficiency); GPU-hours is 16 times that. Prices checked October 27, 2026. The efficiencies are assumptions for the example, not measurements.

*Two machines finish sooner but cost 11% to 33% more in this example.*

So the second machine is a time purchase. At 90% efficiency you pay $23.11 more to get the result about 4.4 hours sooner. If nobody is waiting on it, one machine is the cheaper answer.

Add `--nodes` and `--launcher torchrun` to the same command. The [Nodus multi-node guide](https://www.nodus-compute.ai/docs/guides/multi-node/) uses this as its two-node smoke test:

```
nodus run --name torchrun-2-node --gpu H100 --nodes 2 --launcher torchrun --max-cost 2.00 -- torchrun train.py
```

Nodus acquires every machine, checks that they can reach each other, and only then starts your command on all of them together. It sets `MASTER_ADDR`, `NODE_RANK` and the `PET_*` variables that `torchrun` reads, so the script itself does not change. `--gpus-per-node 8` gives each machine eight GPUs, and `--total-gpus` lets Nodus pick the machine shape for you.

Multi-node training is in Beta, and the docs are specific about its limits. Gangs top out at 8 machines.

By default all machines are placed on one private network in one region. If you allow wider placement, traffic between machines on public addresses is not encrypted in the Beta. The guide also says a two-machine DDP result does not yet qualify cross-machine FSDP or several GPUs per machine across machines, so check the exercised combinations before a large FSDP run.

The whole run stops, because every process waits on the others in each all-reduce. That makes checkpoints more important as you add GPUs: one lost machine out of two throws away everything since the last save.

On Nodus, the multi-node guide describes the recovery. If a machine is lost or reclaimed, Nodus stops every rank, keeps the surviving machines, replaces only the lost one, and restarts all ranks with the latest checkpoint restored into the state directory. Your script has to save there and load from it on start; the checkpoint guide in the Nodus docs shows how, and our post on [automatic checkpointing and resume](https://www.nodus-compute.ai/blog/automatic-checkpointing-and-resume-for-training/) compares how other platforms handle it.

As many as the machine has, one process per GPU. On Nodus, the GPU count you ask for is 1, 2, 4 or 8 per machine, and anything larger needs a multi-node job.

DDP and FSDP run over NCCL on whatever links the GPUs share, so NVLink is not required. It does speed up the gradient and all-gather traffic, which matters most for FSDP because it moves weights as well as gradients.

Nodus accepts `Accelerate` and `Deepspeed` as launchers for multi-node jobs, setting their variables or a hostfile, but the multi-node guide marks them accepted in the Beta and qualified later. `torchrun`, Ray and verl are documented without that caveat.

Often, yes. LoRA freezes the base weights and trains small adapters, so gradients and optimizer state shrink to a fraction of a percent of the model. If LoRA quality is good enough for your task, it usually fits a model on one GPU that full training would need a cluster for.

*Originally published on the [Nodus blog](https://www.nodus-compute.ai/blog/multi-gpu-training-pytorch/).*