FSDP vs DeepSpeed: which should you use for LLM training? A Nodus Compute engineering writeup compares PyTorch FSDP and DeepSpeed ZeRO for LLM training, mapping ZeRO stages 0-3 onto FSDP's NO_SHARD, SHARD_GRAD_OP and FULL_SHARD settings and noting that only DeepSpeed's optimizer-only offload has no FSDP equivalent. Worked memory math for 8 GPUs shows a 70B full fine-tune fits on 80 GB H100s only with optimizer offload (17.5 GB per GPU at ZeRO-3 plus CPU optimizer versus 140 GB at plain ZeRO-3), while an 8B model fits at nearly any sharded setting. PyTorch FSDP should be your default for LLM training, and DeepSpeed ZeRO earns its place when the model only fits if you move the optimizer, and nothing else, into CPU memory or onto NVMe, which FSDP cannot do. Nodus runs either one on a single machine with the same nodus run command, while its managed TrainingJobs reject DeepSpeed ZeRO sharding and its multi-machine recovery expects PyTorch Distributed Checkpoint. In our worked math, DeepSpeed's optimizer-only offload is what puts a full 70B fine-tune on one 8 x NVIDIA H100 machine instead of 8 x B200. Both tools implement the same idea from the ZeRO paper https://arxiv.org/abs/1910.02054 : split the weights, gradients and Adam state across GPUs instead of copying them to each one. So the choice comes down to the extra knobs each one has, and the tooling around it. This post covers what our FSDP vs DDP guide https://www.nodus-compute.ai/blog/fsdp-vs-ddp/ does not: how ZeRO stages map onto FSDP settings, what CPU offload buys for 8B and 70B models, the memory we got from DeepSpeed's own estimator, and how to run both on Nodus. Three of the four line up one to one. The Hugging Face Accelerate FSDP guide https://huggingface.co/docs/accelerate/usage guides/fsdp mapping-between-fsdp-sharding-strategies-and-deepspeed-zero-stages gives the mapping, and the DeepSpeed ZeRO tutorial https://www.deepspeed.ai/tutorials/zero/ defines what each stage splits. | What is split across GPUs | DeepSpeed | PyTorch FSDP | Model state per GPU N GPUs | |---|---|---|---| | Nothing | ZeRO stage 0 | NO SHARD same as DDP | 16 bytes x params | | Optimizer state | ZeRO stage 1 | no equivalent | 4 + 12/N bytes x params | | Optimizer state and gradients | ZeRO stage 2 | SHARD GRAD OP | 2 + 14/N bytes x params | | Optimizer state, gradients and weights | ZeRO stage 3 | FULL SHARD | 16/N bytes x params | | Stage 3 inside a machine, copies across machines | ZeRO++ with zero hpz partition size set to GPUs per node | HYBRID SHARD | 16 / GPUs per machine bytes x params | | Optimizer state on CPU or NVMe, weights on GPU | offload optimizer | no equivalent | 2/N bytes x params with stage 3 | | Everything on CPU | offload param plus offload optimizer | CPUOffload offload params=True | the layer being computed | The bytes column uses the ZeRO paper's count for mixed precision with Adam: 2 bytes of 16-bit weights, 2 of 16-bit gradients and 12 of fp32 master weights and Adam moments. The offload rows follow the DeepSpeed memory docs https://deepspeed.readthedocs.io/en/latest/memory.html . Activations come on top of every row. The two gaps decide most cases, and partial offload is the one that matters. PyTorch's FSDP docs https://docs.pytorch.org/docs/stable/fsdp.html say CPUOffload offload params=True "offloads gradients to CPU as well, meaning that the optimizer step runs on CPU", and FSDP2's CPUOffloadPolicy moves parameters, gradients and optimizer state together. The Accelerate FSDP vs DeepSpeed guide https://huggingface.co/docs/accelerate/concept guides/fsdp and deepspeed calls FSDP's offload "all-or-nothing" and notes that DeepSpeed can also offload to NVMe. On one 8-GPU machine, an 8B model fits at almost any sharded setting, and a 70B model fits on 80 GB GPUs only with optimizer offload. Here is the table above worked for 8 GPUs, in decimal GB: | Setting on 8 GPUs | 8B per GPU | 70B per GPU | |---|---|---| | Stage 0 / NO SHARD | 128 GB | 1,120 GB | | ZeRO-1 | 44 GB | 385 GB | | ZeRO-2 / SHARD GRAD OP | 30 GB | 262.5 GB | | ZeRO-3 / FULL SHARD | 16 GB | 140 GB | | ZeRO-3 with CPU optimizer | 2 GB | 17.5 GB | For 70B the arithmetic is 70 x 16 = 1,120; 70 x 4 + 12/8 = 385; 70 x 2 + 14/8 = 262.5; 70 x 16/8 = 140; and 70 x 2/8 = 17.5. The 8B column swaps 70 for 8. These are floors before activations and temporary buffers. Only optimizer offload brings a 70B full fine-tune under an 80 GB H100's memory on one machine. The 8B model needs no offload at all. FULL SHARD or ZeRO-3 holds it at 16 GB per GPU, so for 8B the two frameworks reach the same memory and you pick on tooling. The 70B model is where they part. Fully sharded, it needs 140 GB per GPU, which an 80 GB H100 cannot hold. DeepSpeed can keep the 16-bit weights and gradients sharded on the GPUs and move the 12 bytes of optimizer state per parameter to CPU memory, which leaves 17.5 GB per GPU. It agreed on the GPU side and added a warning about CPU memory. We installed DeepSpeed 0.20.0 with CPU-only PyTorch 2.15.0 in a scratch environment on November 21, 2026 and ran the estimator from the DeepSpeed memory docs https://deepspeed.readthedocs.io/en/latest/memory.html . No GPU is needed: python from deepspeed.runtime.zero.stage3 import estimate zero3 model states mem needs all cold estimate zero3 model states mem needs all cold total params=70e9, largest layer params=8192 128256, num gpus per node=8, num nodes=1 The largest layer is the embedding of Llama 3.1 70B: a hidden size of 8,192 times a vocabulary of 128,256, from its config.json https://huggingface.co/unsloth/Meta-Llama-3.1-70B-Instruct/blob/main/config.json . The estimator prints "GB", but its source divides by 2^30, so these are GiB: | ZeRO-3 option | Per GPU | CPU memory | |---|---|---| | No offload, zero.Init on | 150.60 GiB | 46.97 GiB | | CPU optimizer, zero.Init on | 20.21 GiB | 1,564.62 GiB | | CPU optimizer, zero.Init off | 20.21 GiB | 3,129.24 GiB | | CPU weights and optimizer, zero.Init on | 3.91 GiB | 1,760.20 GiB | The GPU numbers run higher than our table because the estimator is stricter. It counts 18 bytes per parameter for stage 3, since DeepSpeed's docs say gradients are converted to fp32 for the update. It also reserves 4 bytes per parameter of the largest layer for gathering it, which is 4.2 GB here. The CPU column is the real lesson. With the optimizer on the CPU, 70B x 16 bytes x DeepSpeed's 1.5 safety factor is 1,680 GB, or 1,564.62 GiB. Without zero.Init , every one of the 8 processes first loads its own fp32 copy of the model, so the requirement doubles to 3,129.24 GiB. In Accelerate that switch is zero3 init flag: true . That matters on real machines. The example listing in the Nodus GPU guide https://www.nodus-compute.ai/docs/guides/gpus/ shows an 8 x H100 machine with 1800Gi of memory. With zero.Init on, the optimizer-offload run fits with about 235 GiB to spare. With it off, it does not fit at all. It is cheaper per hour, and cheaper per run only if the slowdown stays under about 2.1 times. The two single-machine ways to fully fine-tune 70B are 8 x H100 with ZeRO-3 and CPU optimizer offload, or 8 x B200 with plain FULL SHARD , where 140 GB fits in 180 GB. H100 SXM 80 GB is listed at $2.60/hr and B200 180 GB at $5.50/hr on the Nodus pricing page https://www.nodus-compute.ai/pricing/ . Eight of each is $20.80/hr and $44.00/hr. Prices checked November 21, 2026. The catch is speed. DeepSpeed's config docs https://www.deepspeed.ai/docs/config-json/ say the optimizer computation moves to the CPU whenever optimizer state is offloaded, and every step copies gradients out and updated weights back. The two GPUs also differ in speed. We have not measured either gap, so here is the break-even instead: 44.00 / 20.80 = 2.12. Say the B200 run takes 10 hours, which costs $440.00. The offload run on H100s costs: | Offload run takes | Hours | Cost at $20.80/hr | |---|---|---| | 1.5 x as long | 15 | $312.00 | | 2 x as long | 20 | $416.00 | | 2.5 x as long | 25 | $520.00 | | 3 x as long | 30 | $624.00 | The slowdown factors are assumptions for the example, not measurements. Offload wins only while the H100 run takes less than about 2.1 times as long as the B200 run. Measure before you commit. A sweep https://www.nodus-compute.ai/docs/guides/sweeps/ on Nodus runs one Job across GPU types and compares cost, time and throughput per cell, which gives you the real ratio. If you only need to adapt the model rather than retrain every weight, LoRA https://www.nodus-compute.ai/blog/lora-vs-full-fine-tuning/ avoids most of this memory, as our 70B VRAM guide https://www.nodus-compute.ai/blog/vram-needed-for-70b-model/ shows. Change one config file and keep the script. Accelerate wraps both, and its docs show the two configs side by side. This is the FSDP config from the Accelerate FSDP guide https://huggingface.co/docs/accelerate/usage guides/fsdp , trimmed to the sharding keys: distributed type: FSDP fsdp config: fsdp auto wrap policy: TRANSFORMER BASED WRAP fsdp cpu ram efficient loading: true fsdp offload params: false fsdp sharding strategy: FULL SHARD fsdp state dict type: SHARDED STATE DICT mixed precision: bf16 This is the ZeRO-3 example from the Accelerate DeepSpeed guide https://huggingface.co/docs/accelerate/usage guides/deepspeed , with offload param device changed from cpu to none so only the optimizer leaves the GPU: distributed type: DEEPSPEED deepspeed config: gradient accumulation steps: 1 gradient clipping: 1.0 offload optimizer device: cpu offload param device: none zero3 init flag: true zero3 save 16bit model: true zero stage: 3 Both launch with accelerate launch --config file