How much VRAM and how many GPUs to fine-tune a 70B parameter model like LLaMA 3.1 locally?

A developer created a VRAM calculator for fine-tuning 70B parameter models like LLaMA 3.1 locally, showing that full fine-tuning requires about 1.1 TB of VRAM (8× H100/A100-80GB), while QLoRA can run on a single 48 GB card. The tool covers full fine-tuning, LoRA, and QLoRA methods with adjustable quantization and context length.

Rough VRAM math for fine-tuning a 70B, by method: Full fine-tune AdamW, mixed precision : ~16 bytes/param once you count fp16 weights + fp32 master weights + gradients + optimizer moments — that’s ~1.1 TB before activations, so realistically 8× H100/A100-80GB with ZeRO-3/FSDP sharding and activation checkpointing, or fewer GPUs with optimizer offload and a lot of patience. Not a home setup. LoRA fp16 base : the frozen base still needs its ~140 GB resident, plus a small adapter + optimizer state 2× 80 GB, or 4× 48 GB with sharding. QLoRA 4-bit NF4 base : ~40–45 GB for the quantized base + adapter state + activations. A single 48 GB card A6000/RTX PRO 6000 works at moderate sequence lengths; context length is what pushes it over, since KV/activation memory scales with tokens. I built a calculator that does this per method/quant/context and shows the formula and every assumption it covers full/LoRA/QLoRA, not just inference : https://vram.rxdt.dev https://vram.rxdt.dev corrections welcome if your real runs disagree.