Rough VRAM math for fine-tuning a 70B, by method:
Full fine-tune (AdamW, mixed precision): ~16 bytes/param once you count fp16 weights + fp32 master weights + gradients + optimizer moments — that’s ~1.1 TB before activations, so realistically 8× H100/A100-80GB with ZeRO-3/FSDP sharding and activation checkpointing, or fewer GPUs with optimizer offload and a lot of patience. Not a home setup.
LoRA (fp16 base): the frozen base still needs its ~140 GB resident, plus a small adapter + optimizer state 2× 80 GB, or 4× 48 GB with sharding.
QLoRA (4-bit NF4 base): ~40–45 GB for the quantized base + adapter state + activations. A single 48 GB card (A6000/RTX PRO 6000) works at moderate sequence lengths; context length is what pushes it over, since KV/activation memory scales with tokens.
I built a calculator that does this per method/quant/context and shows the formula and every assumption (it covers full/LoRA/QLoRA, not just inference): https://vram.rxdt.dev corrections welcome if your real runs disagree.