Member-only story
I spent an afternoon pricing every byte of a LoRA fine-tuning step from first principles, and the result was not what I expected. For Llama 3.1 8B, 87.3% of the memory that scales with your sequence length is not the model, not the optimizer, and not the attention activations. It is the cross-entropy loss head — one tensor that exists for a few microseconds, produces a single scalar, and never appears in your training script.
For gpt-oss-20b it is 95.2%. At 16K context that one tensor wants 49.1 GB, while the entire 4-bit quantized model it is training fits in 12.8 GB. The loss function is four times the size of the thing being trained. This is the actual axis on which Unsloth, Axolotl, TRL, and LLaMA-Factory differ. Everybody reads the framework comparisons as “Unsloth is 2x faster,” files it under marketing, and moves on. That framing is wrong. The speed numbers are real but modest and model-dependent. The memory numbers are enormous and structural, and they come almost entirely from what each framework does with this one tensor.
I wrote a 120-line calculator that prices it. Then I checked it against Unsloth’s published benchmark table. On an 80 GB GPU my arithmetic predicts the Transformers + FlashAttention-2 baseline at 27,713 tokens…