Most fine-tuning guides start at "install these 14 packages". I wanted to start one step earlier: will it even fit?
So I built two free things that answer that before you download 19 GB of weights.
1. A calculator in the browser: huggingface.co/spaces/byjameson/will-it-fit — type a model id, pick your VRAM and sequence length. It reads config.json and does the arithmetic.
2. A CLI:
pip install -e .
streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512
13.99B params, 48 layers
NF4 blocks ~7.5 GB + bf16 head ~1.6 GB + reserve 0.9 GB (embedding stays on CPU)
-> whole model needs ~9.9 GB on 6 GB
verdict: needs streaming
measured on RTX 3050 6 GB: 6 block(s), 3.86 GB peak, 6.726 s/step at seq 512
Memory is arithmetic. Speed is only printed where it was measured.
The memory number comes from config.json: NF4 weight bytes, the bf16 LM head, the embedding pinned to CPU, and a 0.9 GB reserve. That is deterministic.
The speed number exists only for configurations I actually ran on the reference machine (RTX 3050 Laptop, 6 GB). For anything else it says not measured and tells you to start with one block and let the trainer's autotune measure the real MB/block. Planners that guess throughput are worse than no planner.
streamft fine-tunes models that do not fit by never holding them: the NF4 weights are mmap'd from disk, one transformer block is materialised on the GPU at a time, and the backward recomputes through it. LoRA is trainable, the base stays frozen.
Measured on a 6 GB laptop card: a 32B model at 1024 tokens/step, 36.7 tok/s, 4.61 GB peak. Thirty SFT steps moved held-out loss from 1.8755 to 0.9959, with 64/64 layers receiving gradients.
| model | streamft | PEFT/transformers | streamft peak | PEFT peak |
|---|---|---|---|---|
| 3B | 0.94 s/step | 0.58 s | 3.85 GB | 4.90 GB |
| 7B | 1.70 s | 8.96 s | 5.51 GB | 10.05 GB |
| 14B | 6.73 s | 18.56 s | 3.86 GB | 16.48 GB |
| 32B | 12.5 s | fails before step 1 | 3.90 GB | — |
That third row is the honest one: on a 3B, PEFT is faster than this engine and that's fine. On Windows, PEFT does not OOM for 7B/14B — it silently spills into shared system memory (10 GB and 16 GB reserved on a 6 GB card) and pays 5.3x and 2.8x the step time. At 32B it dies in prepare_model_for_kbit_training before training starts.
Two checks are in the repo and runnable:
scripts/verify_gradients.py — one resident block (with recompute) vs all blocks resident, same adapter weights: loss identical to 10 decimals, adapter gradients scripts/compare_peft.py — the matched runs above. It also caught a bug in my first comparison: PEFT's eval averaged per row while mine averaged per token, which made the two look 0.7 apart. Fixed, both start at the same number (1.42087).lm_head); --head nf4 is broken in the bitsandbytes version I tested.
If you try it on a different card, the interesting thing is the disagreement between the planner and reality — that is exactly what the autotune log shows, and I would like to hear where it is wrong.