Will it fit? A free planner for fine-tuning models that do not fit on your GPU

A developer known as byjameson released streamft, a free open-source fine-tuning engine that trains models too large for a GPU by memory-mapping NF4 weights from disk and materializing one transformer block at a time, alongside a browser calculator and CLI that predict whether a given model will fit in available VRAM. On a 6 GB RTX 3050 laptop card, the tool fine-tuned a 32B model at 1024 tokens/step, 36.7 tok/s and 4.61 GB peak, and thirty SFT steps moved held-out loss from 1.8755 to 0.9959 with gradients reaching all 64 layers. Benchmarks show PEFT/transformers is faster on a 3B model (0.58 vs 0.94 s/step), while at 7B and 14B it spills into shared system memory and runs 5.3x and 2.8x slower, and at 32B it fails before the first step.

Most fine-tuning guides start at "install these 14 packages". I wanted to start one step earlier: will it even fit? So I built two free things that answer that before you download 19 GB of weights. 1. A calculator in the browser: huggingface.co/spaces/byjameson/will-it-fit https://huggingface.co/spaces/byjameson/will-it-fit — type a model id, pick your VRAM and sequence length. It reads config.json and does the arithmetic. 2. A CLI: pip install -e . streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512 13.99B params, 48 layers NF4 blocks ~7.5 GB + bf16 head ~1.6 GB + reserve 0.9 GB embedding stays on CPU - whole model needs ~9.9 GB on 6 GB verdict: needs streaming measured on RTX 3050 6 GB: 6 block s , 3.86 GB peak, 6.726 s/step at seq 512 Memory is arithmetic. Speed is only printed where it was measured. The memory number comes from config.json : NF4 weight bytes, the bf16 LM head, the embedding pinned to CPU, and a 0.9 GB reserve. That is deterministic. The speed number exists only for configurations I actually ran on the reference machine RTX 3050 Laptop, 6 GB . For anything else it says not measured and tells you to start with one block and let the trainer's autotune measure the real MB/block. Planners that guess throughput are worse than no planner. streamft fine-tunes models that do not fit by never holding them: the NF4 weights are mmap'd from disk, one transformer block is materialised on the GPU at a time, and the backward recomputes through it. LoRA is trainable, the base stays frozen. Measured on a 6 GB laptop card: a 32B model at 1024 tokens/step, 36.7 tok/s, 4.61 GB peak . Thirty SFT steps moved held-out loss from 1.8755 to 0.9959 , with 64/64 layers receiving gradients. | model | streamft | PEFT/transformers | streamft peak | PEFT peak | |---|---|---|---|---| | 3B | 0.94 s/step | 0.58 s | 3.85 GB | 4.90 GB | | 7B | 1.70 s | 8.96 s | 5.51 GB | 10.05 GB | | 14B | 6.73 s | 18.56 s | 3.86 GB | 16.48 GB | | 32B | 12.5 s | fails before step 1 | 3.90 GB | — | That third row is the honest one: on a 3B, PEFT is faster than this engine and that's fine . On Windows, PEFT does not OOM for 7B/14B — it silently spills into shared system memory 10 GB and 16 GB reserved on a 6 GB card and pays 5.3x and 2.8x the step time. At 32B it dies in prepare model for kbit training before training starts. Two checks are in the repo and runnable: scripts/verify gradients.py — one resident block with recompute vs all blocks resident, same adapter weights: loss identical to 10 decimals, adapter gradients scripts/compare peft.py — the matched runs above. It also caught a bug in my first comparison: PEFT's eval averaged per row while mine averaged per token, which made the two look 0.7 apart. Fixed, both start at the same number 1.42087 . lm head ; --head nf4 is broken in the bitsandbytes version I tested. If you try it on a different card, the interesting thing is the disagreement between the planner and reality — that is exactly what the autotune log shows, and I would like to hear where it is wrong.