{"slug": "will-it-fit-a-free-planner-for-fine-tuning-models-that-do-not-fit-on-your-gpu", "title": "Will it fit? A free planner for fine-tuning models that do not fit on your GPU", "summary": "A developer known as byjameson released streamft, a free open-source fine-tuning engine that trains models too large for a GPU by memory-mapping NF4 weights from disk and materializing one transformer block at a time, alongside a browser calculator and CLI that predict whether a given model will fit in available VRAM. On a 6 GB RTX 3050 laptop card, the tool fine-tuned a 32B model at 1024 tokens/step, 36.7 tok/s and 4.61 GB peak, and thirty SFT steps moved held-out loss from 1.8755 to 0.9959 with gradients reaching all 64 layers. Benchmarks show PEFT/transformers is faster on a 3B model (0.58 vs 0.94 s/step), while at 7B and 14B it spills into shared system memory and runs 5.3x and 2.8x slower, and at 32B it fails before the first step.", "body_md": "Most fine-tuning guides start at \"install these 14 packages\". I wanted to start one step earlier: **will it even fit?**\n\nSo I built two free things that answer that before you download 19 GB of weights.\n\n**1. A calculator in the browser:** [huggingface.co/spaces/byjameson/will-it-fit](https://huggingface.co/spaces/byjameson/will-it-fit) — type a model id, pick your VRAM and sequence length. It reads `config.json` and does the arithmetic.\n\n**2. A CLI:**\n\n```\npip install -e .\nstreamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512\n13.99B params, 48 layers\nNF4 blocks ~7.5 GB + bf16 head ~1.6 GB + reserve 0.9 GB (embedding stays on CPU)\n-> whole model needs ~9.9 GB on 6 GB\nverdict: needs streaming\nmeasured on RTX 3050 6 GB: 6 block(s), 3.86 GB peak, 6.726 s/step at seq 512\n```\n\n**Memory is arithmetic. Speed is only printed where it was measured.**\n\nThe memory number comes from `config.json`: NF4 weight bytes, the bf16 LM head, the embedding pinned to CPU, and a 0.9 GB reserve. That is deterministic.\n\nThe speed number exists only for configurations I actually ran on the reference machine (RTX 3050 Laptop, 6 GB). For anything else it says *not measured* and tells you to start with one block and let the trainer's autotune measure the real MB/block. Planners that guess throughput are worse than no planner.\n\n`streamft` fine-tunes models that do not fit by never holding them: the NF4 weights are mmap'd from disk, one transformer block is materialised on the GPU at a time, and the backward recomputes through it. LoRA is trainable, the base stays frozen.\n\nMeasured on a 6 GB laptop card: a 32B model at **1024 tokens/step, 36.7 tok/s, 4.61 GB peak**. Thirty SFT steps moved held-out loss from **1.8755 to 0.9959**, with 64/64 layers receiving gradients.\n\n| model | streamft | PEFT/transformers | streamft peak | PEFT peak | \n|---|---|---|---|---|\n| 3B | 0.94 s/step | **0.58 s** | **3.85 GB** | 4.90 GB | \n| 7B | **1.70 s** | 8.96 s | **5.51 GB** | 10.05 GB | \n| 14B | **6.73 s** | 18.56 s | **3.86 GB** | 16.48 GB | \n| 32B | **12.5 s** | fails before step 1 | **3.90 GB** | — | \n\nThat third row is the honest one: on a 3B, **PEFT is faster than this engine and that's fine**. On Windows, PEFT does not OOM for 7B/14B — it silently spills into shared system memory (10 GB and 16 GB reserved on a 6 GB card) and pays 5.3x and 2.8x the step time. At 32B it dies in `prepare_model_for_kbit_training` before training starts.\n\nTwo checks are in the repo and runnable:\n\n`scripts/verify_gradients.py` — one resident block (with recompute) vs all blocks resident, same adapter weights: loss identical to 10 decimals, adapter gradients `scripts/compare_peft.py` — the matched runs above. It also caught a bug in my first comparison: PEFT's eval averaged per row while mine averaged per token, which made the two look 0.7 apart. Fixed, both start at the same number (1.42087).`lm_head`); `--head nf4` is broken in the bitsandbytes version I tested.\nIf you try it on a different card, the interesting thing is the *disagreement* between the planner and reality — that is exactly what the autotune log shows, and I would like to hear where it is wrong.", "url": "https://wpnews.pro/news/will-it-fit-a-free-planner-for-fine-tuning-models-that-do-not-fit-on-your-gpu", "canonical_source": "https://dev.to/mycute_97b39c4e34027570f9/will-it-fit-a-free-planner-for-fine-tuning-models-that-do-not-fit-on-your-gpu-o50", "published_at": "2026-10-25 14:39:23+00:00", "updated_at": "2026-10-25 14:45:13.987018+00:00", "lang": "en", "topics": ["ai-tools", "machine-learning", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["streamft", "Hugging Face", "Qwen2.5-14B-Instruct", "PEFT", "transformers", "bitsandbytes", "RTX 3050", "byjameson"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/will-it-fit-a-free-planner-for-fine-tuning-models-that-do-not-fit-on-your-gpu", "markdown": "https://wpnews.pro/news/will-it-fit-a-free-planner-for-fine-tuning-models-that-do-not-fit-on-your-gpu.md", "text": "https://wpnews.pro/news/will-it-fit-a-free-planner-for-fine-tuning-models-that-do-not-fit-on-your-gpu.txt", "jsonld": "https://wpnews.pro/news/will-it-fit-a-free-planner-for-fine-tuning-models-that-do-not-fit-on-your-gpu.jsonld"}}