An 8B Fine-Tune Now Fits in 4 GB of VRAM

Independent researcher Alpamys Makazhan released Soup, a Show HN project that fine-tunes a full Llama-3.1-8B model in NF4 quantization with a 3.32 GB VRAM peak at 119.6 tokens/sec on a 4 GB RTX 3050 Laptop GPU under Windows, by streaming frozen decoder layers through the GPU one at a time. The method is validated as bit-exact against fully-resident training across nine architecture families, but does not support GRPO or PPO due to online token generation.

AI https://sourcefeed.dev/c/ai Article An 8B Fine-Tune Now Fits in 4 GB of VRAM Soup streams frozen decoder layers through the GPU one at a time, and proves the arithmetic is bit-exact. Mariana Souza https://sourcefeed.dev/u/mariana souza For three years, the floor under local fine-tuning has sat at roughly the same height. QLoRA https://arxiv.org/abs/2305.14314 set it in 2023: quantize the frozen base model to 4-bit NF4, train a small LoRA adapter on top, and suddenly a 33B model fit on a 24 GB card. Unsloth https://unsloth.ai then ground the 8B case down to about 8 GB — its own benchmarks put a Llama-3.1-8B QLoRA run at 6.6 to 9 GB peak depending on config. Below that, nothing. A 4 GB laptop GPU — the RTX 3050 in millions of mid-range machines — was inference-only territory, and marginal inference at that. Soup https://github.com/MakazhanAlpamys/Soup , a Show HN this week from independent researcher Alpamys Makazhan, drops the floor through the basement. The author-reported numbers: a Llama-3.1-8B LoRA fine-tune in NF4 running at 119.6 tokens/sec with a 3.32 GB VRAM peak, on a 4 GB RTX 3050 Laptop GPU under Windows. Not a distilled model, not a 3B stand-in — the full 8B, with a preprint https://doi.org/10.5281/zenodo.21771064 documenting the method. The trick is that frozen weights are just data The insight Soup builds on is old, but it's been stuck on the wrong side of the training/inference divide. In LoRA training the base model is frozen: no gradients, no optimizer states, nothing to update. Its weights are read-only inputs to the computation — which means they don't have to live anywhere, they just have to arrive on time . AirLLM https://github.com/lyogavin/airllm exploited exactly this for inference back in 2023, running 70B models on 4 GB cards by loading one decoder layer at a time and paying a brutal latency tax. Soup does the same thing for training: the frozen base stays in host RAM or spills to NVMe when RAM is short , and decoder layers stream through pre-allocated VRAM buffers as the forward and backward passes need them. Only the LoRA adapters, their gradients, and their optimizer states are permanent GPU residents. Training is a much nastier place to pull this off than inference. The backward pass traverses the stack a second time in reverse, activations have to be checkpointed and recomputed as layers come and go, and — the part that should worry you most — any numerical slippage in the swap logic doesn't crash. It just quietly trains a worse model. This is where Soup earns real credit: the project validates layer streaming as bit-exact against a fully-resident run, max absolute logit difference of 0.0, checked across nine architecture families. Anyone who has chased dtype-mismatch ghosts through DeepSpeed's offload paths knows how unusual it is for an offloading system to make that claim, let alone gate releases on it. There's a genuinely clever bonus for preference training. DPO normally doubles your memory bill because it needs a frozen reference model alongside the policy. But Soup's streamed base with the adapters switched off is the reference — so streamed DPO peaked at 0.914× the SFT peak in the author's tests. The reference model is free in memory. It is not free in time: DPO reads the layer stack 1.52× as often per step. And the approach has a hard boundary — GRPO and PPO are explicitly unsupported, because online RL generates tokens during training, and generation re-reads every layer per token. Streaming can't amortize that. With RL-style post-training being where most of the frontier's attention is in 2026, that's not a small asterisk. The honest math Run the numbers before you get excited. At 119.6 tokens/sec, a modest SFT set — 10,000 examples averaging 512 tokens — is about 12 hours per epoch. A rented A100 does that job in well under an hour for a couple of dollars. Streaming also carries a measured 1.43× overhead versus resident training, though the author could only measure that at 0.5B scale, since resident 8B doesn't fit on the card at all. If your question is "what's the cheapest way to fine-tune an 8B model," the answer is still the cloud, and it isn't close. But that's the wrong question for the people this serves. A 1,000-example persona or style LoRA is roughly 70 minutes per epoch — a lunch break. A 10k-example run is one overnight. And the whole thing happens on hardware you already own, with training data that never leaves the machine. For a student, a developer without a corporate cloud account, or anyone fine-tuning on data they can't legally upload, "slow but local" beats "fast but elsewhere" every time. One HN commenter described running a fine-tuned 4B model for AML compliance work at community banks — exactly the shape of workload where the data-locality argument writes itself. Adoption is a pip install and a YAML flag: training: stream layers: true quantization: 4bit batch size: 4 larger batches amortize the weight reads stream source: auto RAM if it fits, NVMe if not Fair warning before you bet anything real on it: this is a v0.72.x project from a single author, the throughput and memory numbers are self-reported from one Windows laptop, and nobody has independently replicated them yet. The design is credible and the bit-exactness protocol is the right kind of paranoia, but treat the specific figures as a preprint's claims, not established fact. Training keeps inheriting inference's tricks The pattern worth noticing runs deeper than one repo. Every memory trick the inference world invents migrates to training a year or two later: 4-bit quantization went from GPTQ serving to QLoRA training; paged attention's memory discipline echoed into paged optimizers; now AirLLM-style layer streaming has crossed over too. The frozen-base structure of PEFT makes the migration almost mechanical — and it suggests the next crossing is already visible. Per the author, a 14B model in NF4 needs about 7.5 GB streamed, which puts 14B fine-tuning inside the 8 GB cards that ship in ordinary gaming laptops. My read: this is not an economics story, and anyone framing it as "cloud training is dead" is selling something. It's an access-floor story. The set of people who can fine-tune a real model just expanded from "owns a 24 GB GPU or a credit card" to "owns a laptop from the last four years," and the interesting consequences of that — private-data LoRAs, classroom fine-tuning, hobbyist preference-tuning — come from who got added, not from any throughput number. The floor didn't move down a notch. For an entire class of hardware, it just appeared. Sources & further reading - Soup: LLM fine-tuning framework with exact layer streaming https://github.com/MakazhanAlpamys/Soup — github.com - Show HN: Fine-tune an 8B model on a 4 GB laptop GPU https://news.ycombinator.com/item?id=49166984 — news.ycombinator.com - Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU https://doi.org/10.5281/zenodo.21771064 — doi.org - Finetune Llama 3.1 with Unsloth https://unsloth.ai/blog/llama3-1 — unsloth.ai - AirLLM: 70B inference on a single 4GB GPU https://github.com/lyogavin/airllm — github.com Mariana Souza https://sourcefeed.dev/u/mariana souza · Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. Discussion 0 No comments yet Be the first to weigh in.