# An 8B Fine-Tune Now Fits in 4 GB of VRAM > Source: > Published: 2026-08-04 15:09:54+00:00 [AI](https://sourcefeed.dev/c/ai)Article # An 8B Fine-Tune Now Fits in 4 GB of VRAM Soup streams frozen decoder layers through the GPU one at a time, and proves the arithmetic is bit-exact. [Mariana Souza](https://sourcefeed.dev/u/mariana_souza) For three years, the floor under local fine-tuning has sat at roughly the same height. [QLoRA](https://arxiv.org/abs/2305.14314) set it in 2023: quantize the frozen base model to 4-bit NF4, train a small LoRA adapter on top, and suddenly a 33B model fit on a 24 GB card. [Unsloth](https://unsloth.ai) then ground the 8B case down to about 8 GB — its own benchmarks put a Llama-3.1-8B QLoRA run at 6.6 to 9 GB peak depending on config. Below that, nothing. A 4 GB laptop GPU — the RTX 3050 in millions of mid-range machines — was inference-only territory, and marginal inference at that. [Soup](https://github.com/MakazhanAlpamys/Soup), a Show HN this week from independent researcher Alpamys Makazhan, drops the floor through the basement. The author-reported numbers: a Llama-3.1-8B LoRA fine-tune in NF4 running at 119.6 tokens/sec with a 3.32 GB VRAM peak, on a 4 GB RTX 3050 Laptop GPU under Windows. Not a distilled model, not a 3B stand-in — the full 8B, with a [preprint](https://doi.org/10.5281/zenodo.21771064) documenting the method. ## The trick is that frozen weights are just data The insight Soup builds on is old, but it's been stuck on the wrong side of the training/inference divide. In LoRA training the base model is frozen: no gradients, no optimizer states, nothing to update. Its weights are read-only inputs to the computation — which means they don't have to *live* anywhere, they just have to *arrive on time*. [AirLLM](https://github.com/lyogavin/airllm) exploited exactly this for inference back in 2023, running 70B models on 4 GB cards by loading one decoder layer at a time and paying a brutal latency tax. Soup does the same thing for training: the frozen base stays in host RAM (or spills to NVMe when RAM is short), and decoder layers stream through pre-allocated VRAM buffers as the forward and backward passes need them. Only the LoRA adapters, their gradients, and their optimizer states are permanent GPU residents. Training is a much nastier place to pull this off than inference. The backward pass traverses the stack a second time in reverse, activations have to be checkpointed and recomputed as layers come and go, and — the part that should worry you most — any numerical slippage in the swap logic doesn't crash. It just quietly trains a worse model. This is where Soup earns real credit: the project validates layer streaming as *bit-exact* against a fully-resident run, max absolute logit difference of 0.0, checked across nine architecture families. Anyone who has chased dtype-mismatch ghosts through DeepSpeed's offload paths knows how unusual it is for an offloading system to make that claim, let alone gate releases on it. There's a genuinely clever bonus for preference training. DPO normally doubles your memory bill because it needs a frozen reference model alongside the policy. But Soup's streamed base with the adapters switched off *is* the reference — so streamed DPO peaked at 0.914× the SFT peak in the author's tests. The reference model is free in memory. It is not free in time: DPO reads the layer stack 1.52× as often per step. And the approach has a hard boundary — GRPO and PPO are explicitly unsupported, because online RL generates tokens during training, and generation re-reads every layer per token. Streaming can't amortize that. With RL-style post-training being where most of the frontier's attention is in 2026, that's not a small asterisk. ## The honest math Run the numbers before you get excited. At 119.6 tokens/sec, a modest SFT set — 10,000 examples averaging 512 tokens — is about 12 hours per epoch. A rented A100 does that job in well under an hour for a couple of dollars. Streaming also carries a measured 1.43× overhead versus resident training, though the author could only measure that at 0.5B scale, since resident 8B doesn't fit on the card at all. If your question is "what's the cheapest way to fine-tune an 8B model," the answer is still the cloud, and it isn't close. But that's the wrong question for the people this serves. A 1,000-example persona or style LoRA is roughly 70 minutes per epoch — a lunch break. A 10k-example run is one overnight. And the whole thing happens on hardware you already own, with training data that never leaves the machine. For a student, a developer without a corporate cloud account, or anyone fine-tuning on data they can't legally upload, "slow but local" beats "fast but elsewhere" every time. One HN commenter described running a fine-tuned 4B model for AML compliance work at community banks — exactly the shape of workload where the data-locality argument writes itself. Adoption is a pip install and a YAML flag: ``` training: stream_layers: true quantization: 4bit batch_size: 4 # larger batches amortize the weight reads stream_source: auto # RAM if it fits, NVMe if not ``` Fair warning before you bet anything real on it: this is a v0.72.x project from a single author, the throughput and memory numbers are self-reported from one Windows laptop, and nobody has independently replicated them yet. The design is credible and the bit-exactness protocol is the right kind of paranoia, but treat the specific figures as a preprint's claims, not established fact. ## Training keeps inheriting inference's tricks The pattern worth noticing runs deeper than one repo. Every memory trick the inference world invents migrates to training a year or two later: 4-bit quantization went from GPTQ serving to QLoRA training; paged attention's memory discipline echoed into paged optimizers; now AirLLM-style layer streaming has crossed over too. The frozen-base structure of PEFT makes the migration almost mechanical — and it suggests the next crossing is already visible. Per the author, a 14B model in NF4 needs about 7.5 GB streamed, which puts 14B fine-tuning inside the 8 GB cards that ship in ordinary gaming laptops. My read: this is not an economics story, and anyone framing it as "cloud training is dead" is selling something. It's an access-floor story. The set of people who can fine-tune a real model just expanded from "owns a 24 GB GPU or a credit card" to "owns a laptop from the last four years," and the interesting consequences of that — private-data LoRAs, classroom fine-tuning, hobbyist preference-tuning — come from who got added, not from any throughput number. The floor didn't move down a notch. For an entire class of hardware, it just appeared. ## Sources & further reading - [Soup: LLM fine-tuning framework with exact layer streaming](https://github.com/MakazhanAlpamys/Soup)— github.com - [Show HN: Fine-tune an 8B model on a 4 GB laptop GPU](https://news.ycombinator.com/item?id=49166984)— news.ycombinator.com - [Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU](https://doi.org/10.5281/zenodo.21771064)— doi.org - [Finetune Llama 3.1 with Unsloth](https://unsloth.ai/blog/llama3-1)— unsloth.ai - [AirLLM: 70B inference on a single 4GB GPU](https://github.com/lyogavin/airllm)— github.com [Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. ## Discussion 0 No comments yet Be the first to weigh in.