{"slug": "show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu", "title": "Show HN: Fine-tune an 8B model on a 4 GB laptop GPU", "summary": "Soup v0.72.4, an open-source CLI tool, now supports preference alignment (DPO, ORPO, SimPO, KTO) via layer streaming, enabling fine-tuning of 8B models on a 4 GB laptop GPU. The update claims bit-exact results against non-streamed runs, with streamed DPO peaking at 0.914× the supervised fine-tuning peak on an RTX 3050 4 GB, and adds a VRAM pre-flight for paired losses. The tool aims to reduce infrastructure overhead, which the developer says consumes 30-50% of teams' time.", "body_md": "**Fine-tune and post-train LLMs in one command. No SSH, no config hell.**\n\n[Website](https://trysoup.dev) ·\n[Quick Start](#quick-start) ·\n[Config](#configuration) ·\n[Docs](#documentation) ·\n[Commands](/MakazhanAlpamys/Soup/blob/main/docs/commands.md) ·\n[Models](/MakazhanAlpamys/Soup/blob/main/docs/models.md) ·\n[Discord](https://discord.gg/8RgVbFA6Zq)\n\nSoup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.\n\n```\npip install \"soup-cli[train]\"   # add [train] to fine-tune; bare `soup-cli` is the light CLI\nsoup init --template chat\nsoup train\n```\n\nTraining LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that.\n\n**Zero SSH.** Never SSH into a broken GPU box again.**One config.** A simple YAML file is all you need.**Auto everything.** Batch size, GPU detection, quantization — handled.**Works locally.** Train on your own GPU with QLoRA. No cloud required.\n\n**v0.72.4 — align on a laptop: DPO, ORPO, SimPO and KTO over layer streaming.** Layer\nstreaming keeps the frozen base out of VRAM and feeds it to the GPU one decoder layer at\na time. It used to support supervised fine-tuning only; now it runs the preference\nlosses too.\n\n**DPO's reference model is free.** DPO needs a reference to compare against, and a second copy of the model would double memory and defeat the whole point. Soup uses*the same streamed base with its adapters switched off*— one set of weights, one stream. Measured on an RTX 3050 4 GB: streamed DPO peaked at**0.914×** the supervised-fine-tuning peak. Forcing a real second model in the same test cost**+730 MB — exactly one copy of the weights.****KTO is not reference-free**, however it is usually described: it picks its reference the same way DPO does, so it gets the same treatment. ORPO and SimPO genuinely are.**Bit-exact against a normal, non-streamed run** of the same loss —`0.0`\n\ndifference, the bar every release in this series has to clear.**The VRAM pre-flight knows a paired loss is twice the rows**, because chosen and rejected go through the model as one tensor.** Honest cost:**the reference is free in*memory*, not in*time*— DPO reads the layer stack**1.52×** as often per step as supervised fine-tuning does.`grpo`\n\n/`ppo`\n\nstay excluded on purpose: generation re-reads every layer per token, which is exactly what streaming cannot amortise.- Still BETA.\n\n```\n# soup.yaml — then just `soup train --config soup.yaml`\ntraining:\n  stream_layers: true      # base streams out of VRAM; only the adapter trains\n  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card\n  batch_size: 4            # v0.72.3: bigger batches amortise the weight read\n  stream_source: auto      # RAM when it fits, NVMe disk when it does not\n```\n\nTrained withThat adapter is inert — its tensors were saved under keys with an extra`stream_layers: true`\n\non v0.72.0?`.inner.`\n\nsegment, so every loader returned the untuned base. Fixed in v0.72.1; re-run or re-save. Check with:`python -c \"from safetensors.torch import load_file; print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])\"`\n\n## Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)\n\nPoint `soup reward synth`\n\nat a JSONL of reference outputs and it infers a deterministic verifier,\nwrites a readable / committable `.py`\n\nreward function, and — the part nobody else does — *refuses* to\nemit one that can't tell your references from bad answers (four families: `numeric`\n\n/ `json_schema`\n\n/\n`regex`\n\n/ `tool_call`\n\n; a mandatory calibration report is the moat). Reward ensembles\n(`reward_fn: \"accuracy,format\"`\n\n) also train now. (#311)\n\n```\nsoup reward synth references.jsonl -o reward.py --output-report calib.json\n```\n\n## Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)\n\n`soup ship`\n\n's verdict became emittable, committable, and provenance-bound: `--emit-evidence`\n\nmakes a\nrun replay into an identical verdict, `eval.ship`\n\nin `soup.yaml`\n\n+ `--config`\n\nmakes the gate policy\nreviewable, and `--config`\n\nbinds evidence to the exact recipe that produced it (stale evidence → exit 3).\n`soup ship --push owner/repo#N`\n\nposts the SHIP / DON'T-SHIP card on the PR.\n\n## Previous release — v0.71.38, The gate grows teeth (real leg-2 regression gate)\n\n`soup ship`\n\n's regression leg became real: a fixed, extraction-based scorer over seven bundled,\noffline suites (MCQ · arithmetic · tool-calling · JSON validity · safety/refusal). A tune that\nwins your task but quietly breaks tool-calling now gets a **DON'T SHIP**. Zero new deps.\n\n```\nsoup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl\n#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error\n```\n\n## Previous release — v0.71.33, `soup draft`\n\n(measure speculative decoding)\n\n`soup draft measure`\n\nreports a draft model's **acceptance rate** + real plain-vs-assisted tok/s\n(exit 0/2/1 for CI); `soup draft distill`\n\ndistils your target into a dense tiny draft, auto-wired\ninto `soup serve --auto-spec`\n\n. The honest result on a small same-family pair: distillation didn't\nmove acceptance (69.3% → 69.3%) and assisted decoding was a net slowdown — which is exactly the\nnumber you want *before* shipping speculative decoding.\n\n```\nsoup draft measure --target ./my-tuned-model --draft HuggingFaceTB/SmolLM2-135M-Instruct \\\n  --prompts prod-prompts.jsonl        # -> acceptance %, real tok/s, ship-or-not\n```\n\nFull history: [CHANGELOG.md](/MakazhanAlpamys/Soup/blob/main/CHANGELOG.md) · [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).\n\n```\n# Light core: CLI + config + data tools, no PyTorch\npip install soup-cli\n\n# Add the training stack (torch, transformers, peft, trl, datasets, …)\npip install \"soup-cli[train]\"\n\n# Everything (train + serve + ui + data) in one shot\npip install \"soup-cli[all]\"\n\n# Or from GitHub (latest dev)\npip install git+https://github.com/MakazhanAlpamys/Soup.git\n```\n\nThe full extras table (`fast`\n\n, `mlx`\n\n, `serve`\n\n, `eval`\n\n, `ui`\n\n, `vision`\n\n, `audio`\n\n, …) lives in\n[ docs/models.md](/MakazhanAlpamys/Soup/blob/main/docs/models.md#optional-extras).\n\nUse double quotes around the extra.They are the only spelling that works in every shell —`cmd.exe`\n\n, PowerShell, bash, and zsh.Older tutorials and videos (including some of ours) show the single-quoted\n\n`pip install 'soup-cli[train]'`\n\n. That is bash / zsh / PowerShell syntax, and it fails on Windows`cmd.exe`\n\n, which has no single-quote quoting and hands the quotes straight to pip:\n\n```\nERROR: Invalid requirement: \"'soup-cli[train]'\": Expected package name at the start of dependency specifier\n```\n\nIf you hit that, swap the\n\n`'`\n\nfor`\"`\n\n— pip is rejecting a literal quote character, nothing is wrong with the package. (Dropping the quotes entirely works on Windows too, but zsh then reads`[train]`\n\nas a glob and fails.)\n\n`soup init`\n\n, `soup data …`\n\n, and the other data/inspection commands work on the light install.\nFine-tuning (`soup train`\n\n) needs the `[train]`\n\nextra.\n\n```\nsoup init                       # interactive wizard\nsoup init --template chat       # or start from a template\n```\n\nTemplates: `chat`\n\n, `code`\n\n, `tool-calling`\n\n, `medical`\n\n, `reasoning`\n\n, `vision`\n\n, `kto`\n\n, `orpo`\n\n,\n`simpo`\n\n, `ipo`\n\n, `bco`\n\n, `rlhf`\n\n, `pretrain`\n\n, `moe`\n\n, `longcontext`\n\n, `embedding`\n\n, `audio`\n\n.\n\n```\nsoup train --config soup.yaml                 # LoRA, quantization, batching — all handled\nsoup chat  --model ./output                    # talk to your model\nsoup push  --model ./output --repo you/my-model\n\nsoup merge  --adapter ./output                              # merge LoRA into the base\nsoup export --model ./output --format gguf --quant q4_k_m   # GGUF for Ollama / llama.cpp\n```\n\nMore export targets (ONNX, TensorRT, AWQ, GPTQ, BitNet) and deployment options live in\n[ docs/serving-and-export.md](/MakazhanAlpamys/Soup/blob/main/docs/serving-and-export.md).\n\nA complete `soup.yaml`\n\n:\n\n```\nbase: meta-llama/Llama-3.1-8B-Instruct\ntask: sft\n# backend: unsloth  # 2-5x faster, pip install \"soup-cli[fast]\"\n\ndata:\n  train: ./data/train.jsonl\n  format: alpaca\n  val_split: 0.1\n\ntraining:\n  epochs: 3\n  lr: 2e-5\n  batch_size: auto\n  lora:\n    r: 64\n    alpha: 16\n  quantization: 4bit\n\noutput: ./output\n```\n\n`config/schema.py`\n\nis the single source of truth for every field. Advanced data, training,\nand PEFT options are documented under [Documentation](#documentation).\n\nThe full feature reference lives in [ docs/](/MakazhanAlpamys/Soup/blob/main/docs). Start here:\n\n| Guide | Covers |\n|---|---|\n|\n\n[PEFT, long context & efficiency](/MakazhanAlpamys/Soup/blob/main/docs/peft-and-efficiency.md)[Performance & quantization](/MakazhanAlpamys/Soup/blob/main/docs/performance-and-quantization.md)[Data engineering](/MakazhanAlpamys/Soup/blob/main/docs/data.md)[Evaluation & probes](/MakazhanAlpamys/Soup/blob/main/docs/evaluation.md)`soup advise`\n\n[Serving & export](/MakazhanAlpamys/Soup/blob/main/docs/serving-and-export.md)[Adapters, registry & governance](/MakazhanAlpamys/Soup/blob/main/docs/adapters-and-governance.md)`soup loop`\n\n), knowledge editing, steering, supply-chain controls (scan/sign/BOM/attest/audit/airgap)[Compliance & governance quickstart](/MakazhanAlpamys/Soup/blob/main/docs/compliance.md)`init`\n\ntemplates, provenance (BOM/attest/repro-receipt), audit log, air-gap, model-card autogen (`soup card`\n\n), CI gate (`soup ci init`\n\n)[Backends, platform & ops](/MakazhanAlpamys/Soup/blob/main/docs/backends-and-ops.md)[Command reference](/MakazhanAlpamys/Soup/blob/main/docs/commands.md)`soup`\n\ncommand list[Supported models & extras](/MakazhanAlpamys/Soup/blob/main/docs/models.md)All formats are auto-detected from JSONL, JSON, CSV, Parquet, or TXT:\n\n**alpaca**—`{\"instruction\": ..., \"input\": ..., \"output\": ...}`\n\n**sharegpt**—`{\"conversations\": [{\"from\": \"human\", \"value\": ...}, ...]}`\n\n**chatml**—`{\"messages\": [{\"role\": \"user\", \"content\": ...}, ...]}`\n\n**dpo / orpo / simpo / ipo**—`{\"prompt\": ..., \"chosen\": ..., \"rejected\": ...}`\n\n**kto**—`{\"prompt\": ..., \"completion\": ..., \"label\": true}`\n\n**llava / sharegpt4v**(vision),** audio**,** plaintext**(pre-training),** embedding**,** prm**,** pre_tokenized**,** video**,** multimodal**\n\nFull schemas and the Axolotl/LlamaFactory-parity data pipeline (remote URIs, streaming,\nsharding, interleaving, vocab expansion, document ingestion) are in\n[ docs/data.md](/MakazhanAlpamys/Soup/blob/main/docs/data.md).\n\n```\nsoup train  --config soup.yaml        # train (SFT/DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/...)\nsoup infer  --model ./output --input prompts.jsonl   # batch inference\nsoup chat   --model ./output          # interactive chat\nsoup serve  --model ./output          # OpenAI-compatible API server\nsoup merge  --adapter ./output        # merge LoRA into the base model\nsoup export --model ./output --format gguf           # export for deployment\nsoup eval   benchmark --model ./output               # evaluate\nsoup data   inspect ./data/train.jsonl               # dataset stats\nsoup recipes list                     # 100+ ready-made model recipes\nsoup autopilot --model <id> --data d.jsonl --goal chat  # zero-config\nsoup doctor                           # check GPU / deps / environment\n```\n\nThe complete command list is in [ docs/commands.md](/MakazhanAlpamys/Soup/blob/main/docs/commands.md).\n\nSoup works with **any** text-generation model on the\n[HuggingFace Hub](https://huggingface.co/models?pipeline_tag=text-generation) — if it loads with\n`AutoModelForCausalLM`\n\n, it works, zero config changes. Llama 3.x/4, Qwen 2.5/3, Gemma 3, Mistral,\nMixtral, DeepSeek R1/V3, Phi-4, and 100+ others ship as ready-made recipes (`soup recipes list`\n\n).\n\n| VRAM | Max model (QLoRA 4-bit) | Example |\n|---|---|---|\n| 8 GB | ~7B | Llama-3.1-8B, Mistral-7B |\n| 16 GB | ~14B | Phi-4-14B, Qwen2.5-14B |\n| 24 GB | ~34B | CodeLlama-34B, Yi-1.5-34B |\n| 48 GB | ~70B | Llama-3.3-70B |\n| 80 GB+ | 70B+ (full) or MoE | Mixtral-8x22B, DeepSeek-V3 |\n\nFull model + vision tables and the optional-extras matrix are in [ docs/models.md](/MakazhanAlpamys/Soup/blob/main/docs/models.md).\n\nRun Soup without installing CUDA or PyTorch locally (image published to GHCR on every release):\n\n```\ndocker pull ghcr.io/makazhanalpamys/soup:latest\ndocker run --gpus all -v $(pwd):/workspace ghcr.io/makazhanalpamys/soup train --config soup.yaml\ndocker compose up   # or build locally\n```\n\n- Python 3.10+\n- GPU with CUDA (recommended), Apple Silicon (MPS), or CPU (experimental — very slow)\n- 8 GB+ VRAM for 7B models with QLoRA\n\nAll training tasks run on CPU for testing (quantization auto-disabled). Optional extras\n(`train`\n\n, `all`\n\n, `fast`\n\n, `vision`\n\n, `qat`\n\n, `serve`\n\n, `serve-fast`\n\n, `ui`\n\n, `eval`\n\n, `deepspeed`\n\n,\n`liger`\n\n, `mlx`\n\n, `onnx`\n\n, `tensorrt`\n\n, …) are listed in\n[ docs/models.md](/MakazhanAlpamys/Soup/blob/main/docs/models.md#optional-extras).\n\n```\nsoup doctor    # GPU, system resources, dependencies, and version in one place\n```\n\n— reinstall PyTorch for your CUDA version:`ImportError: DLL load failed while importing _C`\n\n(Windows)`pip install torch --index-url https://download.pytorch.org/whl/cu121`\n\n.— multiple Python installs; use a virtualenv.`soup version`\n\n≠`pip show soup-cli`\n\n```\ngit clone https://github.com/MakazhanAlpamys/Soup.git\ncd Soup\npip install -e \".[dev]\"\n\nruff check src/soup_cli/ tests/    # lint\npytest tests/ -v                   # unit tests (fast, no GPU)\npytest tests/ -m smoke -v          # smoke tests (downloads a tiny model, trains)\n\npre-commit install                 # optional: ruff lint+format on commit\n```\n\nSee [CONTRIBUTING.md](/MakazhanAlpamys/Soup/blob/main/CONTRIBUTING.md) for the full workflow and [SECURITY.md](/MakazhanAlpamys/Soup/blob/main/SECURITY.md) to\nreport a vulnerability.\n\nSoup is Apache-2.0 and free — and stays that way. It is built and maintained in the open on a single 4 GB laptop, which is why every performance number in these docs is measured rather than claimed.\n\nIf Soup saved you a training run, [starring the repo](https://github.com/MakazhanAlpamys/Soup)\nhelps most, and it costs nothing.\n\nThe next most useful thing is **hardware**. Multi-GPU, 8B+ validation, and Apple Silicon are\nthe parts a single 4 GB laptop cannot reach, so they ship behind honest \"requires \"\ngates instead of unverified claims. If you have access to a bigger box — or GPU credits going\nunused — running one of the\n[ help wanted](https://github.com/MakazhanAlpamys/Soup/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)\nissues and posting the numbers moves Soup further than anything else. Those issues say exactly\nwhat is blocked on hardware today.\n\nBuilt by the community ❤️ — thank you to everyone who has contributed. See\n[CONTRIBUTORS.md](/MakazhanAlpamys/Soup/blob/main/CONTRIBUTORS.md).\n\nBugs and feature requests belong in the\n[issue tracker](https://github.com/MakazhanAlpamys/Soup/issues), questions in\n[Discussions](https://github.com/MakazhanAlpamys/Soup/discussions) — both get answered faster\nand help the next person with the same problem.\n\nFor live chat, setup help, and everything that reads better as a conversation, join the\n[Discord](https://discord.gg/8RgVbFA6Zq). Anything that should still be findable in six months\nbelongs in Issues or Discussions — a Discord answer helps one person, an issue helps everyone\nwho hits the same thing. The [Code of Conduct](/MakazhanAlpamys/Soup/blob/main/CODE_OF_CONDUCT.md) applies there too.\n\nFor anything that does not fit in public — security reports (see [SECURITY.md](/MakazhanAlpamys/Soup/blob/main/SECURITY.md)),\nCode of Conduct matters, or press — email ** team@trysoup.dev**. That is the project address\nand the right one for anything Soup-related.\n\n**is the maintainer's personal address; it reaches the same person and is a fine fallback.**\n\n[makazanalpamys@gmail.com](mailto:makazanalpamys@gmail.com)Layer streaming — training an 8B model on a 4 GB laptop GPU by streaming the frozen base from host RAM one decoder layer at a time — is described in a preprint, together with the correctness protocol that verifies a streamed run is bit-exact against a resident one:\n\nMakazhan, A. (2026).\n\nExact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU.Zenodo.[https://doi.org/10.5281/zenodo.21771064]\n\nThe measurement records behind every number in it are in [ benchmarks/](/MakazhanAlpamys/Soup/blob/main/benchmarks), published\nas written — including the failures, the assumptions that turned out wrong, and the numbers that\nwere measured and then discarded.\n\n```\n@misc{makazhan2026exact,\n  title        = {Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU},\n  author       = {Makazhan, Alpamys},\n  year         = {2026},\n  publisher    = {Zenodo},\n  doi          = {10.5281/zenodo.21771064},\n  url          = {https://doi.org/10.5281/zenodo.21771064}\n}\n```\n\n[Apache-2.0](/MakazhanAlpamys/Soup/blob/main/LICENSE). Copyright © the Soup contributors.", "url": "https://wpnews.pro/news/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu", "canonical_source": "https://github.com/MakazhanAlpamys/Soup", "published_at": "2026-08-04 11:17:57+00:00", "updated_at": "2026-08-04 11:22:59.571568+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-tools", "developer-tools"], "entities": ["Soup", "RTX 3050", "QLoRA", "DPO", "ORPO", "SimPO", "KTO"], "alternates": {"html": "https://wpnews.pro/news/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu", "markdown": "https://wpnews.pro/news/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu.md", "text": "https://wpnews.pro/news/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu.txt", "jsonld": "https://wpnews.pro/news/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu.jsonld"}}