{"slug": "fine-tuning-gemma-4-on-an-8-gb-rtx-3050-slow-low-end-and-still-viable", "title": "Fine-Tuning Gemma 4 on an 8 GB RTX 3050: Slow, Low End and Still Viable", "summary": "A developer released lazy-gemma, an open-source pipeline that fine-tunes Gemma 4 E2B on a single 8 GB RTX 3050, running the full loop—self-generated training data, training, and evaluation—in 6 hours 44 minutes with a 4,896 MiB peak. The project found that the best tuned run cut tokens per correct answer by only 6.3% while losing one correct answer out of 250 held-out GSM8K problems, whereas llama.cpp's built-in 256-token thinking cap cut tokens by 35.8% with no training at all.", "body_md": "*This is a submission for the [Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady](https://dev.to/challenges/hacktoberfest-week3-2026-10-19)*\n\nThis article provides a step by step guide to fine-tuning Gemma 4 E2B on an 8 GB NVIDIA RTX 3050, the low end of a desktop GPU, so that the model thinks less before it answers. Every log, script and number is committed.\n\n[https://github.com/xbill9/lazy-gemma](https://github.com/xbill9/lazy-gemma)\n\n**The RTX 3050 is an 8 GB inference card: it serves Gemma 4 E2B at 109 tokens per second. That speed makes it a training card too, because cheap fine-tuning starts with inference. On one 3050 the whole loop runs in 6 h 44 min: the model writes its own training data, 3.7 million tokens in 4 h 1 min; trains on it in 2 h 10 min at a 4,896 MiB peak, or 1 h 6 min with the standard QLoRA stack; and is graded on 250 held-out problems in 32 min. The trained file drops straight back into llama.cpp, byte for byte the format it started in. The lazy model itself is not there yet: the best tuned run cuts tokens per correct answer by 6.3% with one fewer right answer out of 250, while llama.cpp's built-in thinking cap of 256 tokens cuts them by 35.8% with no training at all.**\n\nGemma 4 thinks before it answers. On grade-school maths problems the E2B model writes a median of 788 tokens per reply on the 3050, and most of that is thinking. On a fast GPU that costs a few seconds. On a slow device it is most of the wait.\n\nThe lazy idea: teach the model to reach the same answer in fewer words. Ask it the same problem four times, keep the shortest answer that is right, and fine-tune it on those.\n\nThe 3050 question: a used 8 GB card is good for inference, so can it fine-tune too, or does training need rented hardware? This kind of fine-tuning starts with inference: the model has to answer thousands of problems before there is anything to train on, and that took longer than the training itself. A card that serves 109 tokens per second already does that part. The answer here is that the same card runs the whole loop, overnight, and the trained model ships in the same 4-bit file it serves.\n\nWhat the project delivers:\n\nThe baseline and each tuned model answer the same 250 held-out GSM8K problems, greedy, thinking on, through llama-server on the 3050:\n\n| Model | Right | Tokens per correct answer | \n|---|---|---|\n| Gemma 4 E2B, uncapped | 227 (90.8%) | 977.8 | \n| Same, llama.cpp thinking cap 128 | 224 (89.6%) | 548.8 | \n| Same, llama.cpp thinking cap 256 | 228 (91.2%) | 628.0 | \n| Same, llama.cpp thinking cap 512 | 228 (91.2%) | 856.8 | \n| Tuned A: shortest right answer, lr 1e-4 | 220 (88.0%) | 927.2 | \n| Tuned B: budget in the prompt, plain prompt | 229 (91.6%) | 947.2 | \n| Tuned B, prompt says \"at most 256 tokens\" | 224 (89.6%) | 903.2 | \n| Tuned A at lr 1e-3 | 226 (90.4%) | 916.2 | \n| QLoRA at lr 1e-3 | 228 (91.2%) | 963.2 | \n\nTokens include the thinking. A tuned model only counts as a win if it uses fewer tokens per correct answer at the same accuracy.\n\n| Path | What | \n|---|---|\n| `make_splits.py` | Fixed train and held-out sets from GSM8K, written before any training | \n| `lazy_eval.py` | Runs and grades the held-out set against llama-server | \n| `sample_train.py` | Four answers per training problem, with the exact token ids | \n| `qat_lora.py` | The model, built on the served 4-bit weights, with trainable adapters | \n| `train.py` | Picks the shortest right answer per problem and trains | \n| `qlora_train.py` | The same data and settings on the standard QLoRA stack | \n| `export_gguf.py` | Writes the trained weights back into a copy of the served file | \n| `benchmarks/runs/` | Every run: script, logs, per-problem rows, summaries | \n\n`b11696-3-g7182ef71e`\n`xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf`, Gemma 4 E2B with Google's quantization-aware weights written exactly as 4-bit Q4_0`google/gemma-4-E2B-it-qat-q4_0-unquantized` from Hugging Face, for the model's layout and tokenizer`git clone https://github.com/xbill9/lazy-gemma`\n`make_splits.py` takes the 7,473 GSM8K training problems for training and draws 250 of the 1,319 test problems, by seed 0, for the held-out set. Both are written to `data/` with the source files' sha256 in `data/manifest.json`. The trainer never sees a held-out problem.\n\nGrading is code: the number after `Answer:` in the reply, compared with the reference. The prompt and grader come from my earlier GSM8K work (see Prior Work).\n\nThe untrained model first, then llama.cpp's own way to think less: `thinking_budget_tokens` on each request, which ends the thinking block at that many tokens.\n\n```\nuncapped    {\"n\": 250, \"right\": 227, \"accuracy\": 0.908, \"truncated\": 2, \"mean_tokens\": 887.8, \"median_tokens\": 788,   \"tokens_per_correct\": 977.8, \"seconds_per_correct\": 9.06}\nbudget 256  {\"n\": 250, \"right\": 228, \"accuracy\": 0.912, \"truncated\": 0, \"mean_tokens\": 572.8, \"median_tokens\": 517.0, \"tokens_per_correct\": 628.0, \"seconds_per_correct\": 5.8}\n```\n\nThe cap at 256 keeps accuracy and cuts tokens per correct answer by 35.8%. That is the number a fine-tune has to beat.\n\nThe deliverable is a 4-bit file for small cards. A model trained in 16-bit and rounded to 4-bit afterwards ends up with weights it never trained on. So `qat_lora.py` trains through the 4-bit weights themselves:\n\nWith the adapter at zero, the model's weights equal the file's:\n\n```\nblk.0.ffn_gate.weight    exact=True mismatched=0/9437184\nblk.20.attn_q.weight     exact=True mismatched=0/3145728\nblk.34.ffn_down.weight   exact=True mismatched=0/18874368\nblk.7.inp_gate.weight    exact=True mismatched=0/393216\ntoken_embd.weight        exact=True mismatched=0/402653184\n{\"top1_agree\": 510, \"of\": 512, \"rate\": 0.9961}\n```\n\nIts next-token choice matches llama-server's on 510 of 512 tokens of a real reply.\n\n🔎 **Tip: the lazy route is a few hundred lines of PyTorch.** The usual recipe for a 4-bit fine-tune is QLoRA: transformers, PEFT and bitsandbytes. It trains against bitsandbytes' own 4-bit format, NF4, so the adapter learns weights that get rounded a second time on the way into a Q4_0 GGUF. llama.cpp's trainer reads the GGUF directly, and at the commit used here it stops on Gemma 4 with `unsupported unary op for backward pass: TANH`; its `llama-finetune` trains full weights only. Reading the weights out of the GGUF and writing the loop around them took `qat_lora.py` (216 lines) and `train.py` (179 lines), and the two checks above show the model it trains is the shipped file. The loop itself, AdamW with warmup, a cosine schedule and gradient clipping, is what TRL's trainer does in one call. The 4-bit grid is the part none of those libraries offered. Step 8 runs the standard recipe on the same data and measures the difference.\n\nThree choices make training fit on the 3050:\n\nOne step of 8 sequences at 1,024 tokens:\n\n```\n{\"load_s\": 26.5, \"load_peak_mib\": 2434, \"unfilled\": [], \"lora_params\": 24158208}\n{\"step\": 1, \"loss\": 16.508, \"s_per_step\": 35.68, \"peak_mib\": 4725, \"reserved_mib\": 5120}\n```\n\nThat uses 4.7 GB of the card's 8 GB. Overfitting one reply at lr 1e-3 brings the loss from 0.1831 to 0.0235 in 12 steps and moves 24,385,598 of the 1,835,532,288 4-bit weights, so the gradients reach the grid.\n\nThe standard QLoRA stack meets the same two limits on this card. Gemma 4's per-layer table is 4.7 GB in 16-bit, and Hugging Face's Trainer and accelerate place the whole model on the GPU, table included, so the first backward pass runs out of memory. `qlora_train.py` moves the table to the CPU, marks the model as split across CPU and GPU so neither library moves it back, and computes the loss in the same chunks. Then it fits:\n\n```\n{\"load_s\": 3.9, \"load_peak_mib\": 6201, \"lora_params\": 24158208}\n{\"ple_device\": \"cpu\", \"gpu_mib\": 1785}\n```\n\nThe peak is the table passing through the GPU while loading; training starts from 1,785 MiB. Same adapter size, 24,158,208 values, at 17.1 s per step.\n\n`sample_train.py` asks the served model each of 1,000 training problems four times, at temperature 1.0, on a 4-slot llama-server. It keeps the exact prompt and reply token ids, so the trainer sees what the server saw.\n\n```\n{\"samples\": 4000, \"right\": 3524, \"problems\": 1000}\n```\n\n4,000 samples, 3,707,725 generated tokens, took 4 h 1 min on the 3050: 256 tokens per second across the four slots. This is the longest step of the loop, and it is pure inference. 948 problems got at least one right answer. The shortest right answer averages 757.2 tokens against 858.3 for all right answers, so the targets are 12% shorter than the model's typical right answer.\n\n`train.py` picks, in code, the shortest right answer per problem whose thinking finished, then trains two passes at lr 1e-4, batch 8:\n\n```\n{\"problems\": 1000, \"with_right\": 948, \"train_seqs\": 931, \"mean_gen_len\": 723.8}\n{\"step\": 234, \"of\": 234, \"epoch\": 1, \"loss\": 0.2107, \"gnorm\": 1.525, \"lr\": 0.0, \"elapsed_s\": 7822, \"peak_mib\": 4896}\n```\n\n234 steps in 2 h 10 min, 33.4 s per step, 4,896 MiB peak.\n\nA second run, B, writes a thinking budget into the prompt (\"Think for at most 256 tokens\") set to each answer's own length, and pairs each problem's shortest answer with a longer one at a bigger budget. 1,411 sequences, 354 steps, 3 h 28 min, 4,899 MiB peak.\n\n`export_gguf.py` copies the served GGUF and overwrites only the 4-bit weights of the trained layers, under the original scales. A zero adapter writes back the identical file:\n\n``` php\nlevels changed 0 of 1835532288 in trained tensors -> zero.gguf\nzero adapter: byte-identical\n```\n\nFor run A:\n\n``` php\nlevels changed 238829 of 1835532288 in trained tensors -> gemma-4-E2B-it-lazy-q4_0.gguf\n```\n\nThe result loads in the same llama-server with the same flags, at the same size, on any card that ran the original.\n\nThe same 250 problems, the same server flags, greedy:\n\n```\nrun A                      {\"right\": 220, \"accuracy\": 0.88,  \"mean_tokens\": 816.0, \"median_tokens\": 707.0, \"tokens_per_correct\": 927.2}\nrun B, plain prompt        {\"right\": 229, \"accuracy\": 0.916, \"mean_tokens\": 867.6, \"median_tokens\": 780.0, \"tokens_per_correct\": 947.2}\nrun B, prompt budget 128   {\"right\": 221, \"accuracy\": 0.884, \"mean_tokens\": 801.7, \"median_tokens\": 721.0, \"tokens_per_correct\": 906.9}\nrun B, prompt budget 256   {\"right\": 224, \"accuracy\": 0.896, \"mean_tokens\": 809.3, \"median_tokens\": 740.5, \"tokens_per_correct\": 903.2}\nrun B, prompt budget 512   {\"right\": 225, \"accuracy\": 0.9,   \"mean_tokens\": 822.3, \"median_tokens\": 755.0, \"tokens_per_correct\": 913.7}\nrun A at lr 1e-3           {\"right\": 226, \"accuracy\": 0.904, \"mean_tokens\": 828.3, \"median_tokens\": 706.0, \"tokens_per_correct\": 916.2}\nQLoRA at lr 1e-3           {\"right\": 228, \"accuracy\": 0.912, \"mean_tokens\": 878.4, \"median_tokens\": 785.5, \"tokens_per_correct\": 963.2}\n```\n\nRun A cuts tokens per correct answer by 5.2% and loses 7 right answers, so it fails the equal-accuracy rule. Run B's plain prompt cuts them by 3.1% with accuracy held. A budget written in the prompt moves tokens by less than 11 between 128 and 512. Run A at lr 1e-3 cuts them by 6.3% and gets 226 right, one fewer than the untrained model.\n\nThe QLoRA row is the standard recipe from Step 4: transformers' Trainer, PEFT and bitsandbytes' NF4 format, on run A's data and settings at lr 1e-3, exported by the same script. It trains in 1 h 6 min, half the time of run A, and gets 228 right and cuts tokens per correct answer by 1.5%. Run A at the same rate cuts them by 6.3%, or 4.9% fewer tokens than QLoRA: 916.2 against 963.2. Its export moved 52,531,066 of the 4-bit weights, 2.862%, against run A's 7.353%. An adapter trained on NF4 loses part of its correction when the weights are rounded into Q4_0.\n\n🔎 **Tip: count the 4-bit weights that moved.** Run A's export changed 238,829 of 1,835,532,288 weights, 0.013%; run B's changed 0.041%. A correction smaller than half a 4-bit step rounds away, so an adapter can train for hours and leave the shipped file nearly unchanged. At lr 1e-3 the same training moved 134,965,355, or 7.35%, 565 times as many as at lr 1e-4. Tokens per correct answer fell from 927.2 to 916.2. Once the weights move, the limit is the data: the shortest of four right answers is only 11.8% shorter than a typical right answer, so the model has little shorter thinking to learn from.\n\n|  | llama.cpp thinking cap 256 | Tuned A | Tuned A, lr 1e-3 | QLoRA, lr 1e-3 | Tuned B, plain prompt | \n|---|---|---|---|---|---|\n| Right of 250 | 🥈 228 | 220 | 226 | 🥈 228 | 🥇 229 | \n| Tokens per correct answer | 🥇 628.0 | 927.2 | 🥈 916.2 | 963.2 | 947.2 | \n| Training needed | 🟢 none | 2 h 10 min on the 3050 | 2 h 10 min on the 3050 | 1 h 6 min on the 3050 | 3 h 28 min on the 3050 | \n| Works with any client | ⚠️ needs the request field | 🟢 yes | 🟢 yes | 🟢 yes | 🟢 yes | \n| Fewer tokens at equal accuracy | 🟢 yes | ❌ no, accuracy fell | ⚠️ 6.3%, one fewer right | ⚠️ 1.5%, small | ⚠️ 3.1%, small | \n\nTo make Gemma 4 E2B think less today, set `thinking_budget_tokens` to 256 on the request: it costs nothing and cuts tokens per correct answer by 35.8% on these problems. To fine-tune Gemma 4 E2B cheaply, the 8 GB card that serves it is enough: sample, train and grade on the 3050 in under 7 hours. For the training step, the standard QLoRA stack is the quick route, an hour for two passes, once the per-layer table goes to the CPU and the loss is chunked. Training through the shipped 4-bit weights takes twice as long, fits in under 5 GB, and here cut more tokens per correct answer.\n\n`gen_eval.py` in my earlier Gemma 4 TPU work (`gemma4-dev/jev-tpu-v5e1`).` gguf_exact.py` in my earlier Gemma 4 llama.cpp work (`gemma4-dev/local-llamacpp-1650ti-4b-q4_0`), published as Every step of this project needs the weights on the desk.\n\n`ggml_compute_backward: unsupported unary op for backward pass: TANH`, from the model's output softcap. Because the code is open, the line is findable (`src/models/gemma4.cpp:436`), and so is the workaround: train in PyTorch on the same weights.\nThe payoff available today comes from llama.cpp, with no training: the 256-token thinking cap takes the 3050 from 9.06 s to 5.80 s per correct answer, and from 977.8 to 628.0 tokens.\n\nOn a slow device the tokens matter more. The Snapdragon 850 laptop from my earlier work decodes E2B at 5.00 tokens per second (measured 2026-10-11, `gemma4-dev/local-llamacpp-sdm850-2b-q4_0`). By arithmetic, 349.8 fewer tokens per correct answer is about 70 s less waiting per answer there. Not measured on that laptop in this project.\n\n**Gemma**: Gemma 4 E2B, fine-tuned locally on an 8 GB RTX 3050 through its quantization-aware 4-bit weights and served with llama.cpp.\n\nThe goal of this project was to show that a low-end 8 GB GPU that serves Gemma 4 E2B can also fine-tune it, by teaching the model to think less. The 3050's inference speed covers the sampling; keeping Gemma 4's large tables off the GPU makes the training fit, with either trainer. The results were:\n\nScope: one RTX 3050 OEM, llama.cpp `b11696-3-g7182ef71e`, one greedy run of 250 held-out GSM8K problems per model, four samples per training problem on 1,000 problems. Seconds per correct answer on the 3050 are inflated for some early baseline rows, which ran beside a CPU-side export test; token counts are unaffected. The Snapdragon 850 figure comes from another project and is not re-measured here.\n\nThe strategy for fine-tuning Gemma 4 on a low-end 8 GB GPU was validated with an incremental step by step approach.", "url": "https://wpnews.pro/news/fine-tuning-gemma-4-on-an-8-gb-rtx-3050-slow-low-end-and-still-viable", "canonical_source": "https://dev.to/xbill/fine-tuning-gemma-4-on-an-8-gb-rtx-3050-slow-low-end-and-still-viable-291f", "published_at": "2026-10-21 18:30:33+00:00", "updated_at": "2026-10-21 18:39:55.958330+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "mlops", "ai-research", "generative-ai"], "entities": ["Gemma 4 E2B", "NVIDIA RTX 3050", "llama.cpp", "Google", "Hugging Face", "GSM8K", "QLoRA", "lazy-gemma"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fine-tuning-gemma-4-on-an-8-gb-rtx-3050-slow-low-end-and-still-viable", "markdown": "https://wpnews.pro/news/fine-tuning-gemma-4-on-an-8-gb-rtx-3050-slow-low-end-and-still-viable.md", "text": "https://wpnews.pro/news/fine-tuning-gemma-4-on-an-8-gb-rtx-3050-slow-low-end-and-still-viable.txt", "jsonld": "https://wpnews.pro/news/fine-tuning-gemma-4-on-an-8-gb-rtx-3050-slow-low-end-and-still-viable.jsonld"}}