LoRA Fine-Tuning Rank Sweep: 12 Steps, 90 Min [2026]

A tutorial published by Shattered lays out a 12-step LoRA rank sweep that trains the same adapter at five ranks on a single GPU, scores each run against a held-out set, and merges the winning adapter into a standalone model in about 90 minutes, built on Hugging Face's PEFT v0.21.3 and TRL v1.18.0. The guide cites Hugging Face documentation putting QLoRA's 4-bit loading memory saving at up to four times versus a standard LoRA load, and notes that on October 29, 2026 Hugging Face announced day-zero TRL fine-tuning support for Mistral Large 4, with TRL v1.18.0 reportedly adding Gemma 4 variants and newer Qwen3.8 revisions. It also cites the 2025 Stack Overflow Developer Survey finding that 84% of developers use or plan to use AI tools.

Hugging Face’s newest releases are PEFT v0.21.3 https://whatsnew.fyi/product/peft/latest and TRL v1.18.0. That pair now sits underneath most LoRA fine-tuning jobs, so a rank sweep built on it tests the stack you will actually deploy. The lab below trains the same adapter at five ranks on one GPU, scores every run against a held-out set, and merges the winner into a standalone model. Most LoRA guides stop after one training run and one lucky output. That leaves the questions that cost real money unanswered. Does rank 64 beat rank 8? Is alpha doing anything? Did the adapter learn the task, or did it memorize the templates? The sweep answers those with numbers from your own hardware in about 90 minutes. For the single-run, memory-first route, see the walkthrough on fine-tuning an LLM with QLoRA https://shattered.io/fine-tune-llm-qlora-12-steps-2026/ . A rank sweep picks up once that first run works and asks which settings were worth the GPU hours. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F Why LoRA Fine-Tuning Still Wins in November 2026 LoRA freezes every weight in the base model and trains two small matrices beside each adapted layer. The product of those matrices acts as a learned correction to the original weight. Because only the small matrices receive gradients, the optimizer state shrinks from billions of values to a few million, and the saved artifact is a file you can email instead of a 15 GB checkpoint. QLoRA adds one step. The frozen base loads in 4-bit precision, and the adapter trains in higher precision on top of it. Hugging Face’s documentation puts the memory saving from 4-bit loading at up to four times compared with a standard LoRA load, which is the difference between a job that fits on a 24 GB card and one that does not. The method also keeps scaling. On October 29, 2026, Hugging Face announced day-zero TRL fine-tuning support for Mistral Large 4 and told users to upgrade transformers , trl , and peft to use it. TRL’s own release notes reportedly list a LoRA supervised fine-tuning example for Mistral Large 4 on a single node of eight B300 GPUs over FP8 weights. TRL v1.18.0 also reportedly adds support for Gemma 4 variants and newer Qwen3.8 revisions. The same three libraries and the same LoraConfig object cover a 1.5B test model and a trillion-parameter mixture-of-experts release. Demand is not hypothetical. The AI section of the 2025 Stack Overflow Developer Survey https://survey.stackoverflow.co/2025/technology reported that 84% of developers use or plan to use AI tools. Teams that already run open weights want to bend them to private tasks, and LoRA is the cheapest way to do it. Anyone choosing a base model first should skim the open-weight models spec sheet https://shattered.io/open-weight-ai-models-side-by-side-the-spec-sheet/ , then come back here to adapt it. The wider AI and machine learning archive https://shattered.io/category/ai-machine-learning/ tracks which families keep shipping. The table compares LoRA with the other ways to change a model’s behavior. Memory figures are relative, because the absolute numbers depend on model size, sequence length, and batch size. | Approach | What changes | Relative memory | Best used when | |---|---|---|---| | Prompting | Nothing in the model | None | The task fits in a short instruction and a few examples | | Retrieval RAG | The context window | Low | The answer lives in documents that change often | | LoRA | Small adapter matrices | Moderate | You need a fixed output format, tone, or narrow skill | | QLoRA | Adapters over a 4-bit base | Up to four times lower than LoRA per Hugging Face docs | The base model barely fits your GPU in 16-bit | | Full fine-tuning | Every weight | Highest | You have a large dataset, many GPUs, and a reason adapters fall short | Fine-tuning does not replace retrieval. It teaches behavior, not facts. A ticket classifier that must always answer in strict JSON is a textbook LoRA job. A chatbot that must quote this week’s price list is not. How Rank, Alpha and Dropout Change a LoRA Run Every LoRA setting boils down to four numbers and one list. Rank r sets how many columns the adapter matrices carry, which caps how much the adapter can express. Alpha lora alpha scales the adapter’s output, and the effective multiplier is alpha divided by rank. Dropout randomly zeroes adapter activations during training to curb overfitting. The learning rate then decides how hard each step pushes. The list, target modules , names which layers get adapters. Trainable parameters grow linearly with rank. For one adapted weight matrix with d in inputs and d out outputs, the adapter adds r d in + d out values. Doubling the rank doubles that cost across every targeted layer. This is why the sweep prints the trainable-parameter count next to each score. A rank that wins by half a point while tripling the file size rarely deserves to ship. Alpha trips up newcomers. Because the multiplier is alpha over rank, raising rank while holding alpha fixed quietly shrinks the adapter’s influence. The sweep therefore holds the ratio constant at alpha equal to twice the rank, a widely used convention, so each run changes capacity and nothing else. PEFT also offers rank-stabilized scaling through use rslora=True , which divides by the square root of rank instead. Treat it as a second experiment, not part of the first. On dropout, the Hugging Face tooling defaults to 0.05, and its documentation calls 0.05 to 0.1 the typical range. A narrow, templated task like the one in this lab has little to overfit against at low rank, so the default is fine. Check your license before you train, too. The license comparison for six open-weight families https://shattered.io/open-weight-models-license-comparison-6-families-16gb-2026/ shows which ones allow derivative adapters in commercial products. The starting values below are common community defaults, not official Hugging Face recommendations. The sweep exists to replace them with measured values for your task. | Setting | Starting value | Sweep range | What to watch | |---|---|---|---| | Rank r | 16 | 4, 8, 16, 32, 64 | Score gain per extra trainable parameter | | Alpha | 2 x rank | Fixed ratio first | Divergence if the ratio is too high | | Dropout | 0.05 | 0.0 to 0.1 | Gap between training loss and eval score | | Learning rate | 2e-4 | 1e-4 to 3e-4 | Loss spikes in the first 50 steps | | Target modules | all-linear | Attention only vs all | Quality versus adapter size | | Epochs | 2 | 1 to 3 | Eval score falling while loss keeps dropping | Prerequisites and Pinned Versions The lab runs on one NVIDIA GPU under Linux or WSL2. A 16 GB card handles the default 1.5B-class model in bfloat16 without any quantization. Moving to a 7B or 8B model with the --q4 flag typically lands around 8 to 12 GB of VRAM in practice, since 4-bit weights alone take roughly 4 to 5 GB and activations, adapter state, and framework overhead fill the rest. Sequence length and batch size move that number more than anything else. Anyone sizing hardware for bigger models can check the memory table in the local Qwen3.8-27B guide https://shattered.io/run-llm-locally-qwen3-8-27b-llama-cpp-12-steps-2026/ . Pin the Hugging Face stack. PEFT, TRL, and Transformers all moved within the last week, and a mismatched trio is the single most common reason a copied script fails. The table lists what the lab uses. | Component | Version | Notes | |---|---|---| | Python | 3.11 or newer | Any current 3.x release works | | PyTorch | Latest stable build for your CUDA driver | Install it first, from the selector on pytorch.org | | Transformers | 5.23.0 | v5.23.0 unified six new model families https://nownessai.com/article/hugging-face-transformers-v5-23-0-release in its November 13 release | | PEFT | 0.21.3 | Released November 13, 2026, with a fix for saving DTensor tensor-parallel adapters | | TRL | 1.18.0 | Released November 13, 2026, with a stable AsyncGRPOTrainer | | bitsandbytes | Latest release | Only needed for the --q4 path | | datasets, accelerate | Latest release | No special pins | Two version notes save debugging time. PEFT v0.21.2 fixed encoder-decoder models when paired with Transformers 5.18.0 or later, so anything older than that combination is worth upgrading. And FP8 checkpoints require Transformers 5.0.0 or later, which matters if you later move to the largest releases. You also need about 20 GB of free disk for the model cache, the run folders, and the merged copy. A Hugging Face account is only required for gated models. The default Qwen test model downloads without a login. Renting a GPU works just as well as owning one. Any cloud instance with a single NVIDIA card of 16 GB or more can run every script unchanged, and the whole sweep finishes inside one session, so you pay for a few hours at most. Budget extra time for the first model download, which depends on your connection rather than the GPU. If you plan several sweeps on different bases, point HF HOME at a large disk so each model downloads once. Project Layout for the Rank Sweep Lab The project is a ticket classifier. Given a short support message, the model must answer with a JSON object holding a category billing, login, bug, or feature and a priority high, medium, or low . The task is narrow on purpose. It has a crisp pass/fail metric, it needs no private data, and a 1.5B model can learn it in minutes, so five full runs fit in one session. The default base is a small Qwen instruct model, picked for speed rather than prestige. Swap in any other family through one environment variable, and the DeepSeek, Qwen, and Mistral comparison https://shattered.io/deepseek-vs-qwen-vs-mistral-open-weight-models-2026/ shows how those three differ before you commit GPU time to one. Training data comes from a script rather than a download. That choice removes dataset licensing questions, makes every run reproducible from a seed, and lets the evaluation set contain phrasings the training set never shows. A quarter of the sentence templates and a quarter of the urgency hints appear only in the held-out file, which tests whether an adapter generalized or memorized. lora-sweep/ requirements.txt pinned package list check env.py prints versions and GPU facts make data.py builds data/train.jsonl and data/eval.jsonl train.py one LoRA run, parameterized by rank, alpha, lr evaluate.py scores the base model or any adapter sweep.sh loops train.py and evaluate.py over ranks report.py merges run summaries into one table merge.py folds the winning adapter into a full model runs/ adapters and per-run summaries created by train.py results/ per-run metrics created by evaluate.py The split between train.py and evaluate.py is deliberate. Training and scoring run as separate processes, so a crash during evaluation never costs you a finished adapter, and the scorer can grade any model folder on disk, including the merged one at the end. Every random choice flows from a seed. The data script seeds Python’s generator, and the trainer takes a seed flag, so a rerun on the same hardware should land very close to the first result. Small differences still appear between GPU models because floating-point kernels are not bit-identical across architectures. Create the folder and the requirements file first. PyTorch is deliberately absent from the list, because the right build depends on your CUDA driver and you should install it from the official selector. mkdir -p lora-sweep/data && cd lora-sweep cat requirements.txt <<'EOF' transformers==5.23.0 peft==0.21.3 trl==1.18.0 datasets accelerate bitsandbytes EOF Set Up the Environment Steps 1 to 3 Step 1: Confirm the GPU and driver Run one query before installing anything. It tells you the card, its memory, and the driver version, which decides which PyTorch build to pick. nvidia-smi --query-gpu=name,memory.total,driver version --format=csv Example output your card and driver will differ : name, memory.total MiB , driver version NVIDIA GeForce RTX 4090, 24564 MiB, 5xx.xx If the command is missing, the NVIDIA driver is not installed or WSL2 has no GPU passthrough. Fix that first. Nothing later works without it. Step 2: Create a virtual environment and install the pins An isolated environment keeps this lab from fighting whatever else lives on the machine. Install PyTorch first from the build matching your driver, then the pinned Hugging Face stack. python3 -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip install PyTorch here using the command from pytorch.org for your CUDA version pip install -r requirements.txt Running the install inside WSL2 is fine, provided the Windows NVIDIA driver is current and nvidia-smi works from the Linux shell. Do not install a separate Linux driver inside WSL2. Doing so breaks the passthrough that the Windows driver provides. On a shared server, check nvidia-smi for other jobs before you start, because a neighbor holding 10 GB of memory turns a safe batch size into an out-of-memory error. The resolver should finish without conflicts. If it reports one, jump to the troubleshooting section before editing any pins. A forced install of mismatched versions produces errors that look like bugs in your code. Step 3: Verify versions before you write any training code A short script catches version drift in a second. Save it as check env.py . python import importlib.metadata as md import torch for pkg in "torch", "transformers", "peft", "trl", "bitsandbytes", "datasets", "accelerate" : try: print f"{pkg:14s}{md.version pkg }" except md.PackageNotFoundError: print f"{pkg:14s}MISSING" print "cuda available:", torch.cuda.is available if torch.cuda.is available : print "gpu:", torch.cuda.get device name 0 print "bf16 supported:", torch.cuda.is bf16 supported Run python check env.py . The three lines that matter should read transformers 5.23.0 , peft 0.21.3 , and trl 1.18.0 , with cuda available: True below them. If bf16 reports False, your GPU predates bfloat16 support. Swap bf16=True for fp16=True in train.py and expect a little more numerical fuss. Build the Ticket Dataset and the Baseline Steps 4 and 5 Step 4: Generate the training and evaluation data The generator draws a sentence template, an urgency hint, and optional noise such as a ticket number or a mobile signature. Training uses the first three templates of each group. Evaluation uses all four, so one in four eval tickets carries wording the adapter never saw. python import json import random from pathlib import Path random.seed 7 TEMPLATES = { "billing": "I was charged twice for {item}.", "The invoice for {item} shows the wrong amount.", "Please refund my last payment for {item}.", "My card was declined but {item} still billed me.", , "login": "I cannot sign in to {item}.", "The password reset email for {item} never arrives.", "Two-factor codes for {item} keep getting rejected.", "{item} logs me out every few minutes.", , "bug": "{item} crashes when I open the settings page.", "Exporting from {item} produces an empty file.", "{item} freezes after the latest update.", "Charts in {item} show yesterday's numbers.", , "feature": "Can you add dark mode to {item}?", "We need a CSV import in {item}.", "Please support single sign-on for {item}.", "It would help if {item} had a weekly summary email.", , } ITEMS = "the Pro plan", "the mobile app", "the desktop client", "the analytics dashboard", "the team workspace", "the API console" HINTS = { "high": "This blocks our whole team.", "We are losing money every hour.", "Customers are complaining right now.", "Our launch is tomorrow and this stops it." , "medium": "It is annoying but we have a workaround.", "Please look at it this week.", "It slows us down a little.", "We would like an answer soon." , "low": "No rush, just flagging it.", "Whenever you get a chance.", "Nice to have, not urgent.", "Only a minor annoyance." , } INSTRUCTION = 'Classify the support ticket. Reply with JSON only, using the keys ' '"category" billing, login, bug, feature and "priority" high, medium, low .' def make n, held out : k = 4 if held out else 3 rows = for in range n : cat = random.choice list TEMPLATES pri = random.choice list HINTS sentence = random.choice TEMPLATES cat :k .format item=random.choice ITEMS hint = random.choice HINTS pri :k extra = random.choice "", f" Ticket ref {random.randint 1000, 9999 }.", " Thanks, Sam.", " Sent from my phone." rows.append f"{sentence} {hint}{extra}", {"category": cat, "priority": pri} return rows def write path, rows, with label : with open path, "w" as f: for text, label in rows: rec = { "prompt": {"role": "user", "content": f"{INSTRUCTION}\n\nTicket: {text}"} , "completion": {"role": "assistant", "content": json.dumps label } , } if with label: rec "label" = label f.write json.dumps rec + "\n" Path "data" .mkdir exist ok=True write "data/train.jsonl", make 1800, held out=False , with label=False write "data/eval.jsonl", make 200, held out=True , with label=True print "wrote 1800 train rows and 200 eval rows" Run python make data.py and peek at one row with head -n 1 data/eval.jsonl . The prompt and completion are separate fields. That prompt-completion layout matters, because TRL then computes the loss on the completion only, so the model never gets graded on repeating the instruction. Step 5: Score the untouched base model Never train before you measure. The baseline sets the bar every adapter must clear, and it exposes whether the task is already easy for the model. It also beats leaderboard shopping, since the open-model rankings that disagree with each other https://shattered.io/open-source-llm-leaderboards-disagree-kimi-k3-1-pro-glm-5-3-2026/ say little about your private task. Save this as evaluate.py . The sweep reuses it unchanged. python import argparse import json import os import re import torch from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig p = argparse.ArgumentParser p.add argument "--model", default=os.environ.get "BASE MODEL", "Qwen/Qwen2.5-1.5B-Instruct" p.add argument "--adapter", default=None p.add argument "--name", default="base" p.add argument "--q4", action="store true" p.add argument "--limit", type=int, default=0 args = p.parse args tok = AutoTokenizer.from pretrained args.model, padding side="left" if tok.pad token is None: tok.pad token = tok.eos token kw = {"device map": {"": 0}} if args.q4: kw "quantization config" = BitsAndBytesConfig load in 4bit=True, bnb 4bit quant type="nf4", bnb 4bit compute dtype=torch.bfloat16, bnb 4bit use double quant=True else: kw "dtype" = torch.bfloat16 model = AutoModelForCausalLM.from pretrained args.model, kw if args.adapter: model = PeftModel.from pretrained model, args.adapter model.eval rows = json.loads line for line in open "data/eval.jsonl" if args.limit: rows = rows : args.limit def parse text : m = re.search r"\{. ?\}", text, re.S if not m: return None try: return json.loads m.group 0 except json.JSONDecodeError: return None preds = for i in range 0, len rows , 16 : chunk = rows i : i + 16 texts = tok.apply chat template r "prompt" , tokenize=False, add generation prompt=True for r in chunk enc = tok texts, return tensors="pt", padding=True, add special tokens=False .to model.device with torch.no grad : gen = model.generate enc, max new tokens=48, do sample=False preds += tok.batch decode gen :, enc "input ids" .shape 1 : , skip special tokens=True valid = cat = pri = exact = 0 for r, out in zip rows, preds : obj = parse out if obj is None: continue valid += 1 c = obj.get "category" == r "label" "category" q = obj.get "priority" == r "label" "priority" cat += c pri += q exact += c and q n = len rows res = {"name": args.name, "n": n, "valid json": valid / n, "category acc": cat / n, "priority acc": pri / n, "exact match": exact / n} os.makedirs "results", exist ok=True json.dump res, open f"results/{args.name}.json", "w" , indent=2 with open f"results/{args.name}.preds.jsonl", "w" as f: for r, out in zip rows, preds : f.write json.dumps {"gold": r "label" , "pred": out.strip } + "\n" print json.dumps res, indent=2 print "sample output:", preds 0 .strip Run python evaluate.py --name base . The script left-pads the batch, because decoder models generate from the right edge of the prompt. It also passes add special tokens=False , since the chat template already inserts any special tokens the model expects. Example output the format is real, the numbers are illustrative and will shift with the model and hardware : { "name": "base", "n": 200, "valid json": 0.78, "category acc": 0.74, "priority acc": 0.53, "exact match": 0.41 } sample output: Here is the classification: {"category": "bug", "priority": "high"} The sample line shows the typical base-model failure. It knows the answer but wraps it in chatter, and priority guesses lag category guesses because urgency lives in a subtle hint sentence. A fine-tune should fix the first problem almost immediately. The second one is where rank may or may not matter. Configure and Train the First Adapter Steps 6 to 8 Step 6: Write the training script One script handles every run in the sweep. Rank, alpha, learning rate, and dropout arrive as flags, so the sweep changes exactly one variable at a time. Passing a LoraConfig as peft config lets TRL wrap the model, which matches the pattern in a current TRL plus PEFT small-model walkthrough https://guides-ai.com/guides/lora-fine-tune-small-model/ and keeps the loop free of hand-rolled training code. Save this as train.py . python import argparse import json import os import time import torch from datasets import load dataset from peft import LoraConfig, prepare model for kbit training from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig from trl import SFTConfig, SFTTrainer p = argparse.ArgumentParser p.add argument "--model", default=os.environ.get "BASE MODEL", "Qwen/Qwen2.5-1.5B-Instruct" p.add argument "--rank", type=int, default=16 p.add argument "--alpha", type=int, default=None p.add argument "--dropout", type=float, default=0.05 p.add argument "--lr", type=float, default=2e-4 p.add argument "--epochs", type=float, default=2 p.add argument "--batch", type=int, default=8 p.add argument "--accum", type=int, default=2 p.add argument "--max-length", type=int, default=512 p.add argument "--seed", type=int, default=42 p.add argument "--q4", action="store true" p.add argument "--out", default=None args = p.parse args alpha = args.alpha if args.alpha is not None else 2 args.rank out = args.out or f"runs/r{args.rank}" tok = AutoTokenizer.from pretrained args.model if tok.pad token is None: tok.pad token = tok.eos token kw = {"device map": {"": 0}} if args.q4: kw "quantization config" = BitsAndBytesConfig load in 4bit=True, bnb 4bit quant type="nf4", bnb 4bit compute dtype=torch.bfloat16, bnb 4bit use double quant=True else: kw "dtype" = torch.bfloat16 model = AutoModelForCausalLM.from pretrained args.model, kw if args.q4: model = prepare model for kbit training model lora = LoraConfig r=args.rank, lora alpha=alpha, lora dropout=args.dropout, target modules="all-linear", bias="none", task type="CAUSAL LM" cfg = SFTConfig output dir=out, num train epochs=args.epochs, per device train batch size=args.batch, gradient accumulation steps=args.accum, learning rate=args.lr, lr scheduler type="cosine", warmup steps=10, logging steps=10, save strategy="no", bf16=True, gradient checkpointing=True, gradient checkpointing kwargs={"use reentrant": False}, max length=args.max length, seed=args.seed, report to="none", train ds = load dataset "json", data files="data/train.jsonl", split="train" trainer = SFTTrainer model=model, args=cfg, train dataset=train ds, processing class=tok, peft config=lora trainer.model.print trainable parameters trainable = sum p .numel for p in trainer.model.parameters if p .requires grad total = sum p .numel for p in trainer.model.parameters torch.cuda.reset peak memory stats t0 = time.time trainer.train logs = h for h in trainer.state.log history if "loss" in h summary = { "rank": args.rank, "alpha": alpha, "lr": args.lr, "dropout": args.dropout, "epochs": args.epochs, "seed": args.seed, "trainable params": trainable, "total params": total, "first loss": logs 0 "loss" , "final loss": logs -1 "loss" , "seconds": round time.time - t0 , "peak gb": round torch.cuda.max memory allocated / 1e9, 2 , } os.makedirs out, exist ok=True trainer.model.save pretrained f"{out}/adapter" tok.save pretrained f"{out}/adapter" json.dump summary, open f"{out}/summary.json", "w" , indent=2 print json.dumps summary, indent=2 Four lines deserve a closer look. target modules="all-linear" attaches adapters to every linear layer except the output head, which is the setting Hugging Face’s QLoRA examples favor. gradient checkpointing trades roughly a third more compute for much lower activation memory. The use reentrant flag avoids a known gradient-flow warning with PEFT. And save strategy="no" skips checkpoints, because an adapter this small is saved once at the end. Step 7: Launch one run and read the first lines Start with the middle of the sweep. If rank 16 trains cleanly, the other four will too. python train.py --rank 16 The first printed line reports how little of the model is trainable. Then the trainer logs every ten steps. Example output illustrative, exact values vary : trainable params: <N || all params: <M || trainable%: <p, well under 5 {'loss': 1.91, 'grad norm': 0.58, 'learning rate': 0.00019, 'epoch': 0.09} {'loss': 0.42, 'grad norm': 0.31, 'learning rate': 0.00016, 'epoch': 0.44} {'loss': 0.09, 'grad norm': 0.12, 'learning rate': 0.00007, 'epoch': 1.33} {'loss': 0.05, 'grad norm': 0.09, 'learning rate': 0.00001, 'epoch': 1.96} If you have a smaller card or want the 7B-class path, add --q4 and set BASE MODEL to a 7B or 8B instruct checkpoint. Nothing else changes. Step 8: Inspect the run summary When the run finishes, open runs/r16/summary.json . Four fields tell you whether the run was healthy. - first loss and final loss should differ by an order of magnitude on this task. A final loss that barely moved means the adapter never learned. - peak gb tells you how much headroom you have. If it sits near your card’s limit, drop --batch and raise --accum to keep the effective batch the same. - trainable params against total params shows the whole point of LoRA in one ratio. Expect it well under five percent. - seconds sets the budget for the sweep. Multiply it by five and add evaluation time. The adapter folder holds an adapter model.safetensors file and an adapter config.json . The config records the rank, alpha, and base model name, which is how PEFT later rebuilds the wrapper without you re-typing anything. Evaluate the Adapter Against the Baseline Step 9 Step 9: Score the adapter on held-out tickets Run the same evaluator you used for the baseline, this time pointing at the adapter folder. python evaluate.py --adapter runs/r16/adapter --name r16 Example output illustrative : { "name": "r16", "n": 200, "valid json": 1.0, "category acc": 0.97, "priority acc": 0.93, "exact match": 0.91 } sample output: {"category": "bug", "priority": "high"} Compare the two blocks line by line. The valid-JSON rate should jump to near 100% because the adapter learned to emit only the object. Category accuracy usually follows. Priority accuracy is the stricter test, since the cue is a hint sentence and a quarter of the eval hints use wording the training set never contained. The evaluator also writes every prediction next to its gold label in results/r16.preds.jsonl . A short script turns that file into a ranked list of confusions, which tells you what the adapter still gets wrong far better than one accuracy figure does. python import json from collections import Counter miss = Counter for line in open "results/r16.preds.jsonl" : rec = json.loads line try: pred = json.loads rec "pred" except json.JSONDecodeError: miss "invalid json" += 1 continue for key in "category", "priority" : if pred.get key = rec "gold" key : miss f"{key}: {rec 'gold' key } - {pred.get key }" += 1 for k, v in miss.most common 8 : print v, k On this task, the top entries usually pair neighbors such as medium mistaken for high, or bug mistaken for login when the sentence mentions a session. Those pairs point at ambiguous training signal, not at missing capacity. A cleaner label or a clearer hint beats a bigger rank every time. Pay attention to the gap between training behavior and eval behavior. Near-zero training loss with a modest exact-match score points at memorized templates. If that happens, lower the epoch count to 1, raise dropout to 0.1, or add more varied training phrasings. The fix is almost never a bigger rank. A different base can help more, and the roundup of the best open-source LLMs https://shattered.io/best-open-source-llm-october-2026-kimi-glm-deepseek/ lists five candidates worth a second sweep. One honest caveat: 200 eval tickets is a small sample. For an exact-match rate near 0.9, the standard error is about 2 points, computed as the square root of 0.9 times 0.1 divided by 200. Any two runs within that distance are statistically tied. The report in the next section applies that rule. Run the Rank Sweep and Pick a Winner Steps 10 and 11 Step 10: Loop over ranks with one script The sweep holds alpha at twice the rank, keeps the learning rate and seed fixed, and trains one adapter per rank. Save this as sweep.sh . bash /usr/bin/env bash set -euo pipefail for r in 4 8 16 32 64; do python train.py --rank "$r" --out "runs/r${r}" python evaluate.py --adapter "runs/r${r}/adapter" --name "r${r}" done python report.py Run the five ranks one after another, never side by side on one GPU. Parallel jobs fight over memory and make the peak gb and seconds columns meaningless, and they can push a run over the memory edge that a solo run would clear. If you own two cards, pin one rank per card with CUDA VISIBLE DEVICES and compare timings only across runs that shared the same hardware. Run it with bash sweep.sh and let it work. Each pass loads the base model fresh, so a crash in rank 32 never corrupts the rank 8 results. If the machine sleeps or the terminal dies, rerun only the missing ranks. Because every adapter lands in its own folder, nothing is overwritten. Before reading any numbers, decide whether this base model deserves the sweep. If you plan to adapt a coding model instead of a classifier, the best local LLM for coding https://shattered.io/best-local-llm-for-coding-free-weights-vs-2-api-2026/ comparison narrows the field. Run the same sweep on two candidates and keep the one that wins on your data. Step 11: Build the comparison table and apply the elbow rule The report script joins each run’s training summary with its evaluation scores, prints a table, and picks a winner. The rule is simple. Take the best exact-match score, then choose the smallest rank within 2 points of it. That favors a smaller adapter whenever the extra capacity buys nothing measurable. python import glob import json rows = for path in sorted glob.glob "runs/ /summary.json" : sm = json.load open path name = f"r{sm 'rank' }" try: ev = json.load open f"results/{name}.json" except FileNotFoundError: continue rows.append { sm, ev} rows.sort key=lambda r: r "rank" best = max r "exact match" for r in rows print "| rank | alpha | trainable | final loss | valid | cat | pri | exact | peak GB | sec |" print "|---|---|---|---|---|---|---|---|---|---|" for r in rows: print f"| {r 'rank' } | {r 'alpha' } | {r 'trainable params' :,} | {r 'final loss' :.3f} | " f"{r 'valid json' :.2f} | {r 'category acc' :.2f} | {r 'priority acc' :.2f} | " f"{r 'exact match' :.2f} | {r 'peak gb' } | {r 'seconds' } |" winner = min r for r in rows if r "exact match" = best - 0.02 , key=lambda r: r "rank" print f"\nwinner: {winner 'name' } rank {winner 'rank' }, exact match {winner 'exact match' :.2f} " Save it as report.py . The sweep calls it at the end, and you can rerun it any time. Example output the format is exact, the values are illustrative and not a benchmark : | rank | alpha | trainable | final loss | valid | cat | pri | exact | peak GB | sec | |---|---|---|---|---|---|---|---|---|---| | 4 | 8 | ... | 0.081 | 1.00 | 0.96 | 0.89 | 0.87 | ... | ... | | 8 | 16 | ... | 0.062 | 1.00 | 0.97 | 0.92 | 0.90 | ... | ... | | 16 | 32 | ... | 0.051 | 1.00 | 0.97 | 0.93 | 0.91 | ... | ... | | 32 | 64 | ... | 0.048 | 1.00 | 0.97 | 0.93 | 0.91 | ... | ... | | 64 | 128 | ... | 0.047 | 1.00 | 0.97 | 0.92 | 0.90 | ... | ... | winner: r8 rank 8, exact match 0.90 The shape of that table is the lesson, and it repeats across narrow tasks. Scores climb quickly at low ranks, flatten near rank 8 to 16, and sometimes dip at 64 as the larger adapter starts fitting noise. Your run may differ. A task that needs real new knowledge, such as a domain with unfamiliar vocabulary, can keep improving to higher ranks. Whatever the table says, ship the elbow, not the peak. Merge the Winner and Ship It Step 12 Step 12: Fold the adapter into a standalone model An adapter needs its base model at load time. A merged model does not, which makes it easier to hand to a runtime that knows nothing about PEFT. The merge adds the scaled adapter product into each targeted weight and then removes the adapter wrapper. Save this as merge.py . python import argparse import torch from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer p = argparse.ArgumentParser p.add argument "--model", required=True p.add argument "--adapter", required=True p.add argument "--out", required=True args = p.parse args tok = AutoTokenizer.from pretrained args.model base = AutoModelForCausalLM.from pretrained args.model, dtype=torch.bfloat16, device map={"": 0} merged = PeftModel.from pretrained base, args.adapter .merge and unload merged.save pretrained args.out, safe serialization=True tok.save pretrained args.out print "saved", args.out Run the merge against the winner, then score the merged folder with the same evaluator. Reusing evaluate.py with --model pointing at a local path proves the merge changed nothing. python merge.py --model Qwen/Qwen2.5-1.5B-Instruct --adapter runs/r8/adapter --out merged/r8 python evaluate.py --model merged/r8 --name merged r8 The merged score should land within a point of the adapter’s own score. Small differences come from rounding in bfloat16. A big drop means the merge loaded the wrong base or the wrong adapter, so check both paths before blaming the method. Merge into the 16-bit base, never into a 4-bit one. Rounding a 4-bit weight after adding an adapter delta throws away part of what the adapter learned. If you trained with --q4 , reload the base in bfloat16 for the merge, as the script does. Check the sizes before you ship. The adapter folder is small, typically tens of megabytes for a model this size, while the merged folder is the full 16-bit model. That gap is the practical argument for keeping the adapter as your versioned artifact and merging only at deploy time. Record the base model name, the exact package versions from check env.py , the rank, and the eval scores in a short text file beside each artifact. Six months from now, that note answers the question of which base the adapter belongs to. Where the merged folder goes next depends on your runtime. For a local chat endpoint, the walkthrough on running llama.cpp with Open WebUI https://shattered.io/llama-cpp-server-open-webui-gguf-12-steps-2026/ covers GGUF conversion and serving. The comparison of four local runtimes https://shattered.io/ollama-vs-lm-studio-vs-llama-cpp-vs-vllm-local-llm-runtimes-2026/ helps you pick between them. Teams that serve many adapters can skip merging entirely. Hugging Face’s SageMaker documentation describes serving multiple LoRA adapters from one vLLM endpoint, so each customer’s adapter loads on top of one shared base. The full run order Once every file exists, the whole lab runs top to bottom. Nothing below needs editing between commands. python check env.py python make data.py python evaluate.py --name base bash sweep.sh python merge.py --model Qwen/Qwen2.5-1.5B-Instruct --adapter runs/r8/adapter --out merged/r8 python evaluate.py --model merged/r8 --name merged r8 Replace r8 with whichever rank the report names as the winner. The finished project holds a deterministic dataset, five scored adapters, one comparison table, and a merged model verified against its own adapter. Common Pitfalls That Waste GPU Hours Seven mistakes account for most failed LoRA runs. Each one produces output that looks plausible, which is why they survive so long. Raising rank without watching alpha. The adapter’s output is scaled by alpha divided by rank. Bump rank from 8 to 64 with alpha fixed at 16, and the multiplier falls from 2 to 0.25. The bigger adapter then trains slower and often scores worse, which reads as proof that rank does not matter. Keep the ratio constant, as the sweep does, or test the ratio as its own variable. Training on the prompt as well as the answer. If the loss covers instruction tokens, the model spends capacity learning to recite the instruction. Use the prompt-completion layout from make data.py so TRL scores the completion only. When you convert someone else’s data, check that the format survives the trip. Letting the pad token mask the end-of-sequence token. Many tokenizers have no pad token, and the common fix is to reuse EOS. In some setups, that causes the collator to mask every EOS label, so the model never learns to stop and rambles past the JSON. Cap max new tokens in evaluation, and read a few raw outputs, not just the score. Copying a learning rate from a full fine-tune. Full fine-tuning runs use rates around 1e-5 because every weight moves. Adapters start from zero-initialized matrices and tolerate a rate more than ten times higher, which is why 2e-4 is a common LoRA start. The reverse mistake also bites. Paste 2e-4 into a full fine-tune and the model collapses within a few dozen steps. Judging the adapter on data it trained on. A model that has seen the template scores well on the template. The held-out phrasings in eval.jsonl exist to catch this. On your own project, split by source or by time, not by random row, so near-duplicates cannot leak across the line. Merging into a 4-bit base. Merging requires adding a small delta to each weight. A 4-bit weight cannot hold that delta without rounding error, and the rounding eats the adapter’s gains. Reload the base in bfloat16, merge, then re-quantize the merged result if you need a small file. Following an old tutorial against new packages. Examples from before the Transformers 5 line pass torch dtype where current code passes dtype , and older Trainer examples pass tokenizer= where current ones pass processing class= . Deprecation warnings are your cue to check the version, not to ignore the message. The pinned list in this lab exists so the script and the libraries agree. Troubleshooting LoRA Fine-Tuning The table covers the failures that show up most often, ordered roughly by how early in the lab they appear. | Symptom | Likely cause | Fix | |---|---|---| | CUDA out of memory during training | Batch, sequence length, or model too large for the card | Halve --batch and double --accum , lower --max-length , or add --q4 | | pip reports a dependency conflict | PyTorch installed after the pinned packages, or leftovers from an older environment | Create a fresh venv, install PyTorch first, then pip install -r requirements.txt | | bitsandbytes fails to find a GPU backend | Driver and CUDA runtime mismatch | Update the driver, reinstall bitsandbytes, or skip --q4 for the 1.5B default | | Loss shows nan or spikes early | Learning rate too high, or fp16 overflow | Drop --lr to 1e-4 and keep bf16=True | | Loss barely moves | Adapters not attached, or rate too low | Check that print trainable parameters reports a nonzero count, then raise --lr | | ValueError: Target modules not found | Layer names differ in this architecture | Keep all-linear , or print model and list the real module names | | Adapter scores match the baseline exactly | Evaluation ran without --adapter , or the path is wrong | Re-run with --adapter runs/r8/adapter and confirm adapter config.json exists | | Output never stops or repeats | EOS masked during training, or the wrong chat template | Cap max new tokens , inspect raw outputs, and use the model’s own template | | Error that tensors do not require grad | Gradient checkpointing without non-reentrant mode | Keep use reentrant: False , and call enable input require grads on quantized models | | Download is refused with a 401 error | Gated model without a login | Log in with the Hugging Face CLI and accept the model’s terms on its page | | Saving a tensor-parallel adapter writes sharded or full weights | Older PEFT behavior with DTensor models | Upgrade to PEFT 0.21.3, which gathers adapter weights instead of the whole sharded model | | Training crawls on Windows | Native Windows kernels and bitsandbytes support lag Linux | Run the lab inside WSL2 with GPU passthrough | When none of those match, shrink the problem. Run python train.py --rank 8 --epochs 0.1 to get a very short run, and add --limit 16 to evaluation. A failure that reproduces in under a minute takes minutes to diagnose. One that appears only after an hour of training takes the afternoon. Advanced Tips for Bigger Models and Better Adapters Repeat the winning rank with three seeds before you trust it. One seed hides variance, and the elbow rule above assumes the noise is about 2 points. Three seeds per rank at five ranks is fifteen runs, which still fits an overnight window on a single card. Report the mean and the spread, not the best run. Test the scaling and decomposition variants next. PEFT exposes rank-stabilized scaling with use rslora=True and weight-decomposed adapters with use dora=True , both as extra arguments to LoraConfig . Treat each as a separate arm of the sweep. PEFT v0.21.0 also reportedly added Riemannian-preconditioned LoRA, KaSA, Super-Tuning, and ShadowPEFT, so the menu keeps growing, and a method that wins on someone else’s benchmark deserves your own held-out test before it earns a place in production. Narrow the target modules when the file size matters. Attention-only adapters the query, key, value, and output projections are smaller than all-linear ones, and a classifier like this lab’s often loses nothing. Compare the two on the same ranks and let the table decide. Use TRL’s memory work on long sequences. Recent TRL releases added a fused language-model head, aimed at cutting memory use and letting training run at longer sequence lengths. If your real data has long documents, upgrade and rerun a memory test before you buy a bigger GPU. Add reinforcement learning after supervised tuning only when a metric needs it. TRL v1.18.0 promoted AsyncGRPOTrainer to a stable top-level trainer. For a format-following task, supervised LoRA is enough. For tasks where correctness can be checked automatically, such as code that must pass tests, a GRPO stage on top of the merged model is the natural next experiment. Scale the same recipe up when the task earns it. Hugging Face wrote the Mistral Large 4 launch around day-zero TRL support, and a Beam post on the hub reportedly describes training LoRA adapters with PEFT instead of updating all 501 billion parameters of that model. Models at that size need multi-GPU nodes, but the workflow is the one you just ran. The open-weight release of Kimi K3.1 Pro shows the same direction on the Moonshot side, which means adapter skills transfer between families. Finally, gate every adapter behind the evaluator. Keep evaluate.py and a frozen eval file in version control, and fail a pull request when exact match drops more than 2 points. A regression caught in review is cheaper than one caught by a customer. Frequently Asked Questions What is LoRA fine-tuning? LoRA fine-tuning adapts a language model by freezing its original weights and training small low-rank matrices beside the targeted layers. Only those matrices receive gradient updates, so memory use and file size drop sharply. The result is an adapter you can load on top of the base model or merge into it. What is the difference between LoRA and QLoRA? QLoRA is LoRA applied to a base model loaded in 4-bit precision. The adapter still trains in higher precision. Hugging Face’s documentation puts the memory saving from the 4-bit load at up to four times, which lets a 7B or 8B model fit on a single consumer GPU. What rank should I start with? Start at 16 with alpha at 32, then sweep. Narrow format-following tasks often plateau between 8 and 16. Tasks that add new knowledge or a new language may keep improving at higher ranks. Pick the smallest rank within about 2 points of your best score. How much VRAM does LoRA fine-tuning need? A 1.5B model in bfloat16 fits on a 16 GB card at the lab’s settings. A 7B to 8B model with 4-bit loading typically needs roughly 8 to 12 GB, because the 4-bit weights take about 4 to 5 GB and activations plus optimizer state fill the rest. Sequence length and batch size shift the total more than any other setting. Should I merge the adapter or keep it separate? Merge when you want one self-contained model for a runtime that does not understand PEFT, such as a GGUF conversion. Keep adapters separate when you serve many variants from one base, since swapping a small adapter is cheaper than loading a full model per customer. Always merge into a 16-bit base, never a 4-bit one. Is LoRA better than RAG? They solve different problems. LoRA changes how a model behaves, such as output format, tone, or a narrow skill. RAG changes what a model can see at answer time, so it suits facts that change. Many production systems use both. How many training examples do I need? Narrow format tasks often work with hundreds to a few thousand clean examples, and this lab uses 1,800. Quality beats volume. A few hundred consistent examples outperform thousands of noisy ones, and the held-out eval set tells you when more data stops helping. Can I do this without a local GPU? Yes. Any rented NVIDIA GPU with 16 GB or more runs the same scripts, and the 4-bit path stretches smaller cards. Training on CPU alone is possible in theory but impractical for anything beyond a toy example. Copy the project folder, run check env.py first, and everything else follows.