LoRA Fine-Tuning Rank Sweep: 12 Steps, 90 Min [2026]

Hugging Face’s newest releases are PEFT v0.21.3 and TRL v1.18.0. That pair now sits underneath most LoRA fine-tuning jobs, so a rank sweep built on it tests the stack you will actually deploy. The lab below trains the same adapter at five ranks on one GPU, scores every run against a held-out set, and merges the winner into a standalone model.

Most LoRA guides stop after one training run and one lucky output. That leaves the questions that cost real money unanswered. Does rank 64 beat rank 8? Is alpha doing anything? Did the adapter learn the task, or did it memorize the templates? The sweep answers those with numbers from your own hardware in about 90 minutes.

For the single-run, memory-first route, see the walkthrough on fine-tuning an LLM with QLoRA. A rank sweep picks up once that first run works and asks which settings were worth the GPU hours.

Don't miss new tech stories on Google

Add Shattered once in the Google app and our stories show up in your news suggestions.

Add now

Why LoRA Fine-Tuning Still Wins in November 2026 #

LoRA freezes every weight in the base model and trains two small matrices beside each adapted layer. The product of those matrices acts as a learned correction to the original weight. Because only the small matrices receive gradients, the optimizer state shrinks from billions of values to a few million, and the saved artifact is a file you can email instead of a 15 GB checkpoint.

QLoRA adds one step. The frozen base loads in 4-bit precision, and the adapter trains in higher precision on top of it. Hugging Face’s documentation puts the memory saving from 4-bit at up to four times compared with a standard LoRA load, which is the difference between a job that fits on a 24 GB card and one that does not.

The method also keeps scaling. On October 29, 2026, Hugging Face announced day-zero TRL fine-tuning support for Mistral Large 4 and told users to upgrade transformers, trl, and peft to use it. TRL’s own release notes reportedly list a LoRA supervised fine-tuning example for Mistral Large 4 on a single node of eight B300 GPUs over FP8 weights. TRL v1.18.0 also reportedly adds support for Gemma 4 variants and newer Qwen3.8 revisions. The same three libraries and the same LoraConfig object cover a 1.5B test model and a trillion-parameter mixture-of-experts release.

Demand is not hypothetical. The AI section of the 2025 Stack Overflow Developer Survey reported that 84% of developers use or plan to use AI tools. Teams that already run open weights want to bend them to private tasks, and LoRA is the cheapest way to do it. Anyone choosing a base model first should skim the open-weight models spec sheet, then come back here to adapt it. The wider AI and machine learning archive tracks which families keep shipping.

The table compares LoRA with the other ways to change a model’s behavior. Memory figures are relative, because the absolute numbers depend on model size, sequence length, and batch size.

Approach	What changes	Relative memory	Best used when
Prompting	Nothing in the model	None	The task fits in a short instruction and a few examples
Retrieval (RAG)	The context window	Low	The answer lives in documents that change often
LoRA	Small adapter matrices	Moderate	You need a fixed output format, tone, or narrow skill
QLoRA	Adapters over a 4-bit base	Up to four times lower than LoRA per Hugging Face docs	The base model barely fits your GPU in 16-bit
Full fine-tuning	Every weight	Highest	You have a large dataset, many GPUs, and a reason adapters fall short

Fine-tuning does not replace retrieval. It teaches behavior, not facts. A ticket classifier that must always answer in strict JSON is a textbook LoRA job. A chatbot that must quote this week’s price list is not.

How Rank, Alpha and Dropout Change a LoRA Run #

Every LoRA setting boils down to four numbers and one list. Rank (r) sets how many columns the adapter matrices carry, which caps how much the adapter can express. Alpha (lora_alpha) scales the adapter’s output, and the effective multiplier is alpha divided by rank. Dropout randomly zeroes adapter activations during training to curb overfitting. The learning rate then decides how hard each step pushes. The list, target_modules, names which layers get adapters.

Trainable parameters grow linearly with rank. For one adapted weight matrix with d_in inputs and d_out outputs, the adapter adds r * (d_in + d_out) values. Doubling the rank doubles that cost across every targeted layer. This is why the sweep prints the trainable-parameter count next to each score. A rank that wins by half a point while tripling the file size rarely deserves to ship.

Alpha trips up newcomers. Because the multiplier is alpha over rank, raising rank while holding alpha fixed quietly shrinks the adapter’s influence. The sweep therefore holds the ratio constant at alpha equal to twice the rank, a widely used convention, so each run changes capacity and nothing else. PEFT also offers rank-stabilized scaling through use_rslora=True, which divides by the square root of rank instead. Treat it as a second experiment, not part of the first.

On dropout, the Hugging Face tooling defaults to 0.05, and its documentation calls 0.05 to 0.1 the typical range. A narrow, templated task like the one in this lab has little to overfit against at low rank, so the default is fine. Check your license before you train, too. The license comparison for six open-weight families shows which ones allow derivative adapters in commercial products.

The starting values below are common community defaults, not official Hugging Face recommendations. The sweep exists to replace them with measured values for your task.

Setting	Starting value	Sweep range	What to watch
Rank ( r )	16	4, 8, 16, 32, 64	Score gain per extra trainable parameter
Alpha	2 x rank	Fixed ratio first	Divergence if the ratio is too high
Dropout	0.05	0.0 to 0.1	Gap between training loss and eval score
Learning rate	2e-4	1e-4 to 3e-4	Loss spikes in the first 50 steps
Target modules	all-linear	Attention only vs all	Quality versus adapter size
Epochs	2	1 to 3	Eval score falling while loss keeps dropping

Prerequisites and Pinned Versions #

The lab runs on one NVIDIA GPU under Linux or WSL2. A 16 GB card handles the default 1.5B-class model in bfloat16 without any quantization. Moving to a 7B or 8B model with the --q4 flag typically lands around 8 to 12 GB of VRAM in practice, since 4-bit weights alone take roughly 4 to 5 GB and activations, adapter state, and framework overhead fill the rest. Sequence length and batch size move that number more than anything else. Anyone sizing hardware for bigger models can check the memory table in the local Qwen3.8-27B guide.

Pin the Hugging Face stack. PEFT, TRL, and Transformers all moved within the last week, and a mismatched trio is the single most common reason a copied script fails. The table lists what the lab uses.

Component	Version	Notes
Python	3.11 or newer	Any current 3.x release works
PyTorch	Latest stable build for your CUDA driver	Install it first, from the selector on pytorch.org
Transformers	5.23.0	v5.23.0 unified six new model families in its November 13 release
PEFT	0.21.3	Released November 13, 2026, with a fix for saving DTensor tensor-parallel adapters
TRL	1.18.0	Released November 13, 2026, with a stable AsyncGRPOTrainer
bitsandbytes	Latest release	Only needed for the --q4 path
datasets, accelerate	Latest release	No special pins

Two version notes save debugging time. PEFT v0.21.2 fixed encoder-decoder models when paired with Transformers 5.18.0 or later, so anything older than that combination is worth upgrading. And FP8 checkpoints require Transformers 5.0.0 or later, which matters if you later move to the largest releases.

You also need about 20 GB of free disk for the model cache, the run folders, and the merged copy. A Hugging Face account is only required for gated models. The default Qwen test model downloads without a login.

Renting a GPU works just as well as owning one. Any cloud instance with a single NVIDIA card of 16 GB or more can run every script unchanged, and the whole sweep finishes inside one session, so you pay for a few hours at most. Budget extra time for the first model download, which depends on your connection rather than the GPU. If you plan several sweeps on different bases, point HF_HOME at a large disk so each model downloads once.

Project Layout for the Rank Sweep Lab #

The project is a ticket classifier. Given a short support message, the model must answer with a JSON object holding a category (billing, login, bug, or feature) and a priority (high, medium, or low). The task is narrow on purpose. It has a crisp pass/fail metric, it needs no private data, and a 1.5B model can learn it in minutes, so five full runs fit in one session. The default base is a small Qwen instruct model, picked for speed rather than prestige. Swap in any other family through one environment variable, and the DeepSeek, Qwen, and Mistral comparison shows how those three differ before you commit GPU time to one.

Training data comes from a script rather than a download. That choice removes dataset licensing questions, makes every run reproducible from a seed, and lets the evaluation set contain phrasings the training set never shows. A quarter of the sentence templates and a quarter of the urgency hints appear only in the held-out file, which tests whether an adapter generalized or memorized.

lora-sweep/
  requirements.txt   pinned package list
  check_env.py       prints versions and GPU facts
  make_data.py       builds data/train.jsonl and data/eval.jsonl
  train.py           one LoRA run, parameterized by rank, alpha, lr
  evaluate.py        scores the base model or any adapter
  sweep.sh           loops train.py and evaluate.py over ranks
  report.py          merges run summaries into one table
  merge.py           folds the winning adapter into a full model
  runs/              adapters and per-run summaries (created by train.py)
  results/           per-run metrics (created by evaluate.py)

The split between train.py and evaluate.py is deliberate. Training and scoring run as separate processes, so a crash during evaluation never costs you a finished adapter, and the scorer can grade any model folder on disk, including the merged one at the end. Every random choice flows from a seed. The data script seeds Python’s generator, and the trainer takes a seed flag, so a rerun on the same hardware should land very close to the first result. Small differences still appear between GPU models because floating-point kernels are not bit-identical across architectures.

Create the folder and the requirements file first. PyTorch is deliberately absent from the list, because the right build depends on your CUDA driver and you should install it from the official selector.

mkdir -p lora-sweep/data && cd lora-sweep
cat > requirements.txt <<'EOF'
transformers==5.23.0
peft==0.21.3
trl==1.18.0
datasets
accelerate
bitsandbytes
EOF

Set Up the Environment (Steps 1 to 3) #

Step 1: Confirm the GPU and driver

Run one query before installing anything. It tells you the card, its memory, and the driver version, which decides which PyTorch build to pick.

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv

Example output (your card and driver will differ):

name, memory.total [MiB], driver_version
NVIDIA GeForce RTX 4090, 24564 MiB, 5xx.xx

If the command is missing, the NVIDIA driver is not installed or WSL2 has no GPU passthrough. Fix that first. Nothing later works without it.

Step 2: Create a virtual environment and install the pins

An isolated environment keeps this lab from fighting whatever else lives on the machine. Install PyTorch first from the build matching your driver, then the pinned Hugging Face stack.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt

Running the install inside WSL2 is fine, provided the Windows NVIDIA driver is current and nvidia-smi works from the Linux shell. Do not install a separate Linux driver inside WSL2. Doing so breaks the passthrough that the Windows driver provides. On a shared server, check nvidia-smi for other jobs before you start, because a neighbor holding 10 GB of memory turns a safe batch size into an out-of-memory error.

The resolver should finish without conflicts. If it reports one, jump to the troubleshooting section before editing any pins. A forced install of mismatched versions produces errors that look like bugs in your code.

Step 3: Verify versions before you write any training code

A short script catches version drift in a second. Save it as check_env.py.

import importlib.metadata as md
import torch

for pkg in ["torch", "transformers", "peft", "trl", "bitsandbytes", "datasets", "accelerate"]:
    try:
        print(f"{pkg:14s}{md.version(pkg)}")
    except md.PackageNotFoundError:
        print(f"{pkg:14s}MISSING")

print("cuda available:", torch.cuda.is_available())
if torch.cuda.is_available():
    print("gpu:", torch.cuda.get_device_name(0))
    print("bf16 supported:", torch.cuda.is_bf16_supported())

Run python check_env.py. The three lines that matter should read transformers 5.23.0, peft 0.21.3, and trl 1.18.0, with cuda available: True below them. If bf16 reports False, your GPU predates bfloat16 support. Swap bf16=True for fp16=True in train.py and expect a little more numerical fuss.

Build the Ticket Dataset and the Baseline (Steps 4 and 5) #

Step 4: Generate the training and evaluation data

The generator draws a sentence template, an urgency hint, and optional noise such as a ticket number or a mobile signature. Training uses the first three templates of each group. Evaluation uses all four, so one in four eval tickets carries wording the adapter never saw.

import json
import random
from pathlib import Path

random.seed(7)

TEMPLATES = {
    "billing": [
        "I was charged twice for {item}.",
        "The invoice for {item} shows the wrong amount.",
        "Please refund my last payment for {item}.",
        "My card was declined but {item} still billed me.",
    ],
    "login": [
        "I cannot sign in to {item}.",
        "The password reset email for {item} never arrives.",
        "Two-factor codes for {item} keep getting rejected.",
        "{item} logs me out every few minutes.",
    ],
    "bug": [
        "{item} crashes when I open the settings page.",
        "Exporting from {item} produces an empty file.",
        "{item} freezes after the latest update.",
        "Charts in {item} show yesterday's numbers.",
    ],
    "feature": [
        "Can you add dark mode to {item}?",
        "We need a CSV import in {item}.",
        "Please support single sign-on for {item}.",
        "It would help if {item} had a weekly summary email.",
    ],
}
ITEMS = ["the Pro plan", "the mobile app", "the desktop client",
         "the analytics dashboard", "the team workspace", "the API console"]
HINTS = {
    "high": ["This blocks our whole team.", "We are losing money every hour.",
             "Customers are complaining right now.", "Our launch is tomorrow and this stops it."],
    "medium": ["It is annoying but we have a workaround.", "Please look at it this week.",
               "It slows us down a little.", "We would like an answer soon."],
    "low": ["No rush, just flagging it.", "Whenever you get a chance.",
            "Nice to have, not urgent.", "Only a minor annoyance."],
}
INSTRUCTION = ('Classify the support ticket. Reply with JSON only, using the keys '
               '"category" (billing, login, bug, feature) and "priority" (high, medium, low).')

def make(n, held_out):
    k = 4 if held_out else 3
    rows = []
    for _ in range(n):
        cat = random.choice(list(TEMPLATES))
        pri = random.choice(list(HINTS))
        sentence = random.choice(TEMPLATES[cat][:k]).format(item=random.choice(ITEMS))
        hint = random.choice(HINTS[pri][:k])
        extra = random.choice(["", f" Ticket ref {random.randint(1000, 9999)}.",
                               " Thanks, Sam.", " Sent from my phone."])
        rows.append((f"{sentence} {hint}{extra}", {"category": cat, "priority": pri}))
    return rows

def write(path, rows, with_label):
    with open(path, "w") as f:
        for text, label in rows:
            rec = {
                "prompt": [{"role": "user", "content": f"{INSTRUCTION}\n\nTicket: {text}"}],
                "completion": [{"role": "assistant", "content": json.dumps(label)}],
            }
            if with_label:
                rec["label"] = label
            f.write(json.dumps(rec) + "\n")

Path("data").mkdir(exist_ok=True)
write("data/train.jsonl", make(1800, held_out=False), with_label=False)
write("data/eval.jsonl", make(200, held_out=True), with_label=True)
print("wrote 1800 train rows and 200 eval rows")

Run python make_data.py and peek at one row with head -n 1 data/eval.jsonl. The prompt and completion are separate fields. That prompt-completion layout matters, because TRL then computes the loss on the completion only, so the model never gets graded on repeating the instruction.

Step 5: Score the untouched base model

Never train before you measure. The baseline sets the bar every adapter must clear, and it exposes whether the task is already easy for the model. It also beats leaderboard shopping, since the open-model rankings that disagree with each other say little about your private task. Save this as evaluate.py. The sweep reuses it unchanged.

import argparse
import json
import os
import re

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

p = argparse.ArgumentParser()
p.add_argument("--model", default=os.environ.get("BASE_MODEL", "Qwen/Qwen2.5-1.5B-Instruct"))
p.add_argument("--adapter", default=None)
p.add_argument("--name", default="base")
p.add_argument("--q4", action="store_true")
p.add_argument("--limit", type=int, default=0)
args = p.parse_args()

tok = AutoTokenizer.from_pretrained(args.model, padding_side="left")
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

kw = {"device_map": {"": 0}}
if args.q4:
    kw["quantization_config"] = BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True)
else:
    kw["dtype"] = torch.bfloat16
model = AutoModelForCausalLM.from_pretrained(args.model, **kw)
if args.adapter:
    model = PeftModel.from_pretrained(model, args.adapter)
model.eval()

rows = [json.loads(line) for line in open("data/eval.jsonl")]
if args.limit:
    rows = rows[: args.limit]

def parse(text):
    m = re.search(r"\{.*?\}", text, re.S)
    if not m:
        return None
    try:
        return json.loads(m.group(0))
    except json.JSONDecodeError:
        return None

preds = []
for i in range(0, len(rows), 16):
    chunk = rows[i : i + 16]
    texts = [tok.apply_chat_template(r["prompt"], tokenize=False, add_generation_prompt=True)
             for r in chunk]
    enc = tok(texts, return_tensors="pt", padding=True, add_special_tokens=False).to(model.device)
    with torch.no_grad():
        gen = model.generate(**enc, max_new_tokens=48, do_sample=False)
    preds += tok.batch_decode(gen[:, enc["input_ids"].shape[1]:], skip_special_tokens=True)

valid = cat = pri = exact = 0
for r, out in zip(rows, preds):
    obj = parse(out)
    if obj is None:
        continue
    valid += 1
    c = obj.get("category") == r["label"]["category"]
    q = obj.get("priority") == r["label"]["priority"]
    cat += c
    pri += q
    exact += c and q

n = len(rows)
res = {"name": args.name, "n": n, "valid_json": valid / n, "category_acc": cat / n,
       "priority_acc": pri / n, "exact_match": exact / n}
os.makedirs("results", exist_ok=True)
json.dump(res, open(f"results/{args.name}.json", "w"), indent=2)
with open(f"results/{args.name}.preds.jsonl", "w") as f:
    for r, out in zip(rows, preds):
        f.write(json.dumps({"gold": r["label"], "pred": out.strip()}) + "\n")
print(json.dumps(res, indent=2))
print("sample output:", preds[0].strip())

Run python evaluate.py --name base. The script left-pads the batch, because decoder models generate from the right edge of the prompt. It also passes add_special_tokens=False, since the chat template already inserts any special tokens the model expects.

Example output (the format is real, the numbers are illustrative and will shift with the model and hardware):

{
  "name": "base",
  "n": 200,
  "valid_json": 0.78,
  "category_acc": 0.74,
  "priority_acc": 0.53,
  "exact_match": 0.41
}
sample output: Here is the classification: {"category": "bug", "priority": "high"}

The sample line shows the typical base-model failure. It knows the answer but wraps it in chatter, and priority guesses lag category guesses because urgency lives in a subtle hint sentence. A fine-tune should fix the first problem almost immediately. The second one is where rank may or may not matter.

Configure and Train the First Adapter (Steps 6 to 8) #

Step 6: Write the training script

One script handles every run in the sweep. Rank, alpha, learning rate, and dropout arrive as flags, so the sweep changes exactly one variable at a time. Passing a LoraConfig as peft_config lets TRL wrap the model, which matches the pattern in a current TRL plus PEFT small-model walkthrough and keeps the loop free of hand-rolled training code. Save this as train.py.

import argparse
import json
import os
import time

import torch
from datasets import load_dataset
from peft import LoraConfig, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

p = argparse.ArgumentParser()
p.add_argument("--model", default=os.environ.get("BASE_MODEL", "Qwen/Qwen2.5-1.5B-Instruct"))
p.add_argument("--rank", type=int, default=16)
p.add_argument("--alpha", type=int, default=None)
p.add_argument("--dropout", type=float, default=0.05)
p.add_argument("--lr", type=float, default=2e-4)
p.add_argument("--epochs", type=float, default=2)
p.add_argument("--batch", type=int, default=8)
p.add_argument("--accum", type=int, default=2)
p.add_argument("--max-length", type=int, default=512)
p.add_argument("--seed", type=int, default=42)
p.add_argument("--q4", action="store_true")
p.add_argument("--out", default=None)
args = p.parse_args()

alpha = args.alpha if args.alpha is not None else 2 * args.rank
out = args.out or f"runs/r{args.rank}"

tok = AutoTokenizer.from_pretrained(args.model)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

kw = {"device_map": {"": 0}}
if args.q4:
    kw["quantization_config"] = BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True)
else:
    kw["dtype"] = torch.bfloat16
model = AutoModelForCausalLM.from_pretrained(args.model, **kw)
if args.q4:
    model = prepare_model_for_kbit_training(model)

lora = LoraConfig(r=args.rank, lora_alpha=alpha, lora_dropout=args.dropout,
                  target_modules="all-linear", bias="none", task_type="CAUSAL_LM")

cfg = SFTConfig(
    output_dir=out,
    num_train_epochs=args.epochs,
    per_device_train_batch_size=args.batch,
    gradient_accumulation_steps=args.accum,
    learning_rate=args.lr,
    lr_scheduler_type="cosine",
    warmup_steps=10,
    logging_steps=10,
    save_strategy="no",
    bf16=True,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    max_length=args.max_length,
    seed=args.seed,
    report_to="none",
)

train_ds = load_dataset("json", data_files="data/train.jsonl", split="train")
trainer = SFTTrainer(model=model, args=cfg, train_dataset=train_ds,
                     processing_class=tok, peft_config=lora)
trainer.model.print_trainable_parameters()

trainable = sum(p_.numel() for p_ in trainer.model.parameters() if p_.requires_grad)
total = sum(p_.numel() for p_ in trainer.model.parameters())

torch.cuda.reset_peak_memory_stats()
t0 = time.time()
trainer.train()

logs = [h for h in trainer.state.log_history if "loss" in h]
summary = {
    "rank": args.rank, "alpha": alpha, "lr": args.lr, "dropout": args.dropout,
    "epochs": args.epochs, "seed": args.seed,
    "trainable_params": trainable, "total_params": total,
    "first_loss": logs[0]["loss"], "final_loss": logs[-1]["loss"],
    "seconds": round(time.time() - t0),
    "peak_gb": round(torch.cuda.max_memory_allocated() / 1e9, 2),
}
os.makedirs(out, exist_ok=True)
trainer.model.save_pretrained(f"{out}/adapter")
tok.save_pretrained(f"{out}/adapter")
json.dump(summary, open(f"{out}/summary.json", "w"), indent=2)
print(json.dumps(summary, indent=2))

Four lines deserve a closer look. target_modules="all-linear" attaches adapters to every linear layer except the output head, which is the setting Hugging Face’s QLoRA examples favor. gradient_checkpointing trades roughly a third more compute for much lower activation memory. The use_reentrant flag avoids a known gradient-flow warning with PEFT. And save_strategy="no" skips checkpoints, because an adapter this small is saved once at the end.

Step 7: Launch one run and read the first lines

Start with the middle of the sweep. If rank 16 trains cleanly, the other four will too.

python train.py --rank 16

The first printed line reports how little of the model is trainable. Then the trainer logs every ten steps.

Example output (illustrative, exact values vary):

trainable params: <N> || all params: <M> || trainable%: <p, well under 5>
{'loss': 1.91, 'grad_norm': 0.58, 'learning_rate': 0.00019, 'epoch': 0.09}
{'loss': 0.42, 'grad_norm': 0.31, 'learning_rate': 0.00016, 'epoch': 0.44}
{'loss': 0.09, 'grad_norm': 0.12, 'learning_rate': 0.00007, 'epoch': 1.33}
{'loss': 0.05, 'grad_norm': 0.09, 'learning_rate': 0.00001, 'epoch': 1.96}

If you have a smaller card or want the 7B-class path, add --q4 and set BASE_MODEL to a 7B or 8B instruct checkpoint. Nothing else changes.

Step 8: Inspect the run summary

When the run finishes, open runs/r16/summary.json. Four fields tell you whether the run was healthy.

• first_loss andfinal_loss should differ by an order of magnitude on this task. A final loss that barely moved means the adapter never learned.
• peak_gb tells you how much headroom you have. If it sits near your card’s limit, drop--batch and raise--accum to keep the effective batch the same.
• trainable_params againsttotal_params shows the whole point of LoRA in one ratio. Expect it well under five percent.
• seconds sets the budget for the sweep. Multiply it by five and add evaluation time.

The adapter folder holds an adapter_model.safetensors file and an adapter_config.json. The config records the rank, alpha, and base model name, which is how PEFT later rebuilds the wrapper without you re-typing anything.

Evaluate the Adapter Against the Baseline (Step 9) #

Step 9: Score the adapter on held-out tickets

Run the same evaluator you used for the baseline, this time pointing at the adapter folder.

python evaluate.py --adapter runs/r16/adapter --name r16

Example output (illustrative):

{
  "name": "r16",
  "n": 200,
  "valid_json": 1.0,
  "category_acc": 0.97,
  "priority_acc": 0.93,
  "exact_match": 0.91
}
sample output: {"category": "bug", "priority": "high"}

Compare the two blocks line by line. The valid-JSON rate should jump to near 100% because the adapter learned to emit only the object. Category accuracy usually follows. Priority accuracy is the stricter test, since the cue is a hint sentence and a quarter of the eval hints use wording the training set never contained.

The evaluator also writes every prediction next to its gold label in results/r16.preds.jsonl. A short script turns that file into a ranked list of confusions, which tells you what the adapter still gets wrong far better than one accuracy figure does.

import json
from collections import Counter

miss = Counter()
for line in open("results/r16.preds.jsonl"):
    rec = json.loads(line)
    try:
        pred = json.loads(rec["pred"])
    except json.JSONDecodeError:
        miss["invalid json"] += 1
        continue
    for key in ("category", "priority"):
        if pred.get(key) != rec["gold"][key]:
            miss[f"{key}: {rec['gold'][key]} -> {pred.get(key)}"] += 1

for k, v in miss.most_common(8):
    print(v, k)

On this task, the top entries usually pair neighbors such as medium mistaken for high, or bug mistaken for login when the sentence mentions a session. Those pairs point at ambiguous training signal, not at missing capacity. A cleaner label or a clearer hint beats a bigger rank every time.

Pay attention to the gap between training behavior and eval behavior. Near-zero training loss with a modest exact-match score points at memorized templates. If that happens, lower the epoch count to 1, raise dropout to 0.1, or add more varied training phrasings. The fix is almost never a bigger rank. A different base can help more, and the roundup of the best open-source LLMs lists five candidates worth a second sweep.

One honest caveat: 200 eval tickets is a small sample. For an exact-match rate near 0.9, the standard error is about 2 points, computed as the square root of 0.9 times 0.1 divided by 200. Any two runs within that distance are statistically tied. The report in the next section applies that rule.

Run the Rank Sweep and Pick a Winner (Steps 10 and 11) #

Step 10: Loop over ranks with one script

The sweep holds alpha at twice the rank, keeps the learning rate and seed fixed, and trains one adapter per rank. Save this as sweep.sh.

#!/usr/bin/env bash
set -euo pipefail

for r in 4 8 16 32 64; do
  python train.py --rank "$r" --out "runs/r${r}"
  python evaluate.py --adapter "runs/r${r}/adapter" --name "r${r}"
done

python report.py

Run the five ranks one after another, never side by side on one GPU. Parallel jobs fight over memory and make the peak_gb and seconds columns meaningless, and they can push a run over the memory edge that a solo run would clear. If you own two cards, pin one rank per card with CUDA_VISIBLE_DEVICES and compare timings only across runs that shared the same hardware.

Run it with bash sweep.sh and let it work. Each pass loads the base model fresh, so a crash in rank 32 never corrupts the rank 8 results. If the machine sleeps or the terminal dies, rerun only the missing ranks. Because every adapter lands in its own folder, nothing is overwritten.

Before reading any numbers, decide whether this base model deserves the sweep. If you plan to adapt a coding model instead of a classifier, the best local LLM for coding comparison narrows the field. Run the same sweep on two candidates and keep the one that wins on your data.

Step 11: Build the comparison table and apply the elbow rule

The report script joins each run’s training summary with its evaluation scores, prints a table, and picks a winner. The rule is simple. Take the best exact-match score, then choose the smallest rank within 2 points of it. That favors a smaller adapter whenever the extra capacity buys nothing measurable.

import glob
import json

rows = []
for path in sorted(glob.glob("runs/*/summary.json")):
    sm = json.load(open(path))
    name = f"r{sm['rank']}"
    try:
        ev = json.load(open(f"results/{name}.json"))
    except FileNotFoundError:
        continue
    rows.append({**sm, **ev})

rows.sort(key=lambda r: r["rank"])
best = max(r["exact_match"] for r in rows)

print("| rank | alpha | trainable | final loss | valid | cat | pri | exact | peak GB | sec |")
print("|---|---|---|---|---|---|---|---|---|---|")
for r in rows:
    print(f"| {r['rank']} | {r['alpha']} | {r['trainable_params']:,} | {r['final_loss']:.3f} | "
          f"{r['valid_json']:.2f} | {r['category_acc']:.2f} | {r['priority_acc']:.2f} | "
          f"{r['exact_match']:.2f} | {r['peak_gb']} | {r['seconds']} |")

winner = min((r for r in rows if r["exact_match"] >= best - 0.02), key=lambda r: r["rank"])
print(f"\nwinner: {winner['name']} (rank {winner['rank']}, exact match {winner['exact_match']:.2f})")

Save it as report.py. The sweep calls it at the end, and you can rerun it any time.

Example output (the format is exact, the values are illustrative and not a benchmark):

| rank | alpha | trainable | final loss | valid | cat | pri | exact | peak GB | sec |
|---|---|---|---|---|---|---|---|---|---|
| 4 | 8 | ... | 0.081 | 1.00 | 0.96 | 0.89 | 0.87 | ... | ... |
| 8 | 16 | ... | 0.062 | 1.00 | 0.97 | 0.92 | 0.90 | ... | ... |
| 16 | 32 | ... | 0.051 | 1.00 | 0.97 | 0.93 | 0.91 | ... | ... |
| 32 | 64 | ... | 0.048 | 1.00 | 0.97 | 0.93 | 0.91 | ... | ... |
| 64 | 128 | ... | 0.047 | 1.00 | 0.97 | 0.92 | 0.90 | ... | ... |

winner: r8 (rank 8, exact match 0.90)

The shape of that table is the lesson, and it repeats across narrow tasks. Scores climb quickly at low ranks, flatten near rank 8 to 16, and sometimes dip at 64 as the larger adapter starts fitting noise. Your run may differ. A task that needs real new knowledge, such as a domain with unfamiliar vocabulary, can keep improving to higher ranks. Whatever the table says, ship the elbow, not the peak.

Merge the Winner and Ship It (Step 12) #

Step 12: Fold the adapter into a standalone model

An adapter needs its base model at load time. A merged model does not, which makes it easier to hand to a runtime that knows nothing about PEFT. The merge adds the scaled adapter product into each targeted weight and then removes the adapter wrapper. Save this as merge.py.

import argparse

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

p = argparse.ArgumentParser()
p.add_argument("--model", required=True)
p.add_argument("--adapter", required=True)
p.add_argument("--out", required=True)
args = p.parse_args()

tok = AutoTokenizer.from_pretrained(args.model)
base = AutoModelForCausalLM.from_pretrained(args.model, dtype=torch.bfloat16, device_map={"": 0})
merged = PeftModel.from_pretrained(base, args.adapter).merge_and_unload()
merged.save_pretrained(args.out, safe_serialization=True)
tok.save_pretrained(args.out)
print("saved", args.out)

Run the merge against the winner, then score the merged folder with the same evaluator. Reusing evaluate.py with --model pointing at a local path proves the merge changed nothing.

python merge.py --model Qwen/Qwen2.5-1.5B-Instruct --adapter runs/r8/adapter --out merged/r8
python evaluate.py --model merged/r8 --name merged_r8

The merged score should land within a point of the adapter’s own score. Small differences come from rounding in bfloat16. A big drop means the merge loaded the wrong base or the wrong adapter, so check both paths before blaming the method.

Merge into the 16-bit base, never into a 4-bit one. Rounding a 4-bit weight after adding an adapter delta throws away part of what the adapter learned. If you trained with --q4, reload the base in bfloat16 for the merge, as the script does.

Check the sizes before you ship. The adapter folder is small, typically tens of megabytes for a model this size, while the merged folder is the full 16-bit model. That gap is the practical argument for keeping the adapter as your versioned artifact and merging only at deploy time. Record the base model name, the exact package versions from check_env.py, the rank, and the eval scores in a short text file beside each artifact. Six months from now, that note answers the question of which base the adapter belongs to.

Where the merged folder goes next depends on your runtime. For a local chat endpoint, the walkthrough on running llama.cpp with Open WebUI covers GGUF conversion and serving. The comparison of four local runtimes helps you pick between them. Teams that serve many adapters can skip merging entirely. Hugging Face’s SageMaker documentation describes serving multiple LoRA adapters from one vLLM endpoint, so each customer’s adapter loads on top of one shared base.

The full run order

Once every file exists, the whole lab runs top to bottom. Nothing below needs editing between commands.

python check_env.py
python make_data.py
python evaluate.py --name base
bash sweep.sh
python merge.py --model Qwen/Qwen2.5-1.5B-Instruct --adapter runs/r8/adapter --out merged/r8
python evaluate.py --model merged/r8 --name merged_r8

Replace r8 with whichever rank the report names as the winner. The finished project holds a deterministic dataset, five scored adapters, one comparison table, and a merged model verified against its own adapter.

Common Pitfalls That Waste GPU Hours #

Seven mistakes account for most failed LoRA runs. Each one produces output that looks plausible, which is why they survive so long.

Raising rank without watching alpha. The adapter’s output is scaled by alpha divided by rank. Bump rank from 8 to 64 with alpha fixed at 16, and the multiplier falls from 2 to 0.25. The bigger adapter then trains slower and often scores worse, which reads as proof that rank does not matter. Keep the ratio constant, as the sweep does, or test the ratio as its own variable.

Training on the prompt as well as the answer. If the loss covers instruction tokens, the model spends capacity learning to recite the instruction. Use the prompt-completion layout from make_data.py so TRL scores the completion only. When you convert someone else’s data, check that the format survives the trip.

Letting the pad token mask the end-of-sequence token. Many tokenizers have no pad token, and the common fix is to reuse EOS. In some setups, that causes the collator to mask every EOS label, so the model never learns to stop and rambles past the JSON. Cap max_new_tokens in evaluation, and read a few raw outputs, not just the score.

Copying a learning rate from a full fine-tune. Full fine-tuning runs use rates around 1e-5 because every weight moves. Adapters start from zero-initialized matrices and tolerate a rate more than ten times higher, which is why 2e-4 is a common LoRA start. The reverse mistake also bites. Paste 2e-4 into a full fine-tune and the model collapses within a few dozen steps.

Judging the adapter on data it trained on. A model that has seen the template scores well on the template. The held-out phrasings in eval.jsonl exist to catch this. On your own project, split by source or by time, not by random row, so near-duplicates cannot leak across the line.

Merging into a 4-bit base. Merging requires adding a small delta to each weight. A 4-bit weight cannot hold that delta without rounding error, and the rounding eats the adapter’s gains. Reload the base in bfloat16, merge, then re-quantize the merged result if you need a small file.

Following an old tutorial against new packages. Examples from before the Transformers 5 line pass torch_dtype where current code passes dtype, and older Trainer examples pass tokenizer= where current ones pass processing_class=. Deprecation warnings are your cue to check the version, not to ignore the message. The pinned list in this lab exists so the script and the libraries agree.

Troubleshooting LoRA Fine-Tuning #

The table covers the failures that show up most often, ordered roughly by how early in the lab they appear.

Symptom	Likely cause	Fix
CUDA out of memory during training	Batch, sequence length, or model too large for the card	Halve --batch and double--accum , lower--max-length , or add--q4
pip reports a dependency conflict	PyTorch installed after the pinned packages, or leftovers from an older environment	Create a fresh venv, install PyTorch first, then pip install -r requirements.txt
bitsandbytes fails to find a GPU backend	Driver and CUDA runtime mismatch	Update the driver, reinstall bitsandbytes, or skip --q4 for the 1.5B default
Loss shows nan or spikes early	Learning rate too high, or fp16 overflow	Drop --lr to 1e-4 and keepbf16=True
Loss barely moves	Adapters not attached, or rate too low	Check that print_trainable_parameters reports a nonzero count, then raise--lr
ValueError: Target modules not found	Layer names differ in this architecture	Keep all-linear , or printmodel and list the real module names
Adapter scores match the baseline exactly	Evaluation ran without --adapter , or the path is wrong	Re-run with --adapter runs/r8/adapter and confirmadapter_config.json exists
Output never stops or repeats	EOS masked during training, or the wrong chat template	Cap max_new_tokens , inspect raw outputs, and use the model’s own template
Error that tensors do not require grad	Gradient checkpointing without non-reentrant mode	Keep use_reentrant: False , and callenable_input_require_grads() on quantized models
Download is refused with a 401 error	Gated model without a login	Log in with the Hugging Face CLI and accept the model’s terms on its page
Saving a tensor-parallel adapter writes sharded or full weights	Older PEFT behavior with DTensor models	Upgrade to PEFT 0.21.3, which gathers adapter weights instead of the whole sharded model
Training crawls on Windows	Native Windows kernels and bitsandbytes support lag Linux	Run the lab inside WSL2 with GPU passthrough

When none of those match, shrink the problem. Run python train.py --rank 8 --epochs 0.1 to get a very short run, and add --limit 16 to evaluation. A failure that reproduces in under a minute takes minutes to diagnose. One that appears only after an hour of training takes the afternoon.

Advanced Tips for Bigger Models and Better Adapters #

Repeat the winning rank with three seeds before you trust it. One seed hides variance, and the elbow rule above assumes the noise is about 2 points. Three seeds per rank at five ranks is fifteen runs, which still fits an overnight window on a single card. Report the mean and the spread, not the best run.

Test the scaling and decomposition variants next. PEFT exposes rank-stabilized scaling with use_rslora=True and weight-decomposed adapters with use_dora=True, both as extra arguments to LoraConfig. Treat each as a separate arm of the sweep. PEFT v0.21.0 also reportedly added Riemannian-preconditioned LoRA, KaSA, Super-Tuning, and ShadowPEFT, so the menu keeps growing, and a method that wins on someone else’s benchmark deserves your own held-out test before it earns a place in production.

Narrow the target modules when the file size matters. Attention-only adapters (the query, key, value, and output projections) are smaller than all-linear ones, and a classifier like this lab’s often loses nothing. Compare the two on the same ranks and let the table decide.

Use TRL’s memory work on long sequences. Recent TRL releases added a fused language-model head, aimed at cutting memory use and letting training run at longer sequence lengths. If your real data has long documents, upgrade and rerun a memory test before you buy a bigger GPU.

Add reinforcement learning after supervised tuning only when a metric needs it. TRL v1.18.0 promoted AsyncGRPOTrainer to a stable top-level trainer. For a format-following task, supervised LoRA is enough. For tasks where correctness can be checked automatically, such as code that must pass tests, a GRPO stage on top of the merged model is the natural next experiment.

Scale the same recipe up when the task earns it. Hugging Face wrote the Mistral Large 4 launch around day-zero TRL support, and a Beam post on the hub reportedly describes training LoRA adapters with PEFT instead of updating all 501 billion parameters of that model. Models at that size need multi-GPU nodes, but the workflow is the one you just ran. The open-weight release of Kimi K3.1 Pro shows the same direction on the Moonshot side, which means adapter skills transfer between families.

Finally, gate every adapter behind the evaluator. Keep evaluate.py and a frozen eval file in version control, and fail a pull request when exact match drops more than 2 points. A regression caught in review is cheaper than one caught by a customer.

Frequently Asked Questions #

What is LoRA fine-tuning?

LoRA fine-tuning adapts a language model by freezing its original weights and training small low-rank matrices beside the targeted layers. Only those matrices receive gradient updates, so memory use and file size drop sharply. The result is an adapter you can load on top of the base model or merge into it.

What is the difference between LoRA and QLoRA?

QLoRA is LoRA applied to a base model loaded in 4-bit precision. The adapter still trains in higher precision. Hugging Face’s documentation puts the memory saving from the 4-bit load at up to four times, which lets a 7B or 8B model fit on a single consumer GPU.

What rank should I start with?

Start at 16 with alpha at 32, then sweep. Narrow format-following tasks often plateau between 8 and 16. Tasks that add new knowledge or a new language may keep improving at higher ranks. Pick the smallest rank within about 2 points of your best score.

How much VRAM does LoRA fine-tuning need?

A 1.5B model in bfloat16 fits on a 16 GB card at the lab’s settings. A 7B to 8B model with 4-bit typically needs roughly 8 to 12 GB, because the 4-bit weights take about 4 to 5 GB and activations plus optimizer state fill the rest. Sequence length and batch size shift the total more than any other setting.

Should I merge the adapter or keep it separate?

Merge when you want one self-contained model for a runtime that does not understand PEFT, such as a GGUF conversion. Keep adapters separate when you serve many variants from one base, since swapping a small adapter is cheaper than a full model per customer. Always merge into a 16-bit base, never a 4-bit one.

Is LoRA better than RAG?

They solve different problems. LoRA changes how a model behaves, such as output format, tone, or a narrow skill. RAG changes what a model can see at answer time, so it suits facts that change. Many production systems use both.

How many training examples do I need?

Narrow format tasks often work with hundreds to a few thousand clean examples, and this lab uses 1,800. Quality beats volume. A few hundred consistent examples outperform thousands of noisy ones, and the held-out eval set tells you when more data stops helping.

Can I do this without a local GPU?

Yes. Any rented NVIDIA GPU with 16 GB or more runs the same scripts, and the 4-bit path stretches smaller cards. Training on CPU alone is possible in theory but impractical for anything beyond a toy example. Copy the project folder, run check_env.py first, and everything else follows.

source & further reading

shattered.io — original article Cloudflare D1 Adds AI Search Source, Caps Results at 50 MB Speculative Decoding Tutorial: 12 Steps, 75 Min [2026] Microsoft Outage Hits Day 2, No Cyber Link Seen [2026]