Fine-Tune an LLM With QLoRA: 12 Steps, 90 Min [2026]

You can fine-tune an LLM on a single consumer GPU in an afternoon, then run the result on your own machine through Ollama. The recipe is QLoRA: freeze a 4-bit copy of the base model, train small low-rank adapter matrices on top, merge the adapter, and export a GGUF file. The practical target here is a support-ticket triage model that returns strict JSON, trained on a synthetic dataset your own script generates in seconds.

The 2023 QLoRA paper from Tim Dettmers and coauthors made the core claim plainly: the method “reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.” Three years later the base models have changed, but the recipe has held up. What changed is the tooling around it, and that is where most readers get stuck.

The stack moved fast this fall. Hugging Face’s latest TRL release is v1.16.0. PEFT v0.21.2 landed on October 1. Ollama v0.40.11 is the current release as of October 24. On the model side, Qwen3.8-27B, a dense, native vision-language model from Alibaba, is the newest 27B-class open-weight checkpoint covered here. If you only want inference on a big card, the walkthrough on running LLMs locally on an RTX 5090 covers the serving half. What follows covers the training half, end to end.

Don't miss new tech stories on Google

Add Shattered once in the Google app and our stories show up in your news suggestions.

Add now

Why Fine-Tune an LLM Instead of Prompting or RAG #

Most teams reach for fine-tuning too early. A good system prompt plus a few examples solves a large share of formatting problems, and retrieval-augmented generation handles anything that depends on fresh or private documents. So the first question is not how to fine tune LLM weights. It is whether you need to touch the weights at all.

Fine-tuning earns its cost in four situations. The first is a rigid output format that the base model breaks even with careful prompting, such as strict JSON that a downstream parser consumes. The second is a narrow classification or extraction task where you have hundreds of labeled examples and want a small, cheap model to match a large one. The third is a house style or tone that a prompt cannot hold across thousands of calls. The fourth is latency and cost: a tuned model needs a short prompt, because the behavior lives in the weights instead of in 800 tokens of instructions you pay for on every request.

It is the wrong tool for teaching a model new facts. Adapters shift behavior, and they absorb knowledge poorly and unreliably. If your goal is answering questions about a changing knowledge base, build retrieval and leave the weights alone. If you need both, tune for format and behavior, then add retrieval for facts. The two approaches stack cleanly.

Open-weight models also make the economics different from two years ago. A ranking such as the one in the best open source LLM roundup shows how many capable checkpoints now exist, and several of them ship under permissive licenses. That means you can tune, merge, and redistribute inside your company without negotiating an API contract. It also means you own the privacy story, since the training data never leaves your machine.

A quick decision rule helps. Try the prompt approach first and write down the failure rate on 100 realistic inputs. If the failure rate is under a few percent, stop. If it sits above that and the failures are about format or behavior, collect examples and tune. If the failures are about missing knowledge, build retrieval. Keep that failure log, because the evaluation script later in this build reuses the same idea to score the tuned model against the untouched base.

The Project: A QLoRA Ticket-Triage Model and Its Folder Layout #

The model reads one customer message and returns a single JSON object with four keys: category, priority, needs_human, and summary. Categories come from a fixed set of five (billing, bug, account_access, feature_request, security). Priority is one of low, medium, high, or urgent. That task is small enough to train quickly and strict enough to measure without a judge model.

A synthetic dataset keeps the project reproducible. The generator in Step 3 builds 800 training rows and 100 held-out test rows from templates, with random openers, closers, plan names, and reference numbers. Labels follow deterministic rules: security tickets are urgent, account problems start at high, crashes bump a bug up one level, and a closing line saying the issue is blocking the user bumps any ticket up one more. Because the rules are explicit, you can tell when the model learned the rule and when it merely memorized phrasing.

Here is the folder layout the scripts assume. Run every command from the project root.

ticket-triage/
  .venv/
  requirements.txt
  Modelfile
  run_all.sh
  data/
    train.jsonl        (prompt + completion rows, 800)
    test.jsonl         (prompt + label rows, 100)
  scripts/
    preflight.py
    make_dataset.py
    train.py
    merge.py
    eval.py
  out/
    adapter/           (LoRA weights from training)
    merged/            (full bf16 model after merge)
    triage-f16.gguf
    triage-q4_k_m.gguf

Success Metrics Before You Start

Decide what a win looks like before training, or you will rationalize whatever comes out. This build tracks three numbers on the 100 test rows: the share of replies that parse as JSON with all four keys, category accuracy, and priority accuracy. The base model will probably score well on category and poorly on strict formatting. A good tuned model should reach near-perfect JSON validity and lift priority accuracy, because priority depends on rules the base model has never seen. Write your targets down now, for example a very high share of valid JSON, then compare at the end.

Budget your time too. Environment and data take about 15 minutes, training is the long step and depends on your card, merge and export take 15 to 20 minutes, and Ollama import plus evaluation take about 10. Those are planning figures for the 90-minute target, not measured benchmarks, and a 27B model on a 24 GB card will push the training step toward the top of the range or past it.

Prerequisites and Exact Versions for QLoRA in October 2026 #

You need an NVIDIA GPU with enough memory for a 4-bit copy of your chosen base model, a Linux box or WSL 2 with current drivers, and about 150 GB of free disk. The disk figure covers the downloaded base weights, the merged bf16 copy, the f16 GGUF, and the quantized GGUF. Skimping on disk is the most common reason a pipeline dies at Step 9 after an hour of work.

Component	Version to use	Why it matters
OS	Ubuntu Linux or WSL 2 with current NVIDIA drivers	bitsandbytes 4-bit kernels target CUDA
Python	3.11 or newer	Matches what current Hugging Face releases test against
PyTorch	CUDA build matching your driver	A CPU-only wheel silently disables the GPU
Transformers	5.17.0 or newer	PEFT’s stated compatibility target for current releases
PEFT	v0.21.2	Latest stable, fixes encoder-decoder use with Transformers 5.18.0 or newer
TRL	v1.16.0	Latest stable, supplies SFTTrainer and the fused LM head default from v1.15
Ollama	v0.40.11	Current release, runs GGUF imports locally
llama.cpp	Latest master build	Provides the GGUF converter and quantizer

Version pairing deserves attention. The PEFT maintainers have said that support for older Transformers releases will shrink in a future PEFT 1.0, although no date or minimum version has been confirmed. Pin your versions in requirements.txt once the pipeline works, and bump them deliberately instead of by accident. The TRL project also fixed two cases of silently wrong training and three crashes in v1.14.2 on October 6, which is a good reason not to run an older 1.x release for a new project.

You also need a Hugging Face account and, for gated models, a read token. Accept the model license on the model page first, then log in from the terminal with hf auth login. Without that step the download fails with an authorization error that looks like a network problem.

Pick a Base Model: Qwen3.8-27B, Gemma 4 12B and Others #

Check what is newest before you download anything. Alibaba announced Qwen3.8-Max on August 3, 2026, and the family includes an open-weight 2.4T-A95B checkpoint that is far beyond single-GPU territory. The practical Qwen option is Qwen3.8-27B, a dense, native vision-language model with image and video understanding and controllable reasoning. Its official Hugging Face repository is dated mid-August 2026, and the Qwen3.8 announcement lays out the family. A third-party catalog lists it under Apache 2.0, but read the license on the model card yourself before you ship anything.

Google’s side of the shelf is Gemma 4, launched on April 2, 2026 under Apache 2.0. A 12B variant followed on June 3, described as a multimodal model for laptops, and Google released quantization-aware training checkpoints on June 5. Google’s own write-up on bringing Gemma 4 12B to a laptop explains the positioning. One caution applies. The 12B model, the QAT checkpoints, and the multi-token-prediction drafters are related but distinct release pieces, so do not treat them as interchangeable fine-tuning checkpoints.

Model	Released	Class	Fit for one-GPU QLoRA
Qwen3.8-27B	August 2026	27B dense, native vision-language	Tight on a 24 GB card, comfortable on larger ones
Gemma 4 12B	June 3, 2026	12B, multimodal, laptop-oriented	Good first choice for 16 GB cards
Gemma 4 launch family	April 2, 2026	Open models under Apache 2.0	Pick the size that matches your card
Qwen3.8-2.4T-A95B	Open weights with the August 2026 family	Mixture of experts, flagship scale	Multi-GPU territory, skip for this build
Reflection Beam	Ollama Cloud, October 16, 2026	Open-weight, Apache 2.0, 1M-token context	Hosted use in practice, not a one-GPU target

The big open releases keep arriving. Reflection’s Beam went open source under Apache 2.0, a story covered in the Reflection Beam open-source report, and Ollama offered it on its cloud service for a limited free period. A model that size is a hosted-inference story for most readers. Fine-tuning happens one tier down, where a single card can hold the quantized weights.

The wider picture is that open releases now come from several labs at once, which is the theme of the analysis of the AI open-source bet built around Beam and Mistral. For you, that means the base-model choice is a hardware decision first. Pick the largest model whose 4-bit weights plus activations fit your card with room to spare, then confirm the license allows your use.

The scripts below read the model name from a BASE_MODEL environment variable. The default is Qwen/Qwen3.8-27B, which is the official repository name. If you have a smaller card, export the repository ID of a smaller model before every command. Nothing else in the project changes.

Steps 1 to 3: Environment, GPU Preflight and the Dataset #

Step 1: Create the project and install dependencies. A virtual environment keeps the CUDA-linked packages away from your system Python. Install a PyTorch build that matches your driver first if your distribution does not provide one, then the rest.

mkdir -p ~/ticket-triage/data ~/ticket-triage/scripts ~/ticket-triage/out
cd ~/ticket-triage
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

cat > requirements.txt <<'EOF'
torch
transformers>=5.17
peft>=0.21.2
trl>=1.16.0
bitsandbytes
accelerate
datasets
requests
EOF

pip install -r requirements.txt

Step 2: Run a preflight check. This script fails early and loudly. It prints library versions, confirms a CUDA device, reports memory, and loads only the model config so you learn the architecture name without down 50 GB.

import os
import sys

import peft
import torch
import transformers
import trl
from transformers import AutoConfig

BASE_MODEL = os.environ.get("BASE_MODEL", "Qwen/Qwen3.8-27B")

print("python", sys.version.split()[0])
print("torch", torch.__version__, "cuda", torch.cuda.is_available())
print("transformers", transformers.__version__)
print("peft", peft.__version__)
print("trl", trl.__version__)

if not torch.cuda.is_available():
    sys.exit("No CUDA GPU visible. QLoRA with bitsandbytes needs one.")

props = torch.cuda.get_device_properties(0)
print("gpu", props.name, round(props.total_memory / 1024**3, 1), "GiB")
print("bf16 supported", torch.cuda.is_bf16_supported())

config = AutoConfig.from_pretrained(BASE_MODEL)
print("model_type", config.model_type)
print("architectures", config.architectures)

Run it with python scripts/preflight.py. A healthy run prints your GPU name and a memory figure, then the architecture. Save the architecture string, since you will need it again if the llama.cpp converter complains in Step 8.

python 3.12.x
torch 2.x.x cuda True
transformers 5.17.x
peft 0.21.2
trl 1.16.0
gpu NVIDIA GeForce RTX 4090 24.0 GiB
bf16 supported True
model_type (printed from the model config)
architectures [(class name printed from the model config)]

That block shows the shape of a good result. Your versions, GPU line, and architecture string will differ, so treat the numbers as placeholders.

Step 3: Generate the dataset. The generator writes two files. Training rows use the prompt and completion format that TRL supports, which computes loss only on the completion by default. That matters, because it means the model learns to produce the JSON and does not waste capacity predicting the system prompt. Test rows keep the label so the evaluator can score them.

import json
import random

random.seed(42)

SYSTEM = (
    "You are a support triage engine. Reply with one JSON object and nothing else. "
    "Keys: category, priority, needs_human, summary."
)

CATEGORIES = {
    "billing": [
        "I was charged twice for my {plan} plan this month",
        "My invoice shows {amount} but I expected {amount2}",
        "Please refund the renewal that went through on {day}",
    ],
    "bug": [
        "The export button crashes the app when I pick {fmt}",
        "The dashboard shows a blank page after the latest update",
        "Search returns results from the wrong workspace",
    ],
    "account_access": [
        "I cannot log in since I changed my phone",
        "The password reset email never arrives",
        "My teammate lost access to the {plan} workspace",
    ],
    "feature_request": [
        "Could you add {fmt} support to the importer",
        "We would love dark mode on the mobile app",
        "Please let admins schedule reports weekly",
    ],
    "security": [
        "I think someone else is signed in to my account from another country",
        "A phishing email is using your company logo",
        "An API key of mine leaked in a public repository",
    ],
}

FILLERS = {
    "plan": ["Starter", "Team", "Business"],
    "amount": ["$49", "$120", "$300"],
    "amount2": ["$29", "$99", "$250"],
    "day": ["the 3rd", "the 14th", "the 27th"],
    "fmt": ["CSV", "PDF", "XLSX"],
}

LEVELS = ["low", "medium", "high", "urgent"]
BASE_LEVEL = {"billing": 1, "bug": 1, "account_access": 2, "feature_request": 0, "security": 3}
OPENERS = ["Hi team,", "Hello,", "Hey,", ""]
CLOSERS = ["Thanks.", "This is blocking me.", "Any update would help.", ""]

def make_example():
    category = random.choice(list(CATEGORIES))
    values = {key: random.choice(options) for key, options in FILLERS.items()}
    core = random.choice(CATEGORIES[category]).format(**values)
    level = BASE_LEVEL[category]
    if "crashes" in core:
        level += 1
    closer = random.choice(CLOSERS)
    if closer == "This is blocking me.":
        level += 1
    priority = LEVELS[min(level, 3)]
    ref = "Ref " + str(random.randint(1000, 9999)) + "."
    text = " ".join(part for part in [random.choice(OPENERS), core + ".", ref, closer] if part)
    label = {
        "category": category,
        "priority": priority,
        "needs_human": priority in ("high", "urgent") or category == "billing",
        "summary": core,
    }
    return text, label

def build(count, banned):
    rows, seen = [], set(banned)
    while len(rows) < count:
        text, label = make_example()
        if text in seen:
            continue
        seen.add(text)
        rows.append((text, label))
    return rows

def prompt_for(text):
    return [{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}]

test = build(100, [])
train = build(800, {text for text, _ in test})

with open("data/train.jsonl", "w") as handle:
    for text, label in train:
        record = {
            "prompt": prompt_for(text),
            "completion": [{"role": "assistant", "content": json.dumps(label)}],
        }
        handle.write(json.dumps(record) + "\n")

with open("data/test.jsonl", "w") as handle:
    for text, label in test:
        handle.write(json.dumps({"prompt": prompt_for(text), "label": label}) + "\n")

print("train", len(train), "test", len(test))

Run python scripts/make_dataset.py and expect the line train 800 test 100. Open data/train.jsonl and read ten rows by eye before moving on. Dataset bugs cost far more than training bugs, because a wrong label teaches the wrong rule at full speed and nothing in the loss curve will warn you.

Steps 4 to 6: Load 4-Bit Weights, Attach LoRA and Train #

Step 4: Load the base model in 4-bit. QLoRA stores the frozen weights in NF4 precision, adds double quantization to shave the overhead of the quantization constants, and computes in bf16. Three settings do the work, and each one trades a little quality or speed for memory. The Hugging Face team described the same approach in its 2023 write-up on 4-bit Transformers with bitsandbytes, which put 33B fine-tuning on a single 24GB GPU. A 27B model should land in a similar envelope, though that is an inference from those numbers and not a guarantee for every card.

Step 5: Attach LoRA adapters. LoRA leaves the base weights alone and learns two thin matrices per target layer, so you train a small fraction of the parameters. The configuration below uses rank 16, alpha 32, and all-linear targeting. That targets every linear layer except the output head, which is a sensible default when you do not know a model’s layer names. On a vision-language checkpoint it also touches the vision tower’s linear layers, which wastes a little memory for a text-only task. Restrict the targets with a regular expression if that bothers you.

Step 6: Train with SFTTrainer. The script below handles both plain language models and vision-language checkpoints by falling back to the image-text class if the causal-LM refuses the architecture. It trains for two epochs over 800 rows with a batch of 1 and 16 accumulation steps, which gives 50 optimizer steps per epoch.

import os

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import (
    AutoModelForCausalLM,
    AutoModelForImageTextToText,
    AutoTokenizer,
    BitsAndBytesConfig,
)
from trl import SFTConfig, SFTTrainer

BASE_MODEL = os.environ.get("BASE_MODEL", "Qwen/Qwen3.8-27B")
OUT_DIR = os.environ.get("OUT_DIR", "out/adapter")

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

load_args = dict(quantization_config=bnb, device_map="auto", dtype=torch.bfloat16)
try:
    model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, **load_args)
except (ValueError, KeyError):
    model = AutoModelForImageTextToText.from_pretrained(BASE_MODEL, **load_args)
model.config.use_cache = False

lora = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

dataset = load_dataset("json", data_files="data/train.jsonl", split="train")

args = SFTConfig(
    output_dir=OUT_DIR,
    num_train_epochs=2,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    logging_steps=5,
    save_strategy="epoch",
    bf16=True,
    max_length=1024,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    optim="paged_adamw_8bit",
    report_to="none",
)

trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=dataset,
    processing_class=tokenizer,
    peft_config=lora,
)
trainer.train()
trainer.save_model(OUT_DIR)
tokenizer.save_pretrained(OUT_DIR)
print("adapter saved to", OUT_DIR)

Start it with python scripts/train.py inside a tmux or screen session so a dropped SSH connection does not kill the run. TRL v1.15 made a fused Triton LM head the default for SFT, DPO, KTO, GRPO, RLOO, and distillation. Its release notes advertise up to 6.9 times longer sequences on the same GPU, because the trainer no longer materializes the full logits tensor. That is the project’s own claim and your gain will depend on vocabulary size and sequence length, but it is one reason a current TRL release matters for a memory-bound job.

Setting	Value	Reason
Quantization	NF4 with double quantization	Cuts weight memory roughly to a quarter of bf16 with small quality loss
LoRA rank and alpha	16 and 32	A common starting ratio, enough capacity for a format-and-rules task
Dropout	0.05	Light regularization for a small dataset
Learning rate	2e-4 with cosine decay	Typical for LoRA, much higher than full fine-tuning rates
Epochs	2	Enough to learn rules from synthetic rows without memorizing them
Effective batch	16 (1 times 16 accumulation)	Fits small cards while keeping gradients stable
Max length	1024 tokens	Tickets are short, so long limits only waste memory
Optimizer	paged_adamw_8bit	Pages optimizer state to CPU RAM during memory spikes

Resist the urge to raise rank, epochs, and learning rate at once. Change one setting per run and keep the evaluation script as the judge. A rank of 64 on 800 rows mostly buys you overfitting.

Hardware also shapes this step. NVIDIA has been pushing local AI toward cards with 24 GB and more, a trend covered in the report on NVIDIA local AI on 24GB+ RTX GPUs. If your card has less memory, drop max_length to 512 first, then choose a smaller base model before you touch anything else.

Step 7: Read the Training Log Without Guessing #

The trainer prints a dictionary every five steps. The number that matters is loss, and a healthy run shows it falling fast for the first 10 to 20 steps, then flattening. With this synthetic data the loss should drop low, because the rules are deterministic and the completion is short JSON. A flat line near the starting value means the model is not learning. A line that hits zero almost instantly means something leaked into the prompt.

{'loss': 2.31, 'grad_norm': 1.9, 'learning_rate': 0.00019, 'epoch': 0.1}
{'loss': 0.84, 'grad_norm': 0.9, 'learning_rate': 0.00018, 'epoch': 0.2}
{'loss': 0.21, 'grad_norm': 0.5, 'learning_rate': 0.00016, 'epoch': 0.3}
{'loss': 0.08, 'grad_norm': 0.3, 'learning_rate': 0.00011, 'epoch': 1.0}
{'loss': 0.04, 'grad_norm': 0.2, 'learning_rate': 0.00002, 'epoch': 2.0}
adapter saved to out/adapter

Those values only show the expected shape. Your first loss depends on the model and tokenizer, and the exact decay depends on your card and settings. Do not compare absolute numbers with someone else’s run. Compare the slope, and then compare the evaluation results in Step 11.

Watch three other signals. First, grad_norm should stay bounded. A sudden jump to a huge value or NaN means the learning rate is too high for this model, so halve it. Second, GPU memory should plateau after the first few steps. If it climbs steadily, a long outlier row is probably stretching a batch, which is why a hard max_length helps. Third, watch wall-clock time per step. If a step takes far longer than the first few, the paged optimizer may be swapping to CPU RAM, and you should shrink the sequence length or the rank.

Check the output directory when training ends. You should see adapter_config.json, adapter_model.safetensors, tokenizer files, and an epoch checkpoint folder or two. The adapter file is small compared with the base model, which is the whole point of LoRA. If the adapter file is missing and only checkpoints exist, the final save failed, and you can recover from the latest checkpoint directory instead of rerunning the job.

Keep a note of the exact command, environment variables, and package versions for each run. When a result looks surprising two weeks later, that note is the only way to reproduce it. A single line from pip freeze | grep -E "torch|transformers|peft|trl|bitsandbytes" pasted into your notes costs ten seconds and saves hours.

Steps 8 and 9: Merge the Adapter and Export GGUF #

Step 8: Merge the adapter into a full-precision copy. Never merge into the 4-bit weights. Rounding the merged values back into NF4 throws away part of what the adapter learned. Load the base model in bf16 on the CPU, apply the adapter, call merge_and_unload, and save the result. The CPU route needs system RAM for the full model at roughly two bytes per parameter, so plan for tens of gigabytes. It is slower than merging on the GPU, but it never fails with an out-of-memory error on a small card.

import os

import torch
from peft import PeftModel
from transformers import (
    AutoModelForCausalLM,
    AutoModelForImageTextToText,
    AutoTokenizer,
)

BASE_MODEL = os.environ.get("BASE_MODEL", "Qwen/Qwen3.8-27B")
ADAPTER = os.environ.get("ADAPTER", "out/adapter")
MERGED = os.environ.get("MERGED", "out/merged")

load_args = dict(dtype=torch.bfloat16, device_map="cpu")
try:
    base = AutoModelForCausalLM.from_pretrained(BASE_MODEL, **load_args)
except (ValueError, KeyError):
    base = AutoModelForImageTextToText.from_pretrained(BASE_MODEL, **load_args)

model = PeftModel.from_pretrained(base, ADAPTER)
model = model.merge_and_unload()
model.save_pretrained(MERGED, safe_serialization=True)
AutoTokenizer.from_pretrained(BASE_MODEL).save_pretrained(MERGED)
print("merged model written to", MERGED)

Step 9: Convert to GGUF and quantize. llama.cpp provides both tools. Clone the repository, build it with CUDA if you want GPU-accelerated tools, convert the merged Hugging Face folder to an f16 GGUF, then quantize to Q4_K_M. Ollama v0.30 improved GGUF compatibility through llama.cpp, so a recent Ollama will read the result without extra flags. Q4_K_M is the usual balance of size and quality for local use.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
python -m pip install -r requirements.txt
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
cd ..

python llama.cpp/convert_hf_to_gguf.py out/merged \
  --outfile out/triage-f16.gguf --outtype f16

llama.cpp/build/bin/llama-quantize \
  out/triage-f16.gguf out/triage-q4_k_m.gguf Q4_K_M

ls -lh out/*.gguf

Expect the f16 file to be several times larger than the quantized one. The conversion log prints the architecture it detected. If it prints an error about an unsupported architecture, your llama.cpp checkout is older than the model. Pull the latest master, rebuild, and try again. The project moves quickly, so a fresh checkout is the safest fix.

Vision-language checkpoints add one wrinkle. The converter handles the language model, and the vision encoder usually travels as a separate projector file in the GGUF world. For a text-only triage task you can ignore the vision side, though you should confirm in the llama.cpp documentation that your specific architecture converts the text tower cleanly. If it does not, a smaller dense text model removes the problem entirely.

Steps 10 and 11: Import Into Ollama and Score the Result #

Step 10: Build the Modelfile. Ollama needs a chat template that matches the format the model trained on. The easiest way to get it right is to copy the template from the base model’s Ollama entry. Pull the base model, dump its Modelfile, and swap the FROM line to point at your GGUF. Replace BASE_TAG with the Ollama tag of the base model you started from.

export BASE_TAG=your-base-model-tag
ollama pull "$BASE_TAG"
ollama show --modelfile "$BASE_TAG" > out/base.Modelfile

sed 's|^FROM .*|FROM ./out/triage-q4_k_m.gguf|' out/base.Modelfile > Modelfile

cat >> Modelfile <<'EOF'
PARAMETER temperature 0
PARAMETER num_ctx 4096
EOF

ollama create ticket-triage -f Modelfile
ollama run ticket-triage "Hi team, I was charged twice for my Team plan this month. Ref 4821. This is blocking me."

A working model replies with one JSON object and nothing else. If it rambles, repeats itself, or emits template tokens, the template is wrong. Open Modelfile and compare its TEMPLATE block with the base model’s. If your base model emits reasoning text before the answer, remember that the tuned model learned to answer directly, so the two models will not behave the same way under the same template.

Once the model answers, you can plug it into the tools you already use. The OpenCode setup with Ollama shows how a local model slots into an editor workflow, and the same ollama create flow works for any tuned model you want available there.

Step 11: Score both models on the held-out rows. The evaluator calls Ollama’s chat endpoint at temperature 0, parses each reply as JSON, and counts valid objects, category hits, and priority hits. Run it twice, once for the base tag and once for ticket-triage.

import json
import os
import sys

import requests

HOST = os.environ.get("OLLAMA_URL", "http://localhost:11434")
MODEL = sys.argv[1]
REQUIRED = {"category", "priority", "needs_human", "summary"}

def ask(messages):
    reply = requests.post(
        HOST + "/api/chat",
        json={
            "model": MODEL,
            "messages": messages,
            "stream": False,
            "options": {"temperature": 0},
        },
        timeout=300,
    )
    reply.raise_for_status()
    return reply.json()["message"]["content"]

total = valid = category_hits = priority_hits = 0
with open("data/test.jsonl") as handle:
    for line in handle:
        row = json.loads(line)
        total += 1
        raw = ask(row["prompt"])
        try:
            parsed = json.loads(raw)
        except json.JSONDecodeError:
            continue
        if not isinstance(parsed, dict) or not REQUIRED.issubset(parsed):
            continue
        valid += 1
        category_hits += parsed["category"] == row["label"]["category"]
        priority_hits += parsed["priority"] == row["label"]["priority"]

print("model=" + MODEL, "total=" + str(total))
print("valid_json=" + format(valid / total, ".1%"))
print("category_acc=" + format(category_hits / total, ".1%"))
print("priority_acc=" + format(priority_hits / total, ".1%"))

The evaluator counts accuracy over all 100 rows, so a reply that fails to parse counts as a miss on every metric. That choice punishes format failures honestly. A base model that gets the category right but wraps the JSON in a markdown fence scores zero on that row, and that is exactly the failure fine-tuning is meant to remove.

Step 12: Run the Whole Pipeline and Compare Base vs Tuned #

With every script in place, one shell file reruns the project from scratch. This is the complete working project in a single entry point. Keep it in version control alongside the scripts, because a pipeline you can rerun with one command is a pipeline you can improve safely.

#!/usr/bin/env bash
set -euo pipefail

export BASE_MODEL="${BASE_MODEL:-Qwen/Qwen3.8-27B}"
export BASE_TAG="${BASE_TAG:?set BASE_TAG to the Ollama tag of the base model}"

python scripts/preflight.py
python scripts/make_dataset.py
python scripts/train.py
python scripts/merge.py

python llama.cpp/convert_hf_to_gguf.py out/merged \
  --outfile out/triage-f16.gguf --outtype f16
llama.cpp/build/bin/llama-quantize \
  out/triage-f16.gguf out/triage-q4_k_m.gguf Q4_K_M

ollama show --modelfile "$BASE_TAG" > out/base.Modelfile
sed 's|^FROM .*|FROM ./out/triage-q4_k_m.gguf|' out/base.Modelfile > Modelfile
printf 'PARAMETER temperature 0\nPARAMETER num_ctx 4096\n' >> Modelfile
ollama create ticket-triage -f Modelfile

python scripts/eval.py "$BASE_TAG"
python scripts/eval.py ticket-triage

Run it with BASE_TAG=your-base-model-tag bash run_all.sh. The last two commands print the comparison. The shape of a successful result looks like the block below, with your own numbers in place of the placeholders.

model=your-base-model-tag total=100
valid_json=(lower, base model wraps or pads its output)
category_acc=(often high even before tuning)
priority_acc=(low, the rules are unknown to the base model)

model=ticket-triage total=100
valid_json=(near 100%)
category_acc=(near 100%)
priority_acc=(near 100% if the rules were learned)

Read the result with care. A tuned model that scores near 100 percent on synthetic rows proves it learned the rules. It does not prove it will handle real customer messages, which are messier than templates. The honest next step is to collect 50 to 100 real tickets, label them by hand, and run the same evaluator on them. If accuracy drops sharply, your synthetic data is too clean, and the fix is more varied training rows, not more epochs.

A second comparison is worth running if you care about cost, since the reason to fine tune LLM behavior into the weights is often a shorter prompt. Count the prompt tokens the base model needs to behave, including your system prompt and any few-shot examples. The tuned model needs only the short system prompt, so every request is cheaper and faster. For a high-volume classifier, that saving can matter more than the accuracy gain. The local Aider and Ollama coding setup is another example of the same pattern, where a model running on your own hardware replaces a metered API call.

The Faster Path: Unsloth for QLoRA and GGUF Export #

The Hugging Face stack above is the most transparent route, since every step is a separate, inspectable script. Unsloth collapses several of those steps into one library. Its own site says it trains models 2 times faster with 70 percent less VRAM and no accuracy loss. Those are vendor claims, so measure them on your hardware before you plan around them. The core package is Apache 2.0 licensed, while the Studio interface carries AGPL-3.0. The Unsloth pricing page lists a free open-source edition, and the Pro and Enterprise tiers are quote-based with no public prices.

The project has also grown beyond a single-GPU trainer. A third-party tracker reports a desktop app and a Studio web interface by early October, and Unsloth lists GGUF among its export options. According to AMD’s developer site, Unsloth training and inference on AMD GPUs shipped in July. AMD’s own article on training and running models on AMD GPUs with Unsloth is the place to start if you do not own an NVIDIA card. That removes the biggest hardware constraint in the stack above, which assumes CUDA.

A minimal Unsloth version of Steps 4 to 9 looks like the sketch below. The method names follow the library’s documented pattern, but the project ships beta tags frequently, so confirm the current signatures in the Unsloth docs before you copy this into production. Reuse the same BASE_MODEL variable and the same dataset.

import os

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
from unsloth import FastLanguageModel

BASE_MODEL = os.environ.get("BASE_MODEL", "Qwen/Qwen3.8-27B")

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=BASE_MODEL,
    max_seq_length=1024,
    load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    lora_alpha=32,
    lora_dropout=0.0,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)

dataset = load_dataset("json", data_files="data/train.jsonl", split="train")
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    processing_class=tokenizer,
    args=SFTConfig(
        output_dir="out/adapter-unsloth",
        num_train_epochs=2,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=16,
        learning_rate=2e-4,
        logging_steps=5,
        report_to="none",
    ),
)
trainer.train()

model.save_pretrained_gguf("out/unsloth-gguf", tokenizer, quantization_method="q4_k_m")

The tradeoff is control. Unsloth hides the merge and conversion work, which saves time when it works and hides the cause when it does not. If you are shipping a model that other people depend on, run the manual pipeline once so you know what each stage produces. Then switch to the faster path for iteration.

Serving is the other half of the speed story. Training gets the attention, but inference engines change what a tuned model costs to run. DeepSeek’s recent DeepInfer open-source inference release is one sign that the serving layer is now as competitive as the models, and a GGUF file you export today can move to a different runtime later without retraining.

Common Pitfalls When You Fine-Tune an LLM #

Most failed attempts to fine tune LLM checkpoints trace back to a short list of mistakes. Each one below has burned real projects, and each is cheap to avoid if you know it exists. Read the list before your first run, then reread it when a result looks wrong. For more articles on model internals and tooling, the AI and machine learning section collects related coverage.

1. Leaking test rows into training. If a test prompt also appears in the training file, the evaluation score means nothing. The dataset script removes overlap by text. Do the same with any real data, and deduplicate near-matches too.
2. Using the wrong chat template. A model trained on one template and served with another produces garbled or rambling output. Copy the template from the base model, as Step 10 does, and test with a prompt before you run the evaluator.
3. Merging into 4-bit weights. Always merge the adapter into a bf16 copy. Merging into quantized weights loses precision that the adapter just learned.
4. Treating a high training score as success. Synthetic data is clean. A model can hit near-perfect accuracy on it and still fail on real messages. Always run a small hand-labeled set of real examples.
5. Raising the learning rate to speed things up. LoRA tolerates higher rates than full fine-tuning, but 2e-4 is already high. Going further risks NaN loss and a ruined run that looks fine for 20 steps.
6. Forgetting the end-of-sequence token. If training rows do not end the completion properly, the tuned model never stops generating. The prompt and completion format in TRL handles this when the chat template is correct, so a rambling model is another sign of a template mismatch.
7. Ignoring the license. Your tuned model inherits the base model’s terms. Gemma 4 launched under Apache 2.0 and a third-party catalog lists Qwen3.8-27B the same way, but verify the license on the model card for the exact checkpoint you use.
8. Filling the disk. The base weights, merged copy, f16 GGUF, and quantized GGUF add up fast. Delete the f16 file after quantizing, and delete old epoch checkpoints you no longer need.

A ninth habit protects you from all of them: change one variable per run and write down what you changed. When you tweak rank, learning rate, and data in the same run, a better score teaches you nothing, and a worse one leaves you debugging three things at once.

Troubleshooting: 10 Failures and Their Fixes #

Work through these in order. Most errors in this pipeline come from version mismatches or memory limits, and the fixes are short.

1. CUDA out of memory during training. Lowermax_length to 512, keep the batch at 1, raise gradient accumulation, and confirm gradient checkpointing is on. If it still fails, choose a smaller base model.
2. bitsandbytes reports no GPU support or a CUDA setup failure. Your PyTorch build and driver disagree, or you installed a CPU-only wheel. Reinstall a CUDA build of PyTorch that matches your driver, then reinstall bitsandbytes.
3. Loss becomes NaN or jumps. Halve the learning rate, confirm bf16 is supported on your card (the preflight script prints it), and check for an empty or malformed row in the dataset.
4. The raises an error about an unrecognized model type. Your Transformers release is older than the model. Upgrade to 5.17.0 or newer, and read the model card for any minimum version.
5. The causal-LM fails on a vision-language checkpoint. The scripts fall back to the image-text class automatically. If that also fails, print the architecture from the preflight script and search the model card for the right class.
6. TypeError about an unexpected max_seq_length argument. Current TRL usesmax_length inSFTConfig . Update the argument name, and pin TRL v1.16.0 or newer.
7. Merge script runs out of memory. Keepdevice_map="cpu" and add swap or RAM. The merge needs the full bf16 model in memory, at roughly two bytes per parameter.
8. The GGUF converter reports an unsupported architecture. Pull the latest llama.cpp master, rebuild, and rerun the conversion. If the architecture still fails, check the project’s issue tracker for support status.
9. Ollama prints garbage or never stops. The template or stop tokens in your Modelfile do not match the model. Rebuild the Modelfile from the base model’s, as in Step 10, and keep theFROM line as the only change.
10. Valid JSON rate stays low after tuning. Check the template first, then the loss curve. If the loss never dropped, the dataset format is wrong. If it dropped but replies still fail, train one more epoch or lower the temperature further.

Gated downloads deserve a mention too. If a download from the Hugging Face hub fails, run hf auth login and retry.

Advanced Tips: Preference Tuning, Adapters and Safe Data #

Once the basic loop works, the same stack opens several upgrades. TRL covers more than supervised fine-tuning. It also provides DPO, KTO, GRPO, RLOO, and distillation trainers, so you can add a preference-tuning pass after SFT without leaving the library. A common pattern is to run SFT for format, then run DPO on pairs of good and bad replies to sharpen edge cases such as ambiguous billing messages. TRL v1.16.0 also added AWS Trainium support, which matters if your training budget lives on that hardware instead of an NVIDIA card.

You can also skip the merge. Ollama can load a LoRA adapter on top of a base model through an ADAPTER line in the Modelfile, which lets one base model serve several tuned behaviors. Support depends on the architecture, so test it before you rely on it. The merged GGUF route in this build is the safer default, and the adapter route is worth trying when you maintain many small variants.

FROM your-base-model-tag
ADAPTER ./out/adapter
PARAMETER temperature 0

Sweep hyperparameters with discipline. Try rank 8, 16, and 32 with everything else fixed, and keep the one that wins on your hand-labeled set instead of the synthetic one. Try one epoch against two against three, and watch where accuracy stops improving. Past that point, extra epochs only memorize your templates. Sequence packing can speed up training on short rows like these, but check that your chat template and loss masking still behave after you enable it.

Keep Training Data Clean and Output Validated

Training data is an attack surface and a liability. Scrub personal data before it reaches a training file, because a model can reproduce memorized strings, and removing them afterward is nearly impossible. Strip API keys, tokens, email addresses, and account numbers from real tickets. Treat any dataset you did not create as untrusted, since a poisoned row can plant a behavior that only triggers on a specific phrase.

Treat the model’s output as untrusted too. A tuned classifier still reads customer text, and a customer can write instructions into a ticket. Validate every reply against a JSON schema in your application code, reject anything with unknown keys or values outside your allowed sets, and never let the model’s output trigger an action without that check. Pin base-model revisions in your download call, record the hash of each adapter and GGUF you ship, and store them where you store other build artifacts. Those habits turn a pile of weights into something you can audit.

import json

ALLOWED = {
    "category": {"billing", "bug", "account_access", "feature_request", "security"},
    "priority": {"low", "medium", "high", "urgent"},
}

def parse_ticket(raw):
    data = json.loads(raw)
    if set(data) != {"category", "priority", "needs_human", "summary"}:
        raise ValueError("unexpected keys")
    if data["category"] not in ALLOWED["category"]:
        raise ValueError("bad category")
    if data["priority"] not in ALLOWED["priority"]:
        raise ValueError("bad priority")
    if not isinstance(data["needs_human"], bool):
        raise ValueError("needs_human must be a boolean")
    return data

Scale Evaluation Before You Trust the Model

One hundred synthetic rows is a smoke test, not a benchmark. Before the model touches production traffic, build an evaluation set from real, hand-labeled tickets, include the ugly cases (typos, mixed languages, two problems in one message), and track accuracy per category. A model that scores well overall but fails on security tickets is the dangerous kind, because those are the ones you cannot afford to misroute. Rerun the evaluator after every change to data, rank, or base model, and keep the results in a table in your repository.

Finally, plan for the next base model. The open-weight field is moving month to month, and Qwen’s team has been reported to be training a successor, though nothing official has shipped. Your pipeline already isolates the base model behind one environment variable, so a swap costs one training run, not a rewrite. That is the real payoff of building the project as scripts instead of a notebook.

FAQ #

How much VRAM do I need to fine tune LLM weights with QLoRA?

It depends on model size, sequence length, and batch size. The 2023 QLoRA paper reported a 65B model on a single 48GB GPU, and Hugging Face’s 2023 write-up on 4-bit Transformers described 33B fine-tuning on a single 24GB GPU. A 12B model is comfortable on a 16 GB card, and a 27B model needs a 24 GB card with short sequences. Check your card with the preflight script before you download anything large.

Is fine-tuning better than RAG?

They solve different problems. Fine-tuning changes behavior and format, while retrieval supplies facts the model does not have. Use retrieval for changing knowledge and tuning for consistent output. Many production systems use both.

Can I fine-tune on a Mac or an AMD GPU?

Yes, with caveats. bitsandbytes targets CUDA, so the scripts here assume an NVIDIA card. According to AMD’s developer site, Unsloth added AMD GPU training and inference in July. For Apple Silicon, Ollama’s own MLX engine is the more established route for inference. Check each project’s current documentation for supported hardware before you commit.

How many training examples do I need?

For a narrow format-and-rules task, a few hundred clean examples often work, and this project uses 800. Quality beats quantity. A smaller set of varied, correctly labeled rows outperforms a large set of repetitive or noisy ones. Add rows where the evaluator shows failures.

Does my tuned model keep the base model’s license?

Your tuned weights fall under the base model’s license terms. Gemma 4 launched under Apache 2.0, and a third-party catalog lists Qwen3.8-27B under Apache 2.0, but confirm the license on the exact model card you downloaded and read any use restrictions.

Should I merge the adapter or keep it separate?

Merge when you want one portable file that runs anywhere GGUF runs. Keep the adapter separate when you serve several behaviors from one base model and your runtime supports adapters for that architecture. Merging into a bf16 copy first is the safe path for both.

Is Ollama free for running my tuned model?

Running models locally on your own hardware is free. Ollama also sells cloud plans, and its pricing page lists the Pro tier at $20 a month or $200 a year, with Team plans starting at $500 a month. Your tuned GGUF runs on the local runtime and does not need any of them.

source & further reading

shattered.io — original article Tesla 2026.38.5.6: FSD v14.3.12, 20% Faster Reaction [2026] Write Cursor Skills: 12 Steps, 45 Min [2026] Aider Local AI Coding Assistant: 12 Steps, 75 Min [2026]