{"slug": "fine-tune-an-llm-with-qlora-12-steps-90-min-2026", "title": "Fine-Tune an LLM With QLoRA: 12 Steps, 90 Min [2026]", "summary": "A walkthrough published by Shattered outlines a 12-step QLoRA recipe for fine-tuning a large language model on a single consumer GPU in about 90 minutes, then exporting the merged adapter as a GGUF file to run locally through Ollama v0.40.11. The guide cites the 2023 QLoRA paper by Tim Dettmers and coauthors, which claimed the method \"reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance,\" and targets a support-ticket triage model that returns strict JSON trained on a synthetic dataset. It names Hugging Face TRL v1.16.0, PEFT v0.21.2 (released October 1), and Qwen3.8-27B, a dense native vision-language model from Alibaba, as the current stack, and advises teams to try prompting first and tune only when the failure rate on 100 realistic inputs exceeds a few percent and failures concern format or behavior rather than missing knowledge.", "body_md": "You can fine-tune an LLM on a single consumer GPU in an afternoon, then run the result on your own machine through Ollama. The recipe is QLoRA: freeze a 4-bit copy of the base model, train small low-rank adapter matrices on top, merge the adapter, and export a GGUF file. The practical target here is a support-ticket triage model that returns strict JSON, trained on a synthetic dataset your own script generates in seconds.\n\nThe [2023 QLoRA paper](https://arxiv.org/abs/2305.14314) from Tim Dettmers and coauthors made the core claim plainly: the method “reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.” Three years later the base models have changed, but the recipe has held up. What changed is the tooling around it, and that is where most readers get stuck.\n\nThe stack moved fast this fall. Hugging Face’s latest TRL release is v1.16.0. PEFT v0.21.2 landed on October 1. Ollama v0.40.11 is the current release as of October 24. On the model side, Qwen3.8-27B, a dense, native vision-language model from Alibaba, is the newest 27B-class open-weight checkpoint covered here. If you only want inference on a big card, the walkthrough on [running LLMs locally on an RTX 5090](https://shattered.io/run-llm-locally-rtx-5090/) covers the serving half. What follows covers the training half, end to end.\n\nDon't miss new tech stories on Google\n\nAdd Shattered once in the Google app and our stories show up in your news suggestions.\n\n[Add now](https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F)\n\n## Why Fine-Tune an LLM Instead of Prompting or RAG\n\nMost teams reach for fine-tuning too early. A good system prompt plus a few examples solves a large share of formatting problems, and retrieval-augmented generation handles anything that depends on fresh or private documents. So the first question is not how to fine tune LLM weights. It is whether you need to touch the weights at all.\n\nFine-tuning earns its cost in four situations. The first is a rigid output format that the base model breaks even with careful prompting, such as strict JSON that a downstream parser consumes. The second is a narrow classification or extraction task where you have hundreds of labeled examples and want a small, cheap model to match a large one. The third is a house style or tone that a prompt cannot hold across thousands of calls. The fourth is latency and cost: a tuned model needs a short prompt, because the behavior lives in the weights instead of in 800 tokens of instructions you pay for on every request.\n\nIt is the wrong tool for teaching a model new facts. Adapters shift behavior, and they absorb knowledge poorly and unreliably. If your goal is answering questions about a changing knowledge base, build retrieval and leave the weights alone. If you need both, tune for format and behavior, then add retrieval for facts. The two approaches stack cleanly.\n\nOpen-weight models also make the economics different from two years ago. A ranking such as the one in the [best open source LLM roundup](https://shattered.io/best-open-source-llm-october-2026-kimi-glm-deepseek/) shows how many capable checkpoints now exist, and several of them ship under permissive licenses. That means you can tune, merge, and redistribute inside your company without negotiating an API contract. It also means you own the privacy story, since the training data never leaves your machine.\n\nA quick decision rule helps. Try the prompt approach first and write down the failure rate on 100 realistic inputs. If the failure rate is under a few percent, stop. If it sits above that and the failures are about format or behavior, collect examples and tune. If the failures are about missing knowledge, build retrieval. Keep that failure log, because the evaluation script later in this build reuses the same idea to score the tuned model against the untouched base.\n\n## The Project: A QLoRA Ticket-Triage Model and Its Folder Layout\n\nThe model reads one customer message and returns a single JSON object with four keys: `category`, `priority`, `needs_human`, and `summary`. Categories come from a fixed set of five (billing, bug, account_access, feature_request, security). Priority is one of low, medium, high, or urgent. That task is small enough to train quickly and strict enough to measure without a judge model.\n\nA synthetic dataset keeps the project reproducible. The generator in Step 3 builds 800 training rows and 100 held-out test rows from templates, with random openers, closers, plan names, and reference numbers. Labels follow deterministic rules: security tickets are urgent, account problems start at high, crashes bump a bug up one level, and a closing line saying the issue is blocking the user bumps any ticket up one more. Because the rules are explicit, you can tell when the model learned the rule and when it merely memorized phrasing.\n\nHere is the folder layout the scripts assume. Run every command from the project root.\n\n```\nticket-triage/\n  .venv/\n  requirements.txt\n  Modelfile\n  run_all.sh\n  data/\n    train.jsonl        (prompt + completion rows, 800)\n    test.jsonl         (prompt + label rows, 100)\n  scripts/\n    preflight.py\n    make_dataset.py\n    train.py\n    merge.py\n    eval.py\n  out/\n    adapter/           (LoRA weights from training)\n    merged/            (full bf16 model after merge)\n    triage-f16.gguf\n    triage-q4_k_m.gguf\n```\n\n### Success Metrics Before You Start\n\nDecide what a win looks like before training, or you will rationalize whatever comes out. This build tracks three numbers on the 100 test rows: the share of replies that parse as JSON with all four keys, category accuracy, and priority accuracy. The base model will probably score well on category and poorly on strict formatting. A good tuned model should reach near-perfect JSON validity and lift priority accuracy, because priority depends on rules the base model has never seen. Write your targets down now, for example a very high share of valid JSON, then compare at the end.\n\nBudget your time too. Environment and data take about 15 minutes, training is the long step and depends on your card, merge and export take 15 to 20 minutes, and Ollama import plus evaluation take about 10. Those are planning figures for the 90-minute target, not measured benchmarks, and a 27B model on a 24 GB card will push the training step toward the top of the range or past it.\n\n## Prerequisites and Exact Versions for QLoRA in October 2026\n\nYou need an NVIDIA GPU with enough memory for a 4-bit copy of your chosen base model, a Linux box or WSL 2 with current drivers, and about 150 GB of free disk. The disk figure covers the downloaded base weights, the merged bf16 copy, the f16 GGUF, and the quantized GGUF. Skimping on disk is the most common reason a pipeline dies at Step 9 after an hour of work.\n\n| Component | Version to use | Why it matters | \n|---|---|---|\n| OS | Ubuntu Linux or WSL 2 with current NVIDIA drivers | bitsandbytes 4-bit kernels target CUDA | \n| Python | 3.11 or newer | Matches what current Hugging Face releases test against | \n| PyTorch | CUDA build matching your driver | A CPU-only wheel silently disables the GPU | \n| Transformers | 5.17.0 or newer | PEFT’s stated compatibility target for current releases | \n| PEFT | v0.21.2 | Latest stable, fixes encoder-decoder use with Transformers 5.18.0 or newer | \n| TRL | v1.16.0 | Latest stable, supplies SFTTrainer and the fused LM head default from v1.15 | \n| Ollama | v0.40.11 | Current release, runs GGUF imports locally | \n| llama.cpp | Latest master build | Provides the GGUF converter and quantizer | \n\nVersion pairing deserves attention. The PEFT maintainers have said that support for older Transformers releases will shrink in a future PEFT 1.0, although no date or minimum version has been confirmed. Pin your versions in `requirements.txt` once the pipeline works, and bump them deliberately instead of by accident. The TRL project also fixed two cases of silently wrong training and three crashes in v1.14.2 on October 6, which is a good reason not to run an older 1.x release for a new project.\n\nYou also need a Hugging Face account and, for gated models, a read token. Accept the model license on the model page first, then log in from the terminal with `hf auth login`. Without that step the download fails with an authorization error that looks like a network problem.\n\n## Pick a Base Model: Qwen3.8-27B, Gemma 4 12B and Others\n\nCheck what is newest before you download anything. Alibaba announced Qwen3.8-Max on August 3, 2026, and the family includes an open-weight 2.4T-A95B checkpoint that is far beyond single-GPU territory. The practical Qwen option is Qwen3.8-27B, a dense, native vision-language model with image and video understanding and controllable reasoning. Its official Hugging Face repository is dated mid-August 2026, and the [Qwen3.8 announcement](https://qwen.ai/blog?id=qwen3.8) lays out the family. A third-party catalog lists it under Apache 2.0, but read the license on the model card yourself before you ship anything.\n\nGoogle’s side of the shelf is Gemma 4, launched on April 2, 2026 under Apache 2.0. A 12B variant followed on June 3, described as a multimodal model for laptops, and Google released quantization-aware training checkpoints on June 5. Google’s own write-up on [bringing Gemma 4 12B to a laptop](https://developers.googleblog.com/bringing-gemma-4-12b-to-your-laptop-unlocking-local-agentic-workflows-with-google-ai-edge/) explains the positioning. One caution applies. The 12B model, the QAT checkpoints, and the multi-token-prediction drafters are related but distinct release pieces, so do not treat them as interchangeable fine-tuning checkpoints.\n\n| Model | Released | Class | Fit for one-GPU QLoRA | \n|---|---|---|---|\n| Qwen3.8-27B | August 2026 | 27B dense, native vision-language | Tight on a 24 GB card, comfortable on larger ones | \n| Gemma 4 12B | June 3, 2026 | 12B, multimodal, laptop-oriented | Good first choice for 16 GB cards | \n| Gemma 4 launch family | April 2, 2026 | Open models under Apache 2.0 | Pick the size that matches your card | \n| Qwen3.8-2.4T-A95B | Open weights with the August 2026 family | Mixture of experts, flagship scale | Multi-GPU territory, skip for this build | \n| Reflection Beam | Ollama Cloud, October 16, 2026 | Open-weight, Apache 2.0, 1M-token context | Hosted use in practice, not a one-GPU target | \n\nThe big open releases keep arriving. Reflection’s Beam went open source under Apache 2.0, a story covered in the [Reflection Beam open-source report](https://shattered.io/reflection-beam-open-source-apache-2-0-40-partners-2026/), and Ollama offered it on its cloud service for a limited free period. A model that size is a hosted-inference story for most readers. Fine-tuning happens one tier down, where a single card can hold the quantized weights.\n\nThe wider picture is that open releases now come from several labs at once, which is the theme of the [analysis of the AI open-source bet](https://shattered.io/ai-open-source-bet-86-percent-beam-mistral-2026/) built around Beam and Mistral. For you, that means the base-model choice is a hardware decision first. Pick the largest model whose 4-bit weights plus activations fit your card with room to spare, then confirm the license allows your use.\n\nThe scripts below read the model name from a `BASE_MODEL` environment variable. The default is `Qwen/Qwen3.8-27B`, which is the official repository name. If you have a smaller card, export the repository ID of a smaller model before every command. Nothing else in the project changes.\n\n## Steps 1 to 3: Environment, GPU Preflight and the Dataset\n\n**Step 1: Create the project and install dependencies.** A virtual environment keeps the CUDA-linked packages away from your system Python. Install a PyTorch build that matches your driver first if your distribution does not provide one, then the rest.\n\n```\nmkdir -p ~/ticket-triage/data ~/ticket-triage/scripts ~/ticket-triage/out\ncd ~/ticket-triage\npython3 -m venv .venv\nsource .venv/bin/activate\npython -m pip install --upgrade pip\n\ncat > requirements.txt <<'EOF'\ntorch\ntransformers>=5.17\npeft>=0.21.2\ntrl>=1.16.0\nbitsandbytes\naccelerate\ndatasets\nrequests\nEOF\n\npip install -r requirements.txt\n```\n\n**Step 2: Run a preflight check.** This script fails early and loudly. It prints library versions, confirms a CUDA device, reports memory, and loads only the model config so you learn the architecture name without downloading 50 GB.\n\n``` python\n# scripts/preflight.py\nimport os\nimport sys\n\nimport peft\nimport torch\nimport transformers\nimport trl\nfrom transformers import AutoConfig\n\nBASE_MODEL = os.environ.get(\"BASE_MODEL\", \"Qwen/Qwen3.8-27B\")\n\nprint(\"python\", sys.version.split()[0])\nprint(\"torch\", torch.__version__, \"cuda\", torch.cuda.is_available())\nprint(\"transformers\", transformers.__version__)\nprint(\"peft\", peft.__version__)\nprint(\"trl\", trl.__version__)\n\nif not torch.cuda.is_available():\n    sys.exit(\"No CUDA GPU visible. QLoRA with bitsandbytes needs one.\")\n\nprops = torch.cuda.get_device_properties(0)\nprint(\"gpu\", props.name, round(props.total_memory / 1024**3, 1), \"GiB\")\nprint(\"bf16 supported\", torch.cuda.is_bf16_supported())\n\nconfig = AutoConfig.from_pretrained(BASE_MODEL)\nprint(\"model_type\", config.model_type)\nprint(\"architectures\", config.architectures)\n```\n\nRun it with `python scripts/preflight.py`. A healthy run prints your GPU name and a memory figure, then the architecture. Save the architecture string, since you will need it again if the llama.cpp converter complains in Step 8.\n\n```\npython 3.12.x\ntorch 2.x.x cuda True\ntransformers 5.17.x\npeft 0.21.2\ntrl 1.16.0\ngpu NVIDIA GeForce RTX 4090 24.0 GiB\nbf16 supported True\nmodel_type (printed from the model config)\narchitectures [(class name printed from the model config)]\n```\n\nThat block shows the shape of a good result. Your versions, GPU line, and architecture string will differ, so treat the numbers as placeholders.\n\n**Step 3: Generate the dataset.** The generator writes two files. Training rows use the prompt and completion format that TRL supports, which computes loss only on the completion by default. That matters, because it means the model learns to produce the JSON and does not waste capacity predicting the system prompt. Test rows keep the label so the evaluator can score them.\n\n``` python\n# scripts/make_dataset.py\nimport json\nimport random\n\nrandom.seed(42)\n\nSYSTEM = (\n    \"You are a support triage engine. Reply with one JSON object and nothing else. \"\n    \"Keys: category, priority, needs_human, summary.\"\n)\n\nCATEGORIES = {\n    \"billing\": [\n        \"I was charged twice for my {plan} plan this month\",\n        \"My invoice shows {amount} but I expected {amount2}\",\n        \"Please refund the renewal that went through on {day}\",\n    ],\n    \"bug\": [\n        \"The export button crashes the app when I pick {fmt}\",\n        \"The dashboard shows a blank page after the latest update\",\n        \"Search returns results from the wrong workspace\",\n    ],\n    \"account_access\": [\n        \"I cannot log in since I changed my phone\",\n        \"The password reset email never arrives\",\n        \"My teammate lost access to the {plan} workspace\",\n    ],\n    \"feature_request\": [\n        \"Could you add {fmt} support to the importer\",\n        \"We would love dark mode on the mobile app\",\n        \"Please let admins schedule reports weekly\",\n    ],\n    \"security\": [\n        \"I think someone else is signed in to my account from another country\",\n        \"A phishing email is using your company logo\",\n        \"An API key of mine leaked in a public repository\",\n    ],\n}\n\nFILLERS = {\n    \"plan\": [\"Starter\", \"Team\", \"Business\"],\n    \"amount\": [\"$49\", \"$120\", \"$300\"],\n    \"amount2\": [\"$29\", \"$99\", \"$250\"],\n    \"day\": [\"the 3rd\", \"the 14th\", \"the 27th\"],\n    \"fmt\": [\"CSV\", \"PDF\", \"XLSX\"],\n}\n\nLEVELS = [\"low\", \"medium\", \"high\", \"urgent\"]\nBASE_LEVEL = {\"billing\": 1, \"bug\": 1, \"account_access\": 2, \"feature_request\": 0, \"security\": 3}\nOPENERS = [\"Hi team,\", \"Hello,\", \"Hey,\", \"\"]\nCLOSERS = [\"Thanks.\", \"This is blocking me.\", \"Any update would help.\", \"\"]\n\ndef make_example():\n    category = random.choice(list(CATEGORIES))\n    values = {key: random.choice(options) for key, options in FILLERS.items()}\n    core = random.choice(CATEGORIES[category]).format(**values)\n    level = BASE_LEVEL[category]\n    if \"crashes\" in core:\n        level += 1\n    closer = random.choice(CLOSERS)\n    if closer == \"This is blocking me.\":\n        level += 1\n    priority = LEVELS[min(level, 3)]\n    ref = \"Ref \" + str(random.randint(1000, 9999)) + \".\"\n    text = \" \".join(part for part in [random.choice(OPENERS), core + \".\", ref, closer] if part)\n    label = {\n        \"category\": category,\n        \"priority\": priority,\n        \"needs_human\": priority in (\"high\", \"urgent\") or category == \"billing\",\n        \"summary\": core,\n    }\n    return text, label\n\ndef build(count, banned):\n    rows, seen = [], set(banned)\n    while len(rows) < count:\n        text, label = make_example()\n        if text in seen:\n            continue\n        seen.add(text)\n        rows.append((text, label))\n    return rows\n\ndef prompt_for(text):\n    return [{\"role\": \"system\", \"content\": SYSTEM}, {\"role\": \"user\", \"content\": text}]\n\ntest = build(100, [])\ntrain = build(800, {text for text, _ in test})\n\nwith open(\"data/train.jsonl\", \"w\") as handle:\n    for text, label in train:\n        record = {\n            \"prompt\": prompt_for(text),\n            \"completion\": [{\"role\": \"assistant\", \"content\": json.dumps(label)}],\n        }\n        handle.write(json.dumps(record) + \"\\n\")\n\nwith open(\"data/test.jsonl\", \"w\") as handle:\n    for text, label in test:\n        handle.write(json.dumps({\"prompt\": prompt_for(text), \"label\": label}) + \"\\n\")\n\nprint(\"train\", len(train), \"test\", len(test))\n```\n\nRun `python scripts/make_dataset.py` and expect the line `train 800 test 100`. Open `data/train.jsonl` and read ten rows by eye before moving on. Dataset bugs cost far more than training bugs, because a wrong label teaches the wrong rule at full speed and nothing in the loss curve will warn you.\n\n## Steps 4 to 6: Load 4-Bit Weights, Attach LoRA and Train\n\n**Step 4: Load the base model in 4-bit.** QLoRA stores the frozen weights in NF4 precision, adds double quantization to shave the overhead of the quantization constants, and computes in bf16. Three settings do the work, and each one trades a little quality or speed for memory. The Hugging Face team described the same approach in its 2023 write-up on 4-bit Transformers with bitsandbytes, which put 33B fine-tuning on a single 24GB GPU. A 27B model should land in a similar envelope, though that is an inference from those numbers and not a guarantee for every card.\n\n**Step 5: Attach LoRA adapters.** LoRA leaves the base weights alone and learns two thin matrices per target layer, so you train a small fraction of the parameters. The configuration below uses rank 16, alpha 32, and `all-linear` targeting. That targets every linear layer except the output head, which is a sensible default when you do not know a model’s layer names. On a vision-language checkpoint it also touches the vision tower’s linear layers, which wastes a little memory for a text-only task. Restrict the targets with a regular expression if that bothers you.\n\n**Step 6: Train with SFTTrainer.** The script below handles both plain language models and vision-language checkpoints by falling back to the image-text class if the causal-LM loader refuses the architecture. It trains for two epochs over 800 rows with a batch of 1 and 16 accumulation steps, which gives 50 optimizer steps per epoch.\n\n``` python\n# scripts/train.py\nimport os\n\nimport torch\nfrom datasets import load_dataset\nfrom peft import LoraConfig\nfrom transformers import (\n    AutoModelForCausalLM,\n    AutoModelForImageTextToText,\n    AutoTokenizer,\n    BitsAndBytesConfig,\n)\nfrom trl import SFTConfig, SFTTrainer\n\nBASE_MODEL = os.environ.get(\"BASE_MODEL\", \"Qwen/Qwen3.8-27B\")\nOUT_DIR = os.environ.get(\"OUT_DIR\", \"out/adapter\")\n\nbnb = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_use_double_quant=True,\n    bnb_4bit_compute_dtype=torch.bfloat16,\n)\n\ntokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)\nif tokenizer.pad_token is None:\n    tokenizer.pad_token = tokenizer.eos_token\n\nload_args = dict(quantization_config=bnb, device_map=\"auto\", dtype=torch.bfloat16)\ntry:\n    model = AutoModelForCausalLM.from_pretrained(BASE_MODEL, **load_args)\nexcept (ValueError, KeyError):\n    model = AutoModelForImageTextToText.from_pretrained(BASE_MODEL, **load_args)\nmodel.config.use_cache = False\n\nlora = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    lora_dropout=0.05,\n    target_modules=\"all-linear\",\n    task_type=\"CAUSAL_LM\",\n)\n\ndataset = load_dataset(\"json\", data_files=\"data/train.jsonl\", split=\"train\")\n\nargs = SFTConfig(\n    output_dir=OUT_DIR,\n    num_train_epochs=2,\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,\n    learning_rate=2e-4,\n    lr_scheduler_type=\"cosine\",\n    warmup_ratio=0.05,\n    logging_steps=5,\n    save_strategy=\"epoch\",\n    bf16=True,\n    max_length=1024,\n    gradient_checkpointing=True,\n    gradient_checkpointing_kwargs={\"use_reentrant\": False},\n    optim=\"paged_adamw_8bit\",\n    report_to=\"none\",\n)\n\ntrainer = SFTTrainer(\n    model=model,\n    args=args,\n    train_dataset=dataset,\n    processing_class=tokenizer,\n    peft_config=lora,\n)\ntrainer.train()\ntrainer.save_model(OUT_DIR)\ntokenizer.save_pretrained(OUT_DIR)\nprint(\"adapter saved to\", OUT_DIR)\n```\n\nStart it with `python scripts/train.py` inside a `tmux` or `screen` session so a dropped SSH connection does not kill the run. TRL v1.15 made a fused Triton LM head the default for SFT, DPO, KTO, GRPO, RLOO, and distillation. Its release notes advertise up to 6.9 times longer sequences on the same GPU, because the trainer no longer materializes the full logits tensor. That is the project’s own claim and your gain will depend on vocabulary size and sequence length, but it is one reason a current TRL release matters for a memory-bound job.\n\n| Setting | Value | Reason | \n|---|---|---|\n| Quantization | NF4 with double quantization | Cuts weight memory roughly to a quarter of bf16 with small quality loss | \n| LoRA rank and alpha | 16 and 32 | A common starting ratio, enough capacity for a format-and-rules task | \n| Dropout | 0.05 | Light regularization for a small dataset | \n| Learning rate | 2e-4 with cosine decay | Typical for LoRA, much higher than full fine-tuning rates | \n| Epochs | 2 | Enough to learn rules from synthetic rows without memorizing them | \n| Effective batch | 16 (1 times 16 accumulation) | Fits small cards while keeping gradients stable | \n| Max length | 1024 tokens | Tickets are short, so long limits only waste memory | \n| Optimizer | paged_adamw_8bit | Pages optimizer state to CPU RAM during memory spikes | \n\nResist the urge to raise rank, epochs, and learning rate at once. Change one setting per run and keep the evaluation script as the judge. A rank of 64 on 800 rows mostly buys you overfitting.\n\nHardware also shapes this step. NVIDIA has been pushing [local AI](https://shattered.io/nvidia-vs-amd-gpu-local-ai-32gb-vs-24gb-2026/) toward cards with 24 GB and more, a trend covered in the report on [NVIDIA local AI on 24GB+ RTX GPUs](https://shattered.io/nvidia-local-ai-24gb-vram-vllm-llama-cpp-2026/). If your card has less memory, drop `max_length` to 512 first, then choose a smaller base model before you touch anything else.\n\n## Step 7: Read the Training Log Without Guessing\n\nThe trainer prints a dictionary every five steps. The number that matters is `loss`, and a healthy run shows it falling fast for the first 10 to 20 steps, then flattening. With this synthetic data the loss should drop low, because the rules are deterministic and the completion is short JSON. A flat line near the starting value means the model is not learning. A line that hits zero almost instantly means something leaked into the prompt.\n\n```\n{'loss': 2.31, 'grad_norm': 1.9, 'learning_rate': 0.00019, 'epoch': 0.1}\n{'loss': 0.84, 'grad_norm': 0.9, 'learning_rate': 0.00018, 'epoch': 0.2}\n{'loss': 0.21, 'grad_norm': 0.5, 'learning_rate': 0.00016, 'epoch': 0.3}\n{'loss': 0.08, 'grad_norm': 0.3, 'learning_rate': 0.00011, 'epoch': 1.0}\n{'loss': 0.04, 'grad_norm': 0.2, 'learning_rate': 0.00002, 'epoch': 2.0}\nadapter saved to out/adapter\n```\n\nThose values only show the expected shape. Your first loss depends on the model and tokenizer, and the exact decay depends on your card and settings. Do not compare absolute numbers with someone else’s run. Compare the slope, and then compare the evaluation results in Step 11.\n\nWatch three other signals. First, `grad_norm` should stay bounded. A sudden jump to a huge value or NaN means the learning rate is too high for this model, so halve it. Second, GPU memory should plateau after the first few steps. If it climbs steadily, a long outlier row is probably stretching a batch, which is why a hard `max_length` helps. Third, watch wall-clock time per step. If a step takes far longer than the first few, the paged optimizer may be swapping to CPU RAM, and you should shrink the sequence length or the rank.\n\nCheck the output directory when training ends. You should see `adapter_config.json`, `adapter_model.safetensors`, tokenizer files, and an epoch checkpoint folder or two. The adapter file is small compared with the base model, which is the whole point of LoRA. If the adapter file is missing and only checkpoints exist, the final save failed, and you can recover from the latest checkpoint directory instead of rerunning the job.\n\nKeep a note of the exact command, environment variables, and package versions for each run. When a result looks surprising two weeks later, that note is the only way to reproduce it. A single line from `pip freeze | grep -E \"torch|transformers|peft|trl|bitsandbytes\"` pasted into your notes costs ten seconds and saves hours.\n\n## Steps 8 and 9: Merge the Adapter and Export GGUF\n\n**Step 8: Merge the adapter into a full-precision copy.** Never merge into the 4-bit weights. Rounding the merged values back into NF4 throws away part of what the adapter learned. Load the base model in bf16 on the CPU, apply the adapter, call `merge_and_unload`, and save the result. The CPU route needs system RAM for the full model at roughly two bytes per parameter, so plan for tens of gigabytes. It is slower than merging on the GPU, but it never fails with an out-of-memory error on a small card.\n\n``` python\n# scripts/merge.py\nimport os\n\nimport torch\nfrom peft import PeftModel\nfrom transformers import (\n    AutoModelForCausalLM,\n    AutoModelForImageTextToText,\n    AutoTokenizer,\n)\n\nBASE_MODEL = os.environ.get(\"BASE_MODEL\", \"Qwen/Qwen3.8-27B\")\nADAPTER = os.environ.get(\"ADAPTER\", \"out/adapter\")\nMERGED = os.environ.get(\"MERGED\", \"out/merged\")\n\nload_args = dict(dtype=torch.bfloat16, device_map=\"cpu\")\ntry:\n    base = AutoModelForCausalLM.from_pretrained(BASE_MODEL, **load_args)\nexcept (ValueError, KeyError):\n    base = AutoModelForImageTextToText.from_pretrained(BASE_MODEL, **load_args)\n\nmodel = PeftModel.from_pretrained(base, ADAPTER)\nmodel = model.merge_and_unload()\nmodel.save_pretrained(MERGED, safe_serialization=True)\nAutoTokenizer.from_pretrained(BASE_MODEL).save_pretrained(MERGED)\nprint(\"merged model written to\", MERGED)\n```\n\n**Step 9: Convert to GGUF and quantize.** llama.cpp provides both tools. Clone the repository, build it with CUDA if you want GPU-accelerated tools, convert the merged Hugging Face folder to an f16 GGUF, then quantize to Q4_K_M. Ollama v0.30 improved GGUF compatibility through llama.cpp, so a recent Ollama will read the result without extra flags. Q4_K_M is the usual balance of size and quality for local use.\n\n```\ngit clone https://github.com/ggml-org/llama.cpp\ncd llama.cpp\npython -m pip install -r requirements.txt\ncmake -B build -DGGML_CUDA=ON\ncmake --build build --config Release -j\ncd ..\n\npython llama.cpp/convert_hf_to_gguf.py out/merged \\\n  --outfile out/triage-f16.gguf --outtype f16\n\nllama.cpp/build/bin/llama-quantize \\\n  out/triage-f16.gguf out/triage-q4_k_m.gguf Q4_K_M\n\nls -lh out/*.gguf\n```\n\nExpect the f16 file to be several times larger than the quantized one. The conversion log prints the architecture it detected. If it prints an error about an unsupported architecture, your llama.cpp checkout is older than the model. Pull the latest master, rebuild, and try again. The project moves quickly, so a fresh checkout is the safest fix.\n\nVision-language checkpoints add one wrinkle. The converter handles the language model, and the vision encoder usually travels as a separate projector file in the GGUF world. For a text-only triage task you can ignore the vision side, though you should confirm in the llama.cpp documentation that your specific architecture converts the text tower cleanly. If it does not, a smaller dense text model removes the problem entirely.\n\n## Steps 10 and 11: Import Into Ollama and Score the Result\n\n**Step 10: Build the Modelfile.** Ollama needs a chat template that matches the format the model trained on. The easiest way to get it right is to copy the template from the base model’s Ollama entry. Pull the base model, dump its Modelfile, and swap the `FROM` line to point at your GGUF. Replace `BASE_TAG` with the Ollama tag of the base model you started from.\n\n```\nexport BASE_TAG=your-base-model-tag\nollama pull \"$BASE_TAG\"\nollama show --modelfile \"$BASE_TAG\" > out/base.Modelfile\n\nsed 's|^FROM .*|FROM ./out/triage-q4_k_m.gguf|' out/base.Modelfile > Modelfile\n\ncat >> Modelfile <<'EOF'\nPARAMETER temperature 0\nPARAMETER num_ctx 4096\nEOF\n\nollama create ticket-triage -f Modelfile\nollama run ticket-triage \"Hi team, I was charged twice for my Team plan this month. Ref 4821. This is blocking me.\"\n```\n\nA working model replies with one JSON object and nothing else. If it rambles, repeats itself, or emits template tokens, the template is wrong. Open `Modelfile` and compare its `TEMPLATE` block with the base model’s. If your base model emits reasoning text before the answer, remember that the tuned model learned to answer directly, so the two models will not behave the same way under the same template.\n\nOnce the model answers, you can plug it into the tools you already use. The [OpenCode setup with Ollama](https://shattered.io/open-source-ai-coding-assistant-opencode-ollama-2026/) shows how a local model slots into an editor workflow, and the same `ollama create` flow works for any tuned model you want available there.\n\n**Step 11: Score both models on the held-out rows.** The evaluator calls Ollama’s chat endpoint at temperature 0, parses each reply as JSON, and counts valid objects, category hits, and priority hits. Run it twice, once for the base tag and once for `ticket-triage`.\n\n``` python\n# scripts/eval.py\nimport json\nimport os\nimport sys\n\nimport requests\n\nHOST = os.environ.get(\"OLLAMA_URL\", \"http://localhost:11434\")\nMODEL = sys.argv[1]\nREQUIRED = {\"category\", \"priority\", \"needs_human\", \"summary\"}\n\ndef ask(messages):\n    reply = requests.post(\n        HOST + \"/api/chat\",\n        json={\n            \"model\": MODEL,\n            \"messages\": messages,\n            \"stream\": False,\n            \"options\": {\"temperature\": 0},\n        },\n        timeout=300,\n    )\n    reply.raise_for_status()\n    return reply.json()[\"message\"][\"content\"]\n\ntotal = valid = category_hits = priority_hits = 0\nwith open(\"data/test.jsonl\") as handle:\n    for line in handle:\n        row = json.loads(line)\n        total += 1\n        raw = ask(row[\"prompt\"])\n        try:\n            parsed = json.loads(raw)\n        except json.JSONDecodeError:\n            continue\n        if not isinstance(parsed, dict) or not REQUIRED.issubset(parsed):\n            continue\n        valid += 1\n        category_hits += parsed[\"category\"] == row[\"label\"][\"category\"]\n        priority_hits += parsed[\"priority\"] == row[\"label\"][\"priority\"]\n\nprint(\"model=\" + MODEL, \"total=\" + str(total))\nprint(\"valid_json=\" + format(valid / total, \".1%\"))\nprint(\"category_acc=\" + format(category_hits / total, \".1%\"))\nprint(\"priority_acc=\" + format(priority_hits / total, \".1%\"))\n```\n\nThe evaluator counts accuracy over all 100 rows, so a reply that fails to parse counts as a miss on every metric. That choice punishes format failures honestly. A base model that gets the category right but wraps the JSON in a markdown fence scores zero on that row, and that is exactly the failure fine-tuning is meant to remove.\n\n## Step 12: Run the Whole Pipeline and Compare Base vs Tuned\n\nWith every script in place, one shell file reruns the project from scratch. This is the complete working project in a single entry point. Keep it in version control alongside the scripts, because a pipeline you can rerun with one command is a pipeline you can improve safely.\n\n``` bash\n#!/usr/bin/env bash\n# run_all.sh\nset -euo pipefail\n\nexport BASE_MODEL=\"${BASE_MODEL:-Qwen/Qwen3.8-27B}\"\nexport BASE_TAG=\"${BASE_TAG:?set BASE_TAG to the Ollama tag of the base model}\"\n\npython scripts/preflight.py\npython scripts/make_dataset.py\npython scripts/train.py\npython scripts/merge.py\n\npython llama.cpp/convert_hf_to_gguf.py out/merged \\\n  --outfile out/triage-f16.gguf --outtype f16\nllama.cpp/build/bin/llama-quantize \\\n  out/triage-f16.gguf out/triage-q4_k_m.gguf Q4_K_M\n\nollama show --modelfile \"$BASE_TAG\" > out/base.Modelfile\nsed 's|^FROM .*|FROM ./out/triage-q4_k_m.gguf|' out/base.Modelfile > Modelfile\nprintf 'PARAMETER temperature 0\\nPARAMETER num_ctx 4096\\n' >> Modelfile\nollama create ticket-triage -f Modelfile\n\npython scripts/eval.py \"$BASE_TAG\"\npython scripts/eval.py ticket-triage\n```\n\nRun it with `BASE_TAG=your-base-model-tag bash run_all.sh`. The last two commands print the comparison. The shape of a successful result looks like the block below, with your own numbers in place of the placeholders.\n\n```\nmodel=your-base-model-tag total=100\nvalid_json=(lower, base model wraps or pads its output)\ncategory_acc=(often high even before tuning)\npriority_acc=(low, the rules are unknown to the base model)\n\nmodel=ticket-triage total=100\nvalid_json=(near 100%)\ncategory_acc=(near 100%)\npriority_acc=(near 100% if the rules were learned)\n```\n\nRead the result with care. A tuned model that scores near 100 percent on synthetic rows proves it learned the rules. It does not prove it will handle real customer messages, which are messier than templates. The honest next step is to collect 50 to 100 real tickets, label them by hand, and run the same evaluator on them. If accuracy drops sharply, your synthetic data is too clean, and the fix is more varied training rows, not more epochs.\n\nA second comparison is worth running if you care about cost, since the reason to fine tune LLM behavior into the weights is often a shorter prompt. Count the prompt tokens the base model needs to behave, including your system prompt and any few-shot examples. The tuned model needs only the short system prompt, so every request is cheaper and faster. For a high-volume classifier, that saving can matter more than the accuracy gain. The [local Aider and Ollama coding setup](https://shattered.io/aider-local-ai-coding-assistant-ollama-git-2026/) is another example of the same pattern, where a model running on your own hardware replaces a metered API call.\n\n## The Faster Path: Unsloth for QLoRA and GGUF Export\n\nThe Hugging Face stack above is the most transparent route, since every step is a separate, inspectable script. Unsloth collapses several of those steps into one library. Its own site says it trains models 2 times faster with 70 percent less VRAM and no accuracy loss. Those are vendor claims, so measure them on your hardware before you plan around them. The core package is Apache 2.0 licensed, while the Studio interface carries AGPL-3.0. The [Unsloth pricing page](https://unsloth.ai/pricing) lists a free open-source edition, and the Pro and Enterprise tiers are quote-based with no public prices.\n\nThe project has also grown beyond a single-GPU trainer. A third-party tracker reports a desktop app and a Studio web interface by early October, and Unsloth lists GGUF among its export options. According to AMD’s developer site, Unsloth training and inference on AMD GPUs shipped in July. AMD’s own article on [training and running models on AMD GPUs with Unsloth](https://www.amd.com/en/developer/resources/technical-articles/2026/train-and-run-models-on-amd-gpus-with-unsloth.html) is the place to start if you do not own an NVIDIA card. That removes the biggest hardware constraint in the stack above, which assumes CUDA.\n\nA minimal Unsloth version of Steps 4 to 9 looks like the sketch below. The method names follow the library’s documented pattern, but the project ships beta tags frequently, so confirm the current signatures in the Unsloth docs before you copy this into production. Reuse the same `BASE_MODEL` variable and the same dataset.\n\n```\n# scripts/train_unsloth.py (sketch, check the Unsloth docs for current signatures)\nimport os\n\nfrom datasets import load_dataset\nfrom trl import SFTConfig, SFTTrainer\nfrom unsloth import FastLanguageModel\n\nBASE_MODEL = os.environ.get(\"BASE_MODEL\", \"Qwen/Qwen3.8-27B\")\n\nmodel, tokenizer = FastLanguageModel.from_pretrained(\n    model_name=BASE_MODEL,\n    max_seq_length=1024,\n    load_in_4bit=True,\n)\nmodel = FastLanguageModel.get_peft_model(\n    model,\n    r=16,\n    lora_alpha=32,\n    lora_dropout=0.0,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n                    \"gate_proj\", \"up_proj\", \"down_proj\"],\n)\n\ndataset = load_dataset(\"json\", data_files=\"data/train.jsonl\", split=\"train\")\ntrainer = SFTTrainer(\n    model=model,\n    train_dataset=dataset,\n    processing_class=tokenizer,\n    args=SFTConfig(\n        output_dir=\"out/adapter-unsloth\",\n        num_train_epochs=2,\n        per_device_train_batch_size=1,\n        gradient_accumulation_steps=16,\n        learning_rate=2e-4,\n        logging_steps=5,\n        report_to=\"none\",\n    ),\n)\ntrainer.train()\n\nmodel.save_pretrained_gguf(\"out/unsloth-gguf\", tokenizer, quantization_method=\"q4_k_m\")\n```\n\nThe tradeoff is control. Unsloth hides the merge and conversion work, which saves time when it works and hides the cause when it does not. If you are shipping a model that other people depend on, run the manual pipeline once so you know what each stage produces. Then switch to the faster path for iteration.\n\nServing is the other half of the speed story. Training gets the attention, but inference engines change what a tuned model costs to run. DeepSeek’s recent [DeepInfer open-source inference release](https://shattered.io/deepseek-deepinfer-open-source-inference-pressure-2026/) is one sign that the serving layer is now as competitive as the models, and a GGUF file you export today can move to a different runtime later without retraining.\n\n## Common Pitfalls When You Fine-Tune an LLM\n\nMost failed attempts to fine tune LLM checkpoints trace back to a short list of mistakes. Each one below has burned real projects, and each is cheap to avoid if you know it exists. Read the list before your first run, then reread it when a result looks wrong. For more articles on model internals and tooling, the [AI and machine learning section](https://shattered.io/category/ai-machine-learning/) collects related coverage.\n\n1. **Leaking test rows into training.** If a test prompt also appears in the training file, the evaluation score means nothing. The dataset script removes overlap by text. Do the same with any real data, and deduplicate near-matches too.\n2. **Using the wrong chat template.** A model trained on one template and served with another produces garbled or rambling output. Copy the template from the base model, as Step 10 does, and test with a prompt before you run the evaluator.\n3. **Merging into 4-bit weights.** Always merge the adapter into a bf16 copy. Merging into quantized weights loses precision that the adapter just learned.\n4. **Treating a high training score as success.** Synthetic data is clean. A model can hit near-perfect accuracy on it and still fail on real messages. Always run a small hand-labeled set of real examples.\n5. **Raising the learning rate to speed things up.** LoRA tolerates higher rates than full fine-tuning, but 2e-4 is already high. Going further risks NaN loss and a ruined run that looks fine for 20 steps.\n6. **Forgetting the end-of-sequence token.** If training rows do not end the completion properly, the tuned model never stops generating. The prompt and completion format in TRL handles this when the chat template is correct, so a rambling model is another sign of a template mismatch.\n7. **Ignoring the license.** Your tuned model inherits the base model’s terms. Gemma 4 launched under Apache 2.0 and a third-party catalog lists Qwen3.8-27B the same way, but verify the license on the model card for the exact checkpoint you use.\n8. **Filling the disk.** The base weights, merged copy, f16 GGUF, and quantized GGUF add up fast. Delete the f16 file after quantizing, and delete old epoch checkpoints you no longer need.\n\nA ninth habit protects you from all of them: change one variable per run and write down what you changed. When you tweak rank, learning rate, and data in the same run, a better score teaches you nothing, and a worse one leaves you debugging three things at once.\n\n## Troubleshooting: 10 Failures and Their Fixes\n\nWork through these in order. Most errors in this pipeline come from version mismatches or memory limits, and the fixes are short.\n\n1. **CUDA out of memory during training.** Lower`max_length` to 512, keep the batch at 1, raise gradient accumulation, and confirm gradient checkpointing is on. If it still fails, choose a smaller base model.\n2. **bitsandbytes reports no GPU support or a CUDA setup failure.** Your PyTorch build and driver disagree, or you installed a CPU-only wheel. Reinstall a CUDA build of PyTorch that matches your driver, then reinstall bitsandbytes.\n3. **Loss becomes NaN or jumps.** Halve the learning rate, confirm bf16 is supported on your card (the preflight script prints it), and check for an empty or malformed row in the dataset.\n4. **The loader raises an error about an unrecognized model type.** Your Transformers release is older than the model. Upgrade to 5.17.0 or newer, and read the model card for any minimum version.\n5. **The causal-LM loader fails on a vision-language checkpoint.** The scripts fall back to the image-text class automatically. If that also fails, print the architecture from the preflight script and search the model card for the right loader class.\n6. **TypeError about an unexpected `max_seq_length` argument.** Current TRL uses`max_length` in`SFTConfig` . Update the argument name, and pin TRL v1.16.0 or newer.\n7. **Merge script runs out of memory.** Keep`device_map=\"cpu\"` and add swap or RAM. The merge needs the full bf16 model in memory, at roughly two bytes per parameter.\n8. **The GGUF converter reports an unsupported architecture.** Pull the latest llama.cpp master, rebuild, and rerun the conversion. If the architecture still fails, check the project’s issue tracker for support status.\n9. **Ollama prints garbage or never stops.** The template or stop tokens in your Modelfile do not match the model. Rebuild the Modelfile from the base model’s, as in Step 10, and keep the`FROM` line as the only change.\n10. **Valid JSON rate stays low after tuning.** Check the template first, then the loss curve. If the loss never dropped, the dataset format is wrong. If it dropped but replies still fail, train one more epoch or lower the temperature further.\n\nGated downloads deserve a mention too. If a download from the Hugging Face hub fails, run `hf auth login` and retry.\n\n## Advanced Tips: Preference Tuning, Adapters and Safe Data\n\nOnce the basic loop works, the same stack opens several upgrades. TRL covers more than supervised fine-tuning. It also provides DPO, KTO, GRPO, RLOO, and distillation trainers, so you can add a preference-tuning pass after SFT without leaving the library. A common pattern is to run SFT for format, then run DPO on pairs of good and bad replies to sharpen edge cases such as ambiguous billing messages. TRL v1.16.0 also added AWS Trainium support, which matters if your training budget lives on that hardware instead of an NVIDIA card.\n\nYou can also skip the merge. Ollama can load a LoRA adapter on top of a base model through an `ADAPTER` line in the Modelfile, which lets one base model serve several tuned behaviors. Support depends on the architecture, so test it before you rely on it. The merged GGUF route in this build is the safer default, and the adapter route is worth trying when you maintain many small variants.\n\n```\n# Modelfile (adapter variant, architecture support varies)\nFROM your-base-model-tag\nADAPTER ./out/adapter\nPARAMETER temperature 0\n```\n\nSweep hyperparameters with discipline. Try rank 8, 16, and 32 with everything else fixed, and keep the one that wins on your hand-labeled set instead of the synthetic one. Try one epoch against two against three, and watch where accuracy stops improving. Past that point, extra epochs only memorize your templates. Sequence packing can speed up training on short rows like these, but check that your chat template and loss masking still behave after you enable it.\n\n### Keep Training Data Clean and Output Validated\n\nTraining data is an attack surface and a liability. Scrub personal data before it reaches a training file, because a model can reproduce memorized strings, and removing them afterward is nearly impossible. Strip API keys, tokens, email addresses, and account numbers from real tickets. Treat any dataset you did not create as untrusted, since a poisoned row can plant a behavior that only triggers on a specific phrase.\n\nTreat the model’s output as untrusted too. A tuned classifier still reads customer text, and a customer can write instructions into a ticket. Validate every reply against a JSON schema in your application code, reject anything with unknown keys or values outside your allowed sets, and never let the model’s output trigger an action without that check. Pin base-model revisions in your download call, record the hash of each adapter and GGUF you ship, and store them where you store other build artifacts. Those habits turn a pile of weights into something you can audit.\n\n``` python\n# validate.py: reject anything the model should never return\nimport json\n\nALLOWED = {\n    \"category\": {\"billing\", \"bug\", \"account_access\", \"feature_request\", \"security\"},\n    \"priority\": {\"low\", \"medium\", \"high\", \"urgent\"},\n}\n\ndef parse_ticket(raw):\n    data = json.loads(raw)\n    if set(data) != {\"category\", \"priority\", \"needs_human\", \"summary\"}:\n        raise ValueError(\"unexpected keys\")\n    if data[\"category\"] not in ALLOWED[\"category\"]:\n        raise ValueError(\"bad category\")\n    if data[\"priority\"] not in ALLOWED[\"priority\"]:\n        raise ValueError(\"bad priority\")\n    if not isinstance(data[\"needs_human\"], bool):\n        raise ValueError(\"needs_human must be a boolean\")\n    return data\n```\n\n### Scale Evaluation Before You Trust the Model\n\nOne hundred synthetic rows is a smoke test, not a benchmark. Before the model touches production traffic, build an evaluation set from real, hand-labeled tickets, include the ugly cases (typos, mixed languages, two problems in one message), and track accuracy per category. A model that scores well overall but fails on security tickets is the dangerous kind, because those are the ones you cannot afford to misroute. Rerun the evaluator after every change to data, rank, or base model, and keep the results in a table in your repository.\n\nFinally, plan for the next base model. The open-weight field is moving month to month, and Qwen’s team has been reported to be training a successor, though nothing official has shipped. Your pipeline already isolates the base model behind one environment variable, so a swap costs one training run, not a rewrite. That is the real payoff of building the project as scripts instead of a notebook.\n\n## FAQ\n\n**How much VRAM do I need to fine tune LLM weights with QLoRA?**\n\nIt depends on model size, sequence length, and batch size. The 2023 QLoRA paper reported a 65B model on a single 48GB GPU, and Hugging Face’s 2023 write-up on 4-bit Transformers described 33B fine-tuning on a single 24GB GPU. A 12B model is comfortable on a 16 GB card, and a 27B model needs a 24 GB card with short sequences. Check your card with the preflight script before you download anything large.\n\n**Is fine-tuning better than RAG?**\n\nThey solve different problems. Fine-tuning changes behavior and format, while retrieval supplies facts the model does not have. Use retrieval for changing knowledge and tuning for consistent output. Many production systems use both.\n\n**Can I fine-tune on a Mac or an AMD GPU?**\n\nYes, with caveats. bitsandbytes targets CUDA, so the scripts here assume an NVIDIA card. According to AMD’s developer site, Unsloth added AMD GPU training and inference in July. For Apple Silicon, Ollama’s own MLX engine is the more established route for inference. Check each project’s current documentation for supported hardware before you commit.\n\n**How many training examples do I need?**\n\nFor a narrow format-and-rules task, a few hundred clean examples often work, and this project uses 800. Quality beats quantity. A smaller set of varied, correctly labeled rows outperforms a large set of repetitive or noisy ones. Add rows where the evaluator shows failures.\n\n**Does my tuned model keep the base model’s license?**\n\nYour tuned weights fall under the base model’s license terms. Gemma 4 launched under Apache 2.0, and a third-party catalog lists Qwen3.8-27B under Apache 2.0, but confirm the license on the exact model card you downloaded and read any use restrictions.\n\n**Should I merge the adapter or keep it separate?**\n\nMerge when you want one portable file that runs anywhere GGUF runs. Keep the adapter separate when you serve several behaviors from one base model and your runtime supports adapters for that architecture. Merging into a bf16 copy first is the safe path for both.\n\n**Is Ollama free for running my tuned model?**\n\nRunning models locally on your own hardware is free. Ollama also sells cloud plans, and its pricing page lists the Pro tier at $20 a month or $200 a year, with Team plans starting at $500 a month. Your tuned GGUF runs on the local runtime and does not need any of them.", "url": "https://wpnews.pro/news/fine-tune-an-llm-with-qlora-12-steps-90-min-2026", "canonical_source": "https://shattered.io/fine-tune-llm-qlora-12-steps-2026/", "published_at": "2026-10-25 10:31:34+00:00", "updated_at": "2026-10-25 10:42:03.204483+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "machine-learning", "ai-research"], "entities": ["Shattered", "Tim Dettmers", "Hugging Face", "TRL", "PEFT", "Ollama", "Qwen3.8-27B", "Alibaba"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fine-tune-an-llm-with-qlora-12-steps-90-min-2026", "markdown": "https://wpnews.pro/news/fine-tune-an-llm-with-qlora-12-steps-90-min-2026.md", "text": "https://wpnews.pro/news/fine-tune-an-llm-with-qlora-12-steps-90-min-2026.txt", "jsonld": "https://wpnews.pro/news/fine-tune-an-llm-with-qlora-12-steps-90-min-2026.jsonld"}}