cd /news/ai-tools/merge-a-lora-adapter-into-gguf-12-st… · home › topics › ai-tools › article
[ARTICLE · art-171732] src=shattered.io ↗ pub= topic=ai-tools verified=true sentiment=· neutral

Merge a LoRA Adapter Into GGUF: 12 Steps, 80 Min

Llama.cpp v1.0.0, released November 13, 2026 as the project's first stable version, requires Hugging Face PEFT LoRA adapters to be converted to GGUF before llama.cpp or Ollama can run them locally, and a walkthrough script called ship_adapter.py takes a public QLoRA adapter for Qwen2.5-1.5B-Instruct through preflight checks, a merge_and_unload merge, a logit comparison against the unmerged adapter, a bf16 GGUF, a Q4_K_M GGUF and an Ollama model in about 80 minutes on a 16 GB laptop. The guide says most failures come from merging into the wrong precision, dropping the tokenizer, shipping a DoRA adapter through a path that cannot carry it, or running a converter from a random master branch, each producing a model that loads fine and answers badly. It recommends merging into the original 16-bit base weights rather than a 4-bit NF4 base, since quantizing twice stacks error on error.

recommendation

Merge LoRA adapters into the 16-bit base weights, not the 4-bit NF4 base, then convert to bf16 GGUF and quantize once to Q4_K_M to avoid stacking quantization error.

more in playbook: Inference and self-hosting
by read32 min views9 publishedNov 18, 2026
Merge a LoRA Adapter Into GGUF: 12 Steps, 80 Min
Image: Shattered (auto-discovered)

The llama.cpp v1.0.0 release landed on November 13, 2026 as the project’s first stable version, and it tightened a rule that trips up many fine-tuners: a LoRA adapter trained in Hugging Face PEFT cannot run on local hardware until it becomes a GGUF file that llama.cpp or Ollama can read. The adapter alone is just a set of small low-rank matrices. It needs its exact base model, the right tokenizer, and a conversion path that respects how it was trained.

Most failures happen at that handoff. People merge into the wrong precision, forget the tokenizer, ship a DoRA adapter through a path that cannot carry it, or run a converter from a random master branch. Each mistake produces a model that loads fine and answers badly, which is the worst kind of bug because nothing crashes.

One Python script, ship_adapter.py, takes a public QLoRA adapter for Qwen2.5-1.5B-Instruct through every hop: preflight checks, a merge_and_unload merge, a logit comparison against the unmerged adapter, a bf16 GGUF, a Q4_K_M GGUF, and an Ollama model. The same script works on adapters you trained yourself, including the ones from a QLoRA fine-tune of an open model. Budget about 80 minutes on a laptop with 16 GB of RAM.

Don't miss new tech stories on Google

Add Shattered once in the Google app and our stories show up in your news suggestions.

Add now

What a LoRA Adapter Is and Why You Merge It #

LoRA freezes the original weight matrix W and learns two thin matrices, A and B, whose product forms a correction. At inference the effective weight is W plus (alpha divided by r) times B times A, where r is the rank and alpha is a scaling constant. A LoRA adapter file stores only A and B for each targeted module, plus an adapter_config.json that names the base model it was trained against. That is why adapters are small and why they are useless without the right base.

You have two ways to ship one. The first is to merge: add the correction into W once, save a normal checkpoint, and forget the adapter existed. The merged model is the same size as the base, runs at base-model speed, and needs no adapter-aware runtime. The second is to keep the adapter separate: convert it to its own GGUF and let llama.cpp apply it at load time with the --lora flag. That costs a little speed but lets you swap behaviors on one base model.

Rank matters during training and disappears after a merge. If you ran a LoRA rank sweep and picked rank 16 over rank 64, the merged checkpoint is exactly the same size either way. Rank changes how much the adapter can learn, not how big the finished model is.

QLoRA adds one wrinkle. The adapter was trained against a base model quantized to 4-bit NF4, but the correction itself is stored in 16-bit precision. When you merge, load the original 16-bit base weights and merge into those. Merging into a 4-bit base either fails or bakes quantization noise into the new weights, and you will quantize again later for GGUF anyway. Quantizing twice stacks error on error.

Merge-first is the default path here for a practical reason. It works for every adapter type PEFT can merge, including DoRA and adapters that carry modified embeddings. Adapter-only GGUF conversion is narrower, so it appears later as Option B, with a compatibility table that tells you when it is safe.

What llama.cpp v1.0.0 Changed for Adapter Workflows #

The November 13 release put llama.cpp and ggml on semantic versioning, with libllama and libggml as the two published libraries. A wire report on the release describes the new promise: breaking API or ABI changes now require a major version bump. For people who only run the command-line tools, the headline matters less than the cleanup that came with it.

The 1.0 transition removed deprecated public interfaces, which can break downstream integrations that linked against old symbols. One breakdown of what breaks in llama.cpp 1.0 walks through the affected areas. For adapter work, the relevant removal is the old --lora-base merge path. Current documentation, dated November 16, no longer offers it, so any older tutorial that tells you to merge by passing a base file to the will fail on a v1.0.0 build.

Two earlier changes shape the workflow too. The v0.8.0 release on October 28 strengthened GGUF validation for the split, export-lora and merge operations, so a malformed file now fails fast instead of half-broken. And the project keeps shipping numbered pre-release builds between tags. Build b12503, marked a pre-release, was the newest on November 18. Treat those as a moving target and pin the v1.0.0 tag.

The server changed as well. On November 13, llama-server gained hardening against SSRF, CORS abuse and DNS rebinding, added a /v1/audio/speech endpoint, and shipped a redesigned web interface with chat groups and a Markdown editor. Coverage of the release says it did not change the GGUF file format itself, which means GGUF files you converted last month still load.

The conversion scripts remain under active maintenance. Issue threads from November 8 to 11 reportedly cover how convert_lora_to_gguf.py handles rsLoRA scaling, including a proposed --fold-scale option. Since that work was still open at the time of writing, the safest assumption is that adapter-only conversion of rsLoRA adapters needs a manual check, which Step 12 covers.

The takeaway: pin v1.0.0, avoid tutorials written before November 2026, and keep the merge path as your baseline.

Prerequisites and Tested Versions #

The merge itself is light. A 1.5B-parameter model in bf16 needs roughly 3 GB of RAM, so any modern laptop handles it, with or without a GPU. The heavy parts are disk space and the llama.cpp build. Scale matters if you bring your own adapter: a 27B model such as Qwen3.8-27B needs about 54 GB of RAM for a bf16 merge (27 billion parameters at 2 bytes each), so check the arithmetic against your hardware before you start. If you are still choosing where the finished model will live, the three ways to run open-source models frame the choice between local, rented and hosted hardware.

Component Version or requirement Why it is needed
Python 3.12 Runs PEFT, Transformers and the llama.cpp converters
llama.cpp v1.0.0 tag, built from source GGUF conversion, quantization and serving
Hugging Face PEFT Current release from PyPI Loads the adapter and runs merge_and_unload
Transformers and PyTorch Current releases from PyPI Loads the base model and runs the smoke test
CMake and a C++ compiler CMake 3.28 or newer, GCC 13 or Clang 17 Builds llama.cpp
Ollama A current release Imports the quantized GGUF as a named model
Free disk space 15 GB Base, merged copy, bf16 GGUF, Q4_K_M GGUF and the build tree
RAM 16 GB (more for larger models) Holds base and adapter copies during the merge

Run pip freeze after Step 1 and keep the output. PEFT and Transformers move quickly, and an adapter that merged cleanly in one release can behave differently in another. A saved lockfile turns “it worked last week” into something you can reproduce.

The time budget breaks down like this: 10 minutes for setup, 15 for downloads, 10 for the merge and smoke test, 20 for the llama.cpp build, and 25 for conversion, quantization, serving and the Ollama import. A GPU shortens the build and inference steps but is not required for any step.

On model choice: the worked example uses Qwen2.5-1.5B-Instruct because a public QLoRA adapter for it exists, it merges in minutes, and a bad result is obvious within three prompts. It is an older Qwen generation. Newer families work with the same steps, but they raise the runtime requirements. One community listing for a Qwen 3.8 model notes that older Ollama builds fail at prompt rendering with an unknown renderer "qwen3.8" error and need Ollama 0.32.12 or newer. Match your runtime to your model family before you blame your merge.

Prepare the Workspace #

Step 1: Create the project and the Python environment

Keep the merge environment separate from the llama.cpp converter environment. The converter’s requirements file can pin its own PyTorch build, and mixing the two is a classic way to wreck a working setup.

mkdir adapter-shipper && cd adapter-shipper
mkdir -p models adapters merged gguf tools

python3.12 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install torch transformers peft accelerate safetensors sentencepiece huggingface_hub
pip freeze > requirements.lock

python -c "import peft, transformers, torch; print(peft.__version__, transformers.__version__, torch.__version__)"

The last command prints three version strings. Write them into your notes. If the import fails, the install did not finish, and every later step will fail with confusing errors.

Step 2: Download the base model and the adapter

Use snapshot_download from Python rather than a command-line tool, because the CLI name has changed across huggingface_hub releases and a script keeps working across all of them. Save this as fetch.py.

import sys
from huggingface_hub import snapshot_download

base_id, adapter_id = sys.argv[1], sys.argv[2]

base_path = snapshot_download(base_id, local_dir="models/base")
adapter_path = snapshot_download(adapter_id, local_dir="adapters/demo")
print("base   ->", base_path)
print("adapter->", adapter_path)
python fetch.py Qwen/Qwen2.5-1.5B-Instruct ms-afridi/qwen2.5-1.5b-arxiv-title-lora
ls adapters/demo

The adapter repository is a public QLoRA adapter that was trained with 4-bit NF4 quantization and loads through PeftModel.from_pretrained. Expected listing:

README.md
adapter_config.json
adapter_model.safetensors
tokenizer.json
tokenizer_config.json

Your listing may differ slightly, and that is fine. The two files that matter are adapter_config.json and adapter_model.safetensors. If you only see a .bin weights file, the adapter predates safetensors defaults, and the preflight script in Step 4 needs a one-line change to read it.

Inspect and Preflight the Adapter #

Step 3: Read adapter_config.json before touching any weights

Five fields in the config decide which shipping path is safe: peft_type, base_model_name_or_path, use_dora, use_rslora and modules_to_save. Print them first. Save this as inspect_adapter.py.

import json
import sys
from pathlib import Path

adapter_dir = Path(sys.argv[1])
cfg = json.loads((adapter_dir / "adapter_config.json").read_text())

for key in ["peft_type", "base_model_name_or_path", "r", "lora_alpha",
            "lora_dropout", "target_modules", "use_dora", "use_rslora",
            "modules_to_save", "bias", "task_type"]:
    print(f"{key:26} {cfg.get(key)}")

r, alpha = cfg["r"], cfg["lora_alpha"]
scale = alpha / (r ** 0.5 if cfg.get("use_rslora") else r)
print(f"{'effective scale':26} {scale:.3f}")
print("files:", ", ".join(sorted(p.name for p in adapter_dir.iterdir())))
python inspect_adapter.py adapters/demo

Example output (your adapter’s values will differ):

peft_type                  LORA
base_model_name_or_path    Qwen/Qwen2.5-1.5B-Instruct
r                          16
lora_alpha                 32
lora_dropout               0.05
target_modules             ['q_proj', 'k_proj', 'v_proj', 'o_proj']
use_dora                   False
use_rslora                 False
modules_to_save            None
bias                       none
task_type                  CAUSAL_LM
effective scale            2.000

Check that base_model_name_or_path matches the base you downloaded. An adapter trained on the Instruct variant and merged into the base variant produces text that looks plausible and reads wrong. The effective scale is alpha divided by r for classic LoRA and alpha divided by the square root of r for rsLoRA. Keep that number in mind for the rsLoRA check in Step 12.

Step 4: Run a preflight check for unsupported adapter features

The preflight script returns two lists. Merge blockers stop everything. Adapter-GGUF blockers only rule out Option B, because the merge path handles those features fine. Save this as preflight.py.

import json
import sys
from pathlib import Path

from safetensors import safe_open

def preflight(adapter_dir: Path):
    cfg = json.loads((adapter_dir / "adapter_config.json").read_text())
    merge_blockers, adapter_gguf_blockers = [], []

    if cfg.get("peft_type") != "LORA":
        merge_blockers.append(f"peft_type is {cfg.get('peft_type')}, expected LORA")

    weights = adapter_dir / "adapter_model.safetensors"
    if not weights.exists():
        merge_blockers.append("adapter_model.safetensors is missing")
        return merge_blockers, adapter_gguf_blockers

    with safe_open(str(weights), framework="pt") as f:
        names = list(f.keys())

    if cfg.get("use_dora") or any("lora_magnitude_vector" in n for n in names):
        adapter_gguf_blockers.append("DoRA magnitude vectors: merge instead of converting the adapter")
    if cfg.get("modules_to_save"):
        adapter_gguf_blockers.append(f"modules_to_save={cfg['modules_to_save']}: full tensors live outside the low-rank pair")
    if any("lora_embedding" in n for n in names):
        adapter_gguf_blockers.append("embedding LoRA tensors present")
    if cfg.get("bias", "none") != "none":
        adapter_gguf_blockers.append(f"bias={cfg['bias']}: bias deltas are not part of a plain LoRA pair")
    if cfg.get("use_rslora"):
        adapter_gguf_blockers.append("rsLoRA: verify the converter's scale handling before trusting it")

    return merge_blockers, adapter_gguf_blockers

if __name__ == "__main__":
    mb, ab = preflight(Path(sys.argv[1]))
    print("merge blockers      :", mb or "none")
    print("adapter-gguf blockers:", ab or "none")
    sys.exit(1 if mb else 0)
python preflight.py adapters/demo
merge blockers      : none
adapter-gguf blockers: none

A third-party checker called lora-preflight reportedly does a similar job, flagging adapters with tensors that llama.cpp and Ollama cannot load and recommending merge_and_unload for them. The script above covers the same ground in 40 lines and has no extra dependency. The compatibility table below summarizes what the checks mean.

Adapter trait Merge, then GGUF Adapter-only GGUF What to do
Plain LoRA Works Works Either path, pick by deployment needs
QLoRA-trained LoRA Works Works Merge into the 16-bit base, never the 4-bit one
rsLoRA Works Verify the scale Merge, or compare logits after conversion
DoRA Works Not reliable Merge, and do not convert DoRA to plain LoRA to dodge the problem
modules_to_save (embeddings, lm_head) Works Not supported Merge only
Added tokens Works after an embedding resize Not supported Merge only, keep the adapter’s tokenizer

The DoRA row deserves a warning. An issue reported on November 16, 2026 says PEFT’s convert_to_lora and save_as_lora helpers can produce a model that differs from the trained DoRA model. Converting DoRA to LoRA looks like a clever way to unlock the adapter-only path, but until that issue closes, merging is the only route you can verify.

Merge the LoRA Adapter Into the Base Model #

Step 5: Merge with merge_and_unload

The merge script loads the 16-bit base, grows the embedding matrix only if the adapter added tokens, applies the adapter, folds it in, and saves the model together with the right tokenizer. Save this as merge.py.

import sys
from pathlib import Path

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

def merge(base_dir: Path, adapter_dir: Path, out_dir: Path, dtype=torch.bfloat16):
    tok_src = adapter_dir if (adapter_dir / "tokenizer_config.json").exists() else base_dir
    tokenizer = AutoTokenizer.from_pretrained(tok_src)

    model = AutoModelForCausalLM.from_pretrained(base_dir, dtype=dtype, low_cpu_mem_usage=True)

    rows = model.get_input_embeddings().num_embeddings
    if len(tokenizer) > rows:
        model.resize_token_embeddings(len(tokenizer))

    model = PeftModel.from_pretrained(model, adapter_dir)
    model = model.merge_and_unload()

    out_dir.mkdir(parents=True, exist_ok=True)
    model.save_pretrained(out_dir, safe_serialization=True)
    tokenizer.save_pretrained(out_dir)
    return out_dir

if __name__ == "__main__":
    base, adapter, out = (Path(p) for p in sys.argv[1:4])
    dtype = torch.float32 if len(sys.argv) > 4 and sys.argv[4] == "float32" else torch.bfloat16
    print("merged ->", merge(base, adapter, out, dtype))
python merge.py models/base adapters/demo merged/arxiv-titles
ls -lh merged/arxiv-titles

The guard on the embedding resize uses a greater-than check on purpose. Many model families pad the embedding matrix beyond the tokenizer’s vocabulary for hardware alignment, so a naive “sizes differ, so resize” rule would shrink the matrix and corrupt it. Only grow it when the adapter’s tokenizer is genuinely larger.

Example listing:

-rw-r--r-- 1 you you  707 config.json
-rw-r--r-- 1 you you  242 generation_config.json
-rw-r--r-- 1 you you 3.1G model.safetensors
-rw-r--r-- 1 you you  11M tokenizer.json
-rw-r--r-- 1 you you 7.2K tokenizer_config.json

The model file should match the base model’s size. If it comes out noticeably larger, something was saved in float32 by accident. If it comes out smaller, the merge produced a partial checkpoint and you should rerun it.

Step 6: Smoke-test the merged model against the unmerged adapter

A merge that finishes without errors proves nothing. The smoke test loads the base with the adapter attached on the fly, loads the merged checkpoint, and compares the last-token logits for a few prompts. Save this as smoke_test.py.

import sys
from pathlib import Path

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base, adapter, merged = (Path(p) for p in sys.argv[1:4])
THRESHOLD = 1.0

tok = AutoTokenizer.from_pretrained(merged)
ref = AutoModelForCausalLM.from_pretrained(base, dtype=torch.bfloat16)
ref = PeftModel.from_pretrained(ref, adapter).eval()
mrg = AutoModelForCausalLM.from_pretrained(merged, dtype=torch.bfloat16).eval()

prompts = [
    "Write a title for a paper on low-rank adapters for small language models.",
    "Suggest a title for a study of quantization error in 4-bit language models.",
    "Give a title for a paper about merging fine-tuned adapters into base weights.",
]

worst = 0.0
for p in prompts:
    text = tok.apply_chat_template([{"role": "user", "content": p}],
                                   tokenize=False, add_generation_prompt=True)
    ids = tok(text, return_tensors="pt")
    with torch.no_grad():
        a = ref(**ids).logits[0, -1].float()
        b = mrg(**ids).logits[0, -1].float()
        gen = mrg.generate(**ids, max_new_tokens=32, do_sample=False)
    gap = (a - b).abs().max().item()
    worst = max(worst, gap)
    reply = tok.decode(gen[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)
    print(f"gap={gap:.4f}  reply={reply.strip()!r}")

print(f"worst gap: {worst:.4f} (threshold {THRESHOLD})")
sys.exit(0 if worst < THRESHOLD else 1)
python smoke_test.py models/base adapters/demo merged/arxiv-titles

Example output (numbers and wording will differ):

gap=0.1250  reply='Low-Rank Adapters for Efficient Small Language Model Tuning'
gap=0.0938  reply='Measuring Quantization Error in 4-Bit Language Models'
gap=0.1562  reply='Folding Fine-Tuned Adapters Into Base Weights: A Practical Study'
worst gap: 0.1562 (threshold 1.0)

The threshold of 1.0 is a heuristic, not a standard. Logits in bf16 carry rounding noise, so identical math through two code paths rarely matches to the last digit. A broken merge, such as the wrong base or a missed tokenizer, usually produces much larger gaps. If your gaps sit near 1.0, rerun the merge in float32 by adding float32 as the fourth argument to merge.py.

Convert the Merged Model to GGUF #

Step 7: Build llama.cpp at the v1.0.0 tag

Clone into tools/, check out the tag, and build. Use a second virtual environment for the converter requirements.

git clone https://github.com/ggml-org/llama.cpp tools/llama.cpp
cd tools/llama.cpp
git checkout v1.0.0

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

python3.12 -m venv ../../.venv-llama
source ../../.venv-llama/bin/activate
pip install -r requirements.txt
cd ../..

Drop -DGGML_CUDA=ON for a CPU-only build. On Apple silicon, the Metal backend turns on by default. If git checkout v1.0.0 reports an unknown ref, run git tag --list "v1*" and use the tag it shows. The build takes about 20 minutes on a laptop without ccache.

tools/llama.cpp/build/bin/llama-quantize --help | head -n 5

If that prints usage text, the binaries exist. A “no such file” error means the build failed earlier, and the CMake output has the reason.

Step 8: Convert the merged checkpoint to a bf16 GGUF

Convert to bf16 first and quantize afterward. This keeps one lossless-ish master copy you can re-quantize at will.

source .venv-llama/bin/activate
python tools/llama.cpp/convert_hf_to_gguf.py merged/arxiv-titles \
  --outfile gguf/arxiv-titles-bf16.gguf --outtype bf16

Example output (abridged):

INFO:hf-to-gguf: model: arxiv-titles
INFO:hf-to-gguf:Model architecture: Qwen2ForCausalLM
INFO:hf-to-gguf:gguf:  model part 'model.safetensors'
INFO:hf-to-gguf:token_embd.weight, torch.bfloat16 --> BF16, shape = {1536, 151936}
...
INFO:hf-to-gguf:Model successfully exported to gguf/arxiv-titles-bf16.gguf

The converter reads the tokenizer and chat template from the merged folder and bakes them into the GGUF header. That is why Step 5 saved the tokenizer next to the weights. A merged folder without tokenizer files converts to a file that either errors or guesses the vocabulary.

The converter picks its code path from the architecture string in config.json, which is why one command handles families as different as Qwen, DeepSeek and Mistral, the three open-weight lines compared in DeepSeek vs Qwen vs Mistral. A brand-new family can still outrun the converter, so a model released last week may need a newer llama.cpp build than a model released last year.

Step 9: Quantize to Q4_K_M

tools/llama.cpp/build/bin/llama-quantize \
  gguf/arxiv-titles-bf16.gguf gguf/arxiv-titles-Q4_K_M.gguf Q4_K_M

Quantization packs weights into small blocks that share a scale. The llama.cpp documentation lists the bits per weight for each k-quant family, and from those you can estimate file sizes for a 1.5B-parameter model.

Quant family Bits per weight Estimated floor for 1.5B parameters Typical use
bf16 (master) 16 about 3.0 GB Archive copy, re-quantize from here
Q8_0 8.5 about 1.6 GB Near-lossless local serving
Q6_K 6.5625 about 1.2 GB High quality with a real size cut
Q5_K 5.5 about 1.0 GB Quality-first on small GPUs
Q4_K 4.5 Not listed The default for most laptops
Q3_K 3.4375 Not listed Tight memory, expect visible quality loss
Q2_K 2.625 about 0.49 GB Experiments only

These sizes are arithmetic floors (parameters times bits per weight, divided by 8). Real files run larger because embeddings and a few sensitive tensors stay at higher precision, and Q4_K_M deliberately mixes Q4_K and Q6_K blocks. Pick Q4_K_M unless you have a reason not to, then compare outputs against the bf16 file before you commit to anything lower.

Perplexity gives you a number to compare instead of a feeling. Save 50 to 100 lines of text from your fine-tuning domain as eval.txt and run the same file through both GGUFs:

for f in gguf/arxiv-titles-bf16.gguf gguf/arxiv-titles-Q4_K_M.gguf; do
  tools/llama.cpp/build/bin/llama-perplexity -m "$f" -f eval.txt -ngl 99 | tail -n 2
done

The tool ends each run with a final perplexity estimate. A small rise from bf16 to Q4_K_M is normal. A large jump means the quantization hurt your task, and the next step up (Q5_K_M or Q6_K) is cheap insurance. Use text the model has not seen during fine-tuning, or both numbers will look better than they deserve.

Run the GGUF Locally #

Step 10: Serve with llama-server and test over HTTP

The server speaks an OpenAI-compatible API, which makes it the easiest target for a repeatable test. Bind it to localhost and set an API key. The November hardening work on llama-server does not excuse an open port on a shared network.

export LLAMA_API_KEY="$(openssl rand -hex 16)"
tools/llama.cpp/build/bin/llama-server \
  -m gguf/arxiv-titles-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 -ngl 99 -c 4096 \
  --api-key "$LLAMA_API_KEY"

In a second terminal:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLAMA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Write a title for a paper on low-rank adapters for small language models."}],"temperature":0,"max_tokens":32}'
{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Low-Rank Adapters for Efficient Small Language Model Tuning"}}], ...}

With temperature 0, the Q4_K_M answer should land close to the bf16 reply from Step 6, though a few words may differ. Identical text is not the goal. The goal is that the model still shows the fine-tuned behavior, so a title-shaped answer is a pass and a rambling generic reply is a fail. The llama.cpp web interface at the same address also works for a quick manual check. For a fuller front end, the llama.cpp server and Open WebUI setup covers the next layer up. A quick command-line check works too:

tools/llama.cpp/build/bin/llama-cli -m gguf/arxiv-titles-Q4_K_M.gguf \
  -no-cnv -n 48 --temp 0 \
  -p "Write a title for a paper on low-rank adapters for small language models."

Flag names in llama-cli shift between builds, so check --help if your build behaves differently.

Step 11: Import the model into Ollama

Ollama imports GGUF files through a Modelfile. Its import documentation says it does not quantize during import, so the Q4_K_M file you made in Step 9 is exactly what gets served. Ollama 0.30, announced on June 5, 2026, improved GGUF compatibility through llama.cpp, so any current release handles standard architectures. Create Modelfile next to the GGUF:

FROM ./gguf/arxiv-titles-Q4_K_M.gguf
TEMPLATE """{{- range .Messages }}<|im_start|>{{ .Role }}
{{ .Content }}<|im_end|>
{{ end }}<|im_start|>assistant
"""
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.2
ollama create arxiv-titles -f Modelfile
ollama run arxiv-titles "Write a title for a paper on low-rank adapters for small language models."
gathering model components
copying file sha256:...
parsing GGUF
writing manifest
success
Low-Rank Adapters for Efficient Small Language Model Tuning

The TEMPLATE block above is the ChatML format that Qwen models use. It must match the format the adapter was trained with, or the model answers as if the fine-tuning never happened. For another model family, copy the template from that family’s official Ollama listing with ollama show --modelfile.

Option B: Ship the Adapter as Its Own GGUF #

Step 12: Convert the adapter alone and apply it at load time

Use this path when you serve several adapters on one base or want to swap behavior without re gigabytes of weights. It is only safe for the rows marked “Works” in the compatibility table. Run the preflight first and stop if it lists adapter-GGUF blockers.

source .venv-llama/bin/activate
python tools/llama.cpp/convert_lora_to_gguf.py adapters/demo \
  --base models/base --outfile gguf/arxiv-titles-lora-f16.gguf --outtype f16

python tools/llama.cpp/convert_hf_to_gguf.py models/base \
  --outfile gguf/base-bf16.gguf --outtype bf16
tools/llama.cpp/build/bin/llama-quantize gguf/base-bf16.gguf gguf/base-Q4_K_M.gguf Q4_K_M

tools/llama.cpp/build/bin/llama-server -m gguf/base-Q4_K_M.gguf \
  --lora gguf/arxiv-titles-lora-f16.gguf --host 127.0.0.1 --port 8080

The --base argument points at the original Hugging Face folder, because the converter needs its config to map tensor names. Use --lora-scaled instead of --lora when you want to dial the adapter’s strength up or down. The older merge-through-the- route is gone in v1.0.0, so a tutorial that passes a base file to the for merging is out of date.

Two cautions apply. First, the adapter was trained against full-precision or NF4 weights, and you are now applying it to a Q4_K_M base. The result is usually fine and sometimes visibly worse than the merged-then-quantized file, so run the same three prompts through both and compare. Second, llama.cpp applies the adapter separately at inference time and does not fold it into the base, which keeps memory-mapped of the base intact. That is the whole point of the route, and it explains why rsLoRA and DoRA adapters need extra care. For an rsLoRA adapter, compare logits against the merged model before you trust the converted file, since the converter’s scale handling was still being fixed in November.

Merge or Keep It Separate: A Decision Guide #

The merged GGUF and the adapter-only GGUF solve different problems, and picking by habit wastes either disk or speed. Five questions settle it quickly.

  • How many behaviors share one base model? One behavior means merge. Three or more tasks on the same base favor separate adapters, because each extra adapter costs a small file while each extra merged model costs a full copy of the base.
  • Who runs the file? A teammate importing into Ollama wants one self-contained GGUF and a Modelfile. A service you operate on llama.cpp can juggle adapters at load time.
  • Which adapter traits does the preflight report? Any adapter-GGUF blocker from Step 4 ends the debate in favor of merging.
  • How much does speed matter? Applying an adapter at inference adds work on every targeted layer. For latency-sensitive serving, the merged file removes that overhead.
  • How often does the adapter change? A weekly retrain argues for adapters, since you ship megabytes instead of gigabytes. A frozen release argues for a merge, a single checksum and a single artifact to support.

Default to the merge. It carries the fewest hidden assumptions, it works with every runtime that reads GGUF, and the smoke test proves it matches the adapter you trained. Move to adapter-only when you have a concrete reason, such as several tasks on one base or a retraining cadence that makes gigabyte uploads painful.

The base model choice interacts with all of this. A bigger base raises the merge’s RAM floor, raises the GGUF size and lowers how many copies fit on one disk, which pushes teams with many tasks toward adapters. A smaller base makes merged copies cheap enough to keep one per task. The best open-source LLM roundup helps you weigh base candidates, and the broader AI and machine learning coverage tracks new open-weight releases that could change the answer.

Whichever route you pick, record it in the manifest. A file named arxiv-titles-Q4_K_M.gguf that came from a merge and a file with the same name that came from base-plus-adapter behave differently under load, and six months from now nobody will remember which one is in production.

The Complete Project: One Script From Adapter to Ollama Model #

The steps above chain into a single entry point. ship_adapter.py imports preflight from Step 4 and merge from Step 5, runs the smoke test from Step 6 as a subprocess, then calls the llama.cpp tools and writes a Modelfile plus a manifest with checksums. The project layout after a run looks like this:

adapter-shipper/
  fetch.py  inspect_adapter.py  preflight.py  merge.py  smoke_test.py  ship_adapter.py
  models/base/            adapters/demo/         merged/arxiv-titles/
  gguf/arxiv-titles-bf16.gguf   gguf/arxiv-titles-Q4_K_M.gguf
  Modelfile   manifest.json   requirements.lock
  tools/llama.cpp/        .venv/   .venv-llama/

Save the orchestrator as ship_adapter.py.

#!/usr/bin/env python3
import argparse
import hashlib
import json
import subprocess
import sys
from pathlib import Path

import peft

from merge import merge
from preflight import preflight

MODELFILE = '''FROM ./__GGUF__
TEMPLATE """{{- range .Messages }}<|im_start|>{{ .Role }}
{{ .Content }}<|im_end|>
{{ end }}<|im_start|>assistant
"""
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.2
'''

def run(cmd):
    print("+", " ".join(str(c) for c in cmd), flush=True)
    subprocess.run([str(c) for c in cmd], check=True)

def sha256(path: Path) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", required=True, type=Path)
    ap.add_argument("--adapter", required=True, type=Path)
    ap.add_argument("--name", required=True)
    ap.add_argument("--llama-cpp", default=Path("tools/llama.cpp"), type=Path)
    ap.add_argument("--llama-python", default=".venv-llama/bin/python")
    ap.add_argument("--quant", default="Q4_K_M")
    ap.add_argument("--skip-smoke", action="store_true")
    args = ap.parse_args()

    blockers, adapter_notes = preflight(args.adapter)
    if blockers:
        sys.exit(f"preflight failed: {blockers}")
    for note in adapter_notes:
        print("note, adapter-only GGUF is unsafe:", note)

    merged = Path("merged") / args.name
    Path("gguf").mkdir(exist_ok=True)
    merge(args.base, args.adapter, merged)

    if not args.skip_smoke:
        run([sys.executable, "smoke_test.py", args.base, args.adapter, merged])

    bf16 = Path("gguf") / f"{args.name}-bf16.gguf"
    quant = Path("gguf") / f"{args.name}-{args.quant}.gguf"
    run([args.llama_python, args.llama_cpp / "convert_hf_to_gguf.py",
         merged, "--outfile", bf16, "--outtype", "bf16"])
    run([args.llama_cpp / "build" / "bin" / "llama-quantize", bf16, quant, args.quant])

    Path("Modelfile").write_text(MODELFILE.replace("__GGUF__", str(quant)))

    cfg = json.loads((args.adapter / "adapter_config.json").read_text())
    manifest = {
        "name": args.name,
        "base_model": cfg.get("base_model_name_or_path"),
        "peft_version": peft.__version__,
        "llama_cpp_tag": "v1.0.0",
        "quant": args.quant,
        "files": {p.name: {"bytes": p.stat().st_size, "sha256": sha256(p)} for p in (bf16, quant)},
    }
    Path("manifest.json").write_text(json.dumps(manifest, indent=2))
    print(json.dumps(manifest, indent=2))
    print(f"next: ollama create {args.name} -f Modelfile")

if __name__ == "__main__":
    main()
source .venv/bin/activate
python ship_adapter.py --base models/base --adapter adapters/demo --name arxiv-titles
ollama create arxiv-titles -f Modelfile

Example final output (hashes and sizes are placeholders, yours will differ):

{
  "name": "arxiv-titles",
  "base_model": "Qwen/Qwen2.5-1.5B-Instruct",
  "peft_version": "...",
  "llama_cpp_tag": "v1.0.0",
  "quant": "Q4_K_M",
  "files": {
    "arxiv-titles-bf16.gguf": {"bytes": 3087000000, "sha256": "..."},
    "arxiv-titles-Q4_K_M.gguf": {"bytes": 986000000, "sha256": "..."}
  }
}
next: ollama create arxiv-titles -f Modelfile

The manifest is the part people skip and later regret. When a colleague reports that the model “got worse” in January, the manifest tells you which base, which PEFT release and which llama.cpp tag produced the file, and the checksum proves they are testing the same artifact you shipped.

The script fails loudly at the first bad hop. A failed preflight exits before any weights load, a failed smoke test stops before any GGUF exists, and the converter’s non-zero exit code halts the quantize step. That ordering is deliberate, because the expensive steps come last.

Common Pitfalls #

  • Merging into a 4-bit base. QLoRA trains against NF4 weights, so the temptation is to reload the same quantized model and merge. Load the original 16-bit weights instead. Merging into quantized weights either errors out or adds rounding noise that Q4_K_M will then compound.
  • Pairing the adapter with the wrong base revision. An adapter trained on the Instruct checkpoint and merged into the base checkpoint, or into a base that was re-released after training, loads without complaint and answers worse. Comparebase_model_name_or_path against what you downloaded every time.
  • Dropping the tokenizer. The converter copies the vocabulary and chat template into the GGUF header. Save the tokenizer from the adapter folder when it exists, since fine-tunes sometimes add special tokens.
  • Building llama.cpp from master. Numbered pre-release builds ship daily, and the 1.0 cleanup removed deprecated APIs. Pin the v1.0.0 tag so a conversion that worked today still works in March.
  • Merging in fp16. bf16 has the same exponent range as float32, while fp16 has a narrower one, and some model families overflow in fp16. Use bf16, or float32 when you want the highest fidelity.
  • Quantizing an already quantized file.llama-quantize refuses to requantize unless you pass--allow-requantize , and for good reason. Always quantize from the bf16 master.
  • Mismatched chat templates in Ollama. If the Modelfile TEMPLATE differs from the format used during fine-tuning, the model behaves like the untuned base. The GGUF is fine and the prompt is wrong.
  • Using the DoRA-to-LoRA trick. Converting a DoRA adapter to plain LoRA to unlock adapter-only GGUF is attractive, and a reported November 16 issue says the converted result can differ from the trained model. Merge instead.
  • Ignoring the license. A merged model is a derivative of its base and carries the base model’s terms. Check theopen-weight license comparison before you publish a merged checkpoint or a GGUF.
  • Sharing one Python environment. The llama.cpp requirements file can pull a different PyTorch build than your merge environment needs. Two virtual environments cost 10 seconds of setup and save an afternoon.

Troubleshooting #

  • “Can’t find adapter_config.json” when the adapter. You pointedPeftModel.from_pretrained at the parent folder or at a partial download. Pass the directory that directly containsadapter_config.json , and rerunfetch.py if files are missing.
  • Size mismatch for embed_tokens or lm_head while . The adapter was trained after tokens were added. Load the adapter’s tokenizer, grow the embedding matrix before attaching the adapter (asmerge.py does), and merge instead of converting the adapter alone.
  • Out of memory during the merge. Load withlow_cpu_mem_usage=True and bf16, close other applications, and keep the merge on CPU. For a 27B model, plan for roughly 54 GB of RAM, or merge on a larger machine and copy the result back.
  • The converter reports an unsupported architecture. Either you pointed it at the adapter folder instead of the merged folder, or the model family is newer than your checkout. Check thatconfig.json lists an architecture, then try a newer llama.cpp tag.
  • git checkout v1.0.0 reports an unknown ref. Your clone did not fetch tags. Rungit fetch --tags , or list tags withgit tag --list "v1*" .
  • CMake cannot find CUDA. Install a CUDA toolkit that matches your driver, or rebuild without-DGGML_CUDA=ON . A CPU build is slower to run but converts and quantizes identically.
  • The Q4_K_M model is gibberish but the bf16 file is fine. Quantization hit a sensitive tensor. Try Q5_K_M, Q6_K or Q8_0, and read the importance-matrix tip in the next section.
  • Ollama creates the model but ignores the fine-tuning. Check that the Modelfile points at the merged GGUF and not the base file, then diff the TEMPLATE against the format you trained with.
  • Ollama fails with unknown renderer "qwen3.8". The runtime is older than the model family. One community listing says Ollama 0.32.12 or newer is required, so upgrade Ollama before you suspect the GGUF.
  • llama-server --lora cannot load the adapter or reports missing tensors. The base GGUF architecture does not match the adapter, or the adapter carries features the converter cannot express, such as DoRA magnitude vectors ormodules_to_save . Merge instead.
  • The smoke test gap is huge (far above 1.0). Suspect the wrong base or a missing tokenizer first. Rerun Step 3 and confirmbase_model_name_or_path matches the folder you passed tomerge.py .
  • An old tutorial’s --lora-base flag errors out. That merge path was removed from the current documentation. Merge with PEFT as in Step 5.

Advanced Tips #

Use an importance matrix for low-bit quantization. The llama.cpp documentation describes quantization types that rely on an importance matrix, which measures which weights matter on real text. Generate one from domain text that resembles your fine-tuning data, then quantize with it:

tools/llama.cpp/build/bin/llama-imatrix -m gguf/arxiv-titles-bf16.gguf \
  -f calibration.txt -o gguf/imatrix.dat
tools/llama.cpp/build/bin/llama-quantize --imatrix gguf/imatrix.dat \
  gguf/arxiv-titles-bf16.gguf gguf/arxiv-titles-Q3_K_M.gguf Q3_K_M

Calibration text from your own task is the whole trick. A matrix built from generic web text protects the wrong weights.

Combine two adapters before merging. PEFT can blend adapters that share a base. The linear combination type needs matching ranks, so check r in both configs first.

from peft import PeftModel
model = PeftModel.from_pretrained(base_model, "adapters/style", adapter_name="style")
model.load_adapter("adapters/domain", adapter_name="domain")
model.add_weighted_adapter(["style", "domain"], [0.6, 0.4],
                           adapter_name="blend", combination_type="linear")
model.set_adapter("blend")
merged = model.merge_and_unload()

Re-run the smoke test against the blend with prompts from both tasks. Blends often keep one skill and dilute the other, and the weights need tuning.

Hot-swap adapters on one base. With Option B, llama-server accepts several --lora arguments, and recent builds can load them without applying them at startup and expose an endpoint for changing scales per request. Flag names vary by build, so confirm with llama-server --help before you design around it. Compare the cost against merging: one base plus three small adapters beats three full merged models on disk, and loses on raw speed.

Skip the manual steps only when you do not need to debug. Several training tools, Unsloth among them, can export GGUF directly after training. That shortcut removes the intermediate merged checkpoint, which is the artifact you want when a quantized model misbehaves. Hugging Face also publishes a convert_to_gguf.py helper in its skills repository, which is a useful reference for a scripted pipeline. A community repository, reportedly, documents the same pattern for a Granite 4.0 H 1B math LoRA: merge with merge_and_unload, convert to F16 GGUF, then quantize into several files. The recipe is standard.

Keep golden prompts in version control. Store 20 prompts and their bf16 answers, run them through every new quantization, and fail the build when the answers drift. The smoke test in Step 6 becomes a regression suite for free. When you move to a newer base model, the open-weight spec sheet helps you check parameter counts and context windows before you retrain.

FAQ #

Can I merge a LoRA adapter into a quantized model?

Avoid it. Merge into the original 16-bit weights, then quantize the result to GGUF. Merging into a 4-bit base adds rounding error that the later quantization step compounds.

Do I have to merge before creating a GGUF?

No. You can convert the adapter to its own GGUF and apply it with --lora. That path only fits plain LoRA and QLoRA-trained adapters. Adapters with DoRA vectors, modified embeddings or added tokens should be merged.

Which llama.cpp version should I use for conversion?

Use the v1.0.0 tag, the first stable release, published on November 13, 2026. Daily pre-release builds exist, but a pinned tag keeps your conversions reproducible.

Which quantization should I choose for a merged adapter?

Start with Q4_K_M for laptops and Q8_0 when memory allows. Run your golden prompts against the bf16 file and the quantized file, and only drop to Q3 or lower with an importance matrix built from your own domain text.

Does merging change the model size?

No. The merged checkpoint has the same parameter count and file size as the base model, whatever rank the adapter used. The adapter’s rank only affects how much the adapter could learn during training.

How do I confirm the merged model still behaves like the adapter?

Compare last-token logits between the base with the adapter attached and the merged checkpoint, as the smoke test does. Large gaps point to the wrong base, a missing tokenizer or a skipped embedding resize.

Can I run the result in Ollama and llama.cpp both?

Yes. Both read the same GGUF file. Ollama needs a Modelfile with a TEMPLATE that matches the training format, while llama.cpp reads the chat template straight from the GGUF header.

── more in #ai-tools 4 stories · sorted by recency
── more on @llama.cpp 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/merge-a-lora-adapter…] indexed:0 read:32min 2026-11-18 · —