{"slug": "merge-a-lora-adapter-into-gguf-12-steps-80-min", "title": "Merge a LoRA Adapter Into GGUF: 12 Steps, 80 Min", "summary": "Llama.cpp v1.0.0, released November 13, 2026 as the project's first stable version, requires Hugging Face PEFT LoRA adapters to be converted to GGUF before llama.cpp or Ollama can run them locally, and a walkthrough script called ship_adapter.py takes a public QLoRA adapter for Qwen2.5-1.5B-Instruct through preflight checks, a merge_and_unload merge, a logit comparison against the unmerged adapter, a bf16 GGUF, a Q4_K_M GGUF and an Ollama model in about 80 minutes on a 16 GB laptop. The guide says most failures come from merging into the wrong precision, dropping the tokenizer, shipping a DoRA adapter through a path that cannot carry it, or running a converter from a random master branch, each producing a model that loads fine and answers badly. It recommends merging into the original 16-bit base weights rather than a 4-bit NF4 base, since quantizing twice stacks error on error.", "body_md": "The [llama.cpp v1.0.0 release](https://newreleases.io/project/github/ggml-org/llama.cpp/release/v1.0.0) landed on November 13, 2026 as the project’s first stable version, and it tightened a rule that trips up many fine-tuners: a LoRA adapter trained in Hugging Face PEFT cannot run on local hardware until it becomes a GGUF file that llama.cpp or Ollama can read. The adapter alone is just a set of small low-rank matrices. It needs its exact base model, the right tokenizer, and a conversion path that respects how it was trained.\n\nMost failures happen at that handoff. People merge into the wrong precision, forget the tokenizer, ship a DoRA adapter through a path that cannot carry it, or run a converter from a random master branch. Each mistake produces a model that loads fine and answers badly, which is the worst kind of bug because nothing crashes.\n\nOne Python script, `ship_adapter.py`, takes a public QLoRA adapter for Qwen2.5-1.5B-Instruct through every hop: preflight checks, a `merge_and_unload` merge, a logit comparison against the unmerged adapter, a bf16 GGUF, a Q4_K_M GGUF, and an Ollama model. The same script works on adapters you trained yourself, including the ones from a [QLoRA fine-tune of an open model](https://shattered.io/fine-tune-llm-qlora-12-steps-2026/). Budget about 80 minutes on a laptop with 16 GB of RAM.\n\nDon't miss new tech stories on Google\n\nAdd Shattered once in the Google app and our stories show up in your news suggestions.\n\n[Add now](https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F)\n\n## What a LoRA Adapter Is and Why You Merge It\n\nLoRA freezes the original weight matrix W and learns two thin matrices, A and B, whose product forms a correction. At inference the effective weight is W plus (alpha divided by r) times B times A, where r is the rank and alpha is a scaling constant. A LoRA adapter file stores only A and B for each targeted module, plus an `adapter_config.json` that names the base model it was trained against. That is why adapters are small and why they are useless without the right base.\n\nYou have two ways to ship one. The first is to merge: add the correction into W once, save a normal checkpoint, and forget the adapter existed. The merged model is the same size as the base, runs at base-model speed, and needs no adapter-aware runtime. The second is to keep the adapter separate: convert it to its own GGUF and let llama.cpp apply it at load time with the `--lora` flag. That costs a little speed but lets you swap behaviors on one base model.\n\nRank matters during training and disappears after a merge. If you ran a [LoRA rank sweep](https://shattered.io/lora-fine-tuning-rank-sweep-12-steps-2026/) and picked rank 16 over rank 64, the merged checkpoint is exactly the same size either way. Rank changes how much the adapter can learn, not how big the finished model is.\n\nQLoRA adds one wrinkle. The adapter was trained against a base model quantized to 4-bit NF4, but the correction itself is stored in 16-bit precision. When you merge, load the original 16-bit base weights and merge into those. Merging into a 4-bit base either fails or bakes quantization noise into the new weights, and you will quantize again later for GGUF anyway. Quantizing twice stacks error on error.\n\nMerge-first is the default path here for a practical reason. It works for every adapter type PEFT can merge, including DoRA and adapters that carry modified embeddings. Adapter-only GGUF conversion is narrower, so it appears later as Option B, with a compatibility table that tells you when it is safe.\n\n## What llama.cpp v1.0.0 Changed for Adapter Workflows\n\nThe November 13 release put llama.cpp and ggml on semantic versioning, with `libllama` and `libggml` as the two published libraries. A [wire report on the release](https://zmedia.co/news/wire/cmwf2de18004mxkumresrfgjm-llamacpp-v100-a-stronger-foundation-for-the-local-ai-ecosyst/) describes the new promise: breaking API or ABI changes now require a major version bump. For people who only run the command-line tools, the headline matters less than the cleanup that came with it.\n\nThe 1.0 transition removed deprecated public interfaces, which can break downstream integrations that linked against old symbols. One [breakdown of what breaks in llama.cpp 1.0](https://buttondown.com/insiderllm/archive/llamacpp-hits-10-heres-what-breaks/) walks through the affected areas. For adapter work, the relevant removal is the old `--lora-base` merge path. Current documentation, dated November 16, no longer offers it, so any older tutorial that tells you to merge by passing a base file to the loader will fail on a v1.0.0 build.\n\nTwo earlier changes shape the workflow too. The v0.8.0 release on October 28 strengthened GGUF validation for the `split`, `export-lora` and `merge` operations, so a malformed file now fails fast instead of loading half-broken. And the project keeps shipping numbered pre-release builds between tags. Build `b12503`, marked a pre-release, was the newest on November 18. Treat those as a moving target and pin the v1.0.0 tag.\n\nThe server changed as well. On November 13, `llama-server` gained hardening against SSRF, CORS abuse and DNS rebinding, added a `/v1/audio/speech` endpoint, and shipped a redesigned web interface with chat groups and a Markdown editor. Coverage of the release says it did not change the GGUF file format itself, which means GGUF files you converted last month still load.\n\nThe conversion scripts remain under active maintenance. Issue threads from November 8 to 11 reportedly cover how `convert_lora_to_gguf.py` handles rsLoRA scaling, including a proposed `--fold-scale` option. Since that work was still open at the time of writing, the safest assumption is that adapter-only conversion of rsLoRA adapters needs a manual check, which Step 12 covers.\n\nThe takeaway: pin `v1.0.0`, avoid tutorials written before November 2026, and keep the merge path as your baseline.\n\n## Prerequisites and Tested Versions\n\nThe merge itself is light. A 1.5B-parameter model in bf16 needs roughly 3 GB of RAM, so any modern laptop handles it, with or without a GPU. The heavy parts are disk space and the llama.cpp build. Scale matters if you bring your own adapter: a 27B model such as [Qwen3.8-27B](https://shattered.io/run-llm-locally-qwen3-8-27b-llama-cpp-12-steps-2026/) needs about 54 GB of RAM for a bf16 merge (27 billion parameters at 2 bytes each), so check the arithmetic against your hardware before you start. If you are still choosing where the finished model will live, the [three ways to run open-source models](https://shattered.io/open-source-ai-models-3-run-paths-mistral-large-4-kimi-k3-1-glm-5-3-2026/) frame the choice between local, rented and hosted hardware.\n\n| Component | Version or requirement | Why it is needed | \n|---|---|---|\n| Python | 3.12 | Runs PEFT, Transformers and the llama.cpp converters | \n| llama.cpp | v1.0.0 tag, built from source | GGUF conversion, quantization and serving | \n| Hugging Face PEFT | Current release from PyPI | Loads the adapter and runs `merge_and_unload` | \n| Transformers and PyTorch | Current releases from PyPI | Loads the base model and runs the smoke test | \n| CMake and a C++ compiler | CMake 3.28 or newer, GCC 13 or Clang 17 | Builds llama.cpp | \n| Ollama | A current release | Imports the quantized GGUF as a named model | \n| Free disk space | 15 GB | Base, merged copy, bf16 GGUF, Q4_K_M GGUF and the build tree | \n| RAM | 16 GB (more for larger models) | Holds base and adapter copies during the merge | \n\nRun `pip freeze` after Step 1 and keep the output. PEFT and Transformers move quickly, and an adapter that merged cleanly in one release can behave differently in another. A saved lockfile turns “it worked last week” into something you can reproduce.\n\nThe time budget breaks down like this: 10 minutes for setup, 15 for downloads, 10 for the merge and smoke test, 20 for the llama.cpp build, and 25 for conversion, quantization, serving and the Ollama import. A GPU shortens the build and inference steps but is not required for any step.\n\nOn model choice: the worked example uses Qwen2.5-1.5B-Instruct because a public QLoRA adapter for it exists, it merges in minutes, and a bad result is obvious within three prompts. It is an older Qwen generation. Newer families work with the same steps, but they raise the runtime requirements. One community listing for a Qwen 3.8 model notes that older Ollama builds fail at prompt rendering with an `unknown renderer \"qwen3.8\"` error and need Ollama 0.32.12 or newer. Match your runtime to your model family before you blame your merge.\n\n## Prepare the Workspace\n\n### Step 1: Create the project and the Python environment\n\nKeep the merge environment separate from the llama.cpp converter environment. The converter’s requirements file can pin its own PyTorch build, and mixing the two is a classic way to wreck a working setup.\n\n```\nmkdir adapter-shipper && cd adapter-shipper\nmkdir -p models adapters merged gguf tools\n\npython3.12 -m venv .venv\nsource .venv/bin/activate\npip install --upgrade pip\npip install torch transformers peft accelerate safetensors sentencepiece huggingface_hub\npip freeze > requirements.lock\n\npython -c \"import peft, transformers, torch; print(peft.__version__, transformers.__version__, torch.__version__)\"\n```\n\nThe last command prints three version strings. Write them into your notes. If the import fails, the install did not finish, and every later step will fail with confusing errors.\n\n### Step 2: Download the base model and the adapter\n\nUse `snapshot_download` from Python rather than a command-line tool, because the CLI name has changed across `huggingface_hub` releases and a script keeps working across all of them. Save this as `fetch.py`.\n\n``` python\nimport sys\nfrom huggingface_hub import snapshot_download\n\nbase_id, adapter_id = sys.argv[1], sys.argv[2]\n\nbase_path = snapshot_download(base_id, local_dir=\"models/base\")\nadapter_path = snapshot_download(adapter_id, local_dir=\"adapters/demo\")\nprint(\"base   ->\", base_path)\nprint(\"adapter->\", adapter_path)\npython fetch.py Qwen/Qwen2.5-1.5B-Instruct ms-afridi/qwen2.5-1.5b-arxiv-title-lora\nls adapters/demo\n```\n\nThe adapter repository is a public QLoRA adapter that was trained with 4-bit NF4 quantization and loads through `PeftModel.from_pretrained`. Expected listing:\n\n```\nREADME.md\nadapter_config.json\nadapter_model.safetensors\ntokenizer.json\ntokenizer_config.json\n```\n\nYour listing may differ slightly, and that is fine. The two files that matter are `adapter_config.json` and `adapter_model.safetensors`. If you only see a `.bin` weights file, the adapter predates safetensors defaults, and the preflight script in Step 4 needs a one-line change to read it.\n\n## Inspect and Preflight the Adapter\n\n### Step 3: Read adapter_config.json before touching any weights\n\nFive fields in the config decide which shipping path is safe: `peft_type`, `base_model_name_or_path`, `use_dora`, `use_rslora` and `modules_to_save`. Print them first. Save this as `inspect_adapter.py`.\n\n``` python\nimport json\nimport sys\nfrom pathlib import Path\n\nadapter_dir = Path(sys.argv[1])\ncfg = json.loads((adapter_dir / \"adapter_config.json\").read_text())\n\nfor key in [\"peft_type\", \"base_model_name_or_path\", \"r\", \"lora_alpha\",\n            \"lora_dropout\", \"target_modules\", \"use_dora\", \"use_rslora\",\n            \"modules_to_save\", \"bias\", \"task_type\"]:\n    print(f\"{key:26} {cfg.get(key)}\")\n\nr, alpha = cfg[\"r\"], cfg[\"lora_alpha\"]\nscale = alpha / (r ** 0.5 if cfg.get(\"use_rslora\") else r)\nprint(f\"{'effective scale':26} {scale:.3f}\")\nprint(\"files:\", \", \".join(sorted(p.name for p in adapter_dir.iterdir())))\npython inspect_adapter.py adapters/demo\n```\n\nExample output (your adapter’s values will differ):\n\n```\npeft_type                  LORA\nbase_model_name_or_path    Qwen/Qwen2.5-1.5B-Instruct\nr                          16\nlora_alpha                 32\nlora_dropout               0.05\ntarget_modules             ['q_proj', 'k_proj', 'v_proj', 'o_proj']\nuse_dora                   False\nuse_rslora                 False\nmodules_to_save            None\nbias                       none\ntask_type                  CAUSAL_LM\neffective scale            2.000\n```\n\nCheck that `base_model_name_or_path` matches the base you downloaded. An adapter trained on the Instruct variant and merged into the base variant produces text that looks plausible and reads wrong. The effective scale is alpha divided by r for classic LoRA and alpha divided by the square root of r for rsLoRA. Keep that number in mind for the rsLoRA check in Step 12.\n\n### Step 4: Run a preflight check for unsupported adapter features\n\nThe preflight script returns two lists. Merge blockers stop everything. Adapter-GGUF blockers only rule out Option B, because the merge path handles those features fine. Save this as `preflight.py`.\n\n``` python\nimport json\nimport sys\nfrom pathlib import Path\n\nfrom safetensors import safe_open\n\ndef preflight(adapter_dir: Path):\n    cfg = json.loads((adapter_dir / \"adapter_config.json\").read_text())\n    merge_blockers, adapter_gguf_blockers = [], []\n\n    if cfg.get(\"peft_type\") != \"LORA\":\n        merge_blockers.append(f\"peft_type is {cfg.get('peft_type')}, expected LORA\")\n\n    weights = adapter_dir / \"adapter_model.safetensors\"\n    if not weights.exists():\n        merge_blockers.append(\"adapter_model.safetensors is missing\")\n        return merge_blockers, adapter_gguf_blockers\n\n    with safe_open(str(weights), framework=\"pt\") as f:\n        names = list(f.keys())\n\n    if cfg.get(\"use_dora\") or any(\"lora_magnitude_vector\" in n for n in names):\n        adapter_gguf_blockers.append(\"DoRA magnitude vectors: merge instead of converting the adapter\")\n    if cfg.get(\"modules_to_save\"):\n        adapter_gguf_blockers.append(f\"modules_to_save={cfg['modules_to_save']}: full tensors live outside the low-rank pair\")\n    if any(\"lora_embedding\" in n for n in names):\n        adapter_gguf_blockers.append(\"embedding LoRA tensors present\")\n    if cfg.get(\"bias\", \"none\") != \"none\":\n        adapter_gguf_blockers.append(f\"bias={cfg['bias']}: bias deltas are not part of a plain LoRA pair\")\n    if cfg.get(\"use_rslora\"):\n        adapter_gguf_blockers.append(\"rsLoRA: verify the converter's scale handling before trusting it\")\n\n    return merge_blockers, adapter_gguf_blockers\n\nif __name__ == \"__main__\":\n    mb, ab = preflight(Path(sys.argv[1]))\n    print(\"merge blockers      :\", mb or \"none\")\n    print(\"adapter-gguf blockers:\", ab or \"none\")\n    sys.exit(1 if mb else 0)\npython preflight.py adapters/demo\nmerge blockers      : none\nadapter-gguf blockers: none\n```\n\nA third-party checker called lora-preflight reportedly does a similar job, flagging adapters with tensors that llama.cpp and Ollama cannot load and recommending `merge_and_unload` for them. The script above covers the same ground in 40 lines and has no extra dependency. The compatibility table below summarizes what the checks mean.\n\n| Adapter trait | Merge, then GGUF | Adapter-only GGUF | What to do | \n|---|---|---|---|\n| Plain LoRA | Works | Works | Either path, pick by deployment needs | \n| QLoRA-trained LoRA | Works | Works | Merge into the 16-bit base, never the 4-bit one | \n| rsLoRA | Works | Verify the scale | Merge, or compare logits after conversion | \n| DoRA | Works | Not reliable | Merge, and do not convert DoRA to plain LoRA to dodge the problem | \n| `modules_to_save` (embeddings, lm_head) | Works | Not supported | Merge only | \n| Added tokens | Works after an embedding resize | Not supported | Merge only, keep the adapter’s tokenizer | \n\nThe DoRA row deserves a warning. An issue reported on November 16, 2026 says PEFT’s `convert_to_lora` and `save_as_lora` helpers can produce a model that differs from the trained DoRA model. Converting DoRA to LoRA looks like a clever way to unlock the adapter-only path, but until that issue closes, merging is the only route you can verify.\n\n## Merge the LoRA Adapter Into the Base Model\n\n### Step 5: Merge with merge_and_unload\n\nThe merge script loads the 16-bit base, grows the embedding matrix only if the adapter added tokens, applies the adapter, folds it in, and saves the model together with the right tokenizer. Save this as `merge.py`.\n\n``` python\nimport sys\nfrom pathlib import Path\n\nimport torch\nfrom peft import PeftModel\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ndef merge(base_dir: Path, adapter_dir: Path, out_dir: Path, dtype=torch.bfloat16):\n    tok_src = adapter_dir if (adapter_dir / \"tokenizer_config.json\").exists() else base_dir\n    tokenizer = AutoTokenizer.from_pretrained(tok_src)\n\n    model = AutoModelForCausalLM.from_pretrained(base_dir, dtype=dtype, low_cpu_mem_usage=True)\n\n    rows = model.get_input_embeddings().num_embeddings\n    if len(tokenizer) > rows:\n        model.resize_token_embeddings(len(tokenizer))\n\n    model = PeftModel.from_pretrained(model, adapter_dir)\n    model = model.merge_and_unload()\n\n    out_dir.mkdir(parents=True, exist_ok=True)\n    model.save_pretrained(out_dir, safe_serialization=True)\n    tokenizer.save_pretrained(out_dir)\n    return out_dir\n\nif __name__ == \"__main__\":\n    base, adapter, out = (Path(p) for p in sys.argv[1:4])\n    dtype = torch.float32 if len(sys.argv) > 4 and sys.argv[4] == \"float32\" else torch.bfloat16\n    print(\"merged ->\", merge(base, adapter, out, dtype))\npython merge.py models/base adapters/demo merged/arxiv-titles\nls -lh merged/arxiv-titles\n```\n\nThe guard on the embedding resize uses a greater-than check on purpose. Many model families pad the embedding matrix beyond the tokenizer’s vocabulary for hardware alignment, so a naive “sizes differ, so resize” rule would shrink the matrix and corrupt it. Only grow it when the adapter’s tokenizer is genuinely larger.\n\nExample listing:\n\n```\n-rw-r--r-- 1 you you  707 config.json\n-rw-r--r-- 1 you you  242 generation_config.json\n-rw-r--r-- 1 you you 3.1G model.safetensors\n-rw-r--r-- 1 you you  11M tokenizer.json\n-rw-r--r-- 1 you you 7.2K tokenizer_config.json\n```\n\nThe model file should match the base model’s size. If it comes out noticeably larger, something was saved in float32 by accident. If it comes out smaller, the merge produced a partial checkpoint and you should rerun it.\n\n### Step 6: Smoke-test the merged model against the unmerged adapter\n\nA merge that finishes without errors proves nothing. The smoke test loads the base with the adapter attached on the fly, loads the merged checkpoint, and compares the last-token logits for a few prompts. Save this as `smoke_test.py`.\n\n``` python\nimport sys\nfrom pathlib import Path\n\nimport torch\nfrom peft import PeftModel\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nbase, adapter, merged = (Path(p) for p in sys.argv[1:4])\nTHRESHOLD = 1.0\n\ntok = AutoTokenizer.from_pretrained(merged)\nref = AutoModelForCausalLM.from_pretrained(base, dtype=torch.bfloat16)\nref = PeftModel.from_pretrained(ref, adapter).eval()\nmrg = AutoModelForCausalLM.from_pretrained(merged, dtype=torch.bfloat16).eval()\n\nprompts = [\n    \"Write a title for a paper on low-rank adapters for small language models.\",\n    \"Suggest a title for a study of quantization error in 4-bit language models.\",\n    \"Give a title for a paper about merging fine-tuned adapters into base weights.\",\n]\n\nworst = 0.0\nfor p in prompts:\n    text = tok.apply_chat_template([{\"role\": \"user\", \"content\": p}],\n                                   tokenize=False, add_generation_prompt=True)\n    ids = tok(text, return_tensors=\"pt\")\n    with torch.no_grad():\n        a = ref(**ids).logits[0, -1].float()\n        b = mrg(**ids).logits[0, -1].float()\n        gen = mrg.generate(**ids, max_new_tokens=32, do_sample=False)\n    gap = (a - b).abs().max().item()\n    worst = max(worst, gap)\n    reply = tok.decode(gen[0][ids[\"input_ids\"].shape[1]:], skip_special_tokens=True)\n    print(f\"gap={gap:.4f}  reply={reply.strip()!r}\")\n\nprint(f\"worst gap: {worst:.4f} (threshold {THRESHOLD})\")\nsys.exit(0 if worst < THRESHOLD else 1)\npython smoke_test.py models/base adapters/demo merged/arxiv-titles\n```\n\nExample output (numbers and wording will differ):\n\n```\ngap=0.1250  reply='Low-Rank Adapters for Efficient Small Language Model Tuning'\ngap=0.0938  reply='Measuring Quantization Error in 4-Bit Language Models'\ngap=0.1562  reply='Folding Fine-Tuned Adapters Into Base Weights: A Practical Study'\nworst gap: 0.1562 (threshold 1.0)\n```\n\nThe threshold of 1.0 is a heuristic, not a standard. Logits in bf16 carry rounding noise, so identical math through two code paths rarely matches to the last digit. A broken merge, such as the wrong base or a missed tokenizer, usually produces much larger gaps. If your gaps sit near 1.0, rerun the merge in float32 by adding `float32` as the fourth argument to `merge.py`.\n\n## Convert the Merged Model to GGUF\n\n### Step 7: Build llama.cpp at the v1.0.0 tag\n\nClone into `tools/`, check out the tag, and build. Use a second virtual environment for the converter requirements.\n\n```\ngit clone https://github.com/ggml-org/llama.cpp tools/llama.cpp\ncd tools/llama.cpp\ngit checkout v1.0.0\n\ncmake -B build -DGGML_CUDA=ON\ncmake --build build --config Release -j\n\npython3.12 -m venv ../../.venv-llama\nsource ../../.venv-llama/bin/activate\npip install -r requirements.txt\ncd ../..\n```\n\nDrop `-DGGML_CUDA=ON` for a CPU-only build. On Apple silicon, the Metal backend turns on by default. If `git checkout v1.0.0` reports an unknown ref, run `git tag --list \"v1*\"` and use the tag it shows. The build takes about 20 minutes on a laptop without ccache.\n\n```\ntools/llama.cpp/build/bin/llama-quantize --help | head -n 5\n```\n\nIf that prints usage text, the binaries exist. A “no such file” error means the build failed earlier, and the CMake output has the reason.\n\n### Step 8: Convert the merged checkpoint to a bf16 GGUF\n\nConvert to bf16 first and quantize afterward. This keeps one lossless-ish master copy you can re-quantize at will.\n\n```\nsource .venv-llama/bin/activate\npython tools/llama.cpp/convert_hf_to_gguf.py merged/arxiv-titles \\\n  --outfile gguf/arxiv-titles-bf16.gguf --outtype bf16\n```\n\nExample output (abridged):\n\n```\nINFO:hf-to-gguf:Loading model: arxiv-titles\nINFO:hf-to-gguf:Model architecture: Qwen2ForCausalLM\nINFO:hf-to-gguf:gguf: loading model part 'model.safetensors'\nINFO:hf-to-gguf:token_embd.weight, torch.bfloat16 --> BF16, shape = {1536, 151936}\n...\nINFO:hf-to-gguf:Model successfully exported to gguf/arxiv-titles-bf16.gguf\n```\n\nThe converter reads the tokenizer and chat template from the merged folder and bakes them into the GGUF header. That is why Step 5 saved the tokenizer next to the weights. A merged folder without tokenizer files converts to a file that either errors or guesses the vocabulary.\n\nThe converter picks its code path from the architecture string in `config.json`, which is why one command handles families as different as Qwen, DeepSeek and Mistral, the three open-weight lines compared in [DeepSeek vs Qwen vs Mistral](https://shattered.io/deepseek-vs-qwen-vs-mistral-open-weight-models-2026/). A brand-new family can still outrun the converter, so a model released last week may need a newer llama.cpp build than a model released last year.\n\n### Step 9: Quantize to Q4_K_M\n\n```\ntools/llama.cpp/build/bin/llama-quantize \\\n  gguf/arxiv-titles-bf16.gguf gguf/arxiv-titles-Q4_K_M.gguf Q4_K_M\n```\n\nQuantization packs weights into small blocks that share a scale. The llama.cpp documentation lists the bits per weight for each k-quant family, and from those you can estimate file sizes for a 1.5B-parameter model.\n\n| Quant family | Bits per weight | Estimated floor for 1.5B parameters | Typical use | \n|---|---|---|---|\n| bf16 (master) | 16 | about 3.0 GB | Archive copy, re-quantize from here | \n| Q8_0 | 8.5 | about 1.6 GB | Near-lossless local serving | \n| Q6_K | 6.5625 | about 1.2 GB | High quality with a real size cut | \n| Q5_K | 5.5 | about 1.0 GB | Quality-first on small GPUs | \n| Q4_K | 4.5 | Not listed | The default for most laptops | \n| Q3_K | 3.4375 | Not listed | Tight memory, expect visible quality loss | \n| Q2_K | 2.625 | about 0.49 GB | Experiments only | \n\nThese sizes are arithmetic floors (parameters times bits per weight, divided by 8). Real files run larger because embeddings and a few sensitive tensors stay at higher precision, and Q4_K_M deliberately mixes Q4_K and Q6_K blocks. Pick Q4_K_M unless you have a reason not to, then compare outputs against the bf16 file before you commit to anything lower.\n\nPerplexity gives you a number to compare instead of a feeling. Save 50 to 100 lines of text from your fine-tuning domain as `eval.txt` and run the same file through both GGUFs:\n\n```\nfor f in gguf/arxiv-titles-bf16.gguf gguf/arxiv-titles-Q4_K_M.gguf; do\n  tools/llama.cpp/build/bin/llama-perplexity -m \"$f\" -f eval.txt -ngl 99 | tail -n 2\ndone\n```\n\nThe tool ends each run with a final perplexity estimate. A small rise from bf16 to Q4_K_M is normal. A large jump means the quantization hurt your task, and the next step up (Q5_K_M or Q6_K) is cheap insurance. Use text the model has not seen during fine-tuning, or both numbers will look better than they deserve.\n\n## Run the GGUF Locally\n\n### Step 10: Serve with llama-server and test over HTTP\n\nThe server speaks an OpenAI-compatible API, which makes it the easiest target for a repeatable test. Bind it to localhost and set an API key. The November hardening work on `llama-server` does not excuse an open port on a shared network.\n\n```\nexport LLAMA_API_KEY=\"$(openssl rand -hex 16)\"\ntools/llama.cpp/build/bin/llama-server \\\n  -m gguf/arxiv-titles-Q4_K_M.gguf \\\n  --host 127.0.0.1 --port 8080 -ngl 99 -c 4096 \\\n  --api-key \"$LLAMA_API_KEY\"\n```\n\nIn a second terminal:\n\n```\ncurl -s http://127.0.0.1:8080/v1/chat/completions \\\n  -H \"Authorization: Bearer $LLAMA_API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"messages\":[{\"role\":\"user\",\"content\":\"Write a title for a paper on low-rank adapters for small language models.\"}],\"temperature\":0,\"max_tokens\":32}'\n{\"choices\":[{\"finish_reason\":\"stop\",\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\"Low-Rank Adapters for Efficient Small Language Model Tuning\"}}], ...}\n```\n\nWith temperature 0, the Q4_K_M answer should land close to the bf16 reply from Step 6, though a few words may differ. Identical text is not the goal. The goal is that the model still shows the fine-tuned behavior, so a title-shaped answer is a pass and a rambling generic reply is a fail. The llama.cpp web interface at the same address also works for a quick manual check. For a fuller front end, the [llama.cpp server and Open WebUI setup](https://shattered.io/llama-cpp-server-open-webui-gguf-12-steps-2026/) covers the next layer up. A quick command-line check works too:\n\n```\ntools/llama.cpp/build/bin/llama-cli -m gguf/arxiv-titles-Q4_K_M.gguf \\\n  -no-cnv -n 48 --temp 0 \\\n  -p \"Write a title for a paper on low-rank adapters for small language models.\"\n```\n\nFlag names in `llama-cli` shift between builds, so check `--help` if your build behaves differently.\n\n### Step 11: Import the model into Ollama\n\nOllama imports GGUF files through a Modelfile. Its import documentation says it does not quantize during import, so the Q4_K_M file you made in Step 9 is exactly what gets served. Ollama 0.30, announced on June 5, 2026, improved GGUF compatibility through llama.cpp, so any current release handles standard architectures. Create `Modelfile` next to the GGUF:\n\n```\nFROM ./gguf/arxiv-titles-Q4_K_M.gguf\nTEMPLATE \"\"\"{{- range .Messages }}<|im_start|>{{ .Role }}\n{{ .Content }}<|im_end|>\n{{ end }}<|im_start|>assistant\n\"\"\"\nPARAMETER stop \"<|im_end|>\"\nPARAMETER temperature 0.2\nollama create arxiv-titles -f Modelfile\nollama run arxiv-titles \"Write a title for a paper on low-rank adapters for small language models.\"\ngathering model components\ncopying file sha256:...\nparsing GGUF\nwriting manifest\nsuccess\nLow-Rank Adapters for Efficient Small Language Model Tuning\n```\n\nThe TEMPLATE block above is the ChatML format that Qwen models use. It must match the format the adapter was trained with, or the model answers as if the fine-tuning never happened. For another model family, copy the template from that family’s official Ollama listing with `ollama show --modelfile`.\n\n## Option B: Ship the Adapter as Its Own GGUF\n\n### Step 12: Convert the adapter alone and apply it at load time\n\nUse this path when you serve several adapters on one base or want to swap behavior without reloading gigabytes of weights. It is only safe for the rows marked “Works” in the compatibility table. Run the preflight first and stop if it lists adapter-GGUF blockers.\n\n```\nsource .venv-llama/bin/activate\npython tools/llama.cpp/convert_lora_to_gguf.py adapters/demo \\\n  --base models/base --outfile gguf/arxiv-titles-lora-f16.gguf --outtype f16\n\npython tools/llama.cpp/convert_hf_to_gguf.py models/base \\\n  --outfile gguf/base-bf16.gguf --outtype bf16\ntools/llama.cpp/build/bin/llama-quantize gguf/base-bf16.gguf gguf/base-Q4_K_M.gguf Q4_K_M\n\ntools/llama.cpp/build/bin/llama-server -m gguf/base-Q4_K_M.gguf \\\n  --lora gguf/arxiv-titles-lora-f16.gguf --host 127.0.0.1 --port 8080\n```\n\nThe `--base` argument points at the original Hugging Face folder, because the converter needs its config to map tensor names. Use `--lora-scaled` instead of `--lora` when you want to dial the adapter’s strength up or down. The older merge-through-the-loader route is gone in v1.0.0, so a tutorial that passes a base file to the loader for merging is out of date.\n\nTwo cautions apply. First, the adapter was trained against full-precision or NF4 weights, and you are now applying it to a Q4_K_M base. The result is usually fine and sometimes visibly worse than the merged-then-quantized file, so run the same three prompts through both and compare. Second, llama.cpp applies the adapter separately at inference time and does not fold it into the base, which keeps memory-mapped loading of the base intact. That is the whole point of the route, and it explains why rsLoRA and DoRA adapters need extra care. For an rsLoRA adapter, compare logits against the merged model before you trust the converted file, since the converter’s scale handling was still being fixed in November.\n\n## Merge or Keep It Separate: A Decision Guide\n\nThe merged GGUF and the adapter-only GGUF solve different problems, and picking by habit wastes either disk or speed. Five questions settle it quickly.\n\n- **How many behaviors share one base model?** One behavior means merge. Three or more tasks on the same base favor separate adapters, because each extra adapter costs a small file while each extra merged model costs a full copy of the base.\n- **Who runs the file?** A teammate importing into Ollama wants one self-contained GGUF and a Modelfile. A service you operate on llama.cpp can juggle adapters at load time.\n- **Which adapter traits does the preflight report?** Any adapter-GGUF blocker from Step 4 ends the debate in favor of merging.\n- **How much does speed matter?** Applying an adapter at inference adds work on every targeted layer. For latency-sensitive serving, the merged file removes that overhead.\n- **How often does the adapter change?** A weekly retrain argues for adapters, since you ship megabytes instead of gigabytes. A frozen release argues for a merge, a single checksum and a single artifact to support.\n\nDefault to the merge. It carries the fewest hidden assumptions, it works with every runtime that reads GGUF, and the smoke test proves it matches the adapter you trained. Move to adapter-only when you have a concrete reason, such as several tasks on one base or a retraining cadence that makes gigabyte uploads painful.\n\nThe base model choice interacts with all of this. A bigger base raises the merge’s RAM floor, raises the GGUF size and lowers how many copies fit on one disk, which pushes teams with many tasks toward adapters. A smaller base makes merged copies cheap enough to keep one per task. The [best open-source LLM roundup](https://shattered.io/best-open-source-llm-october-2026-kimi-glm-deepseek/) helps you weigh base candidates, and the broader [AI and machine learning coverage](https://shattered.io/category/ai-machine-learning/) tracks new open-weight releases that could change the answer.\n\nWhichever route you pick, record it in the manifest. A file named `arxiv-titles-Q4_K_M.gguf` that came from a merge and a file with the same name that came from base-plus-adapter behave differently under load, and six months from now nobody will remember which one is in production.\n\n## The Complete Project: One Script From Adapter to Ollama Model\n\nThe steps above chain into a single entry point. `ship_adapter.py` imports `preflight` from Step 4 and `merge` from Step 5, runs the smoke test from Step 6 as a subprocess, then calls the llama.cpp tools and writes a Modelfile plus a manifest with checksums. The project layout after a run looks like this:\n\n```\nadapter-shipper/\n  fetch.py  inspect_adapter.py  preflight.py  merge.py  smoke_test.py  ship_adapter.py\n  models/base/            adapters/demo/         merged/arxiv-titles/\n  gguf/arxiv-titles-bf16.gguf   gguf/arxiv-titles-Q4_K_M.gguf\n  Modelfile   manifest.json   requirements.lock\n  tools/llama.cpp/        .venv/   .venv-llama/\n```\n\nSave the orchestrator as `ship_adapter.py`.\n\n``` python\n#!/usr/bin/env python3\nimport argparse\nimport hashlib\nimport json\nimport subprocess\nimport sys\nfrom pathlib import Path\n\nimport peft\n\nfrom merge import merge\nfrom preflight import preflight\n\nMODELFILE = '''FROM ./__GGUF__\nTEMPLATE \"\"\"{{- range .Messages }}<|im_start|>{{ .Role }}\n{{ .Content }}<|im_end|>\n{{ end }}<|im_start|>assistant\n\"\"\"\nPARAMETER stop \"<|im_end|>\"\nPARAMETER temperature 0.2\n'''\n\ndef run(cmd):\n    print(\"+\", \" \".join(str(c) for c in cmd), flush=True)\n    subprocess.run([str(c) for c in cmd], check=True)\n\ndef sha256(path: Path) -> str:\n    h = hashlib.sha256()\n    with open(path, \"rb\") as f:\n        for chunk in iter(lambda: f.read(1 << 20), b\"\"):\n            h.update(chunk)\n    return h.hexdigest()\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--base\", required=True, type=Path)\n    ap.add_argument(\"--adapter\", required=True, type=Path)\n    ap.add_argument(\"--name\", required=True)\n    ap.add_argument(\"--llama-cpp\", default=Path(\"tools/llama.cpp\"), type=Path)\n    ap.add_argument(\"--llama-python\", default=\".venv-llama/bin/python\")\n    ap.add_argument(\"--quant\", default=\"Q4_K_M\")\n    ap.add_argument(\"--skip-smoke\", action=\"store_true\")\n    args = ap.parse_args()\n\n    blockers, adapter_notes = preflight(args.adapter)\n    if blockers:\n        sys.exit(f\"preflight failed: {blockers}\")\n    for note in adapter_notes:\n        print(\"note, adapter-only GGUF is unsafe:\", note)\n\n    merged = Path(\"merged\") / args.name\n    Path(\"gguf\").mkdir(exist_ok=True)\n    merge(args.base, args.adapter, merged)\n\n    if not args.skip_smoke:\n        run([sys.executable, \"smoke_test.py\", args.base, args.adapter, merged])\n\n    bf16 = Path(\"gguf\") / f\"{args.name}-bf16.gguf\"\n    quant = Path(\"gguf\") / f\"{args.name}-{args.quant}.gguf\"\n    run([args.llama_python, args.llama_cpp / \"convert_hf_to_gguf.py\",\n         merged, \"--outfile\", bf16, \"--outtype\", \"bf16\"])\n    run([args.llama_cpp / \"build\" / \"bin\" / \"llama-quantize\", bf16, quant, args.quant])\n\n    Path(\"Modelfile\").write_text(MODELFILE.replace(\"__GGUF__\", str(quant)))\n\n    cfg = json.loads((args.adapter / \"adapter_config.json\").read_text())\n    manifest = {\n        \"name\": args.name,\n        \"base_model\": cfg.get(\"base_model_name_or_path\"),\n        \"peft_version\": peft.__version__,\n        \"llama_cpp_tag\": \"v1.0.0\",\n        \"quant\": args.quant,\n        \"files\": {p.name: {\"bytes\": p.stat().st_size, \"sha256\": sha256(p)} for p in (bf16, quant)},\n    }\n    Path(\"manifest.json\").write_text(json.dumps(manifest, indent=2))\n    print(json.dumps(manifest, indent=2))\n    print(f\"next: ollama create {args.name} -f Modelfile\")\n\nif __name__ == \"__main__\":\n    main()\nsource .venv/bin/activate\npython ship_adapter.py --base models/base --adapter adapters/demo --name arxiv-titles\nollama create arxiv-titles -f Modelfile\n```\n\nExample final output (hashes and sizes are placeholders, yours will differ):\n\n```\n{\n  \"name\": \"arxiv-titles\",\n  \"base_model\": \"Qwen/Qwen2.5-1.5B-Instruct\",\n  \"peft_version\": \"...\",\n  \"llama_cpp_tag\": \"v1.0.0\",\n  \"quant\": \"Q4_K_M\",\n  \"files\": {\n    \"arxiv-titles-bf16.gguf\": {\"bytes\": 3087000000, \"sha256\": \"...\"},\n    \"arxiv-titles-Q4_K_M.gguf\": {\"bytes\": 986000000, \"sha256\": \"...\"}\n  }\n}\nnext: ollama create arxiv-titles -f Modelfile\n```\n\nThe manifest is the part people skip and later regret. When a colleague reports that the model “got worse” in January, the manifest tells you which base, which PEFT release and which llama.cpp tag produced the file, and the checksum proves they are testing the same artifact you shipped.\n\nThe script fails loudly at the first bad hop. A failed preflight exits before any weights load, a failed smoke test stops before any GGUF exists, and the converter’s non-zero exit code halts the quantize step. That ordering is deliberate, because the expensive steps come last.\n\n## Common Pitfalls\n\n- **Merging into a 4-bit base.** QLoRA trains against NF4 weights, so the temptation is to reload the same quantized model and merge. Load the original 16-bit weights instead. Merging into quantized weights either errors out or adds rounding noise that Q4_K_M will then compound.\n- **Pairing the adapter with the wrong base revision.** An adapter trained on the Instruct checkpoint and merged into the base checkpoint, or into a base that was re-released after training, loads without complaint and answers worse. Compare`base_model_name_or_path` against what you downloaded every time.\n- **Dropping the tokenizer.** The converter copies the vocabulary and chat template into the GGUF header. Save the tokenizer from the adapter folder when it exists, since fine-tunes sometimes add special tokens.\n- **Building llama.cpp from master.** Numbered pre-release builds ship daily, and the 1.0 cleanup removed deprecated APIs. Pin the v1.0.0 tag so a conversion that worked today still works in March.\n- **Merging in fp16.** bf16 has the same exponent range as float32, while fp16 has a narrower one, and some model families overflow in fp16. Use bf16, or float32 when you want the highest fidelity.\n- **Quantizing an already quantized file.**`llama-quantize` refuses to requantize unless you pass`--allow-requantize` , and for good reason. Always quantize from the bf16 master.\n- **Mismatched chat templates in Ollama.** If the Modelfile TEMPLATE differs from the format used during fine-tuning, the model behaves like the untuned base. The GGUF is fine and the prompt is wrong.\n- **Using the DoRA-to-LoRA trick.** Converting a DoRA adapter to plain LoRA to unlock adapter-only GGUF is attractive, and a reported November 16 issue says the converted result can differ from the trained model. Merge instead.\n- **Ignoring the license.** A merged model is a derivative of its base and carries the base model’s terms. Check the[open-weight license comparison](https://shattered.io/open-weight-models-license-comparison-6-families-16gb-2026/) before you publish a merged checkpoint or a GGUF.\n- **Sharing one Python environment.** The llama.cpp requirements file can pull a different PyTorch build than your merge environment needs. Two virtual environments cost 10 seconds of setup and save an afternoon.\n\n## Troubleshooting\n\n- **“Can’t find adapter_config.json” when loading the adapter.** You pointed`PeftModel.from_pretrained` at the parent folder or at a partial download. Pass the directory that directly contains`adapter_config.json` , and rerun`fetch.py` if files are missing.\n- **Size mismatch for `embed_tokens` or `lm_head` while loading.** The adapter was trained after tokens were added. Load the adapter’s tokenizer, grow the embedding matrix before attaching the adapter (as`merge.py` does), and merge instead of converting the adapter alone.\n- **Out of memory during the merge.** Load with`low_cpu_mem_usage=True` and bf16, close other applications, and keep the merge on CPU. For a 27B model, plan for roughly 54 GB of RAM, or merge on a larger machine and copy the result back.\n- **The converter reports an unsupported architecture.** Either you pointed it at the adapter folder instead of the merged folder, or the model family is newer than your checkout. Check that`config.json` lists an architecture, then try a newer llama.cpp tag.\n- **`git checkout v1.0.0` reports an unknown ref.** Your clone did not fetch tags. Run`git fetch --tags` , or list tags with`git tag --list \"v1*\"` .\n- **CMake cannot find CUDA.** Install a CUDA toolkit that matches your driver, or rebuild without`-DGGML_CUDA=ON` . A CPU build is slower to run but converts and quantizes identically.\n- **The Q4_K_M model is gibberish but the bf16 file is fine.** Quantization hit a sensitive tensor. Try Q5_K_M, Q6_K or Q8_0, and read the importance-matrix tip in the next section.\n- **Ollama creates the model but ignores the fine-tuning.** Check that the Modelfile points at the merged GGUF and not the base file, then diff the TEMPLATE against the format you trained with.\n- **Ollama fails with `unknown renderer \"qwen3.8\"`.** The runtime is older than the model family. One community listing says Ollama 0.32.12 or newer is required, so upgrade Ollama before you suspect the GGUF.\n- **`llama-server --lora` cannot load the adapter or reports missing tensors.** The base GGUF architecture does not match the adapter, or the adapter carries features the converter cannot express, such as DoRA magnitude vectors or`modules_to_save` . Merge instead.\n- **The smoke test gap is huge (far above 1.0).** Suspect the wrong base or a missing tokenizer first. Rerun Step 3 and confirm`base_model_name_or_path` matches the folder you passed to`merge.py` .\n- **An old tutorial’s `--lora-base` flag errors out.** That merge path was removed from the current documentation. Merge with PEFT as in Step 5.\n\n## Advanced Tips\n\n**Use an importance matrix for low-bit quantization.** The llama.cpp documentation describes quantization types that rely on an importance matrix, which measures which weights matter on real text. Generate one from domain text that resembles your fine-tuning data, then quantize with it:\n\n```\ntools/llama.cpp/build/bin/llama-imatrix -m gguf/arxiv-titles-bf16.gguf \\\n  -f calibration.txt -o gguf/imatrix.dat\ntools/llama.cpp/build/bin/llama-quantize --imatrix gguf/imatrix.dat \\\n  gguf/arxiv-titles-bf16.gguf gguf/arxiv-titles-Q3_K_M.gguf Q3_K_M\n```\n\nCalibration text from your own task is the whole trick. A matrix built from generic web text protects the wrong weights.\n\n**Combine two adapters before merging.** PEFT can blend adapters that share a base. The `linear` combination type needs matching ranks, so check `r` in both configs first.\n\n``` python\nfrom peft import PeftModel\nmodel = PeftModel.from_pretrained(base_model, \"adapters/style\", adapter_name=\"style\")\nmodel.load_adapter(\"adapters/domain\", adapter_name=\"domain\")\nmodel.add_weighted_adapter([\"style\", \"domain\"], [0.6, 0.4],\n                           adapter_name=\"blend\", combination_type=\"linear\")\nmodel.set_adapter(\"blend\")\nmerged = model.merge_and_unload()\n```\n\nRe-run the smoke test against the blend with prompts from both tasks. Blends often keep one skill and dilute the other, and the weights need tuning.\n\n**Hot-swap adapters on one base.** With Option B, `llama-server` accepts several `--lora` arguments, and recent builds can load them without applying them at startup and expose an endpoint for changing scales per request. Flag names vary by build, so confirm with `llama-server --help` before you design around it. Compare the cost against merging: one base plus three small adapters beats three full merged models on disk, and loses on raw speed.\n\n**Skip the manual steps only when you do not need to debug.** Several training tools, Unsloth among them, can export GGUF directly after training. That shortcut removes the intermediate merged checkpoint, which is the artifact you want when a quantized model misbehaves. Hugging Face also publishes a [convert_to_gguf.py helper](https://github.com/huggingface/skills/blob/25910b3d0ba31a2ba102f901cdc9e008c278a69c/skills/huggingface-llm-trainer/scripts/convert_to_gguf.py) in its skills repository, which is a useful reference for a scripted pipeline. A community repository, reportedly, documents the same pattern for a Granite 4.0 H 1B math LoRA: merge with `merge_and_unload`, convert to F16 GGUF, then quantize into several files. The recipe is standard.\n\n**Keep golden prompts in version control.** Store 20 prompts and their bf16 answers, run them through every new quantization, and fail the build when the answers drift. The smoke test in Step 6 becomes a regression suite for free. When you move to a newer base model, the [open-weight spec sheet](https://shattered.io/open-weight-ai-models-side-by-side-the-spec-sheet/) helps you check parameter counts and context windows before you retrain.\n\n## FAQ\n\n**Can I merge a LoRA adapter into a quantized model?**\n\nAvoid it. Merge into the original 16-bit weights, then quantize the result to GGUF. Merging into a 4-bit base adds rounding error that the later quantization step compounds.\n\n**Do I have to merge before creating a GGUF?**\n\nNo. You can convert the adapter to its own GGUF and apply it with `--lora`. That path only fits plain LoRA and QLoRA-trained adapters. Adapters with DoRA vectors, modified embeddings or added tokens should be merged.\n\n**Which llama.cpp version should I use for conversion?**\n\nUse the v1.0.0 tag, the first stable release, published on November 13, 2026. Daily pre-release builds exist, but a pinned tag keeps your conversions reproducible.\n\n**Which quantization should I choose for a merged adapter?**\n\nStart with Q4_K_M for laptops and Q8_0 when memory allows. Run your golden prompts against the bf16 file and the quantized file, and only drop to Q3 or lower with an importance matrix built from your own domain text.\n\n**Does merging change the model size?**\n\nNo. The merged checkpoint has the same parameter count and file size as the base model, whatever rank the adapter used. The adapter’s rank only affects how much the adapter could learn during training.\n\n**How do I confirm the merged model still behaves like the adapter?**\n\nCompare last-token logits between the base with the adapter attached and the merged checkpoint, as the smoke test does. Large gaps point to the wrong base, a missing tokenizer or a skipped embedding resize.\n\n**Can I run the result in Ollama and llama.cpp both?**\n\nYes. Both read the same GGUF file. Ollama needs a Modelfile with a TEMPLATE that matches the training format, while llama.cpp reads the chat template straight from the GGUF header.", "url": "https://wpnews.pro/news/merge-a-lora-adapter-into-gguf-12-steps-80-min", "canonical_source": "https://shattered.io/merge-lora-adapter-gguf-llama-cpp-ollama-12-steps/", "published_at": "2026-11-18 10:27:24+00:00", "updated_at": "2026-11-18 10:45:50.183585+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-infrastructure", "machine-learning"], "entities": ["llama.cpp", "ggml", "Hugging Face PEFT", "Qwen2.5-1.5B-Instruct", "Ollama", "LoRA", "QLoRA", "DoRA"], "also_reported_by": [], "takeaway": "Merge a QLoRA adapter into the original 16-bit base weights rather than the 4-bit NF4 base, then convert to bf16 GGUF and quantize once to Q4_K_M, because quantizing twice stacks error on error.", "recommendation": {"advice": "Merge LoRA adapters into the 16-bit base weights, not the 4-bit NF4 base, then convert to bf16 GGUF and quantize once to Q4_K_M to avoid stacking quantization error.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/merge-a-lora-adapter-into-gguf-12-steps-80-min", "markdown": "https://wpnews.pro/news/merge-a-lora-adapter-into-gguf-12-steps-80-min.md", "text": "https://wpnews.pro/news/merge-a-lora-adapter-into-gguf-12-steps-80-min.txt", "jsonld": "https://wpnews.pro/news/merge-a-lora-adapter-into-gguf-12-steps-80-min.jsonld"}}