{"slug": "convert-and-quantize-hugging-face-models-to-gguf-for-llama-cpp", "title": "Convert and Quantize Hugging Face Models to GGUF for llama.cpp", "summary": "Mariana Souza published a tutorial on SourceFeed showing how to convert and quantize Hugging Face models to GGUF for llama.cpp, using Qwen3-0.6B as an example. The process turns a 1.5 GB safetensors checkpoint into a ~400 MB 4-bit GGUF file that runs locally on CPU or GPU, verified against llama.cpp release b10375 with Python 3.12 on macOS 15 and Ubuntu 24.04.", "body_md": "# Convert and Quantize Hugging Face Models to GGUF for llama.cpp\n\nTurn any Hugging Face checkpoint into a 4-bit GGUF that runs fast and small on your own hardware.\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)\n\n## What you'll build / learn\n\nYou'll take a stock [Hugging Face](https://huggingface.co) model (Qwen3-0.6B), convert it to GGUF, and quantize it to 4-bit with [llama.cpp](https://github.com/ggml-org/llama.cpp) — turning a 1.5 GB safetensors checkpoint into a ~400 MB file that runs locally on your CPU or GPU. The same three commands work for any supported architecture, so swap in whatever model you actually care about.\n\n## Prerequisites\n\nVerified against llama.cpp release `b10375`\n\n(August 2026) with Python 3.12 on macOS 15 and Ubuntu 24.04.\n\n**Git, CMake (≥ 3.14), and a C++17 compiler**— Xcode Command Line Tools on macOS,`build-essential`\n\n+`cmake`\n\non Debian/Ubuntu.**Python 3.10+** with`venv`\n\n. The conversion deps pin`torch 2.11.0`\n\n(CPU wheel) and`transformers 4.57.6`\n\n.**~4 GB free disk** for this model: 1.5 GB download, 1.2 GB converted file, 0.4 GB quantized file. Budget roughly 4× a model's parameter count in bytes if you bring your own.- No GPU required. Conversion and quantization are CPU-only operations; a GPU only helps at inference time.\n- No Hugging Face account needed for Qwen3-0.6B (Apache-2.0, ungated). Gated models like Llama need\n`hf auth login`\n\nfirst.\n\n## 1. Clone and build llama.cpp\n\nYou need the source checkout either way — the conversion script lives in the repo root — so build the binaries from it too:\n\n```\ngit clone https://github.com/ggml-org/llama.cpp\ncd llama.cpp\ncmake -B build\ncmake --build build --config Release -j 8\n```\n\nThis produces `llama-quantize`\n\n, `llama-cli`\n\n, and friends under `build/bin/`\n\n. On macOS, Metal support is compiled in by default; on NVIDIA boxes add `-DGGML_CUDA=ON`\n\nto the first cmake call if you want GPU inference later. Prebuilt binaries from the releases page also work, but you still need this repo for the Python script.\n\n## 2. Install the Python conversion dependencies\n\nThe converter is a Python script with its own pinned requirements. Keep them in a venv so the pinned torch/transformers versions don't fight your global site-packages:\n\n```\npython3 -m venv .venv\nsource .venv/bin/activate\npython3 -m pip install -r requirements.txt\n```\n\nThis pulls a CPU-only PyTorch wheel (via the `download.pytorch.org/whl/cpu`\n\nindex the requirements file specifies), so it won't drag in CUDA libraries. It also installs `transformers`\n\n, which ships the `hf`\n\nCLI you'll use next. One caveat from the llama.cpp docs: the pins install transformers 4, and some very new models (Gemma 4, for example) need transformers 5 — `pip install -U transformers`\n\nis safe if conversion complains about an unrecognized model type.\n\n## 3. Download the model from Hugging Face\n\nGrab the full repo — the converter needs `config.json`\n\nand the tokenizer files, not just the weights:\n\n```\nhf download Qwen/Qwen3-0.6B --local-dir Qwen3-0.6B\n```\n\nYou'll end up with `model.safetensors`\n\n(1.5 GB) plus config and tokenizer files in `Qwen3-0.6B/`\n\n. For a gated model, run `hf auth login`\n\nwith a token from [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) before downloading.\n\n## 4. Convert to GGUF\n\nGGUF is llama.cpp's single-file format: weights, tokenizer, and metadata together, laid out for mmap-friendly loading. Convert at full precision first — you always quantize *from* a bf16/f16 GGUF, never re-quantize an already-quantized file, because each lossy pass compounds the error:\n\n```\npython3 convert_hf_to_gguf.py Qwen3-0.6B \\\n  --outfile Qwen3-0.6B-BF16.gguf \\\n  --outtype bf16\n```\n\nThe script logs each tensor as it maps and writes it, then finishes with:\n\n```\nINFO:hf-to-gguf:Model successfully exported to Qwen3-0.6B-BF16.gguf\n```\n\n`--outtype auto`\n\n(the default) also works — it matches whatever precision the source weights use. And if you'd rather skip step 3 entirely, `--remote`\n\nstreams tensors straight from the Hub: `python3 convert_hf_to_gguf.py Qwen/Qwen3-0.6B --remote --outfile Qwen3-0.6B-BF16.gguf --outtype bf16`\n\n.\n\n## 5. Quantize to 4-bit\n\nNow shrink it. `Q4_K_M`\n\nis the community default for a reason: roughly 4.5 bits per weight with the quality-critical tensors kept at higher precision, which costs little accuracy on most models:\n\n```\n./build/bin/llama-quantize Qwen3-0.6B-BF16.gguf Qwen3-0.6B-Q4_K_M.gguf Q4_K_M\n```\n\nIt runs in under a minute for a model this size, printing per-tensor lines as it converts. Other useful targets: `Q8_0`\n\n(near-lossless, ~2× smaller than bf16), `Q5_K_M`\n\n(middle ground), `Q3_K_M`\n\nand below (only when you're desperate for RAM). Run `./build/bin/llama-quantize --help`\n\nfor the full list with per-type size/quality estimates.\n\n## Verify it works\n\nCheck the sizes first:\n\n```\nls -lh Qwen3-0.6B-*.gguf\n-rw-r--r--  1 you  staff   1.2G Aug 12 10:41 Qwen3-0.6B-BF16.gguf\n-rw-r--r--  1 you  staff   397M Aug 12 10:44 Qwen3-0.6B-Q4_K_M.gguf\n```\n\nThen actually run the quantized model. `-st`\n\n(single turn) answers one prompt and exits instead of dropping into interactive chat:\n\n```\n./build/bin/llama-cli -m Qwen3-0.6B-Q4_K_M.gguf \\\n  -p \"Explain what GGUF is in one sentence.\" -st -n 256\n```\n\nAfter the loader output you should see a coherent answer (Qwen3 emits a `<think>…</think>`\n\nreasoning block first — that's normal), something like:\n\n```\nGGUF is a binary file format used to store quantized large language\nmodels for efficient local inference with llama.cpp.\n```\n\nIf the model loads, generates fluent text, and exits cleanly, your conversion and quantization are good.\n\n## Troubleshooting\n\n** ERROR:hf-to-gguf:Model <name>ForCausalLM is not supported** — the architecture isn't in your checkout's converter, usually because the model is newer than your clone.\n\n`git pull`\n\n, re-run `pip install -r requirements.txt`\n\n, and rebuild. If it still fails, the architecture genuinely isn't supported yet — search the llama.cpp issues/PRs for it.** ValueError: The checkpoint you are trying to load has model type '<x>' but Transformers does not recognize this architecture** — your transformers is too old for the model. Run\n\n`pip install -U transformers`\n\ninside the venv (the llama.cpp docs explicitly bless this over the pinned version).** ModuleNotFoundError: No module named 'gguf'** (or\n\n`'torch'`\n\n) — you're running the script outside the venv, or skipped step 2. Run `source .venv/bin/activate`\n\nand retry; check `which python3`\n\npoints into `.venv`\n\n.** GatedRepoError: 403 Client Error … Access to model <x> is restricted** — the model requires accepting a license on its Hugging Face page. Accept it in the browser while logged in, then\n\n`hf auth login`\n\nwith a read token and re-download.## Next steps\n\nServe your quantized model over an OpenAI-compatible API with `./build/bin/llama-server -m Qwen3-0.6B-Q4_K_M.gguf`\n\n, then point any OpenAI client at `localhost:8080`\n\n. To squeeze quality out of aggressive quants (Q3 and below), generate an importance matrix with `llama-imatrix`\n\non a calibration text file and pass it to `llama-quantize --imatrix`\n\n. For sharing, `hf upload`\n\npushes your GGUF to a Hub repo so others can pull it with `llama cli -hf you/your-model-GGUF`\n\n. And if you want quants without any local setup, the [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space runs this exact pipeline in your browser.\n\n## Sources & further reading\n\n-\n[llama.cpp quantize tool documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md)— github.com -\n[llama.cpp build guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md)— github.com -\n[llama-cli reference](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md)— github.com -\n[Hugging Face hf CLI guide](https://huggingface.co/docs/huggingface_hub/guides/cli)— huggingface.co -\n[Qwen3-0.6B model card](https://huggingface.co/Qwen/Qwen3-0.6B)— huggingface.co\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor\n\nMariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/convert-and-quantize-hugging-face-models-to-gguf-for-llama-cpp", "canonical_source": "https://sourcefeed.dev/a/convert-and-quantize-hugging-face-models-to-gguf-for-llamacpp", "published_at": "2026-08-12 17:41:58+00:00", "updated_at": "2026-08-12 17:43:44.315024+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "artificial-intelligence"], "entities": ["Mariana Souza", "Hugging Face", "Qwen3-0.6B", "llama.cpp", "SourceFeed"], "alternates": {"html": "https://wpnews.pro/news/convert-and-quantize-hugging-face-models-to-gguf-for-llama-cpp", "markdown": "https://wpnews.pro/news/convert-and-quantize-hugging-face-models-to-gguf-for-llama-cpp.md", "text": "https://wpnews.pro/news/convert-and-quantize-hugging-face-models-to-gguf-for-llama-cpp.txt", "jsonld": "https://wpnews.pro/news/convert-and-quantize-hugging-face-models-to-gguf-for-llama-cpp.jsonld"}}