Convert and Quantize Hugging Face Models to GGUF for llama.cpp

Mariana Souza published a tutorial on SourceFeed showing how to convert and quantize Hugging Face models to GGUF for llama.cpp, using Qwen3-0.6B as an example. The process turns a 1.5 GB safetensors checkpoint into a ~400 MB 4-bit GGUF file that runs locally on CPU or GPU, verified against llama.cpp release b10375 with Python 3.12 on macOS 15 and Ubuntu 24.04.

Convert and Quantize Hugging Face Models to GGUF for llama.cpp Turn any Hugging Face checkpoint into a 4-bit GGUF that runs fast and small on your own hardware. Mariana Souza https://sourcefeed.dev/u/mariana souza What you'll build / learn You'll take a stock Hugging Face https://huggingface.co model Qwen3-0.6B , convert it to GGUF, and quantize it to 4-bit with llama.cpp https://github.com/ggml-org/llama.cpp — turning a 1.5 GB safetensors checkpoint into a ~400 MB file that runs locally on your CPU or GPU. The same three commands work for any supported architecture, so swap in whatever model you actually care about. Prerequisites Verified against llama.cpp release b10375 August 2026 with Python 3.12 on macOS 15 and Ubuntu 24.04. Git, CMake ≥ 3.14 , and a C++17 compiler — Xcode Command Line Tools on macOS, build-essential + cmake on Debian/Ubuntu. Python 3.10+ with venv . The conversion deps pin torch 2.11.0 CPU wheel and transformers 4.57.6 . ~4 GB free disk for this model: 1.5 GB download, 1.2 GB converted file, 0.4 GB quantized file. Budget roughly 4× a model's parameter count in bytes if you bring your own.- No GPU required. Conversion and quantization are CPU-only operations; a GPU only helps at inference time. - No Hugging Face account needed for Qwen3-0.6B Apache-2.0, ungated . Gated models like Llama need hf auth login first. 1. Clone and build llama.cpp You need the source checkout either way — the conversion script lives in the repo root — so build the binaries from it too: git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build cmake --build build --config Release -j 8 This produces llama-quantize , llama-cli , and friends under build/bin/ . On macOS, Metal support is compiled in by default; on NVIDIA boxes add -DGGML CUDA=ON to the first cmake call if you want GPU inference later. Prebuilt binaries from the releases page also work, but you still need this repo for the Python script. 2. Install the Python conversion dependencies The converter is a Python script with its own pinned requirements. Keep them in a venv so the pinned torch/transformers versions don't fight your global site-packages: python3 -m venv .venv source .venv/bin/activate python3 -m pip install -r requirements.txt This pulls a CPU-only PyTorch wheel via the download.pytorch.org/whl/cpu index the requirements file specifies , so it won't drag in CUDA libraries. It also installs transformers , which ships the hf CLI you'll use next. One caveat from the llama.cpp docs: the pins install transformers 4, and some very new models Gemma 4, for example need transformers 5 — pip install -U transformers is safe if conversion complains about an unrecognized model type. 3. Download the model from Hugging Face Grab the full repo — the converter needs config.json and the tokenizer files, not just the weights: hf download Qwen/Qwen3-0.6B --local-dir Qwen3-0.6B You'll end up with model.safetensors 1.5 GB plus config and tokenizer files in Qwen3-0.6B/ . For a gated model, run hf auth login with a token from huggingface.co/settings/tokens https://huggingface.co/settings/tokens before downloading. 4. Convert to GGUF GGUF is llama.cpp's single-file format: weights, tokenizer, and metadata together, laid out for mmap-friendly loading. Convert at full precision first — you always quantize from a bf16/f16 GGUF, never re-quantize an already-quantized file, because each lossy pass compounds the error: python3 convert hf to gguf.py Qwen3-0.6B \ --outfile Qwen3-0.6B-BF16.gguf \ --outtype bf16 The script logs each tensor as it maps and writes it, then finishes with: INFO:hf-to-gguf:Model successfully exported to Qwen3-0.6B-BF16.gguf --outtype auto the default also works — it matches whatever precision the source weights use. And if you'd rather skip step 3 entirely, --remote streams tensors straight from the Hub: python3 convert hf to gguf.py Qwen/Qwen3-0.6B --remote --outfile Qwen3-0.6B-BF16.gguf --outtype bf16 . 5. Quantize to 4-bit Now shrink it. Q4 K M is the community default for a reason: roughly 4.5 bits per weight with the quality-critical tensors kept at higher precision, which costs little accuracy on most models: ./build/bin/llama-quantize Qwen3-0.6B-BF16.gguf Qwen3-0.6B-Q4 K M.gguf Q4 K M It runs in under a minute for a model this size, printing per-tensor lines as it converts. Other useful targets: Q8 0 near-lossless, ~2× smaller than bf16 , Q5 K M middle ground , Q3 K M and below only when you're desperate for RAM . Run ./build/bin/llama-quantize --help for the full list with per-type size/quality estimates. Verify it works Check the sizes first: ls -lh Qwen3-0.6B- .gguf -rw-r--r-- 1 you staff 1.2G Aug 12 10:41 Qwen3-0.6B-BF16.gguf -rw-r--r-- 1 you staff 397M Aug 12 10:44 Qwen3-0.6B-Q4 K M.gguf Then actually run the quantized model. -st single turn answers one prompt and exits instead of dropping into interactive chat: ./build/bin/llama-cli -m Qwen3-0.6B-Q4 K M.gguf \ -p "Explain what GGUF is in one sentence." -st -n 256 After the loader output you should see a coherent answer Qwen3 emits a <think …</think reasoning block first — that's normal , something like: GGUF is a binary file format used to store quantized large language models for efficient local inference with llama.cpp. If the model loads, generates fluent text, and exits cleanly, your conversion and quantization are good. Troubleshooting ERROR:hf-to-gguf:Model <name ForCausalLM is not supported — the architecture isn't in your checkout's converter, usually because the model is newer than your clone. git pull , re-run pip install -r requirements.txt , and rebuild. If it still fails, the architecture genuinely isn't supported yet — search the llama.cpp issues/PRs for it. ValueError: The checkpoint you are trying to load has model type '<x ' but Transformers does not recognize this architecture — your transformers is too old for the model. Run pip install -U transformers inside the venv the llama.cpp docs explicitly bless this over the pinned version . ModuleNotFoundError: No module named 'gguf' or 'torch' — you're running the script outside the venv, or skipped step 2. Run source .venv/bin/activate and retry; check which python3 points into .venv . GatedRepoError: 403 Client Error … Access to model <x is restricted — the model requires accepting a license on its Hugging Face page. Accept it in the browser while logged in, then hf auth login with a read token and re-download. Next steps Serve your quantized model over an OpenAI-compatible API with ./build/bin/llama-server -m Qwen3-0.6B-Q4 K M.gguf , then point any OpenAI client at localhost:8080 . To squeeze quality out of aggressive quants Q3 and below , generate an importance matrix with llama-imatrix on a calibration text file and pass it to llama-quantize --imatrix . For sharing, hf upload pushes your GGUF to a Hub repo so others can pull it with llama cli -hf you/your-model-GGUF . And if you want quants without any local setup, the GGUF-my-repo https://huggingface.co/spaces/ggml-org/gguf-my-repo space runs this exact pipeline in your browser. Sources & further reading - llama.cpp quantize tool documentation https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md — github.com - llama.cpp build guide https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md — github.com - llama-cli reference https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md — github.com - Hugging Face hf CLI guide https://huggingface.co/docs/huggingface hub/guides/cli — huggingface.co - Qwen3-0.6B model card https://huggingface.co/Qwen/Qwen3-0.6B — huggingface.co Mariana Souza https://sourcefeed.dev/u/mariana souza · Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. Discussion 0 No comments yet Be the first to weigh in.