# Convert and Quantize Hugging Face Models to GGUF for llama.cpp > Source: > Published: 2026-08-12 17:41:58+00:00 # Convert and Quantize Hugging Face Models to GGUF for llama.cpp Turn any Hugging Face checkpoint into a 4-bit GGUF that runs fast and small on your own hardware. [Mariana Souza](https://sourcefeed.dev/u/mariana_souza) ## What you'll build / learn You'll take a stock [Hugging Face](https://huggingface.co) model (Qwen3-0.6B), convert it to GGUF, and quantize it to 4-bit with [llama.cpp](https://github.com/ggml-org/llama.cpp) — turning a 1.5 GB safetensors checkpoint into a ~400 MB file that runs locally on your CPU or GPU. The same three commands work for any supported architecture, so swap in whatever model you actually care about. ## Prerequisites Verified against llama.cpp release `b10375` (August 2026) with Python 3.12 on macOS 15 and Ubuntu 24.04. **Git, CMake (≥ 3.14), and a C++17 compiler**— Xcode Command Line Tools on macOS,`build-essential` +`cmake` on Debian/Ubuntu.**Python 3.10+** with`venv` . The conversion deps pin`torch 2.11.0` (CPU wheel) and`transformers 4.57.6` .**~4 GB free disk** for this model: 1.5 GB download, 1.2 GB converted file, 0.4 GB quantized file. Budget roughly 4× a model's parameter count in bytes if you bring your own.- No GPU required. Conversion and quantization are CPU-only operations; a GPU only helps at inference time. - No Hugging Face account needed for Qwen3-0.6B (Apache-2.0, ungated). Gated models like Llama need `hf auth login` first. ## 1. Clone and build llama.cpp You need the source checkout either way — the conversion script lives in the repo root — so build the binaries from it too: ``` git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build cmake --build build --config Release -j 8 ``` This produces `llama-quantize` , `llama-cli` , and friends under `build/bin/` . On macOS, Metal support is compiled in by default; on NVIDIA boxes add `-DGGML_CUDA=ON` to the first cmake call if you want GPU inference later. Prebuilt binaries from the releases page also work, but you still need this repo for the Python script. ## 2. Install the Python conversion dependencies The converter is a Python script with its own pinned requirements. Keep them in a venv so the pinned torch/transformers versions don't fight your global site-packages: ``` python3 -m venv .venv source .venv/bin/activate python3 -m pip install -r requirements.txt ``` This pulls a CPU-only PyTorch wheel (via the `download.pytorch.org/whl/cpu` index the requirements file specifies), so it won't drag in CUDA libraries. It also installs `transformers` , which ships the `hf` CLI you'll use next. One caveat from the llama.cpp docs: the pins install transformers 4, and some very new models (Gemma 4, for example) need transformers 5 — `pip install -U transformers` is safe if conversion complains about an unrecognized model type. ## 3. Download the model from Hugging Face Grab the full repo — the converter needs `config.json` and the tokenizer files, not just the weights: ``` hf download Qwen/Qwen3-0.6B --local-dir Qwen3-0.6B ``` You'll end up with `model.safetensors` (1.5 GB) plus config and tokenizer files in `Qwen3-0.6B/` . For a gated model, run `hf auth login` with a token from [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) before downloading. ## 4. Convert to GGUF GGUF is llama.cpp's single-file format: weights, tokenizer, and metadata together, laid out for mmap-friendly loading. Convert at full precision first — you always quantize *from* a bf16/f16 GGUF, never re-quantize an already-quantized file, because each lossy pass compounds the error: ``` python3 convert_hf_to_gguf.py Qwen3-0.6B \ --outfile Qwen3-0.6B-BF16.gguf \ --outtype bf16 ``` The script logs each tensor as it maps and writes it, then finishes with: ``` INFO:hf-to-gguf:Model successfully exported to Qwen3-0.6B-BF16.gguf ``` `--outtype auto` (the default) also works — it matches whatever precision the source weights use. And if you'd rather skip step 3 entirely, `--remote` streams tensors straight from the Hub: `python3 convert_hf_to_gguf.py Qwen/Qwen3-0.6B --remote --outfile Qwen3-0.6B-BF16.gguf --outtype bf16` . ## 5. Quantize to 4-bit Now shrink it. `Q4_K_M` is the community default for a reason: roughly 4.5 bits per weight with the quality-critical tensors kept at higher precision, which costs little accuracy on most models: ``` ./build/bin/llama-quantize Qwen3-0.6B-BF16.gguf Qwen3-0.6B-Q4_K_M.gguf Q4_K_M ``` It runs in under a minute for a model this size, printing per-tensor lines as it converts. Other useful targets: `Q8_0` (near-lossless, ~2× smaller than bf16), `Q5_K_M` (middle ground), `Q3_K_M` and below (only when you're desperate for RAM). Run `./build/bin/llama-quantize --help` for the full list with per-type size/quality estimates. ## Verify it works Check the sizes first: ``` ls -lh Qwen3-0.6B-*.gguf -rw-r--r-- 1 you staff 1.2G Aug 12 10:41 Qwen3-0.6B-BF16.gguf -rw-r--r-- 1 you staff 397M Aug 12 10:44 Qwen3-0.6B-Q4_K_M.gguf ``` Then actually run the quantized model. `-st` (single turn) answers one prompt and exits instead of dropping into interactive chat: ``` ./build/bin/llama-cli -m Qwen3-0.6B-Q4_K_M.gguf \ -p "Explain what GGUF is in one sentence." -st -n 256 ``` After the loader output you should see a coherent answer (Qwen3 emits a `…` reasoning block first — that's normal), something like: ``` GGUF is a binary file format used to store quantized large language models for efficient local inference with llama.cpp. ``` If the model loads, generates fluent text, and exits cleanly, your conversion and quantization are good. ## Troubleshooting ** ERROR:hf-to-gguf:Model ForCausalLM is not supported** — the architecture isn't in your checkout's converter, usually because the model is newer than your clone. `git pull` , re-run `pip install -r requirements.txt` , and rebuild. If it still fails, the architecture genuinely isn't supported yet — search the llama.cpp issues/PRs for it.** ValueError: The checkpoint you are trying to load has model type '' but Transformers does not recognize this architecture** — your transformers is too old for the model. Run `pip install -U transformers` inside the venv (the llama.cpp docs explicitly bless this over the pinned version).** ModuleNotFoundError: No module named 'gguf'** (or `'torch'` ) — you're running the script outside the venv, or skipped step 2. Run `source .venv/bin/activate` and retry; check `which python3` points into `.venv` .** GatedRepoError: 403 Client Error … Access to model is restricted** — the model requires accepting a license on its Hugging Face page. Accept it in the browser while logged in, then `hf auth login` with a read token and re-download.## Next steps Serve your quantized model over an OpenAI-compatible API with `./build/bin/llama-server -m Qwen3-0.6B-Q4_K_M.gguf` , then point any OpenAI client at `localhost:8080` . To squeeze quality out of aggressive quants (Q3 and below), generate an importance matrix with `llama-imatrix` on a calibration text file and pass it to `llama-quantize --imatrix` . For sharing, `hf upload` pushes your GGUF to a Hub repo so others can pull it with `llama cli -hf you/your-model-GGUF` . And if you want quants without any local setup, the [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space runs this exact pipeline in your browser. ## Sources & further reading - [llama.cpp quantize tool documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md)— github.com - [llama.cpp build guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md)— github.com - [llama-cli reference](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md)— github.com - [Hugging Face hf CLI guide](https://huggingface.co/docs/huggingface_hub/guides/cli)— huggingface.co - [Qwen3-0.6B model card](https://huggingface.co/Qwen/Qwen3-0.6B)— huggingface.co [Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. ## Discussion 0 No comments yet Be the first to weigh in.