Turn any Hugging Face checkpoint into a 4-bit GGUF that runs fast and small on your own hardware.
What you'll build / learn #
You'll take a stock Hugging Face model (Qwen3-0.6B), convert it to GGUF, and quantize it to 4-bit with llama.cpp — turning a 1.5 GB safetensors checkpoint into a ~400 MB file that runs locally on your CPU or GPU. The same three commands work for any supported architecture, so swap in whatever model you actually care about.
Prerequisites #
Verified against llama.cpp release b10375
(August 2026) with Python 3.12 on macOS 15 and Ubuntu 24.04.
Git, CMake (≥ 3.14), and a C++17 compiler— Xcode Command Line Tools on macOS,build-essential
+cmake
on Debian/Ubuntu.Python 3.10+ withvenv
. The conversion deps pintorch 2.11.0
(CPU wheel) andtransformers 4.57.6
.~4 GB free disk for this model: 1.5 GB download, 1.2 GB converted file, 0.4 GB quantized file. Budget roughly 4× a model's parameter count in bytes if you bring your own.- No GPU required. Conversion and quantization are CPU-only operations; a GPU only helps at inference time.
- No Hugging Face account needed for Qwen3-0.6B (Apache-2.0, ungated). Gated models like Llama need
hf auth login
first.
1. Clone and build llama.cpp #
You need the source checkout either way — the conversion script lives in the repo root — so build the binaries from it too:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j 8
This produces llama-quantize
, llama-cli
, and friends under build/bin/
. On macOS, Metal support is compiled in by default; on NVIDIA boxes add -DGGML_CUDA=ON
to the first cmake call if you want GPU inference later. Prebuilt binaries from the releases page also work, but you still need this repo for the Python script.
2. Install the Python conversion dependencies #
The converter is a Python script with its own pinned requirements. Keep them in a venv so the pinned torch/transformers versions don't fight your global site-packages:
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt
This pulls a CPU-only PyTorch wheel (via the download.pytorch.org/whl/cpu
index the requirements file specifies), so it won't drag in CUDA libraries. It also installs transformers
, which ships the hf
CLI you'll use next. One caveat from the llama.cpp docs: the pins install transformers 4, and some very new models (Gemma 4, for example) need transformers 5 — pip install -U transformers
is safe if conversion complains about an unrecognized model type.
3. Download the model from Hugging Face #
Grab the full repo — the converter needs config.json
and the tokenizer files, not just the weights:
hf download Qwen/Qwen3-0.6B --local-dir Qwen3-0.6B
You'll end up with model.safetensors
(1.5 GB) plus config and tokenizer files in Qwen3-0.6B/
. For a gated model, run hf auth login
with a token from huggingface.co/settings/tokens before down.
4. Convert to GGUF #
GGUF is llama.cpp's single-file format: weights, tokenizer, and metadata together, laid out for mmap-friendly . Convert at full precision first — you always quantize from a bf16/f16 GGUF, never re-quantize an already-quantized file, because each lossy pass compounds the error:
python3 convert_hf_to_gguf.py Qwen3-0.6B \
--outfile Qwen3-0.6B-BF16.gguf \
--outtype bf16
The script logs each tensor as it maps and writes it, then finishes with:
INFO:hf-to-gguf:Model successfully exported to Qwen3-0.6B-BF16.gguf
--outtype auto
(the default) also works — it matches whatever precision the source weights use. And if you'd rather skip step 3 entirely, --remote
streams tensors straight from the Hub: python3 convert_hf_to_gguf.py Qwen/Qwen3-0.6B --remote --outfile Qwen3-0.6B-BF16.gguf --outtype bf16
.
5. Quantize to 4-bit #
Now shrink it. Q4_K_M
is the community default for a reason: roughly 4.5 bits per weight with the quality-critical tensors kept at higher precision, which costs little accuracy on most models:
./build/bin/llama-quantize Qwen3-0.6B-BF16.gguf Qwen3-0.6B-Q4_K_M.gguf Q4_K_M
It runs in under a minute for a model this size, printing per-tensor lines as it converts. Other useful targets: Q8_0
(near-lossless, ~2× smaller than bf16), Q5_K_M
(middle ground), Q3_K_M
and below (only when you're desperate for RAM). Run ./build/bin/llama-quantize --help
for the full list with per-type size/quality estimates.
Verify it works #
Check the sizes first:
ls -lh Qwen3-0.6B-*.gguf
-rw-r--r-- 1 you staff 1.2G Aug 12 10:41 Qwen3-0.6B-BF16.gguf
-rw-r--r-- 1 you staff 397M Aug 12 10:44 Qwen3-0.6B-Q4_K_M.gguf
Then actually run the quantized model. -st
(single turn) answers one prompt and exits instead of dropping into interactive chat:
./build/bin/llama-cli -m Qwen3-0.6B-Q4_K_M.gguf \
-p "Explain what GGUF is in one sentence." -st -n 256
After the output you should see a coherent answer (Qwen3 emits a <think>…</think>
reasoning block first — that's normal), something like:
GGUF is a binary file format used to store quantized large language
models for efficient local inference with llama.cpp.
If the model loads, generates fluent text, and exits cleanly, your conversion and quantization are good.
Troubleshooting #
** ERROR:hf-to-gguf:Model <name>ForCausalLM is not supported** — the architecture isn't in your checkout's converter, usually because the model is newer than your clone.
git pull
, re-run pip install -r requirements.txt
, and rebuild. If it still fails, the architecture genuinely isn't supported yet — search the llama.cpp issues/PRs for it.** ValueError: The checkpoint you are trying to load has model type '<x>' but Transformers does not recognize this architecture** — your transformers is too old for the model. Run
pip install -U transformers
inside the venv (the llama.cpp docs explicitly bless this over the pinned version).** ModuleNotFoundError: No module named 'gguf'** (or
'torch'
) — you're running the script outside the venv, or skipped step 2. Run source .venv/bin/activate
and retry; check which python3
points into .venv
.** GatedRepoError: 403 Client Error … Access to model <x> is restricted** — the model requires accepting a license on its Hugging Face page. Accept it in the browser while logged in, then
hf auth login
with a read token and re-download.## Next steps
Serve your quantized model over an OpenAI-compatible API with ./build/bin/llama-server -m Qwen3-0.6B-Q4_K_M.gguf
, then point any OpenAI client at localhost:8080
. To squeeze quality out of aggressive quants (Q3 and below), generate an importance matrix with llama-imatrix
on a calibration text file and pass it to llama-quantize --imatrix
. For sharing, hf upload
pushes your GGUF to a Hub repo so others can pull it with llama cli -hf you/your-model-GGUF
. And if you want quants without any local setup, the GGUF-my-repo space runs this exact pipeline in your browser.
Sources & further reading #
llama.cpp quantize tool documentation— github.com - llama.cpp build guide— github.com - llama-cli reference— github.com - Hugging Face hf CLI guide— huggingface.co - Qwen3-0.6B model card— huggingface.co
Mariana Souza· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.