Alibaba's compact 8B reasoning model ships in quantized GGUF formats, offering hybrid thinking mode, 131K context, and local inference on modest hardware.
- Qwen3-8B released as GGUF quantizations (Q4_K_M through Q8_0) for local inference on Hugging Face .
- Hybrid thinking: single checkpoint switches between reasoning and fast chat via /think and /no_think.
- 8.2B dense params, 36 layers, GQA 32/8, 32K native context extendable to 131K with YaRN.
- Outperforms QwQ and Qwen2.5-Instruct on math, code, and commonsense reasoning per the technical report .
- Runs out of the box on llama.cpp, Ollama, LM Studio, vLLM; Apache 2.0 license allows commercial use.
- Avoid greedy decoding; use presence_penalty 1.5 on quantized builds to stop repetition.
Alibaba’s Qwen team has published official quantized builds of Qwen3-8B on the model page. The repository provides Q4_K_M and Q8_0 GGUF files for llama.cpp, recent Ollama releases, LM Studio, and compatible runtimes. Developers can run the same checkpoint in an extended reasoning mode or a faster direct-answer mode without converting the original weights.
GGUF removes a conversion step #
GGUF packages model weights and runtime metadata in a format designed for local inference. Quantization stores weights at reduced precision, lowering storage and memory requirements while introducing a potential quality tradeoff.
| Build | Approximate size | Practical tradeoff |
|---|---|---|
Q4_K_M |
5.03 GB | Lower memory use and faster ; suitable as a local default |
Q8_0 |
8.71 GB | Higher weight precision with greater RAM and VRAM requirements |
Total runtime memory exceeds the file size because inference also allocates a key-value cache, compute buffers, and prompt state. With a default 16-bit KV cache, a 131,072-token window for this architecture can consume roughly 18 GiB for the cache alone, before weights and other buffers. KV-cache quantization and partial CPU offload can reduce VRAM pressure, with corresponding speed or quality costs.
Reasoning becomes a request-level switch #
Qwen3-8B supports thinking and non-thinking behavior within one checkpoint. Thinking mode emits intermediate reasoning inside <think> blocks before the final response, while non-thinking mode answers directly with lower token use and latency.
| Mode | Behavior | Control |
|---|---|---|
| Thinking | Generates an extended reasoning trace before the answer | enable_thinking=true or/think |
| Non-thinking | Returns a direct answer | enable_thinking=false |
This story is for Pro members #
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.