Alibaba Ships Qwen3-8B as Ready-to-Run Local AI With Dual Reasoning Modes

Alibaba's Qwen team published official quantized GGUF builds of Qwen3-8B on Hugging Face, offering Q4_K_M (5.03 GB) and Q8_0 (8.71 GB) files that run locally on llama.cpp, Ollama, LM Studio and vLLM under an Apache 2.0 license. The 8.2B-parameter dense model has 36 layers, grouped-query attention at 32/8, a 32K native context extendable to 131K with YaRN, and switches between extended reasoning and direct answers via enable_thinking=true or /think. The Qwen team reports the model outperforms QwQ and Qwen2.5-Instruct on math, code and commonsense reasoning, and advises avoiding greedy decoding with a presence_penalty of 1.5 on quantized builds to stop repetition.

Alibaba Ships Qwen3-8B as Ready-to-Run Local AI With Dual Reasoning Modes Alibaba's compact 8B reasoning model ships in quantized GGUF formats, offering hybrid thinking mode, 131K context, and local inference on modest hardware. - Qwen3-8B released as GGUF quantizations Q4 K M through Q8 0 for local inference on Hugging Face https://huggingface.co/Qwen/Qwen3-8B-GGUF?utm source=alphasignal . - Hybrid thinking: single checkpoint switches between reasoning and fast chat via /think and /no think. - 8.2B dense params, 36 layers, GQA 32/8, 32K native context extendable to 131K with YaRN. - Outperforms QwQ and Qwen2.5-Instruct on math, code, and commonsense reasoning per the technical report https://arxiv.org/abs/2505.09388?utm source=alphasignal . - Runs out of the box on llama.cpp, Ollama, LM Studio, vLLM; Apache 2.0 license allows commercial use. - Avoid greedy decoding; use presence penalty 1.5 on quantized builds to stop repetition. Alibaba’s Qwen team has published official quantized builds of Qwen3-8B on the model page https://huggingface.co/Qwen/Qwen3-8B-GGUF?utm source=alphasignal . The repository provides Q4 K M and Q8 0 GGUF files for llama.cpp, recent Ollama releases, LM Studio, and compatible runtimes. Developers can run the same checkpoint in an extended reasoning mode or a faster direct-answer mode without converting the original weights. GGUF removes a conversion step GGUF packages model weights and runtime metadata in a format designed for local inference. Quantization stores weights at reduced precision, lowering storage and memory requirements while introducing a potential quality tradeoff. | Build | Approximate size | Practical tradeoff | |---|---|---| | Q4 K M | 5.03 GB | Lower memory use and faster loading; suitable as a local default | | Q8 0 | 8.71 GB | Higher weight precision with greater RAM and VRAM requirements | Total runtime memory exceeds the file size because inference also allocates a key-value cache, compute buffers, and prompt state. With a default 16-bit KV cache, a 131,072-token window for this architecture can consume roughly 18 GiB for the cache alone, before weights and other buffers. KV-cache quantization and partial CPU offload can reduce VRAM pressure, with corresponding speed or quality costs. Reasoning becomes a request-level switch Qwen3-8B supports thinking and non-thinking behavior within one checkpoint. Thinking mode emits intermediate reasoning inside <think blocks before the final response, while non-thinking mode answers directly with lower token use and latency. | Mode | Behavior | Control | |---|---|---| | Thinking | Generates an extended reasoning trace before the answer | enable thinking=true or /think | | Non-thinking | Returns a direct answer | enable thinking=false | This story is for Pro members You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.