Guide to running Qwen3.8 quants including Qwen3.8-27B on your local setup.
Qwen3.8 is Qwen’s new model family, featuring Qwen3.8-27B, Qwen3.8-** 2.4T-A95B** and Qwen3.8-Max. Qwen3.8-27B has vision and reasoning capabilities, a 256K context window, and runs locally on 17GB RAM/VRAM setups. Qwen3.8 excels at agentic coding, vision and chat tasks, and can now run via Unsloth GGUFs, NVFP4 and Unsloth Desktop. Qwen3.8-2.4T-A95B is a 2.4T parameter (95B active) model with rivaling GPT-5.6 Sol.
Qwen3.8 GGUFs use Unsloth Dynamic V3.0 (preview) for SOTA quant performance. Thank you Qwen for day zero access. Unsloth quants also include:
Developer Role Support for agentic tools like CodexTool calling: Improved parsing nested objects to make tools succeed more
Full-precision Qwen3.8-2.4T-A95B requires 4.9TB of storage and 1-bit Unsloth Dynamic GGUFs takes 397GB (91% smaller), and larger IQ1_S takes 508GB.
⚙️ Usage Guide
Qwen3.8-27B Requirements:
Qwen3.8-27B 4-bit quants will work on most devices 17-19GB VRAM like RTX 5080, 4090 or a Mac with 24GB RAM.
Table: Hardware requirements (units = total memory: RAM + VRAM, or unified memory) 11-13 GB
13-16 GB
17-19 GB
24 GB
31 GB
56 GB
Qwen3.8-2.4T Requirements:
397GB
508GB
657 GB
2.6 TB
4.9 TB
Recommended Settings
Qwen3.8-27B Settings:
Qwen3.8-27B is a** hybrid thinking** model with different default settings for thinking and non-thinking modes:
temperature
1.0
0.7
top_p
0.95
0.80
top_k
20
20
min_p
0.0
0.0
presence_penalty
0.0
1.5
repetition_penalty
1.0
1.0
Maximum context window:262,144
(can be extended to 1M via YaRN)Thinking Mode:
temperature=1.0
,top_p=0.95
,top_k=20
,min_p=0.0
,presence_penalty=0.0
,repetition_penalty=1.0
Instruct (or non-thinking) mode:
temperature=0.7
,top_p=0.80
,top_k=20
,min_p=0.0
,presence_penalty=1.5
,repetition_penalty=1.0
Qwen3.8-2.4T Settings:
Qwen3.8-2.4T is thinking-only, while Qwen3.8-Max is hybrid.
temperature = 1.0
top_p = 0.95
top_k = 20
min_p = 0.0
presence_penalty = 0.0
Context length = up to
1,010,000
temperature=1.0
,top_p=0.95
,top_k=20
,min_p=0.0
,presence_penalty=0.0
,repetition_penalty=1.0
If the model fits, you will get ~20 tokens/s generation when using B200s and >120 tokens / s throughput. Best rule of thumb: RAM+VRAM ≈ the quant size; otherwise it’ll still work, just much slower due to disk off.
💡 Thinking + Preserve Thinking
Qwen3.8 has Preserve Thinking which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. Unsloth has 'Think' and Preserved Thinking toggles for Qwen3.8.
Qwen3.8-27B comes with support for reasoning_effort
, which can be used to adjust reasoning depth and control cost. These toggles are automatically enabled in Unsloth:
xhigh
(default): for complex tasks demanding thorough analysismedium
: balancing accuracy and speedlow
: efficient reasoning optimizing for speed and costnone
Run Qwen3.8 Guide #
You can now run Qwen3.8 in llama.cpp and Unsloth Desktop. For the large Qwen3.8-2.T model, we will be utilizing the 397GB IQ1_XXXS
quant (named Q1_0) for best results in terms of accessibility and accuracy and it will require at least 450GB RAM. Feel free to change quantization type. GGUF: Qwen3.8-GGUF
Run in Unsloth DesktopRun in llama.cppNVFP4 Guide
🦥 Run Qwen3.8 in Unsloth Desktop
Qwen3.8 can run in Unsloth Desktop, an open-source UI app for local AI. Unsloth automatically offloads to RAM and detects multiGPU setups. With Unsloth Desktop, you can run models locally on MacOS, Windows, Linux and:
Search, download,
run GGUFsand safetensor models+Self-healing tool callingweb search(Python, Bash)** Code execution**Automatic inferenceparameter tuning (temp, top-p, etc.)Fast CPU + GPU inference via MLX and llama.cpp
Train LLMs2x faster with 70% less VRAM
Install Unsloth
The easiest way to get started is by down the Unsloth Desktop app. Works on macOS, Windows, and Linux.
Or, if you prefer to install manually:
MacOS, Linux, WSL:
Windows PowerShell:
Search and download Qwen3.8
Go to Unsloth Chat or Model hub and search for Qwen3.8 in the search bar and download your desired model and quant.
Run Qwen3.8
Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.
For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat).
For example using Unsloth Desktop with the 397GB Qwen3.8 (-91% smaller) allows you to toggle thinking modes, allow inline canvas, web search and code execution and much more.
### Qwen3.8-2.4T-A95B New 1-bit data-types
We extended IQ1_S in llama.cpp which is 1.5625 bits per weight to 1.1875 bpw by reducing the number of entries in the codebook - we found this works well for large models, and can still retain a lot of accuracy - we also found these new data-types to be fine for post training quantization (PTQ) without the need for QAT or QAD (quantization aware training / distillation)
Due to naming issues, we used TQ2_0, TQ1_0 and Q1_0 otherwise it won't pop up in the HF repo.
IQ1_S
IQ1_S
1.5625
2048
11
50 B
UD-IQ1_XS
TQ2_0
1.4375
1024
10
46 B
UD-IQ1_XXS
TQ1_0
1.3125
512
9
42 B
UD-IQ1_XXXS
Q1_0
1.1875
256
8
38 B
We are still running benchmarks for the new data-types, but for other large models, we get good results without any QAT / QAD:
IQ1_S
553.204
2.578876
0.564553
78.882
UD-IQ1_XS
513.583
2.931261
0.690161
75.726
UD-IQ1_XXS
473.961
3.540383
0.876007
71.284
UD-IQ1_XXXS
434.340
4.488796
1.109944
66.257
🦙 Run Qwen3.8 in llama.cpp
We need to use the specific IQ1_XXXS branch here. You can follow the build instructions below as well. Change -DGGML_CUDA=ON
to `-DGGML_CUDA=OFF`
if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF`
then continue as usual - Metal support is on by default.
If you just want to run the standard IQ1_S
and other quants, then compile llama.cpp normally:
Download the model via (after installing pip install huggingface_hub
). You can choose Q1_0
for IQ1_XXXS
or other quantized versions like Q8_0
. If downloads get stuck, see: [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)
**Qwen3.8-27B:**
**Qwen3.8-2.4T:**
To run the model in llama-cli, follow the code snippets below:
Remember to [change settings](/docs/models/qwen3.8#recommended-settings) according to your use-case.
**Qwen3.8-27B:**
**Qwen3.8-2.4T:**
To run the general UD-IQ1_S you can do:
**Qwen3.8-27B:**
**Qwen3.8-2.4T:**
Then to run it:
⚡️NVFP4
Like Qwen3.6, we’re also releasing new dynamic NVFP4 Qwen3.8-27B quants that run ~1.5× faster than BF16 checkpoints, with better performance and comparable file sizes. Run Qwen3.8-27B NVFP4** 1.5x faster **on **24GB VRAM. **We also added FP8 KV cache calibration for 2x longer context lengths! NVFP4 requires NVIDIA's Blackwell GPUs like RTX 50X, DGX Spark (see Qwen3.8), B200, B300 GPUs. For older GPUs, our GGUFs work well! You can run NVFP4 quants in vLLM and SGLang.
Qwen3.8-27B-NVFP4 quant 1
89.8
133.7
1.49x
89.8
133.7
8
649.4
938.8
1.45x
81.2
117.3
32
1983.0
2787.0
1.41x
62.0
87.1
64
3048.5
4407.2
1.45x
47.6
68.9
See below for previous benchmarks conducted for Qwen3.6 as well comparing to other NVFP4 implementations which use 16bit activations vs our NVFP4 activations:
All benchmarks use 1x B200 128 concurrency. Higher concurrency can boost 35B to 17,561 tokens / s.
For accuracy benchmarks we ran KLD and Top-1% agreement over Code, Chat and many domains. NVFP4 is consistency 92% to 97% accuracy recovery vs BF16 zh
0.01628
93.55%
code
0.02600
96.68%
refgen
0.03993
94.46%
chat
0.05818
92.15%
ja / ko / ru / es
0.0124-0.0155
94-95%
For accuracy benchmarks For Qwen 3.6, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all:
For more information, you can read our [Dynamic NVFP4 quants blog](/docs/basics/nvfp4).
To run NVFP4 quants, see below for commands to run Qwen3.8-27B in [vLLM](/docs/basics/inference-and-deployment/vllm-guide) or [SGLang](/docs/basics/inference-and-deployment/sglang-guide):
vLLM:
To install vLLM in a separate venv:
Then to serve the 27B variant:
To enable MTP / speculative decoding (faster decode but somewhat less throughput), use:
If you get Torchcodec issues, be sure to do the below then relaunch vllm. SGLang:
Install SGLang then run:
📊 Benchmarks
Qwen3.8-27B
Qwen3.8-2.4T-A95B
Last updated
Was this helpful?