# Unsloth - Qwen3.8 - How to Run Locally

> Source: <https://unsloth.ai/docs/models/qwen3.8>
> Published: 2026-08-14 15:03:16+00:00

# Qwen3.8 - How to Run Locally

Guide to running Qwen3.8 quants including Qwen3.8-27B on your local setup.

Qwen3.8 is Qwen’s new model family, featuring Qwen3.8-**27B**, Qwen3.8-** 2.4T-A95B** and Qwen3.8-**Max**. Qwen3.8-27B has **vision** and reasoning capabilities, a **256K context** window, and runs locally on **17GB RAM/VRAM** setups. Qwen3.8 excels at agentic coding, vision and chat tasks, and can now run via Unsloth GGUFs, NVFP4 and [Unsloth Desktop](/docs/models/qwen3.8#run-qwen3.8-in-unsloth-desktop). Qwen3.8-2.4T-A95B is a 2.4T parameter (95B active) model with rivaling GPT-5.6 Sol.

Qwen3.8 GGUFs use Unsloth Dynamic V3.0 (preview) for SOTA quant performance. Thank you Qwen for day zero access. Unsloth quants also include:

**Developer Role Support** for agentic tools like Codex**Tool calling:** Improved parsing nested objects to make tools succeed more

Full-precision Qwen3.8-2.4T-A95B requires 4.9TB of storage and 1-bit [Unsloth](https://github.com/unslothai/unsloth) Dynamic GGUFs takes **397GB (91% smaller)**, and larger IQ1_S takes 508GB.

### ⚙️ Usage Guide

#### Qwen3.8-27B Requirements:

Qwen3.8-**27B** 4-bit quants will work on most devices 17-19GB VRAM like RTX 5080, 4090 or a Mac with 24GB RAM.
**Table: Hardware requirements** (units = total memory: RAM + VRAM, or unified memory)

11-13 GB

13-16 GB

17-19 GB

24 GB

31 GB

56 GB

#### Qwen3.8-**2.4T** Requirements:

397GB

508GB

657 GB

2.6 TB

4.9 TB

### Recommended Settings

#### Qwen3.8-**27B Settings:**

Qwen3.8-27B is a** hybrid thinking** model with different default settings for thinking and non-thinking modes:

`temperature`

1.0

0.7

`top_p`

0.95

0.80

`top_k`

20

20

`min_p`

0.0

0.0

`presence_penalty`

0.0

1.5

`repetition_penalty`

1.0

1.0

**Maximum context window:**`262,144`

(can be extended to 1M via YaRN)Thinking Mode:

`temperature=1.0`

,`top_p=0.95`

,`top_k=20`

,`min_p=0.0`

,`presence_penalty=0.0`

,`repetition_penalty=1.0`

Instruct (or non-thinking) mode:

`temperature=0.7`

,`top_p=0.80`

,`top_k=20`

,`min_p=0.0`

,`presence_penalty=1.5`

,`repetition_penalty=1.0`

#### Qwen3.8-**2.4T Settings:**

Qwen3.8-2.4T is **thinking-only**, while Qwen3.8-Max is hybrid.

temperature = 1.0

top_p = 0.95

top_k = 20

min_p = 0.0

presence_penalty = 0.0

Context length = up to

`1,010,000`

`temperature=1.0`

,`top_p=0.95`

,`top_k=20`

,`min_p=0.0`

,`presence_penalty=0.0`

,`repetition_penalty=1.0`

If the model fits, you will get ~20 tokens/s generation when using B200s and >120 tokens / s throughput. Best rule of thumb: RAM+VRAM ≈ the quant size; otherwise it’ll still work, just much slower due to disk offloading.

### 💡 Thinking + Preserve Thinking

Qwen3.8 has **Preserve Thinking** which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. [Unsloth](/docs/models/qwen3.8#run-qwen3.8-in-unsloth-desktop) has 'Think' and Preserved Thinking toggles for Qwen3.8.

Qwen3.8-27B comes with support for `reasoning_effort`

, which can be used to adjust reasoning depth and control cost. These toggles are automatically enabled in Unsloth:

`xhigh`

(default): for complex tasks demanding thorough analysis`medium`

: balancing accuracy and speed`low`

: efficient reasoning optimizing for speed and costnone

## Run Qwen3.8 Guide

You can now run Qwen3.8 in llama.cpp and Unsloth Desktop. For the large Qwen3.8-2.T model, we will be utilizing the 397GB `IQ1_XXXS`

quant (named Q1_0) for best results in terms of accessibility and accuracy and it will require at least 450GB RAM. Feel free to change quantization type. GGUF: **Qwen3.8-GGUF**

[Run in Unsloth Desktop](/docs/models/qwen3.8#run-qwen3.8-in-unsloth-desktop)[Run in llama.cpp](/docs/models/qwen3.8#run-qwen3.8-in-llama.cpp)[NVFP4 Guide](/docs/models/qwen3.8#run-qwen3.8-in-llama.cpp)

### 🦥 Run Qwen3.8 in Unsloth Desktop

Qwen3.8 can run in [Unsloth Desktop](/docs/models/qwen3.8#run-qwen3.8-in-unsloth-desktop), an open-source UI app for local AI. **Unsloth automatically offloads to RAM and detects multiGPU setups**. With Unsloth Desktop, you can run models locally on **MacOS, Windows**, Linux and:

Search, download,

[run GGUFs](/docs/new/studio#run-models-locally)and safetensor models+**Self-healing** tool calling**web search**(Python, Bash)** Code execution**[Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive)parameter tuning (temp, top-p, etc.)Fast CPU + GPU inference via MLX and llama.cpp

[Train LLMs](/docs/new/studio#no-code-training)2x faster with 70% less VRAM

#### Install Unsloth

The easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).

Or, if you prefer to install manually:

MacOS, Linux, WSL:

Windows PowerShell:

#### Search and download Qwen3.8

Go to [Unsloth Chat](/docs/new/studio/chat) or Model hub and search for Qwen3.8 in the search bar and download your desired model and quant.

#### Run Qwen3.8

Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.

For more information, you can view our [Unsloth inference guide](/docs/new/studio/chat).

For example using Unsloth Desktop with the 397GB Qwen3.8 (-91% smaller) allows you to toggle thinking modes, allow inline canvas, web search and code execution and much more.

### Qwen3.8-2.4T-A95B New 1-bit data-types

We extended IQ1_S in llama.cpp which is 1.5625 bits per weight to 1.1875 bpw by reducing the number of entries in the codebook - we found this works well for large models, and can still retain a lot of accuracy - we also found these new data-types to be fine for post training quantization (PTQ) without the need for QAT or QAD (quantization aware training / distillation)

Due to naming issues, we used TQ2_0, TQ1_0 and Q1_0 otherwise it won't pop up in the HF repo.

IQ1_S

IQ1_S

**1.5625**

2048

11

50 B

UD-IQ1_XS

TQ2_0

1.4375

1024

10

46 B

UD-IQ1_XXS

TQ1_0

1.3125

512

9

42 B

UD-IQ1_XXXS

Q1_0

**1.1875**

256

8

38 B

We are still running benchmarks for the new data-types, but for other large models, we get **good results without any QAT / QAD**:

IQ1_S

553.204

2.578876

0.564553

78.882

UD-IQ1_XS

513.583

2.931261

0.690161

75.726

UD-IQ1_XXS

473.961

3.540383

0.876007

71.284

UD-IQ1_XXXS

434.340

4.488796

1.109944

66.257

### 🦙 Run Qwen3.8 in llama.cpp

We need to use the specific IQ1_XXXS branch [here](https://github.com/unslothai/llama.cpp/pull/61). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON`

to `-DGGML_CUDA=OFF`

if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF`

then continue as usual - Metal support is on by default.

If you just want to run the standard `IQ1_S`

and other quants, then compile llama.cpp normally:

Download the model via (after installing `pip install huggingface_hub`

). You can choose `Q1_0`

for `IQ1_XXXS`

or other quantized versions like `Q8_0`

. If downloads get stuck, see: [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)

**Qwen3.8-27B:**

**Qwen3.8-2.4T:**

To run the model in llama-cli, follow the code snippets below:
Remember to [change settings](/docs/models/qwen3.8#recommended-settings) according to your use-case.

**Qwen3.8-27B:**

**Qwen3.8-2.4T:**

To run the general UD-IQ1_S you can do:

**Qwen3.8-27B:**

**Qwen3.8-2.4T:**

Then to run it:

### ⚡️NVFP4

Like Qwen3.6, we’re also releasing new [dynamic NVFP4 Qwen3.8](/docs/basics/nvfp4)-27B quants that run **~1.5× faster** than BF16 checkpoints, with **better performance** and comparable file sizes. Run Qwen3.8-27B NVFP4** 1.5x faster **on **24GB VRAM. **We also added **FP8 KV cache calibration** for 2x longer context lengths! NVFP4 requires NVIDIA's Blackwell GPUs like RTX 50X, DGX Spark (see [Qwen3.8](/docs/models/qwen3.8#dgx-spark-with-nvfp4-quants)), B200, B300 GPUs. For older GPUs, our GGUFs work well! You can run NVFP4 quants in [vLLM](/docs/models/qwen3.8#vllm) and [SGLang](/docs/models/qwen3.8#sglang).

[Qwen3.8-27B-](https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4)**NVFP4** quant

1

89.8

**133.7**

1.49x

89.8

**133.7**

8

649.4

**938.8**

1.45x

81.2

**117.3**

32

1983.0

**2787.0**

1.41x

62.0

**87.1**

64

3048.5

**4407.2**

1.45x

47.6

**68.9**

See below for previous benchmarks conducted for Qwen3.6 as well comparing to other NVFP4 implementations which use 16bit activations vs our NVFP4 activations:

All benchmarks use 1x B200 128 concurrency. Higher concurrency can boost 35B to 17,561 tokens / s.

For accuracy benchmarks we ran KLD and Top-1% agreement over Code, Chat and many domains. NVFP4 is consistency 92% to 97% accuracy recovery vs BF16

zh

0.01628

93.55%

code

0.02600

96.68%

refgen

0.03993

94.46%

chat

0.05818

92.15%

ja / ko / ru / es

0.0124-0.0155

94-95%

For accuracy benchmarks For Qwen 3.6, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all:

For more information, you can read our [Dynamic NVFP4 quants blog](/docs/basics/nvfp4).

To run NVFP4 quants, see below for commands to run Qwen3.8-27B in [vLLM](/docs/basics/inference-and-deployment/vllm-guide) or [SGLang](/docs/basics/inference-and-deployment/sglang-guide):

**vLLM:**

To install vLLM in a separate venv:

Then to serve the 27B variant:

To enable MTP / speculative decoding (faster decode but somewhat less throughput), use:

If you get Torchcodec issues, be sure to do the below then relaunch vllm.

**SGLang:**

Install SGLang then run:

### 📊 Benchmarks

#### Qwen3.8-**27B**

#### Qwen3.8-**2.4T-A95B**

Last updated

Was this helpful?