Unsloth - Qwen3.8 - How to Run Locally

Unsloth released dynamic GGUF quantizations for Qwen3.8, enabling the 27B model to run locally on 17-19GB VRAM setups and the 2.4T parameter model to run in 397GB via 1-bit quantization. The Qwen3.8 family includes 27B, 2.4T-A95B, and Max variants, with the 27B offering vision, reasoning, and a 256K context window. Unsloth's quants support developer role and improved tool calling, and the models can be run via Unsloth Desktop or llama.cpp.

Qwen3.8 - How to Run Locally Guide to running Qwen3.8 quants including Qwen3.8-27B on your local setup. Qwen3.8 is Qwen’s new model family, featuring Qwen3.8- 27B , Qwen3.8- 2.4T-A95B and Qwen3.8- Max . Qwen3.8-27B has vision and reasoning capabilities, a 256K context window, and runs locally on 17GB RAM/VRAM setups. Qwen3.8 excels at agentic coding, vision and chat tasks, and can now run via Unsloth GGUFs, NVFP4 and Unsloth Desktop /docs/models/qwen3.8 run-qwen3.8-in-unsloth-desktop . Qwen3.8-2.4T-A95B is a 2.4T parameter 95B active model with rivaling GPT-5.6 Sol. Qwen3.8 GGUFs use Unsloth Dynamic V3.0 preview for SOTA quant performance. Thank you Qwen for day zero access. Unsloth quants also include: Developer Role Support for agentic tools like Codex Tool calling: Improved parsing nested objects to make tools succeed more Full-precision Qwen3.8-2.4T-A95B requires 4.9TB of storage and 1-bit Unsloth https://github.com/unslothai/unsloth Dynamic GGUFs takes 397GB 91% smaller , and larger IQ1 S takes 508GB. ⚙️ Usage Guide Qwen3.8-27B Requirements: Qwen3.8- 27B 4-bit quants will work on most devices 17-19GB VRAM like RTX 5080, 4090 or a Mac with 24GB RAM. Table: Hardware requirements units = total memory: RAM + VRAM, or unified memory 11-13 GB 13-16 GB 17-19 GB 24 GB 31 GB 56 GB Qwen3.8- 2.4T Requirements: 397GB 508GB 657 GB 2.6 TB 4.9 TB Recommended Settings Qwen3.8- 27B Settings: Qwen3.8-27B is a hybrid thinking model with different default settings for thinking and non-thinking modes: temperature 1.0 0.7 top p 0.95 0.80 top k 20 20 min p 0.0 0.0 presence penalty 0.0 1.5 repetition penalty 1.0 1.0 Maximum context window: 262,144 can be extended to 1M via YaRN Thinking Mode: temperature=1.0 , top p=0.95 , top k=20 , min p=0.0 , presence penalty=0.0 , repetition penalty=1.0 Instruct or non-thinking mode: temperature=0.7 , top p=0.80 , top k=20 , min p=0.0 , presence penalty=1.5 , repetition penalty=1.0 Qwen3.8- 2.4T Settings: Qwen3.8-2.4T is thinking-only , while Qwen3.8-Max is hybrid. temperature = 1.0 top p = 0.95 top k = 20 min p = 0.0 presence penalty = 0.0 Context length = up to 1,010,000 temperature=1.0 , top p=0.95 , top k=20 , min p=0.0 , presence penalty=0.0 , repetition penalty=1.0 If the model fits, you will get ~20 tokens/s generation when using B200s and 120 tokens / s throughput. Best rule of thumb: RAM+VRAM ≈ the quant size; otherwise it’ll still work, just much slower due to disk offloading. 💡 Thinking + Preserve Thinking Qwen3.8 has Preserve Thinking which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. Unsloth /docs/models/qwen3.8 run-qwen3.8-in-unsloth-desktop has 'Think' and Preserved Thinking toggles for Qwen3.8. Qwen3.8-27B comes with support for reasoning effort , which can be used to adjust reasoning depth and control cost. These toggles are automatically enabled in Unsloth: xhigh default : for complex tasks demanding thorough analysis medium : balancing accuracy and speed low : efficient reasoning optimizing for speed and costnone Run Qwen3.8 Guide You can now run Qwen3.8 in llama.cpp and Unsloth Desktop. For the large Qwen3.8-2.T model, we will be utilizing the 397GB IQ1 XXXS quant named Q1 0 for best results in terms of accessibility and accuracy and it will require at least 450GB RAM. Feel free to change quantization type. GGUF: Qwen3.8-GGUF Run in Unsloth Desktop /docs/models/qwen3.8 run-qwen3.8-in-unsloth-desktop Run in llama.cpp /docs/models/qwen3.8 run-qwen3.8-in-llama.cpp NVFP4 Guide /docs/models/qwen3.8 run-qwen3.8-in-llama.cpp 🦥 Run Qwen3.8 in Unsloth Desktop Qwen3.8 can run in Unsloth Desktop /docs/models/qwen3.8 run-qwen3.8-in-unsloth-desktop , an open-source UI app for local AI. Unsloth automatically offloads to RAM and detects multiGPU setups . With Unsloth Desktop, you can run models locally on MacOS, Windows , Linux and: Search, download, run GGUFs /docs/new/studio run-models-locally and safetensor models+ Self-healing tool calling web search Python, Bash Code execution Automatic inference https://unsloth.ai/docs/desktop feature-deep-dive parameter tuning temp, top-p, etc. Fast CPU + GPU inference via MLX and llama.cpp Train LLMs /docs/new/studio no-code-training 2x faster with 70% less VRAM Install Unsloth The easiest way to get started is by downloading the Unsloth Desktop app /docs/desktop . Works on macOS /docs/get-started/install/mac , Windows /docs/get-started/install/windows-installation , and Linux /docs/get-started/install/linux . Or, if you prefer to install manually: MacOS, Linux, WSL: Windows PowerShell: Search and download Qwen3.8 Go to Unsloth Chat /docs/new/studio/chat or Model hub and search for Qwen3.8 in the search bar and download your desired model and quant. Run Qwen3.8 Inference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings. For more information, you can view our Unsloth inference guide /docs/new/studio/chat . For example using Unsloth Desktop with the 397GB Qwen3.8 -91% smaller allows you to toggle thinking modes, allow inline canvas, web search and code execution and much more. Qwen3.8-2.4T-A95B New 1-bit data-types We extended IQ1 S in llama.cpp which is 1.5625 bits per weight to 1.1875 bpw by reducing the number of entries in the codebook - we found this works well for large models, and can still retain a lot of accuracy - we also found these new data-types to be fine for post training quantization PTQ without the need for QAT or QAD quantization aware training / distillation Due to naming issues, we used TQ2 0, TQ1 0 and Q1 0 otherwise it won't pop up in the HF repo. IQ1 S IQ1 S 1.5625 2048 11 50 B UD-IQ1 XS TQ2 0 1.4375 1024 10 46 B UD-IQ1 XXS TQ1 0 1.3125 512 9 42 B UD-IQ1 XXXS Q1 0 1.1875 256 8 38 B We are still running benchmarks for the new data-types, but for other large models, we get good results without any QAT / QAD : IQ1 S 553.204 2.578876 0.564553 78.882 UD-IQ1 XS 513.583 2.931261 0.690161 75.726 UD-IQ1 XXS 473.961 3.540383 0.876007 71.284 UD-IQ1 XXXS 434.340 4.488796 1.109944 66.257 🦙 Run Qwen3.8 in llama.cpp We need to use the specific IQ1 XXXS branch here https://github.com/unslothai/llama.cpp/pull/61 . You can follow the build instructions below as well. Change -DGGML CUDA=ON to -DGGML CUDA=OFF if you don't have a GPU or just want CPU inference. For Apple Mac / Metal devices , set -DGGML CUDA=OFF then continue as usual - Metal support is on by default. If you just want to run the standard IQ1 S and other quants, then compile llama.cpp normally: Download the model via after installing pip install huggingface hub . You can choose Q1 0 for IQ1 XXXS or other quantized versions like Q8 0 . If downloads get stuck, see: Hugging Face Hub, XET debugging /docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging Qwen3.8-27B: Qwen3.8-2.4T: To run the model in llama-cli, follow the code snippets below: Remember to change settings /docs/models/qwen3.8 recommended-settings according to your use-case. Qwen3.8-27B: Qwen3.8-2.4T: To run the general UD-IQ1 S you can do: Qwen3.8-27B: Qwen3.8-2.4T: Then to run it: ⚡️NVFP4 Like Qwen3.6, we’re also releasing new dynamic NVFP4 Qwen3.8 /docs/basics/nvfp4 -27B quants that run ~1.5× faster than BF16 checkpoints, with better performance and comparable file sizes. Run Qwen3.8-27B NVFP4 1.5x faster on 24GB VRAM. We also added FP8 KV cache calibration for 2x longer context lengths NVFP4 requires NVIDIA's Blackwell GPUs like RTX 50X, DGX Spark see Qwen3.8 /docs/models/qwen3.8 dgx-spark-with-nvfp4-quants , B200, B300 GPUs. For older GPUs, our GGUFs work well You can run NVFP4 quants in vLLM /docs/models/qwen3.8 vllm and SGLang /docs/models/qwen3.8 sglang . Qwen3.8-27B- https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4 NVFP4 quant 1 89.8 133.7 1.49x 89.8 133.7 8 649.4 938.8 1.45x 81.2 117.3 32 1983.0 2787.0 1.41x 62.0 87.1 64 3048.5 4407.2 1.45x 47.6 68.9 See below for previous benchmarks conducted for Qwen3.6 as well comparing to other NVFP4 implementations which use 16bit activations vs our NVFP4 activations: All benchmarks use 1x B200 128 concurrency. Higher concurrency can boost 35B to 17,561 tokens / s. For accuracy benchmarks we ran KLD and Top-1% agreement over Code, Chat and many domains. NVFP4 is consistency 92% to 97% accuracy recovery vs BF16 zh 0.01628 93.55% code 0.02600 96.68% refgen 0.03993 94.46% chat 0.05818 92.15% ja / ko / ru / es 0.0124-0.0155 94-95% For accuracy benchmarks For Qwen 3.6, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all: For more information, you can read our Dynamic NVFP4 quants blog /docs/basics/nvfp4 . To run NVFP4 quants, see below for commands to run Qwen3.8-27B in vLLM /docs/basics/inference-and-deployment/vllm-guide or SGLang /docs/basics/inference-and-deployment/sglang-guide : vLLM: To install vLLM in a separate venv: Then to serve the 27B variant: To enable MTP / speculative decoding faster decode but somewhat less throughput , use: If you get Torchcodec issues, be sure to do the below then relaunch vllm. SGLang: Install SGLang then run: 📊 Benchmarks Qwen3.8- 27B Qwen3.8- 2.4T-A95B Last updated Was this helpful?