{"slug": "unsloth-qwen3-8-how-to-run-locally", "title": "Unsloth - Qwen3.8 - How to Run Locally", "summary": "Unsloth released dynamic GGUF quantizations for Qwen3.8, enabling the 27B model to run locally on 17-19GB VRAM setups and the 2.4T parameter model to run in 397GB via 1-bit quantization. The Qwen3.8 family includes 27B, 2.4T-A95B, and Max variants, with the 27B offering vision, reasoning, and a 256K context window. Unsloth's quants support developer role and improved tool calling, and the models can be run via Unsloth Desktop or llama.cpp.", "body_md": "# Qwen3.8 - How to Run Locally\n\nGuide to running Qwen3.8 quants including Qwen3.8-27B on your local setup.\n\nQwen3.8 is Qwen’s new model family, featuring Qwen3.8-**27B**, Qwen3.8-** 2.4T-A95B** and Qwen3.8-**Max**. Qwen3.8-27B has **vision** and reasoning capabilities, a **256K context** window, and runs locally on **17GB RAM/VRAM** setups. Qwen3.8 excels at agentic coding, vision and chat tasks, and can now run via Unsloth GGUFs, NVFP4 and [Unsloth Desktop](/docs/models/qwen3.8#run-qwen3.8-in-unsloth-desktop). Qwen3.8-2.4T-A95B is a 2.4T parameter (95B active) model with rivaling GPT-5.6 Sol.\n\nQwen3.8 GGUFs use Unsloth Dynamic V3.0 (preview) for SOTA quant performance. Thank you Qwen for day zero access. Unsloth quants also include:\n\n**Developer Role Support** for agentic tools like Codex**Tool calling:** Improved parsing nested objects to make tools succeed more\n\nFull-precision Qwen3.8-2.4T-A95B requires 4.9TB of storage and 1-bit [Unsloth](https://github.com/unslothai/unsloth) Dynamic GGUFs takes **397GB (91% smaller)**, and larger IQ1_S takes 508GB.\n\n### ⚙️ Usage Guide\n\n#### Qwen3.8-27B Requirements:\n\nQwen3.8-**27B** 4-bit quants will work on most devices 17-19GB VRAM like RTX 5080, 4090 or a Mac with 24GB RAM.\n**Table: Hardware requirements** (units = total memory: RAM + VRAM, or unified memory)\n\n11-13 GB\n\n13-16 GB\n\n17-19 GB\n\n24 GB\n\n31 GB\n\n56 GB\n\n#### Qwen3.8-**2.4T** Requirements:\n\n397GB\n\n508GB\n\n657 GB\n\n2.6 TB\n\n4.9 TB\n\n### Recommended Settings\n\n#### Qwen3.8-**27B Settings:**\n\nQwen3.8-27B is a** hybrid thinking** model with different default settings for thinking and non-thinking modes:\n\n`temperature`\n\n1.0\n\n0.7\n\n`top_p`\n\n0.95\n\n0.80\n\n`top_k`\n\n20\n\n20\n\n`min_p`\n\n0.0\n\n0.0\n\n`presence_penalty`\n\n0.0\n\n1.5\n\n`repetition_penalty`\n\n1.0\n\n1.0\n\n**Maximum context window:**`262,144`\n\n(can be extended to 1M via YaRN)Thinking Mode:\n\n`temperature=1.0`\n\n,`top_p=0.95`\n\n,`top_k=20`\n\n,`min_p=0.0`\n\n,`presence_penalty=0.0`\n\n,`repetition_penalty=1.0`\n\nInstruct (or non-thinking) mode:\n\n`temperature=0.7`\n\n,`top_p=0.80`\n\n,`top_k=20`\n\n,`min_p=0.0`\n\n,`presence_penalty=1.5`\n\n,`repetition_penalty=1.0`\n\n#### Qwen3.8-**2.4T Settings:**\n\nQwen3.8-2.4T is **thinking-only**, while Qwen3.8-Max is hybrid.\n\ntemperature = 1.0\n\ntop_p = 0.95\n\ntop_k = 20\n\nmin_p = 0.0\n\npresence_penalty = 0.0\n\nContext length = up to\n\n`1,010,000`\n\n`temperature=1.0`\n\n,`top_p=0.95`\n\n,`top_k=20`\n\n,`min_p=0.0`\n\n,`presence_penalty=0.0`\n\n,`repetition_penalty=1.0`\n\nIf the model fits, you will get ~20 tokens/s generation when using B200s and >120 tokens / s throughput. Best rule of thumb: RAM+VRAM ≈ the quant size; otherwise it’ll still work, just much slower due to disk offloading.\n\n### 💡 Thinking + Preserve Thinking\n\nQwen3.8 has **Preserve Thinking** which leaves the thinking trace from the previous conversation. This increases the number of tokens you use, but could increase accuracy in continued conversations. [Unsloth](/docs/models/qwen3.8#run-qwen3.8-in-unsloth-desktop) has 'Think' and Preserved Thinking toggles for Qwen3.8.\n\nQwen3.8-27B comes with support for `reasoning_effort`\n\n, which can be used to adjust reasoning depth and control cost. These toggles are automatically enabled in Unsloth:\n\n`xhigh`\n\n(default): for complex tasks demanding thorough analysis`medium`\n\n: balancing accuracy and speed`low`\n\n: efficient reasoning optimizing for speed and costnone\n\n## Run Qwen3.8 Guide\n\nYou can now run Qwen3.8 in llama.cpp and Unsloth Desktop. For the large Qwen3.8-2.T model, we will be utilizing the 397GB `IQ1_XXXS`\n\nquant (named Q1_0) for best results in terms of accessibility and accuracy and it will require at least 450GB RAM. Feel free to change quantization type. GGUF: **Qwen3.8-GGUF**\n\n[Run in Unsloth Desktop](/docs/models/qwen3.8#run-qwen3.8-in-unsloth-desktop)[Run in llama.cpp](/docs/models/qwen3.8#run-qwen3.8-in-llama.cpp)[NVFP4 Guide](/docs/models/qwen3.8#run-qwen3.8-in-llama.cpp)\n\n### 🦥 Run Qwen3.8 in Unsloth Desktop\n\nQwen3.8 can run in [Unsloth Desktop](/docs/models/qwen3.8#run-qwen3.8-in-unsloth-desktop), an open-source UI app for local AI. **Unsloth automatically offloads to RAM and detects multiGPU setups**. With Unsloth Desktop, you can run models locally on **MacOS, Windows**, Linux and:\n\nSearch, download,\n\n[run GGUFs](/docs/new/studio#run-models-locally)and safetensor models+**Self-healing** tool calling**web search**(Python, Bash)** Code execution**[Automatic inference](https://unsloth.ai/docs/desktop#feature-deep-dive)parameter tuning (temp, top-p, etc.)Fast CPU + GPU inference via MLX and llama.cpp\n\n[Train LLMs](/docs/new/studio#no-code-training)2x faster with 70% less VRAM\n\n#### Install Unsloth\n\nThe easiest way to get started is by downloading the [Unsloth Desktop app](/docs/desktop). Works on [macOS](/docs/get-started/install/mac), [Windows](/docs/get-started/install/windows-installation), and [Linux](/docs/get-started/install/linux).\n\nOr, if you prefer to install manually:\n\nMacOS, Linux, WSL:\n\nWindows PowerShell:\n\n#### Search and download Qwen3.8\n\nGo to [Unsloth Chat](/docs/new/studio/chat) or Model hub and search for Qwen3.8 in the search bar and download your desired model and quant.\n\n#### Run Qwen3.8\n\nInference parameters should be auto-set when using Unsloth, however you can still change it manually. You can also edit the context length, chat template and other settings.\n\nFor more information, you can view our [Unsloth inference guide](/docs/new/studio/chat).\n\nFor example using Unsloth Desktop with the 397GB Qwen3.8 (-91% smaller) allows you to toggle thinking modes, allow inline canvas, web search and code execution and much more.\n\n### Qwen3.8-2.4T-A95B New 1-bit data-types\n\nWe extended IQ1_S in llama.cpp which is 1.5625 bits per weight to 1.1875 bpw by reducing the number of entries in the codebook - we found this works well for large models, and can still retain a lot of accuracy - we also found these new data-types to be fine for post training quantization (PTQ) without the need for QAT or QAD (quantization aware training / distillation)\n\nDue to naming issues, we used TQ2_0, TQ1_0 and Q1_0 otherwise it won't pop up in the HF repo.\n\nIQ1_S\n\nIQ1_S\n\n**1.5625**\n\n2048\n\n11\n\n50 B\n\nUD-IQ1_XS\n\nTQ2_0\n\n1.4375\n\n1024\n\n10\n\n46 B\n\nUD-IQ1_XXS\n\nTQ1_0\n\n1.3125\n\n512\n\n9\n\n42 B\n\nUD-IQ1_XXXS\n\nQ1_0\n\n**1.1875**\n\n256\n\n8\n\n38 B\n\nWe are still running benchmarks for the new data-types, but for other large models, we get **good results without any QAT / QAD**:\n\nIQ1_S\n\n553.204\n\n2.578876\n\n0.564553\n\n78.882\n\nUD-IQ1_XS\n\n513.583\n\n2.931261\n\n0.690161\n\n75.726\n\nUD-IQ1_XXS\n\n473.961\n\n3.540383\n\n0.876007\n\n71.284\n\nUD-IQ1_XXXS\n\n434.340\n\n4.488796\n\n1.109944\n\n66.257\n\n### 🦙 Run Qwen3.8 in llama.cpp\n\nWe need to use the specific IQ1_XXXS branch [here](https://github.com/unslothai/llama.cpp/pull/61). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON`\n\nto `-DGGML_CUDA=OFF`\n\nif you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF`\n\nthen continue as usual - Metal support is on by default.\n\nIf you just want to run the standard `IQ1_S`\n\nand other quants, then compile llama.cpp normally:\n\nDownload the model via (after installing `pip install huggingface_hub`\n\n). You can choose `Q1_0`\n\nfor `IQ1_XXXS`\n\nor other quantized versions like `Q8_0`\n\n. If downloads get stuck, see: [Hugging Face Hub, XET debugging](/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging)\n\n**Qwen3.8-27B:**\n\n**Qwen3.8-2.4T:**\n\nTo run the model in llama-cli, follow the code snippets below:\nRemember to [change settings](/docs/models/qwen3.8#recommended-settings) according to your use-case.\n\n**Qwen3.8-27B:**\n\n**Qwen3.8-2.4T:**\n\nTo run the general UD-IQ1_S you can do:\n\n**Qwen3.8-27B:**\n\n**Qwen3.8-2.4T:**\n\nThen to run it:\n\n### ⚡️NVFP4\n\nLike Qwen3.6, we’re also releasing new [dynamic NVFP4 Qwen3.8](/docs/basics/nvfp4)-27B quants that run **~1.5× faster** than BF16 checkpoints, with **better performance** and comparable file sizes. Run Qwen3.8-27B NVFP4** 1.5x faster **on **24GB VRAM. **We also added **FP8 KV cache calibration** for 2x longer context lengths! NVFP4 requires NVIDIA's Blackwell GPUs like RTX 50X, DGX Spark (see [Qwen3.8](/docs/models/qwen3.8#dgx-spark-with-nvfp4-quants)), B200, B300 GPUs. For older GPUs, our GGUFs work well! You can run NVFP4 quants in [vLLM](/docs/models/qwen3.8#vllm) and [SGLang](/docs/models/qwen3.8#sglang).\n\n[Qwen3.8-27B-](https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4)**NVFP4** quant\n\n1\n\n89.8\n\n**133.7**\n\n1.49x\n\n89.8\n\n**133.7**\n\n8\n\n649.4\n\n**938.8**\n\n1.45x\n\n81.2\n\n**117.3**\n\n32\n\n1983.0\n\n**2787.0**\n\n1.41x\n\n62.0\n\n**87.1**\n\n64\n\n3048.5\n\n**4407.2**\n\n1.45x\n\n47.6\n\n**68.9**\n\nSee below for previous benchmarks conducted for Qwen3.6 as well comparing to other NVFP4 implementations which use 16bit activations vs our NVFP4 activations:\n\nAll benchmarks use 1x B200 128 concurrency. Higher concurrency can boost 35B to 17,561 tokens / s.\n\nFor accuracy benchmarks we ran KLD and Top-1% agreement over Code, Chat and many domains. NVFP4 is consistency 92% to 97% accuracy recovery vs BF16\n\nzh\n\n0.01628\n\n93.55%\n\ncode\n\n0.02600\n\n96.68%\n\nrefgen\n\n0.03993\n\n94.46%\n\nchat\n\n0.05818\n\n92.15%\n\nja / ko / ru / es\n\n0.0124-0.0155\n\n94-95%\n\nFor accuracy benchmarks For Qwen 3.6, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all:\n\nFor more information, you can read our [Dynamic NVFP4 quants blog](/docs/basics/nvfp4).\n\nTo run NVFP4 quants, see below for commands to run Qwen3.8-27B in [vLLM](/docs/basics/inference-and-deployment/vllm-guide) or [SGLang](/docs/basics/inference-and-deployment/sglang-guide):\n\n**vLLM:**\n\nTo install vLLM in a separate venv:\n\nThen to serve the 27B variant:\n\nTo enable MTP / speculative decoding (faster decode but somewhat less throughput), use:\n\nIf you get Torchcodec issues, be sure to do the below then relaunch vllm.\n\n**SGLang:**\n\nInstall SGLang then run:\n\n### 📊 Benchmarks\n\n#### Qwen3.8-**27B**\n\n#### Qwen3.8-**2.4T-A95B**\n\nLast updated\n\nWas this helpful?", "url": "https://wpnews.pro/news/unsloth-qwen3-8-how-to-run-locally", "canonical_source": "https://unsloth.ai/docs/models/qwen3.8", "published_at": "2026-08-14 15:03:16+00:00", "updated_at": "2026-08-14 15:13:32.718988+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["Unsloth", "Qwen", "Qwen3.8", "Qwen3.8-27B", "Qwen3.8-2.4T-A95B", "Qwen3.8-Max", "llama.cpp", "Unsloth Desktop"], "alternates": {"html": "https://wpnews.pro/news/unsloth-qwen3-8-how-to-run-locally", "markdown": "https://wpnews.pro/news/unsloth-qwen3-8-how-to-run-locally.md", "text": "https://wpnews.pro/news/unsloth-qwen3-8-how-to-run-locally.txt", "jsonld": "https://wpnews.pro/news/unsloth-qwen3-8-how-to-run-locally.jsonld"}}