{"slug": "benchmarking-qwen3-8-27b-quantizations-4-bit-holds-up-1-bit-collapses", "title": "Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses", "summary": "A developer's benchmark of Qwen3.8 27B quantizations found that the 17 GB Q4_K_M 4-bit version matches the full BF16 model on Terminal-Bench 2.1, while 1-bit versions collapse to near-random performance on GPQA Diamond. The tests, run on Modal GPUs at a cost of about $3,000, showed little difference down to 4-bit, but 2-bit scored slightly lower and reasoning effort significantly affected results.", "body_md": "How much GPU RAM do you actually need to run Qwen3.8 27B without sacrificing quality?\n\nThe full `BF16`\n\nmodel weighs 55 GB, putting it beyond most consumer hardware.\nYet the 17 GB `Q4_K_M`\n\nmatches the full model on a popular agentic coding benchmark, Terminal-Bench 2.1. It fits on a 24 GB card such as RTX 4090, still leaving room for about 64k tokens of context.\n\nCompression eventually hits a cliff. At 1 bit, the model performs around random chance on GPQA Diamond, and longer reasoning makes it worse.\n\n## Background\n\nPreviously, I investigated the Qwen3.6 27B model, which was [good at generating SVG pelicans even at 12GB](https://quesma.com/blog/qwen-quantization-quality/), and [maintained most of its knowledge up to 16GB](https://quesma.com/blog/quantization-hurts-knowledge/).\nAt the same time, in Reddit threads, many complain that all quantizations, even the 8-bit ones, give worse results - with people asking [why your local LLM feels dumber than it is](https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917). Are these complaints grounded?\n\nMeasuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks. Some noise might be irrelevant for solving tasks, as (say) a quantized model generates an answer of precisely the same quality, paraphrased a bit. In other cases, a single different token might be a logical error, or even abruptly end the output.\n\nSo, I focus on directly measuring results on popular benchmarks - [GPQA Diamond](https://artificialanalysis.ai/evaluations/gpqa-diamond), instruction-following [IFBench](https://github.com/allenai/IFBench), programming [Terminal-Bench 2.1](https://www.tbench.ai/leaderboard/terminal-bench/2.1).\nFirst, to replicate official results of the full model `BF16`\n\n, and then to see how quantization affects results.\n\nI burned around $3,000 on [Modal](https://modal.com/) GPUs when I ran models with [llama.cpp](https://github.com/ggml-org/llama.cpp) using a build from 16 August 2026 as [earlier builds do not work for this model](https://www.paulsprogrammingnotes.com/2026/08/running-qwen-3-8-27b-16gb.html). I could have run it on my own laptop, in principle, but (unlike pelican-generation), these are time-consuming benchmarks.\n\nNote that I use `F16`\n\nKV-cache regardless of model quantization, weighing around 2.3 GB per 32k tokens.\n\nI used [Unsloth quantizations](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF): v2 for the 2-, 4-, and 8-bit models, and [v3](https://unsloth.ai/docs/basics/dynamic-3.0-ggufs) for the 1-bit models. [Unsloth replaced the v2 files on 19 August 2026](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/discussions/74), so the exact files used for most tests are no longer available.\n\nIn short, if you go with a 4-bit quantization `Q4_K_M`\n\n(17GB), you won’t notice a difference on these benchmarks.\nAt the same time, the effort setting matters a lot (note that the default is `xhigh`\n\n) - and it is a tricky choice, as [it can overthink](https://simonwillison.net/2026/Aug/16/qwen-38-27b/).\n\n## One-shot tests\n\nThe easiest ones are one-shot tests: in this case, graduate-level science GPQA Diamond and instruction-following IFBench.\nI run each at three reasoning efforts: `low`\n\n, `medium`\n\n, and the default `xhigh`\n\n.\n\n### GPQA Diamond\n\nFirst and foremost, I was happy I replicated the official results. Running benchmarks is hard; there are many hidden settings or assumptions that can change the results drastically. Here, on the first go, results were as reported by Qwen.\n\nSecond, besides noise (bars are [Wilson 95% confidence intervals](https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Wilson_score_interval), very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower.\n\nAt the same time, thinking level changed the score dractically. The best results, for `xhigh`\n\n, needed around 8k reasoning tokens.\n\n### IFBench\n\nHere, to my great surprise, there is no change between models, down to a decent 2-bit one, weighing less than 11 GB. Yet, context is even lower, around 4k tokens.\n\n## Agentic coding and Terminal-Bench 2.1\n\nHow does it work for programming? Terminal-Bench 2.1 is a standard agentic benchmark, with 89 tasks.\nHere I use 3h timeout, `xhigh`\n\neffort. I reserve 98k context.\n\nNot only does my measurement of `BF16`\n\nreplicate the stated result, but, to my surprise, `Q4_K_M`\n\ndoes as well.\nI accidentally skipped running `Q8_0`\n\n; yet, in this case, I can safely interpolate between 4-bit and the full model’s values. Running it would be both costly and unnecessary (and would exceed an informal blog post’s budget).\nOnly at 2-bit `UD-Q2_K_XL`\n\nthings break a bit. A noticeable fall, but still [the level of Opus 4.7 or Gemini 3.1 Pro](https://www.tbench.ai/leaderboard/terminal-bench/2.1?lf=%7B%22agent_display%22:%7B%22kind%22:%22categorical%22,%22values%22:%5B%22Terminus+2%22%5D%7D%7D). Again, far from frontier, but also - far from useless.\n\nResults are one thing, but what about the process? Do smaller models need more turns, tokens or time to get the result?\n\nOn the same solved tasks, `UD-Q2_K_XL`\n\ntakes as many turns as BF16 but writes about a quarter more tokens.\nThe number of turns stay roughly the same.\n\n## The 1-bit cliff\n\nQuality drops off a cliff at 1-bit. [As with knowledge](https://quesma.com/blog/quantization-hurts-knowledge/), quantization damage is nonlinear: first there is no measurable change, then a small decline, and finally a collapse.\n\nWhile 2-bit quantizations work to some extent, even the best 1-bit model is useless for these benchmarks:\n\nAs you may see, the scores are around the random guessing level, with the smallest model being below that threshold.\nAnd longer reasoning makes it worse: at `xhigh`\n\n, scores drop below `low`\n\n, as the model more often reasons until the token budget runs out and returns an empty answer.\nSure, Unsloth boasts that:\n\nWe also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller.\n\nBut in this case, these remaining 28% matter a lot. And this matches another user’s experience, vide [Qwen3.8 27b 1bit brain damage quant on r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/comments/1vtr3h0/ladies_and_gentlemen_i_present_to_you_qwen38_27b/).\n\n## Costs\n\nRunning these benchmarks isn’t cheap.\nRunning benchmarks via API is costly, as I know from my [previous benchmarks](https://quesma.com/benchmarks/). Running on rented GPU is much costlier.\n\nI used Modal, as it is easy to run it from the CLI, including from agents. Other setups may have different pricing. Obviously, this calculation changes if you have your own devices.\n\nIt takes some testing to find the optimal way to run models. Usually, instead of using Multi-Token Prediction (MTP), which works well for a single stream, I use a few parallel streams. The key constraint is whether the GPU has enough memory for both the model and the required KV caches.\n\nI used NVIDIA L40S (the same Ada Lovelace chip as the RTX 4090 but twice as much memory: 48 GB), H100 (80 GB) and H200 (141 GB). I would like to share costs to give you a ballpark estimate if you want to run benchmarks yourself.\n\nFor comparison, [DeepSeek V4 Flash 0731](https://openrouter.ai/deepseek/deepseek-v4-flash-0731#providers), a 284B model, costs around $0.1/Mtok for output from the cheapest providers on OpenRouter. I am not sure how much of this difference comes from the efficiency of running models at scale, pricing strategy, or popularity.\n\n## Conclusion\n\nIf you run experiments locally, usually pick the best model that fits in your GPU memory together with the required context.\nFor most tasks Unsloth’s `Q4_K_M`\n\nshould be good enough, without any noticeable difference; for some simpler tasks `UD-Q2_K_XL`\n\nshould be more than fine.\nSince people report that KV-caches are more susceptible to quantization, I may test it as well.\n\nBut in general, I believe that quantization should be embraced, rather than feared.\n\nAnd what is your experience?", "url": "https://wpnews.pro/news/benchmarking-qwen3-8-27b-quantizations-4-bit-holds-up-1-bit-collapses", "canonical_source": "https://quesma.com/blog/qwen38-27b-quantizations-benchmarked/", "published_at": "2026-08-26 00:00:00+00:00", "updated_at": "2026-08-26 17:45:35.124149+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools"], "entities": ["Qwen3.8 27B", "Terminal-Bench 2.1", "GPQA Diamond", "IFBench", "Modal", "Unsloth", "llama.cpp", "RTX 4090"], "alternates": {"html": "https://wpnews.pro/news/benchmarking-qwen3-8-27b-quantizations-4-bit-holds-up-1-bit-collapses", "markdown": "https://wpnews.pro/news/benchmarking-qwen3-8-27b-quantizations-4-bit-holds-up-1-bit-collapses.md", "text": "https://wpnews.pro/news/benchmarking-qwen3-8-27b-quantizations-4-bit-holds-up-1-bit-collapses.txt", "jsonld": "https://wpnews.pro/news/benchmarking-qwen3-8-27b-quantizations-4-bit-holds-up-1-bit-collapses.jsonld"}}