{"slug": "qwen-3-8-quant-selection-guide-for-rtx-5090", "title": "Qwen 3.8 Quant Selection Guide for RTX 5090", "summary": "A benchmark comparison of seven quantized Qwen3.8-27B checkpoints on an RTX 5090 found the cyankiwi/Qwen3.8-27B-AWQ-INT4 quant to be the most accurate, with only 156 top-1 token flips (2.193%) across 7,114 tokens, while the sakamakismile/Qwen3.8-27B-MTP-NVFP4 quant performed worst with 451 flips (6.340%). The tests, run by a Level1Techs forum user, compared each candidate's next-token distribution against the stock BF16 reference across two multi-step workflows with context depths up to 123.6K tokens.", "body_md": "Since I have been running a rather in depth set of benchmarks comparing Qwen’s [Why your local LLM feels dumber than it is](https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917) I thought it might be worth giving an example of how to actually apply this approach to quant selection given the 3.8 release of a popular model.\n\nThe title of this post is a targeted guide, somebody who wants to fit Q3.8-27b into a single 5090 and would like to see the trade-offs between precision and performance.\n\nAs stated in my much larger more detailed series on precision, your specific tests should use your specific captured workflows for evaluating model selection, mine are agentic and technical rather than what you might need. Every evaluation is workload specific, know yours, test against it.\n\nThis model is in its very early days of being quantized, fine tuned, deployed, etc. So everything in here WILL change at some point as new quants are released, bugs are fixed, chat templates inference engines cuda kernels etc. Change is continual, and while this post focuses on one specific benchmark result, it should be considered generally as a way to approach selecting and reviewing quants for a model.\n\nThe testing scored 7,114 natural assistant-output tokens per checkpoint across 13 output ranges in two real multi-step workflows. Context depth ranged from about 18.8K to 123.6K. At every output position we compared the candidate’s full next-token distribution with stock Qwen3.8-27B BF16 and recorded whether its Top-1 token changed.\n\nAll candidates used Triton attention, TP1, eager execution, BF16 KV, reasoning off, MTP off, and the same tokenizer. Candidates ran on the RTX 5090; the existing BF16 reference ran on an RTX PRO 6000, so absolute small deltas include a labeled cross-card component.\n\nThe labels elsewhere in this report are short local aliases. These are the actual Hugging Face repositories and the revisions represented by the files we tested:\n\n| Short label | Hugging Face checkpoint | |\n|---|---|---|\n| BF16 reference |\n`Qwen/Qwen3.8-27B` |\n\n`cyankiwi/Qwen3.8-27B-AWQ-INT4`\n\n`philbert440/Qwen3.8-27B-W4A16-AWQ`\n\n`dbirks/Qwen3.8-27B-W4A16-AutoRound`\n\n`unsloth/Qwen3.8-27B-NVFP4`\n\n`RadixArk/Qwen3.8-27B-NVFP4`\n\n`gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090`\n\n`sakamakismile/Qwen3.8-27B-MTP-NVFP4`\n\nThe MTP-bearing repositories were evaluated with MTP disabled for this checkpoint-quality comparison. The GitTensor and SakamakiSmile checkpoints are not interchangeable merely because both say NVFP4; their export recipes and preserved higher-precision tensors differ.\n\n| Rank | Checkpoint | Arithmetic class | Top-1 flips | Flip rate | KLD p50 | KLD p95 | KLD p99 | Verdict |\n|---|---|---|---|---|---|---|---|---|\n| 1 | `Qwen3.8-27b-awq-cyanwiki` |\nasymmetric G32 W4A16 | 156 / 7,114 | 2.193% |\n0.00003475 |\n0.04826 |\n0.13570 |\nDeploy |\n| 2 | `Qwen3.8-27b-nvfp4-unsloth` |\nmixed W8A8/W4A4 | 184 / 7,114 | 2.586% | 0.00004580 | 0.06029 | 0.17359 | Numerically good, operationally awkward |\n| 3 | `Qwen3.8-27b-w4a16-dbirks` |\nsymmetric G128 W4A16 | 211 / 7,114 | 2.966% | 0.00007520 | 0.11872 | 0.35675 | W4A16 backup |\n| 4 | `Qwen3.8-27b-awq-philbert440` |\nasymmetric G128 W4A16 | 241 / 7,114 | 3.388% | 0.00007198 | 0.09133 | 0.24967 | Not bad, but loses at depth |\n| 5 | `Qwen3.8-27b-nvfp4-radixark` |\nmixed W8A8/W4A4 | 319 / 7,114 | 4.484% | 0.00007147 | 0.14041 | 0.53206 | No compelling advantage |\n| 6 | `Qwen3.8-27b-nvfp4-gittensor` |\nW4A4 | 325 / 7,114 | 4.568% | 0.00013996 | 0.18852 | 0.68631 | Skip |\n| 7 | `Qwen3.8-27b-nvfp4-sakamakismile` |\nW4A4 | 451 / 7,114 | 6.340% |\n0.00028275 | 0.46118 |\n1.37355 |\nDumpsterize |\n\nThere was a clear and obvious winner for the quality comparison: cyanwiki scored well in other tests across other models/weights as well in related testing.\n\n| Checkpoint | SP06: 4,339 tokens, 18.8K–83.7K | SP03: 2,775 tokens, 19.9K–123.6K |\n|---|---|---|\n| AWQ G32 | 99 flips (2.282%) | 57 flips (2.054%) |\n| Unsloth NVFP4 | 105 (2.420%) | 79 (2.847%) |\n| dbirks W4A16 | 117 (2.696%) | 94 (3.387%) |\n| philbert440 AWQ | 127 (2.927%) | 114 (4.108%) |\n| RadixArk NVFP4 | 153 (3.526%) | 166 (5.982%) |\n| GitTensor NVFP4 | 149 (3.434%) | 176 (6.342%) |\n| SakamakiSmile NVFP4 | 278 (6.407%) | 173 (6.234%) |\n\nThe G32 AWQ model wins on both workflows rather than obtaining its aggregate lead from one lucky prompt. The data is not monotonic with context length: a hard open-ended answer near 20K produced far more flips than a more constrained range near 30K. Content and baseline decision margin matter at least as much as raw depth.\n\nUnsloth is second by text fidelity, but it is not the a clean W4A16 shape for this worker. It mixes activation-quantized W8A8/W4A4 paths, and at 123K with BF16 KV it failed at the original 98% allocation because too little execution workspace remained. It completed after reserving more headroom and setting the exact context ceiling. Several NVFP4 candidates also forced the FlashInfer GEMM autotuner to reject large-workspace tactics near the 5090 memory ceiling.\n\nThat behavior is useful real-world evidence, but it makes Unsloth less attractive than the slightly more faithful, simpler G32 W4A16 checkpoint.\n\nWe took 200 physical-MTG-card scans from a real collection and replayed the exact production bottom-strip card/set/printing identification task. The runner uses the saved dewarped image, crops the bottom 12%, JPEG-encodes at quality 92, disables thinking, and sends Recall’s current set-code/collector-number JSON prompt. The candidate model’s read then receives weight 3 in the real candidate-constrained vote; fixed RapidOCR and Tesseract reads receive weights 2 and 1.\n\nThis is not generic visual question answering and it is not image-logit KLD. It is the operational question that matters here: did changing the decoder alter what Recall concludes about a scanned Magic card? All four models returned parseable JSON on all 200 images.\n\nRuntime coordinate: pinned `vllm/vllm-openai:nightly-x86_64`\n\nbuild `acb0f1dcd`\n\n, vLLM `0.27.2rc1.dev110+gacb0f1dcd`\n\n, eager TP1, BF16 KV, `TRITON_ATTN`\n\nfull attention, FlashAttention for the vision encoder, Triton/FLA GDN prefill and CUDA GDN decode. All three W4A16 checkpoints resolved to `MarlinLinearKernel`\n\nthrough CompressedTensors WNA16. BF16, dbirks and Philbert ran on otherwise-idle RTX PRO 6000 Blackwells; Cyanwiki ran on the RTX 5090. Consequently, latency from this quality gate is not used as a product comparison; the planned speed/capacity matrix is same-card.\n\n| Coordinate | Raw set exact | Raw collector exact | Raw both exact | Ensemble decisive | Ensemble correct | Correct when decisive |\n|---|---|---|---|---|---|---|\n| Tesseract + RapidOCR only | — | — | — | 63 (31.5%) | 52 (26.0%) | 82.5% |\n| Qwen3.8 BF16 | 28 (14.0%) | 95 (47.5%) | 28 (14.0%) | 108 (54.0%) | 100 (50.0%) | 92.6% |\n| Cyanwiki G32 W4A16 | 27 (13.5%) | 94 (47.0%) | 27 (13.5%) | 109 (54.5%) | 100 (50.0%) | 91.7% |\n| dbirks G128 W4A16 | 28 (14.0%) | 101 (50.5%) | 28 (14.0%) | 111 (55.5%) | 102 (51.0%) | 91.9% |\n| Philbert G128 W4A16 | 27 (13.5%) | 87 (43.5%) | 27 (13.5%) | 104 (52.0%) | 95 (47.5%) | 91.3% |\n\nThe absolute raw set-code score looks harsh because this panel deliberately contains 148 same-name reprint groups and the tiny printed set code is often the difficult field. The VLM is not used alone: adding any tested Qwen3.8 coordinate to the two conventional OCR engines increases decisive, correct outcomes substantially over the fixed-OCR-only control.\n\nThe aggregate two-card lead for dbirks is not statistically or operationally convincing on this sample. Against Cyanwiki, dbirks uniquely fixes five cards and uniquely breaks three; the other 192 outcomes agree. BF16 versus Cyanwiki is exactly six unique wins apiece. Treat the panel as evidence that Cyanwiki did not create a visible regression in this workflow, not as evidence that quantization improves vision.\n\nThe primary W4A16 checkpoints exclude the `visual.*`\n\ntower from weight quantization, but their visual embeddings still flow through different quantized language decoders. These results show why the real prompt, crop, parser and downstream vote are more decision-useful than merely observing that the vision tower stayed BF16.\n\n`Qwen3.8-27b-awq-cyanwiki`\n\n.For this mixed agentic-plus-card workload, Cyanwiki remains the best first deployment. If the worker were dedicated only to bottom-strip card OCR, dbirks would merit a larger head-to-head panel before selection; this 200-card tie is not enough reason to accept its clearly weaker long-context text result.\n\nChanging any of these produces a new runtime coordinate; it should not inherit the exact percentages in this table without a controlled check.\n\nLets face it, most casual users seem to survive a small number of token flips just fine. Thats why most reviews focus on performance. So lets actually measure the precision lost when performance is gained.\n\nThis matrix asks what a 5090 owner trades for speed and usable context capacity; it is not part of the checkpoint ranking.\n\nPreflight found that the appealing 3-by-2 table is not actually symmetric in the pinned nightly. Cyanwiki is a weight-only checkpoint with `kv_cache_scheme: null`\n\n; it contains no static K/V scales. This vLLM build no longer exposes the older `--calculate-kv-scales`\n\nswitch. Plain\n\n`--kv-cache-dtype fp8`\n\ntherefore falls back to scalar scale 1.0 and warns that the checkpoint did not provide calibrated K/V scaling factors.\n\nThe build also offers `fp8_per_token_head`\n\n, which calculates dynamic scales at cache-write time, but only Triton advertises support. FlashInfer accepts scalar `fp8`\n\n/`fp8_e4m3`\n\n, not the dynamically scaled mode. FlashAttention 2 accepts BF16 KV on SM120 but rejects FP8 KV: vLLM requires FA3 on SM90 or FA4 on SM100 for that combination. The honest practical panel therefore records seven requested coordinates as six runnable measurements plus one compatibility failure rather than pretending all six backend-by-cache cells mean the same\n\nthing:\n\n| Cell | Full-attention backend | KV cache | Meaning |\n|---|---|---|---|\n| 1 — fidelity reference | Triton | `bfloat16` |\nhigh-fidelity control |\n| 2 | FlashAttention 2 | `bfloat16` |\nbackend-only change |\n| 3 | FlashInfer | `bfloat16` |\nbackend-only change |\n| 4 | Triton | `fp8_per_token_head` |\ndynamically scaled practical FP8 |\n| 5 | Triton | `fp8` |\nnaive one-flag, unit-scale FP8 |\n| 6 | FlashAttention 2 | `fp8` |\nunsupported on SM120; compatibility result |\n| 7 | FlashInfer | `fp8` |\nnaive one-flag, unit-scale FP8 |\n\nCells 5–7 are valuable as a documented pitfall, not as a recommendation. A clean calibrated scalar-FP8 3-by-2 experiment would require creating and freezing a derived checkpoint with calibrated per-layer K/V scales. That is a separate preparation variable and should not be quietly smuggled into a “change one launch flag” deployment table.\n\nHold checkpoint, GPU, runtime image digest, tokenizer/processor, prompt, seed, samplers, context ceiling, concurrency, MTP/reasoning policy, graph policy and all non-attention kernels fixed. Treat a launch as valid only when its log confirms that the requested backend and KV dtype actually resolved; unsupported or silently substituted coordinates are compatibility results, not measurements of the requested cell.\n\nAccuracy reference: Cyanwiki G32, Triton, BF16 KV. To finish the arithmetic panel in parallel, all seven quality cells run on one RTX PRO 6000 SM120 proxy; all performance and Recall cells run on the target RTX 5090 SM120. Report per-range Top-1 flips and KLD rather than only a global mean. The proxy keeps architecture and requested kernels fixed but is not an identical board or memory-pressure coordinate, so any near-noise quality delta should be repeated\n\non the target 5090 before making a strong claim.\n\nThe frozen accuracy slice contains 3,512 recorded assistant-output tokens in five natural ranges at roughly 18.8K, 27.5K, 42.2K, 42.8K and 48.6K context. It stops at 50,196 tokens, avoiding the 5090 capacity edge while still testing diagnostic prose, parallel tool calls, a mutating call, failure recovery and a 1,570-token Python migration-script tool call. The same 200-card Recall panel should also be replayed for every supported/recommended cell; that exposes a vision-workflow regression that text KLD alone cannot see.\n\nPractical 32K table columns:\n\nBenchmarking must warm model load, JIT/autotuning and the selected graph shapes before timed repetitions. Record all repetitions rather than publishing only the best. The launch recipe should be one pinned container command with explicit pick-one substitutions for attention backend and KV dtype. It must reject unsupported FA2/FlashInfer plus `fp8_per_token_head`\n\ncombinations and retain the unit-scale warning in the evidence for plain `fp8`\n\ncells.\n\nPerformance scope: exact-token-targeted 32K context, C1 sustained decode, 60-second measured cells, a hidden decode warmup, one exact 32K cold-prefill sample per repetition and three repetitions per coordinate. There is no prefill-depth sweep in this practical comparison. Use the benchmark at commit `0b4185b5b435e948b199c9077a00b084864aa963`\n\n; publish both its JSON and the server’s resolved-runtime log. Quality capture uses the same graph policy, prefix-cache policy, `max_num_seqs=4`\n\n, scheduler token budget and dtypes as the\n\npractical server rather than inheriting the earlier eager-only checkpoint bakeoff. Its 96-GB proxy necessarily has a larger physical KV allocation; capacity numbers come only from the actual 5090 performance cells.\n\nBefore execution, cross-check this seven-cell plan and the exact launch flags against the local community `llm-inference-bench`\n\ntooling so its PP/TG definitions, prompt construction and warmup behavior are captured precisely.\n\nThe completed matrix used one exact 32,768-token integrated prefill scout, a hidden 10-second decode warmup, and a 60-second concurrency-one decode window. Each runnable coordinate was repeated three times. The values below are medians; the raw JSON retains all repetitions.\n\n| Coordinate | PP32K tok/s | TG32K C1 tok/s | Server KV budget | KV-token capacity | Top-1 flips vs Triton/BF16 | Per-range KLD p95 | Recall correct / decisive |\n|---|---|---|---|---|---|---|---|\n| Triton + BF16 KV | 2,614 | 66.4 | 8.21 GiB | 120,897 | reference | reference | 100 / 109 |\n| FA2 + BF16 KV | 3,179 | 70.3 | 8.21 GiB | 120,897 | 19 / 3,512 (0.541%) | 0.000295–0.002075 | 98 / 107 |\n| FlashInfer + BF16 KV | 3,157 | 69.8 | 7.88 GiB | 115,948 | 14 / 3,512 (0.399%) | 0.000579–0.002123 | 100 / 108 |\n| Triton + dynamically scaled FP8 KV | 2,117 | 57.9 | 8.21 GiB | 218,807 | 22 / 3,512 (0.626%) | 0.001278–0.003927 | 99 / 108 |\n| Triton + unit-scale FP8 KV | 2,824 | 70.8 | 8.18 GiB | 218,807 | 29 / 3,512 (0.826%) | 0.001806–0.007343 | 99 / 108 |\n| FA2 + unit-scale FP8 KV | unsupported |\nunsupported |\n— | — | — | — | — |\n| FlashInfer + unit-scale FP8 KV | 3,121 | 73.6 | 7.88 GiB | 211,084 | 18 / 3,512 (0.513%) | 0.001573–0.006023 | 99 / 107 |\n\nThe exact Triton/BF16 repeat control changed Top-1 at 13 of 3,512 positions (0.370%). Its requested configuration and Triton source, PTX and IR hashes were identical. This is an empirical same-path repeatability floor, not a correction that can simply be subtracted from every candidate. It does mean that the 14 FlashInfer/BF16 flips are effectively at the observed floor, while 19 FA2/BF16 flips are only modestly above it. The FP8 cells show clearer distributional movement, especially Triton’s uncalibrated unit-scale cache.\n\nThe large practical capacity gain is 1.81x for Triton FP8 versus Triton BF16 (218,807 versus 120,897 usable KV tokens), not a full 2x after graph and runtime workspace are accounted for. Dynamic per-token/head scaling was the higher-integrity FP8 method, but it cost about 19% of Triton/BF16 prefill and 13% of decode throughput. The one-flag unit-scale paths were faster, but the model provides no calibrated K/V scales, so their surprisingly good aggregate scores\n\nshould not be generalized into a safety claim.\n\nOn this exact 5090 coordinate, FA2/BF16 is the strongest conventional speed/capacity choice: about 21.6% faster prefill and 5.9% faster decode than Triton/BF16 while retaining the same server KV allocation. FlashInfer/BF16 is very close in speed and closest to the observed repeatability floor, but its workspace left about 4.1% fewer usable KV tokens. Keep Triton/BF16 when fidelity continuity with the original quant ranking matters more than the\n\nsmall throughput gain.\n\nAll six runnable cells returned 200/200 valid JSON objects on the Recall card panel. Their final ensemble score ranged only from 98 to 100 correct cases, so that panel found no decision-useful visual regression among these runtime settings. It does not vindicate unit-scale FP8 KV for unrelated agentic or high-stakes workloads.\n\nThis is the tested server shape. Mount the frozen `cyankiwi/Qwen3.8-27B-AWQ-INT4`\n\nrevision at `/models/qwen38-cyanwiki`\n\n, then make only the indicated pick-one substitutions:\n\n```\nIMAGE='vllm/vllm-openai@sha256:0db5553091d59de260f67696fff89026e7376ec0d326111ba78fb004a27778a0'\nATTENTION_BACKEND='FLASH_ATTN'   # TRITON_ATTN | FLASH_ATTN | FLASHINFER\nKV_DTYPE='bfloat16'              # bfloat16 | fp8 | fp8_per_token_head\n\ndocker run --rm --gpus 'device=0' --network host --ipc host --shm-size 32g \\\n  -v /absolute/path/to/Qwen3.8-27B-AWQ-INT4:/models/qwen38-cyanwiki:ro \\\n  -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e VLLM_NO_USAGE_STATS=1 \\\n  \"$IMAGE\" /models/qwen38-cyanwiki \\\n  --served-model-name qwen38-cyanwiki \\\n  --tensor-parallel-size 1 \\\n  --dtype bfloat16 \\\n  --kv-cache-dtype \"$KV_DTYPE\" \\\n  --attention-backend \"$ATTENTION_BACKEND\" \\\n  --gdn-prefill-backend triton \\\n  --max-model-len 50197 \\\n  --max-num-seqs 4 \\\n  --max-num-batched-tokens 8192 \\\n  --gpu-memory-utilization 0.95 \\\n  --enable-prefix-caching \\\n  --seed 1 \\\n  --trust-remote-code \\\n  --host 127.0.0.1 --port 8000\n```\n\nCompatibility is not a Cartesian product. On this SM120 stack, `fp8_per_token_head`\n\nis Triton-only and `FLASH_ATTN`\n\nplus FP8 KV is rejected. Plain `fp8`\n\nis the uncalibrated unit-scale case documented above. The practical default from this panel is therefore `FLASH_ATTN`\n\nplus `bfloat16`\n\n; use `TRITON_ATTN`\n\nplus `bfloat16`\n\nto reproduce the fidelity-reference coordinate.\n\nThe checkpoint’s quantized decoder linears resolved to `MarlinLinearKernel`\n\nthrough CompressedTensors WNA16 in every cell. The backend switch applied to the 16 decoder full-attention layers: `TRITON_ATTN`\n\n, FlashAttention version 2, or FlashInfer. FlashInfer resolved BF16 prefill and decode queries to its XQA decode backend on `sm120`\n\n; with plain FP8 KV the cache dtype resolved to `torch.float8_e4m3fn`\n\n.\n\nQwen3.8 is hybrid, so those labels do not describe the whole model. Its 48 GatedDeltaNet layers remained on Triton/FLA prefill and the CUDA recurrent decode implementation in all cells. The vision encoder separately used FlashAttention, and top-k/top-p sampling separately used FlashInfer. Thus “Triton versus FA2 versus FlashInfer” here means the decoder’s full-attention backend only; it is not a claim that every CUDA kernel in the process changed.\n\nFA2/FP8 failed before serving with the explicit vLLM validation error that FP8 KV requires FA3 on SM90 or FA4 on SM100. No silent fallback result is included.\n\nIn this particular use case, this particular model, and these specific quants: There is a clear quality winner. If you care more cache size than absolute precision, I can safely drop down to FP8 here knowing the cost.\n\nI am still working to figure out how to bundle up my datasets and testing at the larger scale but this one shook out, was a natural question people might ask, and is timely given the new weights dropping.", "url": "https://wpnews.pro/news/qwen-3-8-quant-selection-guide-for-rtx-5090", "canonical_source": "https://forum.level1techs.com/t/qwen-3-8-quant-selection-guide-for-rtx-5090/254095#post_1", "published_at": "2026-08-19 14:38:40+00:00", "updated_at": "2026-08-19 14:42:15.787911+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research"], "entities": ["Qwen", "RTX 5090", "cyankiwi/Qwen3.8-27B-AWQ-INT4", "philbert440/Qwen3.8-27B-W4A16-AWQ", "dbirks/Qwen3.8-27B-W4A16-AutoRound", "unsloth/Qwen3.8-27B-NVFP4", "RadixArk/Qwen3.8-27B-NVFP4", "gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090"], "alternates": {"html": "https://wpnews.pro/news/qwen-3-8-quant-selection-guide-for-rtx-5090", "markdown": "https://wpnews.pro/news/qwen-3-8-quant-selection-guide-for-rtx-5090.md", "text": "https://wpnews.pro/news/qwen-3-8-quant-selection-guide-for-rtx-5090.txt", "jsonld": "https://wpnews.pro/news/qwen-3-8-quant-selection-guide-for-rtx-5090.jsonld"}}