Qwen 3.8 Quant Selection Guide for RTX 5090

A benchmark comparison of seven quantized Qwen3.8-27B checkpoints on an RTX 5090 found the cyankiwi/Qwen3.8-27B-AWQ-INT4 quant to be the most accurate, with only 156 top-1 token flips (2.193%) across 7,114 tokens, while the sakamakismile/Qwen3.8-27B-MTP-NVFP4 quant performed worst with 451 flips (6.340%). The tests, run by a Level1Techs forum user, compared each candidate's next-token distribution against the stock BF16 reference across two multi-step workflows with context depths up to 123.6K tokens.

Since I have been running a rather in depth set of benchmarks comparing Qwen’s Why your local LLM feels dumber than it is https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917 I thought it might be worth giving an example of how to actually apply this approach to quant selection given the 3.8 release of a popular model. The title of this post is a targeted guide, somebody who wants to fit Q3.8-27b into a single 5090 and would like to see the trade-offs between precision and performance. As stated in my much larger more detailed series on precision, your specific tests should use your specific captured workflows for evaluating model selection, mine are agentic and technical rather than what you might need. Every evaluation is workload specific, know yours, test against it. This model is in its very early days of being quantized, fine tuned, deployed, etc. So everything in here WILL change at some point as new quants are released, bugs are fixed, chat templates inference engines cuda kernels etc. Change is continual, and while this post focuses on one specific benchmark result, it should be considered generally as a way to approach selecting and reviewing quants for a model. The testing scored 7,114 natural assistant-output tokens per checkpoint across 13 output ranges in two real multi-step workflows. Context depth ranged from about 18.8K to 123.6K. At every output position we compared the candidate’s full next-token distribution with stock Qwen3.8-27B BF16 and recorded whether its Top-1 token changed. All candidates used Triton attention, TP1, eager execution, BF16 KV, reasoning off, MTP off, and the same tokenizer. Candidates ran on the RTX 5090; the existing BF16 reference ran on an RTX PRO 6000, so absolute small deltas include a labeled cross-card component. The labels elsewhere in this report are short local aliases. These are the actual Hugging Face repositories and the revisions represented by the files we tested: | Short label | Hugging Face checkpoint | | |---|---|---| | BF16 reference | Qwen/Qwen3.8-27B | cyankiwi/Qwen3.8-27B-AWQ-INT4 philbert440/Qwen3.8-27B-W4A16-AWQ dbirks/Qwen3.8-27B-W4A16-AutoRound unsloth/Qwen3.8-27B-NVFP4 RadixArk/Qwen3.8-27B-NVFP4 gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 sakamakismile/Qwen3.8-27B-MTP-NVFP4 The MTP-bearing repositories were evaluated with MTP disabled for this checkpoint-quality comparison. The GitTensor and SakamakiSmile checkpoints are not interchangeable merely because both say NVFP4; their export recipes and preserved higher-precision tensors differ. | Rank | Checkpoint | Arithmetic class | Top-1 flips | Flip rate | KLD p50 | KLD p95 | KLD p99 | Verdict | |---|---|---|---|---|---|---|---|---| | 1 | Qwen3.8-27b-awq-cyanwiki | asymmetric G32 W4A16 | 156 / 7,114 | 2.193% | 0.00003475 | 0.04826 | 0.13570 | Deploy | | 2 | Qwen3.8-27b-nvfp4-unsloth | mixed W8A8/W4A4 | 184 / 7,114 | 2.586% | 0.00004580 | 0.06029 | 0.17359 | Numerically good, operationally awkward | | 3 | Qwen3.8-27b-w4a16-dbirks | symmetric G128 W4A16 | 211 / 7,114 | 2.966% | 0.00007520 | 0.11872 | 0.35675 | W4A16 backup | | 4 | Qwen3.8-27b-awq-philbert440 | asymmetric G128 W4A16 | 241 / 7,114 | 3.388% | 0.00007198 | 0.09133 | 0.24967 | Not bad, but loses at depth | | 5 | Qwen3.8-27b-nvfp4-radixark | mixed W8A8/W4A4 | 319 / 7,114 | 4.484% | 0.00007147 | 0.14041 | 0.53206 | No compelling advantage | | 6 | Qwen3.8-27b-nvfp4-gittensor | W4A4 | 325 / 7,114 | 4.568% | 0.00013996 | 0.18852 | 0.68631 | Skip | | 7 | Qwen3.8-27b-nvfp4-sakamakismile | W4A4 | 451 / 7,114 | 6.340% | 0.00028275 | 0.46118 | 1.37355 | Dumpsterize | There was a clear and obvious winner for the quality comparison: cyanwiki scored well in other tests across other models/weights as well in related testing. | Checkpoint | SP06: 4,339 tokens, 18.8K–83.7K | SP03: 2,775 tokens, 19.9K–123.6K | |---|---|---| | AWQ G32 | 99 flips 2.282% | 57 flips 2.054% | | Unsloth NVFP4 | 105 2.420% | 79 2.847% | | dbirks W4A16 | 117 2.696% | 94 3.387% | | philbert440 AWQ | 127 2.927% | 114 4.108% | | RadixArk NVFP4 | 153 3.526% | 166 5.982% | | GitTensor NVFP4 | 149 3.434% | 176 6.342% | | SakamakiSmile NVFP4 | 278 6.407% | 173 6.234% | The G32 AWQ model wins on both workflows rather than obtaining its aggregate lead from one lucky prompt. The data is not monotonic with context length: a hard open-ended answer near 20K produced far more flips than a more constrained range near 30K. Content and baseline decision margin matter at least as much as raw depth. Unsloth is second by text fidelity, but it is not the a clean W4A16 shape for this worker. It mixes activation-quantized W8A8/W4A4 paths, and at 123K with BF16 KV it failed at the original 98% allocation because too little execution workspace remained. It completed after reserving more headroom and setting the exact context ceiling. Several NVFP4 candidates also forced the FlashInfer GEMM autotuner to reject large-workspace tactics near the 5090 memory ceiling. That behavior is useful real-world evidence, but it makes Unsloth less attractive than the slightly more faithful, simpler G32 W4A16 checkpoint. We took 200 physical-MTG-card scans from a real collection and replayed the exact production bottom-strip card/set/printing identification task. The runner uses the saved dewarped image, crops the bottom 12%, JPEG-encodes at quality 92, disables thinking, and sends Recall’s current set-code/collector-number JSON prompt. The candidate model’s read then receives weight 3 in the real candidate-constrained vote; fixed RapidOCR and Tesseract reads receive weights 2 and 1. This is not generic visual question answering and it is not image-logit KLD. It is the operational question that matters here: did changing the decoder alter what Recall concludes about a scanned Magic card? All four models returned parseable JSON on all 200 images. Runtime coordinate: pinned vllm/vllm-openai:nightly-x86 64 build acb0f1dcd , vLLM 0.27.2rc1.dev110+gacb0f1dcd , eager TP1, BF16 KV, TRITON ATTN full attention, FlashAttention for the vision encoder, Triton/FLA GDN prefill and CUDA GDN decode. All three W4A16 checkpoints resolved to MarlinLinearKernel through CompressedTensors WNA16. BF16, dbirks and Philbert ran on otherwise-idle RTX PRO 6000 Blackwells; Cyanwiki ran on the RTX 5090. Consequently, latency from this quality gate is not used as a product comparison; the planned speed/capacity matrix is same-card. | Coordinate | Raw set exact | Raw collector exact | Raw both exact | Ensemble decisive | Ensemble correct | Correct when decisive | |---|---|---|---|---|---|---| | Tesseract + RapidOCR only | — | — | — | 63 31.5% | 52 26.0% | 82.5% | | Qwen3.8 BF16 | 28 14.0% | 95 47.5% | 28 14.0% | 108 54.0% | 100 50.0% | 92.6% | | Cyanwiki G32 W4A16 | 27 13.5% | 94 47.0% | 27 13.5% | 109 54.5% | 100 50.0% | 91.7% | | dbirks G128 W4A16 | 28 14.0% | 101 50.5% | 28 14.0% | 111 55.5% | 102 51.0% | 91.9% | | Philbert G128 W4A16 | 27 13.5% | 87 43.5% | 27 13.5% | 104 52.0% | 95 47.5% | 91.3% | The absolute raw set-code score looks harsh because this panel deliberately contains 148 same-name reprint groups and the tiny printed set code is often the difficult field. The VLM is not used alone: adding any tested Qwen3.8 coordinate to the two conventional OCR engines increases decisive, correct outcomes substantially over the fixed-OCR-only control. The aggregate two-card lead for dbirks is not statistically or operationally convincing on this sample. Against Cyanwiki, dbirks uniquely fixes five cards and uniquely breaks three; the other 192 outcomes agree. BF16 versus Cyanwiki is exactly six unique wins apiece. Treat the panel as evidence that Cyanwiki did not create a visible regression in this workflow, not as evidence that quantization improves vision. The primary W4A16 checkpoints exclude the visual. tower from weight quantization, but their visual embeddings still flow through different quantized language decoders. These results show why the real prompt, crop, parser and downstream vote are more decision-useful than merely observing that the vision tower stayed BF16. Qwen3.8-27b-awq-cyanwiki .For this mixed agentic-plus-card workload, Cyanwiki remains the best first deployment. If the worker were dedicated only to bottom-strip card OCR, dbirks would merit a larger head-to-head panel before selection; this 200-card tie is not enough reason to accept its clearly weaker long-context text result. Changing any of these produces a new runtime coordinate; it should not inherit the exact percentages in this table without a controlled check. Lets face it, most casual users seem to survive a small number of token flips just fine. Thats why most reviews focus on performance. So lets actually measure the precision lost when performance is gained. This matrix asks what a 5090 owner trades for speed and usable context capacity; it is not part of the checkpoint ranking. Preflight found that the appealing 3-by-2 table is not actually symmetric in the pinned nightly. Cyanwiki is a weight-only checkpoint with kv cache scheme: null ; it contains no static K/V scales. This vLLM build no longer exposes the older --calculate-kv-scales switch. Plain --kv-cache-dtype fp8 therefore falls back to scalar scale 1.0 and warns that the checkpoint did not provide calibrated K/V scaling factors. The build also offers fp8 per token head , which calculates dynamic scales at cache-write time, but only Triton advertises support. FlashInfer accepts scalar fp8 / fp8 e4m3 , not the dynamically scaled mode. FlashAttention 2 accepts BF16 KV on SM120 but rejects FP8 KV: vLLM requires FA3 on SM90 or FA4 on SM100 for that combination. The honest practical panel therefore records seven requested coordinates as six runnable measurements plus one compatibility failure rather than pretending all six backend-by-cache cells mean the same thing: | Cell | Full-attention backend | KV cache | Meaning | |---|---|---|---| | 1 — fidelity reference | Triton | bfloat16 | high-fidelity control | | 2 | FlashAttention 2 | bfloat16 | backend-only change | | 3 | FlashInfer | bfloat16 | backend-only change | | 4 | Triton | fp8 per token head | dynamically scaled practical FP8 | | 5 | Triton | fp8 | naive one-flag, unit-scale FP8 | | 6 | FlashAttention 2 | fp8 | unsupported on SM120; compatibility result | | 7 | FlashInfer | fp8 | naive one-flag, unit-scale FP8 | Cells 5–7 are valuable as a documented pitfall, not as a recommendation. A clean calibrated scalar-FP8 3-by-2 experiment would require creating and freezing a derived checkpoint with calibrated per-layer K/V scales. That is a separate preparation variable and should not be quietly smuggled into a “change one launch flag” deployment table. Hold checkpoint, GPU, runtime image digest, tokenizer/processor, prompt, seed, samplers, context ceiling, concurrency, MTP/reasoning policy, graph policy and all non-attention kernels fixed. Treat a launch as valid only when its log confirms that the requested backend and KV dtype actually resolved; unsupported or silently substituted coordinates are compatibility results, not measurements of the requested cell. Accuracy reference: Cyanwiki G32, Triton, BF16 KV. To finish the arithmetic panel in parallel, all seven quality cells run on one RTX PRO 6000 SM120 proxy; all performance and Recall cells run on the target RTX 5090 SM120. Report per-range Top-1 flips and KLD rather than only a global mean. The proxy keeps architecture and requested kernels fixed but is not an identical board or memory-pressure coordinate, so any near-noise quality delta should be repeated on the target 5090 before making a strong claim. The frozen accuracy slice contains 3,512 recorded assistant-output tokens in five natural ranges at roughly 18.8K, 27.5K, 42.2K, 42.8K and 48.6K context. It stops at 50,196 tokens, avoiding the 5090 capacity edge while still testing diagnostic prose, parallel tool calls, a mutating call, failure recovery and a 1,570-token Python migration-script tool call. The same 200-card Recall panel should also be replayed for every supported/recommended cell; that exposes a vision-workflow regression that text KLD alone cannot see. Practical 32K table columns: Benchmarking must warm model load, JIT/autotuning and the selected graph shapes before timed repetitions. Record all repetitions rather than publishing only the best. The launch recipe should be one pinned container command with explicit pick-one substitutions for attention backend and KV dtype. It must reject unsupported FA2/FlashInfer plus fp8 per token head combinations and retain the unit-scale warning in the evidence for plain fp8 cells. Performance scope: exact-token-targeted 32K context, C1 sustained decode, 60-second measured cells, a hidden decode warmup, one exact 32K cold-prefill sample per repetition and three repetitions per coordinate. There is no prefill-depth sweep in this practical comparison. Use the benchmark at commit 0b4185b5b435e948b199c9077a00b084864aa963 ; publish both its JSON and the server’s resolved-runtime log. Quality capture uses the same graph policy, prefix-cache policy, max num seqs=4 , scheduler token budget and dtypes as the practical server rather than inheriting the earlier eager-only checkpoint bakeoff. Its 96-GB proxy necessarily has a larger physical KV allocation; capacity numbers come only from the actual 5090 performance cells. Before execution, cross-check this seven-cell plan and the exact launch flags against the local community llm-inference-bench tooling so its PP/TG definitions, prompt construction and warmup behavior are captured precisely. The completed matrix used one exact 32,768-token integrated prefill scout, a hidden 10-second decode warmup, and a 60-second concurrency-one decode window. Each runnable coordinate was repeated three times. The values below are medians; the raw JSON retains all repetitions. | Coordinate | PP32K tok/s | TG32K C1 tok/s | Server KV budget | KV-token capacity | Top-1 flips vs Triton/BF16 | Per-range KLD p95 | Recall correct / decisive | |---|---|---|---|---|---|---|---| | Triton + BF16 KV | 2,614 | 66.4 | 8.21 GiB | 120,897 | reference | reference | 100 / 109 | | FA2 + BF16 KV | 3,179 | 70.3 | 8.21 GiB | 120,897 | 19 / 3,512 0.541% | 0.000295–0.002075 | 98 / 107 | | FlashInfer + BF16 KV | 3,157 | 69.8 | 7.88 GiB | 115,948 | 14 / 3,512 0.399% | 0.000579–0.002123 | 100 / 108 | | Triton + dynamically scaled FP8 KV | 2,117 | 57.9 | 8.21 GiB | 218,807 | 22 / 3,512 0.626% | 0.001278–0.003927 | 99 / 108 | | Triton + unit-scale FP8 KV | 2,824 | 70.8 | 8.18 GiB | 218,807 | 29 / 3,512 0.826% | 0.001806–0.007343 | 99 / 108 | | FA2 + unit-scale FP8 KV | unsupported | unsupported | — | — | — | — | — | | FlashInfer + unit-scale FP8 KV | 3,121 | 73.6 | 7.88 GiB | 211,084 | 18 / 3,512 0.513% | 0.001573–0.006023 | 99 / 107 | The exact Triton/BF16 repeat control changed Top-1 at 13 of 3,512 positions 0.370% . Its requested configuration and Triton source, PTX and IR hashes were identical. This is an empirical same-path repeatability floor, not a correction that can simply be subtracted from every candidate. It does mean that the 14 FlashInfer/BF16 flips are effectively at the observed floor, while 19 FA2/BF16 flips are only modestly above it. The FP8 cells show clearer distributional movement, especially Triton’s uncalibrated unit-scale cache. The large practical capacity gain is 1.81x for Triton FP8 versus Triton BF16 218,807 versus 120,897 usable KV tokens , not a full 2x after graph and runtime workspace are accounted for. Dynamic per-token/head scaling was the higher-integrity FP8 method, but it cost about 19% of Triton/BF16 prefill and 13% of decode throughput. The one-flag unit-scale paths were faster, but the model provides no calibrated K/V scales, so their surprisingly good aggregate scores should not be generalized into a safety claim. On this exact 5090 coordinate, FA2/BF16 is the strongest conventional speed/capacity choice: about 21.6% faster prefill and 5.9% faster decode than Triton/BF16 while retaining the same server KV allocation. FlashInfer/BF16 is very close in speed and closest to the observed repeatability floor, but its workspace left about 4.1% fewer usable KV tokens. Keep Triton/BF16 when fidelity continuity with the original quant ranking matters more than the small throughput gain. All six runnable cells returned 200/200 valid JSON objects on the Recall card panel. Their final ensemble score ranged only from 98 to 100 correct cases, so that panel found no decision-useful visual regression among these runtime settings. It does not vindicate unit-scale FP8 KV for unrelated agentic or high-stakes workloads. This is the tested server shape. Mount the frozen cyankiwi/Qwen3.8-27B-AWQ-INT4 revision at /models/qwen38-cyanwiki , then make only the indicated pick-one substitutions: IMAGE='vllm/vllm-openai@sha256:0db5553091d59de260f67696fff89026e7376ec0d326111ba78fb004a27778a0' ATTENTION BACKEND='FLASH ATTN' TRITON ATTN | FLASH ATTN | FLASHINFER KV DTYPE='bfloat16' bfloat16 | fp8 | fp8 per token head docker run --rm --gpus 'device=0' --network host --ipc host --shm-size 32g \ -v /absolute/path/to/Qwen3.8-27B-AWQ-INT4:/models/qwen38-cyanwiki:ro \ -e HF HUB OFFLINE=1 -e TRANSFORMERS OFFLINE=1 -e VLLM NO USAGE STATS=1 \ "$IMAGE" /models/qwen38-cyanwiki \ --served-model-name qwen38-cyanwiki \ --tensor-parallel-size 1 \ --dtype bfloat16 \ --kv-cache-dtype "$KV DTYPE" \ --attention-backend "$ATTENTION BACKEND" \ --gdn-prefill-backend triton \ --max-model-len 50197 \ --max-num-seqs 4 \ --max-num-batched-tokens 8192 \ --gpu-memory-utilization 0.95 \ --enable-prefix-caching \ --seed 1 \ --trust-remote-code \ --host 127.0.0.1 --port 8000 Compatibility is not a Cartesian product. On this SM120 stack, fp8 per token head is Triton-only and FLASH ATTN plus FP8 KV is rejected. Plain fp8 is the uncalibrated unit-scale case documented above. The practical default from this panel is therefore FLASH ATTN plus bfloat16 ; use TRITON ATTN plus bfloat16 to reproduce the fidelity-reference coordinate. The checkpoint’s quantized decoder linears resolved to MarlinLinearKernel through CompressedTensors WNA16 in every cell. The backend switch applied to the 16 decoder full-attention layers: TRITON ATTN , FlashAttention version 2, or FlashInfer. FlashInfer resolved BF16 prefill and decode queries to its XQA decode backend on sm120 ; with plain FP8 KV the cache dtype resolved to torch.float8 e4m3fn . Qwen3.8 is hybrid, so those labels do not describe the whole model. Its 48 GatedDeltaNet layers remained on Triton/FLA prefill and the CUDA recurrent decode implementation in all cells. The vision encoder separately used FlashAttention, and top-k/top-p sampling separately used FlashInfer. Thus “Triton versus FA2 versus FlashInfer” here means the decoder’s full-attention backend only; it is not a claim that every CUDA kernel in the process changed. FA2/FP8 failed before serving with the explicit vLLM validation error that FP8 KV requires FA3 on SM90 or FA4 on SM100. No silent fallback result is included. In this particular use case, this particular model, and these specific quants: There is a clear quality winner. If you care more cache size than absolute precision, I can safely drop down to FP8 here knowing the cost. I am still working to figure out how to bundle up my datasets and testing at the larger scale but this one shook out, was a natural question people might ask, and is timely given the new weights dropping.