{"slug": "quickreduce-int3-quantization-and-benchmarking-on-mi355", "title": "QuickReduce INT3 Quantization and Benchmarking on MI355", "summary": "AMD researchers implemented INT3 quantization in QuickReduce, a high-performance all-reduce library for ROCm, achieving a 22% reduction in on-wire data volume compared to INT4 on MI355 GPUs. The INT3 mode trades compression ratio for lower communication latency, with model-dependent accuracy trade-offs, and is integrated into vLLM for inference.", "body_md": "# QuickReduce INT3 Quantization and Benchmarking on MI355[#](#quickreduce-int3-quantization-and-benchmarking-on-mi355)\n\nLarge Language Models (LLMs) typically contain billions — or even tens of billions — of parameters. During inference, tensor parallelism (TP) is a widely used technique that distributes the compute across multiple GPUs. This approach, however, requires frequent, large-scale data synchronization between layers, introducing significant communication latency and placing enormous pressure on interconnect bandwidth.\n\nAmong the various communication patterns, **all-reduce** stands out as one of the most critical. It aggregates data from all participating devices (for example, via summation) and broadcasts the result back to each device, enabling synchronized multi-GPU computation.\n\n[QuickReduce](https://github.com/mk1-project/quickreduce/) is a high-performance all-reduce library designed for AMD ROCm that supports **inline compression**: the send path quantizes the data, the reduction is performed in the compressed domain, and the receive path dequantizes the result. Compared to RCCL (AMD’s collective communication primitives for multi-GPU and multi-node communication), QuickReduce achieves up to **2.25x speedup** on 2×MI300X and 4×MI300X configurations, and outperforms RCCL across all multi-GPU (single-node) configurations when optimized, as reported in [our previous work](https://rocm.blogs.amd.com/artificial-intelligence/quick-reduce/README.html).\n\nIn [our previous work](https://rocm.blogs.amd.com/artificial-intelligence/quick-reduce/README.html), we discussed the design principles and performance characteristics of QuickReduce in detail, integrated it into the popular inference frameworks [vLLM](https://github.com/vllm-project/vllm) and [SGLang](https://github.com/sgl-project/sglang), and provided comprehensive performance and accuracy benchmarks for integer quantization (INT8/INT6/INT4) on MI300.\n\nIn a [subsequent post](https://rocm.blogs.amd.com/artificial-intelligence/quick-reduce-2/README.html), we further introduced a hardware-accelerated FP4 quantization path on MI355 and presented updated performance and accuracy evaluations, showing that on MI355 FP4 and INT4 are essentially interchangeable.\n\nIn this blog post, we go one step further and implement the **INT3** quantization mode of QuickReduce. INT3 trades a higher compression ratio for a smaller on-wire transfer volume. We describe its design and vLLM integration, and demonstrate its latency advantage over INT4 and its model-dependent accuracy on AMD Instinct MI355.\n\n## QuickReduce with INT3: Design and Implementation[#](#quickreduce-with-int3-design-and-implementation)\n\n### Design Highlights[#](#design-highlights)\n\nINT3 reuses the same “block-level FP16 scale + symmetric quantization” framework as INT4, with the key difference that each value is represented in just **3 bits**:\n\nEach quantization block covers 32 FP16/BF16 values (8 threads × 4\n\n`fp16x2`\n\nlanes) that share one FP16 scale.Values are scaled, clipped to the symmetric range [-4, +3], rounded to integers, and biased to the unsigned domain [0, 7] for packing.\n\nUnlike INT4 (eight 4-bit values packed into one\n\n`int32`\n\n), INT3 splits each 3-bit value into a low and a high part:low 2 bits → one\n\n`uint16`\n\nper thread (denoted`q2w`\n\n)high 1 bit → one\n\n`uint8`\n\nper thread (denoted`q1w`\n\n)\n\nEach group of 32 values shares one decoding scale, written by the thread-group leader.\n\nThis packing shrinks the per-rank transmitted tile from 1152 bytes (INT4) to **896 bytes**, a reduction of about **22%**.\nThe smaller on-wire data directly reduces the communication volume of the two-shot QuickReduce kernel in Phase-1/Phase-2;\nthis benefit only takes effect once the message length exceeds the vLLM activation threshold.\n\n### Quantization and Dequantization Paths[#](#quantization-and-dequantization-paths)\n\nThe send path is implemented in `CodecQ3::send`\n\n(`csrc/quickreduce/quick_reduce_impl.cuh`\n\n): it first computes the absolute maximum within the thread group and derives the FP16 encode/decode scale; it then scales and clips the values to [-4, +3], rounds them, and biases them to [0, 7]; finally it splits each 3-bit value into `q2w`\n\nand `q1w`\n\n, writes them into the communication buffer, and the group leader writes the block scale.\n\nThe receive path is implemented in `CodecQ3::recv`\n\n: it reads `q2w`\n\n, `q1w`\n\n, and the scale from the buffer using non-temporal loads, recombines the low 2 bits and high 1 bit into an unsigned quantized value, maps it back to the symmetric integer domain, and multiplies by the decoding scale to reconstruct FP16/BF16. The two-shot kernel calls `recv`\n\nduring Phase-1B (in-segment reduction) and Phase-2 (gather), consistent with the other QuickReduce quantization modes.\n\n## QuickReduce INT3 Benchmarking on MI355[#](#quickreduce-int3-benchmarking-on-mi355)\n\n### Kernel Performance[#](#kernel-performance)\n\nAll experiments in this post were run on a **single-node** MI355 server (8× AMD Instinct MI355X GPUs in one node), so every TP configuration below uses GPUs within that single node (for the full software and system configuration, see the [Appendix](#appendix-configuration-details)). On this server we benchmark three all-reduce implementations:\n\n**RCCL****Custom AllReduce (CR)**— A widely adopted all-reduce acceleration kernel used in vLLM and SGLang, which significantly outperforms RCCL at low data volumes.**QuickReduce (QR)**— This post focuses on comparing the** INT3**and** INT4**quantization configurations.\n\nThe tests cover message sizes ranging from **4 KB to 1 GB**, and all latency values are reported in **μs (microseconds)**.\n\nTo better visualize relative performance across the full message-size range, we plot **speedup over RCCL** rather than raw latency. The y-axis is defined as **RCCL latency ÷ each method’s latency**, so a value greater than 1 indicates better performance than RCCL.\n\n#### TP = 2[#](#tp-2)\n\nThe figure below shows how the TP=2 speedup over RCCL varies with message size, highlighting where QuickReduce INT3 and INT4 overtake CR.\n\n#### TP > 2[#](#id1)\n\nBased on our kernel experiments, INT3 and INT4 perform very similarly under TP=4 and TP=8. We believe that as TP grows, the number of GPUs that must communicate with one another increases multiplicatively, requiring more synchronization and scheduling time; in this regime, synchronization and latency overheads — rather than the transferred data volume — increasingly become the bottleneck, so the transfer time saved by further bit-width compression (INT3) becomes a smaller fraction of the total and the benefit is diluted.\n\nBecause INT3 compresses the bit-width more aggressively and is more sensitive to accuracy, taking both performance and accuracy into account, we do not recommend using INT3 outside of TP=2.\n\n### Key Observations[#](#key-observations)\n\nFrom the results, we draw the following conclusions:\n\n**For message sizes beyond the crossover point, QuickReduce delivers the highest large-message speedup over RCCL**: at a 1 GB message size, QR INT3 reaches roughly** 5x speedup**at TP=2.** Compared to INT4, INT3 only shows a clear additional speedup at large communication volumes**(above ~4 MB); for smaller messages the two are essentially on par.\n\n### End-to-End Performance and Accuracy[#](#end-to-end-performance-and-accuracy)\n\nWe selected five representative models spanning both Dense (Qwen3-32B, Qwen2.5-72B-Instruct) and MoE (DeepSeek-V4-Flash, MiniMax-M2.5, Qwen3-235B-A22B) architectures, and used vLLM for both performance and accuracy evaluation.\nAs with INT4, we set a **1 MB activation threshold** for INT3: even with QR INT3 enabled, the QR INT3 communication path is only used when a single communication exceeds 1 MB;\nbelow 1 MB, the framework falls back to RCCL or CR.\n\n**Server launch command:**\n\n```\nVLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT3 vllm serve <model_path> \\\n    --no-enable-prefix-caching \\\n    --trust-remote-code \\\n    -tp 2 \\\n    --dtype auto \\\n    --port 12340\n```\n\n**Client commands:**\n\n```\n# Accuracy evaluation\npython3 vllm/tests/evals/gsm8k/gsm8k_eval.py --port 12340\n\n# Performance benchmark\nvllm bench serve \\\n    --model <model_path> \\\n    --backend vllm \\\n    --endpoint /v1/completions \\\n    --dataset-name random \\\n    --random-input-len 4096 \\\n    --random-output-len 10 \\\n    --num-prompts 500 \\\n    --ignore-eos \\\n    --request-rate inf \\\n    --max-concurrency 128 \\\n    --port 12340\n```\n\nAll end-to-end results below were collected at **TP=2** under a throughput-oriented load (4096-token random inputs, `--request-rate inf`\n\n, `--max-concurrency 128`\n\n). For each model we compare the no-quantization baseline (NONE), INT4, and INT3. Speedup and throughput gain are normalized to the NONE baseline, and **Accuracy Recovery** is GSM8K accuracy relative to the NONE baseline.\n\n#### DeepSeek-V4-Flash (MoE)[#](#deepseek-v4-flash-moe)\n\nQuantization |\nTTFT (ms) |\nTPOT (ms) |\nOutput Throughput (tok/s) |\nTTFT Speedup |\nTPOT Speedup |\nThroughput Gain |\nGSM8K |\nAccuracy Recovery |\n|---|---|---|---|---|---|---|---|---|\nNONE |\n26226.98 |\n519.49 |\n37.50 |\n1.000 |\n1.000 |\n1.000 |\n0.9447 |\n1.0000 |\nINT4 |\n24329.79 |\n479.63 |\n40.55 |\n1.078 |\n1.083 |\n1.081 |\n0.9443 |\n0.9996 |\n|\n|\n|\n|\n|\n|\n|\n0.9373 |\n0.9922 |\n\n#### Qwen3-32B (Dense)[#](#qwen3-32b-dense)\n\nQuantization |\nTTFT (ms) |\nTPOT (ms) |\nOutput Throughput (tok/s) |\nTTFT Speedup |\nTPOT Speedup |\nThroughput Gain |\nGSM8K |\nAccuracy Recovery |\n|---|---|---|---|---|---|---|---|---|\nNONE |\n25671.79 |\n513.31 |\n37.94 |\n1.000 |\n1.000 |\n1.000 |\n0.6200 |\n1.0000 |\nINT4 |\n19903.11 |\n395.44 |\n48.83 |\n1.290 |\n1.298 |\n1.287 |\n0.6670 |\n1.0758 |\n|\n|\n|\n|\n|\n|\n|\n0.5867 |\n0.9462 |\n\n#### Qwen2.5-72B-Instruct (Dense)[#](#qwen2-5-72b-instruct-dense)\n\nQuantization |\nTTFT (ms) |\nTPOT (ms) |\nOutput Throughput (tok/s) |\nTTFT Speedup |\nTPOT Speedup |\nThroughput Gain |\nGSM8K |\nAccuracy Recovery |\n|---|---|---|---|---|---|---|---|---|\nNONE |\n48670.30 |\n971.36 |\n20.06 |\n1.000 |\n1.000 |\n1.000 |\n0.7120 |\n1.0000 |\nINT4 |\n35127.15 |\n698.54 |\n27.69 |\n1.386 |\n1.391 |\n1.380 |\n0.7290 |\n1.0239 |\n|\n|\n|\n|\n|\n|\n|\n0.7207 |\n1.0122 |\n\n#### MiniMax-M2.5 (MoE)[#](#minimax-m2-5-moe)\n\nQuantization |\nTTFT (ms) |\nTPOT (ms) |\nOutput Throughput (tok/s) |\nTTFT Speedup |\nTPOT Speedup |\nThroughput Gain |\nGSM8K |\nAccuracy Recovery |\n|---|---|---|---|---|---|---|---|---|\nNONE |\n18668.83 |\n372.06 |\n52.06 |\n1.000 |\n1.000 |\n1.000 |\n0.9317 |\n1.0000 |\nINT4 |\n15575.94 |\n304.56 |\n62.67 |\n1.199 |\n1.222 |\n1.204 |\n0.9307 |\n0.9989 |\n|\n|\n|\n|\n|\n|\n|\n0.9273 |\n0.9953 |\n\n#### Qwen3-235B-A22B (MoE)[#](#qwen3-235b-a22b-moe)\n\nQuantization |\nTTFT (ms) |\nTPOT (ms) |\nOutput Throughput (tok/s) |\nTTFT Speedup |\nTPOT Speedup |\nThroughput Gain |\nGSM8K |\nAccuracy Recovery |\n|---|---|---|---|---|---|---|---|---|\nNONE |\n33496.04 |\n673.56 |\n28.88 |\n1.000 |\n1.000 |\n1.000 |\n0.9067 |\n1.0000 |\nINT4 |\n26412.86 |\n531.00 |\n36.50 |\n1.268 |\n1.268 |\n1.264 |\n0.9057 |\n0.9989 |\n|\n|\n|\n|\n|\n|\n|\n0.9117 |\n1.0055 |\n\n### Analysis[#](#analysis)\n\nOn performance, QR INT3 provides a further speedup over QR INT4 because it puts less data on the wire. Across the five models, relative to INT4, INT3 reduces TTFT by about **3.6%** on average (up to ~5% on Qwen2.5-72B-Instruct), reduces TPOT by about **3.7%**, and improves output throughput by about **3.7%**. The TTFT and TPOT gains are comparable here: under this throughput-oriented load (many concurrent requests), the decode phase also accumulates substantial all-reduce traffic above the activation threshold, so it benefits from the smaller INT3 payload as well — not just the prefill phase. These gains come on top of the speedup INT4 already delivers over the no-quantization baseline.\n\nOn accuracy, INT3 preserves accuracy well across the MoE models and the stronger dense model; the only model with a clearly visible drop is Qwen3-32B. Our analysis is as follows:\n\nOn one hand, compared to INT4, INT3 has fewer quantization levels (8 vs 16) and therefore injects more quantization noise. The three MoE models (DeepSeek-V4-Flash, MiniMax-M2.5, and Qwen3-235B-A22B) are the most robust — each retains at least 99% of its baseline accuracy (Accuracy Recovery 0.9922, 0.9953, and 1.0055, respectively). One likely explanation is that, for MoE models, the output is a weighted sum of multiple experts; combined with their larger parameter redundancy and hidden dimensions, this may further dilute and average out the quantization noise during computation, making them less sensitive.\n\nOn the other hand, the determining factor is the strength of the baseline rather than the Dense-vs-MoE distinction. Qwen3-32B is the only model with a notable drop (Accuracy Recovery 0.9462), and it already has a low GSM8K baseline (0.62), indicating limited inherent capability: many problems sit near the decision boundary, so the extra noise introduced by INT3 can easily flip the model’s answers. By contrast, the other dense model, Qwen2.5-72B-Instruct, has a stronger baseline (0.712) and is essentially unaffected (Accuracy Recovery 1.0122). For models that are inherently strong and have high baseline scores, most problems are answered with high confidence, and 3-bit noise is not enough to change the correct outcome.\n\n## Summary[#](#summary)\n\nIn this blog post, we benchmarked the INT3 quantization configuration of QuickReduce on MI355 for both performance and accuracy.\n\nFrom the kernel tests: under TP=2, INT3 is clearly faster than INT4 starting from about 4 MB, and reaches roughly 5x speedup over RCCL on 1 GB large messages; however, under TP=4 / TP=8, INT3 and INT4 converge.\n\nFrom the end-to-end tests (TP=2, throughput-oriented load): compared to INT4, INT3 reduces TTFT by about 3.6% and TPOT by about 3.7% on average, while improving output throughput by about 3.7%.\n\nOn accuracy, based on the models evaluated in this post, INT3 performed best on the tested MoE models and on models with stronger baseline accuracy; on models with weaker baselines (such as Qwen3-32B in our tests), the added 3-bit noise can flip near-boundary answers.\n\n## Appendix - Configuration Details[#](#appendix-configuration-details)\n\nFor full reproducibility, the exact software stack and system configuration used for all benchmarks in this post are listed below.\n\nResults were obtained using a system configured with AMD Instinct™ MI355X GPUs. Tests for vLLM were conducted by AMD on June 10, 2026. Actual results may vary depending on system configuration, usage, software version, and applied optimizations.\n\n### Software Setup for vLLM[#](#software-setup-for-vllm)\n\n```\nHIPBLASLT_BRANCH: 1.2.2.70203-90~22.04 (libhipblaslt.so.1.2.70203)\nTRITON_BRANCH: 3.6.0\nTRITON_REPO: https://github.com/triton-lang/triton.git\nPYTORCH_BRANCH: 2.10.0+git8514f05\nPYTORCH_REPO: https://github.com/ROCm/pytorch.git (commit 8514f05131610dab50233027b2fab9c01235081b)\nPYTORCH_VISION_BRANCH: 0.24.1+d801a34\nPYTORCH_VISION_REPO: https://github.com/pytorch/vision.git\nAITER_BRANCH: 0.1.13 (amd_aiter)\nAITER_REPO: https://github.com/ROCm/aiter.git\nVLLM_BRANCH: 0.20.2rc1.dev713+ge380238bd.d20260609\nVLLM_REPO: https://github.com/vllm-project/vllm\nROCm: 7.2.3 (HIP runtime 7.2.53211-c2d9476115)\n```\n\n### System Configuration[#](#system-configuration)\n\n```\nAMD Instinct MI355X platform (gfx950)\nCPU: 2x AMD EPYC 9575F 64-Core Processor (256 logical CPUs)\nNUMA: 2 NUMA node(s), 1 NUMA node per socket\n    node0 CPUs: 0-63,128-191\n    node1 CPUs: 64-127,192-255\nMemory: ~3.0 TiB total\nDisk: 1x Micron 7450 14 TB NVMe + 2x Micron 7450 3.5 TB NVMe\nGPU: 8x AMD Instinct MI355X (gfx950, 256 CUs each, ~288 GiB HBM each — VRAM total 309 GB/GPU reported)\nHost OS: Ubuntu 22.04.5 LTS (Jammy)\nHost Kernel: 5.15.0-144-generic\nHost GPU Driver (amdgpu / KFD): 6.14.14\n```\n\n## Disclaimers[#](#disclaimers)\n\nThe information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, ROCm, Instinct, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved", "url": "https://wpnews.pro/news/quickreduce-int3-quantization-and-benchmarking-on-mi355", "canonical_source": "https://rocm.blogs.amd.com/artificial-intelligence/quick-reduce-3/README.html", "published_at": "2026-07-13 00:00:00+00:00", "updated_at": "2026-07-13 16:23:19.900933+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-research"], "entities": ["AMD", "QuickReduce", "MI355", "ROCm", "vLLM", "SGLang", "MI300X", "RCCL"], "alternates": {"html": "https://wpnews.pro/news/quickreduce-int3-quantization-and-benchmarking-on-mi355", "markdown": "https://wpnews.pro/news/quickreduce-int3-quantization-and-benchmarking-on-mi355.md", "text": "https://wpnews.pro/news/quickreduce-int3-quantization-and-benchmarking-on-mi355.txt", "jsonld": "https://wpnews.pro/news/quickreduce-int3-quantization-and-benchmarking-on-mi355.jsonld"}}