QuickReduce INT3 Quantization and Benchmarking on MI355 AMD researchers implemented INT3 quantization in QuickReduce, a high-performance all-reduce library for ROCm, achieving a 22% reduction in on-wire data volume compared to INT4 on MI355 GPUs. The INT3 mode trades compression ratio for lower communication latency, with model-dependent accuracy trade-offs, and is integrated into vLLM for inference. QuickReduce INT3 Quantization and Benchmarking on MI355 quickreduce-int3-quantization-and-benchmarking-on-mi355 Large Language Models LLMs typically contain billions — or even tens of billions — of parameters. During inference, tensor parallelism TP is a widely used technique that distributes the compute across multiple GPUs. This approach, however, requires frequent, large-scale data synchronization between layers, introducing significant communication latency and placing enormous pressure on interconnect bandwidth. Among the various communication patterns, all-reduce stands out as one of the most critical. It aggregates data from all participating devices for example, via summation and broadcasts the result back to each device, enabling synchronized multi-GPU computation. QuickReduce https://github.com/mk1-project/quickreduce/ is a high-performance all-reduce library designed for AMD ROCm that supports inline compression : the send path quantizes the data, the reduction is performed in the compressed domain, and the receive path dequantizes the result. Compared to RCCL AMD’s collective communication primitives for multi-GPU and multi-node communication , QuickReduce achieves up to 2.25x speedup on 2×MI300X and 4×MI300X configurations, and outperforms RCCL across all multi-GPU single-node configurations when optimized, as reported in our previous work https://rocm.blogs.amd.com/artificial-intelligence/quick-reduce/README.html . In our previous work https://rocm.blogs.amd.com/artificial-intelligence/quick-reduce/README.html , we discussed the design principles and performance characteristics of QuickReduce in detail, integrated it into the popular inference frameworks vLLM https://github.com/vllm-project/vllm and SGLang https://github.com/sgl-project/sglang , and provided comprehensive performance and accuracy benchmarks for integer quantization INT8/INT6/INT4 on MI300. In a subsequent post https://rocm.blogs.amd.com/artificial-intelligence/quick-reduce-2/README.html , we further introduced a hardware-accelerated FP4 quantization path on MI355 and presented updated performance and accuracy evaluations, showing that on MI355 FP4 and INT4 are essentially interchangeable. In this blog post, we go one step further and implement the INT3 quantization mode of QuickReduce. INT3 trades a higher compression ratio for a smaller on-wire transfer volume. We describe its design and vLLM integration, and demonstrate its latency advantage over INT4 and its model-dependent accuracy on AMD Instinct MI355. QuickReduce with INT3: Design and Implementation quickreduce-with-int3-design-and-implementation Design Highlights design-highlights INT3 reuses the same “block-level FP16 scale + symmetric quantization” framework as INT4, with the key difference that each value is represented in just 3 bits : Each quantization block covers 32 FP16/BF16 values 8 threads × 4 fp16x2 lanes that share one FP16 scale.Values are scaled, clipped to the symmetric range -4, +3 , rounded to integers, and biased to the unsigned domain 0, 7 for packing. Unlike INT4 eight 4-bit values packed into one int32 , INT3 splits each 3-bit value into a low and a high part:low 2 bits → one uint16 per thread denoted q2w high 1 bit → one uint8 per thread denoted q1w Each group of 32 values shares one decoding scale, written by the thread-group leader. This packing shrinks the per-rank transmitted tile from 1152 bytes INT4 to 896 bytes , a reduction of about 22% . The smaller on-wire data directly reduces the communication volume of the two-shot QuickReduce kernel in Phase-1/Phase-2; this benefit only takes effect once the message length exceeds the vLLM activation threshold. Quantization and Dequantization Paths quantization-and-dequantization-paths The send path is implemented in CodecQ3::send csrc/quickreduce/quick reduce impl.cuh : it first computes the absolute maximum within the thread group and derives the FP16 encode/decode scale; it then scales and clips the values to -4, +3 , rounds them, and biases them to 0, 7 ; finally it splits each 3-bit value into q2w and q1w , writes them into the communication buffer, and the group leader writes the block scale. The receive path is implemented in CodecQ3::recv : it reads q2w , q1w , and the scale from the buffer using non-temporal loads, recombines the low 2 bits and high 1 bit into an unsigned quantized value, maps it back to the symmetric integer domain, and multiplies by the decoding scale to reconstruct FP16/BF16. The two-shot kernel calls recv during Phase-1B in-segment reduction and Phase-2 gather , consistent with the other QuickReduce quantization modes. QuickReduce INT3 Benchmarking on MI355 quickreduce-int3-benchmarking-on-mi355 Kernel Performance kernel-performance All experiments in this post were run on a single-node MI355 server 8× AMD Instinct MI355X GPUs in one node , so every TP configuration below uses GPUs within that single node for the full software and system configuration, see the Appendix appendix-configuration-details . On this server we benchmark three all-reduce implementations: RCCL Custom AllReduce CR — A widely adopted all-reduce acceleration kernel used in vLLM and SGLang, which significantly outperforms RCCL at low data volumes. QuickReduce QR — This post focuses on comparing the INT3 and INT4 quantization configurations. The tests cover message sizes ranging from 4 KB to 1 GB , and all latency values are reported in μs microseconds . To better visualize relative performance across the full message-size range, we plot speedup over RCCL rather than raw latency. The y-axis is defined as RCCL latency ÷ each method’s latency , so a value greater than 1 indicates better performance than RCCL. TP = 2 tp-2 The figure below shows how the TP=2 speedup over RCCL varies with message size, highlighting where QuickReduce INT3 and INT4 overtake CR. TP 2 id1 Based on our kernel experiments, INT3 and INT4 perform very similarly under TP=4 and TP=8. We believe that as TP grows, the number of GPUs that must communicate with one another increases multiplicatively, requiring more synchronization and scheduling time; in this regime, synchronization and latency overheads — rather than the transferred data volume — increasingly become the bottleneck, so the transfer time saved by further bit-width compression INT3 becomes a smaller fraction of the total and the benefit is diluted. Because INT3 compresses the bit-width more aggressively and is more sensitive to accuracy, taking both performance and accuracy into account, we do not recommend using INT3 outside of TP=2. Key Observations key-observations From the results, we draw the following conclusions: For message sizes beyond the crossover point, QuickReduce delivers the highest large-message speedup over RCCL : at a 1 GB message size, QR INT3 reaches roughly 5x speedup at TP=2. Compared to INT4, INT3 only shows a clear additional speedup at large communication volumes above ~4 MB ; for smaller messages the two are essentially on par. End-to-End Performance and Accuracy end-to-end-performance-and-accuracy We selected five representative models spanning both Dense Qwen3-32B, Qwen2.5-72B-Instruct and MoE DeepSeek-V4-Flash, MiniMax-M2.5, Qwen3-235B-A22B architectures, and used vLLM for both performance and accuracy evaluation. As with INT4, we set a 1 MB activation threshold for INT3: even with QR INT3 enabled, the QR INT3 communication path is only used when a single communication exceeds 1 MB; below 1 MB, the framework falls back to RCCL or CR. Server launch command: VLLM ROCM QUICK REDUCE QUANTIZATION=INT3 vllm serve