{"slug": "gpu-initiated-dma-copy-engines-in-rccl-and-rocshmem", "title": "GPU-Initiated DMA Copy Engines in RCCL and rocSHMEM", "summary": "AMD reported that GPU-initiated SDMA copy engines in RCCL and rocSHMEM achieve large-message bandwidth comparable to or higher than default implementations while using one compute unit instead of 64, tested on eight-GPU Instinct MI350X systems. In rocSHMEM putnbi tests, SDMA delivered the highest bandwidth among tested configurations for messages of 512 KiB and larger using one CU and one wavefront. The capability lets GPU kernels issue transfers while dedicated SDMA engines move data, freeing compute units for computation in fused compute-and-communication kernels such as inference workloads.", "body_md": "# GPU-Initiated DMA Copy Engines in RCCL and rocSHMEM[#](#gpu-initiated-dma-copy-engines-in-rccl-and-rocshmem)\n\nAs GPU-accelerated workloads scale across multiple devices and nodes, the efficiency of inter-GPU communication becomes a decisive factor in overall application performance. This is especially true for inference workloads, where communication sits on the critical path and traditional approaches to overlapping compute and communication fall short without fine-grained, kernel-level orchestration.\n\nIn this blog, you will explore a powerful new capability available in both RCCL and rocSHMEM: GPU-initiated SDMA (System DMA) copy engines for intra-node data movement. You will learn how this approach decouples data transfer from compute unit execution, freeing up GPU resources for computation while still saturating available bandwidth. Through performance results collected on AMD Instinct™ MI350X hardware, you will see how GPU-initiated SDMA achieves bandwidth comparable to or exceeding traditional implementations, using as few as a single compute unit instead of 64.\n\nDevice-side APIs are critical because they enable the fusion of computation and communication. This is a capability that is particularly essential for optimizing inference workloads, where communication lies on the critical path and where overlapping compute and communication is impractical without fine-grained orchestration between the two.\n\nRecently, RCCL has introduced support for GPU-initiated communication, enabling data transfers within the scale-up domain through regular load/store operations as well as across nodes through GPU-initiated networking (GIN). rocSHMEM was architected with this capability from the outset. Both GPU-initiated communication APIs provide a clean abstraction to access hardware features that enable high efficiency scale-up data transfers, maximizing bandwidth while using fewer resources.\n\n## Highlights[#](#highlights)\n\n- High bandwidth with fewer compute units (CUs): in eight-GPU MI350X tests, GPU-initiated SDMA achieves comparable or higher large-message bandwidth using one versus 64 CUs for default RCCL `AlltoAll` and`AllGather` . Similarly, in rocSHMEM`putnbi` tests, SDMA delivers the highest bandwidth among tested configurations for messages 512 KiB and larger, using one CU and one wavefront.\n- More resources for computation: GPU kernels initiate transfers while dedicated SDMA engines move the data, reducing CU usage in fused compute-and-communication kernels.\n- Demonstrated across libraries: generic solution that works across libraries.\n\n## A New Approach: GPU-Initiated DMA Copy Engines[#](#a-new-approach-gpu-initiated-dma-copy-engines)\n\nGPUs can move data using either CUs, which execute load/store instructions, or dedicated DMA copy engines called SDMA on AMD hardware. A separate distinction is who initiates the transfer: the CPU or a running GPU kernel.\n\n| Transfer approach | Who initiates the transfer? | Who performs the data movement? | CU involvement | \n|---|---|---|---|\n| GPU load/store communication | GPU kernel | CUs executing load/store instructions | CUs execute the copying work | \n| Host-initiated SDMA | CPU | SDMA engines | CUs do not execute the copy | \n| GPU-initiated SDMA | GPU kernel | SDMA engines | CUs issue commands and coordinate completion | \n\nWhile GPU-initiated communication itself is not novel, what distinguishes this release is the addition of support for GPU-initiated communication within the scale-up domain using SDMA copy engines. This is a significant advancement: this release enables GPU-initiated communication that leverages SDMA engines rather than compute units for data movement. AMD GPUs are equipped with enough SDMA engines to fully saturate the available bandwidth within the scale-up domain.\n\nThe key advantage of this approach lies in resource utilization. In conventional fused compute-and-communication kernels, compute units must service both the computation and the data transfer, creating contention for shared resources. By offloading the communication to SDMA engines through GPU-initiated commands, compute resources are fully available for the computational portion of the kernel. Notably, this approach can saturate the available bandwidth using only a single CU, the one required to issue the SDMA commands.\n\n## Performance Evaluation[#](#performance-evaluation)\n\n### RCCL Results[#](#rccl-results)\n\nTo demonstrate the benefits of this approach, Figure 1 and Figure 2 compare the performance of three different implementations for `AlltoAll` and `Allgather`, respectively:\n\n- The default RCCL implementation based on load/store instructions\n- The host-initiated SDMA (Host-SDMA)\n- The GPU-initiated SDMA (GIN-SDMA)\n\nHost-SDMA and GIN-SDMA rely on symmetric memory registration, whereas the default implementation does not. Experimental results were collected on eight GPUs of an AMD Instinct™ MI350X node, with bus bandwidth reported as a function of message size. As shown in the figures, the SDMA implementations achieve bandwidth comparable to—and slightly higher than—the default implementation for the large message sizes evaluated. For alltoall, GIN-SDMA backend is used for message sizes greater than 32MiB; smaller sizes use vector load-store operations (red line in Figure-1). For allgather, GIN-SDMA is used for messages 512KiB or larger. Notably, GIN-SDMA delivers higher bandwidth than the default while using only a single CU, compared to the 64 CUs required by the default RCCL algorithm.\n\n### rocSHMEM Results[#](#rocshmem-results)\n\nGPU-initiated SDMA is also supported in rocSHMEM, as illustrated in Figure 3 and Figure 4.\n\nFigure 3 shows the bus bandwidth of `rocshmem_alltoall_wg` running on eight GPUs of an AMD Instinct™ MI350X GPU node with a single CU. Performance is compared across two configurations: (a) SDMA enabled (SDMA ON) with one CU and one wavefront, and (b) SDMA disabled (SDMA OFF) with one CU and 16 wavefronts. As the figure shows, SDMA ON delivers significantly higher performance because a single CU cannot saturate the available bandwidth using regular loads and stores.\n\n| Configuration | Details | Results | \n|---|---|---|\n| SDMA OFF: one CU, 16 wavefronts | Threads on the CU execute the copying instructions | Lower bandwidth | \n| SDMA ON: one CU, one wavefront | Threads issue commands; SDMA engines copy the data | Significantly higher bandwidth | \n\nFigure 4 shows the bandwidth of a `putnbi` benchmark between two GPUs on an AMD Instinct™ MI350X GPU node. Results are shown for SDMA ON with one CU using one and four wavefronts, and for SDMA OFF with various CU and wavefront configurations. At 128 KiB, the bandwidth achieved by a single CU is similar whether SDMA is enabled or not, although using a larger number of CUs with SDMA OFF can drive higher bandwidth at this message size. For messages of 512 KiB or larger, SDMA ON delivers the highest bandwidth regardless of configuration. The current rocSHMEM implementation does not use SDMA for messages smaller than 128 KiB; consequently, the configuration with four wavefronts outperforms the single-wavefront configuration at 64 KiB, as the load/store path benefits from additional wavefronts. However, once SDMA is used, a single CU with a single wavefront is sufficient to achieve peak bandwidth.\n\n## Architectural Perspective[#](#architectural-perspective)\n\nEach graphics compute die (GCD) has multiple SDMA engines available to perform copy operations. At a low level, an SDMA engine is a programmable DMA engine (similar in spirit to an RDMA NIC) but designed for inter-GPU communication within a node or pod. Figure 5 illustrates how SDMA engines are controlled at the low level. A ring buffer is populated with DMA commands (e.g., linear copies, copies with fused signal, or atomic operations), and a doorbell mechanism is used to trigger consumption of the pending entries in the ring buffer. For host-initiated operations, the `hipMemcpyAsync` API from the standard ROCm runtime abstracts this complexity, making SDMA engines readily accessible to the programmer. For device-initiated operations, RCCL-GIN and rocSHMEM prepare the SDMA packets internally, providing a similarly transparent interface to device kernels.\n\n## Availability and Reproducibility[#](#availability-and-reproducibility)\n\nThis feature is available in RCCL and rocSHMEM from [ROCm 10.1](https://rocm.docs.amd.com/en/docs-10.1.0/about/release-notes.html). To use the GPU-initiated SDMA feature, download [RCCL](https://github.com/ROCm/rocm-systems/tree/develop/projects/rccl) or [rocSHMEM](https://github.com/ROCm/rocm-systems/tree/develop/projects/rocshmem) from GitHub and build from source.\n\nMeasurements were done by AMD engineers from the collectives team. Experiments were repeated between 5 and 7 times. Since the results were stable across the executions, one set of the results is reported.\n\n### RCCL Configuration[#](#rccl-configuration)\n\nSymmetric memory registration and cuMem (`NCCL_CUMEM_ENABLE=1`) are prerequisites for GIN enablement in RCCL. RCCL currently supports multiple GIN backends, such as proxy, SDMA, and GDA. The SDMA backend can be enabled by using the environment variable `NCCL_GIN_TYPE=7`.\n\nBuild command:\n\n```\ninstall.sh --rocshmem-gin\n```\n\nVersions used for results:\n\n```\nRCCL version  : 2.32.3-develop:d20dae4\nHIP version   : 7.0.51831-b20d3d7ac\nROCm version  : 7.0.2.2.0-16-9428210\nMPI version   : open MPI 4.1.8\n```\n\n#### RCCL Hardware Configuration[#](#rccl-hardware-configuration)\n\nSingle server with eight AMD Instinct MI350X GPUs. Configuration: Quanta server with two AMD EPYC 9655 96-Core processors per server, 3 TB of system memory, BIOS F0TC3A07, CentOS Stream 9.\n\n`rccl-tests` `AlltoAll` command (uses RCCL internal `AlltoAll` kernel using GIN APIs):\n\n```\nmpirun -np 8 --map-by ppr:8:node -x NCCL_IGNORE_CPU_AFFINITY=1 \\\n  -x HSA_NO_SCRATCH_RECLAIM=1 -x NCCL_ENV_PLUGIN=none -x NCCL_CUMEM_ENABLE=1 -x NCCL_GIN_TYPE=7 \\\n  rccl-tests/build/alltoall_perf -b 1 -e 4G -f 2 -g 1 -R 2 -w 100 -n 20\n```\n\n`rccl-tests` `Allgather` command (uses `rccl-tests` `Allgather` kernel using GIN APIs):\n\n```\nmpirun -np 8 --map-by ppr:8:node -x NCCL_IGNORE_CPU_AFFINITY=1 \\\n  -x HSA_NO_SCRATCH_RECLAIM=1 -x NCCL_ENV_PLUGIN=none -x NCCL_CUMEM_ENABLE=1 -x NCCL_GIN_TYPE=7 \\\n  rccl-tests/build/all_gather_perf -b 1 -e 4G -f 2 -g 1 -R 2 -D 3 -w 100 -n 20\n```\n\n### rocSHMEM configuration[#](#rocshmem-configuration)\n\nBuild command:\n\n```\n./scripts/build_configs/all_backends -DUSE_SDMA=ON\n```\n\nVersions used for results:\n\n```\nversion      : rocSHMEM_v3.8.0_GDA(MUX:IONIC/BNXT/MLX5)+RO+IPC+SDMA\ngit Hash     : 1b28700e0c9d6ce2e7e5917bb555181859e5f9da\nROCm version : 10.1.0\n```\n\n#### rocSHMEM Hardware Configuration[#](#rocshmem-hardware-configuration)\n\nSingle server with eight AMD Instinct MI350X GPUs. Configuration: single Supermicro AS-8126GS-TNMR server, 2x AMD EPYC 9575F 64-Core Processor, 3TB memory (1.5TB per NUMA node), BIOS version 1.8. Ubuntu® 24.04 LTS, ROCm™ 10.1.0.\n\n`AlltoAll` rocSHMEM functional test:\n\n```\nmpiexec -n 8 --map-by numa --bind-to none \\\n  -x ROCSHMEM_TEST_UUID=1 \\\n  -x ROCSHMEM_HEAP_SIZE=$((64 * 1024 * 1024 * 1024)) \\\n  -x ROCSHMEM_SDMA_ENABLED=1 -x ROCSHMEM_SDMA_THRESHOLD=2048 \\\n  tests/functional_tests/rocshmem_functional_tests -a teamalltoall -w 1 -z 64 -v $((1 * 1024 * 1024 * 1024)) -n 20\n```\n\n`Putnbi` rocSHMEM functional test:\n\n```\nmpiexec -n 2 --map-by numa --bind-to none \\\n  -x ROCSHMEM_TEST_UUID=1 \\\n  -x ROCSHMEM_HEAP_SIZE=$((64 * 1024 * 1024 * 1024)) \\\n  -x ROCSHMEM_SDMA_ENABLED=1 -x ROCSHMEM_SDMA_THRESHOLD=1024 \\\n  tests/functional_tests/rocshmem_functional_tests -a wgputnbi -w 1 -z 64 -v $((1 * 1024 * 1024 * 1024)) -n 20\n```\n\nImportant parameters for rocSHMEM functional tests:\n\n| Parameter | Description | \n|---|---|\n| `ROCSHMEM_SDMA_ENABLED` | Turn SDMA ON/OFF | \n| `ROCSHMEM_SDMA_THRESHOLD` | Controls switch point between IPC/LSA and SDMA | \n| `-w` | Number of workgroups (workgroups normally map to CUs) | \n| `-z` | Number of threads/workgroup (64 is one wavefront on MI3XX) | \n\n## Summary[#](#summary)\n\nThis blog explored how GPU-initiated SDMA in RCCL and rocSHMEM enables a new class of fused compute-and-communication kernels where data movement is offloaded entirely to dedicated DMA engines. Performance benchmarks on AMD Instinct™ MI350X hardware demonstrated how this approach frees compute units to focus on computation, eliminates resource contention, and achieves full scale-up bandwidth with as little as a single Compute Unit. Available from ROCm 10.1, this capability is especially valuable for inference workloads where communication falls on the critical path and fine-grained overlap between compute and communication is essential.\n\n## Disclaimers[#](#disclaimers)\n\nThe information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies.\n\n© 2026 Advanced Micro Devices, Inc. All rights reserved", "url": "https://wpnews.pro/news/gpu-initiated-dma-copy-engines-in-rccl-and-rocshmem", "canonical_source": "https://rocm.blogs.amd.com/software-tools-optimization/gpu-initiated-sdma/README.html", "published_at": "2026-11-13 00:00:00+00:00", "updated_at": "2026-11-13 18:49:44.268004+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "machine-learning"], "entities": ["AMD", "RCCL", "rocSHMEM", "AMD Instinct MI350X", "SDMA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/gpu-initiated-dma-copy-engines-in-rccl-and-rocshmem", "markdown": "https://wpnews.pro/news/gpu-initiated-dma-copy-engines-in-rccl-and-rocshmem.md", "text": "https://wpnews.pro/news/gpu-initiated-dma-copy-engines-in-rccl-and-rocshmem.txt", "jsonld": "https://wpnews.pro/news/gpu-initiated-dma-copy-engines-in-rccl-and-rocshmem.jsonld"}}