DMA-Latte: Expanding the Reach of DMA Offloads to Latency-Bound ML Communication

A paper posted to arXiv on 10 November 2025 by Suchita Pati and coauthors, DMA-Latte, extends DMA communication offloads on AMD Instinct GPUs from bandwidth-bound transfers (tens of MB to GB) into latency-bound regions (KB to low MB). The optimizations deliver up to 4.5× speedups (3.2× geomean in the latency-bound region) over baseline DMA offload on ML collectives such as all-gather and all-to-all, with 3-10% power savings versus the RCCL GPU core-based communication library, and up to 1.65× lower latency and 1.9× higher throughput for LLM inference over vLLM. The work matters because it makes low-cost DMA engines competitive for the small, latency-sensitive transfers that dominate ML communication.

Computer Science Distributed, Parallel, and Cluster Computing Submitted on 10 Nov 2025 v1 https://arxiv.org/abs/2511.06605v1 , last revised 14 Sep 2026 this version, v3 Title:DMA-Latte: Expanding the Reach of DMA Offloads to Latency-bound ML Communication View PDF https://arxiv.org/pdf/2511.06605 HTML experimental https://arxiv.org/html/2511.06605v3 Abstract:Offloading communication to existing direct memory access DMA engines, available on most state-of-the-art commercial GPUs, has emerged as a practical and low-cost solution to efficiently overlap computation and communication in machine learning ML . However, the reach of DMA offloads has so far been limited to bandwidth-bound scenarios only 10s of MB to GB transfer sizes . In this work, we break this barrier and extend DMA communication offloads to latency-bound regions KB to low MB . Specifically, we leverage hitherto untapped features available in the state-of-the-art AMD Instinct$^{\mathrm{TM}}$ GPUs that render DMA communication offloads competitive even in latency-bound regions. We demonstrate the efficacy of these features both at the operator level ML communication collectives such as all-gather and all-to-all , and at the end-to-end workload level LLM inference . At the operator level, our optimizations provide up to 4.5$\times$ speedups 3.2$\times$ geomean in the latency-bound region over baseline DMA offload, narrowing the performance gap while delivering additional power savings 3-10% for ML collectives compared to state-of-the-art GPU core-based communication library, RCCL. At the workload level, we demonstrate acceleration for LLM inference: up to 1.65$\times$ lower latency and up to 1.9$\times$ higher throughput over the state-of-the-art vLLM inference framework. We conclude with a discussion of AMD Instinct GPU runtime innovations that stand to expose these features. Submission history From: Suchita Pati view email https://arxiv.org/show-email/19b06877/2511.06605 Mon, 10 Nov 2025 01:28:58 UTC 796 KB \ v1\ https://arxiv.org/abs/2511.06605v1 Fri, 10 Apr 2026 17:41:06 UTC 1,536 KB \ v2\ https://arxiv.org/abs/2511.06605v2 v3 Mon, 14 Sep 2026 19:26:33 UTC 1,586 KB Additional Features References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .