[Submitted on 10 Nov 2025 (
[v1](https://arxiv.org/abs/2511.06605v1)), last revised 14 Sep 2026 (this version, v3)]
[View PDF](https://arxiv.org/pdf/2511.06605)
[HTML (experimental)](https://arxiv.org/html/2511.06605v3)
Abstract:Off communication to existing direct memory access (DMA) engines, available on most state-of-the-art commercial GPUs, has emerged as a practical and low-cost solution to efficiently overlap computation and communication in machine learning (ML). However, the reach of DMA offloads has so far been limited to bandwidth-bound scenarios only (10s of MB to GB transfer sizes). In this work, we break this barrier and extend DMA communication offloads to latency-bound regions (KB to low MB). Specifically, we leverage hitherto untapped features available in the state-of-the-art AMD Instinct$^{\mathrm{TM}}$ GPUs that render DMA communication offloads competitive even in latency-bound regions. We demonstrate the efficacy of these features both at the operator level (ML communication collectives such as all-gather and all-to-all), and at the end-to-end workload level (LLM inference). At the operator level, our optimizations provide up to 4.5$\times$ speedups (3.2$\times$ geomean in the latency-bound region) over baseline DMA offload, narrowing the performance gap while delivering additional power savings (3-10%) for ML collectives compared to state-of-the-art GPU core-based communication library, RCCL. At the workload level, we demonstrate acceleration for LLM inference: up to 1.65$\times$ lower latency and up to 1.9$\times$ higher throughput over the state-of-the-art vLLM inference framework. We conclude with a discussion of AMD Instinct GPU runtime innovations that stand to expose these features.
Submission history #
From: Suchita Pati [
[view email](https://arxiv.org/show-email/19b06877/2511.06605)]
**Mon, 10 Nov 2025 01:28:58 UTC (796 KB)**
[\[v1\]](https://arxiv.org/abs/2511.06605v1)
**Fri, 10 Apr 2026 17:41:06 UTC (1,536 KB)**
[\[v2\]](https://arxiv.org/abs/2511.06605v2)
**[v3]** Mon, 14 Sep 2026 19:26:33 UTC (1,586 KB)
Additional Features
References & Citations
...
Bibliographic Explorer
(What is the Explorer?) Connected Papers
(What is Connected Papers?) Litmaps
(What is Litmaps?) scite Smart Citations
(What are Smart Citations?) alphaXiv
(What is alphaXiv?) CatalyzeX Code Finder for Papers
(What is CatalyzeX?) DagsHub
(What is DagsHub?) Gotit.pub
(What is GotitPub?) Hugging Face
(What is Huggingface?) ScienceCast
(What is ScienceCast?) Influence Flower
(What are Influence Flowers?) CORE Recommender
(What is CORE?) arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.