{"slug": "deploying-rdma-for-large-scale-inference-and-training-workloads", "title": "Deploying RDMA for Large-Scale Inference and Training Workloads", "summary": "ByteDance deployed ByteExpress, a software communication layer over hardware RDMA, in its parameter-server-based inference and training systems, cutting mean and P99 model parameter fetch latency by 35.7% and 43.7% respectively and total CPU usage by 10.69% versus kernel TCP in production inference, according to an arXiv paper (2610.12815v1). In training, ByteExpress reduced P99 model parameter push latency by up to 74.0% compared with TCP, with no PFC pauses observed at runtime. The paper reports three deployment lessons: efficient RDMA requires buffer management across application and transport layers, scalable communication requires coordinated management of connections, receive capacity and outstanding data, and service availability requires recovery above the transport layer.", "body_md": "arXiv:2610.12815v1 Announce Type: new \nAbstract: RDMA can reduce communication overhead in distributed machine learning systems, but realizing these benefits in production requires more than replacing TCP. Variable-size messages, concurrent connections, bursty incasts, and service failures introduce challenges that hardware offload alone cannot address. This paper reports our experience deploying ByteExpress, a software communication layer over hardware RDMA, in ByteDance's parameter-server-based inference and training systems. We present three lessons from the deployment: efficient RDMA requires buffer management across application and transport layers; scalable communication requires coordinated management of connections, receive capacity, and outstanding data; and service availability requires recovery above the transport layer. ByteExpress combines an RDMA-aware allocator, zero-copy transfers, QP multiplexing, receive-credit flow control, a software send window, and RPC-assisted recovery. In production inference, ByteExpress reduces mean and P99 model parameter fetch latency by 35.7% and 43.7%, respectively, and total CPU usage by 10.69% compared with kernel TCP. In training, ByteExpress reduces P99 model parameter push latency by up to 74.0% compared with TCP, with no PFC pauses observed at runtime.", "url": "https://wpnews.pro/news/deploying-rdma-for-large-scale-inference-and-training-workloads", "canonical_source": "https://www.machinebrief.com/news/deploying-rdma-for-large-scale-inference-and-training-worklo-5e8q", "published_at": "2026-10-12 04:00:00+00:00", "updated_at": "2026-10-12 05:08:12.948973+00:00", "lang": "en", "topics": ["ai-infrastructure", "machine-learning", "mlops"], "entities": ["ByteDance", "ByteExpress", "RDMA", "TCP", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/deploying-rdma-for-large-scale-inference-and-training-workloads", "markdown": "https://wpnews.pro/news/deploying-rdma-for-large-scale-inference-and-training-workloads.md", "text": "https://wpnews.pro/news/deploying-rdma-for-large-scale-inference-and-training-workloads.txt", "jsonld": "https://wpnews.pro/news/deploying-rdma-for-large-scale-inference-and-training-workloads.jsonld"}}