Ray Direct Transport (RDT), a feature in Ray Core, enables RDMA-backed weight transfer in reinforcement learning for LLMs, achieving up to 7.5x speedup over naive RDMA implementations on GB200 nodes u…
Google has open-sourced TPU Raiden, an inference optimization library for KV-cache transfer in large language model serving, under the Apache-2.0 license on GitHub, positioning it as a direct counterp…
VLLM's slack-aware preemption policies rescued urgent request attainment at high load in a benchmark on 6× A100 SXM4 80GB GPUs running Llama 70B, where the control policy collapsed to 0% urgent attain…
Nvidia is developing its proprietary CMX platform to improve AI inference economics by offloading KV cache to SSDs via BlueField DPUs, while the open CXL standard offers a vendor-agnostic alternative.…
Scality is enhancing its object storage to support AI workloads by integrating with NVIDIA's NIXL library, enabling S3 over RDMA to bypass CPU bottlenecks and move data directly between flash and GPU …