Faster Transport on Cloud Infra AWS EFA's SRD transport moves CUDA buffers roughly 40x faster than tuned TCP, about 13x faster through a shuffle, and roughly 2x faster on the PDS-H Q9 benchmark, according to a blog post recording configuration and testing details for EFA-enabled nodes. The tests ran on two g7e.12xlarge nodes with 2 Blackwell PRO GPUs (48 vCPUs) each, 4 GPUs and 4 ranks total, using GPUDirect RDMA, which AWS notes is only supported on multi-GPU instances. The author attributes the shrinking SRD-versus-TCP gap to growing workload complexity. Faster Transport on Cloud Infra EFA's SRD transport moves CUDA buffers ~40x faster than tuned TCP, ~13x faster through a shuffle, and ~2x faster using PDS-H Q9 as a benchmark. Why is there a performance difference between SRD and TCP, and why does that difference shrink as workload complexity grows? My team and I have been evaluating cuDF-Polars in cloud deployments and we've been experimenting with EFA enabled nodes. EFA is a high-performance network interface that allows for low-latency, high-bandwidth communication between nodes. Importantly, EFA enables GPU-to-GPU communication across nodes which is a great fit for our use case. Generally, this is referred to as GPUDirect RDMA Remote Direct Memory Access which is a key component of distributed high-performance computing. Tools like NCCL, UCX, NIXL with cuda copy or gdrcopy capabilities have built-in support for doing this kind of transfer. In this blog post, I'll mostly be recording configuration and testing details for EFA enabled nodes. AWS has a complete list of EFA enabled instance types https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html ac network and they also provide launch templates https://docs.aws.amazon.com/pcs/latest/userguide/working-with networking efa create-lt.html for easy consumption human or agent . Note: I don't think one can enable or turn on proper network adapters through the UI. Much of this work is also summarized in a helpful getting started page https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html though UCX is not listed in the test setup so I'll write it down here. It's a little buried, but on the g7e aws https://aws.amazon.com/ec2/instance-types/g7e/ page, AWS notes that GPUDirect RDMA is only supported for multi-GPU instances. So, every experiment below runs on two g7e.12xlarge nodes: 2 GPUs Blackwell PRO, 48 vCPUs per node or 4 GPUs / 4 ranks total. You can find all run and launch scripts in the companion folder: EFA benchmark scripts https://quasiben.github.io/static/code-snippets/efa-aws/ . These scripts drive both nodes over SSH from your laptop. I get a little confused with so much jargon and similar acronyms but this is what I've boiled accelerated networking on AWS down to: 1. ENA = standard networking 2. ENA Express = better ENA with SRD Scalable Reliable Datagram for low latency 3. EFA = RDMA with SRD and only some instances support GPUDirect RDMA I'm using an Ubuntu image; official AWS images may already have the modules baked in. To launch with EFA enabled we need the following: 1. Launch a node with EFA enabled: InterfaceType=efa 2. Setup EFA on the node: js echo "== Installing AWS EFA software stack" curl -O https://efa-installer.amazonaws.com/aws-efa-installer-latest.tar.gz tar -xzf aws-efa-installer-latest.tar.gz cd aws-efa-installer sudo ./efa installer.sh -y --skip-limit-conf cd .. rm -rf aws-efa-installer aws-efa-installer-latest.tar.gz echo "Enabling efa nv peermem kernel module required for CUDA " sudo modprobe efa nv peermem echo "Enabling persistent loading of efa nv peermem kernel module" echo "efa nv peermem" | sudo tee /etc/modules-load.d/efa nv peermem.conf Script: 00-launch https://quasiben.github.io/static/code-snippets/efa-aws/00-launch/launch efa nodes.sh does all of the above unattended. You can verify a working EFA setup with the following: bash ubuntu@ip-172-31-29-167:~$ fi info -p efa should list efa 0 device provider: efa fabric: efa-direct domain: rdmap47s0-rdm version: 206.0 type: FI EP RDM protocol: FI PROTO EFA provider: efa fabric: efa domain: rdmap47s0-rdm version: 206.0 type: FI EP RDM protocol: FI PROTO EFA provider: efa fabric: efa domain: rdmap47s0-dgrm version: 206.0 type: FI EP DGRAM protocol: FI PROTO EFA ubuntu@ip-172-31-29-167:~$ ls /dev/infiniband/uverbs should exist /dev/infiniband/uverbs0 UCX Performance ¶ ucx-performance I first want to validate EFA with UCX. Check UCX can "see" srd: bash cudf-polars ubuntu@ip-172-31-29-167:~$ ucx info -d | grep srd Transport: srd Now we can run ucx perftest which will measure bandwidth as we send 100 MiB GPU buffers across the wire in one direction: Note: I needed to tune TCP configs a bit to get more reasonable output. Out of the box, UCX defaults don't prioritize TCP transport. Example output with SRD ¶ example-output-with-srd On Node A ucx perftest -m cuda -t tag bw -n 10 -s $ 1024 1024 100 On Node B ucx perftest -m cuda -t tag bw -n 10 -s $ 1024 1024 100 IP NODE A +--------------+--------------+------------------------------+---------------------+-----------------------+ | | | overhead usec | bandwidth MB/s | message rate msg/s | +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+ | Stage | iterations | 50.0%ile | average | overall | average | overall | average | overall | +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+ Final: 10 0.353 2202.916 2202.916 45394.37 45394.37 454 454 Example output without SRD ¶ example-output-without-srd UCX TLS=tcp,cuda copy,cuda ipc,sm,self \ UCX TCP TX SEG SIZE=256K UCX TCP RX SEG SIZE=256K UCX TCP MAX BW=auto \ UCX TCP SNDBUF=4M UCX TCP RCVBUF=4M \ ucx perftest -m cuda -t tag bw -n 10 -s $ 1024 1024 100 IP NODE A +--------------+--------------+------------------------------+---------------------+-----------------------+ | | | overhead usec | bandwidth MB/s | message rate msg/s | +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+ | Stage | iterations | 50.0%ile | average | overall | average | overall | average | overall | +--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+ Final: 10 0.568 87966.013 87966.013 1136.80 1136.80 11 11 UCX Results ¶ ucx-results | Transport | Bandwidth | Message rate | Average overhead | |---|---|---|---| | SRD EFA | 45.39 GB/s | 454 msg/s | 2.20 ms | | TCP, tuned srd excluded | 1.14 GB/s | 11 msg/s | 87.97 ms | With SRD/EFA, UCX can transfer CUDA buffers at ~45GB/s , and without SRD TCP only bandwidth is severely degraded to ~1.1GB/s , a ~40x performance difference. TCP being slow here is expected and conversely demonstrates why GPU RDMA is critical. With TCP, GPU data is moved from device to host D- H , serialized, then sent across the wire, then deserialized, and finally moved back from host to device H- D . GPU RDMA avoids the costly H- D / D- H movement and the serialization costs. Hmm, it's the same NIC between TCP/SRD. Perhaps a question for later . Effectively, with GPU RDMA, the pipeline is: GPU- NIC- RemoteNIC- RemoteGPU. AWS states that on a g7e.12xlarge the transport is 400Gbps or 50GB/s, very close to what we measure with ucx perftest . Fig 1. TCP and GPUDirect RDMA SRD/EFA data paths between two g7e.12xlarge nodes. Let's increase complexity from simple perf testing... Shuffle Performance ¶ shuffle-performance In these blogs I haven't gone into much of the underlying machinery of cuDF-Polars but a lot of the performance comes from the accelerated and out-of-core shuffle https://docs.rapids.ai/api/rapidsmpf/stable/background/shuffle-architecture/ implemented in RapidsMPF. Let's do a similar experiment to ucx perftest where, instead of just shoving bytes across the wire, we measure the bandwidth of a more complex workload, a shuffle, again with and without SRD/EFA. A shuffle is more revealing than point-to-point transfers, and it's more than just an all-to-all. Here every rank is sending and receiving at once and doing some hash calculations as well. In this experiment we configure a benchmark to shuffle 20 GiB of randomly generated data per rank per GPU . Each 20 GiB is composed of 1 GiB input partitions and is redistributed into 8 output partitions of ~2.5 GiB per rank. The benchmark runs 3 warmups and then shuffles the same data 10 more times, and the number we care about is the local throughput. Below is a simplified example of what was executed: 20 GiB/rank at 1 GiB per input partition, c=1 default, 4 bytes/row : -n = 1024 1024 1024 / 4 = 268435456 rows - exactly 1024 MiB/partition -p 20 partitions 1 GiB = 20 GiB/rank -o 8: number of output partitions experiment with srd ${CONDA PREFIX}/bin/libcudf streaming bench shuffle -w 3 -r 10 -g -s -x -n 268435456 -p 20 -o 8 experiment without srd, TCP tuned see note below UCX TLS=tcp,cuda copy,cuda ipc,sm,self \ UCX TCP TX SEG SIZE=256K UCX TCP RX SEG SIZE=256K UCX TCP MAX BW=auto \ UCX TCP SNDBUF=4M UCX TCP RCVBUF=4M \ ${CONDA PREFIX}/bin/libcudf streaming bench shuffle -w 3 -r 10 -g -s -x -n 268435456 -p 20 -o 8 Example output with SRD ¶ example-output-with-srd 1 3:PRINT:0:2026-08-24 15:12:43.805116391 means: 726.79 ms | local throughput: 27.52 GiB/s | global throughput: 110.07 GiB/s | in parts: 20 | out parts: 8 | nranks: 4 | device memory peak: 35 GiB | device memory total: 135.31 GiB avg 2:PRINT:0:2026-08-23 01:37:19.838332753 Statistics of the last run : - alloc-device: 15 GiB | 4.42 ms | 3.32 TiB/s | avg-stream-delay 26.23 us - event-loop-total: 451.58 ms | avg 11.37 us - metadata-payload-exchange-complete-data-transfers: 145.69 ms | avg 3.67 us - metadata-payload-exchange-progress: 427.41 ms | avg 10.77 us - metadata-payload-exchange-receive-metadata: 229.34 ms | avg 5.78 us - metadata-payload-exchange-send-messages: 3.19 ms - metadata-payload-exchange-setup-data-receives: 14.75 ms | avg 371.56 ns - shuffle-payload-recv: 15 GiB | avg 32.01 MiB - shuffle-payload-send: 15 GiB | avg 32 MiB Shuffle Results ¶ shuffle-results | Transport | Mean elapsed/rank | Local throughput | Global throughput | |---|---|---|---| | SRD EFA | 727.6 ms | 27.52 GiB/s | 109.95 GiB/s | | TCP, tuned srd excluded | 9.45 s | 2.13 GiB/s | 8.52 GiB/s | Again, not unexpected but also so amazing that we have software which can leverage this powerful hardware. libcudf streaming bench shuffle isn't hitting max perf of the card but a shuffle is also doing more than just transporting data. Still, as we increased complexity of the workload we can still see the benefits: ~13x perf difference. Last One, I Promise: PDS-H Query 9 ¶ last-one-i-promise-pds-h-query-9 Ok, one more experiment. Let's increase complexity again and run an end-to-end example. As mentioned at the top, we have been studying cloud performance for real world examples. With that in mind let's run a PDS-H workload, an experimental TPC-H-derived benchmark shipped with cuDF-Polars. Specifically, we will only study query 9, which has several joins and will therefore include many shuffles. PDS-H is not an official or TPC-H-compliant benchmark, so these numbers should not be compared with published TPC-H results. Q9 Physical Plan SF1K ¶ q9-physical-plan-sf1k SORT 'nation', 'o year' 'nation', 'o year', 'sum profit' 120 SELECT 'nation', 'o year', 'sum profit' 120 GROUPBY 'nation', 'o year' 'nation', 'o year', ' 0' 120 SELECT 'nation', 'o year', 'amount' 120 PROJECTION 'n name', 'o orderdate', 'l extendedprice', 'l discount', 'ps supplycost', 'l quantity' 120 JOIN Inner 's nationkey', 'n nationkey', 'ps supplycost', 'l quantity', 'l extendedprice', '...', 's natio nkey', 'n name' 120 PROJECTION 'ps supplycost', 'l quantity', 'l extendedprice', 'l discount', 'o orderdate', 's nationkey' 12 0 JOIN Inner 'l orderkey', 'o orderkey', 'ps supplycost', 's nationkey', 'l quantity', '...', 'l orderke y', 'o orderdate' 120 PROJECTION 'ps supplycost', 's nationkey', 'l quantity', 'l extendedprice', 'l discount', 'l orderkey' 120 JOIN Inner 'p partkey', 'ps suppkey' 'l partkey', 'l suppkey' 'ps supplycost', 's nationkey', 'p p artkey', '...', 'l extendedprice', 'l discount' 120 PROJECTION 'ps supplycost', 's nationkey', 'p partkey', 'ps suppkey' 8 JOIN Inner 'ps suppkey', 's suppkey', 'p partkey', 'ps suppkey', 'ps supplycost', 's nationkey ' 8 JOIN Inner 'p partkey', 'ps partkey', 'p partkey', 'ps suppkey', 'ps supplycost' 8 PROJECTION 'p partkey', 4 STREAMINGSCAN 'p partkey', 'p name' 4 STREAMINGSCAN 'ps suppkey', 'ps supplycost', 'ps partkey' 8 STREAMINGSCAN 's nationkey', 's suppkey' 1 STREAMINGSCAN 'l orderkey', 'l quantity', 'l extendedprice', 'l discount', 'l partkey', 'l suppkey' 120 STREAMINGSCAN 'o orderdate', 'o orderkey' 8 STREAMINGSCAN 'n name', 'n nationkey' 1 In cuDF-Polars, PDS-H is baked in, making it easy to test. Running Q9 at SF1000 on the same two-node, four-GPU cluster: python -m cudf polars.streaming.benchmarks.pdsh ${QUERY} This time the gap is much smaller than the raw bandwidth and shuffle numbers above would suggest: | Transport | Q9 Iter 0 | Q9 Iter 1 | |---|---|---| | SRD EFA | 7.94s | 5.23s | | TCP, tuned srd excluded | 16.79s | 14.31s | Nice We still observe a healthy performance gain: a 2x-2.7x perf difference comparing iterations. The gap between the iterations is a little bothersome. The OS isn't caching a file -- we're reading chunks of data from S3. There are also costs to warmup the engine, such as launching threads, memory allocation, as well as possibly caching of the data within S3. All those costs contribute roughly equally to iteration 0 of both SRD and TCP, which we are not measuring in this experiment. Wrapping Up ¶ wrapping-up We started at a 40x raw perf difference comparing SRD/EFA to TCP, then we saw a 13x perf difference when performing a shuffle. Now, with an end-to-end example we see 2x-2.7x perf differences. This is reasonable given that the workload isn't shuffling the entire time. We can see in the physical plan above that Q9 is also pulling data from S3, doing some selections, group-bys, etc. Still, 2x is a great win for distributed accelerated data processing, and I'd expect it to be the floor rather than the ceiling. The more data a query moves and the more shuffles it induces, the more of the workload is network-bound -- and that is where SRD/EFA shines and delivers higher bandwidth. A Secret Deep Dive: Why Doesn't Tuned TCP Hit Max Perf? ¶ a-secret-deep-dive-why-doesnt-tuned-tcp-hit-max-perf Even after tuning TCP, we still hit a ceiling, and TCP never got close to SRD's throughput. Is this a UCX inefficiency, or a hard limit somewhere lower in the stack? ENA the underlying NIC on these instances is capable of the same raw ~400Gbps line rate as EFA. To isolate the transport from UCX entirely, we can evaluate plain iperf3 between the same two nodes. Single stream, varying transfer size ¶ single-stream-varying-transfer-size iperf3 -c