Hello All,
Not posted in a long time. but I cant find answers anywhere else.
I have been following the AI clustering topics about USB4.
I need someone to help get me real latency numbers for Mac Thunderbolt RDMA.
Can someone run ib_read_lat, ib_write_lat and ib_send_lat or some equivalent on a 2 node setup.
I need to know if it is real RDMA or just really good software RDMA. My understanding right now is that there is no way that thunderbolt can bypass the CPU/kernel and go to the GPU vRAM with a zero-copy. There is a limit to how low they can get without adding a RDMA chip to the USB port, which apple might do.
If results are >5 than it is just software RDMA.
If results are lower than they must have RDMA hardware.
If it is <2uS then we are really cooking.
Notes:
I have tested both thunderbolt-ibverbs and OdinLink-Five. and they are good but most of the improvements they get in latency is from enabling busy polling. I am running it with Strix Halo and getting 7.8uS over TCP. I want to know if Apple is just doing the same trick.
`sudo sysctl -w net.core.busy_poll=100`
`sudo sysctl -w net.core.busy_read=100`
I look forward to getting real numbers for latency.
boatman 2
Update: I was not able to get numbers but I can confirm that the USB RDMA on Macs is just good software. not real RDMA hardware offload.