I’m doing a great disservice to how the flow control actually works in this explanation, but the speed you’d get when crossing PCIe generations in a switch will come down to the buffer size the switch has for the different “channels” on it and traffic patterns.
teromee 23 To clarify: Think of CXL as just a faster, smarter ‘highway’ (transport layer) for PCIe. The hardware handles the movement of data, but vLLM is the dispatcher.
Currently, vLLM treats all memory like it’s in the same room. For an ‘AI Council’ or CXL setup to work, we need a Custom Scheduler in vLLM that understands ‘Distance.’ It needs to realize that while CXL memory is huge, it has slightly higher latency than local VRAM.
So, the software mods aren’t changing the AI model itself; they are changing how vLLM manages the KV Cache and Compute Location based on the hardware it detects. There are already scripts that can ‘fingerprint’ your hardware and suggest the best model—we just need that same logic baked into the vLLM scheduler so it can route data to the right ‘expert’ on the fly.
1 Like
There again my working assumptin is that PCIe isn’t a network routing switch but based on a PCI world where once you’ve won an arbitration round, your data gets through within a minimal time or size window and that the switch won’t buffer buffer (incurring potentially huge latencies and cost for storage), but to cut-through processing where data going in on one lane, goes out on another (or others) within a few cylcles at most. Full reception of deep buffers and forwarding only after full reception not only slows things down terribly, but incurs all types of responsibilities and failure scenarios which just don’t fit a technology designed for affordable personal computing decades ago. In PCIe fabrics source and targerts are assumed to be much “brainier” than the switch, with data only released at the source when the recipient has acknowledged transfer, so that any switch buffers would be just what it takes to do lane splits/joins and rate adaptations, not to keep and safeguard for asynchronous delivery.
In that it’s very different from say a fiber channel network where flow control with redit management is requried because the fabric takes ownership and responsibility over data en route.
But that’s just from personal memory and I could be all wrong.
I hope this scales to thunderbolt PCIe enclosures.
I’d love a mini PC where it has a direct x16 for the GPU and have a thunderbolt port that turns into a lot of x1 Gen2 or Gen1 ports. (or even conventional PCI) A 2U half height rack sounds good for about all the legacy things I would want to plug in and when the CPU gets too old, get another mini PC and keep the same enclosure forever. I’m tired of having gigantic cases with obsolete motherboards.
I just care about supporting a lot of obsolete cards and one modern GPU.
nereith 28 Based on an old document from PLX on PCIE switches behaving as bandwidth bridges, I would just assume such switches would be able to handle Gen5 x4 ↔ Gen3 x 16 transactions without flow control issues.
1 Like
Wendel, why don’t you do a bit of testing and benchmarking to relieve us of the guesswork?
Hi Y’all,
Starting a new project I think may interest the forum, working on connecting 4 H100 air cooled cards or 8 liquid cooled cards in a 4U chassis above my R7515 through either PEX88096 ( Amazon.com: PCIe 4.0 to SlimSAS Adapter Card, (4) X16 Slots and (4) SFF-8654 Ports (2Up/2Down), with Broadcom PEX88096 Switch Chip, Compatible Windows/Linux/VMware/Freebsd/Ubuntu : Electronics ) OR c-payne gen 5 device ( PCIe gen5 MCIO Switch 100-Lane - Microchip Switchtec PM50100 – c-payne.com ). I already ordered a few just to check them out. These are all SXM cards with an additional purchased conversion pcie adapter. I think some of the items I am tackling would be interesting for feedback to @wendell as I plan to purchase 8 additional H100s and I would love to volunteer for your PCIE PLX solution testing (and pay for them). The reason for the R7515 is because I had 4 of them in my garage with RAM sitting idle . My plan is to externally liquid cool them depending on how tight I can fit the systems. Naturally these will be for LLMs, so we can checkout these PLX cards and the AI bandwidth scenario.
Thanks for looking at my insane project!
ces319 31 Do you have any pictures of your server? Do you have any recommendations on chassis for these backplanes?
ces319 32 Did you ever implement this solution? I’m attempting to go down this road. Pictures or build recommendations would be appreciated for reference. The build side of this is tricky unless you go with an open frame.
If stuff like broadcom’s pci-e pxl card step-down things got actually affordable people who want a system that combines very high single-core performance and lots of pci-e connectivity without having to drop used car money on either threadripper or sapphire/granite rapids. And I really hope prices for these magic expansion cards drop one way or another.