Hey all, posing the question up front: what’s the cheapest way to get a high scaling efficiency tensor parallel rig for 8 GPUs?
So, I recently impulse bought 8x r9700 with the plan to run tensor parallel tp8 on ~200GB models. It seems like for inference performance the best implementation would be to optimize for low latency and decent bandwidth through a pcie 5.0 fabric where p2p communications never hit the cpu. I’m thinking on skimping on the host as a result (cheapest cpu+mobo that exposes a pcie 16x 5.0 slot and 64gb ram).
I’m looking at the githib project “local-inference-lab/rtx6kpro/blob/master/hardware/topology.md” for inspiration using a single Microchip Switchtec PM50100 running 8x lanes per gpu flike this:
Ryzen host (64gb ddr5 ram, cheap mobo + processor)
│
PCIe 5.0 x16 slot
│
x16 → 2× MCIO x8 card
│
▼
┌────────────────────┐
│ C-Payne PM50100 │
│ 100-lane Gen5 │
│ switch │
└────────────────────┘
│ │ │ │ │ │ │ │
x8 each over MCIO
│ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
PCIe x16 mechanical
endpoint adapters
│ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
R9700 × 8
And housing all of the bits in a small cheap gpu cluster housing like a MM-A515-CPW and swapping the electronics.
I reached out to guva systems to see if I could get a quote for a true 16x lanes per gpu switch since thats a clear optimization option, but I’m unsure what the performance unlocks for vllm optimization would be.
I’m a bit of a greenbeard in this area though, so I could use advice on hardware choice optimizations really matter for a vllm-radiance deployment on qwen3.8-next-flash or quantized glm5.3-flash like I am currently targeting.
Just realized the title is now misleading to the post content and I can’t edit it, I wandered a bit while researching this post.