What’s the cheapest way to get a high scaling efficiency tensor parallel rig for 8 GPUs?

An 8-GPU tensor-parallel (TP8) rig should prioritize PCIe topology over CPU choice, according to guidance recommending that GPUs communicate directly through a Gen5 switch rather than routing peer-to-peer traffic through the host. The guidance advises benchmarking actual GPU-to-GPU bandwidth and latency, P2P support, NUMA effects, and 4-to-8-GPU scaling with NCCL P2P tests before spending more on the host, and verifying that the motherboard and BIOS expose the required PCIe lanes and ACS/IOMMU settings. For roughly 200 GB models, memory capacity and inter-GPU communication can become the bottleneck, so a cheap CPU is acceptable if it does not limit tokenization, scheduling, or data movement.

For an 8-GPU TP8 setup, focus less on the CPU and more on the PCIe topology. If the GPUs can communicate directly through a Gen5 switch without routing P2P traffic through the host, that should matter much more for tensor-parallel inference than having an expensive CPU. The important thing is to benchmark the actual topology rather than assume x16 per GPU will automatically translate into better vLLM performance. Check GPU-to-GPU bandwidth/latency, P2P support, NUMA effects, and how the workload scales from 4 → 8 GPUs. Also make sure the motherboard and BIOS expose the required PCIe lanes and ACS/IOMMU settings correctly. For ~200 GB models, memory capacity and inter-GPU communication can easily become the bottleneck, so I’d validate the switch configuration with NCCL P2P tests before spending more on the host. A cheap CPU is reasonable if it doesn’t become the bottleneck for tokenization, scheduling, or data movement.