A practical guide to running 8x RTX PRO 6000's

A practical guide from Jerry James at an unspecified outlet details how to run 8x NVIDIA RTX PRO 6000 Blackwell GPUs with an AMD EPYC 9555, concluding that splitting 400B+ models across PCIe Gen 5 introduces heavy latency, so the platform's strength is high-density parallel execution, supporting up to 1.15 million active tokens per card and enabling on-premise 70B fine-tuning.

A practical guide to running 8x RTX PRO 6000's Maximize parallelism across 8x RTX PRO 6000's: high-concurrency inference, model fleets, and 70B fine-tuning. By Jerry James · When we got our hands on an 8x NVIDIA RTX PRO 6000 Blackwell /inventory/gpu/pny-nvidia-rtx-pro-6000-blackwell-workstation-edition-96gb-gddr7 system paired with an AMD EPYC 9555 /inventory/cpu/amd-epyc-9575f-64-core-3-30ghz-processor 64 cores, 128 threads, Zen 5 , the first question we asked was simple: what can we run with this? With 768 GB of aggregate GDDR7 VRAM on tap, the knee-jerk instinct was to test whether we could shard larger models like Llama-3.1 405B or DeepSeek-R1 671B across all eight cards. But that seems a job more suited to this config's much more expensive sibling in NVIDIA's data center lineup /inventory/systems/msi-cg480-s6053-dual-epyc-9575f-8x-h200-nvl-ai-training-server . That realization led us to reframe the architecture. Instead of forcing deep model parallelism over PCIe, it makes more sense to explore what workloads we can handle at a fraction of the price of something like an HGX B200/B300 without sacrificing too much performance. TLDR: Splitting massive 400B+ models across eight PCIe GPUs introduces heavy latency, so that type of workload is out. Let's leave that to the cream of NVIDIA's monstrous server lineup. This platform's strength is high-density parallel execution, where it keeps up with its more powerful counterparts. Isolated serving TP=1 maximizes compute: Running independent single-GPU instances eliminates inter-card bus traffic, delivering more efficient compute utilization and minimal inter-token latency. Unprecedented KV cache density: Dense mid-sized models such as Qwen3.8-27B in FP8 leave ~60 GB of usable VRAM per card for KV caching, sustaining up to 1.15 million active tokens per card ~9.2 million tokens node-wide at 4k context . Multi-tenant model fleets: One node can concurrently handle 24 to 32 specialized 8B microservice endpoints or up to 8 dedicated 32B endpoints or 8 parallel 1080p video diffusion clips. On-premise 70B+ fine-tuning: Sharding model states across 768 GB VRAM while offloading AdamW optimizer moments to the EPYC 9555's host DDR5 memory enables native 70B fine-tuning without multi-node cloud clusters. 1. Interconnect Constraints: PCIe Gen 5 vs. NVLink Interconnect bandwidth and collective latency determine whether a workload scales efficiently via model parallelism or independent concurrency: | | | | | | |---|---|---|---|---|---| PCIe Gen 5 x16 | 64 GB/s | 128 GB/s | 1x baseline | 12-25 μs | $$ | NVLink 4 Hopper | 450 GB/s | 900 GB/s | ~7x | 1.5-3.0 μs | $$$ | NVLink 5 Blackwell | 900 GB/s | 1,800 GB/s | ~14x | 1.0-2.0 μs | $$$$ | PCIe Gen 5 x16 64 GB/s 128 GB/s 1x baseline 12-25 μs $$ NVLink 4 Hopper 450 GB/s 900 GB/s ~7x 1.5-3.0 μs $$$ NVLink 5 Blackwell 900 GB/s 1,800 GB/s ~14x 1.0-2.0 μs $$$$ Although going with PCIe 5.0 clearly has limitations, the advantages and capabilities of NVLink you see above also come attached to an abnormally high price tag. Now, this is unavoidable when you're training or serving frontier models that span 100s of gigabytes or even terabytes. But you can handle concurrent agentic workloads and so much more at a fraction of the price with Blackwell GPUs attached to the good old PCIe slot. Tensor parallelism in motion: one model sharded across GPUs and stitched back together over the interconnect. Animation: NVIDIA https://developer.nvidia.com/blog/demystifying-ai-inference-deployments-for-trillion-parameter-large-language-models/ . 2. Recommended Parallelism Strategies So, if you aren't sharding massive 400B+ monolithic models across this server, what CAN it do? Well, servicing very high concurrency needs, of course, and some more: A. Isolated Single-GPU Serving TP=1, DP=8 The 96 GB GDDR7 capacity of each RTX PRO 6000 accommodates dense mid-sized architectures like the new Qwen3.8-27B entirely on a single card without multi-GPU sharding. What this means is that your mileage will vary depending on your model and the precision used: | | | | | | | |---|---|---|---|---|---|---| Qwen3.8-27B | FP8 ~27 GB | FP8 | ~59.4 GB | 32 KB | ~287 / card 2,296 node | 6-7x 256k or 13-14x 128k streams | Qwen3.8-27B | BF16 ~54 GB | FP8 | ~32.4 GB | 32 KB | ~156 / card 1,248 node | 3-4x 256k or 7x 128k streams | Qwen3.8-27B | BF16 ~54 GB | BF16 | ~32.4 GB | 64 KB | ~97 / card 776 node | 3-4x 128k lossless streams | Llama-3.3 70B | FP8 ~70 GB | FP8 | ~16.4 GB | 160 KB | ~25 / card 200 node | 1x 64k stream | Qwen3.8-27B FP8 ~27 GB FP8 ~59.4 GB 32 KB ~287 / card 2,296 node 6-7x 256k or 13-14x 128k streams Qwen3.8-27B BF16 ~54 GB FP8 ~32.4 GB 32 KB ~156 / card 1,248 node 3-4x 256k or 7x 128k streams Qwen3.8-27B BF16 ~54 GB BF16 ~32.4 GB 64 KB ~97 / card 776 node 3-4x 128k lossless streams Llama-3.3 70B FP8 ~70 GB FP8 ~16.4 GB 160 KB ~25 / card 200 node 1x 64k stream B. Multi-Model Microservice Fleets 8B-32B For enterprise inference routing, each 96 GB card can host multiple smaller models e.g., Llama-3.1 8B, Mistral Small, Qwen3 32B or deep continuous-batching workers. With this kind of strategy, the numbers break down to something like this: | | | | | | | |---|---|---|---|---|---|---| Dedicated 8B instances | 1x 8B model ~8 GB, FP8 | ~78.4 GB | 64 KB | 306 / card 2,448 node | 8 dedicated models | High-volume single-task API | 3x 8B model stacking | 3x 8B models ~24 GB, FP8 | ~62.4 GB | 64 KB | 244 / card 1,952 node | 24 distinct endpoints up to 32 with 4x 8B | Multi-tenant microservice mesh | Mid-tier 32B models | 1x 32B model ~32 GB, FP8 | ~54.4 GB | 128 KB | 106 / card 848 node | 8 specialized models | Advanced coding & reasoning | Dedicated 8B instances 1x 8B model ~8 GB, FP8 ~78.4 GB 64 KB 306 / card 2,448 node 8 dedicated models High-volume single-task API 3x 8B model stacking 3x 8B models ~24 GB, FP8 ~62.4 GB 64 KB 244 / card 1,952 node 24 distinct endpoints up to 32 with 4x 8B Multi-tenant microservice mesh Mid-tier 32B models 1x 32B model ~32 GB, FP8 ~54.4 GB 128 KB 106 / card 848 node 8 specialized models Advanced coding & reasoning C. Shard Fine-Tuning with Overlapped Communication During parameter tuning, frameworks like PyTorch FSDP and DeepSpeed ZeRO-3 overlap PCIe gradient all-gathers and reduce-scatters with backward-pass compute. Sharding states across the 768 GB aggregate pool supports fine-tuning 70B+ architectures. However, full 70B AdamW fine-tuning necessitates mandatory CPU offloading ~840 GB optimizer states requiring at least 1.15 TB host RAM via 12x 96GB DIMMs . At 8 GPUs transferring 64 GB/s each, the aggregate 512 GB/s PCIe demand operates at ~83-89% of the EPYC 9555's 576-614 GB/s DDR5 bus, making host memory bandwidth co-limiting alongside PCIe. | | | | | | |---|---|---|---|---|---| Full 70B fine-tuning | ~35 GB shard weights + grads | = 1.15 TB 12x 96GB; ~840 GB AdamW offload | 64 GB/s PCIe Gen 5 per card 512 GB/s aggregate co-limits host DDR5 | Micro-batch 2-4 8k context | Full parameter updates on 70B+ architectures on a single node | Hyper-throughput QLoRA | ~35 GB replicated 4-bit base under DP | Dataset staging & caching | Minimal PCIe overhead only adapter gradients exchanged | Global batch 128-256 | Rapid domain adaptation without gradient checkpointing penalties | On-node RLHF / DPO | Co-located in 768 GB pool | Local system coordination | Zero multi-node network stalls | Continuous generation & updates | Actor vLLM rollout + critic/reward model co-hosted on one machine | Full 70B fine-tuning ~35 GB shard weights + grads = 1.15 TB 12x 96GB; ~840 GB AdamW offload 64 GB/s PCIe Gen 5 per card 512 GB/s aggregate co-limits host DDR5 Micro-batch 2-4 8k context Full parameter updates on 70B+ architectures on a single node Hyper-throughput QLoRA ~35 GB replicated 4-bit base under DP Dataset staging & caching Minimal PCIe overhead only adapter gradients exchanged Global batch 128-256 Rapid domain adaptation without gradient checkpointing penalties On-node RLHF / DPO Co-located in 768 GB pool Local system coordination Zero multi-node network stalls Continuous generation & updates Actor vLLM rollout + critic/reward model co-hosted on one machine 3. Orchestration Using the AMD EPYC 9555 While the unbelievably powerful RTX 6000 Blackwell GPUs handle raw matrix multiplication and token generation, feeding eight independent high-throughput accelerators requires significant host-level coordination. In an asynchronous, multi-tenant system, that's where the CPU comes into the picture. Here's how the 64-core EPYC 9555 and its 12-channel memory subsystem can keep the accelerator pool fully saturated without becoming an operational bottleneck: Host RAM for staging, not active KV paging: The EPYC 9555's 12-channel DDR5 bus achieves up to 614 GB/s theoretical peak host memory bandwidth or ~576 GB/s at DDR5-6000 . While paging dynamic KV caches across the 64 GB/s PCIe boundary during live token generation causes severe latency spikes, host memory excels at prefix-cache staging, asynchronous RAG vector buffering, and FSDP optimizer offloading. High-throughput pre-processing: Managing 8 to 32 parallel inference and diffusion streams requires quite a bit of CPU compute. The 64 Zen 5 cores handle tokenization, JSON grammar enforcement, embeddings, and network routing in parallel without thread contention. 4. Production Workloads & Throughput To translate these architectural constraints into practical capacity planning, the table below maps real-world workloads to their optimal deployment models on this 8-GPU platform. It highlights how throughput, latency, and operational efficiency shift across isolated single-GPU instances TP=1 , paired dual-GPU serving TP=2 , and asynchronous batch pipelines, showing where eliminating inter-GPU PCIe traffic gives you the best return in tokens. | | | | |---|---|---|---| Isolated 70B serving TP=1 | 8x independent 70B FP8 instances | Consistent low-latency serving | Zero inter-GPU communication; isolated memory pools | Dual-GPU 70B serving | 4x dual-GPU pairs TP=2 | ~1,500-3,200 tok/s aggregate | Expanded KV cache headroom per instance | Multi-model 8B-14B fleets TP=1 | 8x independent 8B-14B models | 10,000-18,000+ tok/s aggregate | Linear throughput scaling; zero synchronization stalls | Asynchronous render farms | 8x independent diffusion workers | 8 parallel 1080p/4K video streams | Massive local VRAM for un-tiled latent generation | Agentic swarms & RAG | 8x parallel reasoning workers | Slashes swarm execution latency | Concurrent branch execution across multiple models | Isolated 70B serving TP=1 8x independent 70B FP8 instances Consistent low-latency serving Zero inter-GPU communication; isolated memory pools Dual-GPU 70B serving 4x dual-GPU pairs TP=2 ~1,500-3,200 tok/s aggregate Expanded KV cache headroom per instance Multi-model 8B-14B fleets TP=1 8x independent 8B-14B models 10,000-18,000+ tok/s aggregate Linear throughput scaling; zero synchronization stalls Asynchronous render farms 8x independent diffusion workers 8 parallel 1080p/4K video streams Massive local VRAM for un-tiled latent generation Agentic swarms & RAG 8x parallel reasoning workers Slashes swarm execution latency Concurrent branch execution across multiple models Summary This platform isn't the 'best' for massive frontier-model experiments; that's a job for the cutting-edge in speed and memory bandwidth from NVIDIA's stable, which of course, do cost quite a bit more. Where this platform shines is concurrency. And you can get yourself an excellent bargain in compute when you run the right workloads on this type of architecture. You know, things like: Stall-free serving: Run eight separate 27B or 70B models at full speed, eliminating inter-card PCIe synchronization bubbles and maximizing memory bandwidth saturation. Deep concurrency: Host thousands of simultaneous chat sessions and agent reasoning loops with massive dedicated KV cache headroom. Microservice fleets: Co-locate dozens of specialized 8B-32B coding, vision, and tool-calling models on a single machine. Local fine-tuning: Train 70B+ models natively using your host RAM and 768 GB VRAM pool without paying cloud multi-node premiums. Availability You can find out more about these systems on our product pages: the CG480-S6053 4U GPU server /inventory/systems/msi-cg480-s6053-ai-server this build lives in, and the RTX PRO 6000 Blackwell cards /inventory/gpu/pny-nvidia-rtx-pro-6000-blackwell-workstation-edition-96gb-gddr7 themselves. We currently have 14x nodes of CG480-S6053 ready to ship . No lead time. The full L10 spec sheet PDF https://cdn.gpupartner.com/prod/msi-cg480-s6053-l10-rtx-pro-6000-se-spec-sheet-v2.pdf covers the exact build. Sources | | |---|---| Bandwidth specifications for NVLink 4 900 GB/s and NVLink 5 1,800 GB/s | | 96 GB GDDR7 buffer, 1,597 GB/s bandwidth, PCIe Gen 5.0 interface | | 64-core Zen 5 architecture and 12-channel DDR5 memory | | 671B MoE architecture and collective communication overheads | | Dense architectures, GQA, and multi-GPU parallelism overheads | | Parameter counts, hybrid layout, and FP8 checkpoint format | | Overlapping communication with compute and AdamW optimizer offloading | | Blackwell memory allocation, throughput, and FP8 KV-cache quantization | Bandwidth specifications for NVLink 4 900 GB/s and NVLink 5 1,800 GB/s 96 GB GDDR7 buffer, 1,597 GB/s bandwidth, PCIe Gen 5.0 interface 64-core Zen 5 architecture and 12-channel DDR5 memory 671B MoE architecture and collective communication overheads Dense architectures, GQA, and multi-GPU parallelism overheads Parameter counts, hybrid layout, and FP8 checkpoint format Overlapping communication with compute and AdamW optimizer offloading Blackwell memory allocation, throughput, and FP8 KV-cache quantization