What makes this interesting isn't the peak number — it's the sustained utilization. Traditional GPU clusters hit 30-40% MFU on real LLM workloads because NVLink and InfiniBand become bottlenecks once you cross 256 GPUs. The WSE-3's on-wafer fabric eliminates that hop entirely. Memory bandwidth stays at 21 PB/s per system whether you're running a 7B or 70B model, and the compiler maps tensor parallelism across the wafer without MPI rank shuffling. I've seen internal benchmarks where a 16-system CS-4 cluster held 85% MFU on a 400B parameter training run for weeks. That's the difference between "theoretical peak" and "what you actually bill for."
The rack-level CDU is the unsung hero here. Each CS-4 draws ~23 kW at the cold plate. The CDU delivers 1.5 L/min per system at 35°C inlet, and the manifold design lets you service one node without draining the loop. That's a operational detail that matters when you're running 24/7 and a single node failure costs $180k/day in lost training time. Cerebras also added hot-swap power shelves — 6× 3.2 kW titanium units per system — so a PSU failure doesn't take the whole wafer offline.
Software stack caught up too. The 2.4 SDK release added native PyTorch 2.3 support, FlashAttention-3 kernels tuned for the 48 KB per-core SRAM, and a new csrun
launcher that handles gang scheduling across racks without Slurm wrappers. You still write standard PyTorch; the graph compiler lowers to the wafer's dataflow fabric. Debug tooling improved — csdbg
now shows per-core stall cycles and SRAM pressure in a flame graph, which saved me two days tracking down a pipeline bubble in a MoE expert routing kernel.
Trade-offs remain real. You're locked to Cerebras' compiler pipeline; custom CUDA kernels don't port. Model parallelism strategy is fixed at wafer granularity — no pipeline parallelism across systems yet, so activation memory scales with model size per wafer. And the $2.5M per system price tag means you need sustained, large-scale workloads to amortize. But if you're training frontier models at 100B+ parameters and your GPU cluster spends half its cycles waiting on all-reduce, the CS-4 rack density starts looking like the only way to keep the power bill honest.
[NVIDIA B200 vs LPUs: Why Software Optimization Changes Everything 12d ago](/en/news/5355/)
[Next Relay. →](/en/news/6935/)