{"slug": "tens-of-thousands-of-nvidia-gpus-sit-idle-in-meta-s-prometheus-supercluster-just", "title": "Tens of thousands of Nvidia GPUs sit idle in Meta's Prometheus supercluster, just waiting for a failure", "summary": "Meta reported that more than 11 percent of the GPUs in its Prometheus supercluster in New Albany, Ohio, are either dead or held idle as hot spares, part of an effort to keep training runs alive at a site Meta says will eventually scale to a gigawatt and about a million accelerators. In a blog post published this week, Meta said it cut interruptions from roughly 1,000 per day per 8,000 GPUs at the start of the first Prometheus phase to about 11 per day per 8,000 GPUs, a 99 percent reduction, by keeping only 64 of the 72 GPUs in each GB200 and GB300 NVLink domain in use and letting a single faulty data-parallel replica restart on spare hosts instead of halting the whole cluster.", "body_md": "# Tens of thousands of Nvidia GPUs sit idle in Meta's Prometheus supercluster, just waiting for a failure\n\n[The Register](https://www.theregister.com)\n\nAnd other fun facts about how Zuckercorp aims to keep a million GPUs running long enough to train a model\n\nMore than 11 percent of the GPUs at Meta’s Prometheus supercluster in New Albany, Ohio, are either dead or sitting idle waiting to step in when the next drops out, Zuckercorp revealed this week.\n\nThe massive pool of redundant GPUs is just one of several measures the hyperscaler has taken to keep its AI [training](https://www.machinebrief.com/glossary/training) runs on track.\n\nPrometheus is among Meta's largest [compute](https://www.machinebrief.com/glossary/compute) deployments to date. Powered by [Nvidia](https://www.machinebrief.com/glossary/nvidia)'s Grace Blackwell and (soon) Vera Rubin rack systems, Meta says the supercluster will eventually scale to a gigawatt of capacity spanning about a million accelerators.\n\nBut as anyone who has worked on a large scale compute cluster understands, whether for LLM training or traditional HPC workloads, more chips inevitably means more problems.\n\nIn an eight-[GPU](https://www.machinebrief.com/glossary/gpu) node, the chance of having a GPU fail is relatively low. In a 100,000-GPU cluster, failures are unavoidable. Meta is building something that's 10x larger and requires distributed training runs across multiple datacenters. Keeping the entire thing humming long enough to reach the next checkpoint becomes a major headache.\n\nIn a blog post [published](https://engineering.fb.com/2026/10/19/data-center-engineering/inside-prometheus-meta-gigawatt-scale-supercluster/) this week, Meta offered a unique look at the work it's done to keep failures from derailing the development of models like Muse Spark or its most recent release Muse Wonder.\n\n### Idle GPUs are the cost of continuity\n\nWhen Meta began the first phase of its Prometheus deployment in New Albany, the company says, it was experiencing roughly 1,000 interruptions per day for every 8,000 GPUs.\n\nInterruptions don't mean GPU failure. They simply mean something crashed, failed, or timed out, forcing the training run to stop and roll back to the last known good checkpoint.\n\nMeta claims in the year since it's been able to cut down the number of interruptions by 99 percent, to about 11 per day per 8,000 GPUs.\n\nOne of the ways it's done this is by keeping some of Prometheus' GPUs in reserve. The cluster is currently powered by a combination of GB200 and GB300 systems, each with 72 GPUs per NVLink domain. However, only 64 of them are ever in use at any given moment. The other eight are essentially hot spares.\n\nMeta did this because a single failed GPU would take out an entire rack and interrupt the training run. Now, when a GPU fails, one of the spares can slot in and take its place. While this saves time — Meta doesn't have to drop the entire rack from the cluster or wait for someone to swap the faulty component to start training again — a failed GPU would still result in an interruption, albeit a shorter one.\n\n### Parallel resiliency\n\nTo mitigate this, Meta took advantage of the fact that large scale training jobs usually use data parallelism, where multiple copies of the model are resident within the cluster.\n\n\"When a fault can be isolated to a data-parallel model replica, we reduce the impact of a hardware failure by allowing a single faulty replica to be taken offline, while the remaining cluster repeats that step and continues productive training,\" Meta explained. \"The compromised slice simply restarts on available spare hosts and resynchronizes, converting a potential total system halt into a minor, temporary reduction in aggregate throughput.\"\n\nIn other words, Meta's training runs only get interrupted if the fault exceeds the data parallel domain. When this happens, the company has implemented a comprehensive checkpointing scheme, which you can think of a bit like auto-save or a revision history. That way if something does fail, Meta doesn't have to start over from scratch.\n\n\"Our distributed checkpointing architecture asynchronously saves the model state — including gradients and optimizer data — to multi-tier storage,\" the company wrote.\n\n### NVLink growing pains\n\nMeta says one of the most difficult challenges was making the NVLink fabrics used to connect 72 GPUs behave like one great big accelerator. Earlier H100 and H200-based systems featured just eight GPUs per NVLink domain, which meant the links were far shorter and less numerous. But this functionality was expanded nine-fold with Nvidia's Blackwell generation.\n\n\"With over 600,000 NVLink connections, even marginal signal degradation triggers frequent NVLink recovery cycles,\" the company wrote.\n\nTo overcome this, Meta had to implement a continuous telemetry system and conduct L11 diagnostic burns to isolate weak links before the systems were put into production.\n\nPower was another challenge. Ramping up and down tens of thousands of GPUs for a training run can introduce massive spikes in power consumption called transients, which can overwhelm power supply equipment.\n\nAccording to Meta, by focusing on the stability and reliability of its cluster, it was able to achieve more than 85 percent \"goodput\" — a measure of useful throughput — for pretraining and more than 90 percent for [reinforcement learning](https://www.machinebrief.com/glossary/reinforcement-learning) on its latest model Muse Wonder.\n\nThe company's ambitions don't stop here. As it works its way toward a million GPUs, Meta hopes to get its interruption rate down to just two per day for every 8,000 GPUs, or about 100,000 GPU-hours mean time between interrupts. ®\n\nGet AI news in your inbox\n\nDaily digest of what matters in AI.", "url": "https://wpnews.pro/news/tens-of-thousands-of-nvidia-gpus-sit-idle-in-meta-s-prometheus-supercluster-just", "canonical_source": "https://www.machinebrief.com/news/tens-of-thousands-of-nvidia-gpus-sit-idle-in-metas-prometheu-djhv", "published_at": "2026-10-20 20:39:42+00:00", "updated_at": "2026-10-20 22:51:00.846523+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "ai-research"], "entities": ["Meta", "Nvidia", "Prometheus", "Grace Blackwell", "Vera Rubin", "New Albany, Ohio", "Muse Spark", "Muse Wonder"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tens-of-thousands-of-nvidia-gpus-sit-idle-in-meta-s-prometheus-supercluster-just", "markdown": "https://wpnews.pro/news/tens-of-thousands-of-nvidia-gpus-sit-idle-in-meta-s-prometheus-supercluster-just.md", "text": "https://wpnews.pro/news/tens-of-thousands-of-nvidia-gpus-sit-idle-in-meta-s-prometheus-supercluster-just.txt", "jsonld": "https://wpnews.pro/news/tens-of-thousands-of-nvidia-gpus-sit-idle-in-meta-s-prometheus-supercluster-just.jsonld"}}