Meta is wiring a million GPUs together in Ohio to work as one AI machine

Meta has completed the first phase of Prometheus, a gigawatt-scale supercluster in New Albany, Ohio, that links hundreds of thousands of Nvidia GPUs to train AI as one machine and will scale to roughly 1 million distributed GPUs, the company said in an engineering post on Monday. Prometheus trained Meta's Muse family, including the Muse Wonder models launched last week, and ran at more than 85% goodput for pre-training and more than 90% for reinforcement learning, with interruptions cut 99% from a peak of nearly 1,000 a day per 8,000 GPUs in mid-2025. Meta runs jobs on 64 of the 72 GPUs in each Nvidia rack and keeps eight as spares, and manages power swings of 30% to 40% with software called Dynamo Dimmer.

Rows of GPU racks at Meta's New Albany, Ohio, data centre, home of the Prometheus supercluster. Image: Meta Meta is wiring close to a million GPUs together in Ohio so they can train AI as one machine. In a detailed engineering post https://engineering.fb.com/2026/10/19/data-center-engineering/inside-prometheus-meta-gigawatt-scale-supercluster/ on Monday, the company opened up Prometheus, the gigawatt-scale supercluster in New Albany, Ohio, that trained its Muse models, and set out how it keeps hundreds of thousands of chips from tripping each other up. The post, by four Meta infrastructure engineers, says the first phase of Prometheus is finished and was used to train the Muse family, including the Muse Wonder models https://madrobot.blog/2026/10/12/meta-muse-wonder-frontier-ai-model-zuckerberg-price-availability/ Meta launched last week. The next stage is being built with "our next frontier models from Meta Superintelligence Labs in mind". Hundreds of thousands of GPUs acting as one Two years ago New Albany was an ordinary data centre site serving web traffic. Today, Meta says, Prometheus spans "hundreds of thousands of GPUs" across several buildings: older halls, a new AI-optimised building, quick-build "rapid deployment structures" and capacity leased from partners, all joined by miles of fibre. At full scale it will hold "roughly 1 million distributed GPUs, all functioning in lockstep". The network linking the buildings stretches from 2km to more than 100km, which meant Meta had to tune its training software around the speed of light. Meta counts millions of miles of fibre strand, tens of thousands of switches and millions of optical parts in the cluster. Three generations of Nvidia hardware Prometheus runs on Nvidia's rack-scale systems, where 72 GPUs act as a single accelerator: - Catalina GB200 NVL72 : 72 Blackwell GPUs with 192GB of memory each, about 120 kilowatts across a two-rack pair. - Clemente GB300 NVL72 : 72 Blackwell Ultra GPUs with 288GB each, more than an exaflop of FP4 compute and about 135 kilowatts in a single rack. - San Miguel Vera Rubin NVL72 : arriving "soon", fully liquid-cooled at over 170 kilowatts a rack, with 10% to 20% better performance per watt than GB300. Each GPU draws about 1,400 watts. Meta runs jobs on 64 of the 72 GPUs in each rack and keeps the other eight as spares, so a failed chip is swapped at once and the job restarts from its last checkpoint. For a 32,768-GPU training run, that means provisioning 523 racks, or 37,656 GPUs. A 99% cut in crashes The most candid part of the post is about reliability. When Prometheus first came online in mid-2025, Meta says interruptions peaked at nearly 1,000 a day for every 8,000 GPUs. A year of work, including a "stability sprint" in March and Nvidia staff embedded at the site, cut that by 99%. The biggest early culprit was signal problems across more than 600,000 NVLink connections between GB200 chips; memory faults took over as GB300 racks arrived. During Muse Wonder's training, Meta says, the cluster ran at more than 85% "goodput" for pre-training and more than 90% for reinforcement learning, with about 47,000 GPU hours between interruptions, since raised to over 54,000. Its target is about 100,000. Power swings of tens of megawatts Because tens of thousands of GPUs compute and pause in unison, Prometheus produces power swings of 30% to 40%, and starting or stopping a big job can move "more than tens of megawatts in just seconds". Meta handles this with software called Dynamo Dimmer, which gradually lowers GPU power limits as the cluster nears its electrical ceiling instead of halting work, backed by rack batteries, capacitor trays and site-level battery storage. That is the problem Meta, Google, Microsoft and Nvidia promised grid operators they would fix https://madrobot.blog/2026/10/12/google-meta-microsoft-nvidia-ai-data-centres-power-grid-ocp-vera-rubin/ earlier this month. Cooling is a patchwork: older buildings use air-assisted liquid cooling, while the purpose-built AI building pipes liquid straight from the facility. Meta says it pays the full cost of its energy in New Albany and that its newest halls use closed-loop liquid cooling to save water, without giving figures. Hyperion, its next supercluster in Richland Parish, Louisiana, is being designed with lessons from Prometheus. Why it matters Frontier AI is now limited as much by power, networking and failure rates as by chips, and Meta has published unusually hard numbers on all three. A cluster that loses less time to crashes trains bigger models faster, and Prometheus is the machine Meta is counting on to keep Muse level with OpenAI and Anthropic. Sources: Meta Engineering: Inside Prometheus https://engineering.fb.com/2026/10/19/data-center-engineering/inside-prometheus-meta-gigawatt-scale-supercluster/