Meta Ohio facility slashes GPU faults to boost Muse model training reliability
Engineering improvements at Meta’s Ohio facility reduced daily GPU faults significantly, boosting reliability for Muse model training runs.
Key points
- Meta’s Prometheus cluster in Ohio will scale to roughly one million Nvidia GPUs.
- Engineering work reduced training interruptions by 99% compared to mid-2025 levels.
- The facility uses software to manage power fluctuations caused by large compute jobs.
Meta published engineering details on its Prometheus supercluster in New Albany, Ohio, which supports the training of Muse foundational models. The facility, which began as a conventional web traffic data centre, is expanding to approximately one million GPUs. According to Meta, this infrastructure now operates with significantly higher reliability than during its initial phase.
How has reliability improved?
During the first phase in mid-2025, the cluster experienced nearly 1,000 interruption events daily across 8,000 GPUs. A year of engineering work has reduced that rate by 99%. The biggest early issues involved signal quality on NVLink connections and memory faults in GB300 racks. Firmware fixes have increased the average time between interruptions to above 54,000 GPU hours.
What is the hardware architecture?
Prometheus runs Nvidia GB200 and GB300 NVL72 platforms, with Vera Rubin NVL72 racks planned for the San Miguel platform. Each rack holds 72 GPUs acting as one accelerator. The buildings are linked by fibre optics, with an aggregation network connecting them over distances of 2km to 100km. Meta also leases capacity from partners to support the distributed system.
How is power managed?
Large compute jobs cause power draw to swing by 30% to 40%, shifting tens of megawatts in seconds. Meta uses software called Dynamo Dimmer to cap GPU power gradually. This is backed by batteries at the rack, facility, and grid levels. The company pays for the full energy cost in New Albany and matches it with new clean and renewable energy sources.
Questions to ask your suppliers
Security and operations teams should verify whether suppliers monitor hardware signal integrity and memory faults at scale. Ask if vendors have automated firmware update processes that reduce manual intervention risks. Confirm that suppliers can provide detailed logs on interruption causes and mean time between failures. Ensure that procurement teams understand the hardware reserve requirements for large-scale deployments. Inform risk managers about potential power fluctuation impacts on shared infrastructure.
What to do and how to stay safe: Meta
- Review current hardware monitoring logs for patterns in signal quality or memory errors.
- Verify that firmware update procedures are automated and tested before deployment.
- Assess power infrastructure capacity to handle sudden load changes from compute jobs.
- Check supplier contracts for clauses regarding hardware reserve ratios and failure reporting.
Step-by-step guide: Browser Security Settings for Small Business Teams
General security guidance from the TechSec Feed newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
How many GPUs will Prometheus have at full scale?
Meta states the cluster will comprise roughly 1 million distributed GPUs at full scale.
What caused the early interruptions in the cluster?
The biggest early problems were signal quality on NVLink connections and memory faults in GB300 racks.
How does Meta manage power fluctuations from large jobs?
Meta uses software called Dynamo Dimmer to cap GPU power gradually, backed by batteries at multiple levels.
