AI Factories are some of the most technologically complex operational environments ever created. Hundreds of thousands of GPUs, NVIDIA Spectrum-X Ethernet scale-out, scale-across and scale-in fabrics, NVIDIA NVLink scale-up fabrics, distributed schedulers, and multi-week training runs operate in continuous lockstep.These layers function as one production system even when different teams own their telemetry. In synchronous distributed training, a job advances at the pace of its slowest rank. One degraded GPU, interface, storage device, or switch path can waste useful time across many healthy resources.
The failures that hurt yield are often subtle: A GPU runs a few degrees hotter than its peers. An NVMe device loses performance only under sustained load. A switch generates a -frame storm that looks like dozens of unrelated symptoms. None of these signals necessarily crosses a static threshold. Together, they create stragglers, extended step times, and delayed checkpoints.
Diagnosis becomes harder because the evidence is split across teams. Network operators see interface counters and fabric events. Compute operators see node and GPU health. ML teams see changing step times, failed jobs, or unstable inference latency. Each view is accurate, but the relationships between them remain hidden. Triage becomes a chain of dashboard searches and cross-team handoffs, which can turn a contained fault into hours or days of lost production time before the affected node or fabric path is identified.
This post covers how NVIDIA Minerva, an AI-native AIOps platform designed around AI factory topology and workloads, addresses these operational gaps.
NVIDIA Minerva collects telemetry across the stack, maps infrastructure to active jobs in a dynamic knowledge graph, applies purpose-built machine learning, and correlates dependent symptoms into high-fidelity incidents. An AI agent-powered assistant and MCP server connected to external agentic tools give operators a conversational path through the same live evidence. The goal is practical: preserve effective GPU hours and keep model training moving.
Why conventional monitoring breaks at AI factory scale #
Rule-based monitoring assumes that a known metric crossing a known boundary indicates a known condition. That model works for clear failures. It struggles with so-called “gray failures,” where a component stays available, and the relevant issue may appear only under load or in comparison with peers. Absolute thresholds can miss the shift. Aggressive thresholds often add warnings during normal workload changes, leaving the on-call engineer to decide which deviation matters.
Scale adds alert amplification. One physical fault can produce interface errors, RX (receive) frames, congestion indicators, GPU utilization changes, and slower job progress. These observations describe the same dependency chain, but separate dashboards or alerts present them as unrelated events. The on-call engineer receives a volume of alerts and needs to manually correlate the alerts, determine the signal from the noise, analyze the root cause and the jobs exposed to it.
For operations, the workload is the unit that matters. Teams need to know which job is affected, which resources it occupies, where those resources sit in the physical and logical topology, and how current behavior differs from a healthy baseline. That context must remain current as jobs start and stop and as nodes are drained, replaced, or returned to service. Without it, a healthy component dashboard can coexist with poor job throughput and offer no explanation.
Building a live model of infrastructure and jobs #
Solving these challenges requires end-to-end visibility across the AI Factory. NVIDIA Minerva ingests metrics, logs, and topology from GPUs, Spectrum-X and NVLink switches, SuperNICs, DPUs, storage, NVIDIA NCCL, schedulers, and AI workloads. Auto-discovery supplies the relationships that raw telemetry lacks. The resulting knowledge graph maps infrastructure, scheduler, and workload elements, then updates those relationships as the environment changes. It can span multiple data centers and support navigation from a factory-wide view to an individual interface.
The topology graph is the key to changing the investigation path. Starting with an anomalous interface, an operator can identify its node, the fabric path it serves, current job placement, and nearby components in the same failure domain. Starting with a job ID, an ML engineer can trace the workload through GPU, host, NIC, switch, and log evidence. Network and compute teams can start from an infrastructure event and see the affected jobs, or conversely see an affected job and investigate the cause using NVIDIA Minerva. Everyone works from the same current causal map instead of reconciling separate inventories.
Detecting gray failures with purpose-built machine learning #
NVIDIA Minerva processes billions of metric and log data points per hour with supervised and unsupervised models developed for AI factory infrastructure and workloads. Supervised detection identifies known critical error patterns and can associate recognized conditions with specific remediation guidance. Unsupervised models cover behavior with no predefined signature. They learn normal patterns for a hardware or logical component and compare it with the relevant peers.
Peer comparison matters because a measurement can look acceptable in isolation and still be anomalous in context. Sustained errors on a network interface may look normal in isolation but compared to other interfaces, they stand out as anomalous. A GPU temperature may remain within specification while running substantially hotter than sibling GPUs under the same load. Multivariate models can combine temperature, power, clocks, utilization, and workload state to detect that degradation before the component fails outright.
Logs receive similar treatment. NLP-based anomaly detection surfaces previously unseen patterns, while supervised matching identifies known critical errors. A detection can feed scheduler workflows so an affected node is drained before another job lands on it or at a safe point between checkpoints.
Collapsing event storms into actionable incidents #
Topology-aware correlation connects detection to operator action. The correlation engine’s algorithms leverage the knowledge graph to determine which events share a dependency path. Downstream symptoms are then grouped as one incident instead of competing for attention.
The result is less alert noise and more context. NVIDIA Minerva can collapse redundant events and related metric series into a single signal with affected topology and workloads attached. NVIDIA testing shows significant reductions in notified events and tickets for supported scenarios. Operators still need to determine the root-cause using other capabilities provided by NVIDIA Minerva, but they begin with a much smaller search space. Once NVIDIA Minerva creates a high-confidence incident, teams can consume it through the platform UI or existing workflows.
Agentic debugging #
Correlated evidence still requires interpretation. NVIDIA Minerva includes a native AI agent called Sherlock which is grounded in live telemetry and the knowledge graph, so an operator can ask why a specific job slowed during a particular interval, which components changed state, or whether other nodes share the same failure domain. Sherlock enables network ops, compute ops and site reliability engineering (SRE) teams to query the system for root-cause-analysis in human language.
Sherlock can present supporting evidence and recommend remediation without requiring every operator to know the exact metric, dashboard, or query language. This gives less experienced operators a route to domain knowledge that has traditionally required escalation to a specialist. Its value in an operations workflow depends on the evidence that Sherlock provides. A recommendation leads back to the telemetry and topology that support it.
NVIDIA Minerva’s MCP server lets external agentic tools query the same operational context. Teams can incorporate live AI factory evidence into their own workflows. Mitigation can remain human approved or selected actions can be automated according to local policy and the risk of the action.
Use cases from NVIDIA AI factories #
Detecting a single link degradation in a 400K interface AI factory
In a large AI data center running many multi-node training jobs at once, an SRE noticed one job’s step time slowly getting worse. There was no hard failure, no alert and no obvious culprit. Tracking it down by hand would have meant hours of triage across GPUs, NVLink switches, SuperNICs and the network fabric while the job kept burning GPU hours at reduced throughput.
Instead, NVIDIA Minerva’s Full Stack AIOps Debugging took in telemetry from every layer of the workload topology. Anomaly detection found the one NVLink connection out of roughly 400,000 interfaces that was quietly losing bandwidth. It then linked that NVLink switch to the training jobs whose traffic crossed it. Now that the SRE could see in one UI view the failing link, the affected jobs and where both sat in the datacenter hierarchy, they drained and rescheduled the affected nodes, isolated the link for repair, and brought the jobs back to full speed. The team caught the problem before it spread across the cluster, diagnosed it in minutes instead of hours, and kept the fleet’s goodput and token production up.
SuperNIC Proactive Maintenance
A GPU node logged one kernel line: a hardware error reported by the platform’s firmware (APEI, error source 5). It was a PCIe-level fault on the node’s SuperNIC path. The error was rare enough that no static rule or threshold would have caught it, and it came with no job failure or customer complaint.
NVIDIA Minerva’s real-time rare log anomaly detection marked the message as a rare, never-before-seen event. The platform posted an alert to the cluster operations team’s Slack channel. Alongside the alert, NVIDIA Minerva explained what the error meant and laid out step-by-step remediation, starting with the error context and the system logs to check. The on-call engineer confirmed the fault and removed the node from the allocation pool before any customer workload landed on it. This saved hundreds of GPU hours a failing node would have cost in crashed and restarted jobs.
Straggler job due to NVMe power fault
The ops team was trying to explain why several training jobs were running slower than expected. The slowdown pointed to straggler nodes, but not to a cause.
NVIDIA Minerva’s full-stack job RCA dashboard traced the problem to one host whose logs showed an NVMe device failing. The drive hadn’t been able to wake from a low-power state. The node lost access to its storage, which set off a series of I/O write errors that dragged down every job running on it. With the root cause confirmed, the team drained the node and sent the faulty hardware back for RMA. The fix saved an estimated 5,530 hours of job runtime.
-frame storm
A switch issue produced RX frames, GPU utilization spikes, and thousands of metric series. The incident surfaced as many as 81 noisy events. Alerts came in from many devices, but none of them explained the slowdown. The cause was a page fault error that had forced a network switch to restart. The restart set off congestion that spread across the fabric, with RX frames propagating between switches, repeated link downs, and spikes in GPU temperature and utilization anomalies.
NVIDIA Minerva’s topology-aware correlation linked 81 separate events and anomalies to the affected job and sent them to the team as a single incident. Handling it as one incident instead of 81 cut time to repair and saved about 54 hours of total cluster uptime.
Turning operational gains into useful compute #
NVIDIA’s internal testing shows an increase in mean time between interrupts (MTBI) through proactive maintenance, coupled with a decrease in mean time to repair (MTTR) through automated root-cause analysis, and a significant reduction in alert noise. These translate into an increase in effective GPU hours by minimizing downtime.
Convergence with DSX OS #
NVIDIA Minerva is being developed in alignment convergence with NVIDIA DSX OS. It reuses and plans on integrating selected DSX OS health and resilience components such as NVIDIA Attribution. NVIDIA Minerva capabilities beginning with the data collector will become open-source parts of DSX OS. This provides a modular telemetry foundation aligned with NVIDIA reference architectures while keeping observability, analytics, and automation composable.
Bridging the gap #
As AI factories grow in scale and complexity, the gap between traditional monitoring and operational reality widens. NVIDIA Minerva bridges that gap by unifying telemetry across GPUs, networking, storage, and workloads provides the end-to-end observability to detect subtle degradations that static thresholds miss.
By collapsing noisy event storms into single actionable incidents and enabling agentic debugging through conversational AI, NVIDIA Minerva transforms triage from hours of cross-team handoffs into minutes of guided investigation.
To learn how AIOps can improve GPU utilization and reliability in your AI factory, visit https://www.nvidia.com/en-us/networking/products/software/minerva/.
For technical documentation and deployment guides, see https://networking-docs.nvidia.com/minerva/1.0.1.
To explore how NVIDIA Minerva integrates with the broader DSX OS telemetry stack, see https://docs.nvidia.com/dsx#dsx-os.