As GPU clusters scale, operations span compute, network, hardware, and scheduling. Site reliability engineers (SREs) often have to examine multiple dashboards to identify the cause of training performance drops. What if a team of AI agents could perform that correlation through one interface?
NVIDIA developed the AI Factory Operations Agent to streamline its own internal NVIDIA DGX SuperPOD operations and reduce the manual effort by NVIDIA SREs. Sub-agents communicate through Model Context Protocol (MCP) that any organization can adopt to maximize AI factory uptime and performance.
Instead of requiring SREs to check each layer of the AI factory software and hardware separately, the AI Factory Operations Agent uses specialized agents for different operational domains. In production, these agents detect failures, correlate root causes across layers, and surface clear recommendations through a chat interface. SREs can interact with any agent using natural language, switch between investigations, and track every action for auditability.
This blog will share learnings and teach you how to implement the AI Factory Operations Agent in your own workflow.
The architecture: Using NVIDIA blueprints and open models #
The AI Factory Operations Agent developer example was built around the NVIDIA NemoClaw blueprint. At the center sits the user-facing agent runtime, which receives SRE queries through a chat interface and invokes the appropriate domain agents and tools.
Agents connect to infrastructure through OpenClaw plugins and MCP servers, with each MCP exposing a focused set of approved operational tools. You can work through the built-in web UI or integrate existing coding-agent interfaces through the API. This modular design lets you add new agents and infrastructure integrations without changing the existing agents.
The framework is built using NVIDIA OpenShell and is fully on-premises capable, supporting extensibility and data sovereignty. Agent inference uses locally served NVIDIA Nemotron open models, while the framework runs across internal NVIDIA DGX H100, NVIDIA DGX B200, and NVIDIA DGX GB300 production clusters. See the next section for how OpenShell provides agent governance and controls so agents work safely.
How NVIDIA built it #
The AI Factory Operations Agent runs as a Kubernetes control-plane application. Helm deploys the UI, OpenClaw gateway, MCP tool servers, inference endpoint, and NVIDIA OpenShell runtime as Kubernetes services. The chat UI sends each request to NemoClaw, which routes it to the appropriate domain agent.
For shell execution, OpenShell creates a per-session Kubernetes sandbox instead of running commands in the main gateway pod. The runtime receives the AI Factory Operations Agent workspace, selected skills, and command policy. Its lifecycle does not modify existing services or production workloads.
Security is enforced in layers. Kubernetes inspection uses the server-side run_kubectl tool, which invokes a pinned kubectl binary against registered read-only credentials. OpenShell receives neither kubectl nor Kubernetes credentials. Kubernetes service accounts and role-based access control (RBAC) define the cluster resources the tool can inspect.
Because these checks run through the same Kubernetes access path used by the runtime, the results show what the agent can and cannot do inside the cluster.
(agent) $ kubectl auth can-i get pods
yes
(agent) $ kubectl auth can-i get configmaps
yes
(agent) $ kubectl auth can-i get secrets
no
(agent) $ kubectl auth can-i create pods
no
(agent) $ kubectl auth can-i update pods
no
The AI Factory Operations Agent, sub-agents, and tools #
Internal NVIDIA deployment of the AI Factory Operations Agent uses a growing ecosystem of NVIDIA data center domain-specific agents, each providing recommendations.
Job Recovery Agent: Detects training job failures and goodput degradation; performs multi-layer root cause analysis across scheduler, application, hardware, and network layers; and recommends recovery steps when root cause is confirmed.
Hardware Agent: Triages hardware faults at the device level—PCIe errors, ECC errors, GPU issues, NVMe failures, and thermal anomalies—tracing symptoms down to the specific component.
Observability Agent: Queries historical Prometheus, telemetry, and integrated Grafana Loki logs to create dashboards dynamically and display them directly in the web UI.
Research Agent: Searches preloaded documentation including NVIDIA Mission Control manuals and DGX SuperPOD reference architectures and deployment guides. You can also upload your own runbooks and site-specific guides.
Kubernetes Agent: Provides read-only operational awareness across pods, services, deployments, GPU resources, node status, and scheduling state.
Cluster Management Agent: Interfaces with NVIDIA Base Command Manager APIs through diagnostic MCP tools for advisory cluster management and troubleshooting.
Beyond the domain agents, the framework integrates cross-cutting MCP tools including web search, code execution, and Grafana dashboards.
You can also connect your own MCP servers and APIs, adapting the framework to the schedulers, observability platforms, cluster managers, and operational procedures already used in those environments.
Learnings from using agents in the data center #
NVIDIA continues to refine the framework by measuring root cause analysis (RCA) accuracy, tracking SRE agreement, and expanding the agent ecosystem. From production use, five lessons stand out:
- Clear agent boundaries: Overlapping ownership creates noise, not signal.
- Read-only first: Accurate investigation builds SRE trust before automation.
- Cross-layer correlation: The framework connects workload, hardware, and network signals to find root causes missed by isolated tools.
- Precise MCP tools: Clear tool definitions improve routing and reduce wrong tool selection.
- Transparent reasoning : SREs need to see tool calls, evidence, and conclusions before taking action.
Bringing agent-driven operations to your AI factory #
For organizations running AI factories at scale, the preferred deployment model is a dedicated GPU-accelerated control plane isolated from the production cluster. The control plane handles agent orchestration and reasoning while the production cluster provides telemetry without consuming training GPUs or disrupting workloads.
Internal NVIDIA deployments run model inference on dedicated GPU capacity and use the existing control-plane CPU Kubernetes admin space for runtime services. Other deployments can use a local or cloud inference endpoint based on data sovereignty, cluster size, and available GPU capacity, with smaller deployments using an RTX PRO Workstation or DGX Spark.
For multi-tenant AI factories, scoped agents, tools, sandboxes, and Kubernetes permissions control the cluster data, tools, and nodes available to each tenant. Tenant SREs see only their namespaces, jobs, and nodes, while central operations retain broader cluster visibility. Each tenant can use its own set of domain-specific agents and dedicated inference hardware when required. OpenShell enforces these boundaries by running each tenant’s agents and tools in an isolated sandbox with kernel-level restrictions.
Get started #
To get started on streamlining cluster operations management, explore the AI Factory Operations Agent developer example, or access the open source framework and code directly on GitHub. The orchestration framework is available as experimental open source software; APIs, configuration, and deployment requirements may change between releases. NVIDIA customers can contact their account manager to request access to NVIDIA-developed cluster operations agents.