Configure, Certify, and Operate Kubernetes GPU Clusters with NVIDIA DSX OS

NVIDIA released DSX OS, an open source software layer of the NVIDIA DSX AI Factory Platform, comprising three Apache 2.0-licensed projects for configuring, certifying and operating accelerated Kubernetes GPU clusters. NVIDIA AI Cluster Runtime (AICR) renders version-locked configuration recipes into bundles for existing deployment tools, NVIDIA Cluster Readiness Engine (NVCRE) tests GPUs, networking and storage with AI workloads and records certification results, and NVSentinel publishes node conditions and can cordon, taint, drain and remediate unhealthy nodes before triggering NVCRE revalidation. NVIDIA frames the tools around goodput — the volume of work successfully completed — arguing that utilization metrics can mask losses from failing nodes and that clusters need revalidation after driver patches, capacity additions and node repairs.

One underperforming GPU can slow an entire distributed training job. The node may pass basic health checks and show normal utilization, leaving operators to discover the problem only after the workload is running. These issues can appear at any point in a cluster’s lifecycle. A GPU cluster that passes readiness checks at handover changes as nodes are replaced, drivers are patched, and capacity is added. Without repeated validation, the original certification no longer reflects the cluster’s condition. This visibility gap wastes compute on failing nodes and leaves healthy nodes idle out of caution. Utilization metrics can mask these losses. Goodput, which is the volume of work successfully completed, provides a more useful measure of cluster efficiency. Maintaining cluster readiness requires validation throughout the cluster’s operational lifecycle—after configuration changes, as capacity grows, and before repaired nodes return to service. The NVIDIA DSX AI Factory Platform https://docs.nvidia.com/dsx dsx brings together technologies for designing, building, and operating efficient, resilient AI factories. Its open source software layer, NVIDIA DSX OS, includes tools for configuring, certifying, and maintaining the health of accelerated Kubernetes infrastructure. - Configure : NVIDIA AI Cluster Runtime AICR establishes a reproducible baseline for GPU-optimized Kubernetes infrastructure. - Defines version-locked configuration recipes for components such as networking, storage, and drivers. - Renders recipes as bundles for existing deployment tools. - Certify : NVIDIA Cluster Readiness Engine NVCRE validates whether the cluster is ready to run production workloads. - Tests GPUs, networking, and storage using AI workloads and diagnostics. - Records certification results and helps identify faulty nodes. - Operate : NVSentinel monitors cluster health and supports fault remediation and revalidation. - Publishes node conditions based on GPU, node, and network health. - When configured, cordons or taints unhealthy nodes, drains workloads, and remediates faults. - Can trigger revalidation with NVCRE before nodes return to service. All three are open source projects licensed under Apache 2.0 and welcome external contributions. Before you begin, install and configure AICR https://github.com/nvidia/aicr readme , NVCRE https://github.com/NVIDIA/cluster-readiness-engine readme , and NVSentinel https://github.com/NVIDIA/NVSentinel readme using their installation guides. The examples assume access to a Kubernetes GPU cluster with the required CLIs, controllers, dependencies, and NVCRE test provider configured. This walkthrough follows one cluster through its lifecycle: configuration from a recipe, certification before production use, monitoring during operation, and recertification after repair. Each command is a step in that sequence. Configure: A baseline you can reproduce AICR defines the cluster’s software baseline in version-controlled files and renders them into artifacts your existing deployment tooling applies. It generates and validates; your deployer applies and reconciles them. Declarative templates help detect configuration drift from manual changes and keep node configurations consistent after repairs. You describe the target environment. AICR resolves it against a library of validated overlays and emits a recipe: aicr recipe --service eks --accelerator h100 --os ubuntu \ --intent inference --platform dynamo -o recipe.yaml Each component entry in the resulting recipe pins a version and declares which dependencies must be healthy before the component is installed: apiVersion: aicr.run/v1alpha2 componentRefs: - name: dynamo-platform chart: dynamo-platform namespace: dynamo-system source: https://helm.ngc.nvidia.com/nvidia/ai-dynamo type: Helm version: 1.4.2 valuesFile: components/dynamo-platform/values.yaml dependencyRefs: - grove - cert-manager - kube-prometheus-stack - gpu-operator - kai-scheduler The recipe includes three key properties: - Pinned versions: The version is fixed, helping you reproduce the software configuration and investigate failures. In contrast, a floating tag can point to a different software version over time, making failures harder to reproduce. - A dependency graph: dependencyRefs declares what must be installed and healthy first. AICR resolves the graph when it renders the bundle, so the ordering is already encoded in the artifacts your deployer applies. You don’t need to specify that cert-manager comes before NVSentinel, and adding a component later doesn’t require renumbering. - Health assertions per component: Each entry carries a healthCheckAsserts block that runs after installation and checks whether the component meets its configured health criteria. Kubernetes can report a pod stuck in CrashLoopBackOff as Running , so a check based only on the pod phase can miss a broken component. The shipped assertions filter on the container waiting reason rather than the pod phase alone. Rendering the recipe is a separate step that connects AICR to your existing pipeline: aicr bundle --recipe recipe.yaml --deployer argocd --output ./bundles --deployer accepts helm , argocd , argocd-helm , flux , or helmfile , so the same validated recipe becomes Argo CD Application manifests, Flux resources, a Helmfile release graph, or plain Helm commands. You can keep your existing deployment tooling and use AICR to generate its configuration artifacts. Because the recipe is a file, you can use version history, code review, and drift detection to manage the baseline. Check a live cluster against the recipe's constraints aicr validate --recipe recipe.yaml Compare a golden baseline against the live cluster, non-zero exit on drift aicr diff --baseline ./golden.yaml --target cm://default/aicr-snapshot --fail-on-drift --fail-on-drift turns the drift check into a continuous integration CI gate rather than a report. Certify: Prove the cluster will run the job The cluster is now built to a known baseline. It still needs workload validation. Standard health checks confirm that basic services are running. They don’t confirm that the cluster can run a demanding multinode job across its interconnect without stalling. NVCRE validates workload readiness by running training jobs and NVIDIA Collective Communication Library NCCL https://github.com/NVIDIA/nccl readme tests. NVSentinel manages validation requests, scheduling gates, quarantine, and repair, while NVCRE runs tests and records results. Table 1 shows how these responsibilities fit into the node validation lifecycle. | Step | Responsible component | Action | Outcome | |---|---|---|---| | 1. Request and gate | NVSentinel lifecycle manager; operators for upgrades or fabric changes | Create a ValidationRequest for new or repaired nodes. After upgrades or fabric changes, operators cordon the nodes and create the request. | A cordon or taint blocks scheduling on the nodes during validation. | | 2. Test and attribute | NVCRE through nvcre-provider | Run workloads and diagnostic tests. Bisect failing groups to identify suspect nodes. | Test results identify suspect nodes. NVCRE doesn’t change the node state. | | 3. Record and evaluate | NVCRE and NVSentinel lifecycle manager | NVCRE records pass/fail results, failing node IDs, and failure reasons in the Certification resource. The lifecycle manager evaluates the results. | The validation outcome determines whether the scheduling gate can be removed. | | 4. Resolve the gate | NVSentinel lifecycle manager | Remove or retain the scheduling gate based on validation results and any remaining tests. | Pass: remove the configured gate once no other tests hold the node. Failure after retries: keep the node cordoned. | Table 1. Node validation duties, actions, and outcomes A certification identifies the nodes to test, the categories to run, and the thresholds that determine whether the tests pass or fail: apiVersion: nvcre.nvidia.com/v1alpha1 kind: Certification metadata: name: gpu-cluster-cert spec: target: nodeSelector: nvidia.com/gpu.present: "true" categories: - domain: communication variant: nccl-all-reduce - domain: training variant: nemotron5-8b options: thresholds: busBandwidthGBps: "value = 300" nvcrectl certification run --cert-file certification.yaml --wait nvcrectl certification report gpu-cluster-cert -n <namespace The report shows each category’s status, runtime, measured bandwidth, and the reason for any failed nodes. Thresholds make certification a performance gate. None are configured by default. Without the options.thresholds block above, a run reports its measurements and fails when the tests can’t start. A node performing below the target can pass. Thresholds help catch performance degradation rather than breakage. The catalog covers NCCL communication variants, the NVIDIA Data Center GPU Manager DCGM level 4 diagnostic suite, and pretraining workloads using NVIDIA NeMo Framework with 8-billion- and 56-billion-parameter models. A WorkloadRun resource turns your own containerized test into a certification check. NVCRE infers the rest from the target nodes, so the manifest stays concise on any supported platform. These tests can help identify performance issues that aren’t apparent in telemetry. A GPU running just below its performance target might not produce Xid errors, breach monitoring thresholds, or cause node metrics to appear abnormal. That one GPU can slow a synchronous training job to its slowest rank, and operators may find out hours later, or only after a user complains. An NCCL bandwidth test can uncover this performance degradation. Isolating the cause matters as much. If a 64-node all-reduce yields poor bandwidth, every node is suspect, and bisecting by hand can be time-consuming. NVCRE splits failing groups and reruns the tests until it reaches a minimum group size, then reports the suspect nodes with a reason for each. Run NVCRE before handing a cluster or new capacity to a tenant and when repaired hardware rejoins the pool. Operate: Detection and remediation under load The certified cluster is now running workloads and needs to stay healthy. AICR and NVCRE provide point-in-time validation. NVSentinel runs continuously alongside workloads. NVSentinel watches signals the cluster already produces: DCGM health checks, Xid errors, network interface card NIC states, and cloud provider maintenance events. It doesn’t consume GPU time, leaving GPUs available for workloads. NVSentinel installs in monitoring mode by default: it reports faults as node conditions and won’t cordon, evict, or remediate nodes. You can roll it out on a live cluster while you evaluate its reports. You can enable quarantine, draining, and remediation once you trust the reported signals. Detection publishes a health event; a Fault Quarantine ruleset decides what happens to the node. Rulesets use Common Expression Language CEL expressions in your values file: fault-quarantine: ruleSets: - enabled: true version: "1" name: "GPU fatal error ruleset" priority: 100 match: all: - kind: "HealthEvent" expression: "event.agent == 'gpu-health-monitor' && event.componentClass == 'GPU' && event.isFatal == true" - kind: "Node" expression: | 'k8saas.nvidia.com/ManagedByNVSentinel' in node.metadata.labels && node.metadata.labels 'k8saas.nvidia.com/ManagedByNVSentinel' == "false" cordon: shouldCordon: true taint: key: "nvidia.com/gpu-error" value: "fatal" effect: "NoSchedule" A cordon or taint provides a signal that your scheduler, alerting, and lifecycle tooling can use without integrating directly with NVSentinel. A circuit breaker limits the impact by capping how many GPU nodes can be quarantined in a sliding window. The Fault Quarantine configuration guide https://docs.nvidia.com/nvsentinel/configuration/fault-quarantine covers the bounds and the rest of the ruleset options. Not every fault takes the same path. A known fatal error goes straight to remediation. The health events analyzer handles other cases by detecting patterns across a node’s failure history, so a node that fails repeatedly becomes eligible for investigation based on the accumulated record rather than any single event. A health event with the recommended action CONTACT SUPPORT requires manual investigation or support intervention. The loop: How readiness becomes continuous A node fails, gets repaired, and needs to rejoin the pool. Validation before the node returns to service closes the operational loop, but manual handoffs can interrupt the process. Manual certification produces a point-in-time result that routine operations can make outdated. Someone has to notice that the cluster changed, rerun validation, and then read the output and act on it. NVSentinel and NVCRE can help automate this process. Two lifecycle events trigger revalidation automatically when configured. The NVSentinel lifecycle controller https://github.com/NVIDIA/NVSentinel/blob/main/docs/validation.md creates a ValidationRequest when a node joins the cluster and again when a node returns after remediation. It gates scheduling on the node using a cordon or taint, then uses NVCRE to run the tests and record the results. The controller evaluates those results and removes the gate after the tests pass, so a node that still fails validation cannot silently rejoin the pool. When you configure nvcre-provider https://github.com/NVIDIA/NVSentinel/blob/main/distros/kubernetes/nvsentinel/values-nvcre.yaml L59 as its test provider, those tests are NCCL or NVIDIA Nemotron runs through NVCRE: global: lifecycleManager: enabled: true fault-quarantine: validation: enabled: true lifecycle-manager: config: schedulingGate: cordon: remove: true The lifecycle-manager chart turns on new-node validation; the fault-quarantine flag opts in to post-remediation validation separately. schedulingGate.cordon sets the cordon as the scheduling gate for validation, and remove: true uncordons the node once its tests pass. The process is different for in-place driver upgrades, operator upgrades, or fabric changes. These changes don’t automatically trigger validation. Because the baseline is a file, aicr diff --fail-on-drift fails a pipeline stage when a live cluster drifts from its recipe, providing a trigger for a certification run in the same pipeline that made the change. Cordon the affected nodes, then create a validation request naming them and the tests to run: apiVersion: nvsentinel.nvidia.com/v1alpha1 kind: ValidationRequest metadata: name: post-driver-upgrade spec: nodes: - name: node-a - name: node-b tests: - nccl-all-reduce - nemotron-5b The controller also acts on the result. If the request passes, it removes the scheduling gate once no other tests hold the node. If the test fails after its retries, the node remains cordoned. A fault can also surface during validation: GPUs may start thermally throttling during a training test. NVSentinel quarantines the node, and the controller retries the test after remediation. The node stays gated until validation passes. The request and the node both carry state you can query: kubectl get validationrequests kubectl get nodes -l nvsentinel.nvidia.com/validation-state=validation-failed Recovery requires no manual node edits: a later request that passes the same tests clears the failure and removes the gate. Deleting all failed validation requests that reference the node can release it without rerunning validation. The handoff has a clear boundary. NVCRE stops at attribution, recording which nodes failed and why; NVSentinel owns quarantine and repair. Two systems cordoning the same node could produce double-cordons and desynchronized cleanup. This separation also enables you to use your own node lifecycle tooling. A platform team can consume the same certification results and take over NVSentinel’s role. Extending the stack means writing a ruleset or a policy, rather than forking a project. What this changes for operators AICR, NVCRE, and NVSentinel help operators make configuration, validation, and recovery repeatable. With remediation and lifecycle validation enabled, NVSentinel can coordinate repair and testing before nodes return to service, reducing manual handoffs. Table 2 outlines what these tools handle and what operators still manage. | Operational task | What changes | What operators still manage | |---|---|---| | Maintain the software baseline | AICR provides version-locked recipes and generates configuration artifacts for existing deployment tools. Operators can review deviations from a defined baseline. | Approve baseline changes, deploy the artifacts, run drift checks, and reconcile differences. | | Validate new or repaired capacity | NVCRE runs selected workloads and diagnostics and records readiness results. With lifecycle validation configured, NVSentinel can coordinate testing before nodes return to service. | Select tests and thresholds appropriate to the cluster and investigate failed results. | | Recover unhealthy nodes | When enabled, NVSentinel coordinates quarantine, draining, remediation, and revalidation, reducing manual handoffs between these actions. | Define quarantine rules, remediation limits, and escalation procedures; handle faults that automated remediation cannot resolve. | | Validate infrastructure changes | Operators can incorporate baseline checks and workload validation into their upgrade workflows. | Trigger validation after in-place driver, operator, or fabric changes; these changes do not automatically trigger revalidation. | Table 2. How AICR, NVCRE, and NVSentinel change operational tasks and which responsibilities remain with operators Get started Start with the project repositories for installation instructions and examples: You can also: - Get hands-on with these tools at the NVIDIA DSX OS AI Inference and Contributor Workshop https://events.nvidia.com/dsx-os-inference-workshop , held alongside KubeCon + CloudNativeCon North America. Separate registration is required, and seating is limited. - Visit NVIDIA booth 170 at KubeCon https://www.nvidia.com/en-us/events/kubecon-and-cloudnative-con-north-america/ for more Kubernetes deep dives and live demos. - Join the DSX AI Factory Slack community https://inviter.co/dsx-ai-factory to continue the conversation. - Explore open-source, modular components in the NVIDIA DSX AI Factory GitHub https://github.com/dsx-ai-factory .