{"slug": "configure-certify-and-operate-kubernetes-gpu-clusters-with-nvidia-dsx-os", "title": "Configure, Certify, and Operate Kubernetes GPU Clusters with NVIDIA DSX OS", "summary": "NVIDIA released DSX OS, an open source software layer of the NVIDIA DSX AI Factory Platform, comprising three Apache 2.0-licensed projects for configuring, certifying and operating accelerated Kubernetes GPU clusters. NVIDIA AI Cluster Runtime (AICR) renders version-locked configuration recipes into bundles for existing deployment tools, NVIDIA Cluster Readiness Engine (NVCRE) tests GPUs, networking and storage with AI workloads and records certification results, and NVSentinel publishes node conditions and can cordon, taint, drain and remediate unhealthy nodes before triggering NVCRE revalidation. NVIDIA frames the tools around goodput — the volume of work successfully completed — arguing that utilization metrics can mask losses from failing nodes and that clusters need revalidation after driver patches, capacity additions and node repairs.", "body_md": "One underperforming GPU can slow an entire distributed training job. The node may pass basic health checks and show normal utilization, leaving operators to discover the problem only after the workload is running.\n\nThese issues can appear at any point in a cluster’s lifecycle. A GPU cluster that passes readiness checks at handover changes as nodes are replaced, drivers are patched, and capacity is added. Without repeated validation, the original certification no longer reflects the cluster’s condition.\n\nThis visibility gap wastes compute on failing nodes and leaves healthy nodes idle out of caution. Utilization metrics can mask these losses. Goodput, which is the volume of work successfully completed, provides a more useful measure of cluster efficiency. Maintaining cluster readiness requires validation throughout the cluster’s operational lifecycle—after configuration changes, as capacity grows, and before repaired nodes return to service.\n\nThe [NVIDIA DSX AI Factory Platform](https://docs.nvidia.com/dsx#dsx) brings together technologies for designing, building, and operating efficient, resilient AI factories. Its open source software layer, NVIDIA DSX OS, includes tools for configuring, certifying, and maintaining the health of accelerated Kubernetes infrastructure.\n\n- **Configure** : NVIDIA AI Cluster Runtime (AICR) establishes a reproducible baseline for GPU-optimized Kubernetes infrastructure.\n  - Defines version-locked configuration recipes for components such as networking, storage, and drivers.\n  - Renders recipes as bundles for existing deployment tools.\n- **Certify** : NVIDIA Cluster Readiness Engine (NVCRE) validates whether the cluster is ready to run production workloads.\n  - Tests GPUs, networking, and storage using AI workloads and diagnostics.\n  - Records certification results and helps identify faulty nodes.\n- **Operate** : NVSentinel monitors cluster health and supports fault remediation and revalidation.\n  - Publishes node conditions based on GPU, node, and network health.\n  - When configured, cordons or taints unhealthy nodes, drains workloads, and remediates faults.\n  - Can trigger revalidation with NVCRE before nodes return to service.\n\nAll three are open source projects licensed under Apache 2.0 and welcome external contributions.\n\nBefore you begin, install and configure [AICR](https://github.com/nvidia/aicr#readme), [NVCRE](https://github.com/NVIDIA/cluster-readiness-engine#readme), and [NVSentinel](https://github.com/NVIDIA/NVSentinel#readme) using their installation guides. The examples assume access to a Kubernetes GPU cluster with the required CLIs, controllers, dependencies, and NVCRE test provider configured.\n\nThis walkthrough follows one cluster through its lifecycle: configuration from a recipe, certification before production use, monitoring during operation, and recertification after repair. Each command is a step in that sequence.\n\n## Configure: A baseline you can reproduce\n\nAICR defines the cluster’s software baseline in version-controlled files and renders them into artifacts your existing deployment tooling applies. It generates and validates; your deployer applies and reconciles them. Declarative templates help detect configuration drift from manual changes and keep node configurations consistent after repairs.\n\nYou describe the target environment. AICR resolves it against a library of validated overlays and emits a recipe:\n\n```\naicr recipe --service eks --accelerator h100 --os ubuntu \\\n  --intent inference --platform dynamo -o recipe.yaml\n```\n\nEach component entry in the resulting recipe pins a version and declares which dependencies must be healthy before the component is installed:\n\n```\napiVersion: aicr.run/v1alpha2\ncomponentRefs:\n  - name: dynamo-platform\n    chart: dynamo-platform\n    namespace: dynamo-system\n    source: https://helm.ngc.nvidia.com/nvidia/ai-dynamo\n    type: Helm\n    version: 1.4.2\n    valuesFile: components/dynamo-platform/values.yaml\n    dependencyRefs:\n      - grove\n      - cert-manager\n      - kube-prometheus-stack\n      - gpu-operator\n      - kai-scheduler\n```\n\nThe recipe includes three key properties:\n\n- **Pinned versions:** The version is fixed, helping you reproduce the software configuration and investigate failures. In contrast, a floating tag can point to a different software version over time, making failures harder to reproduce.\n- **A dependency graph:**`dependencyRefs` declares what must be installed and healthy first. AICR resolves the graph when it renders the bundle, so the ordering is already encoded in the artifacts your deployer applies. You don’t need to specify that cert-manager comes before NVSentinel, and adding a component later doesn’t require renumbering.\n- **Health assertions per component:** Each entry carries a`healthCheckAsserts` block that runs after installation and checks whether the component meets its configured health criteria. Kubernetes can report a pod stuck in`CrashLoopBackOff` as`Running` , so a check based only on the pod phase can miss a broken component. The shipped assertions filter on the container waiting reason rather than the pod phase alone.\n\nRendering the recipe is a separate step that connects AICR to your existing pipeline:\n\n```\naicr bundle --recipe recipe.yaml --deployer argocd --output ./bundles\n```\n\n`--deployer` accepts `helm`, `argocd`, `argocd-helm`, `flux`, or `helmfile`, so the same validated recipe becomes Argo CD `Application` manifests, Flux resources, a Helmfile release graph, or plain Helm commands. You can keep your existing deployment tooling and use AICR to generate its configuration artifacts.\n\nBecause the recipe is a file, you can use version history, code review, and drift detection to manage the baseline.\n\n```\n# Check a live cluster against the recipe's constraints\naicr validate --recipe recipe.yaml\n\n# Compare a golden baseline against the live cluster, non-zero exit on drift\naicr diff --baseline ./golden.yaml --target cm://default/aicr-snapshot --fail-on-drift\n```\n\n`--fail-on-drift` turns the drift check into a continuous integration (CI) gate rather than a report.\n\n## Certify: Prove the cluster will run the job\n\nThe cluster is now built to a known baseline. It still needs workload validation. Standard health checks confirm that basic services are running. They don’t confirm that the cluster can run a demanding multinode job across its interconnect without stalling. NVCRE validates workload readiness by running training jobs and NVIDIA Collective Communication Library [(NCCL)](https://github.com/NVIDIA/nccl#readme) tests.\n\nNVSentinel manages validation requests, scheduling gates, quarantine, and repair, while NVCRE runs tests and records results. Table 1 shows how these responsibilities fit into the node validation lifecycle.\n\n| **Step** | **Responsible component** | **Action** | **Outcome** | \n|---|---|---|---|\n| 1. Request and gate | NVSentinel lifecycle manager; operators for upgrades or fabric changes | Create a `ValidationRequest` for new or repaired nodes. After upgrades or fabric changes, operators cordon the nodes and create the request. | A cordon or taint blocks scheduling on the nodes during validation. | \n| 2. Test and attribute | NVCRE through `nvcre-provider` | Run workloads and diagnostic tests. Bisect failing groups to identify suspect nodes. | Test results identify suspect nodes. NVCRE doesn’t change the node state. | \n| 3. Record and evaluate | NVCRE and NVSentinel lifecycle manager | NVCRE records pass/fail results, failing node IDs, and failure reasons in the `Certification` resource. The lifecycle manager evaluates the results. | The validation outcome determines whether the scheduling gate can be removed. | \n| 4. Resolve the gate | NVSentinel lifecycle manager | Remove or retain the scheduling gate based on validation results and any remaining tests. | **Pass:** remove the configured gate once no other tests hold the node.**Failure after retries:** keep the node cordoned. | \n\n*Table 1. Node validation duties, actions, and outcomes*\n\nA certification identifies the nodes to test, the categories to run, and the thresholds that determine whether the tests pass or fail:\n\n```\napiVersion: nvcre.nvidia.com/v1alpha1\nkind: Certification\nmetadata:\n  name: gpu-cluster-cert\nspec:\n  target:\n    nodeSelector:\n      nvidia.com/gpu.present: \"true\"\n  categories:\n    - domain: communication\n      variant: nccl-all-reduce\n    - domain: training\n      variant: nemotron5-8b\n  options:\n    thresholds:\n      busBandwidthGBps: \"value >= 300\"\nnvcrectl certification run --cert-file certification.yaml --wait\nnvcrectl certification report gpu-cluster-cert -n <namespace>\n```\n\nThe report shows each category’s status, runtime, measured bandwidth, and the reason for any failed nodes.\n\nThresholds make certification a performance gate. None are configured by default. Without the `options.thresholds` block above, a run reports its measurements and fails when the tests can’t start. A node performing below the target can pass. Thresholds help catch performance degradation rather than breakage.\n\nThe catalog covers NCCL communication variants, the NVIDIA Data Center GPU Manager (DCGM) level 4 diagnostic suite, and pretraining workloads using NVIDIA NeMo Framework with 8-billion- and 56-billion-parameter models. A `WorkloadRun` resource turns your own containerized test into a certification check. NVCRE infers the rest from the target nodes, so the manifest stays concise on any supported platform.\n\nThese tests can help identify performance issues that aren’t apparent in telemetry. A GPU running just below its performance target might not produce Xid errors, breach monitoring thresholds, or cause node metrics to appear abnormal. That one GPU can slow a synchronous training job to its slowest rank, and operators may find out hours later, or only after a user complains. An NCCL bandwidth test can uncover this performance degradation.\n\nIsolating the cause matters as much. If a 64-node all-reduce yields poor bandwidth, every node is suspect, and bisecting by hand can be time-consuming. NVCRE splits failing groups and reruns the tests until it reaches a minimum group size, then reports the suspect nodes with a reason for each.\n\nRun NVCRE before handing a cluster or new capacity to a tenant and when repaired hardware rejoins the pool.\n\n## Operate: Detection and remediation under load\n\nThe certified cluster is now running workloads and needs to stay healthy. AICR and NVCRE provide point-in-time validation. NVSentinel runs continuously alongside workloads.\n\nNVSentinel watches signals the cluster already produces: DCGM health checks, Xid errors, network interface card (NIC) states, and cloud provider maintenance events. It doesn’t consume GPU time, leaving GPUs available for workloads.\n\nNVSentinel installs in monitoring mode by default: it reports faults as node conditions and won’t cordon, evict, or remediate nodes. You can roll it out on a live cluster while you evaluate its reports. You can enable quarantine, draining, and remediation once you trust the reported signals.\n\nDetection publishes a health event; a Fault Quarantine ruleset decides what happens to the node. Rulesets use Common Expression Language (CEL) expressions in your values file:\n\n```\nfault-quarantine:\n  ruleSets:\n    - enabled: true\n      version: \"1\"\n      name: \"GPU fatal error ruleset\"\n      priority: 100\n      match:\n        all:\n          - kind: \"HealthEvent\"\n            expression: \"event.agent == 'gpu-health-monitor' && event.componentClass == 'GPU' && event.isFatal == true\"\n          - kind: \"Node\"\n            expression: |\n              !('k8saas.nvidia.com/ManagedByNVSentinel' in node.metadata.labels &&\n                node.metadata.labels['k8saas.nvidia.com/ManagedByNVSentinel'] == \"false\")\n      cordon:\n        shouldCordon: true\n      taint:\n        key: \"nvidia.com/gpu-error\"\n        value: \"fatal\"\n        effect: \"NoSchedule\"\n```\n\nA cordon or taint provides a signal that your scheduler, alerting, and lifecycle tooling can use without integrating directly with NVSentinel.\n\nA circuit breaker limits the impact by capping how many GPU nodes can be quarantined in a sliding window. The [Fault Quarantine configuration guide](https://docs.nvidia.com/nvsentinel/configuration/fault-quarantine) covers the bounds and the rest of the ruleset options.\n\nNot every fault takes the same path. A known fatal error goes straight to remediation. The health events analyzer handles other cases by detecting patterns across a node’s failure history, so a node that fails repeatedly becomes eligible for investigation based on the accumulated record rather than any single event.\n\nA health event with the recommended action `CONTACT_SUPPORT` requires manual investigation or support intervention.\n\n## The loop: How readiness becomes continuous\n\nA node fails, gets repaired, and needs to rejoin the pool. Validation before the node returns to service closes the operational loop, but manual handoffs can interrupt the process. Manual certification produces a point-in-time result that routine operations can make outdated. Someone has to notice that the cluster changed, rerun validation, and then read the output and act on it. NVSentinel and NVCRE can help automate this process.\n\nTwo lifecycle events trigger revalidation automatically when configured. The NVSentinel [lifecycle controller](https://github.com/NVIDIA/NVSentinel/blob/main/docs/validation.md) creates a `ValidationRequest` when a node joins the cluster and again when a node returns after remediation. It gates scheduling on the node using a cordon or taint, then uses NVCRE to run the tests and record the results. The controller evaluates those results and removes the gate after the tests pass, so a node that still fails validation cannot silently rejoin the pool. When you configure [`nvcre-provider`](https://github.com/NVIDIA/NVSentinel/blob/main/distros/kubernetes/nvsentinel/values-nvcre.yaml#L59) as its test provider, those tests are NCCL or NVIDIA Nemotron runs through NVCRE:\n\n```\nglobal:\n  lifecycleManager:\n    enabled: true\n\nfault-quarantine:\n  validation:\n    enabled: true\n\nlifecycle-manager:\n  config:\n    schedulingGate:\n      cordon:\n        remove: true\n```\n\nThe `lifecycle-manager` chart turns on new-node validation; the `fault-quarantine` flag opts in to post-remediation validation separately. `schedulingGate.cordon` sets the cordon as the scheduling gate for validation, and `remove: true` uncordons the node once its tests pass.\n\nThe process is different for in-place driver upgrades, operator upgrades, or fabric changes. These changes don’t automatically trigger validation. Because the baseline is a file, `aicr diff --fail-on-drift` fails a pipeline stage when a live cluster drifts from its recipe, providing a trigger for a certification run in the same pipeline that made the change.\n\nCordon the affected nodes, then create a validation request naming them and the tests to run:\n\n```\napiVersion: nvsentinel.nvidia.com/v1alpha1\nkind: ValidationRequest\nmetadata:\n  name: post-driver-upgrade\nspec:\n  nodes:\n    - name: node-a\n    - name: node-b\n  tests:\n    - nccl-all-reduce\n    - nemotron-5b\n```\n\nThe controller also acts on the result. If the request passes, it removes the scheduling gate once no other tests hold the node. If the test fails after its retries, the node remains cordoned.\n\nA fault can also surface during validation: GPUs may start thermally throttling during a training test. NVSentinel quarantines the node, and the controller retries the test after remediation. The node stays gated until validation passes.\n\nThe request and the node both carry state you can query:\n\n```\nkubectl get validationrequests\nkubectl get nodes -l nvsentinel.nvidia.com/validation-state=validation-failed\n```\n\nRecovery requires no manual node edits: a later request that passes the same tests clears the failure and removes the gate. Deleting all failed validation requests that reference the node can release it without rerunning validation.\n\nThe handoff has a clear boundary. NVCRE stops at attribution, recording which nodes failed and why; NVSentinel owns quarantine and repair. Two systems cordoning the same node could produce double-cordons and desynchronized cleanup.\n\nThis separation also enables you to use your own node lifecycle tooling. A platform team can consume the same certification results and take over NVSentinel’s role. Extending the stack means writing a ruleset or a policy, rather than forking a project.\n\n## What this changes for operators\n\nAICR, NVCRE, and NVSentinel help operators make configuration, validation, and recovery repeatable. With remediation and lifecycle validation enabled, NVSentinel can coordinate repair and testing before nodes return to service, reducing manual handoffs. Table 2 outlines what these tools handle and what operators still manage.\n\n| **Operational task** | **What changes** | **What operators still manage** | \n|---|---|---|\n| Maintain the software baseline | AICR provides version-locked recipes and generates configuration artifacts for existing deployment tools. Operators can review deviations from a defined baseline. | Approve baseline changes, deploy the artifacts, run drift checks, and reconcile differences. | \n| Validate new or repaired capacity | NVCRE runs selected workloads and diagnostics and records readiness results. With lifecycle validation configured, NVSentinel can coordinate testing before nodes return to service. | Select tests and thresholds appropriate to the cluster and investigate failed results. | \n| Recover unhealthy nodes | When enabled, NVSentinel coordinates quarantine, draining, remediation, and revalidation, reducing manual handoffs between these actions. | Define quarantine rules, remediation limits, and escalation procedures; handle faults that automated remediation cannot resolve. | \n| Validate infrastructure changes | Operators can incorporate baseline checks and workload validation into their upgrade workflows. | Trigger validation after in-place driver, operator, or fabric changes; these changes do not automatically trigger revalidation. | \n\n*Table 2. How AICR, NVCRE, and NVSentinel change operational tasks and which responsibilities remain with operators*\n\n## Get started\n\nStart with the project repositories for installation instructions and examples:\n\nYou can also:\n\n- Get hands-on with these tools at the [NVIDIA DSX OS AI Inference and Contributor Workshop](https://events.nvidia.com/dsx-os-inference-workshop) , held alongside KubeCon + CloudNativeCon North America. Separate registration is required, and seating is limited.\n- Visit NVIDIA booth 170 at [KubeCon](https://www.nvidia.com/en-us/events/kubecon-and-cloudnative-con-north-america/) for more Kubernetes deep dives and live demos.\n- Join the [DSX AI Factory Slack community](https://inviter.co/dsx-ai-factory) to continue the conversation.\n- Explore open-source, modular components in the [NVIDIA DSX AI Factory GitHub](https://github.com/dsx-ai-factory) .", "url": "https://wpnews.pro/news/configure-certify-and-operate-kubernetes-gpu-clusters-with-nvidia-dsx-os", "canonical_source": "https://developer.nvidia.com/blog/configure-certify-and-operate-kubernetes-gpu-clusters-with-nvidia-dsx-os/", "published_at": "2026-11-05 17:00:00+00:00", "updated_at": "2026-11-05 17:22:08.268051+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-chips", "developer-tools", "artificial-intelligence"], "entities": ["NVIDIA", "NVIDIA DSX OS", "NVIDIA DSX AI Factory Platform", "NVIDIA AI Cluster Runtime", "NVIDIA Cluster Readiness Engine", "NVSentinel", "Kubernetes", "Apache 2.0"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/configure-certify-and-operate-kubernetes-gpu-clusters-with-nvidia-dsx-os", "markdown": "https://wpnews.pro/news/configure-certify-and-operate-kubernetes-gpu-clusters-with-nvidia-dsx-os.md", "text": "https://wpnews.pro/news/configure-certify-and-operate-kubernetes-gpu-clusters-with-nvidia-dsx-os.txt", "jsonld": "https://wpnews.pro/news/configure-certify-and-operate-kubernetes-gpu-clusters-with-nvidia-dsx-os.jsonld"}}