# Manage Kubernetes Clusters for GPU Infrastructure with NVIDIA Kubernetes Engine

> Source: <https://developer.nvidia.com/blog/manage-kubernetes-clusters-for-gpu-infrastructure-with-nvidia-kubernetes-engine/>
> Published: 2026-11-10 15:00:00+00:00

NVIDIA is announcing two open source projects for building and operating managed Kubernetes services on GPU infrastructure—NVIDIA Kubernetes Engine (NVKE) and NVIDIA Kubernetes Recipes (NVKR). Both are new projects available in public preview.

AI Factory customers often want to consume GPU capacity through the Kubernetes APIs, tools, and ecosystem they already use, with the factory operating their clusters. Delivering this service requires API-driven provisioning, tenant isolation, upgrades, and recovery, backed by service commitments. It also requires qualifying GPU hardware, networking, drivers, and runtime software across GPU generations and Kubernetes versions.

NVKE and NVKR divide this work between cluster management and reproducible software builds. NVKE provides the software to run a managed Kubernetes service, installed on the factory’s infrastructure, with APIs to provision and manage clusters. NVKR builds the operating system images, container images, and packages those clusters need, using recipes that define validated combinations of software versions and GPU generations. AI Factories retain control of their software, operations, service, and customer experience. Teams can adopt NVKE and NVKR projects together or independently.

Both NVKE and NVKR are components of the [NVIDIA DSX AI Factory Platform](https://www.nvidia.com/en-us/data-center/products/dsx/), which helps AI factories design, build, and operate GPU infrastructure. Both projects are available on GitHub under the Apache 2.0 license, so teams can inspect the code, develop in parallel, and contribute improvements. During the public preview, APIs, configuration, and behavior may change between releases. NVKE and NVKR are not yet recommended for production workloads. See [preview status and known limitations](https://github.com/dsx-ai-factory/nvke/blob/main/docs/overview/preview-status.md).

This walkthrough covers building a node image with NVKR, deploying the NVKE management plane, and provisioning a customer Kubernetes cluster.

## **How NVKE and NVKR work together**

- **NVKR builds the software:** It produces node images, binaries, container images, and Helm charts from specific source versions and records the build inputs.
- **NVKE manages the clusters:** It uses those artifacts to provision and operate Kubernetes clusters.

Teams can use the projects together or separately. NVKE has no runtime dependency on NVKR, and NVKR can build artifacts for other Kubernetes management systems.

`nvkectl` starts a temporary seed cluster on a bootstrap host and uses it to form a management cluster. The NVKE management plane, including the REST API and controllers, runs on the management cluster. The management plane creates workload clusters for AI factory customers, each with its own Kubernetes control plane, etcd, and node pools. Machines come from [NVIDIA Infra Controller (NICo)](https://github.com/dsx-ai-factory/infra-controller), which manages bare-metal lifecycle, through [CAPNICo](https://github.com/dsx-ai-factory/cluster-api-provider-nico), the Cluster API infrastructure provider for NICo.

A node must present an attested SPIFFE identity before it receives join credentials. The NVKE telemetry pipeline, an OpenTelemetry-based Helm chart, collects logs, metrics, and traces and exports them over OTLP to a monitoring backend.

Inside each workload cluster, teams add the GPU-accelerated runtime with [AI Cluster Runtime (AICR)](https://developer.nvidia.com/blog/validate-kubernetes-for-gpu-infrastructure-with-layered-reproducible-recipes/), node packages with [NodeWright](https://developer.nvidia.com/blog/manage-kubernetes-node-fleets-with-nodewright/), and break-fix automation with [NVSentinel](https://developer.nvidia.com/blog/automate-kubernetes-ai-cluster-health-with-nvsentinel/).

NVKE is a [Certified Kubernetes AI Platform and Certified Kubernetes Installer](https://www.cncf.io/training/certification/software-conformance/), conformance tested on Kubernetes 1.36.

## **From bare metal to a customer cluster**

This walkthrough describes how to provision a customer Kubernetes cluster starting from servers installed in a rack by building a node image with NVKR, deploying the NVKE management plane with `nvkectl`, and creating a customer cluster through the NVKE API. The [NVKE documentation](https://docs.nvidia.com/nvke) lists every flag and option, along with the most up-to-date steps.

Walkthrough prerequisites:

- Three CPU servers for the management cluster, plus a separate Linux bootstrap host on the site network with containerd, nerdctl, kind, Helm, and kubectl
- Three CPU servers for the workload cluster control plane
- A pool of GPU servers for worker nodes
- A NICo installation with enrolled capacity
- A Linux build host with Docker and KVM support
- An HTTP server for the node image, a container registry for the controller image, and internet access to download the open source images and Helm charts used during installation

### **Build the node image**

NVKR defines the Kubernetes software configuration that NVIDIA validates for NVKE and provides the reproducible pipeline that turns that definition into deployable artifacts. NVKR builds node images from version-pinned sources and records every input, so teams can identify the software included in each image and can rebuild it later. The management cluster and each node pool can run different images, such as one per CPU architecture or GPU generation.

Teams clone the NVKR repository on the build host. They then run the quickstart script to build `nvkectl`, the NVKE controller image, and its Helm chart, and a second script to build the node image, named `node-image-01` in this walkthrough.

``` bash
$ git clone https://github.com/dsx-ai-factory/nvkr.git && cd nvkr
$ bash scripts/nvkr-quickstart.sh
$ NVKE_OS_ARCH=amd64 scripts/nvke-os-local.sh node-image-01
```

The node image build compiles the NVKE node agent and OVN-Kubernetes from source and adds checksum-pinned upstream releases of the Kubernetes 1.36 packages and the SPIRE agent. It then boots a virtual machine from the Ubuntu installer, installs the agents, and stages the Kubernetes and networking packages on the disk. The result is a QCOW2 disk image. Teams push the controller image to a registry, copy `nvkectl` to the bootstrap host, and put the QCOW2 on an HTTP server.

### **Deploy** **the NVKE management plane**

The NVKE management plane runs on a Kubernetes cluster called the management cluster. Teams run the following command on the bootstrap host to begin creating the management cluster:

``` bash
$ nvkectl bootstrap site -f site-a.yaml \
    --nvke-controller-image registry.example.com/nvke/nvke-controller:<tag>
```

The site specification file, site-a.yaml, includes the inputs that NICo needs to provision the three servers for the management cluster. The bootstrap command starts a temporary Kubernetes cluster on the bootstrap host. This temporary cluster uses kind to run a single Kubernetes node and hosts the NVKE software needed for the setup.

The NVKE controller asks NICo for three servers, which NICo boots from the previously built node image. When all three nodes have joined, the bootstrap process deletes the temporary cluster, leaving the management cluster. Teams then install the NVKE management plane on the management cluster and run nvkectl config set-site to connect the command-line tool to the NVKE API.

### **Publish a release set that customers will run**

Customers choose a release by name from those that the AI factory publishes. They do not work directly with operating system images or cloud-init configuration.

A release set has three components:

1. A machine image holds the iPXE script NICo runs to boot a server.
2. A cluster release pairs a machine image with the cloud-init configuration that prepares a control plane node and records the Kubernetes version that cloud-init installs.
3. A node pool release provides the corresponding image and configuration for worker nodes.

``` bash
$ nvkectl create machineimage -f machineimage.yaml
$ nvkectl create clusterrelease -f clusterrelease.yaml
$ nvkectl create nodepoolrelease -f nodepoolrelease.yaml
```

### **Create a customer cluster**

A customer cluster request includes the same NICo inputs as the site specification, plus a Kubernetes version. The control plane and each node pool have their own instance types, VPC prefix, and release. In this example, the control plane and GPU node pool reference releases named `k8s-1.36.3-amd64`.

```
name: customer-a
controlPlaneKubernetesVersion: 1.36.3
etcdEncrypted: true
controlPlaneVIP: 10.0.20.10
nico: {orgID: <org-id>, siteID: <site-id>, vpcID: <vpc-id>}
controlPlaneConfig:
  desiredCount: 3
  nico: {instanceTypePreference: [<cpu-instance-type>], vpcPrefixIDs: [<prefix-id>], release: k8s-1.36.3-amd64}
nodePools:
  - name: gpu
    nodeNamePrefix: gpu
    desiredNodeCount: 4
    nico: {instanceTypeID: <gpu-instance-type>, vpcPrefixIDs: [<prefix-id>], release: k8s-1.36.3-amd64}
```

Teams save the request as customer-a.yaml, then run the following command to submit it to the NVKE REST API through nvkectl.

``` bash
$ nvkectl create cluster -f customer-a.yaml
```

The cluster status remains `Provisioning` until the first control-plane node has run `kubeadm init` and NVKE has installed OVN-Kubernetes and NodeLocal DNSCache. The status then changes to Provisioned.

### **Give the customer access**

The customer accesses the cluster’s Kubernetes API through a kubeconfig, while the AI factory continues managing it through the NVKE API. To give the customer access, the team retrieves the cluster’s kubeconfig from the management cluster and shares it securely.

``` bash
$ nvkectl get kubeconfig customer-a --output-file customer-a.kubeconfig
kubeconfig written to customer-a.kubeconfig
```

## **AI factories and partners building with NVKE and NVKR**

NVIDIA Cloud Partners Firmus and SK Telecom are integrating NVKE and NVKR into their AI Factories to build and deploy Kubernetes clusters that host AI workloads.

Additionally, NVIDIA Partners Mirantis, Rafay, Spectro Cloud, OpenNebula Systems, and vCluster are integrating components of NVKE and NVKR into their managed Kubernetes offerings to help their AI factory customers run AI workloads on Kubernetes. Netris is enabling NVKE for its network automation solution.

## **Available today in public preview**

NVKE and NVKR are in public preview and are not yet recommended for production workloads. These projects are evolving rapidly, and the NVKE API may change between releases. The preview [status page](https://github.com/dsx-ai-factory/nvke/blob/main/docs/overview/preview-status.md) lists current limitations.

Teams can evaluate the projects against their infrastructure requirements and review planned work on the public [NVKE](https://github.com/dsx-ai-factory/nvke/blob/main/ROADMAP.md) and [NVKR](https://github.com/dsx-ai-factory/nvkr/blob/main/ROADMAP.md) roadmaps. Feedback and contributions can help shape NVKE, NVKR, and the rest of the DSX AI Factory stack.

## **Get started**

NVKE and NVKR are open source components available through the [NVIDIA DSX AI Factory GitHub organization](https://github.com/dsx-ai-factory), alongside CAPNICo. Start with the [NVKE documentation](https://docs.nvidia.com/nvke) or try the [local quickstart](https://github.com/dsx-ai-factory/nvke/blob/main/docs/get-started/quickstart-local.md) on a single machine—no GPU or NVIDIA account needed.

- Join the conversation in `#nvke` on the NVIDIA DSX AI Factory Slack workspace using[the Slack invitation link](https://inviter.co/dsx-ai-factory) . Share what you build with the community.
- Visit the [NVIDIA booth #170 at KubeCon](https://www.nvidia.com/en-us/events/kubecon-and-cloudnative-con-north-america/) for more Kubernetes deep dives and live demos.