Manage Kubernetes Clusters for GPU Infrastructure with NVIDIA Kubernetes Engine

NVIDIA announced two open source projects in public preview, NVIDIA Kubernetes Engine (NVKE) and NVIDIA Kubernetes Recipes (NVKR), for building and operating managed Kubernetes services on GPU infrastructure. Both projects are available on GitHub under the Apache 2.0 license as components of the NVIDIA DSX AI Factory Platform, and NVIDIA states they are not yet recommended for production workloads, with APIs, configuration and behavior subject to change between releases during the preview. NVKE supplies the management-plane software and APIs to provision workload clusters, while NVKR builds node images, container images and Helm charts from recipes defining validated software-version and GPU-generation combinations; NVKE has no runtime dependency on NVKR.

NVIDIA is announcing two open source projects for building and operating managed Kubernetes services on GPU infrastructure—NVIDIA Kubernetes Engine NVKE and NVIDIA Kubernetes Recipes NVKR . Both are new projects available in public preview. AI Factory customers often want to consume GPU capacity through the Kubernetes APIs, tools, and ecosystem they already use, with the factory operating their clusters. Delivering this service requires API-driven provisioning, tenant isolation, upgrades, and recovery, backed by service commitments. It also requires qualifying GPU hardware, networking, drivers, and runtime software across GPU generations and Kubernetes versions. NVKE and NVKR divide this work between cluster management and reproducible software builds. NVKE provides the software to run a managed Kubernetes service, installed on the factory’s infrastructure, with APIs to provision and manage clusters. NVKR builds the operating system images, container images, and packages those clusters need, using recipes that define validated combinations of software versions and GPU generations. AI Factories retain control of their software, operations, service, and customer experience. Teams can adopt NVKE and NVKR projects together or independently. Both NVKE and NVKR are components of the NVIDIA DSX AI Factory Platform https://www.nvidia.com/en-us/data-center/products/dsx/ , which helps AI factories design, build, and operate GPU infrastructure. Both projects are available on GitHub under the Apache 2.0 license, so teams can inspect the code, develop in parallel, and contribute improvements. During the public preview, APIs, configuration, and behavior may change between releases. NVKE and NVKR are not yet recommended for production workloads. See preview status and known limitations https://github.com/dsx-ai-factory/nvke/blob/main/docs/overview/preview-status.md . This walkthrough covers building a node image with NVKR, deploying the NVKE management plane, and provisioning a customer Kubernetes cluster. How NVKE and NVKR work together - NVKR builds the software: It produces node images, binaries, container images, and Helm charts from specific source versions and records the build inputs. - NVKE manages the clusters: It uses those artifacts to provision and operate Kubernetes clusters. Teams can use the projects together or separately. NVKE has no runtime dependency on NVKR, and NVKR can build artifacts for other Kubernetes management systems. nvkectl starts a temporary seed cluster on a bootstrap host and uses it to form a management cluster. The NVKE management plane, including the REST API and controllers, runs on the management cluster. The management plane creates workload clusters for AI factory customers, each with its own Kubernetes control plane, etcd, and node pools. Machines come from NVIDIA Infra Controller NICo https://github.com/dsx-ai-factory/infra-controller , which manages bare-metal lifecycle, through CAPNICo https://github.com/dsx-ai-factory/cluster-api-provider-nico , the Cluster API infrastructure provider for NICo. A node must present an attested SPIFFE identity before it receives join credentials. The NVKE telemetry pipeline, an OpenTelemetry-based Helm chart, collects logs, metrics, and traces and exports them over OTLP to a monitoring backend. Inside each workload cluster, teams add the GPU-accelerated runtime with AI Cluster Runtime AICR https://developer.nvidia.com/blog/validate-kubernetes-for-gpu-infrastructure-with-layered-reproducible-recipes/ , node packages with NodeWright https://developer.nvidia.com/blog/manage-kubernetes-node-fleets-with-nodewright/ , and break-fix automation with NVSentinel https://developer.nvidia.com/blog/automate-kubernetes-ai-cluster-health-with-nvsentinel/ . NVKE is a Certified Kubernetes AI Platform and Certified Kubernetes Installer https://www.cncf.io/training/certification/software-conformance/ , conformance tested on Kubernetes 1.36. From bare metal to a customer cluster This walkthrough describes how to provision a customer Kubernetes cluster starting from servers installed in a rack by building a node image with NVKR, deploying the NVKE management plane with nvkectl , and creating a customer cluster through the NVKE API. The NVKE documentation https://docs.nvidia.com/nvke lists every flag and option, along with the most up-to-date steps. Walkthrough prerequisites: - Three CPU servers for the management cluster, plus a separate Linux bootstrap host on the site network with containerd, nerdctl, kind, Helm, and kubectl - Three CPU servers for the workload cluster control plane - A pool of GPU servers for worker nodes - A NICo installation with enrolled capacity - A Linux build host with Docker and KVM support - An HTTP server for the node image, a container registry for the controller image, and internet access to download the open source images and Helm charts used during installation Build the node image NVKR defines the Kubernetes software configuration that NVIDIA validates for NVKE and provides the reproducible pipeline that turns that definition into deployable artifacts. NVKR builds node images from version-pinned sources and records every input, so teams can identify the software included in each image and can rebuild it later. The management cluster and each node pool can run different images, such as one per CPU architecture or GPU generation. Teams clone the NVKR repository on the build host. They then run the quickstart script to build nvkectl , the NVKE controller image, and its Helm chart, and a second script to build the node image, named node-image-01 in this walkthrough. bash $ git clone https://github.com/dsx-ai-factory/nvkr.git && cd nvkr $ bash scripts/nvkr-quickstart.sh $ NVKE OS ARCH=amd64 scripts/nvke-os-local.sh node-image-01 The node image build compiles the NVKE node agent and OVN-Kubernetes from source and adds checksum-pinned upstream releases of the Kubernetes 1.36 packages and the SPIRE agent. It then boots a virtual machine from the Ubuntu installer, installs the agents, and stages the Kubernetes and networking packages on the disk. The result is a QCOW2 disk image. Teams push the controller image to a registry, copy nvkectl to the bootstrap host, and put the QCOW2 on an HTTP server. Deploy the NVKE management plane The NVKE management plane runs on a Kubernetes cluster called the management cluster. Teams run the following command on the bootstrap host to begin creating the management cluster: bash $ nvkectl bootstrap site -f site-a.yaml \ --nvke-controller-image registry.example.com/nvke/nvke-controller:<tag The site specification file, site-a.yaml, includes the inputs that NICo needs to provision the three servers for the management cluster. The bootstrap command starts a temporary Kubernetes cluster on the bootstrap host. This temporary cluster uses kind to run a single Kubernetes node and hosts the NVKE software needed for the setup. The NVKE controller asks NICo for three servers, which NICo boots from the previously built node image. When all three nodes have joined, the bootstrap process deletes the temporary cluster, leaving the management cluster. Teams then install the NVKE management plane on the management cluster and run nvkectl config set-site to connect the command-line tool to the NVKE API. Publish a release set that customers will run Customers choose a release by name from those that the AI factory publishes. They do not work directly with operating system images or cloud-init configuration. A release set has three components: 1. A machine image holds the iPXE script NICo runs to boot a server. 2. A cluster release pairs a machine image with the cloud-init configuration that prepares a control plane node and records the Kubernetes version that cloud-init installs. 3. A node pool release provides the corresponding image and configuration for worker nodes. bash $ nvkectl create machineimage -f machineimage.yaml $ nvkectl create clusterrelease -f clusterrelease.yaml $ nvkectl create nodepoolrelease -f nodepoolrelease.yaml Create a customer cluster A customer cluster request includes the same NICo inputs as the site specification, plus a Kubernetes version. The control plane and each node pool have their own instance types, VPC prefix, and release. In this example, the control plane and GPU node pool reference releases named k8s-1.36.3-amd64 . name: customer-a controlPlaneKubernetesVersion: 1.36.3 etcdEncrypted: true controlPlaneVIP: 10.0.20.10 nico: {orgID: <org-id , siteID: <site-id , vpcID: <vpc-id } controlPlaneConfig: desiredCount: 3 nico: {instanceTypePreference: <cpu-instance-type , vpcPrefixIDs: <prefix-id , release: k8s-1.36.3-amd64} nodePools: - name: gpu nodeNamePrefix: gpu desiredNodeCount: 4 nico: {instanceTypeID: <gpu-instance-type , vpcPrefixIDs: <prefix-id , release: k8s-1.36.3-amd64} Teams save the request as customer-a.yaml, then run the following command to submit it to the NVKE REST API through nvkectl. bash $ nvkectl create cluster -f customer-a.yaml The cluster status remains Provisioning until the first control-plane node has run kubeadm init and NVKE has installed OVN-Kubernetes and NodeLocal DNSCache. The status then changes to Provisioned. Give the customer access The customer accesses the cluster’s Kubernetes API through a kubeconfig, while the AI factory continues managing it through the NVKE API. To give the customer access, the team retrieves the cluster’s kubeconfig from the management cluster and shares it securely. bash $ nvkectl get kubeconfig customer-a --output-file customer-a.kubeconfig kubeconfig written to customer-a.kubeconfig AI factories and partners building with NVKE and NVKR NVIDIA Cloud Partners Firmus and SK Telecom are integrating NVKE and NVKR into their AI Factories to build and deploy Kubernetes clusters that host AI workloads. Additionally, NVIDIA Partners Mirantis, Rafay, Spectro Cloud, OpenNebula Systems, and vCluster are integrating components of NVKE and NVKR into their managed Kubernetes offerings to help their AI factory customers run AI workloads on Kubernetes. Netris is enabling NVKE for its network automation solution. Available today in public preview NVKE and NVKR are in public preview and are not yet recommended for production workloads. These projects are evolving rapidly, and the NVKE API may change between releases. The preview status page https://github.com/dsx-ai-factory/nvke/blob/main/docs/overview/preview-status.md lists current limitations. Teams can evaluate the projects against their infrastructure requirements and review planned work on the public NVKE https://github.com/dsx-ai-factory/nvke/blob/main/ROADMAP.md and NVKR https://github.com/dsx-ai-factory/nvkr/blob/main/ROADMAP.md roadmaps. Feedback and contributions can help shape NVKE, NVKR, and the rest of the DSX AI Factory stack. Get started NVKE and NVKR are open source components available through the NVIDIA DSX AI Factory GitHub organization https://github.com/dsx-ai-factory , alongside CAPNICo. Start with the NVKE documentation https://docs.nvidia.com/nvke or try the local quickstart https://github.com/dsx-ai-factory/nvke/blob/main/docs/get-started/quickstart-local.md on a single machine—no GPU or NVIDIA account needed. - Join the conversation in nvke on the NVIDIA DSX AI Factory Slack workspace using the Slack invitation link https://inviter.co/dsx-ai-factory . Share what you build with the community. - Visit the NVIDIA booth 170 at KubeCon https://www.nvidia.com/en-us/events/kubecon-and-cloudnative-con-north-america/ for more Kubernetes deep dives and live demos.