{"slug": "validating-gpu-platforms-with-runtime-ai-inventory-via-the-nvidia-dsx-ai-factory", "title": "Validating GPU Platforms with Runtime AI Inventory via the NVIDIA DSX AI Factory", "summary": "NVIDIA's open source AI Cluster Runtime (AICR), a modular component of the NVIDIA DSX AI Factory Platform, now includes a qualified AI bill of materials (AIBOM) component built on Google's k8s-aibom project to inventory AI workloads running on Kubernetes clusters. Two stock recipes, h100-gke-cos-inference and h100-gke-cos-inference-dynamo, select k8s-aibom by default, while other GKE recipes can opt in during generation with --runtime-inventory; k8s-aibom emits a CycloneDX 1.6 AIBOM per recognized AI workload, covering native Dynamo graphs, NIMService, LeaderWorkerSet, DisaggregatedSet, and CronJob resources. The addition addresses the gap where mutable image tags like :latest let two nodes run different code under identical tags, giving platform teams workload-level traceability on top of AICR's version-locked, checksummed deployment bundles.", "body_md": "As teams deploy and scale AI workloads using [NVIDIA Dynamo](https://developer.nvidia.com/dynamo), Kubeflow, or [Slurm](https://developer.nvidia.com/slurm/), tracking what actually runs requires visibility beyond the platform configuration captured in a recipe. Application teams define and update these workloads independently of the platform recipe, so a reproducible deployment bundle alone cannot establish which workload artifacts are running on each node. Mutable image tags like `:latest` add another layer of complexity. Two nodes can end up running different code under identical tags, and a node that pulls next week may retrieve an image that was not covered by an earlier security review.\n\n[NVIDIA AI Cluster Runtime (AICR)](https://docs.nvidia.com/aicr/latest/overview/introduction), an open source modular component of the [NVIDIA DSX AI Factory Platform](https://docs.nvidia.com/dsx), now includes a qualified AI bill of materials (AIBOM) component to close that gap. An open source project created by Google, [k8s-aibom](https://github.com/GoogleCloudPlatform/k8s-aibom) tracks AI workloads running on Kubernetes. Two stock recipes select k8s-aibom by default: `h100-gke-cos-inference` and `h100-gke-cos-inference-dynamo`. Other GKE recipes can opt in during generation with `--runtime-inventory` enabled. Recipes selecting it can opt out with `--runtime-inventory` disabled.\n\nAICR turns known-good GPU platform configurations into version-locked recipes and reproducible deployment bundles. Each recipe records the desired configuration for one hardware, cloud, OS, and intent combination: its components, version pins, values, constraints, deployment ordering, and validation expectations. AICR generates reproducible deployment artifacts from fixed recipe, catalog, and bundle inputs. Bundles include checksums; signing is optional through `aicr bundle --attest`.\n\nQualification with k8s-aibom is a specific claim, not a rubber stamp. AICR binds the exact upstream source, chart, CRDs, and digest-pinned controller image as one tested set, verifies provenance against Google’s expected build identity, reviews rendered RBAC and security defaults, and exercises installation, upgrade, health, and uninstall on the deployment paths it tested.\n\nThis blog will enable you to configure your Kubernetes runtime using AICR to provide a validated, performant configuration for your cluster.\n\n## Two ends of the same problem\n\nAICR and k8s-aibom look at different parts of the AI infrastructure lifecycle. AICR helps platform teams generate and validate known-good configurations for GPU-accelerated Kubernetes. It works at the runtime deployment layer: drivers, operators, kernels, Kubernetes components, deployment configuration.\n\nk8s-aibom looks higher up the stack and later in the lifecycle of the cluster. It watches Kubernetes workload and pod APIs and emits a CycloneDX 1.6 AIBOM for each recognized AI workload. Coverage is bounded: Only supported workload kinds carrying a recognized AI signal are inventoried. Coverage includes native Dynamo graphs, NIMService, LeaderWorkerSet, DisaggregatedSet, and CronJob resources. Tracked child workloads are combined into their owner’s AIBOM.\n\n| **AICR answers** | **k8s-aibom answers** | \n|---|---|\n| Is the GPU infrastructure built from a known and validated configuration? | What AI assets are actually running on that infrastructure? | \n\n*Table 1. The value at each level of the stack.*\n\nOne provides control and reproducibility of the configuration for GPU-accelerated Kubernetes, while the other adds workload discovery and traceability in that cluster. Concretely, AIBOM helps you:\n\n- Find unregistered or shadow AI workloads in namespaces selected for reporting\n- Identify models, inference runtimes, agents, and RAG components\n- Track external AI APIs and vector databases\n- Detect changes or drift in AI dependencies\n- Support security reviews, incident response, and compliance audits\n\nBoth projects arrived at this from opposite sides of the same problem, and [recognized it as such](https://github.com/GoogleCloudPlatform/k8s-aibom/issues/8). AICR records qualified deployment intent and evaluates it against a cluster at a point in time. k8s-aibom observes supported AI workloads and records what it found, including how confidently each attribute was determined. Neither view is sufficient for AI governance on its own, and the boundary between them is exactly where accountability is missing today.\n\n## Using AICR with k8s-aibom\n\nThese examples reflect [AICR v1.2.0](https://github.com/NVIDIA/aicr/tree/b0a00f020b6e8e21c51e8d5d5baad5ca07eeabb6) and its catalog, which pins k8s-aibom 1.6.2. Use a CLI build matching this catalog to reproduce the examples.\n\nStart with a Google Kubernetes Engine (GKE) cluster running Kubernetes 1.32 or later, with NVIDIA H100 GPU nodes using Container-Optimized OS and GKE-managed drivers and device plugins. The following workflow generates a recipe, renders it into a bundle, deploys it, and validates the result. Nothing is hand-assembled and nothing is tuned by hand.\n\nWith the same AICR version, catalog, and generation inputs, recipe resolution produces the same configuration. The last step checks the cluster against it. That is the half AICR brings to this pairing: a validated GPU platform quickly and repeatably. k8s-aibom then keeps inventorying supported AI workloads.\n\n### Generate the recipe\n\n```\naicr recipe \\\n  --service gke \\\n  --accelerator h100 \\\n  --os cos \\\n  --intent inference \\\n  --output recipe.yaml\n```\n\nThe new k8s-aibom component is selected as part of the recipe, not bolted on afterward. You can query any resolved value without generating a file:\n\n```\naicr query \\\n  --service gke --accelerator h100 --os cos --intent inference \\\n  --selector components.k8s-aibom.version\n# 1.6.2\n```\n\n### Bundle and deploy\n\nCreate a deployment bundle with the default Helm deployer. Replace the example node selectors with labels present on your system and GPU node pools:\n\n```\naicr bundle \\\n  --recipe recipe.yaml \\\n  --system-node-selector nodeGroup=system-pool \\\n  --accelerated-node-selector nodeGroup=gpu-worker \\\n  --output ./bundles\n```\n\nThe bundle is one directory per component, ordered by the dependency graph, each carrying its values, checksums, and install script. AICR supports a number of deployers via `--deployer` flag, including `helmfile`, `flux`, and `argocd`.\n\nVerify the generated bundle, then install it:\n\n```\n(cd ./bundles && aicr verify . && chmod +x deploy.sh && ./deploy.sh)\n```\n\nOnce deployed, the k8s-aibom controller runs in the `k8s-aibom-system` namespace. AICR uses the main chart’s CRDs rather than the companion `k8s-aibom-crds` chart. Helm bundles include a CRD apply step; Flux and Argo CD handle updates through their controllers. Helmfile, Terraform, and Fleet require the [documented manual CRD update](https://github.com/NVIDIA/aicr/blob/b0a00f020b6e8e21c51e8d5d5baad5ca07eeabb6/docs/user/component-catalog.md#upgrade-uninstall-and-troubleshooting) when the transition calls for it. Check the installed storage version:\n\n```\nkubectl get crd aiboms.aibom.k8saibom.dev \\\n  -o jsonpath='{.spec.versions[?(@.storage==true)].name}'\n# v1beta1\n```\n\n### Validate\n\n```\naicr snapshot --output snapshot.yaml\naicr validate --recipe recipe.yaml --snapshot snapshot.yaml \\\n  --phase deployment --phase conformance\n```\n\nThe `k8s-aibo` m health check runs as part of the deployment phase. It requires the controller deployment to be available, the cluster-scoped `AIBOMControllerConfig` to report a current `Ready=True`, and both CRDs to report the storage version the pinned chart expects. A matching storage version does not prove the full CRD schema is current. During upgrades, also verify the target schema and check for `Degraded=True` with reason `SchemaPredatesController`; see [health and readiness](https://github.com/NVIDIA/aicr/blob/b0a00f020b6e8e21c51e8d5d5baad5ca07eeabb6/docs/user/component-catalog.md#health-and-readiness).\n\n### Opt a namespace in\n\nInventory is opt-in per namespace. The label controls which namespaces emit AIBOM records; it does not narrow what the controller reads. Informers still cache workload and pod specs cluster-wide, including container arguments and inline environment values. No sink and no secret access are configured by default, so nothing leaves the cluster without an explicit decision.\n\n```\nkubectl create namespace prod-jobs\nkubectl label namespace prod-jobs aibom.k8saibom.dev/enabled=true\n```\n\n### Run an inference workload\n\nSupported workloads carrying a recognized AI signal are inventoried automatically when their namespace opts in. For a simple demonstration, here is a vLLM deployment serving Qwen. Note that it pulls `vllm/vllm-openai:latest` rather than a digest, which is not an uncommon pattern in some environments. The `nvidia-h100-mega-80gb` selector and toleration must match your GPU pool’s labels and taints:\n\n```\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: vllm-qwen\n  namespace: prod-jobs\nspec:\n  replicas: 1\n  selector:\n    matchLabels: {app: vllm-qwen}\n  template:\n    metadata:\n      labels: {app: vllm-qwen}\n    spec:\n      # Adjust to match your GPU node pool's taints and labels.\n      tolerations:\n        - key: dedicated\n          operator: Equal\n          value: gpu-workload\n          effect: NoSchedule\n      nodeSelector:\n        cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb\n      containers:\n        - name: vllm\n          image: vllm/vllm-openai:latest\n          args: [\"--model\", \"Qwen/Qwen2.5-0.5B-Instruct\",\n                 \"--max-model-len\", \"4096\",\n                 \"--gpu-memory-utilization\", \"0.30\"]\n          ports:\n            - containerPort: 8000\n          resources:\n            limits:\n              nvidia.com/gpu: \"1\"\n          volumeMounts:\n            - {name: shm, mountPath: /dev/shm}\n      volumes:\n        - name: shm\n          emptyDir: {medium: Memory, sizeLimit: 2Gi}\n```\n\nSave this manifest as `vllm-qwen.yaml`, then create the workload:\n\n```\nkubectl apply -f vllm-qwen.yaml\n```\n\n### See the inventory\n\nAfter applying the workload, you can track the inventory:\n\n```\nkubectl -n prod-jobs get aiboms.aibom.k8saibom.dev\n```\n\nExample output:\n\n```\nNAME                        WORKLOAD     NAME        CATEGORY    RUNTIME   READY   AGE\napps-deployment-vllm-qwen   Deployment   vllm-qwen   inference   vllm      True    28m\n```\n\nThe controller automatically classified the workload category and serving runtime. One boundary worth knowing: The record is built from the workload spec, so an AIBOM can exist before any pod is running. AIBOM can appear and report `Ready=True` while the pod is still pulling its image. An AIBOM tells you what was declared and, once a Pod exists, what it resolved to. It does not prove successful scheduling or execution.\n\nCheck the status conditions:\n\n```\nkubectl -n prod-jobs get aibom apps-deployment-vllm-qwen \\\n  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {end}'\n# Ready=True Synced=True SinkFailed=False Stale=False\n```\n\n### Verify the AIBOM\n\nThe document is stored inline on the resource when it is small enough, with a published SHA-256 over the canonical bytes:\n\n```\nkubectl -n prod-jobs get aibom apps-deployment-vllm-qwen -o json > aibom.json\njq -r '.status.bomDocument.inline.data' aibom.json | base64 -d > bom.json\n\njq -r '.status.bomDocument.sha256' aibom.json\nshasum -a 256 bom.json | awk '{print $1}'\n```\n\nHere’s where it all comes together:\n\n```\njq -r '.components[] | \"\\(.type)\\t\\(.name)\"' bom.json\n```\n\nThe response should look something like this:\n\n```\napplication              vllm\ncontainer                vllm/vllm-openai\nmachine-learning-model   Qwen/Qwen2.5-0.5B-Instruct\n```\n\nA conventional SBOM tells you which image is running. An AIBOM tells you which model is serving and which runtime is serving it. Attributes carry one of four confidence tiers—`declared`, `inferred`, `unresolved`, or `verified`—along with an evidence locator, so reviewers can see how each claim was reached.\n\n## Resource impact\n\nUpstream [measured k8s-aibom v1.6.1 on GKE](https://github.com/GoogleCloudPlatform/k8s-aibom/blob/d2b5b683d58ee9e71cc95bd4d7272676c4b54d11/docs/evidence/v1.6.1-gke-adr019-dynamo.md) with NVIDIA Dynamo and AICR’s configuration. The test used CPU-only nodes with workload pods remaining pending, so these figures describe controller inventory overhead:\n\n- Approximately four API-server requests per minute during a 15-min steady-state observation window\n- Average controller CPU usage of 2.2 millicores\n- Controller memory working set of 28 to 34 MiB\n- One AIBOM regeneration when the tested workload’s image tag changed\n\n## What comes next\n\nThe integration deliberately stops short of a few things, each its own decision rather than an oversight.\n\n**Signed runtime observations:** The runtime AIBOM is written as a Kubernetes resource, but the observation itself is unsigned. Signing would make captured observations tamper-evident; it would not prove every workload declaration true or establish successful execution. The canonical bytes and published SHA-256 shown above make signing tractable.\n\n**Connecting verified provenance:** k8s-aibom can verify Sigstore signatures on model claims against operator-configured trust roots, enforce signer-identity constraints, check Rekor inclusion, and assign qualifying claims the verified confidence tier. AICR leaves this verification disabled by default. AICR can also produce transparency-logged bundle attestations when signing is enabled; connecting that platform provenance to runtime AI inventory remains future integration work.\n\n**Intent and observed workloads:** An AICR recipe describes deployment intent. A runtime AIBOM records observed workload specifications and available pod-status evidence. Correlating them requires shared artifact identities, freshness rules, and mapping between infrastructure and workload coverage. That comparison remains future integration work; an AIBOM does not establish successful execution or complete cluster conformance.\n\nAICR is available at [NVIDIA/aicr](https://github.com/nvidia/aicr). k8s-aibom is available at [GoogleCloudPlatform/k8s-aibom](https://github.com/GoogleCloudPlatform/k8s-aibom). Both are open source. If you run GPU workloads on Kubernetes, this integration adds an inventory of recognized AI workloads and evidence showing how each attribute was determined.", "url": "https://wpnews.pro/news/validating-gpu-platforms-with-runtime-ai-inventory-via-the-nvidia-dsx-ai-factory", "canonical_source": "https://developer.nvidia.com/blog/validating-gpu-platforms-with-runtime-ai-inventory-via-the-nvidia-dsx-ai-factory/", "published_at": "2026-11-11 17:00:00+00:00", "updated_at": "2026-11-11 17:25:14.157430+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "mlops", "developer-tools", "artificial-intelligence"], "entities": ["NVIDIA", "NVIDIA AI Cluster Runtime (AICR)", "NVIDIA DSX AI Factory Platform", "k8s-aibom", "Google", "Kubernetes", "NVIDIA Dynamo", "Kubeflow"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/validating-gpu-platforms-with-runtime-ai-inventory-via-the-nvidia-dsx-ai-factory", "markdown": "https://wpnews.pro/news/validating-gpu-platforms-with-runtime-ai-inventory-via-the-nvidia-dsx-ai-factory.md", "text": "https://wpnews.pro/news/validating-gpu-platforms-with-runtime-ai-inventory-via-the-nvidia-dsx-ai-factory.txt", "jsonld": "https://wpnews.pro/news/validating-gpu-platforms-with-runtime-ai-inventory-via-the-nvidia-dsx-ai-factory.jsonld"}}