# GPU Cost Monitoring: Track Utilization and Attribute AI Spend > Source: > Published: 2026-07-23 12:11:31+00:00 ## Key takeaways - Average GPU utilization in Kubernetes sits at just 5%, according to the [Cast AI 2026 State of Kubernetes Optimization Report](https://cast.ai/reports/state-of-kubernetes-optimization/). AKS clusters average 2%, EKS 5%, and GKE 6%. - An idle H100 GPU on AWS costs approximately $8,850 per GPU per month at p5 on-demand pricing. A team of 20 engineers with idle GPU habits can burn over $177K per month without noticing. - DCGM (NVIDIA Data Center GPU Manager) is the standard metrics source for GPU utilization in Kubernetes. The dcgm-exporter DaemonSet exposes per-pod, per-namespace Prometheus metrics. - Two GPU cost attribution models exist: requests-based (charges for what a pod reserves) and usage-based (charges on actual utilization). Each serves a different purpose. - Idle GPU detection requires combining `DCGM_FI_DEV_GPU_UTIL` below 20% for 30 minutes with a large`DCGM_FI_DEV_FB_FREE` reading on an allocated pod. - Turning visibility into savings requires automating the response: right-sizing requests, scaling down idle nodes, and setting hard budget guardrails per team. ## Why GPU visibility comes first GPU hardware is the most expensive resource in your cluster. It is also the least utilized. CPU averages 8% utilization across Kubernetes workloads. Memory averages 20%. GPU sits at 5%. That gap is not an accident. Teams provision GPU nodes for peak demand, then forget to scale them down. Training jobs finish but pods remain running. Jupyter notebooks hold GPU allocations for hours with no active kernel. Additionally, `nvidia.com/gpu` resource requests behave differently from CPU: one pod claims the whole GPU, even if it uses 3% of compute and 10% of VRAM. Meanwhile, GPU costs keep rising. According to Cast AI’s GPU Price Report, H200 Capacity Block prices on AWS increased in early 2026, the first GPU price increase since EC2 launched. An idle H100 GPU costs approximately $8,850 per GPU per month on AWS p5 on-demand pricing. For a team of 20 with sloppy GPU habits, that number compounds fast, reaching over $177K per month in wasted spend. However, you cannot address a cost problem you cannot measure. Kubernetes does not expose GPU utilization natively. Standard CPU and memory metrics from the kubelet give you nothing on what the GPU is actually doing. Therefore, GPU cost monitoring requires a dedicated metrics pipeline, starting with DCGM. For broader context on how GPU optimization fits into overall Kubernetes cost management, see [Kubernetes GPU Optimization](https://cast.ai/blog/kubernetes-gpu-optimization/). ## What to track: utilization, memory, idle, DCGM DCGM stands for NVIDIA Data Center GPU Manager. It is the standard telemetry layer for datacenter GPUs. In Kubernetes, the `dcgm-exporter` runs as a DaemonSet, typically deployed as part of the NVIDIA GPU Operator. It connects to the kubelet pod-resources socket and appends pod and namespace labels to every metric it exposes. The result is a Prometheus-compatible `/metrics` endpoint with per-pod GPU telemetry. Retain DCGM metrics for at least 90 days in your time-series database. Shorter retention windows prevent month-over-month trend analysis and make it harder to validate showback reports before implementing chargeback. ### The 8 GPU metrics that matter Not every DCGM metric is equally useful for cost monitoring. The following eight cover compute utilization, memory pressure, power draw, thermal limits, and hardware health. | Metric name | Category | What it measures | Alert threshold | Cost relevance | |---|---|---|---|---| `DCGM_FI_DEV_GPU_UTIL` | Compute | GPU compute utilization, as a percentage | <20% sustained for 30+ min on allocated pod | Primary idle signal. Low value on an allocated pod means you are paying for unused capacity. | `DCGM_FI_DEV_MEM_COPY_UTIL` | Memory | Percentage of time the memory controller is active (memory bus activity indicator) | >90% sustained | Memory bandwidth bottleneck can stall compute and inflate job duration, increasing cost per result. | `DCGM_FI_DEV_FB_USED` | Memory | Framebuffer (VRAM) used, in MB | >90% of total FB | High VRAM usage with low compute signals memory reservation without active workload. | `DCGM_FI_DEV_FB_FREE` | Memory | Framebuffer (VRAM) free, in MB | Large FB_FREE on an allocated pod | Large free VRAM alongside low utilization confirms idle GPU. Key signal for chargeback disputes. | `DCGM_FI_DEV_POWER_USAGE` | Power | Current power draw, in watts | <50W on allocated pod (idle power floor) | Low wattage on an allocated GPU confirms idle state and correlates to energy cost waste. | `DCGM_FI_DEV_GPU_TEMP` | Thermal | GPU die temperature, in Celsius | >83°C for H100, >80°C for A100 | Thermal throttling reduces throughput, increasing time-to-result and cost per training step. | `DCGM_FI_DEV_ECC_DBE_VOL` | Health | Uncorrectable (double-bit) ECC errors | Any non-zero value | Uncorrectable errors cause job crashes and wasted compute time on aborted training runs. | `DCGM_FI_DEV_XID_ERRORS` | Health | Driver XID error codes | XID 79 (GPU off bus), any XID spike | XID 79 means the GPU has fallen off the PCI bus. Jobs silently fail and you pay for a dead node. | *Throttle temperatures vary slightly by SKU and form factor (SXM5 vs PCIe). Verify against NVIDIA’s official thermal specifications for your specific GPU model.* ### Detecting idle GPUs with PromQL Idle GPU detection works by combining a low compute utilization signal with evidence that a pod still holds the allocation. The following PromQL query identifies GPUs running below 20% utilization over a 30-minute window: ``` # Flag GPUs with under 20% average utilization over the last 30 minutes avg_over_time(DCGM_FI_DEV_GPU_UTIL[30m]) < 20 ``` This query returns GPU instances by device. However, it does not yet tell you which pod or team owns the idle GPU. For that, you need label-aware queries scoped to the pod and namespace dimensions that dcgm-exporter appends automatically. Per-pod GPU utilization over the last 30 minutes: ``` # GPU utilization per pod, averaged over 30 minutes avg by (pod, namespace, gpu) ( avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[30m]) ) ``` Per-namespace GPU utilization, useful for team-level dashboards: ``` # Average GPU utilization by namespace over the last hour avg by (namespace) ( avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[1h]) ) ``` To confirm idle state, add the framebuffer check. An allocated pod with low utilization and high free VRAM is definitively idle: ``` # Idle GPU: low utilization AND large free VRAM on an allocated pod (avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[30m]) < 20) and (DCGM_FI_DEV_FB_FREE{pod!=""} > 40000) ``` ## Cost attribution: per workload, team, app GPU metrics visibility is only half the job. The other half is answering: who owns this cost? Attribution turns raw utilization numbers into accountability. ### The attribution hierarchy Namespace is the first attribution boundary in Kubernetes. Each team or application typically lives in its own namespace, so namespace-level GPU spend gives you a fast first cut. For more detail, labels extend attribution further. Common label keys include `team` , `app` , `project` , and `environment` . For a deeper look at how cost allocation works across all resource types in Kubernetes, see [Kubernetes Cost Allocation](https://cast.ai/blog/kubernetes-cost-allocation/). ### Two attribution models: requests-based vs. usage-based Two chargeback models suit different goals. First, the requests-based model charges each team for every `nvidia.com/gpu` resource request in their pods, regardless of actual utilization. This model creates strong accountability: teams pay for what they reserve, not what they use. Therefore, it incentivizes right-sizing requests and releasing GPUs when not needed. Second, the usage-based model charges on actual GPU utilization. This model is better for COGS calculations and for understanding the real cost of an AI feature or model training run. However, it requires more instrumentation and is harder to explain to teams unfamiliar with GPU metrics. Most engineering organizations start with requests-based attribution for team accountability, then add usage-based tracking for product COGS. Use both together for a complete picture. ### Showback before chargeback Start with showback: report GPU costs back to teams without billing them. Run showback for 60-90 days to validate your attribution model before implementing financial accountability. Teams need time to see the data, fix mislabeled workloads, and clean up orphaned GPU reservations. Before activating billing accountability, verify that more than 95% of GPU pod hours carry a team label. Run this check: ``` # Percentage of GPU pod-hours with team label ( count(kube_pod_labels{label_team!=""} * on(pod,namespace) group_left() (DCGM_FI_DEV_GPU_UTIL > 0)) / count(DCGM_FI_DEV_GPU_UTIL > 0) ) * 100 ``` If coverage is below 95%, fix the unlabeled pods before charging teams. An unlabeled GPU pod creates attribution debt that compounds over time. After showback is validated, choose your billing rate model: **Unblended rates**: charge teams the actual hourly rate paid for each GPU. Best for accuracy.** Amortized rates**: spread Reserved Instance or Savings Plan commitments evenly. Best for team budget predictability. For most teams, amortized rates produce fairer internal billing. Finance owns the decision. Loop them in before going live. ### Commitment timing Optimize first, then commit. If you purchase Reserved Instances or Savings Plans against unoptimized GPU baselines, you will over-commit. Run your idle detection pipeline for at least one full 30-day cycle. Before committing, confirm that wasted GPU hours fall below 15% of total GPU hours in the trailing 30-day window. If idle waste is still above 15%, continue optimizing before purchasing. Committing above that threshold locks in overprovision at reserved pricing. ### The chargeback formula For usage-based GPU chargeback, the formula is straightforward: ``` team_gpu_cost = (team_gpu_util / total_gpu_util) × gpu_hour_cost ``` Before running the chargeback formula, verify two things. First, dcgm-exporter must be version 3.x or later, which appends pod and namespace labels to DCGM metrics automatically via the kubelet pod-resources socket. Second, kube-state-metrics must expose your team labels. By default, kube-state-metrics allowlists only a subset of pod labels. Add your label to the kube-state-metrics `--metric-labels-allowlist` flag: ``` --metric-labels-allowlist=pods=[team,app,project,environment] ``` Without this, `kube_pod_labels` will not include your team label and the chargeback query will silently return no results. In PromQL, a per-team chargeback rate looks like this: ``` # Proportional GPU cost share by team label sum by (team) (avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[1h]) * on(pod) group_left(team) kube_pod_labels) / sum(avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod!=""}[1h])) * gpu_hour_cost ``` Replace `gpu_hour_cost` with the actual on-demand or reserved instance price for your GPU node type. For an AWS p5.48xlarge (8x H100), divide the per-node hourly rate by 8 to get the per-GPU cost. ## Turning visibility into savings Dashboards and alerts are inputs, not outcomes. Visibility only produces savings when it triggers action. The following steps move you from monitoring to reclaimed spend. ### Step 1: Set idle GPU alerts Configure Prometheus alerting rules to fire when the idle GPU PromQL condition holds for 30 minutes. Route alerts to the owning team’s Slack channel using the namespace or team label. This creates direct feedback: the team that owns the idle pod receives the alert, not a central platform team. ### Step 2: Right-size GPU requests Most GPU pods request whole GPU units when fractional allocation would suffice for inference workloads. Review `DCGM_FI_DEV_FB_USED` alongside `DCGM_FI_DEV_GPU_UTIL` for each workload. For example, an inference pod using 8 GB of a 80 GB VRAM GPU and 12% compute is a candidate for multi-instance GPU (MIG) or time-slicing. Additionally, setting proper resource limits prevents pods from holding GPU allocations after their primary task completes. ### Step 3: Automate node scale-down Node-level idle detection closes the loop. When all pods on a GPU node show low utilization and the node has been in that state for a defined window, automated scale-down removes the node and stops billing. However, this requires the cluster autoscaler or an intelligent autoscaling layer to understand GPU node costs and act on utilization signals, not just pod count. Cast AI connects GPU utilization data directly to autonomous scaling decisions. It reads DCGM metrics, identifies idle GPU nodes, and removes them without manual intervention. For clusters where GPU spend is the dominant cost driver, this closes the gap between what you monitor and what you actually save. See [Cast AI GPU Optimization](https://cast.ai/gpu-optimization/) to understand how automated GPU rightsizing works in practice. ### Step 4: Set per-team GPU budgets Attribution without limits creates reporting without accountability. Set namespace-level GPU resource quotas to cap the maximum number of GPUs a team can hold simultaneously. Then, pair quotas with showback reports: monthly summaries of GPU spend by team, broken down by workload. Teams that see their GPU costs tend to reduce them. To enforce GPU budgets at the namespace level, apply a ResourceQuota: ``` apiVersion: v1 kind: ResourceQuota metadata: name: gpu-quota namespace: team-ml spec: hard: requests.nvidia.com/gpu: "8" limits.nvidia.com/gpu: "8" ``` This prevents the team-ml namespace from requesting more than 8 GPUs regardless of what the DCGM data shows. Pair quotas with your showback reports: when a team consistently uses 60% of their quota, you have data to justify a quota increase or a consolidation conversation. ## Conclusion GPU cost monitoring in Kubernetes requires three things working together: a metrics pipeline (DCGM), attribution logic (namespace and labels), and automated action on idle signals. Without all three, you have dashboards but not control. At 5% average GPU utilization across Kubernetes clusters, the math is straightforward. Most clusters carry substantial idle GPU cost. Therefore, the first engineering investment is visibility: deploy dcgm-exporter, build the idle detection queries, and connect utilization data to team-level attribution. From there, the path to savings is direct: alert on idle, right-size requests, automate scale-down, and set team budgets. For deeper coverage of the optimization layer that sits above monitoring, see [Kubernetes GPU Optimization](https://cast.ai/blog/kubernetes-gpu-optimization/). ## Frequently Asked Questions **How do you monitor GPU cost in Kubernetes?** Deploy the NVIDIA GPU Operator to run dcgm-exporter as a DaemonSet on each GPU node. The exporter connects to the kubelet pod-resources socket and publishes per-pod GPU metrics to a Prometheus endpoint. From there, build dashboards on `DCGM_FI_DEV_GPU_UTIL` and `DCGM_FI_DEV_FB_USED` scoped by namespace, pod, and team label. Combine utilization data with node pricing to calculate GPU spend per workload. Cast AI ingests these signals automatically and surfaces idle GPU cost without manual dashboard work. **What GPU metrics matter most?** For cost monitoring, the four highest-priority metrics are: `DCGM_FI_DEV_GPU_UTIL` (compute utilization, the primary idle signal), `DCGM_FI_DEV_FB_USED` and `DCGM_FI_DEV_FB_FREE` (VRAM consumption and free capacity), and `DCGM_FI_DEV_POWER_USAGE` (power draw as an idle confirmation). Additionally, `DCGM_FI_DEV_GPU_TEMP` catches thermal throttling that inflates job duration, and `DCGM_FI_DEV_XID_ERRORS` catches hardware failures that cause silent job crashes and wasted GPU time. **How do you attribute GPU cost to teams?** Start with namespace as the first attribution boundary. Each team’s pods run in a dedicated namespace, so namespace-level GPU spend gives an immediate team breakdown. For finer attribution, add pod labels such as `team` , `app` , and `project` . Two chargeback models apply: requests-based (charges for every `nvidia.com/gpu` request regardless of actual use) and usage-based (charges proportional to measured utilization using the formula: team GPU utilization divided by total GPU utilization, multiplied by GPU hour cost). Most organizations use requests-based for team accountability and usage-based for product COGS reporting. **How do you find idle GPUs in Kubernetes?** Query `avg_over_time(DCGM_FI_DEV_GPU_UTIL[30m]) < 20` in Prometheus to identify GPUs averaging below 20% compute utilization over 30 minutes. Then, filter to pods that still hold a GPU allocation (pod label is non-empty) and confirm with `DCGM_FI_DEV_FB_FREE` showing large free VRAM. The combination of low compute utilization and high free memory on an allocated pod is a definitive idle signal. Set a Prometheus alert on this condition and route it to the owning team’s namespace label for direct feedback.