In all Kubernetes platforms, idle GPU waste is one of the toughest capacity issues to resolve. You may have seen it yourself: GPUs allocated by other users apparently run for days, doing pretty much nothing. Kubernetes shows that the pods are still running, and the usage bill is still accumulating due to that allocation, but Data Center GPU Manager NVIDIA metrics reveal close to zero engine activity for hours. Many ML platforms ship idle cullers, but they typically watch UI or session activity or pod lifetime, not GPU engine utilization. GPU-pruner is an open source tool that bridges that gap, where it queries NVIDIA Data Center GPU Manager metrics through Prometheus, tracks the workloads with the unused GPU and scales the parent resource of the workload down to zero rather than deleting it.
This article explains what GPU-pruner is, how it works, where it fits in your stack, and what to be cautious about when you deploy it.
What is GPU-pruner? #
GPU-pruner is a safe idle culler for Kubernetes, querying Prometheus for real-time NVIDIA DCGM utilization metrics to detect pods that have remained inactive past a configured threshold, which defaults to 35 minutes.
I recently made several contributions to enable additional features. Before scale-down, the pruner can notify the user in a configured Slack channel (such as #test-pruner
), giving the user the opportunity to extend their allocation if the idle state is temporary. On scale-down, the pruner walks from the idle pod up to its parent resource and scales that parent to zero by bubbling up to the parent controller and setting the replica to zero. Thus, the parent resource, which holds the workload's configuration, is recreated by scaling up once again.
To evaluate workloads accurately, GPU-pruner queries Prometheus to scrape the NVIDIA Data Center GPU Manager (DCGM), which provides telemetry on per-GPU utilization, power consumption, and profiling activity. Instead of relying on session timeouts or disconnected browser tabs, GPU-pruner defines "idle" by monitoring whether peak hardware engine performance stays below the specified limit (1%) throughout the observation period. When a workload passes the observation window with little utilization, the pruner triggers a scale-down. Depending on the workload type, this mechanism adjusts the replica count to zero for objects such as Deployments, StatefulSets, or LeaderWorkerSets, effectively pausing Kubeflow Notebooks or reducing KServe minReplicas to 0, preserving the workload metadata while reclaiming scarce hardware resources.
Historical context #
GPU clusters became highly utilized due to the rise in machine learning platforms, but the capacity of GPUs has been a roadblock. Platform notebook cullers only detect utilization based on requests, requests per second, load balancer traffic, and browser activity. This works for forgotten browser tabs, but not for pods holding a GPU data or compiling ML models.
GPU-pruner was engineered for plain Kubernetes, where GPU resources are frequently requested but often remain allocated without active processing. This open source tool seamlessly integrates with various APIs—including Kubeflow Notebook and KServe InferenceService—as well as standard Kubernetes resources such as Deployment, ReplicaSet, and StatefulSet objects.
Key concepts and components #
GPUs are clearly scarce and expensive, and a single idle GPU workload can block another team's progress. To solve this issue without modifying any workflow or adding more frustration to developers, platform teams need an automated solution that relies on hardware truth rather than network traffic, as well as maintaining reversible changes like safely scaling workloads to zero rather than outright deleting them. Additionally, it is important to incorporate human-in-the-loop safeguards such as notifying the workload owners prior to scale down to give them a chance to preserve their resources when brief periods of hardware inactivity are intentional, such as during model or interactive debugging.
Metrics-driven
The system executes PromQL queries to scrape hardware telemetry from the ServiceMonitor, focusing on DCGM_FI_PROF_GR_ENGINE_ACTIVE
and DCGM_FI_DEV_GPU_UTIL
metrics. It monitors whether the highest recorded engine performance stays under the 0.01 limit across the observation window, which defaults to 35 minutes. If utilization remains below this floor throughout the period, the pruner identifies the workload as idle and triggers a scale-down for the parent resource.
Pods and Deployments
After identifying an idle pod, GPU-pruner determines the appropriate scale-down target by inspecting the pod's underlying metadata. Rather than acting directly on ephemeral pods, the controller walks up the Kubernetes ownerReferences hierarchy—or parses dedicated KServe labeling schemes—to trace the pod back to its root controller. This metadata traversal allows GPU-pruner to pinpoint the exact top-level resource responsible for managing the workload, whether it is a standard Kubernetes Deployment, a StatefulSet, or an enterprise ML custom resource like a Kubeflow Notebook or InferenceService.
Guardrails before scale-down
To prevent accidental disruptions, GPU-pruner enforces several strict safety guardrails before modifying any cluster resources. A pod will only be scaled down if it existed prior to the lookback window—preventing false positives on newly launched initialization jobs—and if the controller is actively executing in scale-down mode rather than dry-run. Furthermore, if Slack alerts are enabled, the system waits for the grace period to expire without a user acknowledgment before taking action.
Prometheus
To gather operational insights across the cluster, the system relies on Prometheus to track every running pod by pulling data from HTTP endpoints exposing the standard /metrics API. In this setup, Prometheus regularly scrapes raw GPU utilization and telemetry directly from the DCGM exporter pods running in the environment.
Service monitor
To direct Prometheus to these telemetry endpoints, the platform uses Custom Resources called ServiceMonitors. A ServiceMonitor acts as the declarative bridge between Prometheus and cluster workloads, specifying exact target services, ports, and HTTP path endpoints—such as port 9400 at /metrics
—that Prometheus must scrape to collect lower-level GPU data.
Data Center GPU Manager (DCGM) exporters generally operate using DaemonSet across GPU-enabled nodes.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: nvidia-dcgm-exporter
namespace: gpu-operator # or wherever DCGM exporter is deployed
labels:
app: nvidia-dcgm-exporter #important for prometheus to know which app to scrap
spec:
selector:
matchLabels:
app: nvidia-dcgm-exporter
endpoints: #scrap targets based on the endpoint services
- port: metrics # typically port 9400
path: /metrics
interval: 30s
scrapeTimeout: 10s
Prometheus scrape configuration with honor_label #
Because GPU-pruner relies on PromQL metrics to identify hardware inactivity, maintaining the accuracy of workload metadata in the Prometheus instance is paramount.
In a standard deployment, DCGM metrics are scraped from exporters running as a DaemonSet in an infrastructure namespace like gpu-operator
. By default, Prometheus is configured with honor_labels: false
, causing the system to overwrite any target labels that conflict with its own service discovery metadata.
For example, if a machine learning workload titled ml-training-job
is active in the ml-workloads
namespace, then raw telemetry from the DCGM daemonset might appear as follows:
DCGM_FI_DEV_GPU_TEMP{gpu="0", namespace="ml-workloads", pod="ml-training-job"} 68
Under the standard configuration (honor_labels: false
), Prometheus replaces the namespace and pod fields with its discovery targets—typically gpu-operator
and the exporter pod name—relocating the source identifiers to exported_namespace
and exported_pod
:
DCGM_FI_DEV_GPU_TEMP{gpu="0", namespace="gpu-operator", pod="dcgm-exporter-daemonset-abc12", exported_namespace="ml-workloads", exported_pod="ml-training-job"} 68
When your automation expects standard namespace labels to link metrics to user resources, a query for namespace="ml-workloads"
fails to return results.
To ensure metadata integrity, the ServiceMonitor targeting the DCGM exporter must have label preservation enabled:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: nvidia-dcgm-exporter
namespace: gpu-operator
labels:
app: nvidia-dcgm-exporter
spec:
selector:
matchLabels:
app: nvidia-dcgm-exporter
endpoints:
- port: metrics
path: /metrics
interval: 30s
scrapeTimeout: 10s
honorLabels: true
Enabling honorLabels: true
forces Prometheus to respect the original telemetry labels, ensuring GPU profiling data maps directly back to the relevant workload.
How it works #
The pruner initiates PromQL queries against Prometheus to scrape raw NVIDIA DCGM telemetry from cluster-wide exporter pods. To avoid disrupting initialization jobs, the controller filters for pods that are actively running—excluding those in a Pending state—and ensures they have existed longer than the observation window, which defaults to 35 minutes.
The controller then executes a find_root_object
routine, inspecting ownerReferences or specialized labeling schemes. This metadata traversal allows the pruner to trace an idle pod back to its manageable top-level resource, whether it is a standard Deployment, a StatefulSet, or an enterprise ML object like a Kubeflow Notebook, KServe InferenceService, or LeaderWorkerSet.
After pointing these targets, GPU-pruner refreshes its internal metrics (gpu_pruner_idle_gpus
) and pulls data from Prometheus to maintain a leaderboard of cumulative idle GPU hours over a rolling 7-day period.
Finally, the system dispatches an automated notification to the workload owner on the configured Slack channel. If the grace period expires without a user acknowledgment, the controller performs a reversible change by safely scaling the parent resource down to zero replicas (see figure 1).
Benefits and advantages #
GPU-pruner reclaims underutilized hardware without compromising cluster stability. By performing reversible scale-downs instead of permanent deletions, administrators can manage diverse workloads, spanning from vanilla Kubernetes controllers to specialized ML custom resources with any minimal friction. Safety is prioritized through a dry-run mode for logging purposes, while integrated Slack alerts and adjustable grace periods keep developers informed. This design incorporates essential human-in-the-loop safeguards to protect model initialization and active debugging sessions, all while providing comprehensive observability using Grafana telemetry.
Challenges and limitations #
Implementing hardware-aware pruning comes with trade-offs. Because GPU-pruner relies on PromQL queries evaluating DCGM metrics, long startup phases with zero initial GPU engine activity risk of mistaken scale-downs unless explicitly acknowledged by the user. Additionally, metric accuracy hinges on proper Prometheus setup, where misconfigured honor_labels can cause the controller to miss target workloads. Finally, platform teams must manually maintain secrets mapping namespaces to Slack IDs, and coverage for standalone pods or non-standard custom resource definitions remains an ongoing area of expansion.
Get started #
The installation process uses standard Kustomize manifests and provides flexible runtime parameters.
Step 1: Configure Prometheus endpoints and deployment flags
Begin by inspecting your target cluster to confirm whether your DCGM ServiceMonitor uses honorLabels
:
kubectl get servicemonitor -A -o json | jq -r '.items[] | select(.metadata.name | test("dcgm"; "i")) | "\(.metadata.namespace)/\(.metadata.name) honorLabels=\(.spec.endpoints[].honorLabels // false)"'
Next, edit GPU-pruner/hack/deployment.yaml
to supply your Prometheus endpoint and runtime mode. Start in --run-mode=dry-run
to observe telemetry without executing scaling actions:
args:
- 'GPU-pruner'
- '-d'
- '--run-mode=dry-run'
- '--prometheus-url=http://prometheus-Kubernetes.openshift-monitoring.svc:9090'
Important command-line parameters for customizing behavior include:
-t
: Sets the GPU inactivity observation window (defaults to 35m).-e
: Selects target resource types to prune using letter identifiers (d
for Deployments,r
for ReplicaSets,s
for StatefulSets,i
for KServe InferenceServices,n
for Kubeflow Notebooks, andl
for LeaderWorkerSets).--idle-threshold
: Sets the upper limit of GPU engine activity allowed before classifying a workload as idle.--slack-channel
and--slack-interaction-port
: Manages Slack webhook notifications and handles inbound user acknowledgments.
Step 2: Apply manifests and verify logs
Apply the Kustomize overlay to deploy the controller and its associated RBAC roles:
kubectl apply -k GPU-pruner/hack/
Verify that the pruner daemon is active by tailing its logs:
kubectl -n GPU-pruner-system logs -l app=GPU-pruner -f
To enable active scale-down after validating the dry-run behavior, update your deployment configuration or test locally against remote endpoints:
cargo run -p GPU-pruner -- \
--prometheus-url=http://localhost:9090 \
--run-mode=scale-down \
-d
Step 3: Set up telemetry and the dashboard
GPU-pruner includes an embedded web dashboard served over port 8080. Forward the dashboard port to review idle telemetry and target candidate workloads:
kubectl -n GPU-pruner-system port-forward svc/GPU-pruner-dashboard 8080:8080
For platform teams tracking cluster efficiency over time, you can also schedule a weekly summary report. First, create a Slack webhook secret:
Bash
kubectl -n GPU-pruner-system create secret generic GPU-pruner-slack-webhook \
--from-literal=webhook-url='https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
Then trigger a test execution of the reporting CronJob:
kubectl create job --from=cronjob/GPU-pruner-weekly-report GPU-pruner-weekly-report-manual -n GPU-pruner-system
kubectl logs -n GPU-pruner-system -l job-name=GPU-pruner-weekly-report-manual -f
Integrate with Red Hat Openshift #
In production, hardware is critical for platforms such as Red Hat OpenShift AI, which orchestrates data science stacks by combining interactive workbenches (built on Kubeflow Notebooks) with high-performance model-serving endpoints . While OpenShift AI includes UI-based idle notebook culling, that built-in mechanism only tracks active browser sessions or API requests hits. It cannot detect when a notebook sits completely idle while still reserving dedicated hardware.
Because GPU-pruner targets OpenShift AI's underlying custom resources—specifically Kubeflow Notebook objects and KServe InferenceService definitions—it provides true hardware-aware culling driven by cluster-level OpenShift Monitoring and Prometheus metrics. While our team has not yet deployed GPU-pruner into active production across our Red Hat OpenShift Container Platform cluster, the tool was specifically engineered for this integration path. Once deployed, dynamically scaling idle workbench StatefulSets and KServe serving pods down to zero allows platform engineers to dramatically increase tenant density, shorten scheduling queue times for active training jobs, and ensure that premium GPU accelerators are reserved for true compute workloads.
Summary #
By analyzing GPU hardware using metrics instead of browser telemetry, GPU-pruner addresses idle resources across all environments. It detects true inactivity and safely scales parent resources, like Kubeflow Notebooks or standard Kubernetes controllers, down to zero without deleting them, allowing developers to easily restore their setups. By only adjusting only the replica count, the Notebook or controller metadata is preserved, allowing developers to easily restore their setups by scaling the resource back up or utilizing the platform UI. With built-in guardrails, such as Slack alerts and validation windows, platform teams can reclaim capacity while ensuring crucial active workloads remain undisturbed.
Addressing inefficient GPU usage requires some monitoring and human-in-the-loop techniques. While hardware-aware culling aligns actual engine utilization with pod status, its effectiveness depends on reliable metrics and clear team communication around intentional idle times. By starting with dry-run tests and scaling up thoughtfully, the GPU-pruner provides platform administrators with an accurate capacity optimization solution for environments like vanilla Kubernetes making sure accelerators go to active workloads.
For more information on GPU workload management, read Explore OpenShift AI and GPU workload. management