GPU-pruner uses hardware telemetry to reclaim idle Kubernetes GPUs

Platform teams can see a Kubernetes pod holding a GPU, but that does not tell them whether the accelerator is doing useful work. A new Red Hat Developer walkthrough presents GPU-pruner as a hardware-aware alternative to idle cullers that rely on browser sessions, request traffic or pod age.

What the tool changes

GPU-pruner queries NVIDIA Data Center GPU Manager metrics through Prometheus. It watches DCGM_FI_PROF_GR_ENGINE_ACTIVE and DCGM_FI_DEV_GPU_UTIL, treating a workload as idle when peak engine activity stays below a configured threshold across an observation window that defaults to 35 minutes.

Rather than delete an idle pod, the controller follows Kubernetes ownerReferences or KServe-specific labels to find the top-level object that manages it. It can then scale a Deployment, StatefulSet, LeaderWorkerSet, Kubeflow Notebook or KServe InferenceService to zero. The parent resource and its configuration remain in place, so an operator or user can scale the workload back up later.

The walkthrough also describes guardrails. GPU-pruner can begin in dry-run mode, notify a Slack channel before acting, wait through a grace period and exclude workloads that are newer than the lookback window. Those controls matter because model loading, compilation or interactive debugging can temporarily produce low GPU activity without making a workload disposable.

Where OpenShift teams need to be careful

The design depends on trustworthy telemetry labels. The article warns that Prometheus may replace the original workload namespace and pod labels with labels for the DCGM exporter target when honor_labels is disabled. In that configuration, the controller may fail to map a GPU metric back to the user workload. The supplied ServiceMonitor example enables honorLabels: true to preserve that mapping.

There are further limits. Namespace-to-Slack-owner mappings require manual secret management, unusual custom resources may not be recognized, and zero engine activity during a long startup can look like waste. The author also says the team has not yet deployed GPU-pruner into active production on its Red Hat OpenShift Container Platform cluster, even though the tool targets OpenShift AI resources including Kubeflow Notebooks and KServe InferenceServices.

A cautious deployment path

The practical starting point is observation, not automatic reclamation. The walkthrough recommends checking the DCGM ServiceMonitor, deploying the controller with --run-mode=dry-run, and reviewing its logs and dashboard before enabling scale-down.

Platform teams should validate which resource types the controller can touch, test label preservation, and choose an inactivity window that covers their slowest legitimate initialization path. Slack warnings and acknowledgements add a human checkpoint, but they should complement—not replace—workload-specific exclusions and a measured rollout.

GPU-pruner is useful because it asks a narrower question than a conventional notebook culler: not whether a session looks abandoned, but whether the allocated accelerator has actually been working. For shared OpenShift AI clusters, that distinction can turn idle capacity into schedulable capacity without discarding the workload definition.