Kubernetes 1.37 Gang Scheduling Beta: Fix GPU Training Deadlocks

Kubernetes 1.37 “Garhwal,” released August 26, 2026, includes gang scheduling in Beta, which prevents partial-job deadlocks that can leave idle GPUs costing around $467 per event. The feature, detailed in a September 8 Kubernetes team blog post, introduces the PodGroup API at scheduling.k8s.io/v1beta1 with a minCount field to ensure all pods in a group are placed simultaneously. Workload-aware preemption also graduated to Beta, targeting complete PodGroups to avoid evicting partial training runs. Gang scheduling is disabled by default and requires enabling the GenericWorkload feature gate.

Kubernetes 1.37 “Garhwal” dropped August 26. Most writeups are leading with scale-to-zero HPA, and yes, that feature is genuinely useful. But if your team runs distributed AI training on Kubernetes, the release that matters is the one on September 3: gang scheduling graduated to Beta, and the workload-aware scheduling blog https://kubernetes.io/blog/2026/09/08/kubernetes-v1-37-advancing-workload-aware-scheduling/ from the Kubernetes team spells out what just changed. This is the fix for one of the most expensive scheduling bugs in production ML infrastructure — the partial-job deadlock that leaves hundreds of dollars in idle GPUs doing nothing while your training job never actually starts. The $467 Problem Nobody Talks About Here is what happens when you submit a 32-GPU distributed PyTorch job to a standard Kubernetes cluster: the default scheduler places 28 pods across available nodes, then runs out of GPU resources for the remaining 4. Those 28 pods sit in Running state, consuming GPU memory and cluster resources, waiting for the stragglers. The job makes no progress. At H100 cloud rates, 28 idle GPUs for 40 minutes costs around $467 — and that is one deadlock event, on one cluster, on one job. Make it worse: submit two jobs simultaneously. Job A holds 28 GPUs waiting for 4 more. Job B holds 20 GPUs waiting for 12 more. Each is blocking resources the other needs. Neither ever completes. The only fix is manual intervention: delete both jobs, free the cluster, reschedule. This is not a rare edge case. It is a structural problem with how the default Kubernetes scheduler was built — one pod at a time, with no awareness of group placement requirements. Teams have been working around this for years using Volcano, Kueue, or custom batch frameworks. These tools work, but they add operational complexity: a separate scheduler to maintain, version skew to track, a different API surface to debug. Gang scheduling Beta in Kubernetes 1.37 is the beginning of a native answer. What Gang Scheduling Does in 1.37 Gang scheduling introduces the PodGroup API at scheduling.k8s.io/v1beta1 . A PodGroup is a named object with one critical field: minCount . It declares the minimum number of pods that must be placeable simultaneously before any of them get bound to nodes. Pods belong to a group via a label: scheduling.k8s.io/pod-group: <name . When the gang scheduling plugin is active, pods in a PodGroup enter a PreEnqueue holding state. They do not move to the active scheduling queue until the cluster can satisfy minCount simultaneously. If the cluster cannot commit to placing the whole group within scheduleTimeoutSeconds , the group stays queued. No partial placement, no idle GPUs, no deadlock. Workload-aware preemption also graduated to Beta alongside this. When a higher-priority training job needs to schedule and cannot fit, the preemption logic now reasons about entire PodGroups rather than individual pods. It will not evict half of an active training run and leave the rest idle. Preemption targets complete groups — which is the only outcome that makes economic sense for expensive GPU workloads. apiVersion: scheduling.k8s.io/v1beta1 kind: PodGroup metadata: name: pytorch-training-group namespace: ml-training spec: minCount: 8 scheduleTimeoutSeconds: 120 --- apiVersion: v1 kind: Pod metadata: name: trainer-0 namespace: ml-training labels: scheduling.k8s.io/pod-group: pytorch-training-group spec: containers: - name: trainer image: pytorch/pytorch:2.6-cuda12.4 resources: limits: nvidia.com/gpu: "1" How to Enable Gang Scheduling Gang scheduling is disabled by default in 1.37. Enable the GenericWorkload feature gate on your API server and scheduler — this replaces the old GangScheduling and WorkloadAwarePreemption gates that appeared in earlier alpha versions. You also need to enable the scheduling.k8s.io/v1beta1 API group. If you tested gang scheduling on Kubernetes 1.36, stop before upgrading. The v1alpha2 PodGroup API is not supported in 1.37. Any v1alpha2 PodGroup objects in your cluster will trigger API server validation errors on upgrade. Run kubectl get podgroups.scheduling.x-k8s.io --all-namespaces before upgrading, delete any existing objects, and recreate them as v1beta1 after the upgrade completes. DRA Extended Resource Goes GA Quietly significant: DRA Extended Resource KEP-5004 reached Stable in 1.37 after Alpha in 1.35 and Beta in 1.36. This feature lets DRA drivers satisfy requests for traditional extended resources — nvidia.com/gpu , amd.com/gpu , anything in that format — without requiring a separate device plugin running alongside the DRA driver. Per the official DRA update blog https://kubernetes.io/blog/2026/09/03/kubernetes-v1-37-dra-updates/ , you set the extended resource name directly on a DeviceClass, and pods requesting it via the extended resource API get matched through DRA with no ResourceClaim changes required on the workload side. For operators running mixed fleets — some workloads using the old extended resource API, some using the new DRA ResourceClaim API — this is the migration bridge you have been waiting for. Existing workloads keep working without modification while the backend allocation logic moves to DRA transparently. What Else to Watch on 1.37 Gang scheduling is Beta, not GA. For most teams, Beta is acceptable for production evaluation; for teams with strict GA-only policies, this is a 1.39 or 1.40 story based on how Kubernetes typically promotes features. Managed providers — EKS, GKE, AKS — generally need several weeks after a Kubernetes release before the version is available. Check your provider’s roadmap before locking in an upgrade timeline. The full 1.37 release notes https://kubernetes.io/blog/2026/08/26/kubernetes-v1-37-release/ also flag two deprecation paths worth tracking now: IPVS is on a slow removal path in favor of nftables run kubectl get cm -n kube-system kube-proxy -o yaml | grep mode to check your cluster , and cgroup v1 is off by default with a temporary override. Neither is a fire today, but both become urgent before 1.40 ships. The direction is clear. Gang scheduling Beta plus DRA Extended Resource GA plus scale-to-zero HPA from the same release: Kubernetes is assembling the pieces to be a first-class AI training platform. The partial-job deadlock problem has a native solution worth evaluating today. For deeper context on the DRA side, the Cloud Native Now analysis https://cloudnativenow.com/features/kubernetes-v1-37-enhances-dynamic-resource-allocation/ covers the full DRA feature progression across 1.35 through 1.37.