🚦 Meet Kueue: Smart Job Queueing for Kubernetes 🧠⚙️

Kueue, a Kubernetes-native job queueing system maintained as a kubernetes-sigs project, addresses the gap in Kubernetes' ability to manage when batch jobs start. It provides a control tower that queues jobs, enforces quota, and supports priority-based admission, fair sharing, and elastic jobs. The system is designed to prevent resource starvation and improve GPU utilization and cloud cost management.

Hey everyone 👋 If you run batch jobs, data pipelines, or any kind of AI and ML training on Kubernetes, you have probably hit this wall. Kubernetes is fantastic at deciding WHERE a pod should run, but it is surprisingly clueless about WHEN a job should start. 😅 You submit ten jobs, the cluster fills up, and the rest just sit there as Pending. No real queue, no priority, no fairness between teams. One noisy team can eat all your expensive nodes while everyone else waits. 🥲 That is exactly the gap Kueue fills, and today I want to walk you through it with a pile of hands on examples you can run on any cluster, even your homelab. 🏡 👉 Key takeaway up front: Kueue is a job level manager that holds your jobs in a real queue and only admits them when there is enough quota to actually run them. 🧪 Everything in this guide was tested against Kueue v0.18.1 using the v1beta2 API. I pinned every command and manifest to that version so you do not get surprised by API drift. ✅ Why Kubernetes needs a queue ✅ The building blocks in plain language ✅ Installing Kueue ✅ Setting up quota with a ResourceFlavor, a ClusterQueue, and a LocalQueue ✅ Submitting a Job and watching it get queued and admitted ✅ Priority based admission ✅ Partial admission and elastic jobs ✅ Multiple resource flavors for x86 and arm ✅ Fair sharing between teams with cohorts ✅ Dedicated quota with a shared fallback ✅ Queueing a plain Pod ✅ Why this matters a lot for GPUs and your cloud bill Native Kubernetes scheduling is pod centric. The scheduler looks at one pod at a time and tries to place it. That works great for long running services. Batch workloads are different. They have a beginning and an end, they often need a fixed chunk of capacity, and they compete with other teams for the same nodes. Without a queueing layer you get: ✅ Jobs that fail or stay Pending when resources are tight ✅ No quota governance, so one team can starve the others ✅ No admission priority, so a quick experiment can block production training Kueue is a Kubernetes native job queueing system, maintained as a kubernetes-sigs project. It does not replace the scheduler. It sits in front of it. 🛂 Here is the simple mental model. Think of the Kubernetes scheduler as the runway, and Kueue as the control tower deciding which flight is cleared for takeoff and when. ✈️ When a job arrives, Kueue suspends it, creates a matching Workload object, checks if there is enough quota, and only then lets the pods be created. If there is no room, the job waits politely in the queue instead of failing. There are four pieces you need to know, plus one bonus piece for teams. ✅ ResourceFlavor 🍦 Describes a type of resource, usually tied to node labels. For example x86 nodes versus arm nodes, or GPU nodes versus CPU nodes. If you do not need to distinguish node types, you use one empty flavor. ✅ ClusterQueue 🏦 A cluster scoped object that holds the actual quota. This is where you say how much cpu, memory, or how many GPUs are available. Users do not submit to it directly. ✅ LocalQueue 📥 A namespaced object that points to a ClusterQueue. This is what users actually target with their jobs. ✅ Workload 📦 The internal object Kueue creates for each job to track its admission state. You usually just observe it. ✅ Cohort 👥 bonus A group of ClusterQueues that can borrow each other unused quota. This is the magic behind fair sharing between teams. The simplest method is to apply the released manifests with server side apply. kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.1/manifests.yaml The controller runs in the kueue-system namespace. Give it a few seconds and check it is healthy. kubectl get deploy -n kueue-system You should see the controller manager become ready. NAME READY UP-TO-DATE AVAILABLE AGE kueue-controller-manager 1/1 1 1 30s Prefer Helm? Kueue publishes an OCI chart for each release. Just make sure the chart version matches the release you want. helm install kueue oci://registry.k8s.io/kueue/charts/kueue \ --version=0.18.1 \ --namespace kueue-system \ --create-namespace \ --wait --timeout 300s Since we are not distinguishing node types in this first demo, an empty flavor is all we need. default-flavor.yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ResourceFlavor metadata: name: "default-flavor" Apply it. kubectl apply -f default-flavor.yaml Now we define the quota for the whole cluster. Here we allow 9 cpu and 36Gi of memory, all served by our single flavor. cluster-queue.yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ClusterQueue metadata: name: "cluster-queue" spec: namespaceSelector: {} match all namespaces resourceGroups: - coveredResources: "cpu", "memory" flavors: - name: "default-flavor" resources: - name: "cpu" nominalQuota: 9 - name: "memory" nominalQuota: 36Gi Apply it. kubectl apply -f cluster-queue.yaml One important detail. The flavor name under spec.resourceGroups must match the ResourceFlavor name from step 2. If they do not match, the ClusterQueue will not become ready. 🔗 Users cannot send work to a ClusterQueue directly. They need a LocalQueue in their namespace that points to it. We will put ours in the default namespace. default-user-queue.yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: LocalQueue metadata: namespace: "default" name: "user-queue" spec: clusterQueue: "cluster-queue" Apply it. kubectl apply -f default-user-queue.yaml Quick tip: you can apply all three of the above at once using the example bundle from the project. kubectl apply -f https://kueue.sigs.k8s.io/examples/admin/single-clusterqueue-setup.yaml This is the only change your users need to make to an existing Job. Add the kueue.x-k8s.io/queue-name label pointing to the LocalQueue, and make sure each pod declares resource requests. sample-job.yaml apiVersion: batch/v1 kind: Job metadata: generateName: sample-job- namespace: default labels: kueue.x-k8s.io/queue-name: user-queue spec: parallelism: 3 completions: 3 template: spec: containers: - name: dummy-job image: registry.k8s.io/e2e-test-images/agnhost:2.53 command: "/bin/sh" args: "-c", "sleep 60" resources: requests: cpu: "1" memory: "200Mi" restartPolicy: Never Notice that you do not need to set the job to suspended yourself. Kueue manages suspension for you through a webhook and decides the best moment to start it. 🪄 Create the job. kubectl create -f sample-job.yaml List your local queues. The alias queues also works. kubectl -n default get localqueues NAME CLUSTERQUEUE PENDING WORKLOADS user-queue cluster-queue 0 Kueue creates a Workload object for your job. Have a look. kubectl -n default get workloads.kueue.x-k8s.io NAME QUEUE RESERVED IN ADMITTED AGE sample-job-xxxxx user-queue cluster-queue True 3s Want the full story? Describe the workload. When there is not enough quota, you will see it sit unadmitted with a clear message. kubectl -n default describe workload sample-job-xxxxx Status: Conditions: Message: workload didn't fit Reason: Pending Status: False Type: Admitted The moment quota frees up, Kueue admits it automatically. If you describe the Job itself, the event timeline tells the whole story. Events: Type Reason From Message ---- ------ ---- ------- Normal Suspended job-controller Job suspended Normal CreatedWorkload kueue-job-controller Created Workload: default/sample-job-xxxxx Normal Started kueue-job-controller Admitted by clusterQueue cluster-queue Normal Resumed job-controller Job resumed Normal Completed job-controller Job completed No babysitting required. 🎉 Inside a queue, not all jobs are equal. With a WorkloadPriorityClass you can control admission and preemption priority independently from pod priority. Production training jumps the line ahead of throwaway experiments. 🏎️ First create the priority class. sample-priority.yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: WorkloadPriorityClass metadata: name: sample-priority value: 10000 description: "Sample priority" Then point a Job at it with the kueue.x-k8s.io/priority-class label. priority-job.yaml apiVersion: batch/v1 kind: Job metadata: name: sample-job labels: kueue.x-k8s.io/queue-name: user-queue kueue.x-k8s.io/priority-class: sample-priority spec: parallelism: 3 completions: 3 suspend: true template: spec: containers: - name: dummy-job image: registry.k8s.io/e2e-test-images/agnhost:latest args: "pause" restartPolicy: Never Higher value means higher priority for queuing and preemption. The neat part is this priority does not touch the pod priority, so it does not interfere with your normal Kubernetes scheduling. 👌 Sometimes a big job can still make progress with fewer pods. With the kueue.x-k8s.io/job-min-parallelism annotation, Kueue can admit the job at a reduced parallelism instead of leaving it Pending. partial-job.yaml apiVersion: batch/v1 kind: Job metadata: name: sample-job-partial-admission namespace: default labels: kueue.x-k8s.io/queue-name: user-queue annotations: kueue.x-k8s.io/job-min-parallelism: "5" spec: parallelism: 20 completions: 20 template: spec: containers: - name: dummy-job image: registry.k8s.io/e2e-test-images/agnhost:2.53 args: "entrypoint-tester", "hello", "world" resources: requests: cpu: 1 memory: "200Mi" restartPolicy: Never If only 9 cpu is free, this job is admitted with parallelism 9 instead of waiting for all 20. The completions count stays the same. 🙌 Elastic jobs let you change a running Job parallelism without recreating, restarting, or suspending it. This is an alpha feature, so you must enable the ElasticJobsViaWorkloadSlices feature gate and annotate the Job. elastic-job.yaml apiVersion: batch/v1 kind: Job metadata: name: sample-elastic-job namespace: default annotations: kueue.x-k8s.io/elastic-job: "true" labels: kueue.x-k8s.io/queue-name: user-queue spec: parallelism: 3 completions: 100 template: spec: containers: - name: dummy-job image: registry.k8s.io/e2e-test-images/agnhost:2.53 command: "/bin/sh" args: "-c", "sleep 60" resources: requests: cpu: "100m" memory: "100Mi" restartPolicy: Never When you bump parallelism up, Kueue creates a new admitted Workload for the new pod count and marks the old one as Finished. When you scale down, the extra pods terminate and no new Workload is created. Smooth. 🧘 Real clusters often mix node types. Say you have x86 and arm nodes labelled with cpu-arch. You can create one flavor per architecture. flavor-x86.yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ResourceFlavor metadata: name: "x86" spec: nodeLabels: cpu-arch: x86 flavor-arm.yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ResourceFlavor metadata: name: "arm" spec: nodeLabels: cpu-arch: arm Then reference both in a single ClusterQueue. Here cpu is split across the two architectures, while memory uses the simple default flavor because we do not care which architecture provides it. cluster-queue-multi.yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ClusterQueue metadata: name: "cluster-queue" spec: namespaceSelector: {} match all resourceGroups: - coveredResources: "cpu" flavors: - name: "x86" resources: - name: "cpu" nominalQuota: 9 - name: "arm" resources: - name: "cpu" nominalQuota: 12 - coveredResources: "memory" flavors: - name: "default-flavor" resources: - name: "memory" nominalQuota: 84Gi The labels in the ResourceFlavor must match the labels on your nodes. If you use the cluster autoscaler, make sure it adds those labels to new nodes too. 🏷️ This is where Kueue really shines. Put two ClusterQueues in the same cohort and they can borrow each other unused quota. team-a-cq.yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ClusterQueue metadata: name: "team-a-cq" spec: namespaceSelector: {} cohortName: "team-ab" resourceGroups: - coveredResources: "cpu", "memory" flavors: - name: "default-flavor" resources: - name: "cpu" nominalQuota: 9 borrowingLimit: 6 - name: "memory" nominalQuota: 36Gi borrowingLimit: 24Gi team-b-cq.yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ClusterQueue metadata: name: "team-b-cq" spec: namespaceSelector: {} cohortName: "team-ab" resourceGroups: - coveredResources: "cpu", "memory" flavors: - name: "default-flavor" resources: - name: "cpu" nominalQuota: 12 - name: "memory" nominalQuota: 48Gi Both queues belong to the cohort team-ab. Team A has its own guaranteed quota, but it can also borrow idle capacity from Team B, up to the borrowingLimit of 6 cpu and 24Gi. When Team B needs its capacity back, Kueue handles it. ⚖️ A ClusterQueue can borrow from the cohort even when it has zero nominal quota for a flavor. This lets you give each team dedicated capacity on one flavor, plus a shared pool to fall back on. team-a-cq.yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ClusterQueue metadata: name: "team-a-cq" spec: namespaceSelector: {} match all cohortName: "team-ab" resourceGroups: - coveredResources: "cpu" flavors: - name: "arm" resources: - name: "cpu" nominalQuota: 9 borrowingLimit: 0 - name: "x86" resources: - name: "cpu" nominalQuota: 0 - coveredResources: "memory" flavors: - name: "default-flavor" resources: - name: "memory" nominalQuota: 36Gi shared-cq.yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ClusterQueue metadata: name: "shared-cq" spec: namespaceSelector: {} match all cohortName: "team-ab" resourceGroups: - coveredResources: "cpu" flavors: - name: "x86" resources: - name: "cpu" nominalQuota: 6 - coveredResources: "memory" flavors: - name: "default-flavor" resources: - name: "memory" nominalQuota: 24Gi Read it like this: ✅ team-a-cq has a borrowingLimit of 0 on the arm flavor, so its arm capacity is truly dedicated and cannot be borrowed away. ✅ team-a-cq has a nominalQuota of 0 on the x86 flavor, so it has no x86 of its own and can only borrow x86 from shared-cq. This pattern is great for giving each team a guaranteed slice while still pooling the expensive shared hardware. 🤝 You are not limited to Jobs. Kueue can manage plain Pods too. Just add the queue-name label and resource requests. kueue-sleep-pod.yaml apiVersion: v1 kind: Pod metadata: generateName: kueue-sleep- namespace: default labels: kueue.x-k8s.io/queue-name: user-queue spec: containers: - name: sleep image: busybox command: - sleep args: - 3s resources: requests: cpu: 3 restartPolicy: OnFailure Kueue injects a kueue.x-k8s.io/managed=true label to mark the pods it manages. The same label driven approach works for Deployments, StatefulSets, RayJobs, JobSets, and Kubeflow jobs as well. 🧰 Kueue ships a kubectl plugin so you can manage queues without writing kubectl get workloads.kueue.x-k8s.io every time. Once installed, you get handy commands. List workloads in a namespace kubectl kueue list workload Stop a workload it stays in the queue but will not be admitted kubectl kueue stop workload sample-job-xxxxx Resume it later kubectl kueue resume workload sample-job-xxxxx It also covers create, delete, describe, edit, get, and patch for clusterqueues, localqueues, resourceflavors, and workloads. A nice quality of life upgrade for operators. 🧑🔧 Here is the part that makes finance happy. 🤑 GPU and accelerator nodes are expensive, and they are often the scarcest resource in the cluster. The worst outcome is a job that partially grabs a few GPUs, then waits forever for the rest while those GPUs sit idle and billed. With Kueue you get: ✅ Quota governance so no single team hoards the accelerators ✅ Admission only when the capacity a job needs is available ✅ Priority so production training is admitted before throwaway experiments ✅ Borrowing so idle quota is actually used instead of wasted That combination is exactly why Kueue is becoming a key building block for running AI and ML workloads on Kubernetes at scale. 🚀 ✅ Always set resource requests on your pods. If you only set limits, Kueue treats the limits as requests. If you set neither, quota accounting cannot work. ✅ The queue-name label must point to a LocalQueue that exists in the same namespace as the job. ✅ The flavor names in the ClusterQueue must match your ResourceFlavor names exactly. ✅ Elastic jobs are alpha, so remember to enable the ElasticJobsViaWorkloadSlices feature gate. ✅ Stick to one API version. This guide uses v1beta2, which is the current served version in v0.18.1. Kueue takes Kubernetes from I will place pods wherever and whenever to I will admit jobs in a fair, prioritized, quota aware order. For batch, data, and AI workloads that is a huge upgrade, and it costs you almost nothing to adopt since your jobs only need one extra label. 🙌 To recap the flow: ✅ Install Kueue ✅ Create a ResourceFlavor ✅ Create a ClusterQueue with your quota ✅ Create a LocalQueue per namespace ✅ Add the queue-name label to your jobs ✅ Layer on priority, partial admission, elastic scaling, and cohorts as you grow Give it a spin on a small cluster first, watch the Workload objects, and you will quickly get a feel for how admission works. What is next? I am going to bring an AI agent into my own homelab cluster and show the full setup, so stay tuned for that one. 🤖🏡 Happy queueing and stay safe 👋