{"slug": "autoscale-gpu-inference-on-eks-with-karpenter-and-spot-instances", "title": "Autoscale GPU Inference on EKS with Karpenter and Spot Instances", "summary": "Karpenter v1.14.0 can autoscale GPU inference on Amazon EKS by provisioning spot GPU nodes on demand, bin-packing a vLLM v0.25.1 model server, and deleting nodes when traffic drops, eliminating static GPU node groups. The setup requires an EKS cluster with Karpenter installed, a region with G5/G6 capacity, and an EC2 spot quota above zero for GPU instances.", "body_md": "# Autoscale GPU Inference on EKS with Karpenter and Spot Instances\n\nLet Karpenter provision and bin-pack spot GPU nodes under a vLLM server, then delete them when traffic drops.\n\n[Emeka Okafor](https://sourcefeed.dev/u/emeka_okafor)\n\n## What you'll build\n\nA Kubernetes-hosted inference stack on Amazon EKS where [Karpenter](https://karpenter.sh) provisions spot GPU nodes on demand, bin-packs a [vLLM](https://docs.vllm.ai) model server onto the cheapest available capacity, and deletes the nodes when traffic drops — no static GPU node group burning money at 3 a.m.\n\n## Prerequisites\n\nVerified against Karpenter **v1.14.0**, Kubernetes **1.36** on EKS, NVIDIA device plugin **v0.19.3**, and vLLM **v0.25.1** in July 2026. You need:\n\n- An\n[EKS](https://docs.aws.amazon.com/eks/)cluster (Kubernetes 1.33+) with Karpenter v1.14.0 already installed per the[official getting-started guide](https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/)— that guide sets up the`KarpenterNodeRole-<cluster>`\n\nIAM role, the spot-interruption SQS queue (`settings.interruptionQueue`\n\n), and`karpenter.sh/discovery`\n\ntags on your subnets and security groups. This tutorial assumes all of that exists. [kubectl](https://kubernetes.io/docs/tasks/tools/),[Helm](https://helm.sh)3.14+, and[AWS CLI](https://aws.amazon.com/cli/)v2 authenticated against the cluster's account.- A region with G5/G6 capacity (e.g.\n`us-east-1`\n\n,`us-west-2`\n\n). - An EC2 spot quota above zero for GPU instances — check\n**All G and VT Spot Instance Requests**(quota code`L-3819A6DF`\n\n). Fresh accounts often have 0, which blocks everything below:\n\n```\naws service-quotas get-service-quota \\\n  --service-code ec2 --quota-code L-3819A6DF \\\n  --query 'Quota.Value'\n```\n\nIf it returns `0.0`\n\n, request an increase (8+ vCPUs) before starting. Export your cluster name for the steps below:\n\n```\nexport CLUSTER_NAME=<your-cluster>\n```\n\n## 1. Enable spot in the account\n\nIf this account has never launched a spot instance, EC2 needs its service-linked role:\n\n```\naws iam create-service-linked-role --aws-service-name spot.amazonaws.com\n```\n\nAn `InvalidInput`\n\nerror saying the role name \"has been taken\" means it already exists — safe to ignore.\n\n## 2. Create a GPU EC2NodeClass\n\nThe EC2NodeClass tells Karpenter *how* to build GPU nodes. The `al2023@latest`\n\nAMI alias is doing real work here: Karpenter resolves it to the EKS-optimized AL2023 AMI variants — including the NVIDIA accelerated one, which ships the GPU drivers and container toolkit — and picks the right variant per instance type automatically. The 100 GiB root volume matters because the vLLM image alone is over 10 GiB; the AL2023 default of 20 GiB gets you disk-pressure evictions.\n\n```\ncat <<EOF | kubectl apply -f -\napiVersion: karpenter.k8s.aws/v1\nkind: EC2NodeClass\nmetadata:\n  name: gpu-inference\nspec:\n  role: \"KarpenterNodeRole-${CLUSTER_NAME}\"\n  amiSelectorTerms:\n    - alias: al2023@latest\n  subnetSelectorTerms:\n    - tags:\n        karpenter.sh/discovery: \"${CLUSTER_NAME}\"\n  securityGroupSelectorTerms:\n    - tags:\n        karpenter.sh/discovery: \"${CLUSTER_NAME}\"\n  blockDeviceMappings:\n    - deviceName: /dev/xvda\n      ebs:\n        volumeSize: 100Gi\n        volumeType: gp3\nEOF\n```\n\n## 3. Create the spot GPU NodePool\n\nThe NodePool defines *what* Karpenter may launch. Two families (`g6`\n\n= NVIDIA L4, `g5`\n\n= A10G) across all sizes gives EC2 Fleet many spot pools to choose from — diversification is what keeps spot interruption rates low. Allowing both capacity types means Karpenter uses spot when it's available and falls back to on-demand only when it isn't; Fleet picks the actual pool with the price-capacity-optimized strategy (lowest price, lowest interruption risk). The taint keeps non-GPU workloads off these expensive nodes, and the `limits`\n\nblock caps the blast radius at 8 GPUs no matter what gets deployed.\n\n```\ncat <<EOF | kubectl apply -f -\napiVersion: karpenter.sh/v1\nkind: NodePool\nmetadata:\n  name: gpu-inference\nspec:\n  template:\n    spec:\n      taints:\n        - key: nvidia.com/gpu\n          value: \"true\"\n          effect: NoSchedule\n      requirements:\n        - key: kubernetes.io/arch\n          operator: In\n          values: [\"amd64\"]\n        - key: kubernetes.io/os\n          operator: In\n          values: [\"linux\"]\n        - key: karpenter.sh/capacity-type\n          operator: In\n          values: [\"spot\", \"on-demand\"]\n        - key: karpenter.k8s.aws/instance-family\n          operator: In\n          values: [\"g6\", \"g5\"]\n      nodeClassRef:\n        group: karpenter.k8s.aws\n        kind: EC2NodeClass\n        name: gpu-inference\n  limits:\n    nvidia.com/gpu: \"8\"\n  disruption:\n    consolidationPolicy: WhenEmptyOrUnderutilized\n    consolidateAfter: 5m\nEOF\n```\n\n`consolidateAfter: 5m`\n\ngives you five minutes of headroom before Karpenter starts tearing down underutilized nodes — long enough to survive a rolling restart without churning instances.\n\n## 4. Install the NVIDIA device plugin\n\nThe accelerated AMI has drivers, but Kubernetes only learns about the GPUs from the [NVIDIA device plugin](https://github.com/NVIDIA/k8s-device-plugin). Karpenter also won't consider a GPU node fully initialized until the `nvidia.com/gpu`\n\nresource registers, so this DaemonSet is load-bearing. Its default values already tolerate the `nvidia.com/gpu:NoSchedule`\n\ntaint:\n\n```\nhelm repo add nvdp https://nvidia.github.io/k8s-device-plugin\nhelm repo update\nhelm upgrade -i nvdp nvdp/nvidia-device-plugin \\\n  --namespace nvidia-device-plugin --create-namespace \\\n  --version 0.19.3\n```\n\n## 5. Deploy the model server\n\nA vLLM deployment serving Qwen2.5-0.5B-Instruct — small enough to download in a minute and fit any single L4/A10G. The `nvidia.com/gpu: \"1\"`\n\nrequest is the trigger: the pod goes Pending, Karpenter sees an unschedulable GPU pod, and launches a node for it. The in-memory `/dev/shm`\n\nvolume avoids vLLM's shared-memory warnings; the CPU/memory requests are sized so one replica bin-packs cleanly onto an `xlarge`\n\n.\n\n```\ncat <<EOF | kubectl apply -f -\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: vllm\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: vllm\n  template:\n    metadata:\n      labels:\n        app: vllm\n    spec:\n      tolerations:\n        - key: nvidia.com/gpu\n          operator: Exists\n          effect: NoSchedule\n      containers:\n        - name: vllm\n          image: vllm/vllm-openai:v0.25.1\n          args:\n            - --model=Qwen/Qwen2.5-0.5B-Instruct\n            - --max-model-len=4096\n          ports:\n            - containerPort: 8000\n          resources:\n            requests:\n              cpu: \"3\"\n              memory: 10Gi\n              nvidia.com/gpu: \"1\"\n            limits:\n              nvidia.com/gpu: \"1\"\n          readinessProbe:\n            httpGet:\n              path: /health\n              port: 8000\n            initialDelaySeconds: 60\n            periodSeconds: 10\n          volumeMounts:\n            - name: shm\n              mountPath: /dev/shm\n      volumes:\n        - name: shm\n          emptyDir:\n            medium: Memory\n            sizeLimit: 2Gi\nEOF\n```\n\n## Verify it works\n\nWatch Karpenter react — a NodeClaim should appear within seconds and go Ready in about two minutes:\n\n```\nkubectl get nodeclaims -w\nNAME                  TYPE        CAPACITY   ZONE         NODE                          READY   AGE\ngpu-inference-8xkzt   g6.xlarge   spot       us-east-1b   ip-192-168-73-21.ec2.internal True    2m4s\n```\n\n`CAPACITY: spot`\n\nis the money column. Once the pod is Ready (first start takes a few minutes: image pull plus model download), hit the OpenAI-compatible API:\n\n```\nkubectl port-forward deploy/vllm 8000:8000 &\ncurl -s localhost:8000/v1/models\n```\n\nExpected output (trimmed):\n\n```\n{\"object\":\"list\",\"data\":[{\"id\":\"Qwen/Qwen2.5-0.5B-Instruct\",\"object\":\"model\",\"owned_by\":\"vllm\", ...}]}\n```\n\nNow watch the bin-packing. Scale up and Karpenter batches the pending pods, then asks EC2 Fleet for the cheapest combination that fits them — several single-GPU spot nodes or one multi-GPU node, whichever prices lower:\n\n```\nkubectl scale deploy/vllm --replicas=3\nkubectl get nodeclaims\n```\n\nScale back to 1 and, five minutes later (`consolidateAfter`\n\n), watch Karpenter consolidate the now-empty nodes out of existence. That's the whole cost story: GPU capacity exists only while pods need it, and it's spot-priced — AWS quotes up to 90% off on-demand, and G-family discounts routinely land north of 50%.\n\n## Troubleshooting\n\n**Karpenter logs show MaxSpotInstanceCountExceeded** during instance launch — your spot vCPU quota for G instances is 0 or too low. Request an increase on\n\n`L-3819A6DF`\n\n(All G and VT Spot Instance Requests); until it lands, Karpenter will retry and may fall back to on-demand.**Pod event: incompatible with nodepool \"gpu-inference\" ... no instance type satisfied resources** — the pod's requests can't fit any instance the NodePool allows, or the NodePool hit its\n\n`limits`\n\n. Check that requests fit an `xlarge`\n\n(4 vCPU/16 GiB) and that existing NodeClaims haven't consumed the 8-GPU cap.**Node is Ready but the pod stays Pending with Insufficient nvidia.com/gpu** — the device plugin isn't running on the node.\n\n`kubectl get pods -n nvidia-device-plugin -o wide`\n\nshould show one pod per GPU node; if you overrode chart values, make sure the DaemonSet still tolerates `nvidia.com/gpu:NoSchedule`\n\n.**vLLM CrashLoopBackOff with torch.OutOfMemoryError: CUDA out of memory** — the model plus KV cache doesn't fit the GPU. Lower\n\n`--max-model-len`\n\n, pass `--gpu-memory-utilization 0.85`\n\n, or constrain the NodePool to instance types with more VRAM.## Next steps\n\nPin the AMI alias to a specific version (`al2023@v2026...`\n\n) in production — `@latest`\n\ntriggers drift replacement of GPU nodes on every AMI release, which can churn long-running inference pods. Add a second, weighted NodePool with `on-demand`\n\nonly to guarantee a baseline while spot handles overflow. Karpenter only scales *nodes*; pair it with an HPA or [KEDA](https://keda.sh) scaling on queue depth or request latency so replicas track traffic. From there, dig into [disruption budgets](https://karpenter.sh/docs/concepts/disruption/) to bound how many nodes consolidate at once, and NVIDIA time-slicing or MIG in the device plugin to pack multiple small models onto one GPU.\n\n## Sources & further reading\n\n-\n[Getting Started with Karpenter (v1.14)](https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/)— karpenter.sh -\n[Karpenter NodeClasses](https://karpenter.sh/docs/concepts/nodeclasses/)— karpenter.sh -\n[Karpenter Scheduling Concepts](https://karpenter.sh/docs/concepts/scheduling/)— karpenter.sh -\n[NVIDIA device plugin for Kubernetes](https://github.com/NVIDIA/k8s-device-plugin)— github.com -\n[vLLM Releases](https://github.com/vllm-project/vllm/releases)— github.com -\n[Using Amazon EC2 Spot Instances with Karpenter](https://aws.amazon.com/blogs/containers/using-amazon-ec2-spot-instances-with-karpenter/)— aws.amazon.com\n\n[Emeka Okafor](https://sourcefeed.dev/u/emeka_okafor)· Security Editor\n\nEmeka has spent over a decade tracking threat actors, vulnerability disclosures, and the evolving landscape of application security, bringing a sharp continent-spanning perspective to his reporting. He's known for translating dense CVE advisories into clear, actionable context that developers and security teams alike actually read.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/autoscale-gpu-inference-on-eks-with-karpenter-and-spot-instances", "canonical_source": "https://sourcefeed.dev/a/autoscale-gpu-inference-on-eks-with-karpenter-and-spot-instances", "published_at": "2026-07-26 17:41:48+00:00", "updated_at": "2026-07-26 17:56:36.255369+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["Amazon EKS", "Karpenter", "vLLM", "NVIDIA", "EC2", "AWS"], "alternates": {"html": "https://wpnews.pro/news/autoscale-gpu-inference-on-eks-with-karpenter-and-spot-instances", "markdown": "https://wpnews.pro/news/autoscale-gpu-inference-on-eks-with-karpenter-and-spot-instances.md", "text": "https://wpnews.pro/news/autoscale-gpu-inference-on-eks-with-karpenter-and-spot-instances.txt", "jsonld": "https://wpnews.pro/news/autoscale-gpu-inference-on-eks-with-karpenter-and-spot-instances.jsonld"}}