{"slug": "day-12-auto-scaling-ai-inference-using-horizontal-pod-autoscalers-for-spiky-api", "title": "Day 12: Auto-Scaling AI Inference — Using Horizontal Pod Autoscalers for Spiky API Traffic", "summary": "A developer published a hands-on guide showing how to autoscale LLM inference pods on Kubernetes using queue depth rather than CPU or memory, since inference engines pre-allocate memory and saturate devices even under light load. The setup exports vLLM's vllm:num_requests_waiting metric through Prometheus Adapter to a Horizontal Pod Autoscaler, with an asymmetric behavior block that scales up with no stabilization window and scales down over a five-minute window in 25% steps to avoid discarding warm pods that take minutes to load.", "body_md": "AI traffic is volatile, and idle GPU or high-memory instances are expensive. You want replicas to follow demand. The obvious tool is the **Horizontal Pod Autoscaler (HPA)**, but its default signals, CPU and memory, mislead you for inference servers.\n\n**Diagram:** FIG 1 · The inference engine exports queue depth, Prometheus stores it, an adapter exposes it to Kubernetes and the HPA adds or removes pods based on it. [See the animated version](https://sureshpallapothu.in/blog/ai-int-day-12-autoscaling-inference).\n\nRequests reach the inference engine, which exports a queue metric. Prometheus scrapes it, an adapter exposes it to Kubernetes as a custom metric and the HPA adds or removes pods based on it.\n\n**Diagram:** FIG 2 · Inference engines pre-allocate memory and saturate devices even under light load, so CPU and memory utilisation say little about demand. Queue depth and time per output token track what users experience. [See the animated version](https://sureshpallapothu.in/blog/ai-int-day-12-autoscaling-inference).\n\nA better signal is the **depth of the waiting queue**, or the time per output token (TPOT). Both track the wait users feel, which is what scaling should protect.\n\n**Diagram:** FIG 3 · As the waiting queue grows, the HPA adds replicas, and it removes them again slowly once the queue drains. The numbers are illustrative. [See the animated version](https://sureshpallapothu.in/blog/ai-int-day-12-autoscaling-inference).\n\nThis scales on the number of waiting requests per pod. vLLM exports this metric as `vllm:num_requests_waiting`, and Prometheus Adapter exposes it to Kubernetes under a name you configure, shown here as `vllm_num_requests_waiting`.\n\n```\napiVersion: autoscaling/v2\nkind: HorizontalPodAutoscaler\nmetadata:\n  name: inference-engine-hpa\n  namespace: ai-platform\nspec:\n  scaleTargetRef:\n    apiVersion: apps/v1\n    kind: Deployment\n    name: inference-engine-gemma\n  minReplicas: 1\n  maxReplicas: 4\n  metrics:\n    - type: Pods\n      pods:\n        metric:\n          name: vllm_num_requests_waiting\n        target:\n          type: AverageValue\n          averageValue: \"10\"\n  behavior:\n    scaleUp:\n      stabilizationWindowSeconds: 0\n      policies:\n        - { type: Percent, value: 100, periodSeconds: 15 }\n    scaleDown:\n      stabilizationWindowSeconds: 300\n      policies:\n        - { type: Percent, value: 25, periodSeconds: 60 }\n```\n\nExposing a custom metric to the HPA needs an adapter such as **Prometheus Adapter**, or you can use **KEDA**, which can scale directly on a Prometheus query. Check the exact metric name your vLLM version exports.\n\n**Diagram:** FIG 4 · Scale out aggressively to absorb a spike, and scale in cautiously, because a model pod takes minutes to load and flapping wastes that work. [See the animated version](https://sureshpallapothu.in/blog/ai-int-day-12-autoscaling-inference).\n\nA new model pod takes minutes to load, so you want to react to a spike immediately and avoid throwing away warm pods the moment traffic dips. The `behavior` block encodes that: no stabilisation delay when scaling up, and a five-minute window with gentle steps when scaling down.\n\nScaling on the queue ties your replica count to the user experience, and it keeps expensive capacity from sitting idle.\n\nThis is the exact example from the video. A small Python model of the autoscaler's rule, using the lesson's numbers. It is not the real HPA.\n\n``` python\nimport math\n\ntarget, low, high = 10, 1, 4   # waiting requests per pod\nwindow = 5                     # minutes, like 300 seconds\nrecs = []\n\nfor minute, waiting in enumerate([5, 30, 60, 20, 5, 5, 5, 5]):\n    want = math.ceil(waiting / target)\n    recs.append(min(high, max(low, want)))\n    pods = max(recs[-window:])\n    print(f\"min {minute}: waiting {waiting:2d} -> {pods} pods\")\n```\n\nRun it:\n\n```\npython hpa.py\n```\n\nOutput I got while recording the video:\n\n``` php\nmin 0: waiting  5 -> 1 pods\nmin 1: waiting 30 -> 3 pods\nmin 2: waiting 60 -> 4 pods\nmin 3: waiting 20 -> 4 pods\nmin 4: waiting  5 -> 4 pods\nmin 5: waiting  5 -> 4 pods\nmin 6: waiting  5 -> 4 pods\nmin 7: waiting  5 -> 2 pods\n```\n\n**Day 13:** Helm charts: packaging, versioning and updating AI deployments.\n\n*Originally published at [https://sureshpallapothu.in/blog/ai-int-day-12-autoscaling-inference](https://sureshpallapothu.in/blog/ai-int-day-12-autoscaling-inference), where this post includes animated diagrams.*", "url": "https://wpnews.pro/news/day-12-auto-scaling-ai-inference-using-horizontal-pod-autoscalers-for-spiky-api", "canonical_source": "https://dev.to/suresh_kumar_de3920bedd1c/day-12-auto-scaling-ai-inference-using-horizontal-pod-autoscalers-for-spiky-api-traffic-4c7h", "published_at": "2026-11-18 21:33:07+00:00", "updated_at": "2026-11-18 21:42:27.987036+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-tools", "large-language-models"], "entities": ["Kubernetes", "vLLM", "Prometheus Adapter", "KEDA", "Horizontal Pod Autoscaler", "Gemma"], "also_reported_by": [], "takeaway": "Scale inference replicas on vLLM's vllm:num_requests_waiting queue-depth metric via Prometheus Adapter, targeting about 10 waiting requests per pod, with zero scale-up stabilization and a 300-second scale-down window in 25% steps.", "recommendation": {"advice": "Autoscale inference pods on queue depth or TPOT instead of CPU/memory, and pair aggressive scale-up with a five-minute scale-down window so slow-loading model pods aren't discarded.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/day-12-auto-scaling-ai-inference-using-horizontal-pod-autoscalers-for-spiky-api", "markdown": "https://wpnews.pro/news/day-12-auto-scaling-ai-inference-using-horizontal-pod-autoscalers-for-spiky-api.md", "text": "https://wpnews.pro/news/day-12-auto-scaling-ai-inference-using-horizontal-pod-autoscalers-for-spiky-api.txt", "jsonld": "https://wpnews.pro/news/day-12-auto-scaling-ai-inference-using-horizontal-pod-autoscalers-for-spiky-api.jsonld"}}