I Stopped Paying for Idle GPUs - Scale-to-Zero AI Inference on OKE with KEDA

An engineer on Oracle Cloud Infrastructure (OCI) built a scale-to-zero AI inference system on Oracle Kubernetes Engine (OKE) using KEDA to eliminate costs from idle GPUs. The system scales GPU pods down to zero when there is no traffic and spins them up on demand, reducing monthly GPU costs from over $2,000 to near zero for low-traffic environments. A lightweight proxy queues requests during cold starts, which can take 2-3 minutes for GPU pod provisioning.

A single A10 GPU on OCI costs $1.52/hr. Running 24/7, that's $1,094/month. For a production inference service with steady traffic, that's fine. But I had a staging environment and a couple of internal tools that got maybe 20 requests per day. I was paying over $2,000/month for GPUs that sat idle 95% of the time. The obvious solution: scale to zero when there's no traffic, spin up when a request comes in. KEDA does this on Kubernetes, but getting it to work properly with GPU pods took some figuring out. With normal HTTP services, KEDA watches a metric HTTP requests, queue depth, whatever , and Kubernetes can spin up a new pod in seconds. The user barely notices. GPU pods are different: So you can't just scale-to-zero and expect sub-second response times when traffic returns. The trade-off is cost savings vs. cold start latency. For my use case internal tools, staging , a 2-3 minute cold start was acceptable. helm repo add kedacore https://kedacore.github.io/charts helm install keda kedacore/keda \ --namespace keda-system \ --create-namespace I'm using the nginx ingress controller's Prometheus metrics to track request rate. If you're using OCI's native load balancer, you'd use OCI Monitoring metrics instead. prometheus-scaledobject.yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: vllm-scaler namespace: inference spec: scaleTargetRef: name: vllm-inference minReplicaCount: 0 scale to zero maxReplicaCount: 3 cooldownPeriod: 300 wait 5 min of no traffic before scaling down pollingInterval: 15 triggers: - type: prometheus metadata: serverAddress: http://prometheus.monitoring:9090 metricName: http requests total query: | sum rate nginx ingress controller requests{ namespace="inference", service="vllm-inference" } 2m threshold: "1" scale up if 1 req/sec averaged over 2 min activationThreshold: "0.1" activate from zero if any traffic The key settings: minReplicaCount: 0 — this is what enables scale-to-zero cooldownPeriod: 300 — 5 minutes of no traffic before scaling down prevents flapping activationThreshold: "0.1" — even a trickle of traffic triggers scale-up from zeroWhen the pod scales from zero, there's a gap. The request that triggered the scale-up needs to wait for the pod to be ready. I handle this with a simple queue pattern: queue-proxy.yaml — lightweight proxy that holds requests during cold start apiVersion: apps/v1 kind: Deployment metadata: name: inference-proxy namespace: inference spec: replicas: 1 always running, tiny resource footprint template: spec: containers: - name: proxy image: iad.ocir.io/mytenancy/inference-proxy:v1 ports: - containerPort: 8080 env: - name: BACKEND URL value: "http://vllm-inference:8000" - name: TIMEOUT SECONDS value: "180" wait up to 3 min for backend resources: requests: cpu: 50m memory: 64Mi The proxy is a tiny Go service always running, costs almost nothing that: func proxyHandler w http.ResponseWriter, r http.Request { backendURL := os.Getenv "BACKEND URL" timeout, := strconv.Atoi os.Getenv "TIMEOUT SECONDS" deadline := time.Now .Add time.Duration timeout time.Second backoff := 2 time.Second for time.Now .Before deadline { resp, err := http.DefaultClient.Do cloneRequest r, backendURL if err == nil { copyResponse w, resp return } time.Sleep backoff backoff = min backoff 2, 15 time.Second } http.Error w, "inference backend unavailable, try again shortly", 503 } The slowest part of cold start isn't model loading — it's waiting for OKE to provision a GPU node when none exist. This takes 3-5 minutes. My workaround: keep one GPU node always available, but let the inference pods on it scale to zero. The node costs money even when idle, but it's a single node vs. multiple. And when traffic comes in, the pod starts in ~90 seconds model loading instead of 5+ minutes node provisioning + model loading . GPU node pool with min 1 node always warm oci ce node-pool update \ --node-pool-id $GPU NODE POOL ID \ --node-config-details '{ "size": 1, "placementConfigs": ... }' For staging environments where the 5-minute cold start is acceptable, I set the node pool to autoscale from 0 to 2 nodes and let OKE handle it. My three GPU workloads staging vLLM, internal summarizer, internal code review tool were running 24/7 on three A10 instances: | Before | After | |---|---| | 3x A10 always-on | 1x A10 warm node + scale-to-zero pods | | $3,282/month | ~$1,094/month warm node + ~$50 burst usage | $3,282/month | ~$1,144/month | 65% savings. The internal tools scale up when someone uses them a few times a day and scale back down after 5 minutes of idle. The warm node means cold starts are 90 seconds, which is fine for internal users. This works for internal tools, batch endpoints, staging environments, and anything where "please wait a moment" is an okay response. Pavan Madduri — Oracle ACE Associate, CNCF Golden Kubestronaut. I'm also building keda-gpu-scaler for GPU-aware autoscaling. GitHub | LinkedIn | Website | Google Scholar | ResearchGate