Also published on the CNCF blog. Cross-post with canonical link to the CNCF version.
We got paged one Tuesday morning. A critical production service had crashed under traffic—not gradually degraded, but crashed. Hundreds of pending pods. Users were seeing 15–20% error rates. The incident postmortem was brutal: reactive autoscaling had fired, but it was already too late.
The timeline looked like this:
By 06:45, the spike was over. Customers had already hit errors. The system had tried to scale, but the physics of infrastructure didn't cooperate.
The root cause wasn't a bug—it was a mismatch between workload requirements and provisioning speed. Scaling CPU-only services takes minutes. Scaling GPU nodes takes 3–5x longer: firmware loads, drivers initialize, CUDA gets ready. Reactive HPA, by definition, waits for demand to appear before ordering capacity. For GPU workloads, that's reactionary in the worst sense.
We realized that night: we needed to see the spike coming before it arrived.
We had all the data we needed already—Prometheus was collecting CPU, memory, latency, RPS, and NVIDIA GPU utilization continuously. A week of history sat in storage. The question wasn't whether we could predict demand; it was whether we could predict it well enough to matter.
We decided to test a hypothesis: what if a Kubernetes controller running every 60 seconds could look at the past hour of metrics and forecast demand 10 minutes into the future? Not perfectly—just well enough to pre-provision capacity so it's ready by the time traffic actually arrives.
The idea was simple. The execution was... more interesting.
We settled on a three-part architecture: Predict, Provision, Absorb.
We evaluated several options:
We went with Bi-LSTM—a 2-layer LSTM (64 units → 32 units) that looks backward and forward in the sequence. Why? Because we saw patterns that weren't just linear trends. GPU utilization had micro-bursts, recovery valleys, and anomalous plateaus. Bi-LSTM handled those better than simpler approaches. It wasn't the "correct" choice theoretically; it was the right choice for our data.
The model runs inside the controller. We retrain it weekly with the latest data, but the deployed model runs inference-only—no external ML platform, no model serving layer. Just TensorFlow Lite embedded in a Go controller binary.
The tradeoff: Better accuracy came at the cost of longer training time and harder interpretation. We couldn't explain why the model predicted a specific demand value the way we could with ARIMA. But for autoscaling, we only needed to be right 80% of the time, not right 100%.
The model predicts based on learned patterns, but anomalies happen. A marketing campaign launches. A feature goes viral. Traffic patterns shift in ways the training data didn't prepare for.
We added a burst detector that runs in parallel. It maintains an adaptive threshold based on the rolling standard deviation of recent predictions vs. actuals. If real demand suddenly exceeds prediction by some confidence interval, the burst detector triggers and increases the scale-out aggressiveness.
It's not a secondary model—it's a heuristic safety net. When it fires, it signals: "Your model doesn't know what's coming. Scale faster."
Here's where we learned a hard lesson: if you tell Kubernetes to scale 100 pods per second, you'll discover exactly how many scheduler cycles per second your cluster can handle. Spoiler: it's not that many.
The graduated scaler rate-limits scaling to 20 pods per minute. This sounds slow, but it's actually perfect:
The target utilization is 70%, not 100%. This leaves headroom for the actual spike and gives the predictor time to be wrong without cascading failures.
During our week-long hackathon validation, we set up a controlled simulation environment and validated the design against realistic GPU demand patterns:
The design incorporates two critical production guardrails:
Through our validation, we simulated the exact spike pattern from the original incident. The predictor caught it 11 minutes early—validating that this approach would have prevented the error rates and pending pods that actually occurred. This gives us confidence that the design is production-ready.
Model complexity: We started with Bi-LSTM because we had the infrastructure. Honestly? A well-tuned ARIMA model probably gets 80% of the way with 10% of the infrastructure. We should have benchmarked simpler approaches longer.
Retraining: For production deployment, weekly retraining is a baseline, but we recommend retraining on every significant incident. When traffic patterns shift (new feature launch, competitor activity), the model gets stale within days. Build retraining into your incident playbooks from day one.
Explainability: "Why did the predictor forecast 150 pods?" is a question we couldn't answer well. For operators, that's painful. A hybrid approach—LSTM for the forecast, SHAP for explaining the top contributing factors—would've been worth the complexity.
Gradual rollout: For production deployment, we recommend three phases instead of two: shadow (log predictions, don't scale) → capped scale (max 10 pods/predict cycle) → full scale. Each phase gives you a chance to validate stability before expanding scope. Smaller blast radius = faster recovery if something unexpected happens.
We built this without proprietary extensions:
This matters because it means you can run it on any Kubernetes cluster with Prometheus already running. No new infrastructure. No new vendor. Just a controller and a trained model artifact.
Predictive scaling shines when:
It's overkill when:
From our validation during the hackathon, several questions remain for production deployment:
If you're considering this approach for your workloads, we'd be curious how those questions play out in your context.
The core pieces are straightforward:
We've learned that the model architecture matters less than consistent validation. Start simple. If simple works, ship simple. Complexity isn't a feature.
The incident that motivated this work showed a critical gap: reactive autoscaling fails when provisioning is slow. Our validation proves that predictive scaling closes that gap. The approach is straightforward, production-ready, and CNCF-native—no external dependencies.
This isn't about perfect prediction—it's about good-enough prediction happening early enough to matter. For GPU workloads on Kubernetes, that shift from reactive to predictive can be transformative.
If your workloads have slow provisioning (GPU nodes, bare-metal, anything > 2–3 minutes) and somewhat predictable demand patterns, this approach deserves evaluation. We'd love to hear how it works for your teams, and any lessons you discover as you deploy it to production.