Popular DR Strategies in Cloud:
We planned below Disaster Recovery strategies for our Web applications hosted on AKS , backed up by 50 + ML Services running on On-Prem:
Our Infra setup with running hybrid architectures: Many web application layers deployed on Azure Kubernetes Service (AKS) in the cloud, calling back into machine learning inference services still hosted on-premises — often because of data residency rules, GPU capex already sunk into a data center, or ML models trained on sensitive data that can't leave the corporate network.
This hybrid pattern is powerful, but it introduces a DR problem that pure cloud-native architectures don't have: your failure domain now spans two estates you don't fully control together — a cloud region that can fail, and a data center that can fail independently of it, connected by a network link that can also fail on its own.
This post adapts the classic four-tier DR framework (Backup & Restore → Pilot Light → Warm Standby → Multi-Site) to this AKS + on-prem ML topology, and then walks through a concrete highly-available, multi-zone reference architecture.
The Two Foundational Metrics:
Every DR conversation starts with the same two numbers, and in a hybrid AKS/on-prem design you should define them separately for each tier of the stack (web/API layer, data layer, and the ML inference layer), because they will not be the same:
• Recovery Time Objective (RTO) — how long the business can tolerate the web app or the ML-backed feature being down.
• Recovery Point Objective (RPO) — how much data (user data,
feature-store data, model state) you can afford to lose.
A typical mismatch: the web tier might have an RTO of 15 minutes, while the on-prem ML training pipeline has an RTO of 24 hours because it depends on a single data center's GPU cluster. Your DR strategy has to account for that gap explicitly, usually through graceful degradation rather than pretending both tiers fail over at the same speed.
The Four DR Tiers, Adapted for AKS + On-Prem ML
As you move from lowest cost to highest cost, recovery speed increases (lower RTO/RPO). Here's how each tier translates when the backend dependency is an on-prem ML service rather than another cloud region.
1. Backup and Restore (Lowest Cost, Slowest Speed)
How it works in this topology:
• AKS/cluster state: Back up Kubernetes manifests, Helm releases, and persistent volumes with Velero (targeting an Azure Storage Account), plus Azure Backup for any managed disks.
• Container images: Enable geo-replication on Azure Container Registry (ACR) so images are already durable in a second region without a manual restore step.
• Application/config data: Regular exports of Azure SQL/Cosmos DB via native backup, or Azure Backup Vault for PaaS data stores.
• On-prem ML artifacts: Nightly (or more frequent) replication of trained model binaries, feature-store snapshots, and the model registry (e.g., MLflow) to Azure Blob Storage. This is the critical, often-forgotten piece — if the on-prem site is unreachable, you need the last known good model sitting in the cloud, not just the app tier.
Cost vs. Speed: Minimal ongoing spend — nothing is running idle. But recovery means provisioning a fresh AKS cluster, redeploying workloads from Velero, and standing up an emergency inference service (see the "ML fallback" pattern below) from the last replicated model. Expect RTO in hours, RPO measured from the last backup cycle.
2. Pilot Light (Low Cost, Moderate Speed)
How it works:
• A minimal secondary AKS cluster exists in a second Azure region, defined entirely as code (Bicep, Terraform, or ARM templates), scaled to near-zero (e.g., a single system node pool, no user workloads running).
• Core namespaces, secrets (via Azure Key Vault + CSI driver), and networking (VNet, private endpoints) are pre-provisioned so the "light" is already on — you're not starting from a blank slate.
• On-prem ML fallback: A lightweight, always-warm inference stub — a smaller/quantized model or a cached-prediction service — is kept deployed in the secondary region's AKS cluster. It isn't meant to replace the full on-prem model; it's meant to keep the application functional but degraded while the primary ML path is restored.
Cost vs. Speed: Low ongoing cost since only skeleton infrastructure runs continuously. Failover means scaling the node pools up via automation and pointing traffic to the secondary cluster — RTO typically drops to tens of minutes.
3. Warm Standby (Higher Cost, Fast Speed)
• A partially scaled AKS cluster runs continuously in the secondary region with real (not stub) application pods and the Horizontal Pod Autoscaler and Cluster Autoscaler configured to burst on failover.
• Data layer is continuously replicated (e.g., Azure SQL active geo-replication, Cosmos DB multi-region writes).
• For the ML dependency, this tier is where a secondary on-prem site or a colocated edge inference cluster becomes worth the investment — either a second data center with replicated models, or GPU-backed nodes in Azure (via Azure Machine Learning endpoints or GPU node pools in AKS) that mirror the on-prem model's behavior closely enough to serve as a live hot backup, fed by continuous model-artifact sync.
• Azure Traffic Manager or Azure Front Door health-probes the primary path (app + on-prem ML) and can redirect traffic before a human intervenes.
Cost vs. Speed: Meaningfully higher cost — you're running real capacity around the clock. In return, RTO drops to single-digit minutes because the standby path is already serving low-volume traffic and simply needs to absorb the full load.
4. Multi-Site / Active-Active (Highest Cost, Fastest Speed) • Fully active AKS clusters in two or more Azure regions, each capable of independently serving 100% of traffic, fronted by Azure Front Door or Traffic Manager with active-active routing.
• The ML dependency is the hard part here: true multi-site requires either (a) two independent on-prem data centers with synchronized model registries and load-balanced inference traffic over redundant ExpressRoute circuits, or (b) accepting that ML inference is migrated into the cloud (Azure ML / GPU AKS node pools in each region) so the cloud becomes multi-site while on-prem becomes a batch training source rather than a real-time dependency.
• This is also where you decouple the app from synchronous calls to on-prem ML entirely, using an event/queue-based pattern (Azure Service Bus, Event Hubs, or Kafka) so a slow or unreachable on-prem inference path never blocks the web tier — the app degrades to cached/default predictions instead of failing outright.
Cost vs. Speed: Highest cost by far — full redundant capacity, redundant network circuits, and (usually) a redesign to remove hard synchronous coupling to a single on-prem site. In exchange, RTO/RPO approach near-zero, and the on-prem site's failure stops being a single point of failure for the whole product.
Comparison at a Glance :
Tier Ongoing Cost RTO RPO On-Prem ML Handling
Backup & Restore $ Hours Hours (last backup) Model artifacts backed up to cloud storage; emergency stub built on demand
Pilot Light $$ Tens of minutes Tens of minutes Warm stub/quantized model kept ready in secondary region
Warm Standby $$$ Single-digit minutes Minutes Hot secondary inference path (2nd DC or cloud GPU nodes), continuously synced
Multi-Site $$$$ Near-zero Near-zero Dual on-prem sites or ML migrated fully into cloud; async decoupling via queues
Designing a Highly Available, Multi-Zone AKS Web Application
Translating the classic AWS multi-AZ EC2 pattern into Azure/AKS terms, and extending it to cover the on-prem hop:
• Azure Front Door: Acts as the global entry point and CDN — caches static content at the edge, applies WAF policies, and performs global health-probing/failover across regions, similar in role to CloudFront + Route 53 combined.
• Azure Load Balancer / Application Gateway (with AGIC): Distributes traffic across AKS ingress controllers, providing L4/L7 load balancing and SSL termination in front of the cluster.
• AKS with Availability Zones: Deploy node pools across all three Availability Zones in the region, and set topologySpreadConstraints / pod anti-affinity so replicas of each deployment are spread across zones — this is the direct equivalent of an EC2 Auto Scaling Group spanning multiple AZs. Combine with the Cluster Autoscaler and HPA so pod and node counts flex with demand automatically.
• Azure Cache for Redis: Sits in front of both the application data layer and, importantly, in front of the on-prem ML calls — caching recent inference results so that a transient blip in on-prem connectivity or latency doesn't degrade every user request, only new/uncached ones.
• Hybrid connectivity resilience: This is the piece with no AWS-native equivalent, since it's the link back to the data center. Provision redundant ExpressRoute circuits (from different peering locations) with a Site-to-Site VPN as a tertiary fallback, and front the on-prem ML endpoint with Azure Private Link/Private Endpoint so the AKS workloads always resolve to the healthiest available path. Wrap every call to the on-prem service in a circuit breaker (e.g., Polly in .NET, resilience4j-style patterns in other stacks) so a struggling on-prem dependency fails fast and the app falls back to cached or default responses instead of cascading timeouts.
Putting it together
A request flows: Azure Front Door → Application Gateway → AKS ingress (zone-redundant pods) → Redis cache check → ExpressRoute (primary) / VPN (fallback) → on-prem ML inference → response cached in Redis for subsequent requests.
Every hop in that chain has its own failure mode, and the DR tier you choose should be applied consciously at each hop rather than uniformly across the whole system — it's common, and often the right cost/speed trade-off, to run the web/app tier at Warm Standby while the on-prem ML tier only reaches Pilot Light, compensated for with aggressive caching and graceful degradation in the application logic.
Key Takeaways
Define RTO/RPO per tier, not just per application — the web layer and the on-prem ML layer will almost never share the same recovery targets.
Backup on-prem ML artifacts (models, feature stores, registries) to cloud storage regardless of which DR tier you choose elsewhere — it's cheap insurance and enables an emergency fallback at any tier.
Decouple the web tier from synchronous, blocking calls to on-prem ML wherever possible using caching and queue-based patterns — this alone can turn an on-prem outage from a full outage into a graceful degradation.
Treat network connectivity (ExpressRoute/VPN) as its own failure domain with its own redundancy plan — it's the one link in this architecture that doesn't exist in a pure single-cloud design.
Multi-zone AKS deployment (Front Door, App Gateway, zone-spread node pools, Redis) gives you regional high availability essentially "for free" at moderate cost — treat it as the baseline, then layer the four DR tiers on top for cross-region and cross-site resilience.