Key takeaways #
- GPU utilization across production Kubernetes clusters averages just 5%, yet half of GPU cloud providers surveyed by SemiAnalysis report being completely sold out of H100 and H200 capacity.
- OMNI Compute extends existing EKS, GKE, or AKS clusters to source GPU, TPU, and CPU from any AWS, GCP, or OCI region with no application code changes for inference and batch workloads.
- Edge nodes appear as virtual nodes in your existing cluster, so workloads schedule across clouds using standard Kubernetes mechanisms.
- Built-in GPU sharing includes time-slicing (up to 48 replicas per GPU) and MIG partitioning (up to 7 isolated instances on A100, H100, and H200; A30 supports up to 4), which extracts more work from every GPU you secure.
- DRA: beta in K8s 1.31+, GA targeted for K8s 1.34. Edge nodes support DRA-enabled workloads using ResourceClaim objects for declarative multi-GPU scheduling.
- Teams provisioning continuously in the most cost-favorable regions save 2 to 5 times versus the average GPU market price (Cast AI GPU Price Report). These savings reflect compute costs and are most pronounced when staying within the same cloud provider across regions; cross-cloud deployments add egress fees that reduce net savings.
- Networking costs between edge locations and your main cluster are real. Cross-cloud egress fees can offset compute savings, so account for them in your total cost of ownership.
The regional GPU capacity problem #
GPUs are the paradox hardware of the AI era: simultaneously the scarcest and most wasted compute resource in cloud infrastructure. According to Cast AI’s 2026 State of Kubernetes Optimization Report, which draws on first-party data from tens of thousands of production clusters, average GPU utilization sits at just 5% pre-optimization. On AKS that number drops to 2%. On EKS it lands at 5%, and on GKE it reaches 6%. In practice, 95% of paid GPU capacity idles at any given moment.
Yet the supply picture tells the opposite story. AWS H200 Capacity Block prices rose 15% in January 2026, breaking a 20-year trend of falling compute costs. By April 2026, half of GPU cloud providers surveyed by SemiAnalysis reported being completely sold out of H100 and H200 capacity. Hardware lead times run 36 to 52 weeks. Cloud reserved capacity is routinely booked 6 or more months before the workload needs to run.
The core constraint is visibility. GPU capacity exists globally, but most teams can only see what their single cloud region offers. When that region runs dry, workloads stall. For more context on how this shortage developed, see the Cast AI analysis on GPU shortage mitigation.
Why single-region limits hurt more than they appear
Fewer than 2% of GPU workloads run on Spot Instances, according to the same 2026 Cast AI report. That means most teams pay full On-Demand rates even when cheaper capacity exists in nearby regions. Regional scarcity compounds this: when reserved H100s disappear in us-east-1, teams without a fallback simply wait. There is no automatic rerouting. There is no visibility into whether eu-west-1 or ap-southeast-1 has A100s available at a better price. Solving this requires reaching beyond the single-region model entirely.
One cluster, many clouds and regions #
The conventional multi-cloud answer is cluster federation: one cluster per provider, each with its own control plane, RBAC configuration, tooling stack, and operational overhead. Teams then build custom scheduling logic to route workloads across them. This approach adds infrastructure complexity faster than it solves GPU scarcity. For related reading on GPU optimization strategies within a single cluster, start with the Cast AI guide on Kubernetes GPU optimization.
OMNI Compute takes a different approach. Rather than federating separate clusters, it extends a single existing cluster outward. Your EKS, GKE, or AKS cluster becomes the control plane. Edge locations across AWS, GCP, and OCI appear as virtual nodes inside that cluster, labeled under the castai-omni
namespace with liqo.io/type=virtual-node
. Your workloads see standard Kubernetes nodes. They do not know or care that those nodes sit in a different cloud or region. Cast AI provisions and manages cloud accounts in each edge location on your behalf – you do not need separate IAM roles or cloud accounts for the edge regions.
The architecture runs on Liqo, the open-source multi-cluster project. Liqo handles peering between your main cluster and edge locations, projecting remote resources as local virtual nodes. Cast AI adds autoscaling intelligence on top: comparing Spot and On-Demand prices across every connected region, provisioning where capacity is cheapest and available, and managing NVIDIA driver installation automatically at node bring-up time.
For inference, batch processing, and single-node fine-tuning workloads, OMNI Compute requires no application code changes. For distributed multi-node training that uses NCCL all-reduce operations, cross-cloud virtual nodes introduce WAN latency that can significantly slow gradient synchronization. Distributed training is best kept within a single cloud or region where low-latency interconnects (NVLink, EFA, GPUDirect RDMA) are available. OMNI Compute’s primary strength is sourcing GPU capacity for inference and fault-tolerant batch workloads.
Why not build this yourself?
Teams sometimes consider deploying Liqo directly – the open-source federation layer OMNI Compute is built on – or building multi-cloud GPU routing with custom Kubernetes operators and cloud federation scripts. Liqo is open-source and can federate clusters across clouds, but it requires configuring each cloud account, managing VPN peering, handling driver installation per node type, and building your own cost routing logic on top. OMNI Compute is the managed layer that handles this for you. Beyond the federation layer itself, any self-built approach also requires maintaining separate node provisioning logic per cloud, custom quota management, cross-cloud credential management, and per-provider GPU driver automation. The typical build effort runs 2 to 4 engineering months and produces a system that must be maintained through each cloud provider’s API changes and each Kubernetes minor version upgrade.
OMNI Compute provides this as managed infrastructure. The control plane, cross-cloud peering, GPU driver automation, and cost-comparison autoscaling are maintained by Cast AI.
Single-region vs. multi-cloud GPU: a direct comparison
| Dimension | Single-Region Single-Cloud | Multi-Cloud with OMNI Compute |
|---|---|---|
| GPU availability | Constrained by regional quotas | Sources from any cloud or region |
| Capacity fallback | None. Blocked on shortage. | Automatic fallback to alternative regions and clouds |
| Cost optimization | Fixed regional pricing | Picks lowest-cost available capacity; 2-5x savings range vs. average market price |
| Vendor lock-in | High | Low; portable across AWS, GCP, OCI |
| Compliance control | Limited to provider defaults | Configurable placement governance per workload |
| Operational overhead | Separate per-cloud tooling | Single control plane via existing Kubernetes cluster |
| GPU sharing | Manual configuration | Built-in time-slicing and MIG partitioning |
What OMNI Compute does #
OMNI Compute launched on January 12, 2026, alongside Cast AI’s unicorn valuation milestone. It supports three main cluster types: Amazon EKS, Google GKE, and Azure AKS. From any of these, the system extends to edge locations across any AWS region, any GCP region, and any OCI region, with Oracle Cloud as the launch partner.
GPU support: sharing and partitioning
NVIDIA GPUs across AWS, GCP, and OCI connect automatically. OMNI Compute handles driver installation at provisioning time, so you do not maintain separate AMIs or node images per cloud, and you do not manage driver versions manually across providers. The system provisions the node, installs the correct NVIDIA driver, and makes the GPU schedulable.
Two GPU sharing mechanisms address different workload profiles. Time-slicing divides a single GPU into 1 to 48 temporal replicas. Multiple workloads share the hardware over time with no memory isolation between them. This works well for inference workloads with bursty, light memory requirements. MIG (Multi-Instance GPU) partitioning divides A100, H100, and H200 GPUs into up to 7 physically isolated instances (A30 supports up to 4), each with dedicated compute cores, dedicated memory bandwidth, and dedicated L2 cache. For training jobs or latency-sensitive inference that needs guaranteed memory isolation, MIG is the correct tool.
Dynamic Resource Allocation (DRA): In beta since Kubernetes 1.31 and targeted for general availability in Kubernetes 1.34 (planned for late 2026). DRA enables declarative GPU resource claims via ResourceClaim objects, providing more precise multi-GPU scheduling than device plugins. Edge nodes support DRA-enabled workloads.
TPU and CPU access
TPU capacity flows through GCP edge locations. Workloads targeting Google’s tensor processing hardware schedule against GCP-backed virtual nodes in your main cluster, the same way GPU workloads do. This matters specifically for JAX-based training and inference pipelines where TPUs outperform GPUs on large transformer workloads.
CPU capacity covers both Spot and On-Demand instances across all connected regions. When GPU capacity disappears in your primary region, preprocessing pipelines, data s, and lightweight inference workloads continue on cheaper CPU nodes sourced from wherever pricing is favorable. The autoscaler evaluates Spot CPU pricing across all edge locations continuously, selecting the cheapest available instance type at provisioning time.
Cost-optimized autoscaling
Before provisioning any node, the autoscaler compares Spot and On-Demand pricing across your main cluster region and all connected edge locations. It runs GPU-optimized bin-packing first to maximize utilization of existing nodes before scaling out. Teams provisioning continuously in the most cost-favorable regions save 2 to 5 times versus the average GPU market price (Cast AI GPU Price Report). These savings reflect compute costs and are most pronounced when staying within the same cloud provider across regions. Cross-cloud deployments add egress fees that reduce net savings; account for networking costs in your total cost of ownership calculation.
Placement and governance #
Running workloads across clouds and regions introduces two operational realities: control over where workloads execute, and cost from moving data between locations. OMNI Compute addresses placement directly. Networking cost requires honest accounting on your side.
Workload placement controls
OMNI Compute gives platform teams configurable placement governance per workload. Standard Kubernetes node selectors, taints, and tolerations apply to virtual nodes exactly as they apply to physical ones. Here is a minimal pod spec that targets any available OMNI Compute edge node:
apiVersion: v1
kind: Pod
metadata:
name: inference-pod
spec:
tolerations:
- key: "liqo.io/provider"
operator: "Exists"
effect: "NoSchedule"
nodeSelector:
liqo.io/type: "virtual-node"
containers:
- name: inference
image: your-inference-image:latest
resources:
limits:
nvidia.com/gpu: "1"
The Kubernetes scheduler handles placement automatically – no changes to your application code. For workloads with data residency or compliance requirements, you restrict scheduling to approved regions. For cost-sensitive batch jobs, you open scheduling to all edge locations and let the autoscaler find the cheapest available capacity. Per-workload GPU metrics and cost attribution make team-level chargeback possible: utilization tracks at the workload and namespace level across every edge location.
Uniphore’s VP of Product Management, Erik Johnson, described the operational outcome directly: “Cast AI’s ability to provision GPUs across multiple clouds and regions without requiring a single change to our application code is a game changer for running AI inference at scale. OMNI Compute makes that possible. It enables us to serve our customers in different regions around the world, gives us instant flexibility, eliminates regional GPU shortages, and lets us maintain consistent performance globally.”
Resilience and failure handling
When a cross-cloud virtual node becomes unavailable, Kubernetes marks it NotReady using standard node health detection. The Cast AI Autoscaler detects pending unschedulable pods and provisions replacement capacity automatically. Edge nodes are best suited for stateless or checkpoint-tolerant workloads; stateful workloads that require persistent volumes should use the main cluster’s node pool to avoid cross-cloud PVC reattachment delays.
Networking costs: the tradeoff to account for
Cross-region and cross-cloud networking costs are real and can erode compute savings if you ignore them. The cost structure is straightforward:
Main cluster only: Low networking cost with no cross-region traffic.Same cloud, different region: Moderate cost from cross-region data transfer fees.Different cloud providers: High cost from cross-cloud egress fees.
For inference workloads processing data already stored in S3 or GCS, cross-cloud egress can offset the savings from cheaper GPU pricing in a remote region. The practical architecture keeps data and compute in the same cloud wherever possible. Cross-cloud edge locations work best as fallback capacity for GPU shortages, not as the primary execution path for data-heavy workloads. OMNI Compute’s placement controls let you encode this preference directly: reserve cross-cloud locations for overflow scenarios, keep baseline workloads within the same provider.
Samsung Electronics is among the enterprises evaluating OMNI Compute for global AI infrastructure: Kyotack Tylor Kim, Head of Next Gen Cloud Group at Samsung, noted: “OMNI Compute’s unified control plane has the potential to change how enterprises like Samsung run AI infrastructure globally.”
To see what GPU capacity is available across clouds for your cluster today, connect to Cast AI in read-only mode, it takes under five minutes and requires no changes to your cluster.
Conclusion #
Regional GPU scarcity is not a temporary disruption. Hardware lead times of 36 to 52 weeks, cloud reserved capacity booked 6 months ahead, and H200 prices rising 15% in a single month describe a market that will stay constrained for years. Teams that build AI infrastructure on a single cloud region accept a ceiling on what they can run and when they can run it.
OMNI Compute removes that ceiling. A single Kubernetes cluster, extended across clouds and regions, sources GPU, TPU, and CPU from wherever capacity exists at the best available price. Your inference and batch workloads stay exactly as they are. Your operational overhead stays in one control plane. When your primary region runs dry, your inference pipeline does not wait.
The practical next step is understanding how to optimize the GPU workloads you already have before expanding to multi-cloud capacity. Start with the Cast AI GPU optimization guide, then explore how OMNI Compute extends that foundation across every cloud and region.
Frequently Asked Questions #
What is OMNI Compute for AI?
OMNI Compute is a unified compute control plane from Cast AI that automatically discovers available GPU, TPU, and CPU resources across cloud providers and regions, then transparently extends existing Kubernetes clusters to consume them. Launched January 12, 2026, it supports Amazon EKS, Google GKE, and Azure AKS as main clusters, with edge locations in any AWS region, any GCP region, and any OCI region. No application code changes are required to schedule inference and batch workloads on edge nodes. Distributed multi-node training is best kept within a single cloud or region where low-latency interconnects are available.
Can one Kubernetes cluster span clouds?
Yes. OMNI Compute uses Liqo, an open-source multi-cluster project, to peer your main cluster with edge locations across different cloud providers. Edge nodes appear as virtual nodes in your existing cluster under the castai-omni namespace, and workloads schedule to them using standard Kubernetes node selectors, taints, and tolerations. Cast AI provisions and manages cloud accounts in each edge location on your behalf, no separate IAM roles or cloud accounts required. EKS extends to GCP and OCI regions. GKE extends to AWS and OCI regions. AKS extends to AWS, GCP, and OCI regions, all from a single control plane.
Does it support TPUs and CPU? Yes. OMNI Compute exposes TPU resources through GCP edge locations, making Google’s tensor processing hardware schedulable from your main cluster without application code changes. CPU capacity covers both Spot and On-Demand instances across all connected regions. The autoscaler evaluates CPU and GPU pricing together, provisioning workloads on the cheapest appropriate hardware available at scheduling time.
How do you run AI when GPUs are unavailable in a region? OMNI Compute automatically sources capacity from alternative regions and clouds when your primary region lacks GPU availability. The autoscaler compares Spot and On-Demand GPU pricing across all connected edge locations in real time. When one region is sold out, the system provisions nodes from a region or cloud where capacity exists. Teams provisioning continuously in the most cost-favorable regions save 2 to 5 times versus the average GPU market price (Cast AI GPU Price Report). These savings reflect compute costs and are most pronounced within the same cloud provider; cross-cloud deployments add egress fees that reduce net savings. Placement controls let you define which fallback locations each workload can use, so compliance and cost constraints stay respected during automatic failover.