{"slug": "the-open-ai-native-stack-on-kubernetes-from-open-models-to-distributed-inference", "title": "The open AI-native stack on Kubernetes: From open models to distributed inference and agents", "summary": "Google Cloud used KubeCon + CloudNativeCon North America 2026 to detail an open-source AI-native stack for Kubernetes and Google Kubernetes Engine (GKE) built on five CNCF Sandbox projects: OpenRL for RL post-training, llm-d 1.0 for distributed inference with prefix-cache-aware routing and disaggregated serving, Ray on GKE with KubeRay, Agent Substrate for multi-turn agent workflows, and gVisor for kernel-level sandboxing of untrusted agent code. Google Cloud said the stack targets accelerator underutilization in RL, cache-blind load balancing that wipes out key-value cache locality, framework fragmentation across Ray and Slurm clusters, and sandbox escapes in multi-tenant clusters. The post also cites TorchTPU and domain models such as Gemma and DolphinGemma, and points readers to sample manifests for their own clusters.", "body_md": "Over the past decade, Kubernetes established itself as the standard for orchestrating microservices and containerized applications. Today, we're witnessing the next major architectural shift in distributed computing: **AI workloads moving from isolated research prototypes into large-scale, multi-tenant production fleets**. In this post, we'll walk through the open-source stack that powers this shift and show you the sample manifests that you can try on your own Kubernetes clusters. You can also join us at KubeCon + CloudNativeCon North America 2026 to see these projects in action through live demos and technical sessions.\n\nFor modern engineering teams, production AI is rapidly moving beyond generic calls to closed, black-box APIs. Organizations are realizing that their long-term competitive advantage lies in domain specialization. Platform engineers, ML researchers, and AI builders want the autonomy to run high-performing **open-weight models** (such as the [Gemma](https://ai.google.dev/gemma/docs) family). They want to adapt them to proprietary enterprise datasets by using **reinforcement learning (RL)** and serve them with predictable latency, high throughput, and maximum hardware efficiency.\n\nWhether it's scientific initiatives training domain models like [**DolphinGemma**](https://blog.google/innovation-and-ai/products/dolphingemma/) (an LLM helping scientists study how dolphins communicate) on acoustic data, legal platforms evaluating complex contracts, or engineering teams building autonomous coding agents, the objective is the same: **achieving frontier-level task performance at a fraction of the serving cost and latency.**\n\nYet, scaling RL post-training, stateful distributed inference, and autonomous agent execution on traditional container infrastructure introduces substantial hurdles:\n\n**Accelerator underutilization in RL**: When you alternate between training and rollout generation, expensive GPUs and TPUs remain idle for most of the training run.\n\n**Cache-blind load balancing**: When you use standard Kubernetes round-robin and least-connections routing, multi-turn agent calls are scattered across arbitrary pods and wipe out key-value (KV) cache locality.\n\n**Framework fragmentation**: When you operate parallel, siloed compute clusters (such as bespoke [Ray](https://www.ray.io/) or [Slurm](https://slurm.schedmd.com/overview.html) setups) alongside Kubernetes, you fragment capacity and multiply maintenance toil.\n\n**Agent security in multi-tenant clusters:** When you execute untrusted, model-generated shell commands and scripts in shared environments, you expose clusters to sandbox escapes and privilege escalation.\n\nAt [**KubeCon + CloudNativeCon North America 2026**](https://rsvp.withgoogle.com/events/google-cloud-at-kubecon-north-america-2026?pli=1), Google Cloud is addressing these challenges by introducing and expanding a unified, open-source AI-native stack across Kubernetes and [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) based on these components:\n\n**OpenRL** **(CNCF Sandbox):** A Kubernetes-native post training service that simplifies RL post-training and maximizes accelerator utilization.\n\n**llm-d 1.0** **(CNCF Sandbox):** An open distributed inference control plane that provides prefix-cache-aware routing, disaggregated serving, and multi-tier KV cache management.\n\n**Ray on GKE with KubeRay** **(CNCF Sandbox)**: A framework that provides enterprise lifecycle management, elastic autoscaling, and [Kueue](https://kueue.sigs.k8s.io/)-driven scheduling for distributed Ray compute and data processing.\n\n**Agent Substrate** **(CNCF Sandbox)**: An open orchestration runtime that provides lifecycle management and execution targets for multi-turn AI agent workflows.\n\n**gVisor** **(CNCF Sandbox)**: A kernel-level container sandbox that delivers strong system call isolation when you execute untrusted agent code in multi-tenant clusters.\n\n[**TorchTPU**](https://pytorch.org/blog/introducing-torchtpu/): A backend that provides first-class, native PyTorch support across Cloud TPUs and GPUs inside upstream [vLLM](https://vllm.ai/) and [SGLang](https://docs.sglang.io/).\n\nWhile supervised fine-tuning (SFT) has become standard practice, domain excellence especially in multi-step reasoning, mathematical proofing, and autonomous tool use requires RL.\n\nHistorically, it's been difficult to bridge the world of ML experimentation with production Kubernetes fleets. ML researchers want to iterate rapidly in Python and PyTorch by using standard reward functions, while platform teams need multi-tenant governance, quota enforcement, and predictable scheduling.\n\nTo eliminate this friction, Google created [**OpenRL**](https://openrl.dev) and contributed it to the **CNCF Sandbox**.\n\n**Clean 4-primitive Python abstraction**: Researchers write standard Python training code that's centered around four core primitives (`forward_backward`, `optim_step`, `sample`, and `save_state`). OpenRL translates these primitives into Kubernetes scheduling workflows without requiring developers to learn Kubernetes internals or manage infrastructure code.\n\n**Cooperative accelerator time-slicing**: OpenRL interleaves training steps and rollout inferences on the same physical accelerators. This approach avoids the need to dedicate separate, static clusters to compute-bound training and latency-sensitive rollout generation.\n\n**Delta-weight synchronization**: Shipping full multi-gigabyte model weights over cluster networking after every optimization step creates massive network saturation. OpenRL transmits only parameter deltas to rollout workers, eliminating rollout cold starts.\n\n**Zero-config dynamic resource allocation (****DRA****)**: OpenRL automatically profiles model context lengths, batch dimensions, and LoRA versus full fine-tuning configurations. It then dynamically bin-packs workloads across a heterogenous accelerator fleet by using native Kubernetes DRA claims.\n\nAfter you post-train a model, serving it serving it under interactive, bursty agent traffic presents a new set of distributed systems challenges. Standard ingress controllers treat LLM traffic as stateless HTTP requests, scattering sequential turns across different replicas and constantly rebuilding KV cache.\n\nGoogle Cloud, Red Hat, IBM Research, CoreWeave, and NVIDIA co-founded the project, with contributions from over 500 engineers across 25+ organizations. Today, [**llm-d 1.0**](https://llm-d.ai) is the CNCF Sandbox distributed inference control plane that transforms standalone inference engines (like vLLM and SGLang) into a coordinated, cluster-wide serving fabric.\n\n**Prefix-cache-aware routing (****Endpoint Picker****)**: The Endpoint Picker integrates into the [** Kubernetes Gateway API Inference Extension,**](https://gateway-api-inference-extension.sigs.k8s.io/) and GKE manages it natively through [** GKE Inference Gateway**](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-gke-inference-gateway). The Endpoint Picker evaluates real-time KV cache residency across worker pods. In production deployments, it's [doubled KV cache hit rates](https://llm-d.ai/blog/llm-d-v1.0-state-of-the-art-inference-ready-for-day-2) from 35% to 70% and [cut median time-to-first-token (TTFT)](https://cloud.google.com/blog/topics/ai-infrastructure/llm-d-1-0-an-open-source-stack-for-distributed-inference) by 70-85% on multi-turn agent workloads.\n\n**3-Tier KV cache offload**: When accelerator high-bandwidth memory (HBM) nears capacity during long-context interactions, llm-d cascades KV cache blocks from **HBM → host CPU RAM → local NVMe SSD / Google Cloud Managed Lustre / Cloud Storage.** This offload prevents expensive context recalculations.\n\n**EPD Disaggregation with** **DisaggregatedSet**: Decouples **Encoding, Prefill, and Decode (EPD)** onto specialized hardware pools that are connected by high-speed interconnects. DisaggregatedSet is built built on Kubernetes [LeaderWorkerSet (LWS)](https://lws.sigs.k8s.io/stable/docs/). It enables independent horizontal scaling and coordinated, zero-downtime rolling upgrades.\n\n**Native TorchTPU upstream support**: PyTorch models run natively on Google Cloud TPUs inside upstream vLLM and SGLang through TorchTPU. Developers get day-zero parity across TPU v5e, v6e, and v7x alongside NVIDIA GPUs without the need to maintain private forks or rewrite pipelines.\n\n**Latency-tolerant async processing & flow control**: The llm-d-async-processor component pulls background, latency-tolerant tasks from Pub/Sub or Redis into cluster slack capacity. It applies strict traffic prioritization so background processing never degrades interactive user experiences.\n\n**Fleet observability with Prism**: Track cluster-wide throughput, inter-token latency (ITL), TTFT, and cache hit ratios by using [** Prism**](https://github.com/llm-d/llm-d-prism), the llm-d open-source benchmarking and performance dashboard.\n\nWhile specialized runtimes handle inference and RL loops, distributed data preparation, embedding generation, and distributed PyTorch training depend heavily on **Ray**.\n\n**KubeRay** provides an open-source, Kubernetes-native operator that manages the complete lifecycle of Ray clusters, jobs, and services (RayCluster, RayJob, and RayService custom resources) on Kubernetes.\n\n**Declarative cluster lifecycle**: Platform engineers can spin up ephemeral, self-healing Ray clusters on demand for ad-hoc training runs, or run long-lived clusters for batch pipelines.\n\n**Native Kueue queueing & multi-tenancy**: KubeRay integrates directly with **Kueue** for multi-tenant fair sharing, priority preemption, and quota management across shared GPU and TPU capacity.\n\n**Elastic autoscaling**: KubeRay dynamically scales worker pods based on active Ray tasks and actor placement, preventing idle resource consumption.\n\nAs multi-turn AI agents advance from conversational assistants to autonomous systems that generate code, execute shell commands, and orchestrate external APIs running untrusted, model-generated code in shared clusters introduces serious security risks.\n\nGoogle Cloud is contributing **Agent Substrate** and continuing core contributions to **gVisor** in the CNCF Sandbox:\n\n**Kernel-level sandboxing**: gVisor provides a user-space kernel that intercepts and virtualizes system calls, which helps ensure that untrusted agent code can't compromise the host kernel or access neighboring tenant workloads.\n\n**Sub-second ephemeral execution**: Sandboxes spin up in milliseconds per agent turn, providing isolated execution environments for both RL rollout generation in OpenRL and live production agent workflows.\n\nTo evaluate this complete open-source stack on a rigorous enterprise workload, we tested **Gemma 4** against the open-source [**Legal Agent Benchmark (LAB)**](https://github.com/harveyai/harvey-labs) developed by [Harvey.ai](http://harvey.ai). This benchmark spans 24 complex legal practice areas from mergers and acquisitions (M&A)M&A due diligence to contract dispute analysis. The evaluation produced these results:\n\n**Benchmark results**: While standard SFT plateaued around 50% accuracy on multi-step reasoning rubrics, when we applied reproducible RL loops with **OpenRL, we lifted model accuracy to 70%**.\n\n**Operational efficiency**: The entire training pipeline dynamically shared GPU and TPU capacity with concurrent LoRA and inference jobs on GKE without manual partition management.\n\n| **Organization persona** | **Core challenge** | **Architecture with OpenRL + llm-d + KubeRay + Gemma** | \n|---|---|---|\n| **Enterprise platform teams** | Manage multiple internal ML squads without fragmented Slurm/Ray clusters or runaway hardware bills. | Transform existing GKE clusters into self-hosted **Reinforcement Learning-as-a-Service (RLaaS)** by using DRA, Kueue, and accelerator time-slicing. | \n| **Regulated enterprises (finance, healthcare, legal)** | Intellectual property (IP), compliance, and privacy mandates prohibit sending proprietary data to third-party public APIs. | Post-train and serve 1B-30B Gemma models completely inside a private Virtual Private Cloud (VPC) with domain-specific reward functions for superior task accuracy and lower total cost of ownership (TCO). | \n| **AI-native builders & agent developers** | High serving costs and latency bottlenecks on long-context, multi-turn agent interactions. | Deploy llm-d prefix-cache routing that's paired with gVisor-isolated **Agent Sandbox** execution targets. | \n| **Model-as-a-Service & neocloud providers** | Deliver high-margin, multi-tenant post-training and inference across diverse hardware fleets. | Expose OpenRL’s multi-tenant APIs and llm-d’s multi-accelerator control plane across mixed GPU and TPU clusters. | \n\nAttending KubeCon NA in Salt Lake City? Come see these open-source technologies in action:\n\n**Join Google Cloud at KubeCon + CloudNativeCon North America 2026:**\n\nCome say hi to us at booth #122 in Hall B.\n\nDiscover Googlers talks on our [Google Cloud at KubeCon North America 2026](https://rsvp.withgoogle.com/events/google-cloud-at-kubecon-north-america-2026?pli=1) event site.\n\n**Attend featured sessions & keynotes:**\n\nKubernetes with Conviction: How Customization is Shaping the OSS AI Ecosystem's Future - [Salt Palace | Level 1 | Hall DE](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/schedule/?id=1301727).\n\nThe Latest in GPU, TPU, NIC and Other Device Support (WG Device Management) - [Salt Palace | Level 2 | 250 D-F](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/schedule/?id=1289704)\n\nKubernetes Solutions for Agent-Shaped Problems - [Salt Palace | Level 1 | Hall DE](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/schedule/?id=1248040)\n\nHow Anthropic Almost Melted GKE (And the PRs That Saved It) - [Salt Palace | Level 1 | Ballroom H](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/schedule/?id=1247612)\n\nGateway API Project Update: Building the Future of Traffic - [Salt Palace | Level 2 | 250 D-F](https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/program/schedule/?id=1247612)\n\n**Watch live demos at the Google Cloud booth:**\n\nDemo | Inferencing with ARM processors on GKE (Tue 1:30-4:30PM MST)\n\nSandbox your agent workflows with gVisor (Tue 4:30-7:30PM MST)\n\nSecurely running fleets of agents with OpenShell (Wed 10:30-1:30PM MST)\n\nCost optimization via GKE spot nodes & node auto-provisioning (Wed 12:30-2:30PM MST)\n\nFrom Reasoning to Action: Arm-Powered Coding Agents with GKE Agent Sandbox and Axion (Thu 10:30-1:30PM MST)\n\n**Explore OpenRL**: Visit [openrl.dev](https://openrl.dev) and check out the [OpenRL GitHub repository](https://github.com/openrl-dev) for Kubernetes-native RL fine-tuning recipes.\n\n**Deploy llm-d 1.0**: Head to [llm-d.ai](https://llm-d.ai) to test Gateway API Inference extensions and deploy prefix-aware inference pools on your clusters.\n\n**Run KubeRay**: Follow the [KubeRay documentation](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started.html) to manage distributed Ray workloads alongside Kueue.\n\n**Deploy Gemma on GKE**: Explore step-by-step guides for deploying Gemma models on TPUs and GPUs in the [Google Kubernetes Engine documentation](https://cloud.google.com/kubernetes-engine/docs).", "url": "https://wpnews.pro/news/the-open-ai-native-stack-on-kubernetes-from-open-models-to-distributed-inference", "canonical_source": "https://cloud.google.com/blog/topics/developers-practitioners/the-open-ai-native-stack-on-kubernetes-from-open-models-to-distributed-inference-and-agents/", "published_at": "2026-11-10 00:49:58.293869+00:00", "updated_at": "2026-11-10 00:50:01.429412+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "mlops", "ai-tools", "artificial-intelligence"], "entities": ["Google Cloud", "Kubernetes", "Google Kubernetes Engine", "OpenRL", "llm-d", "Ray", "KubeRay", "gVisor"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-open-ai-native-stack-on-kubernetes-from-open-models-to-distributed-inference", "markdown": "https://wpnews.pro/news/the-open-ai-native-stack-on-kubernetes-from-open-models-to-distributed-inference.md", "text": "https://wpnews.pro/news/the-open-ai-native-stack-on-kubernetes-from-open-models-to-distributed-inference.txt", "jsonld": "https://wpnews.pro/news/the-open-ai-native-stack-on-kubernetes-from-open-models-to-distributed-inference.jsonld"}}