Google donated OpenRL to the CNCF Sandbox this week at KubeCon in Salt Lake City — and it did so before the API even stabilized. That detail matters more than the donation itself. When a company puts infrastructure into neutral governance while the design is still malleable, the community gets a seat at the table before the defaults get baked in. That window won’t stay open long.
OpenRL is a self-hosted reinforcement learning post-training framework built for Kubernetes. It lets teams run RL fine-tuning loops on their own GPUs — without Ray, without a managed cloud training service, and without waiting for a hyperscaler to productize the workflow you need. The project started as a GKE Labs research preview in February 2026 and now has over 100 contributors.
The Problem: Your GPUs Are Mostly Idle #
Standard RL post-training loops are sequential by design: generate rollouts, score rewards, update weights, repeat. The bottleneck is obvious once you measure it. For models trained with a 16,000-token max response length, rollout generation consumes 91 percent of execution time for math tasks and 84 percent for code tasks. During those rollouts, the trainer sits idle. During weight updates, the sampler sits idle. The result: GPU clusters burning money at 40 to 75 percent idle time.
OpenRL’s answer is cooperative time-slicing, developed in the llm-d project. Instead of sequential phases on dedicated hardware, the system treats each RL phase as a schedulable unit that can be interleaved across shared physical accelerators. A job yields at a phase boundary, its device state is serialized from GPU memory to host DRAM, the next job resumes exactly where it left off — no container restart, no framework re-initialization. Accelerator duty cycles climb from roughly 40 percent to over 70 percent. The accuracy improvement on the Legal Agent Benchmark is sharper still: 50 percent with standard supervised fine-tuning versus 70 percent after RL loops with OpenRL.
Four Primitives, One Clean Separation #
The API is small by design. OpenRL exposes four primitives: forward_backward for data transfer, optim_step for weight updates, sample for rollout generation, and save_state for weight persistence. The intent is to split ML engineering from infrastructure engineering cleanly — researchers write RL loops in plain Python, infra teams own the Kubernetes backend.
The cooperative time-slicing layer sits one level below the researcher’s code:
orchestrator = TimeSliceOrchestratorClient(target="orchestrator:50051")
@orchestrator.on_accelerators(group_id="trainer-group")
def train_phase(model, trajectories):
return model.update(trajectories)
@orchestrator.on_accelerators(group_id="sampler-group")
def generate_phase(model, prompts):
return model.generate(prompts)
for epoch in range(EPOCHS):
trajectories = generate_phase(policy, dataset)
rewards = compute_rewards(trajectories)
train_phase(policy, rewards)
Two decorators. The orchestrator handles the checkpoint/restore cycle underneath. The RL loop looks sequential to the researcher; on the hardware it isn’t. Deployment is consistent across environments: test on a Mac, point the same code at GKE when ready to scale. Gemma and Qwen are validated out of the box; PyTorch and vLLM are the supported backends.
Where OpenRL Fits in the CNCF AI Stack #
Google donated three projects together at KubeCon: Agent Substrate for sandboxing untrusted agent code, gVisor for kernel-level isolation, and OpenRL for the training layer. The first two were covered here last week. OpenRL completes a CNCF-hosted pipeline that now covers inference (llm-d 1.0), agent execution safety (Agent Substrate, gVisor), and post-training (OpenRL). The training gap is closed.
The Kubernetes-native angle separates OpenRL from the alternatives. OpenRLHF runs on Ray. Slime uses SGLang and Megatron. Neither is natively Kubernetes-aware. OpenRL uses Kubernetes Dynamic Resource Allocation to handle accelerator provisioning automatically — which matters when you’re running multi-tenant fine-tuning alongside inference serving. Nous Research is already running it for Hermes Enterprise deployments.
The Window to Contribute Is Now #
Google was explicit: OpenRL was donated to CNCF before the design hardened specifically to get hardware vendors, inference engine teams, training framework developers, and model providers involved before the API freezes. If your infrastructure constraints differ from Google’s — different accelerators, different scale, different multi-tenancy requirements — contributing now is how those constraints get addressed in the standard instead of patched around it later.
The CNCF Sandbox designation means OpenRL has cleared governance and community requirements but hasn’t reached graduation-level stability. That’s the moment. The roadmap includes selective snapshotting (LoRA adapters only, not full model weights), TPU support, and automated job placement. All of that is still being decided. Get to github.com/gke-labs/open-rl, run the getting-started guide against your hardware, and file issues before the defaults harden.