Accelerating agentic RL and evaluation research velocity with 45x faster GKE Agent Sandbox

When scaling up agentic reinforcement learning (RL) and evaluation across massive parallel rollouts, frontier AI labs inevitably hit a bottleneck: Expensive GPU clusters sit idle, waiting minutes for CPU sandbox cold-starts, plus thousands of multi-gigabyte SWE-bench-style image pulls and scheduling backlogs. It’s a sandbox infrastructure problem that silently slows down your research and burns your training budget.

To solve this fundamental infrastructure bottleneck, today we are introducing GKE Agent Sandbox****optimized for RL along with the Agent Sandbox RL orchestration SDK, plus native integrations for popular RL gyms and harnesses, now generally available.

As the operating system for modern AI, Kubernetes has evolved to power massive GPU/TPU training clusters and distributed inference. Now Kubernetes is expanding to drive the next AI compute frontier: agents. But unlike static workloads, agentic workloads evolve rapidly, so infrastructure must evolve just as fast. Rather than guessing at what RL researchers needed, we placed Kubernetes itself on an auto-research and verification loop driven by performance benchmarks and evaluations. We used heavy agentic benchmarks like SWE-bench to intentionally stress-test and break our own clusters. Every bottleneck that surfaced — from etcd timeouts to GPU idle spikes — was fed back into our development cycle to refine GKE’s core primitives.

This resulted in a purpose-built sandbox layer for agentic RL and eval workloads that features:

With this new primitive, AI labs and agent-native startups can now reliably run large scale agentic RL trajectories and evals simultaneously, minimizing accelerator idle time and drastically accelerating their research velocity.

Before we talk about the solution, let’s be precise about what makes agentic RL so demanding for infrastructure in the first place. In a standard agentic RL loop, an LLM policy generates actions like code snippets on GPUs and executes them inside isolated CPU sandboxes to observe a reward signal. However, when scaling up this loop to support tens of thousands of parallel rollouts, three critical infrastructure bottlenecks emerge:

These are not hypothetical problems. They are the daily reality for frontier AI labs training state-of-the-art agents.

To fan-out the code execution sandboxes, we set up a relatively modest cluster — a 10-node gVisor sandbox pool with GKE image streaming enabled, the Agent Sandbox controller, and an in-cluster SDK driver to claim warm pods. We tested various strategies and setups including a large number of images and high cardinality.

GKE Agent Sandbox is the open Kubernetes primitive for secure agent execution, featuring built-in SandboxWarmPool capabilities that eliminate cold-start overhead by maintaining pre-initialized, healthy environments. By integrating SandboxWarmPool with GKE Image Streaming, we effectively support RL and eval workloads that demand high image cardinality — even with thousands of large images (>1.2GB), while delivering the exceptionally low TTFC required for responsive agentic training. Its snapshot, suspend and resume capabilities support checkpointing for error recovery, and sandbox forking for parallel agent branching logic to explore multiple trails.

This GKE primitive is already powering the agentic RL training infrastructure at Mistral AI, a frontier AI lab:

“To push the boundaries of reinforcement learning, you need infrastructure that can instantly scale to handle unpredictable demand. By leveraging GKE's high-performance Agent Sandbox for RL, we can seamlessly orchestrate hundreds of thousands of secure environments across clusters and handle spikes of over 30,000 sandboxes on a single cluster. It provides the reliable foundation we need to accelerate our model training and iteration cycles.” - Jean-Malo Delignon, Research Engineer, Mistral AI

To help researchers harness this power without writing Kubernetes YAML or embedding custom daemons into container images, we built the Agent Sandbox RL orchestration SDK. It exposes a clean, async Python API with pluggable warm-pooling strategies tailored to your specific evaluation or RL training pattern. We also built native integrations for RL tools like Gymnasium, NVIDIA NeMo Gym, and OpenHands, with more to come.

We ran this on a 10-node gVisor sandbox pool against two workloads: a 500-image SWE-bench environment, and a harsher 4,578-image R2E corpus that does not fit in local disk.

1.TTFC and tail latency: 10x–45x faster

Concurrency ranged from 500 simultaneous tasks and sandboxes up to 18,312 (4,578 R2E images × 4 rollouts). The gain held across every setup.

How we got here. We instrumented the controller, the SDK, and the RL fleet, then ran hundreds of comparison runs against that fixed 10-node budget. Two changes produced nearly all of the gain:

Image hydration moved off the critical path. A high-cardinality corpus far exceeds local disk, so pulling cold images mid-training creates I/O contention and multi-minute tails. The SDK plans image placement across nodes, and GKE Image Streaming plus upfront warm-pooling handle the rest — hydration finishes before the rollout ever asks for it.

The control plane is paced. High-concurrency rollouts trigger API-server thundering herds. Agent Sandbox Controller v1.0.0 adds rate controls that bound how fast warm pools refill, keeping etcd and the API server stable through the burst.

The tuning knobs, the benchmark harness, and the load tests behind these numbers all ship in the agent-sandbox-rl example, which emits human-readable and JSON reports so you can reproduce the comparison on your own cluster.

2. 3x less control-plane and pod lifecycle churn during multi-trajectory rollouts

Raw Kubernetes recreates a pod for every trajectory step. At 18,312 tasks that inflates scheduling overhead, wastes disk I/O, and — in our large-scale tests — triggered unbounded garbage-collection loops. The SDK's in-place recycle strategy keeps the pod alive instead, running an in-pod git reset and repository checkout between rollout episodes. Pod creations drop 3.1x and the control plane stays flat through the burst.

Warming environments up front spends inexpensive CPU and background cluster time so that image streaming and readiness checks never land on the critical path. For an RL fleet that’s a straightforward trade: Accelerator idle time is the expensive resource, and this approach eliminates it.

One detail matters at training scale: The SDK counts and surfaces every failed sandbox as retriable rather than silently dropping it. An untracked drop is not just a lost rollout — it is reward bias.