Inside Replit's New Sandbox Engine, Part 2: Orchestrating MicroVMs at Scale Replit rebuilt its sandbox orchestration on Kubernetes to manage tens of thousands of high-churn microVMs at once, replacing a custom control plane that ran Docker containers on Google Compute Engine VMs autoscaled by Managed Instance Groups. Replit engineers said the old system scaled reactively on observed CPU and memory utilization, required replacing every VM to roll out host-agent changes, and was tied to Google Cloud, blocking multi-cloud support for enterprise customers. The new design starts pods via a supervisor that boots and owns each microVM, letting Kubernetes handle scheduling and bin-packing, and Replit chose it over HashiCorp's Nomad and over Kata Containers to keep direct access to Cloud Hypervisor for device hotplug, memory snapshots and future live migration. Intro In Part 1, we gave you an inside look into how we rebuilt our sandbox runtime https://frontier.replit.com/sandbox-engine-microvms-part-1/ to scale for the agentic era. Whilst microVMs give us the security boundaries we needed, they presented some interesting challenges for orchestration and scaling. They’re slower to boot and less lightweight compared to containers, and at Replit we run 10s of thousands of high-churn microVMs at once. In this second part, we’ll cover why we moved orchestration onto Kubernetes, our architecture and the optimizations we made to orchestrate these at scale Where we started: orchestration in the old engine Our old orchestration system worked well for many years, it even scaled to handle the needs of various use cases before AI as Replit explored the different markets in the Cloud IDE space. We ran Docker containers on Google Compute Engine GCE VMs, autoscaled using Managed Instance Groups MIGs . Each of these VMs ran a host agent that communicated with a custom control plane. Over the years, the system accumulated special cases that made it hard to change safely. Most of the context was in a few people’s heads and as we were growing we realized we needed to rethink our orchestration system. Since we were rewriting the runtime for microVMs anyway, it was the right moment to replace orchestration too. Some of the problems we wanted to address with the new system were: - Reactive scaling. MIGs scaled on observed CPU and memory utilization, so hosts were already overloaded by the time capacity arrived - Rollouts restarted everyone and were slow. Any time we made changes to the host agent, we needed to replace every VM, restarting every sandbox in the fleet - Tied to Google Cloud. Tooling and infrastructure-as-code were built around MIGs, which blocked supporting other clouds for our enterprise customers - Thin tooling. No simple way to list every running sandbox, restart one remotely, or handle operational issues during incidents Why Kubernetes We decided to build the new system on top of Kubernetes. Our main goal was to hand off the problems solved by existing orchestration systems node management, scheduling, rollouts, observability , whilst keeping full control over our runtime. We had initially considered HashiCorp’s Nomad, because it was the closest to our custom control plane in architecture, and wasn’t opinionated about containers. However, we ended up weighing against it as the strengths of Kubernetes and its ecosystem outweighed the complexity it brought. We were already using Kubernetes in other parts of our architecture, to deploy our web servers, agent infrastructure, and even parts of our control plane. Every cloud platform has made it dead simple to spin up managed Kubernetes clusters, which also gives us a much easier path to supporting BYOC bring your own cloud . To run microVMs on Kubernetes nodes, we start pods using our supervisor which in turn boots and owns the microVM. Kubernetes takes care of scheduling the pods, bin-packing across nodes. It also gives us networking within the clusters, giving each pod an IP address. We chose this simpler approach, instead of using something like Kata Containers https://katacontainers.io/ an open source solution for running microVMs as pods in Kubernetes , as it gives us more direct access to Cloud Hypervisor to hotplug devices, snapshot memory and live migration in the future. We found the strongest benefits in the ecosystem, we got a lot of observability and tooling for free OTel , and later we were able to adopt SPIFFE/SPIRE https://spiffe.io/ for identity and Istio https://istio.io/ for our service mesh. Hiring was also a big factor as we grow, our newest team mates are able to reason about our system more easily with prior Kubernetes knowledge. Built for scale and concurrency To spin up a sandbox, we need to boot a Linux kernel, pull images, start a bunch of runc containers, and bring up guest agents. Since sandboxes start from the same base, we can use warm pooling to absorb most of this initialization and wait for the last-mile configuration. This brings the scheduling down to < 50ms when users request sandboxes and keeps Kubernetes pod start-up out of the hot path. To manage the warm pools, we built custom controllers and Custom Resource Definitions CRDs https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ which act as Deployments/ReplicaSets to keep our pools at their desired number of replicas. The CRD lets us define different types of pools to support different resource sizes and runtime tiers. We also configure HPAs to dynamically scale up and down the pools based on usage, enabling the cluster autoscaler to scale up nodes preemptively before the warm pool capacity is depleted. Acquisition from the warm pool just involves attaching and mounting the user’s filesystems and setting up their specific configuration. This takes roughly 500ms, with most of the time spent setting up storage. Given that powerful LLMs can take seconds to produce their first token, the sandbox is usually ready before the model even has anything to say. Once a pooled pod is acquired, it leaves the pool for good and dies once the sandbox is no longer being used by the user. We never recycle the pods, so we can ensure that no stale data is ever left around within a sandbox, guaranteeing strong isolation and protection of our user’s data. Rolling out a new version of our runtime no longer restarts anyone’s sandbox. New warm pods pick up the new version within about 10 minutes of a new build being promoted, which means security fixes and patches to the guest kernel reach new sandboxes quickly. Sandboxes that are already running keep the version they started with, while our host-side security monitor is hot-reloaded across all of them, as we described in Part 1. If needed, we have tooling to force restart sandboxes in batches in cases of incidents or severe security issues. At Replit, we start more than a million sandboxes daily. To operate at this scale, we also keep the system sharded into multiple cells https://replit.com/blog/geo-part-3-deploy . Each has its own pools and control plane state, and represents an isolated failure domain. Prepare once, share the immutable parts Beyond warm pooling, we can do a lot to optimize the shared parts of the infrastructure running within each node. Since we pull down multiple container images to start our runtime, we can use containerd https://containerd.io/ to cache OCI image layers as read-only EROFS images. This means only the first handful of sandboxes have to actually pull layers down from the OCI registry on boot, every other sandbox benefits from the cache. We can also greatly reduce memory pressure on our nodes by using virtio-pmem https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html x1-68900019 and direct access DAX https://docs.kernel.org/filesystems/dax.html to attach these layers inside the guest. This allows us to share the host-backed pages across thousands of sandboxes on the node without building a separate page-cache copy in every VM. In a small benchmark, two microVMs reading the same image kept 95% less of it in their own page cache with pmem + DAX. We’re not the only ones betting on EROFS for sandbox images: Amit Sai recently wrote about speeding up sandbox performance at Modal by moving their gVisor sandboxes to lazily loaded EROFS images https://amitsai.com/writing/erofs/?v=2 . What the load tests taught us Before putting real users on the new engine, we load tested it in staging with synthetic clients that constantly created new sandboxes as old ones expired. These tests pushed a single cluster to nearly 15,000 concurrent sandboxes and sustained over a thousand fresh boots every minute. In early May, we faced Replit’s tenth-birthday promotion, when the product was free for a whole weekend. The new microVM infrastructure was running as a production canary alongside the existing engine. That cluster absorbed the surge automatically, reaching ~10K high-churn concurrent VMs in our canary cluster without noticeable issues or regressions . People building real-world applications gave us our best load test and validation of the new infrastructure. Since then, we have repeatedly exercised the system with synthetic loads before launches. The system is built to successfully sustain ~3x our daily traffic without failures. Through this we’ve been able to make several improvements, such as improving assignment to spread sandboxes across nodes. The weakness of the system comes with warm pool exhaustion, since provisioning physical nodes naturally takes more time. However, this is solvable with both intentional and speculative pre-provisioning, and in practice, we have rarely experienced pool exhaustion in production. Making memory count When we launched microVMs, bare metal nodes weren’t available on GKE Google Kubernetes Engine , so we had to support running them on regular cloud VMs using nested virtualization https://cloud.google.com/compute/docs/instances/nested-virtualization/overview . This worked, but it came at a cost. Every memory access in the guest goes through an extra layer of address translation, and in our early prototypes our in-sandbox agent took twice as long to start as it did on the old engine. We were able to win most of this back by backing guest memory with 2 MiB huge pages on the host. Fewer pages means fewer translations, which brought startup back in line with the old engine. Pre-allocating huge pages for every VM would stop us from overcommitting memory though, so we use transparent huge pages https://docs.kernel.org/admin-guide/mm/transhuge.html THP on a best-effort basis. An Agent often leaves its sandbox waiting while the model generates the next action. Reserving every environment’s maximum CPU and memory as dedicated hardware would waste a lot of resources. We can offload immutable image pages to the host with DAX, but users’ application memory remains private and idle VMs hold onto it. We use virtio-balloon free-page reporting to return unused guest memory to the host, and tune the guest to compact memory aggressively so more of it can be returned. We even turn off THP inside the guest, so the 2 MiB blocks stay free to hand back to the host. By default the guest only reports free 2 MiB blocks, but we lowered this to 128 KiB, which reclaims a lot more memory. The trade-off is that each small report can force the host to split one of the huge pages backing the VM, so we’re choosing density over some of the performance huge pages give us. The runtime also supports DAMON-based reclamation https://docs.kernel.org/admin-guide/mm/damon/reclaim.html with compressed zram https://docs.kernel.org/admin-guide/blockdev/zram.html swap. DAMON identifies cold pages, and anonymous data can move into compressed memory. This approach can really help to reduce memory and free more frames for the balloon to return, but at the cost of higher CPU usage. We are currently testing and tuning this in canary clusters. Operating it day to day Overall our new orchestration system has dramatically increased our ability to operate our infrastructure at scale. We were able to solve for multiple issues in our old system, including how we scale, rollout changes faster and observe what’s happening inside our clusters. Spinning up new cells and clusters has also been greatly simplified, and we’re able to meet our enterprise customer’s needs much faster for providing single tenant environments. We’re looking forward to continuing to build on this system, and are already trialing things like memory snapshots, supporting suspend and resume, bring-your-own-cloud and more. Acknowledgements Authored by Connor Brewster, Jason Dellaluce, Adnan Abdulhussein, Luis Hector Chavez, and the rest of the Compute Platform team at Replit. If you want to work with us on cutting-edge infrastructure, our team is hiring.