Coding agents don't only edit files. They install system packages, run test suites and start Postgres, Redis and BuildKit with docker compose. That needs a real Linux kernel, and running untrusted code for many customers needs a hardware boundary around it.
So every kosem environment is a microVM: Kata Containers on Cloud Hypervisor, on GKE nodes with nested virtualization.<sup>[1]</sup> Each one boots its own Linux 6.6 kernel and runs a real dockerd.
Turning microVMs on is one setting on a GKE node pool. Making them feel like a laptop was the real work, and almost none of it was in the hypervisor. It was in the filesystem and in how memory is counted. Unpacking a kernel source tree went from over 2 minutes to 8 seconds, stat on 20,000 files got 45× faster, and a fresh environment boots in 20 seconds.
A word on isolation first, since Docker inside needs a privileged container. Inside a Kata microVM, that makes it root in its own guest kernel and nothing more: the hypervisor is the boundary with the host. A ValidatingAdmissionPolicy keeps it that way. It rejects any Pod in the namespace that isn't on our Kata RuntimeClass, or that asks for host namespaces, host ports or hostPath volumes. Pods get no service account token, and their egress reaches the public internet only: no private ranges and no metadata server.
Keep hot I/O off virtiofs #
By default, Kata shares a Pod's volumes into the guest over virtiofs. Kubelet formats and mounts the disk on the host, and the guest reaches it through virtiofsd. Every stat, open, unlink and readdir leaves the guest, crosses the VM boundary, runs in the host kernel and comes back. A git status on a large repository makes tens of thousands of those trips.
The fix is to give the guest the disk itself. We attach each Hyperdisk volume with volumeMode: Block rather than Filesystem, and Cloud Hypervisor exposes it to the guest over virtio-blk. Our PID 1 in the guest formats it as ext4 on first boot and mounts it. File operations now stay in the guest kernel.
Unpacking the Linux 6.6 source tree, about 80,000 files, took 126 to 129 s over virtiofs. On the raw block disk it takes 7.6 to 8.8 s, about 15× faster.<sup>[2]</sup> Docker ran with its defaults once /var/lib/docker lived on ext4.
Don't let mke2fs discard a fresh disk
The first cold boot on the new disk took over 2 minutes. 101 seconds of that was mke2fs issuing BLKDISCARD over every block of a new Hyperdisk, which Google's storage had already zeroed. Three flags fixed it:
mke2fs -q -F -t ext4 -L kosem -m 0 \
-E lazy_itable_init=1,lazy_journal_init=1,nodiscard \
/dev/kosem-disk
nodiscard skips the discard sweep. The lazy options let the guest kernel zero inode tables and the journal in the background after mount. Formatting now takes under a second at any size. -m 0 drops ext4's 5% reserve for root, which our unprivileged user can't use anyway: 1 GiB back on a 20 GiB disk.
A cold boot on a fresh disk now takes 20 s, and a restart that reattaches an existing disk takes 4.7 s. One more detail matters here: when PID 1 exits, the VM is gone, and so is any dirty page cache in it. Our init calls sync() and unmounts the disk before it exits.
The root filesystem is hot I/O too #
With /workspace and /var/lib/docker on ext4, big repository operations were fast. Short commands and test loops still dragged. Every binary in /usr/bin, every shared library, every Python import and every file in /tmp still came from the container's root filesystem, which Kata also shares over virtiofs.
Terminal-Bench's cobol-modernization task made it obvious.<sup>[3]</sup> One iteration of the agent's test loop took 106 ms in our microVM and 9 ms on a laptop. Microbenchmarks on our default Debian image showed where the time went:
| Root filesystem benchmark | Over virtiofs | From local disk | Speedup |
|---|---|---|---|
stat 20,000 files |
2.7 s | 0.06 s | 45× |
500× create + unlink in /tmp |
1.8 s | 0.13 s | 14× |
500× /bin/true |
2.2 s | 0.7 s | 3.1× |
| 50× Python startup | 1.9 s | 0.6 s | 3.2× |
Even /bin/true was 3× slower, from the dynamic linker's lookups alone.
Our first fix copied the image's files onto the ext4 disk on first boot. That worked for a 600 MB image, and it took the VM down on a 3.8 GiB one (next section). What we run now packs each image once, when it's registered, into a single read-only EROFS file. A builder microVM runs mkfs.erofs inside the image itself, so registry credentials never enter a VM that reads a customer's image.
At boot, init loop-mounts /rootfs.erofs read-only and makes it the lower layer of an overlayfs. The upper layer lives on the ext4 disk. Every lookup goes to the guest kernel's EROFS driver, and nothing is copied at boot. Environments also behave like real machines: packages from apt, global toolchains and $HOME survive a stop, while /tmp and /run start clean.
On the agent's COBOL test, a full test case, running the COBOL binary and its Python port, now takes 31 ms, down from 106 ms. The COBOL binary on its own runs in 2.5 ms.
Two overlayfs details cost us time:
- Turn off the features that tie an overlay to one host. Kata's guest kernel turns on
index,metacopyandredirect_dir, which bind the upper layer to file handles in the lower one. Restart a stopped environment on another node and those handles break. We mount withindex=off,metacopy=off,redirect_dir=off, the same tradeoff Docker'soverlay2makes. - Date warm caches in the far future. Our own dev image ships warm Go module and build caches, which cut the first
go build ./...of our backend from 49.4 s to 7.5 s. But Go marks a cache entry as used by updating itsmtime, and withmetacopyoff, even that copies the whole file up from the image to the disk. The first build copied up 2.2 GB. Setting every file in the cache to the year 2100 stops Go from touching or trimming them, so first builds stay fast and write almost nothing.
Guest page cache is host memory #
The copy-at-boot design is how we found this one. We booted Microsoft's devcontainers/universal image, 3.8 GiB compressed, in an 8 GiB environment. Within a minute, the node's kernel OOM-killed cloud-hypervisor, and the whole VM went with it. Kubernetes restarted the Pod, the copy started again, and the VM died again.
Two defaults combined:
- Kata gives the guest 2 GiB on top of the Pod's limit (
default_memory = 2048).<sup>[4]</sup> An 8 GiB Pod boots a guest that sees 9.7 GiB. - The host's cgroup for the Pod is its limit plus the RuntimeClass's overhead. GKE's built-in microVM RuntimeClass sets that overhead to
130Miand reconciles it, so you can't raise it in place. We already ran our own RuntimeClass with640Mi, and it still wasn't enough.
Copying gigabytes fills the guest's page cache, which is fine when the guest thinks it has 9.7 GiB. But on the host, the guest's page cache is real memory in the cloud-hypervisor process. At 8.8 GB it crossed the Pod's host cgroup, 8 GiB plus 640Mi, while the guest still saw almost a gigabyte free. The host killed the hypervisor before the guest's OOM killer ever ran.
The fix is to shrink Kata's extra guest memory to its minimum with a Pod annotation<sup>[5]</sup> and give the hypervisor real room on the host:
metadata:
annotations:
io.katacontainers.config.hypervisor.default_memory: "256"
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: machines
handler: kata-clh
overhead:
podFixed:
cpu: 250m
memory: 1024Mi # 256 MiB guest extra + 768 MiB for cloud-hypervisor and virtiofsd
Kata allocates a guest's memory up front, so we stress-tested our largest size, 52 GiB, with the fix. Filling page cache and then anonymous memory made the guest's own OOM killer kill the process that was allocating. cloud-hypervisor and the environment stayed up. Packing images into EROFS also removed the multi-gigabyte copy that set this off.
chroot isn't enough when Docker runs inside #
This bug passed every docker run test and broke everything else. docker exec, docker compose exec, healthchecks, docker compose up --wait and nsenter all failed with executable file not found.
Our init mounted the overlay at /run/kosem-root and started our daemon under a chroot there. Everything it spawned, dockerd included, inherited the chroot, so paths looked right. docker run worked because runc sets up a new container's root directly. docker exec is different: runc joins the running container's mount namespace with setns(2), and paths there resolve against the namespace's real root. Under a chroot, that was still the Pod's original root, with no /bin/sh and none of the container's files.
The fix is what runc itself does: pivot_root(".", ".") in a private mount namespace.[6]<sup>[7]</sup> A small helper starts in a new mount namespace, makes every mount private so nothing leaks back to init, pivots / onto the overlay, detaches the old root and exec s the daemon:
// Runs in a new mount namespace (CLONE_NEWNS). Errors elided.
unix.Mount("", "/", "", unix.MS_REC|unix.MS_PRIVATE, "")
old, _ := unix.Open("/", unix.O_DIRECTORY|unix.O_RDONLY|unix.O_CLOEXEC, 0)
unix.Chdir(newRoot)
unix.PivotRoot(".", ".") // the old root is now stacked on top at /
unix.Fchdir(old)
unix.Unmount(".", unix.MNT_DETACH) // detach the old root
unix.Chdir("/")
syscall.Exec(argv[0], argv, os.Environ())
Once the daemon and dockerd ran in a pivoted namespace, all of it worked. If the pivot fails, the environment refuses to start rather than run with Docker half broken.
Five smaller lessons from running it on GKE #
Fast I/O doesn't help if the agent waits minutes for a node. These are short here, and the details, with the YAML, are for a follow-up post.
- Hold a warm pool with placeholder Pods. A microVM on a node with room starts in seconds. One that needs a new node waits 2 to 4 minutes for GKE to bring up a nested-virtualization VM. We hold spare capacity with a Deployment of
sleep infinityPods at priority-10, sized like a default environment. A real environment preempts one at once, and the autoscaler adds a node for the displaced placeholder.-10is the floor: the autoscaler neither adds nor keeps nodes for Pods below it. Mark placeholderssafe-to-evict: "false", too, or the autoscaler removes the warm nodes as underused. - Name exact machine types. When we asked GKE for at least 8 vCPUs and 60 GB, it chose
c4-standard-16, which cost more for the same six environments. A ComputeClass that listsc4-highmem-8, thenn4,c4dandc3highmem-8, also gets past zonal stockouts. - Check Hyperdisk's default performance. Hyperdisk Balanced includes 3,000 IOPS and 140 MiB/s at no extra cost. A StorageClass that doesn't set them gets
6x + 3,000IOPS and1.5x + 140MiB/s for anxGiB disk, and the extra is billed on every disk, running or stopped.<sup>[8]</sup> That extra was over half our disk bill. Over 17 days, IOPS passed 3,000 in 0.44% of 5-minute windows. A StorageClass pinned to3000and140Micut disk cost by more than half. - Request a quarter of the CPU. Running environments averaged 3.1% of their vCPU: agents mostly wait on the model, then spike for
go testordocker build. With requests at half the limit, CPU filledc4-highmem-8nodes while memory sat unused. At a quarter, about six 2 vCPU, 8 GiB environments fit on a node, and memory is the limit. - Stop idle VMs sooner. Keeping environments up for 15 minutes after the last command made 40% of machine-hours idle. We now stop them 5 minutes after the agent's turn ends. A restart takes 4.7 s.
The short version #
- Keep hot I/O off virtiofs. Attach volumes as raw block devices and format them in the guest with
nodiscard. - Treat the root filesystem as hot I/O. Put a read-only EROFS image under an overlay on the block disk, with
index,metacopyandredirect_diroff. - Remember that guest page cache is host memory. Lower Kata's
default_memoryand give the RuntimeClass room for the hypervisor. - Enter a new root with
pivot_root, notchroot, if Docker runs inside. - Size for bursty, memory-bound work: warm placeholders, Hyperdisk at its included baseline and CPU requests at a quarter of the limit.
This stack runs every kosem environment today, and our own agents ran the benchmarks and test suites behind these fixes. The boundaries around the microVM, such as egress controls, credential brokering and audit logs, are on our security page.
Notes
- Google Cloud, Use nested VMs with GKE Standard clusters .↩
- Measured on GKE 1.37 Standard, a
c4-standard-8node and a 40 GiB Hyperdisk Balanced volume, two runs each, with the same Linux 6.6 tarball over virtiofs and over the raw block disk.↩ - Harbor Framework,
cobol-modernizationtask in Terminal-Bench 2.1 .↩ - Kata Containers issue #6533, Runtime classes seem to define insufficient overhead , on host VMM OOM kills when
default_memory(2048 MiB) exceeds the RuntimeClass's memory overhead under page-cache pressure.↩ - Kata Containers docs, Pod Annotations .↩
- Linux manual page,
pivot_root(2).↩ opencontainers/runcPR #1125,rootfs: make pivot_root not use a temporary directory .↩- Google Cloud, About Hyperdisk Balanced (default performance formulas and the included 3,000 IOPS and 140 MiB/s).↩