cd /news/ai-agents/making-microvms-fast-persistence-and… · home › topics › ai-agents › article
[ARTICLE · art-173414] src=kosem.ai ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

Making microvms fast, persistence and cheap on GKE with Kata and CH

Kosem cut Linux 6.6 kernel source unpacking from 126–129 seconds over virtiofs to 7.6–8.8 seconds by attaching Hyperdisk volumes as block devices over virtio-blk instead of sharing them via virtiofs, according to the company's engineering writeup on running Kata Containers and Cloud Hypervisor microVMs on GKE nodes with nested virtualization. Adding mke2fs flags (nodiscard, lazy_itable_init, lazy_journal_init, -m 0) cut formatting from 101 seconds to under a second, bringing cold boot on a fresh disk to 20 seconds and restart to 4.7 seconds, while stat on 20,000 files got 45× faster. The writeup also reports that the container root filesystem still runs over virtiofs, where one iteration of Terminal-Bench's cobol-modernization test loop took 106 ms in the microVM versus 9 ms on a laptop.

recommendation

For I/O-heavy microVM workloads, mount data disks as raw block devices over virtio-blk rather than virtiofs, and add nodiscard plus lazy init flags to mke2fs to skip zeroing already-clean disks.

more in playbook: Inference and self-hosting
by read11 min views2 publishedNov 20, 2026
Making microvms fast, persistence and cheap on GKE with Kata and CH
Image: source

Coding agents don't only edit files. They install system packages, run test suites and start Postgres, Redis and BuildKit with docker compose. That needs a real Linux kernel, and running untrusted code for many customers needs a hardware boundary around it.

So every kosem environment is a microVM: Kata Containers on Cloud Hypervisor, on GKE nodes with nested virtualization.<sup>[1]</sup> Each one boots its own Linux 6.6 kernel and runs a real dockerd.

Turning microVMs on is one setting on a GKE node pool. Making them feel like a laptop was the real work, and almost none of it was in the hypervisor. It was in the filesystem and in how memory is counted. Unpacking a kernel source tree went from over 2 minutes to 8 seconds, stat on 20,000 files got 45× faster, and a fresh environment boots in 20 seconds.

A word on isolation first, since Docker inside needs a privileged container. Inside a Kata microVM, that makes it root in its own guest kernel and nothing more: the hypervisor is the boundary with the host. A ValidatingAdmissionPolicy keeps it that way. It rejects any Pod in the namespace that isn't on our Kata RuntimeClass, or that asks for host namespaces, host ports or hostPath volumes. Pods get no service account token, and their egress reaches the public internet only: no private ranges and no metadata server.

Keep hot I/O off virtiofs #

By default, Kata shares a Pod's volumes into the guest over virtiofs. Kubelet formats and mounts the disk on the host, and the guest reaches it through virtiofsd. Every stat, open, unlink and readdir leaves the guest, crosses the VM boundary, runs in the host kernel and comes back. A git status on a large repository makes tens of thousands of those trips.

The fix is to give the guest the disk itself. We attach each Hyperdisk volume with volumeMode: Block rather than Filesystem, and Cloud Hypervisor exposes it to the guest over virtio-blk. Our PID 1 in the guest formats it as ext4 on first boot and mounts it. File operations now stay in the guest kernel.

Unpacking the Linux 6.6 source tree, about 80,000 files, took 126 to 129 s over virtiofs. On the raw block disk it takes 7.6 to 8.8 s, about 15× faster.<sup>[2]</sup> Docker ran with its defaults once /var/lib/docker lived on ext4.

Don't let mke2fs discard a fresh disk

The first cold boot on the new disk took over 2 minutes. 101 seconds of that was mke2fs issuing BLKDISCARD over every block of a new Hyperdisk, which Google's storage had already zeroed. Three flags fixed it:

mke2fs -q -F -t ext4 -L kosem -m 0 \
  -E lazy_itable_init=1,lazy_journal_init=1,nodiscard \
  /dev/kosem-disk

nodiscard skips the discard sweep. The lazy options let the guest kernel zero inode tables and the journal in the background after mount. Formatting now takes under a second at any size. -m 0 drops ext4's 5% reserve for root, which our unprivileged user can't use anyway: 1 GiB back on a 20 GiB disk.

A cold boot on a fresh disk now takes 20 s, and a restart that reattaches an existing disk takes 4.7 s. One more detail matters here: when PID 1 exits, the VM is gone, and so is any dirty page cache in it. Our init calls sync() and unmounts the disk before it exits.

The root filesystem is hot I/O too #

With /workspace and /var/lib/docker on ext4, big repository operations were fast. Short commands and test loops still dragged. Every binary in /usr/bin, every shared library, every Python import and every file in /tmp still came from the container's root filesystem, which Kata also shares over virtiofs.

Terminal-Bench's cobol-modernization task made it obvious.<sup>[3]</sup> One iteration of the agent's test loop took 106 ms in our microVM and 9 ms on a laptop. Microbenchmarks on our default Debian image showed where the time went:

Root filesystem benchmark Over virtiofs From local disk Speedup
stat 20,000 files 2.7 s 0.06 s 45×
500× create + unlink in /tmp 1.8 s 0.13 s 14×
500× /bin/true 2.2 s 0.7 s 3.1×
50× Python startup 1.9 s 0.6 s 3.2×

Even /bin/true was 3× slower, from the dynamic linker's lookups alone.

Our first fix copied the image's files onto the ext4 disk on first boot. That worked for a 600 MB image, and it took the VM down on a 3.8 GiB one (next section). What we run now packs each image once, when it's registered, into a single read-only EROFS file. A builder microVM runs mkfs.erofs inside the image itself, so registry credentials never enter a VM that reads a customer's image.

At boot, init loop-mounts /rootfs.erofs read-only and makes it the lower layer of an overlayfs. The upper layer lives on the ext4 disk. Every lookup goes to the guest kernel's EROFS driver, and nothing is copied at boot. Environments also behave like real machines: packages from apt, global toolchains and $HOME survive a stop, while /tmp and /run start clean.

On the agent's COBOL test, a full test case, running the COBOL binary and its Python port, now takes 31 ms, down from 106 ms. The COBOL binary on its own runs in 2.5 ms.

Two overlayfs details cost us time:

  • Turn off the features that tie an overlay to one host. Kata's guest kernel turns onindex ,metacopy andredirect_dir , which bind the upper layer to file handles in the lower one. Restart a stopped environment on another node and those handles break. We mount withindex=off,metacopy=off,redirect_dir=off , the same tradeoff Docker'soverlay2 makes.
  • Date warm caches in the far future. Our own dev image ships warm Go module and build caches, which cut the firstgo build ./... of our backend from 49.4 s to 7.5 s. But Go marks a cache entry as used by updating itsmtime , and withmetacopy off, even that copies the whole file up from the image to the disk. The first build copied up 2.2 GB. Setting every file in the cache to the year 2100 stops Go from touching or trimming them, so first builds stay fast and write almost nothing.

Guest page cache is host memory #

The copy-at-boot design is how we found this one. We booted Microsoft's devcontainers/universal image, 3.8 GiB compressed, in an 8 GiB environment. Within a minute, the node's kernel OOM-killed cloud-hypervisor, and the whole VM went with it. Kubernetes restarted the Pod, the copy started again, and the VM died again.

Two defaults combined:

  1. Kata gives the guest 2 GiB on top of the Pod's limit (default_memory = 2048 ).<sup>[4]</sup> An 8 GiB Pod boots a guest that sees 9.7 GiB.
  2. The host's cgroup for the Pod is its limit plus the RuntimeClass's overhead. GKE's built-in microVM RuntimeClass sets that overhead to130Mi and reconciles it, so you can't raise it in place. We already ran our own RuntimeClass with640Mi , and it still wasn't enough.

Copying gigabytes fills the guest's page cache, which is fine when the guest thinks it has 9.7 GiB. But on the host, the guest's page cache is real memory in the cloud-hypervisor process. At 8.8 GB it crossed the Pod's host cgroup, 8 GiB plus 640Mi, while the guest still saw almost a gigabyte free. The host killed the hypervisor before the guest's OOM killer ever ran.

The fix is to shrink Kata's extra guest memory to its minimum with a Pod annotation<sup>[5]</sup> and give the hypervisor real room on the host:

metadata:
  annotations:
    io.katacontainers.config.hypervisor.default_memory: "256"
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: machines
handler: kata-clh
overhead:
  podFixed:
    cpu: 250m
    memory: 1024Mi # 256 MiB guest extra + 768 MiB for cloud-hypervisor and virtiofsd

Kata allocates a guest's memory up front, so we stress-tested our largest size, 52 GiB, with the fix. Filling page cache and then anonymous memory made the guest's own OOM killer kill the process that was allocating. cloud-hypervisor and the environment stayed up. Packing images into EROFS also removed the multi-gigabyte copy that set this off.

chroot isn't enough when Docker runs inside #

This bug passed every docker run test and broke everything else. docker exec, docker compose exec, healthchecks, docker compose up --wait and nsenter all failed with executable file not found.

Our init mounted the overlay at /run/kosem-root and started our daemon under a chroot there. Everything it spawned, dockerd included, inherited the chroot, so paths looked right. docker run worked because runc sets up a new container's root directly. docker exec is different: runc joins the running container's mount namespace with setns(2), and paths there resolve against the namespace's real root. Under a chroot, that was still the Pod's original root, with no /bin/sh and none of the container's files.

The fix is what runc itself does: pivot_root(".", ".") in a private mount namespace.[6]<sup>[7]</sup> A small helper starts in a new mount namespace, makes every mount private so nothing leaks back to init, pivots / onto the overlay, detaches the old root and exec s the daemon:

// Runs in a new mount namespace (CLONE_NEWNS). Errors elided.
unix.Mount("", "/", "", unix.MS_REC|unix.MS_PRIVATE, "")
old, _ := unix.Open("/", unix.O_DIRECTORY|unix.O_RDONLY|unix.O_CLOEXEC, 0)
unix.Chdir(newRoot)
unix.PivotRoot(".", ".") // the old root is now stacked on top at /
unix.Fchdir(old)
unix.Unmount(".", unix.MNT_DETACH) // detach the old root
unix.Chdir("/")
syscall.Exec(argv[0], argv, os.Environ())

Once the daemon and dockerd ran in a pivoted namespace, all of it worked. If the pivot fails, the environment refuses to start rather than run with Docker half broken.

Five smaller lessons from running it on GKE #

Fast I/O doesn't help if the agent waits minutes for a node. These are short here, and the details, with the YAML, are for a follow-up post.

  • Hold a warm pool with placeholder Pods. A microVM on a node with room starts in seconds. One that needs a new node waits 2 to 4 minutes for GKE to bring up a nested-virtualization VM. We hold spare capacity with a Deployment ofsleep infinity Pods at priority-10 , sized like a default environment. A real environment preempts one at once, and the autoscaler adds a node for the displaced placeholder.-10 is the floor: the autoscaler neither adds nor keeps nodes for Pods below it. Mark placeholderssafe-to-evict: "false" , too, or the autoscaler removes the warm nodes as underused.
  • Name exact machine types. When we asked GKE for at least 8 vCPUs and 60 GB, it chosec4-standard-16 , which cost more for the same six environments. A ComputeClass that listsc4-highmem-8 , thenn4 ,c4d andc3 highmem-8, also gets past zonal stockouts.
  • Check Hyperdisk's default performance. Hyperdisk Balanced includes 3,000 IOPS and 140 MiB/s at no extra cost. A StorageClass that doesn't set them gets6x + 3,000 IOPS and1.5x + 140 MiB/s for anx GiB disk, and the extra is billed on every disk, running or stopped.<sup>[8]</sup> That extra was over half our disk bill. Over 17 days, IOPS passed 3,000 in 0.44% of 5-minute windows. A StorageClass pinned to3000 and140Mi cut disk cost by more than half.
  • Request a quarter of the CPU. Running environments averaged 3.1% of their vCPU: agents mostly wait on the model, then spike forgo test ordocker build . With requests at half the limit, CPU filledc4-highmem-8 nodes while memory sat unused. At a quarter, about six 2 vCPU, 8 GiB environments fit on a node, and memory is the limit.
  • Stop idle VMs sooner. Keeping environments up for 15 minutes after the last command made 40% of machine-hours idle. We now stop them 5 minutes after the agent's turn ends. A restart takes 4.7 s.

The short version #

  1. Keep hot I/O off virtiofs. Attach volumes as raw block devices and format them in the guest withnodiscard .
  2. Treat the root filesystem as hot I/O. Put a read-only EROFS image under an overlay on the block disk, withindex ,metacopy andredirect_dir off.
  3. Remember that guest page cache is host memory. Lower Kata'sdefault_memory and give the RuntimeClass room for the hypervisor.
  4. Enter a new root with pivot_root, not chroot, if Docker runs inside.
  5. Size for bursty, memory-bound work: warm placeholders, Hyperdisk at its included baseline and CPU requests at a quarter of the limit.

This stack runs every kosem environment today, and our own agents ran the benchmarks and test suites behind these fixes. The boundaries around the microVM, such as egress controls, credential brokering and audit logs, are on our security page.

Notes

  1. Google Cloud, Use nested VMs with GKE Standard clusters .↩
  2. Measured on GKE 1.37 Standard, a c4-standard-8 node and a 40 GiB Hyperdisk Balanced volume, two runs each, with the same Linux 6.6 tarball over virtiofs and over the raw block disk.↩
  3. Harbor Framework, cobol-modernization task in Terminal-Bench 2.1 .↩
  4. Kata Containers issue #6533, Runtime classes seem to define insufficient overhead , on host VMM OOM kills whendefault_memory (2048 MiB) exceeds the RuntimeClass's memory overhead under page-cache pressure.↩
  5. Kata Containers docs, Pod Annotations .↩
  6. Linux manual page, pivot_root(2) .↩
  7. opencontainers/runc PR #1125,rootfs: make pivot_root not use a temporary directory .↩
  8. Google Cloud, About Hyperdisk Balanced (default performance formulas and the included 3,000 IOPS and 140 MiB/s).↩
── more in #ai-agents 4 stories · sorted by recency
── more on @kosem 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/making-microvms-fast…] indexed:0 read:11min 2026-11-20 · —