{"slug": "how-we-got-microvms-booting-in-under-a-second", "title": "How we got microVMs booting in under a second", "summary": "Depot CI engineers reduced microVM cold boot times from 7-9 seconds to under 5 seconds by implementing direct kernel boot with a custom minimal Linux kernel, bypassing the standard Ubuntu image's unnecessary modules and drivers. System analysis revealed cloud-init as the primary bottleneck, consuming over 1.4 seconds of boot time, with other services like lvm2-monitor and networkd-dispatcher adding further delays. The optimizations are critical for Depot's just-in-time VM scheduler, which starts VMs only when build requests arrive without pre-warming or standby pools.", "body_md": "## Prologue\n\nWe built a just-in-time VM scheduler for [Depot CI](/products/ci), which means VMs only start when a build request actually arrives. As you might expect, fast boot times are essential to provide a stellar experience for our users. There's no pre-warming and no warm pool of standby VMs. Adding to the complexity, our short-lived sandboxes run on the same VMs that power our long-running builds. In this post, I'll walk you through the optimizations we layered on to get microVM cold starts down to where they are today.\n\n## Baseline and evaluation\n\nOur VMM of choice is Cloud Hypervisor (v51.1.0) running on KVM. The host is an i7i.metal-24xl bare metal instance running Debian 13. For the guest, we use the latest Ubuntu cloud image, converted to qcow2. This qcow2 image is attached to the VM as a disk (with Direct I/O) and serves as the boot device. A second disk, an ISO file, holds the cloud-init configuration. Inside the guest, a small agent receives and executes commands from the host over vsock. For our measurements, we'll consider a VM \"sufficiently booted\" once it can execute `/usr/bin/date`\n\n. By that point, the network is also up and running.\n\nTo conduct the measurements, we built a small helper that interacts with our VM agent. It starts an 8-core, 16GB RAM VM and runs `/usr/bin/date`\n\nagainst it. A measurement looks like this:\n\n``` bash\n$ time helper exec \"/usr/bin/date\"\nTue Apr 28 13:14:24 UTC 2026\n\nreal    0m8.026s\nuser    0m0.004s\nsys     0m0.002s\n```\n\nBy wrapping the helper in `time`\n\n, we get end-to-end boot time. Our unoptimized baseline lands between 7 and 9 seconds.\n\n## Direct kernel boot\n\nThe vanilla Ubuntu image is far from optimized. It loads kernel modules, drivers, and filesystems we don't strictly need, and each one costs us milliseconds at boot. Cloud Hypervisor supports direct kernel boot, which looks like an obvious win. All we need to do is compile a minimal kernel with only the modules we actually use. Cloud Hypervisor's [ kernel config](https://github.com/cloud-hypervisor/linux/blob/ch-6.12.8/arch/x86/configs/ch_defconfig) is a reasonable starting point, and we trim it down further from there. Passing the resulting kernel via the\n\n`--kernel`\n\nflag at VM start, let's see how much time we save:\n\n``` bash\n$ time helper exec \"/usr/bin/date\"\nTue Apr 28 13:20:02 UTC 2026\n\nreal    0m4.751s\nuser    0m0.010s\nsys     0m0.004s\n```\n\nAfter several runs, we can confidently say boot time now lands between 3 and 5 seconds.\n\n## Ask systemd for help\n\nTo dig into where boot time is actually spent, systemd ships with a few useful tools:\n\n```\n# prints a list of all running units, ordered by the time they took to initialize\nsystemd-analyze blame\n# prints a tree of the time-critical chain of units\nsystemd-analyze critical-chain\n```\n\nTogether, they expose the slowest units and processes during boot. For our VM, `systemd-analyze`\n\nreports:\n\n```\nStartup finished in 193ms (kernel) + 3.375s (userspace) = 3.568s\nThese are the top ones:\n systemd-analyze blame\n1.482s cloud-init-local.service\n 580ms cloud-config.service\n 352ms lvm2-monitor.service\n 348ms cloud-init.service\n 252ms dev-vda1.device\n 250ms cloud-final.service\n 138ms networkd-dispatcher.service\n 113ms ssh.service\n  92ms systemd-logind.service\n  78ms user@1000.service\n  75ms systemd-udevd.service\n  73ms keyboard-setup.service\n```\n\nLooking at critical chain paints a similar picture:\n\n```\ngraphical.target @2.676s\n└─multi-user.target @2.676s\n  └─networkd-dispatcher.service @2.537s +138ms\n    └─basic.target @2.509s\n      └─sockets.target @2.509s\n        └─uuidd.socket @2.509s\n          └─sysinit.target @2.507s\n            └─cloud-init.service @2.159s +348ms\n              └─cloud-init-local.service @662ms +1.482s\n                └─systemd-journald.socket @242ms\n                  └─system.slice @218ms\n                    └─-.slice @218ms\n```\n\nCloud-init is the biggest offender, and we'll deal with it in the next section. For now, let's focus on the other services.\n\n### systemd services\n\nWe aren't using LVM, so we can disable it and save 352ms:\n\n```\nsystemctl disable lvm2-monitor.service\nsystemctl mask lvm2-monitor.service\n```\n\nWe also don't need the network dispatcher, since we aren't relying on network hooks. That's another 138ms:\n\n```\nsystemctl disable networkd-dispatcher.service\n```\n\nWhile not applicable to this image, there are a few other systemd services worth disabling by default on our VMs:\n\n```\n# Multipath (for SAN storage, not needed in VMs)\nsystemctl disable multipathd.service\nsystemctl mask multipathd.service\n\n# Apport (Ubuntu crash reporting)\nsystemctl disable apport.service\nsystemctl mask apport.service\n\n# Grub (we boot with an external kernel)\nsystemctl disable grub-common.service\nsystemctl mask grub-common.service grub-initrd-fallback.service\n\n# e2scrub (ext4 online scrubbing, not needed for ephemeral VMs)\nsystemctl disable e2scrub_reap.service\nsystemctl mask e2scrub_all.timer e2scrub_reap.service\n\n# rsyslog (redundant when using journald)\nsystemctl disable rsyslog.service\n```\n\nNot strictly a systemd service, but since our VMs don't support IPv6, we can set\n\n```\ndhcp6: false\n```\n\nin the netplan config so the system doesn't wait for DHCPv6 during boot.\n\nWith these changes baked into our base image, let's rerun the VM:\n\n``` bash\n$ time helper exec \"/usr/bin/date\"\nTue Apr 28 13:45:54 UTC 2026\n\nreal    0m4.050s\nuser    0m0.091s\nsys     0m0.004s\n```\n\n## Optimizing cloud-init\n\nAfter a modest ~0.7 second reduction, cloud-init is still the major contributor to userspace boot time. The ISO has to be mounted, the configuration read, and the resulting actions applied. Let's see how much of that we can trim:\n\nFirst, we narrow the datasource list to just `NoCloud`\n\nso cloud-init doesn't iterate through every cloud provider trying to detect its environment:\n\n```\ndatasource_list: [ NoCloud ]\n```\n\nNext, we run `cloud-init analyze show`\n\n, which produces a detailed breakdown of module timings. Based on that, here are our optimization candidates:\n\n| Module | Time | Notes |\n|---|---|---|\nconfig-ssh | 1.28s | SSH host key generation |\nconfig-grub_dpkg | 0.2s | Unnecessary with external kernel |\n\nAs an extra, we can also remove the following configs from `/etc/cloud/cloud.cfg`\n\n:\n\n`growpart`\n\n`resizefs`\n\n`grub_dpkg`\n\n`apt_configure`\n\n`locale`\n\n`ssh_authkey_fingerprints`\n\nThe biggest win here is skipping host key generation on every VM boot. That said, I wouldn't recommend reusing the same host key across all VMs. A better approach is to maintain a pool of pre-generated unique host keys and assign one to each VM at boot, but that's an implementation detail for another post.\n\nLet's re-measure with the cloud-init optimizations in place:\n\n``` bash\n$ time helper exec \"/usr/bin/date\"\nTue Apr 28 13:56:02 UTC 2026\n\nreal    0m3.022s\nuser    0m0.001s\nsys     0m0.000s\n```\n\n## But… Do we really need cloud-init?\n\nOptimizing cloud-init bought us roughly a second, but it's still taking a considerable chunk of boot time. And really, all we need is a lightweight way to hand a small config to the guest-agent. So why not replace cloud-init entirely?\n\nEnter the [ Firmware Configuration Device](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/fw_cfg.md): a QEMU-compatible device that lets the host pass data directly to the guest operating system. It's a promising fit, though it takes a few steps to wire up.\n\nCloud Hypervisor needs to be rebuilt with this feature enabled:\n\n```\ncargo build --features fw_cfg\n```\n\nOur kernel config needs a new option:\n\n```\nCONFIG_FW_CFG_SYSFS=y\n```\n\nWith both in place, the guest can read the passed data directly from sysfs:\n\n```\n/sys/firmware/qemu_fw_cfg/by_name/opt/org.example/config/raw\n```\n\nWith cloud-init and its ISO disk out of the picture:\n\n``` bash\n$ time helper exec \"/usr/bin/date\"\nTue Apr 28 14:16:02 UTC 2026\n\nreal    0m2.294s\nuser    0m0.003s\nsys     0m0.001s\n```\n\n## Ditching systemd\n\nSystemd does a lot of useful things, but it also does plenty of things we don't actually need. We're building a stripped-down, highly optimized guest, not a general-purpose OS. So let's consider what it would take to replace systemd entirely:\n\n- Set up the rootfs\n- Mount the necessary devices\n- Mount any disks specified in the configuration\n- Configure the network\n- Reap zombie processes\n- Set the hostname\n- Start any services specified in the configuration (e.g. the guest-agent)\n- Anything else not strictly required for boot\n\nItems 1 through 5 are the bare minimum we need at boot, and the ones we can optimize most aggressively. So we're sold on replacing systemd, but how do we actually get our init binary into the VM?\n\n### To initramfs, or not to initramfs?\n\nWe have two obvious options:\n\n- Drop our init binary into\n`/sbin/init`\n\nin the guest - Use an initramfs, which is a single file we can hand to Cloud Hypervisor via a flag\n\nFor Depot CI, we went with the initramfs approach. It enables quick iterations and keeps init system deployment decoupled from the rest of the guest image.\n\n### Parallel initialization\n\nThe great thing about owning the init process is that we control exactly what runs and when. That opens the door to running boot steps in parallel where it makes sense, for example:\n\n- Setting the hostname\n- Starting the Docker daemon\n- Setting up swap\n- Starting sshd\n- Setting kernel parameters\n- And so on\n\nLet's see if ditching systemd was worth it:\n\n``` bash\n$ time helper exec \"/usr/bin/date\"\nTue Apr 29 12:14:02 UTC 2026\n\nreal    0m1.041s\nuser    0m0.002s\nsys     0m0.001s\n```\n\n## Low-hanging fruit\n\nThe numbers are starting to look good! Let's see if we can squeeze out a few more milliseconds here and there.\n\n### Kernel command line\n\nEven though we control the init process, the kernel itself still does a few things at boot that we can trim.\n\n#### Logs\n\nAdding the following options to the kernel cmdline silences boot output, saving a few milliseconds that would otherwise be spent writing to the console:\n\n```\nquiet loglevel=0\n```\n\n#### Time\n\nThese flags tell the kernel to use the KVM paravirtualized clock and skip a couple of timer sanity checks that would otherwise add overhead during boot:\n\n```\nclocksource=kvm-clock no_timer_check tsc=reliable\n```\n\n### Serial console\n\nWe can also disable the serial console entirely so the kernel doesn't waste cycles on output we never read.\n\nTime to check our boot numbers again:\n\n``` bash\n$ time helper exec \"/usr/bin/date\"\nTue Apr 29 14:47:02 UTC 2026\n\nreal    0m0.939s\nuser    0m0.004s\nsys     0m0.002s\n```\n\n### Hugepages and pre-allocate\n\nWe're now under the magical 1-second mark, but there's still room to push further.\n\nAllocating hugepages on the host and using them as backing memory for the VM can meaningfully reduce page faults and allocation overhead. We went with 1GB pages, and the benefit grows with the amount of memory the VM actually uses.\n\nBefore settling on hugepages, we experimented with Cloud Hypervisor's prefault option, which allocates physical memory and sets up page tables before the VM boots. It improves build performance but takes a real toll on boot speed, so we decided against it. Hugepages give us a better balance.\n\n``` bash\n$ time helper exec \"/usr/bin/date\"\nTue Apr 29 14:47:02 UTC 2026\n\nreal    0m0.789s\nuser    0m0.003s\nsys     0m0.001s\n```\n\n## vhost-user-blk is fun\n\nIn practice, boot times aren't constant. Our P50 sits around 0.6 seconds, while our P90 can climb to 1.2 seconds.\n\nSince we started supporting [snapshots](/docs/ci/how-to-guides/custom-images), we've had to rethink how we serve root disks at scale, moving beyond plain qcow2 files. We now store VM root disks and snapshots in the [Depot Registry](/docs/registry/overview) in an OCI-compatible way. Disk chunks are cached on the hosts in a multi-tier setup, and any missing chunks are served on-demand from the registry. This architecture has some impact on boot times today, but we have promising improvements in the pipeline that should let us strike a better balance between fast cold boots and storage pressure. That's a topic worth its own blog post.\n\nWe're also experimenting with VM memory snapshot and restore, which could push boot times down even further.\n\nThanks for following along to learn why starting a build on Depot CI feels so snappy. Until next time!\n\n## FAQ\n\n## What's the biggest single optimization for microVM boot times?\n\nDirect kernel boot, by a wide margin, going from 7 to 9 seconds down to 3 to 5 seconds. Cloud Hypervisor lets you pass a kernel directly via `--kernel`\n\n, and when you do, you can swap the stock Ubuntu kernel for one compiled from a minimal config. That cuts out drivers, modules, and filesystems you don't actually need, and each one was costing milliseconds at boot.\n\n## How do you use the QEMU firmware configuration device to replace cloud-init?\n\nCloud Hypervisor supports the same fw_cfg device that QEMU uses. You rebuild Cloud Hypervisor with `cargo build --features fw_cfg`\n\n, add `CONFIG_FW_CFG_SYSFS=y`\n\nto your kernel config, and the guest reads host-provided data from\n`/sys/firmware/qemu_fw_cfg/by_name/opt/org.example/config/raw`\n\n. The guest-agent reads its config from there on\nstartup, so cloud-init and its ISO disk are out of the picture entirely.\n\n## If you skip SSH host key generation during boot, how do you keep keys unique per VM?\n\nThe post touches on this: generating keys at boot is the slow part, but reusing the same key across all VMs is a bad idea. A better approach is to maintain a pool of pre-generated unique host keys and assign one to each VM at boot. It turns a boot-time problem into a provisioning problem, and the details are worth their own post.\n\n## Related posts\n\n[The differences between QEMU microvm and Cloud Hypervisor](/blog/differences-between-qemu-and-cloud-hypervisor)[Accelerating builds: Improving EC2 boot time from 4s to 2.8s](/blog/accelerating-builds-improve-ec2-boot-time)[Pulling containers faster with eStargz](/blog/booting-containers-faster-with-estargz)", "url": "https://wpnews.pro/news/how-we-got-microvms-booting-in-under-a-second", "canonical_source": "https://depot.dev/blog/optimizing-microvm-boot-times", "published_at": "2026-05-06 00:00:00+00:00", "updated_at": "2026-06-04 16:14:42.105475+00:00", "lang": "en", "topics": ["ai-infrastructure"], "entities": ["Depot CI", "Cloud Hypervisor", "KVM", "Ubuntu", "Debian"], "alternates": {"html": "https://wpnews.pro/news/how-we-got-microvms-booting-in-under-a-second", "markdown": "https://wpnews.pro/news/how-we-got-microvms-booting-in-under-a-second.md", "text": "https://wpnews.pro/news/how-we-got-microvms-booting-in-under-a-second.txt", "jsonld": "https://wpnews.pro/news/how-we-got-microvms-booting-in-under-a-second.jsonld"}}