Building a VMM from scratch, #0.5: Booting Linux in 190 lines of C

BoxLite published vmmlite, a 190-line C program that boots a stock Ubuntu Linux kernel on KVM to a PID 1 that prints one line, as a miniature reference for the VMM BoxLite is building to replace libkrun. The program skips BIOS and GRUB by following Linux's 64-bit boot protocol, copying the bzImage's 64-bit part to 1 MiB, placing struct boot_params at 0x7000, page tables at 0x9000 mapping the first 1 GiB to itself, and a four-entry GDT at 0x500 before jumping to RIP 0x100200; it exits with status 0 about three seconds after start. BoxLite states vmmlite is for learning only, does not validate input files, runs without a sandbox, and emulates only a serial port and a reset, while its Rust VMM's boot path already boots a real Linux kernel to an init on four vCPUs in PR #2052.

In 0 https://blog.boxlite.ai/building-a-vmm-from-scratch-0-why-boxlite-is-replacing-libkrun we set out to replace libkrun with a VMM of our own. Its boot path is landing one PR at a time, and its tests already boot a real Linux kernel to an init on four vCPUs 2052 https://github.com/boxlite-ai/boxlite/pull/2052 . Before the parts that walk through that Rust code, here is the whole path in miniature: vmmlite https://github.com/boxlite-ai/vmmlite , 190 lines of C, not counting comments, that boot a stock Ubuntu kernel on KVM to a PID 1 that prints one line. $ ./vmmlite bzImage initramfs.cpio 0.000000 Linux version 7.0.0-1014-gcp buildd@lcy02-amd64-091 ... ... 1.757880 Run /init as init process Hello from PID 1 1.766049 reboot: Restarting system About three seconds after it starts, vmmlite exits with status 0. It's C on purpose: one file, nothing between you and the ioctl calls. Our Rust VMM follows the same path, with more care. For learning only. vmmlite is not for production. It doesn't validate its input files, runs without a sandbox, and emulates only a serial port and a reset. To run code in a micro-VM, use a production VMM such as BoxLite https://github.com/boxlite-ai/boxlite . main is eight numbered sections. Four talk to KVM; the other four only write bytes into guest memory. Figure 1. The eight sections of vmmlite.c. Sections 1, 2, 7 and 8 call KVM; sections 3 to 6 only write bytes into guest RAM. KVM does the heavy lifting. It runs guest instructions on the real CPU and, after KVM CREATE IRQCHIP and KVM CREATE PIT2 , emulates the interrupt controllers and the timer inside the host kernel. What's left for vmmlite is the job of a PC's firmware and bootloader, plus two devices: a serial port and a reset line. Nothing runs in the guest before Linux: no BIOS, no GRUB. Linux's 64-bit boot protocol https://github.com/gregkh/linux/blob/v6.12.110/Documentation/arch/x86/boot.rst L1374-L1412 allows that, if the loader leaves the machine as they would have: | The protocol asks for | vmmlite | |---|---| | the kernel at its load address | the bzImage's 64-bit part, copied to 1 MiB 3 | | struct boot params , the "zero page" | at 0x7000: the setup header, command line, initramfs and E820 memory map 4 | | 64-bit mode, paging on, identity mapped | page tables at 0x9000 that map the first 1 GiB to itself 6 | | a GDT with flat BOOT CS 0x10 and BOOT DS 0x18 | four entries at 0x500 6 | | interrupts off, RSI at the zero page | RFLAGS 0x2, RSI 0x7000 7 | | a jump to the 64-bit entry, load address + 0x200 | RIP 0x100200 7 | A bzImage starts with a boot sector and 16-bit setup code, written for a BIOS. vmmlite checks the header and skips both: if memcmp k + 0x202, "HdrS", 4 || k 0x236 & 1 // the header's magic; XLF KERNEL 64 die "vmmlite: need a bzImage with a 64-bit entry\n" ; size t setup = k 0x1f1 ? k 0x1f1 : 4 + 1 512; // setup sects 0 means 4 + boot sector memcpy ram + KERNEL, k + setup, ksize - setup ; Figure 2 shows guest RAM when the vCPU starts. The addresses are the ones most VMMs we surveyed share, and the ones in BoxLite's VMM design https://github.com/boxlite-ai/boxlite/blob/main/docs/contributing/architecture/vmm/README.md boot-path . Figure 2. Guest RAM at the first instruction, not to scale. The arrows show how the CPU and the kernel find each piece: registers, zero page fields, and a scan for the MP table's signature. Long mode is a handful of control-register bits, set through KVM: s.cr3 = PML4; // the page tables s.cr4 |= 1 << 5; // CR4.PAE s.cr0 |= 1u << 31 | 1; // CR0.PG, CR0.PE: paging on s.efer |= 1 << 10 | 1 << 8; // EFER.LMA, EFER.LME: long mode IOCTL cpu, KVM SET SREGS, &s ; // apply struct kvm regs r = { .rip = KERNEL + 0x200, // the 64-bit entry point .rsi = ZERO PAGE, // boot protocol: RSI = the zero page .rflags = 2}; // interrupts off bit 1 is always set At 0x100200 sits startup 64 , the decompressor's 64-bit entry head 64.S https://github.com/gregkh/linux/blob/v7.0.14/arch/x86/boot/compressed/head 64.S L277-L278 . It unpacks the real kernel and jumps to it. The vCPU exits along the way, for CPUID or the first touch of a page, but KVM handles those exits itself. CPUID comes from KVM GET SUPPORTED CPUID , with two bits set in leaf 1, as Firecracker does https://github.com/firecracker-microvm/firecracker/blob/6ec271414300fb8f0ff1ff79ad321bb7467d072f/src/vmm/src/cpu config/x86 64/cpuid/normalize.rs L245-L251 : bit 31, "running under a hypervisor", and bit 24, the TSC-deadline timer that KVM emulates. Firmware also tells the kernel what it runs on: the CPUs, the interrupt controllers, and how interrupts are wired. Modern firmware uses ACPI tables. vmmlite writes the older, much smaller Intel MultiProcessor MP table: 240 bytes that list one CPU, an ISA bus, an IOAPIC at 0xFEC00000, and 16 routes from ISA IRQ n to IOAPIC pin n. Linux looks for the table's MP signature in a few fixed places, one of them the last KiB below 640 KiB mpparse.c https://github.com/gregkh/linux/blob/v7.0.14/arch/x86/kernel/mpparse.c L612-L614 . vmmlite writes it at 0x9FC00, as Firecracker, libkrun and our Rust VMM 2052 https://github.com/boxlite-ai/boxlite/pull/2052 do, and leaves that range out of the E820 map. Without the table or ACPI, Linux turns the IOAPIC and SMP off, as the design doc https://github.com/boxlite-ai/boxlite/blob/main/docs/contributing/architecture/vmm/README.md boot-path explains. The routes match how KVM wires its in-kernel chips by default: line n drives PIC input n and IOAPIC pin n irq comm.c https://github.com/gregkh/linux/blob/v6.12.110/arch/x86/kvm/irq comm.c L366-L389 . So IRQ 4 from the serial port lands where Linux expects it. Then vmmlite calls KVM RUN in a loop. It returns only when the guest does something KVM leaves to user space, and vmmlite answers three kinds of port I/O: } else if port = 0x3f8 && port <= 0x3ff { // COM1; an IN reads what com1 io = com1 vm, port & 7, run- io.direction, io ; // returns, an OUT ignores it } else if run- io.direction == KVM EXIT IO IN { // the guest reads another port: memset io, 0, run- io.size ; // it reads 0 } else if port == 0x64 && io == 0xfe { // keyboard controller reset: return 0; // the guest rebooted; done } // any other port write: ignored COM1 is a 16550 UART at ports 0x3F8 to 0x3FF. com1 emulates just enough of five registers for Linux's 8250 driver to find it 8250 port.c https://github.com/gregkh/linux/blob/v7.0.14/drivers/tty/serial/8250/8250 port.c L1104-L1126 and call it an original 8250. The kernel's own messages are the easy part. The console code polls the line status register before each byte 8250 port.c https://github.com/gregkh/linux/blob/v7.0.14/drivers/tty/serial/8250/8250 port.c L3223-L3229 , and com1 always answers 0x60, "transmitter empty". PID 1 is different. Its write lands in a tty buffer, and the driver drains that buffer from its interrupt handler: it turns on the "THR empty" interrupt 8250 port.c https://github.com/gregkh/linux/blob/v7.0.14/drivers/tty/serial/8250/8250 port.c L1416-L1430 and writes the next byte each time the UART reports it 8250 port.c https://github.com/gregkh/linux/blob/v7.0.14/drivers/tty/serial/8250/8250 port.c L1826-L1828 . So com1 keeps one bit of state, a pending "THR empty" event, and drives IRQ 4 from it: struct kvm irq level irq = {.irq = 4, .level = ier & 2 && thre}; IOCTL vm, KVM IRQ LINE, &irq ; // IRQ 4, up or down Figure 3. How "Hello from PID 1 " leaves the VM. Every port access is a VM exit that vmmlite answers; IRQ 4 tells the driver to send more. Each character costs at least three exits: IIR, LSR, then the byte. That's the price of emulating a device one register at a time, and why real VMMs move bulk I/O to virtio. The last exit is the reset. init.c is three calls: int main void { if write 1, "Hello from PID 1 \n", 18 < 0 return 1; // fd 1: the console, ttyS0 tcdrain 1 ; // wait until the UART has sent it all reboot RB AUTOBOOT ; // restart: Linux pulses the i8042 // reset line, and vmmlite exits } Linux tries an ACPI reset, finds no ACPI, and falls back to the keyboard controller: it writes 0xFE to port 0x64, which pulses the CPU's reset line reboot.c https://github.com/gregkh/linux/blob/v7.0.14/arch/x86/kernel/reboot.c L657-L668 . vmmlite takes that write as the end of the VM and returns 0. Here's the boot on a Google Cloud VM with nested KVM and Ubuntu's 7.0.0-1014-gcp kernel, trimmed to the lines vmmlite caused: | Kernel log, trimmed | Caused by | |---|---| | Command line: console=ttyS0 panic=-1 | 4: cmd line ptr | | BIOS-e820: mem 0x0000000000000000-0x000000000009fbff System RAM | 4: the first E820 entry | | BIOS-e820: gap 0x000000000009fc00-0x00000000000fffff | 4: the hole that hides the MP table | | Hypervisor detected: KVM | 7: CPUID leaf 1, bit 31 | | found SMP MP-table at mem 0x0009fc00-0x0009fc0f | 5 | | RAMDISK: mem 0x1ff40000-0x1fffffff | 4: the initramfs, at the top of RAM | | ACPI BIOS Error bug : A valid RSDP was not found | no ACPI tables, on purpose | | IOAPIC 0 : apic id 1, version 17, address 0xfec00000, GSI 0-23 | 5 for the ID and address; KVM's IOAPIC for version 0x11 and 24 pins | | TSC deadline timer available | 7: CPUID leaf 1, bit 24 | | serial8250: ttyS0 at I/O 0x3f8 irq = 4, base baud = 115200 is a 8250 | 8: com1 | | i8042: Can't read CTR while initializing i8042 | 8: unknown ports read 0 | | Hello from PID 1 | init.c, through com1 and IRQ 4 | | reboot: Restarting system | init.c's reboot , just before the write to port 0x64 | The keyboard controller line is the one vmmlite pays for. Every read of its ports returns 0, so Linux waits for a byte that never comes i8042.c https://github.com/gregkh/linux/blob/v7.0.14/drivers/input/serio/i8042.c L230-L238 : 0.73 seconds between i8042: Probing ports directly. and that error, out of 1.76 seconds to Run /init . You need x86 64 Linux with KVM and access to /dev/kvm , plus gcc with static glibc and cpio. vmmlite boots your host's own kernel: git clone https://github.com/boxlite-ai/vmmlite && cd vmmlite gcc -O2 -Wall -Wextra -o vmmlite vmmlite.c gcc -O2 -Wall -Wextra -static -o init init.c echo init | cpio -o -H newc initramfs.cpio sudo install -m 644 /boot/vmlinuz-$ uname -r bzImage ./vmmlite bzImage initramfs.cpio We've booted it on Intel with nested KVM on Google Cloud, and on AMD-V emulated by QEMU. The repo's CI boots a GitHub Actions runner's own kernel on every push. vmmlite is a map, not a VMM you should run. What it skips is what the rest of this series is about: | vmmlite | BoxLite's VMM | Part | |---|---|---| | one loop around KVM RUN ; an exit it doesn't expect ends the VM | one exit contract for KVM, Hypervisor.framework and, later, WHP | 2 | | no way to stop the vCPU from outside | pulling a vCPU out of the guest, safely | 3 | | CPUID copied from KVM, two bits set | registers, CPUID and MSRs on x86 | 4 | | whatever kernel the host has | a kernel we build ourselves | 5 | | a bzImage, on x86 64 only | the VMM as the bootloader, on arm64 and x86 64 | 6 | | COM1 and a reset; other ports read 0 | serial, RTC and reset first, then virtio | 7, 1012 | | KVM only | Hypervisor.framework, where the VMM decodes the exits | 8 | | a PID 1 that prints one line | our agent as PID 1 | 13 | | no sandbox, no hardening | guest input must never panic the host | 14 |