BareMetal OS running inside Firecracker microVMs with <1ms cold start

BareMetal OS now runs inside Firecracker microVMs with a cold start time under 1 millisecond, achieving ~700µs with network and disk enabled on an AMD Ryzen AI Max+ 395. The project enables near-instant-on, hardware-isolated microVMs with as little as 2MiB of RAM, supporting unikernel and monitor modes.

This repository contains the source code for BareMetal-Firecracker. This is a custom version of the BareMetal kernel explicitly for execution within a Firecracker microVM. The goal of this project was to achieve a <1ms cold start for the BareMetal kernel and its payload. That goal was achieved. The purpose of this project is to allow for near "instant on" hardware isolated microVMs to be provisioned with as little as 2MiB of RAM assigned to each. On an AMD Ryzen AI Max+ 395, running Ubuntu Desktop 26.04 with Firecracker v1.15.1, execution times are as follows: - Init: ~100µs from Firecracker handoff to kernel start. - BareMetal: ~700µs with network and disk enabled. ~500µs with only network enabled. This continues the work to support the "Hypervisor as the OS" philosphy as written here https://returninfinity.com/blog/hypervisos-as-data-centre-os . BareMetal https://github.com/ReturnInfinity/BareMetal , an exokernel written in x86-64 Assembly. Firecracker https://firecracker-microvm.github.io , a streamlined virtualization environment. src : Source code for BareMetal init and the BareMetal kernel payload : Payload for the kernel - Currently a minimal version of BareMetal Monitor scripts : Scripts for creating/removing bridge and tap networks img : Screenshot nasm, curl or wget, binutils, screen, firecracker see FIRECRACKER.md A tap0 network device is expected for microVM network connectivity. mkbr0.sh in the scripts directory will configure the bridge and tap. ./build.sh Running this will build the kernel and the monitor utility. The monitor will run on startup. ./build.sh payload.app Running this will configure "unikernel" mode. The app specified in the argument will run on startup. A couple examples are included: webserver.app - lwIP https://github.com/IanSeyler/BareMetal-lwip running as a DHCP client and web server hello.app - "Hello World" in Assembly ./baremetal.sh is the main script. It expects one of several arguments: start - Start the BareMetal VM status - Check if the VM is currently running send <text - Send a line of text to the VM serial console output --full - Print new VM serial console output --full for entire log attach - Attach to the interactive screen session stop - Gracefully shut down the VM Ctrl+Alt+Del Firecracker provides hardware isolated microVMs. What is missing from a "standard" VM: - No firmware BIOS/UEFI - No PCI/PCIe bus - No VGA or LFB - No USB - No HPET - Minimal ACPI What you get: - VirtIO devices block, net, and others addressable via MMIO - PS/2 keyboard controller only used for sending Ctrl-Alt-Del - Serial console Note: It is possible to enable a PCIe bus for Firecracker but it is not a default. Firecracker uses the following memory address on startup: | Start Address | Description | |---|---| | 0x000500 | GDT | | 0x000520 | IDT | | 0x006000 | PVH | | 0x007000 | boot params | | 0x008000 | Stack starts at 0x8FF0 | | 0x009000 | PML4 CR3 points here | | 0x00A000 | PDPTE | | 0x00B000 | PDE | | 0x020000 | cmd line | | 0x0E0000 | RSDP | | 0x100000 | your software | 0xC000 - 0xFFFF should be free Execution starts in 64-bit mode at 0x100000 . RFLAGS is set to 0x2 , RSP/RBP to 0x8FF0 , and RSI to the address of boot params . Init preps the system for the BareMetal Kernel. It sets the system up in a similar way to Pure64 https://github.com/ReturnInfinity/Pure64 . It is also written in Assembly. | Start Address | End Address | Size | Description | |---|---|---|---| | 0x0000000000000000 | 0x0000000000000FFF | 4 KiB | IDT - 256 descriptors each descriptor is 16 bytes | | 0x0000000000001000 | 0x0000000000001FFF | 4 KiB | GDT - 256 descriptors each descriptor is 16 bytes | | 0x0000000000002000 | 0x0000000000002FFF | 4 KiB | PML4 - 512 entries, entry 0 points to PDP at 0x3000, entry 256 points to PDP at 0x4000 | | 0x0000000000003000 | 0x0000000000003FFF | 4 KiB | PDP Low - 512 entries | | 0x0000000000004000 | 0x0000000000004FFF | 4 KiB | PDP High - 512 entries | | 0x0000000000005000 | 0x0000000000005FFF | 4 KiB | Init data | | 0x0000000000006000 | 0x0000000000006FFF | 4 KiB | Stack | | 0x0000000000007000 | 0x0000000000007FFF | 4 KiB | boot params | | 0x0000000000008000 | 0x000000000000FFFF | 32 KiB | Stub | | 0x0000000000010000 | 0x000000000001FFFF | 64 KiB | PD Low - Entries are 8 bytes per 2MiB page | | 0x0000000000020000 | 0x000000000005FFFF | 256 KiB | PD High - Entries are 8 bytes per 2MiB page | | 0x0000000000060000 | 0x000000000009FFFF | 256 KiB | Free | | 0x00000000000A0000 | 0x00000000000FFFFF | 384 KiB | Legacy BIOS ROM Area | | VGA RAM at 0xA0000 128 KiB Color text starts at 0xB8000 | ||| | Video BIOS at 0xC0000 64 KiB | ||| | Motherboard BIOS at F0000 64 KiB | ||| | 0x0000000000100000 | 0xFFFFFFFFFFFFFFFF | 1+ MiB | The software payload is loaded here | | Start Address | End Address | Size | Description | |---|---|---|---| | 0x0000000000005800 | 0x00000000000058FF | 256 B | MMIO devices | | 0x0000000000005900 | 0x00000000000059FF | 256 B | memmap | | 0x0000000000005A00 | 0x0000000000005AFF | 256 B | cmdline | The BareMetal kernel in this repo has been adapted from the general version. VirtIO drivers have been reworked to use MMIO. Virtio-Block and Virtio-Net drivers are present. Virtio-Vsock, and other Firecracker-supported devices, are yet to be added. SMP is not included in this version of BareMetal and will be added at a later date. BareMetal uses 2MiB of memory - A microVM should be provisioned with at least 4MiB of memory so 2MiB can be mapped at 0xFFFF800000000000 . 2MiB is the minimum if the application runs from kernel memory there is some room . The kernel binary actual code + data is currently ~5500 bytes. About 913408 bytes of padding is added to the end of the kernel so that the monitor utility is already at 0x1E0000 in memory. The monitor utility is also padded to be 131072 bytes in length so that any application will be at 0x200000 in memory which is mapped to 0xFFFF800000000000 . - parse ACPI tables for APIC IDs SMP removed from this version - when building in unikernel mode exclude monitor binary - re-org BareMetal memory usage to make more room for payloads when running with only 2MiB of RAM - combine build.sh into baremetal.sh //EOF