BareMetal OS running inside Firecracker microVMs with <1ms cold start

This repository contains the source code for BareMetal-Firecracker. This is a custom version of the BareMetal kernel explicitly for execution within a Firecracker microVM.

The goal of this project was to achieve a <1ms cold start for the BareMetal kernel and its payload. That goal was achieved.

The purpose of this project is to allow for near "instant on" hardware isolated microVMs to be provisioned with as little as 2MiB of RAM assigned to each.

On an AMD Ryzen AI Max+ 395, running Ubuntu Desktop 26.04 with Firecracker v1.15.1, execution times are as follows:

• Init: ~100µs from Firecracker handoff to kernel start.
• BareMetal: ~700µs with network and disk enabled. ~500µs with only network enabled.

This continues the work to support the "Hypervisor as the OS" philosphy as written [here](https://returninfinity.com/blog/hypervisos-as-data-centre-os).

[BareMetal](https://github.com/ReturnInfinity/BareMetal), an exokernel written in x86-64 Assembly.[Firecracker](https://firecracker-microvm.github.io), a streamlined virtualization environment.

src

: Source code for BareMetal init and the BareMetal kernelpayload

: Payload for the kernel - Currently a minimal version of BareMetal Monitorscripts

: Scripts for creating/removing bridge and tap networksimg

: Screenshot

nasm, curl or wget, binutils, screen, firecracker (see FIRECRACKER.md)

A tap0

network device is expected for microVM network connectivity. mkbr0.sh

in the scripts directory will configure the bridge and tap.

./build.sh

Running this will build the kernel and the monitor utility. The monitor will run on startup.

./build.sh payload.app

Running this will configure "unikernel" mode. The app specified in the argument will run on startup. A couple examples are included:

webserver.app

-lwIPrunning as a DHCP client and web serverhello.app

• "Hello World" in Assembly

./baremetal.sh

is the main script. It expects one of several arguments:

start

• Start the BareMetal VMstatus

• Check if the VM is currently runningsend <text>

- Send a line of text to the VM serial console`output [--full]`

- Print new VM serial console output (--full for entire log)`attach`

• Attach to the interactive screen sessionstop

• Gracefully shut down the VM (Ctrl+Alt+Del) Firecracker provides hardware isolated microVMs.

What is missing from a "standard" VM:

• No firmware (BIOS/UEFI)
• No PCI/PCIe bus
• No VGA or LFB
• No USB
• No HPET
• Minimal ACPI

What you get:

• VirtIO devices (block, net, and others) addressable via MMIO
• PS/2 keyboard controller (only used for sending Ctrl-Alt-Del)
• Serial console

Note: It is possible to enable a PCIe bus for Firecracker but it is not a default.

Firecracker uses the following memory address on startup:

Start Address	Description
0x000500	GDT
0x000520	IDT
0x006000	PVH
0x007000	boot_params

| 0x008000 | Stack (starts at 0x8FF0) |
| 0x009000 | PML4 (CR3 points here) |

| 0x00A000 | PDPTE | | 0x00B000 | PDE | | 0x020000 | cmd_line | | 0x0E0000 | RSDP | | 0x100000 | your software |

0xC000

-0xFFFF

should be free

Execution starts in 64-bit mode at 0x100000

. RFLAGS is set to 0x2

, RSP/RBP to 0x8FF0

, and RSI to the address of boot_params

.

Init preps the system for the BareMetal Kernel. It sets the system up in a similar way to Pure64. It is also written in Assembly.

Start Address	End Address	Size	Description
0x0000000000000000	0x0000000000000FFF	4 KiB	IDT - 256 descriptors (each descriptor is 16 bytes)
0x0000000000001000	0x0000000000001FFF	4 KiB	GDT - 256 descriptors (each descriptor is 16 bytes)
0x0000000000002000	0x0000000000002FFF	4 KiB	PML4 - 512 entries, entry 0 points to PDP at 0x3000, entry 256 points to PDP at 0x4000
0x0000000000003000	0x0000000000003FFF	4 KiB	PDP Low - 512 entries
0x0000000000004000	0x0000000000004FFF	4 KiB	PDP High - 512 entries
0x0000000000005000	0x0000000000005FFF	4 KiB	Init data
0x0000000000006000	0x0000000000006FFF	4 KiB	Stack
0x0000000000007000	0x0000000000007FFF	4 KiB	boot_params
0x0000000000008000	0x000000000000FFFF	32 KiB	Stub
0x0000000000010000	0x000000000001FFFF	64 KiB	PD Low - Entries are 8 bytes per 2MiB page
0x0000000000020000	0x000000000005FFFF	256 KiB	PD High - Entries are 8 bytes per 2MiB page
0x0000000000060000	0x000000000009FFFF	256 KiB	Free
0x00000000000A0000	0x00000000000FFFFF	384 KiB	Legacy BIOS ROM Area
VGA RAM at 0xA0000 (128 KiB) Color text starts at 0xB8000
Video BIOS at 0xC0000 (64 KiB)
Motherboard BIOS at F0000 (64 KiB)
0x0000000000100000	0xFFFFFFFFFFFFFFFF	1+ MiB	The software payload is loaded here

Start Address	End Address	Size	Description
0x0000000000005800	0x00000000000058FF	256 B	MMIO devices
0x0000000000005900	0x00000000000059FF	256 B	memmap
0x0000000000005A00	0x0000000000005AFF	256 B	cmdline

The BareMetal kernel in this repo has been adapted from the general version. VirtIO drivers have been reworked to use MMIO.

Virtio-Block and Virtio-Net drivers are present. Virtio-Vsock, and other Firecracker-supported devices, are yet to be added.

SMP is not included in this version of BareMetal and will be added at a later date. BareMetal uses 2MiB of memory - A microVM should be provisioned with at least 4MiB of memory so 2MiB can be mapped at 0xFFFF800000000000

. 2MiB is the minimum if the application runs from kernel memory (there is some room).

The kernel binary (actual code + data) is currently ~5500 bytes. About 913408 bytes of padding is added to the end of the kernel so that the monitor utility is already at 0x1E0000 in memory. The monitor utility is also padded to be 131072 bytes in length so that any application will be at 0x200000 in memory (which is mapped to 0xFFFF800000000000).

• parse ACPI tables for APIC IDs (SMP removed from this version)
• when building in unikernel mode exclude monitor binary
• re-org BareMetal memory usage to make more room for payloads when running with only 2MiB of RAM
• combine build.sh into baremetal.sh

//EOF

source & further reading

github.com — original article