Running 10 million Codex CLIs on one server – LFI processes

wpnews.pro

Sitting on a toilet, I got a great idea - OpenAI spends billions on their servers; let's make them migrate to my infra instead. They have 9 million paying business users, so the business plan is:

Run 10 million Codex CLIs without using any external services
Our data center is a basement server with 16CPU/100GB RAM/1TB SSD
No changes to Codex source code
We can change dependencies - libc, tokio runtime, etc.
We can fork Rust and rebuild anything
We can choose OS/drivers and use our own kernel
The main goal is, of course, money - once we reach the target, we could rent that server for 29.99$ and become rich

An experienced engineer would immediately notice that, given the constraints, the task is impossible to solve. Luckily for us, we are missing the required experience, so that blocker is solved.

Learning is understanding #

What is heavier - light blue or false? - is our current mental model. We know little, and even things we know we've misunderstood. An LLM has absolutely destroyed half of my brain, and yet it is the best tool for learning. Throughout the article you'll see prompts which will help you to learn. Every answer from AI will make you only more confused, and confusion is at the core of learning - keep asking. The idea is not to replace reading papers with LLM talk, but to build enough understanding with LLM talk to be able to read papers.[1]

10 million processes #

Global warming will kill us all in the end - let's help it by doing a warmup.

I have a hunch that it might be a little difficult to run 10 million Codex, so let's start with just 10 million empty running processes. To ensure we understand how things work, let's create an absolutely minimal running process. An empty main wouldn't cut it - we don't want a process to simply exit. We need the minimal process that keeps running while consuming the least amount of resources.

loop {} - 16 CPUs divided by 10M gives us0,0000016CPU per Codex. Plenty, but I think we should start optimising CPU from the beginning, so no busy loop
thread::spawn(|| thread::sleep(Duration::MAX)) - a new thread consumes 48KB of memory; our enormous 100GB divided by 10M gives us a whopping 10KB, which is plenty for super efficient Codex, but too small for running additional threads, skip
std::thread::park() - ah, the perfection, look at that,sleep is 5 letters, whilepark is only 4 - 20% less RAM. For a simple Rustfn main() { park(); } , the produced binary is 300KB; to run it, the OS needs to load part of it into RAM, initialise kernel structs related to the new process, and then our app itself might allocate RAM for its own needs. In our example, total RAM usage is 1.73MB.

Can we do better? First, let's remind ourselves how an OS works:

$ codex "How OS runs my binaries?"
>Cloning https://github.com/torvalds/linux
>Enumerating objects: 11853083, done.
>Receiving objects: 0% (1/11853083)
^C^C
$ codex "How OS runs my binaries? SHORT. LIKE. CAVEMAN"
>Loads binary. Loads binary libs. Execute machine code.

Aha, let's try to remove all the dependencies and also ensure machine code is the smallest with manual Assembly. I remember I've seen [no_std] mentioned in Rust deps. Let's use that - it's probably telling Rust that we do something cool and nonstandard, so we'll get some respect from the compiler and also smaller RAM usage. The following code might look scary, but essentially it is hey oniichan kernel, please run syscall=73 and wait:

#![no_std]
#![no_main]

#[unsafe(naked)]
#[no_mangle]
pub extern "C" fn _start() -> ! {
    /* 
    The simplest way to block a process is to call the kernel syscall ppoll 
    mov loads a syscall number and the svc/syscall instruction makes a call
    */
    #[cfg(target_arch = "aarch64")]
    core::arch::naked_asm!(
        "mov x8, #73", // Essentially syscall_num=73
        "svc #0"       // Execute syscall
    );
    /*
    For the rest of the article, I'll focus mainly on ARM, but I want you to
    appreciate how explicit the instruction for calling a syscall on x86 is
    */
    #[cfg(target_arch = "x86_64")]
    core::arch::naked_asm!(
        "mov eax, 271", // syscall number is different, but same idea
        "syscall"
    );
}

#[panic_handler]
fn panic(_: &core::panic::PanicInfo) -> ! { 
    // Required even if not used
    unsafe { core::hint::unreachable_unchecked() }
}

[2] What is a syscall? Programs are actually free of side effects and surprisingly deterministic, and only by calling syscalls do they get access to the impure and stateful kernel and the outside world - without syscalls the only side effect our programs could have is heating up a PC box (Good ol' days as Haskell devs used to say before stupid monads came in and ruined all the fun). When an app runs, it constantly switches between userland (our app) and privileged kernel mode (processing syscalls), and of course the kernel might reject a syscall if it's not in a mood.

Compiling the no_std example gives us 70KB RAM usage (mostly from dropping the libc dependency) and the binary size is 968 bytes! That panic handler is annoying, though - we are quite chill and don't plan to panic even in such a restricted environment. Can we remove even that? We need an ABSOLUTE MINIMUM:

#![no_std]

#[cfg(target_arch = "aarch64")]
core::arch::global_asm!(
    ".global _start",
    "_start:",
    "mov x8, #73",
    "svc #0",
);

perfection_meme.jpg

Ah... RAM 70KB, binary is 720 bytes. Try to beat that, John Cena!

WAIT!

Wait, don't beat it yet! Super Mario Bros for NES was 40KB and used only 2KB of RAM. How come our essentially smallest-ever program consumes 70KB? People used to send things to space with less memory! Where did it all go?

Name	Size
Kernel page tables	28KB
Kernel stack	16KB
Kernel process info	16KB
Kernel shadow stack	4KB
Userspace RAM	4KB

Not sure who all those people are, but it's obvious now why NES decided not to use Linux - kernel process overhead! Even running absolutely empty programs as an OS process will consume almost 10x our RAM budget. In theory, we could tune things and recompile the kernel to reduce overhead to something like 45KB, which is still bigger than a whole Super Mario.

Like any other great startup, we need to pivot. In fact, OpenAI called me and asked me to change one thing. Well, it was Codex that told me to change it, but it's the same thing.

We are going to run 10M Codex inside... Minecraft.

Sandboxes #

Minecraft/Roblox/Terraria/KVM/gVisor/Docker/cat litter boxes - share a common trait of isolation: Game worlds, your binaries, and your cat's GC results are all isolated from the rest. There are two dimensions in here:

How well things are protected from each other
How much freedom you actually have inside such a cell

We can't run Codex without any isolation. OK, a process provides isolation, but it's rarely enough. Now let's go deeper into available sandbox solutions and see if I manage to pull off the cat's litter box analogy.

Virtual machines

VM - essentially Minecraft in Minecraft. Here is the beautiful visualisation of how Firecracker works https://youtu.be/-BP7DhHTU-I?t=27. We bring another kernel and our guests run there. Having such a setup provides the biggest isolation possible (hardware enforced), but if you didn't have enough resources with one kernel, then bringing the second one is not going to save much.

There is a beautiful concept of a unikernel (see this great example of using Unikraft), which dramatically reduces the amount of overhead of a client kernel, but it still ADDS things, while we need to REMOVE things. Very nice car - but we are looking for a bus.

gVisor #

Fun fact - gVisor started as a fork of Terraria; there are still some signs of it in the code base: https://github.com/google/gvisor/blob/d4d3e28027b4b6c94848a95bb7ada24bf82be84c/pkg/sentry/kernel/syslog.go#L62-L66

		"Preparing for the zombie uprising...",
		"Feeding the init monster...",
		"Creating cloned children...",
		"Daemonizing children...",

[3] How does it work? gVisor uses systrap - when you run your app under gVisor, any syscall from the app will be intercepted and processed by gVisor. Essentially they wrote a userland kernel in Go which is a damn achievement! gVisor provides a virtual file system, network and everything else for each running sandbox. All syscalls are guaranteed to be captured, so rm -rf / will have no effect on the host itself. It's an amazing project and recently got donated to CNCF which I hope will increase its adoption. You run it like:

$ sudo runsc do echo 'World'
>World

You're supposed to run one gVisor per process, so it's the same issue for us - another kernel, even if in userland, per process. Plus Go is a garbage... collecting language. Can we get programming languages flame wars back please? LLMs destroyed that vibe and now I'm forced to rage bait in the hope of hearing people who still care about our craft, sigh...

Containers #

Unix has supported running multiple processes from different users for a long time, and the OS provides a lot of isolation for them, but it's not considered enough. I cannot issue the reboot syscall if I don't have root permission; I cannot read another user's private key, but I can still see a lot on the host - I can see running processes, check available network devices, observe /dev/* files. Containers solved exactly that - the visibility.

When my process runs inside a container, then it still runs on the same machine, same kernel, same syscalls; the only difference is that it runs inside an illusion - the kernel gives it its own view of the file system (ls / will return files somewhere in /var/lib/docker/overlay2/<container_id>), its own virtual devices and network plus a few checks around, but it's still just a process sending syscalls to the same kernel. For our use case there are hardly any savings, plus in terms of isolation it's rarely trusted by the industry, so the tradeoff is not worth it for our case.

SFI

What was the antonym for the word security during the Trojan War? Historical records are scarce, but very likely it was άλογο which roughly translates to taking an untrusted package from the outside, bringing it home and leaving it with all the permissions. That is exactly what is happening when you run random untrusted code INSIDE your own process.

Before you say that you've never heard about SFI, let's talk about Cloudflare workers. Reading https://developers.cloudflare.com/workers/reference/security-model you'll see that clients' code runs in the same process together, WAT?! Workers are running clients' code inside a host process which runs the V8 engine that provides code isolation (it's literally called isolate in V8) and that is probably the most popular SFI implementation in the world now. V8 plus the harness around it is basically a userland OS (V8 is also a VM, but not that VM - hope it helps) - it intercepts all IO (yep, syscalls) from the clients' code, allows/disallows things, forwards to the actual host process (yep, kernel) to perform network or disk IO and manages other resource limits.

So Software Fault Isolation is exactly about bringing untrusted packages and running them all together within a host process. Why bother?

Isolation - kinda should work
Sandbox functionality - it depends, sometimes does not run Doom
Cost - cheap as... as... cat's litter boxes! Made it!

So it looks like SFI is the cheapest isolation mechanism in terms of resources and might be the solution for our two problems: process overhead and its isolation. We would simply run only one Linux process and a bunch of SFI sandboxes inside! Which SFI to choose? V8 is quite complex and I actually didn't know that Rolls-Royce was running on JS https://en.wikipedia.org/wiki/V8_engine

>The first V8 engine used in a car was the 1905 Rolls-Royce

I've asked Google (they are also involved) and, after reading a bunch of papers, journals, magazines and comics, I've finally found the one - LFI (french).

LFI #

V8 allows running untrusted JS inside your process by isolating it, and so LFI allows the same but with Assembly (yep, mov ax, bx thingy). Your beloved C++/Rust/Swift and Brainfuck (yay!) produce Assembly as part of an LLVM build, which then produces machine code which you have in the binaries that the CPU runs.

[4]OK, LFI is not a magical VM that runs Assembly; instead it rewrites Assembly and makes it safe:

It splits the process address space into virtual 4GB address ranges and then rewrites Assembly to ensure that all memory operations are guaranteed to happen within that range for each sandbox
It replaces all syscalls with calls to our provided address in the same process that will decide what to do with the syscall

[5] Because LFI works on Assembly it means we can't take existing binaries - we need to rebuild things which is fine for us. LFI is part of LLVM already https://llvm.org/docs/LFI.html, free as a bird, no subscription, no account and doesn't measure things in tokens.

Here is an example in Rust, its rough representation in Assembly and the result after the LFI rewrite. Spend some time trying to understand it - it's much easier than it looks. OK, your brain is likely on the same LLM drug as mine, so you can skip it. No judgement, only sad acceptance.

fn foo(ar: [i32; 2]) {
    // Memory access, read the first element
    let n = ar[0];           
    // syscall exit_group, number 94
    std::process::exit(n);   
}

That roughly produces the following Assembly:

; Read the first element of ar using the address in x0
ldr w0, [x0]   
; Load the exit_group syscall number, 94
mov x8, #94
; Call kernel syscall
svc #0

After LFI it produces the following

/* LFI rewrites all memory access addresses
x27 is always the LFI guest base, ex. 0x21..
Valid address 0x21.. stays the same 0x21..
An invalid address 0x31.. becomes 0x21..
In the end, read the same first element from ar */
ldr w0, [x27, w0, uxtw]      
; Same, load exit_group syscall number, 94 
mov x8, #94
; New, save the return address           
mov x26, x30
; New, load the runtime gate address
ldr x30, [x27]    
; New, call  runtime instead of kernel
blr x30     
; New, restore the return address
add x30, x27, w26, uxtw

Why did I burn 3 liters of tokens visualising something you successfully ignored in 3 seconds? Because that is the whole of LFI:

Add an additional check for all memory accesses and control flow by aligning access with the sandbox base from x27
Replace all syscalls with a jump to our provided function within a process

That's it, that's the whole genius of LFI: Once rewritten, our guests become isolated from each other and also pure! There is around 7% overhead in number crunching after the LFI rewrite, mostly from slowing memory reads. But besides that there is nothing else! (that's a lie) So let's run some benchmarks!

Let's run 100 separate processes. One process takes 70KB, so together they take 7MB of RAM. Booo, booo!
LFI runs 100 of our processes within the same LFI host process. RAM usage is 15MB. YES! YES! A WIN! A negative 8MB saving!

WAIT!

I lost... I lost?! Wait a minute, I'm not supposed to lose, let me see the script.

Where did all the RAM go?!

Part	RAM
LFI runtime futex tables	4.4MB
LFI runtime callback tables	3.2MB
LFI Host libraries	1.7MB
Kernel page tables	1.6MB
LFI runtime File-descriptor tables	1.2MB
LFI runtime Guest ELF headers/code/data	1.2MB
LFI runtime Working-directory buffers	405KB
LFI runtime Guest stacks	400KB
LFI runtime Syscall entry pages	400KB
LFI runtime Remaining state	336KB

Who are all those people?! So I've lied - for LFI sandboxes to work, we also need to use https://github.com/lfi-project/lfi-runtime/ (that is also a lie, hope my transparency makes things easier to follow): The runtime is responsible for guests into the host process. It provides a syscall catcher (gate). It provides some additional things on top like a virtual file system so guest1 shouldn't see files from guest2, and it also handles how guests actually run inside a process.

Like, think about it: we've got some guest binaries, and they are pure and isolated. Now what? We need to load them into memory, probably? OK cp /tmp/binary.exe [PID=1][Addr=0xFEE1DEAD00], done, what now? We have thousands if not millions of different pieces of guest code in our memory; how do we execute them? There are no threads, so if we just jump to the first guest and run it, then it might have while True {} and we are screwed. Should we create a thread per guest? We are screwed even harder. We have to recreate some of the kernel functionality in userland to schedule all the guests and that is what lfi-runtime does and that is where all the memory went.

The second lie was that LFI needs a runtime, but it doesn't NEED to be lfi-runtime - we can create our own, and considering the task, we were doomed to do it anyway. Just to see a theoretical limit, let's create AN ABSOLUTELY MINIMAL RUNTIME - fully tailored for our benchmark and for that benchmark ONLY - to run as many copies of our minimal process as possible. Take all possible shortcuts ever imaginable, just to pump the numbers as big as possible, to reach numbers that are not even imaginable with a general-purpose micro VM, gVisor or containers.

To follow what is happening and see that things are actually running, we would use slightly modified guest code, using assembly because we all should be Neo by now:

_start:
    mov x8, #172  ; syscall getpid() to get a PID from our runtime
    svc #0
    mov x8, #4096 ; syscall print, made up syscall number. Yep, we can do that
    svc #0
    mov x8, #73   ; syscall ppoll to block forever 
    svc #0

And our minimal runtime would provide an implementation for only 3 syscalls:

getpid - stub, return an index of a sandbox based on its address
print - forward text to stdout via a real syscall
ppoll - stub, make our runtime jump to the next guest.

That way, running guest1 will produce a side effect of printing 1 to stdout and 2 for guest2 and so on. Our runtime will run all of our guests, one by one, sequentially, yep, no threads, no stacks, no memory. Some say it's cheating - I say it's pre-IPO growth numbers.

And here goes the result - 1.2 billion sandboxes on a 10GB RAM VM and an estimated 12 billion sandboxes for 100GB RAM (Need to send that chart to Daytona just to keep them on their toes):

8 bytes per sandbox - it only contained a reference to the runtime syscall gate; all were created in 3.26s (387M/s). Running all guests took 291s (4.4M guests/s), while we spent 64% of the time in the kernel (print, yep). What does it prove? Absolutely nothing.

What does it imply though? That LFI plus a handcrafted LFI runtime tuned for our special case of running 10M Codex is the way. There are a few things left to figure out and we are done, specifically: how to run 10M guests at the same time; how to handle interruptions; custom scheduler; threads; how to reuse memory between guests; how to compress memory; snapshot and unload guests; unload memory to disk; implement a virtual file system; virtual networking; implement hundreds of syscalls in userland and finally make a few PRs to Codex so it will run normally in the common man's 2KB of RAM.

Warmup is done. Now we can actually start working on it.

References #

Ubuntu is 4GB - why do we even need that?! I have my own binary; can't I just run it without an OS and save 4GB? ↩ 2. Can I create a Rust toolchain which returns ENOSYS to all syscalls immediately? Would such binaries be guaranteed to be completely safe for my host system? What if I patch libc as well? ↩ 3. An app calling the write kernel syscall to save a file is risky because of kernel exploits. How does gVisor reduce the risk if, in the end, it will have to call thewrite syscall anyway to write data to disk, even if it wraps it with a bunch of additional checks and logic?↩ 4. VM for assembly - that sounded so bad... or is it? ↩ 5. LFI works on Assembly; why can't I disassemble a random binary like ls , run LFI on the assembly and pack it back?↩

source & further reading

kapteko.com — original article