Why Docker isn't an AI-safe sandbox anymore, and how I fixed it in MindRoom 🔒

Security researchers and AI developers are questioning Docker containers as a security boundary after the security company DepthFirst used its own AI model to find a container escape, publishing the findings under the title "Containers Are No Longer a Security Boundary." The concern centers on the shrinking gap between closed security models such as Anthropic's Claude Mythos and OpenAI's Daybreak Blue, which are gated behind identity checks and vetting, and open-weight models that trail them by only six to seven months, per Epoch AI data cited in the post. The author, who runs untrusted AI agent code in Docker workers for the open-source MindRoom system on Matrix, responded by rethinking the setup, noting that "the code inside the container is now written by an AI, and the bugs that let it out are now found by an AI.

For years, a Docker container was my default answer to “where do I run this?” I never thought it was a perfect sandbox. It was good enough, because getting out of one usually meant finding a specific kind of bug in the Linux kernel itself, and those were hard to find. 1 fn:1 AI models have become very good at finding security bugs, including in Linux. Anthropic and OpenAI now keep their strongest security models, Claude Mythos https://www.anthropic.com/claude/mythos and Daybreak Blue https://openai.com/the-defense-factory/ , behind identity checks and vetting. Open-weight models, which anyone can download and run, come with no such rules, and they trail the closed ones by only six to seven months https://epoch.ai/data-insights/open-closed-cyber-eci-gap . Once there is an open model as good as Mythos, anyone can strip out its refusals with a tool like Heretic https://github.com/p-e-w/heretic people call this abliteration and point it at the Linux kernel. The security company DepthFirst used its own model to find a way out of a container and titled the write-up Containers Are No Longer a Security Boundary https://depthfirst.com/research/containers-are-no-longer-safe . I care about this because running untrusted code in containers is exactly what MindRoom https://www.nijho.lt/post/mindroom/ , my open-source system for AI agents that live in Matrix chat rooms, does. Every agent can get its own Docker worker, and its shell, file, Python, and browser tools run there. It can install anything, start background processes, and drive a real browser. And as I wrote in my ZFS post https://www.nijho.lt/post/btrfs-to-zfs/ , I run coding agents in YOLO mode, exclusively. I do not review the commands they run, I look at the results. So both sides changed at once. The code inside the container is now written by an AI, and the bugs that let it out are now found by an AI. A video from the Low Level channel https://youtu.be/9-2CSJC114k asking whether containers are still safe made me sit down and rethink my setup. I am not an expert in any of this. I have used containers, KVM, and hypervisors for years, including Proxmox before I moved to NixOS https://www.nijho.lt/post/proxmox-to-nixos/ , but I never deeply understood what happens underneath. Building this forced me to learn it, and writing this post is my exercise in understanding it all. All of this is very complicated as is everything new , and I am still learning. I am sure I skip over a lot and simplify the rest; what is here are the parts that helped me understand it. I thought it would be nice to share. When people talk about Linux, they usually mean an operating system like Ubuntu, Fedora, or NixOS. Those differ in almost everything you see: package managers, desktops, config files. What they all have in common is the kernel, the part Linus Torvalds started in 1991 and still leads today. Strictly speaking, that kernel is the only part called Linux. The kernel is the one program on a Linux machine that talks to the hardware directly. Everything else, your shell, Python, Chrome, Docker, runs on top of it and has to ask it for things. Open this file. Give me some memory. Send this packet. Start a new process. Those requests are called system calls, or syscalls for short. There is one kernel per machine, and it decides who is allowed to do what. That makes it the most trusted code on the system. Most kernel bugs only crash the machine or cause glitches. But some let a normal program become root, the admin account, or even run its own code inside the kernel. At that point it can do anything on the machine, including leaving any container it was in. When you docker run -it ubuntu , you get a root shell in what feels like a fresh computer. Under the hood it is one ordinary process. The kernel dresses it up using three features: There is no second kernel. Every syscall from inside a container lands in the same kernel as every other container and the host itself. The kernel enforces the isolation, and it is also what gets attacked. For most uses that is fine. Usually you put fixed software in a container: a database, a web app, some self-hosted service. You know where that code comes from, you keep your kernel patched, and that is good enough. An AI agent is different. It writes new code on the spot and runs it, and nobody knows in advance what that code will do. Since root in a container is root on the host, anyone who can use Docker can start a container as root and mount any folder from the host into it. They can then read and change every file in that folder, including files their own account may not open. Docker’s own documentation says the docker group grants root-level privileges https://docs.docker.com/engine/install/linux-postinstall/ . I knew that in theory, but it only clicked when I watched an AI agent do it on a machine where I had no root, only Docker access. It needed to change files it was not allowed to touch, so it started a small container, mounted their folder, and edited them from inside. No kernel bug needed. Kernel bugs that let a normal program take over the whole machine used to take skilled people weeks or months to find and exploit. The famous ones, like Dirty COW https://dirtycow.ninja in 2016, made the news every year or two, you patched, and that was it. AI is speeding this up, and the DepthFirst write-up I mentioned is a good example. Their model found a bug in the kernel code for Unix sockets, which programs on the same machine use to talk to each other. The kernel freed a piece of memory and then kept using it, a so-called use-after-free, and from inside a container that was enough to get out. Their conclusion is that you should assume attackers can escape containers at will. Blocking more syscalls, or taking more powers away from root, helps a little. But it only narrows which parts of the shared kernel a container can reach. This bug sat in Unix sockets, which almost every program uses, including the browser the agent drives, so blocking them would break the agent. Now put an AI agent in that container. It reads web pages and repositories written by strangers, and one of those could talk it into running an exploit. On a MindRoom server, getting out of a worker means getting onto the machine that runs MindRoom itself. That is where the API keys, Matrix logins, and stored credentials live, which the workers are built to never see. Before changing anything, I wanted to know how real this was for my own setup. I gave a MindRoom agent Daybreak Blue https://openai.com/the-defense-factory/ and asked it to escape its worker container using known kernel vulnerabilities. Daybreak Blue is OpenAI’s security model for verified users, so it does the offensive security work that other models refuse. It did not get out, which was reassuring. But it only tried known bugs. A kernel a few weeks behind on patches, or a bug nobody has published yet, would be a different story. So the attempt tells me the easy path was closed that day, and not much more. The classic answer is a virtual machine. A container shares the host’s kernel, but a VM brings its own, and this is where the hardware comes in. Modern CPUs have a virtualization mode built in, called VT-x on Intel and AMD-V on AMD. In that mode, a second kernel, the guest kernel, runs directly on the real CPU and believes it is in charge, while the CPU quietly limits it. When the guest tries something it may not do on its own, like talking to a disk or a network card, the CPU pauses the guest and hands control to the host. This is called a VM exit. Despite the name, nothing escapes and nothing crashes. The host handles the request, for example by pretending to be that disk, and then resumes the guest where it stopped. From the guest’s point of view, the instruction simply finished. Memory is fenced off too. The host gives the CPU a separate map of which real memory each guest may use, and the CPU checks every memory access against it. So even a guest kernel with full privileges cannot touch the host’s memory. The CPU only provides the mechanism. The software that sets the rules and handles the VM exits is called the hypervisor. “Hypervisor” is a role, not one program, and on Linux it is two pieces: /dev/kvm . Proxmox, which I mentioned above, is KVM plus QEMU. AWS Lambda is KVM plus Firecracker. Code in a VM talks to its own guest kernel. To reach the host, it usually has to break that kernel first and then the hypervisor.<sup 3 fn:3 </sup That is still possible. Google Project Zero turned a bug in how KVM runs VMs inside VMs on AMD https://googleprojectzero.blogspot.com/2021/06/an-epyc-escape-case-study-of-kvm.html into a full escape in 2021. Google’s kvmCTF https://security.googleblog.com/2024/06/virtual-escape-real-reward-introducing.html pays up to $250,000 for a new one. And this year, one researcher, Hyunwoo Kim, published three KVM escapes https://github.com/V4bel/Zapscape , one of them hidden in KVM since 2010. The difference between a container and a VM is how much code an attacker can reach, and that gap is large. On x86 the Intel and AMD chips in most PCs and servers , Linux has about 370 syscalls, and Docker’s default seccomp filter https://docs.docker.com/engine/security/seccomp/ blocks a few dozen of them. Behind the rest sit the kernel’s filesystems, networking, memory management, and core code: about 3.8 million lines, even without counting drivers. Not all of that is reachable from a container, but a lot of it is. KVM on x86 is about 114,000 lines. On top of KVM comes the VMM, but only the few devices it fakes for the guest. Most QEMU escapes came from its many emulated devices, which is why Firecracker and Cloud Hypervisor keep that list short. Google’s bug bounties point the same way. kernelCTF https://google.github.io/security-research/kernelctf/rules.html pays $71,337 for each new kernel exploit that turns a normal user into root, and Google’s public repository https://github.com/google/security-research/tree/master/pocs/linux/kernelctf holds exploits for 93 different kernel bugs. kvmCTF pays more than three times as much, and the same repository holds one KVM exploit, which crashes the host instead of escaping it. kvmCTF started a year later, but the gap is still big. Most of those kernel exploits would not work as written from a default container, because they first use a feature Docker blocks. Still, they show how many exploitable bugs sit behind the syscalls. Regular VMs are heavy. They boot a full operating system and tend to use a lot of memory. A microVM is a stripped-down VM meant to run one workload and start quickly. That still leaves a practical problem. MindRoom, like most tools, speaks Docker: it hands Docker an image and some folders to mount, and gets a running container back. A VM does not take a Docker image. Switching to VMs would normally mean building disk images, booting them, and wiring up files and networking myself. Docker does not start containers by itself, though. It takes care of images, networks, and volumes, and then hands the actual start to a small program called a runtime. The default runtime, runc , does what I described earlier: it starts a process on the host kernel with namespaces, cgroups, and seccomp. But you can tell Docker to use a different runtime. Kata Containers https://katacontainers.io is such a runtime, and it starts a microVM instead. It picks one of the VMMs above my test used QEMU , boots a small guest kernel, and starts the container inside it.<sup 4 fn:4 </sup Docker, and MindRoom, still see an ordinary container. gVisor https://gvisor.dev , from Google, takes a different route, and it is not a VM. It is a reimplementation of the Linux kernel, written in Go, that runs as a normal program on the host. Go manages memory for the programmer, so the use-after-free mistake behind the DepthFirst escape is much harder to make in gVisor. The worker’s syscalls go to gVisor instead of the host kernel, and gVisor itself only makes a small, filtered set of host syscalls. That is the hardening idea taken all the way: gVisor answers the agent’s syscalls itself, so the filter on the host can be much stricter than Docker’s. It needs no /dev/kvm , so it also runs inside cloud VMs that cannot start VMs of their own. My test machine, an ARM server with the kind of chip phones use running NixOS, had no /dev/kvm , which is why I tried gVisor first. Its runtime is called runsc , and it plugs into Docker the same way. Or as a table: | Isolation | What worker code talks to | Needs KVM | Usual escape route | |---|---|---|---| | Plain Docker runc | The host kernel, filtered by seccomp | No | A host kernel bug | | gVisor runsc | gVisor’s own kernel, written in Go | No | A gVisor bug, then a host kernel bug through its small filtered surface | | Kata Containers | Its own guest kernel in a microVM | Yes | A guest kernel bug, then a hypervisor bug | Each option also changes how much code on the host an escape can pick its bug from. The change is PR 2886 https://github.com/mindroom-ai/mindroom/pull/2886 , which is now merged. It adds one setting, MINDROOM DOCKER WORKER RUNTIME , and MindRoom passes its value to Docker as the runtime whenever it starts a worker. That is six lines of source code, plus 62 lines of tests and some docs. Changing the setting recreates each worker the next time it is used, and leaving it unset keeps everything as it was. For gVisor, setting it up is two steps: register the runtime in Docker’s /etc/docker/daemon.json and restart Docker, then point MindRoom at it. { "runtimes": { "runsc": { "path": "/usr/local/bin/runsc" } } } MINDROOM DOCKER WORKER RUNTIME=runsc Kata needs no entry in that file: once it is installed, MINDROOM DOCKER WORKER RUNTIME=io.containerd.kata.v2 is enough. The full setup, and a few catches for each runtime, are in the MindRoom docs https://docs.mindroom.chat/deployment/sandbox-proxy/ security-considerations . The six lines are short because everything else already existed. gVisor and Kata do the isolating. Docker already gives each worker its folders, its credentials, and its network, and it already knows how to hand a container to a different runtime. On the MindRoom side, the hard part happened months earlier, when agent tools moved out of the main process into dedicated workers with explicit mounts and credentials. Once every worker is a single call to Docker’s containers.run , swapping what runs underneath it is one argument. For comparison, Claude Opus estimated that building a microVM backend into MindRoom itself would take 2,500 to 4,000 lines. I tested both runtimes with a real agent in a Matrix room: gVisor on that ARM server, and Kata in a throwaway VM on my desktop. Shell, Python, git, background processes, and the browser all worked, without any MindRoom code change. My Kata numbers below come from a VM inside a VM, so bare-metal numbers will differ. Configuration is the cheap part: one environment variable, plus one entry in Docker’s config for gVisor. The other costs are the usual trade-offs between containers and VMs: /tmp lives in that RAM. The host only hands out memory the guest actually touches, so my idle worker used under 0.8 GiB of host memory. Raising the VM size costs little until a worker needs it. But a VM cannot grow past its size the way a container can, and memory the guest has used, for example to keep recently read files in its cache, mostly stays with the VM. A stronger sandbox protects the host. It does nothing about the tokens and network access you give the worker. An agent needs no kernel exploit to misuse a token that can push to your repositories, or to poke at every device on your home network. Scoped credentials and a firewall allowlist protect you at least as much, and MindRoom handles both separately: Agent Keys https://docs.mindroom.chat/deployment/egress-broker/ let agents use API keys without ever seeing them, and Approved Egress https://docs.mindroom.chat/deployment/approved-egress/ limits which servers a worker on Kubernetes can reach. gVisor has had its own bugs, and it still talks to the host kernel, just through a much smaller door. KVM has bugs too, as this year’s escapes show. Both walls make an escape far harder, and neither makes it impossible. Once I started reading about this, I kept running into people saying containers were never a security boundary. That is true, and for years I did not worry about it. With AI finding kernel bugs faster and agents running code nobody reads, I do now. No sandbox is perfectly safe, though. In a conversation with Dwarkesh Patel https://www.dwarkesh.com/p/noam-brown , Noam Brown doubted that even computers with no network connection at all would be enough to contain an AI. He pointed to studies https://arxiv.org/abs/1503.07919 where two such computers next to each other talk through heat: one runs its CPU hot, and the other picks up the change with its temperature sensor. Every option in this post trades some speed or memory for safety. How far to go depends on your own risk tolerance and how paranoid you are. MindRoom can now move its agents off the shared kernel with one environment variable. I do not use it myself yet, but if you are more paranoid than I am, it is there. The other way out is a bug in the program that starts the container. In 2024, CVE-2024-21626 https://github.com/opencontainers/runc/security/advisories/GHSA-xr7r-f8xq-vfvv in runc, the program Docker uses for that, leaked a handle to a host folder into new containers, so a malicious image could reach the host’s files without any kernel bug. ↩︎ fnref:1 Docker can map the container’s root to an ordinary user on the host with user namespace remapping https://docs.docker.com/engine/security/userns-remap/ , or run without root altogether in rootless mode https://docs.docker.com/engine/security/rootless/ , but neither is the default. Rootless mode would have stopped the agent in my Docker story, because Docker then has only my own rights. Remapping would not, because anyone who can use Docker can turn it off for a single container. Neither helps against a kernel bug, because code running inside the kernel is above every user. ↩︎ fnref:2 A bug in the VMM’s fake devices, or in other VMM code that handles the guest’s traffic, can skip the first step, because a normal program in the guest can feed that code data through the network or a shared folder. CVE-2020-8608 https://access.redhat.com/security/cve/cve-2020-8608 , a buffer overflow in QEMU’s user-mode networking, could be triggered that way with crafted network traffic. ↩︎ fnref:3 Like runc for Docker, Kata’s own programs on the host are a way out too. In 2020, one bug CVE-2020-2023 https://nvd.nist.gov/vuln/detail/CVE-2020-2023 let a container take over the guest without a kernel bug, and a second CVE-2020-2026 https://nvd.nist.gov/vuln/detail/CVE-2020-2026 then let the guest trick the Kata runtime on the host into mounting files over any host path. Neither needed a KVM bug. virtiofsd, the program that shares folders with the guest, also runs on the host and handles the guest’s file requests. ↩︎ fnref:4