10 agents, 30 GB of RAM and one queue: how they share without clashing

A developer released Sluice 0.12.2, a tool that funnels the tests, builds and heavy commands of up to 10 concurrent AI agents on a Linux machine into a single queue with a shared CPU and memory budget, running each check once per git tree. After 11 global out-of-memory kills and 7 systemd-oomd kills over 37 hours on a 30 GiB machine, version 0.10.0 moved enforcement into the kernel by writing each job's cgroup memory.max every second, and the next 387 jobs ran with no OOM and swap never above 36 percent. A later incident on the developer's desktop saw GNOME Shell hang and require a SysRq reboot, which the developer has not yet explained.

For the two weeks after I published Sluice 0.8.0, I ran it every day on my two machines, both with 30 GiB of RAM, with up to 10 AI agents working at once on one of them. Sluice https://github.com/josegarridodigio/sluice puts the tests, builds and heavy commands of all the AI agents on a Linux machine into one queue, with a shared CPU and memory budget, and runs each check once per git tree; I told how it started in the first article https://josegarrido.dev/en/sluice-shared-queue-so-your-agents-stop-repeating-tests/ . Even with that queue, my agents' test suites could still run a machine out of memory. How they share it now, in short: the queue decides who runs, the kernel's ceilings keep each job within its share, and the desktop guard keeps the desktop's share free. This is what I learned, and each lesson became a version, from 0.9.0 to 0.12.2: NODE OPTIONS go into the cache key. A week after publishing 0.8.0 I went through what had happened on my second machine: 12 threads, 30 GiB of RAM, 8 GiB of swap, and between 4 and 10 agents working at once. In 37 hours there were 11 global out-of-memory kills, 7 kills by systemd-oomd , 16 test jobs killed for memory and one agent session killed along with them. Swap was full every time. There were four causes: --filter or --test-concurrency=1 . That was a new command for Sluice, with no history, so it guessed 1.3–1.7 GiB for a suite that needs 11–14 GiB. systemd-oomd window was 5 s, and it ended an agent 6 s after a test job was killed. Version 0.9.0 changed the memory admission: sluice status now says where the reserve comes from, and flags a job that started alone beyond its estimate. No config change needed. With 0.9.0 the same machine hit the next wall. Between 30 October and 1 November one test suite ran 61 times, every time alone on the machine and beyond its estimate: 46 passed, 9 failed their tests and 6 were killed for memory. Each of those six kills was a global OOM, and before each one the system journal fell silent for 22, 30, 37, 67, 72 and about 150 seconds. The whole machine was thrashing, and systemd-oomd and Sluice's own guard were stalled with everything else: anything that runs in userspace arrives too late. Version 0.10.0 hands that job to the kernel. Every second the daemon writes the memory.max of each job's cgroup, so that all jobs together can never take the machine below a floor. The kernel enforces it even while the daemon is frozen. It is on by default with full containment: ~/.config/sluice/config.toml admission memory reserve = "auto" a tenth of the RAM: 3 GiB on 30 GiB memory ceiling = true memory floor = "auto" half the reserve: 1.5 GiB on 30 GiB sluice status shows each running job's ceiling. From 2 November, the next 387 jobs on that machine ran with no OOM and no memory kill, and swap never went above 36 %. The trade-off is deliberate: a run that used to pass by pushing the rest of the machine into swap is now killed inside its own job. The machine stays usable, and the message tells the agent to retry once memory is free, or to tell you that the gate doesn't fit. On 5 November it was my development machine's turn: 20 threads, 30 GiB of RAM, 8 GiB of swap, GNOME on Wayland, Sluice 0.10.0. Between 18:40 and 19:20 a frontend test suite of about 14.5–15 GiB started at three times its estimate, Playwright jobs took 7.7–9.4 GiB each, and a production build ran outside Sluice at the same time. The machine reached 21 GiB used 66 % , swap went from 3.6 to 5.2 GiB, swap-out peaked at 1,759 pages per second, load hit 32, and GNOME Shell's input lag reached 61–80 ms. There was no OOM and nothing was killed: the 0.10.0 ceiling never engaged. The shell ended up not responding, and at 20:37 I rebooted with SysRq. I still don't know what hung the shell: it stayed hung after the memory recovered. But looking into it I found something else. That day systemd had created 385 Chromium scopes on the machine, and that is where the browsers started by the test jobs had gone: outside their jobs. This is what happens. At startup, every Chromium browser process, headless included, prepares the connection to the XDG desktop portal. The portal needs an app id, and Chromium gets one by asking the systemd user manager, over the session bus, for a new scope with its own pid: StartTransientUnit . The process ends up in app.slice/app-org.chromium.Chromium-<pid .scope . There is no flag to turn it off. For a job runner that means the browser's memory leaves the job's accounting, its learned peak, its memory.max and its kill. And Playwright starts browsers in a process group of their own, so a process-group kill misses them too. I checked it on 6 November with a probe: a headless shell running for 6–8 seconds, about 200 MB. The browser process moved to its own scope; the renderers it had already forked stayed in the job. The cheap fixes didn't hold up: DBUS SESSION BUS ADDRESS only works for Chromium, and breaks the tests that need the session bus. FLATPAK SANDBOX DIR trick, which skips the move, stopped working in Chromium on 14 October. What works is a property of cgroup v2. To move a process, the kernel only asks for write access to the cgroup.procs of the common ancestor of both cgroups, and that ancestor is user@UID.service , which belongs to you. So since 0.11.0, every second the daemon walks each job's process tree and writes any descendant it finds outside back into the job's cgroup. This is a job on 0.12.1 that starts Playwright's headless Chromium 156 directly. The browser leaves its cgroup 265 ms after the job starts, and it's back in the same tick: INFO sluice gate daemon::server: moved a process of the job back into its cgroup job=1794597479890-504544-15a79e pid=643155 comm=chrome-headless from=/user.slice/user-$UID.slice/user@$UID.service/app.slice/app-org.chromium.Chromium-643155.scope When the job ends, whatever is left of it is killed, inside its cgroup or outside it: escapees through a pidfd, after checking the process start time, so a reused pid is never signalled. An integration test reproduces the escape with the same StartTransientUnit call. It is on by default containment reclaim escaped = true , and it applies to every job: the setting lives in your user config, not in each gate. If your tests deliberately start something with systemd-run --user --scope , or daemonise a process that must outlive the job, the daemon will pull those back too; for that case, set it to false and Sluice only reports escapes. What Sluice itself places, in its agents scope or in any slice below its own as when a test runs Sluice inside a job , is not an escape. Anything else, Chromium's scopes included, is moved back. The same version, 0.11.0, adds two more limits: admission swap ceiling = true swap floor = "auto" a quarter of the swap desktop guard = true memory.swap.max , so that the jobs together can't take the free swap below the floor. A job at its memory ceiling dies in its own cgroup instead of swapping the machine out. session.slice of the systemd user manager . While it is at or above 10 % some avg10 , until it drops under 5 % , no job starts, not even alone on the machine, and the ceilings keep the whole reserve free. Your jobs will wait while the desktop is under pressure; that's the point. sluice status shows each running job's ceiling and the session's pressure on its memory: line. Here with a single light job on an idle machine, so swap, pressure and escapes, which it only prints when they are not zero, don't show up: Also in 0.11.0: a variable such as NODE OPTIONS or RUSTFLAGS changes what a test checks without changing the tree. With key env its value goes into the cache key, and requests with other values never share a pass. sluice init writes them per toolchain, together with the file the package manager rewrites on every install. On a throwaway npm project: gates.test cmd = "npm test" weight = "auto" timeout = "30m" hooks = "pre-push" key inputs = "node --version", "npm --version" key env = "NODE OPTIONS", "NODE ENV" key files = "node modules/.package-lock.json" env = { CI = "1" } sluice doctor warns when a gate leaves them out: warn key-env gate 'test': key env leaves out RUSTFLAGS, RUSTDOCFLAGS, CARGO BUILD TARGET a change to them reuses a pass made with other values ; needs sluice = 0.11.0 to read key env Adding key env changes each gate's key once, so each gate runs once more. Every job descends from the daemon, and an agent session can start the daemon on its own. Since 0.12.0 the daemon doesn't keep the environment of whoever started it, so one agent session's id, API address and bearer token never reach anyone else's jobs. A check that asks "am I under an agent?" by walking up the process tree no longer mistakes your own runs for an agent's, and merge trains, which run their gates, their prepare and their key inputs in the daemon's environment, never see the token. The daemon keeps only an allowlist PATH , HOME , the locale, XDG , the session bus, SLUICE and a few more , and a sluice daemon started by hand re-executes itself clean. The log names the dropped variables that look like credentials, without showing their values. Gates started by run , verify or a hook still get the environment of whoever asked for them. The API keys your shell profile exports no longer reach the daemon: merge trains get them from your login shell the default tool env , and with queue tool env = "daemon" you have to name the ones they need, such as a proxy, in pass env . ~/.config/sluice/config.toml daemon pass env = "HTTPS PROXY" And a gate can drop variables from its own environment with env remove , for markers that env can't unset. sluice update It verifies the checksum, and it only stops the daemon if it is idle: one running gates keeps going until its next restart. If you use the Claude Code plugin, update it too, and then restart your agent sessions, which keep the old sluice mcp until they reconnect: claude plugin marketplace update sluice claude plugin update sluice@sluice Then check it: sluice doctor daemon-version and plugin both say 0.12.2, plus any key env warnings sluice status the daemon version, and the memory: line with the desktop's pressure 0.12.1 fixes the escape detection of 0.11.0 and 0.12.0. If you're on either and can't update yet, turn off the escape handling in your user config: ~/.config/sluice/config.toml containment reclaim escaped = false and restart the daemon while it is idle with sluice daemon stop it drops queued jobs . With false Sluice neither moves processes back nor kills leftovers at the end of a job; the detection still runs, so you may see a burst of WARN lines. Once updated, remove the line to get the containment back, unless your own tests need it. None of these versions breaks an existing config. The new keys are one-way: a shared .sluice.toml that uses key env needs every machine on 0.11.0 or later, and one that uses env remove , on 0.12.0. You need Linux. Containment needs cgroup v2 and a systemd user manager, and the memory policy needs systemd-oomd with systemd 257 or later; in a container or on cgroup v1 Sluice runs without containment, and sluice doctor tells you why. The memory work in this article comes from Ubuntu 26.04 machines with kernel 7.0. If you're new to it: on a 4-CPU GitHub runner, with 0.9.0 median of 3 rounds , Sluice cut the time until every agent had committed from 103 to 72 s with 4 agents −30 % and from 213 to 119 s with 8 −44 % . curl -fsSL https://raw.githubusercontent.com/josegarridodigio/sluice/main/scripts/install.sh | sh sluice doctor sluice init --dry-run sluice init The code, the CHANGELOG and the decision behind each change ADR-0026 to ADR-0030 are at github.com/josegarridodigio/sluice https://github.com/josegarridodigio/sluice . If it's useful to you, a star there helps other people find it. And if you want the next version when it ships, with what I learned running it, the newsletter is just below.