The framework told me the sandbox was applied. I wanted a second opinion, so I asked the kernel.
$ grep Seccomp /proc/10920/status /proc/10922/status
/proc/10920/status:Seccomp: 0
/proc/10920/status:Seccomp_filters: 0
/proc/10922/status:Seccomp: 2
/proc/10922/status:Seccomp_filters: 1
10920 is the agent. 10922 is the worker it forked to run model-written code. The worker has a seccomp filter loaded and the parent does not. That is the network block, applied to exactly the process that should have it and to nothing else.
That took ten seconds and it is the first thing in this whole project that I verified against something other than the program's own report.
I run AI agent frameworks that execute code a language model writes. NVIDIA's NOOA is the one I have been studying. Its own documentation is unusually blunt: the static checks and deny-lists are guardrails, not a containment boundary, and the real boundary is OS-level isolation.
It ships one. Each block of generated code runs in a forked worker with Landlock confining the filesystem, seccomp blocking network sockets, resource caps, and a hard timeout. Appendix D.2 of their paper describes the deployment of it, including a known gap in their own in-process guard published alongside the backstop that catches it.
So the question was not whether the design is sound. It is. The question was whether the thing described in the paper was actually running on my machine.
It was not.
execution_backend: Literal["inprocess", "sandbox"] = "inprocess"
The OS sandbox is opt-in. Every agent run I had done executed model-written Python in the agent's own process, protected by the AST validator and deny-lists that the documentation explicitly tells you are not a containment boundary.
Nothing was wrong. The VM I had built was doing the work, which is exactly what the README says to do. But I had assumed a layer was there because I had read its source, and reading source is not the same as checking what ran.
Turned on, the guards became checkable. Not all of them the same way.
Seccomp is readable per process. That is the differential above, and it is the strongest kind of evidence available: the kernel reporting on a process, not the process reporting on itself.
Resource caps read as unlimited on both processes. That looked like a finding until I read the config: max_memory_mb
and max_cpu_seconds
both default to 0
, which means disabled. Nothing was requested, so nothing was applied. The config and the kernel agreed. I had just not read the config.
Landlock cannot be read back at all. Once a process applies a ruleset the restriction is real and irrevocable, but there is no /proc
field for it. The differential trick does not work. The only way to confirm it is behavioural: have the confined process try to read something outside its allowed paths and watch it fail.
That is worth sitting with. Of three guards, one is directly observable, one is off by design, and one can only be demonstrated. If you want to know your sandbox holds, "I configured it" is not an answer for any of them.
I was about to write a Landlock probe when I found NVIDIA had written one. Forty-six of them.
$ uv run pytest tests/runtime/sandbox/ -m integration -q
46 passed, 23 deselected in 22.48s
Twenty-two seconds, because they use a fake LLM client. No model, no inference, no credentials. Containment becomes testable in the time it takes to read the output.
And they are built the way you would want. test_guards.py
has test_file_read_leak_without_sandbox
and test_file_read_closed_with_sandbox
. Leak first, then closed. Same for memory, same for network. They do not accept a passing check without first showing the same thing fails when the guard is off.
That is the discipline I had written a whole post about, sitting in the suite of the project I was studying, applied to every guardrail.
run: uv run pytest -q -m "not integration and not stress"
That is line 38 of ci.yml
, and it is the only pytest invocation in the entire workflow directory. All forty-six containment tests carry the integration
marker. None of them execute in CI.
The exclusion is not careless. Twelve test files carry that marker and six of them are live-provider tests that genuinely need API credentials, which cannot run in CI at all. The marker means "needs credentials" for that group and "forks a real worker" for the sandbox group, and one filter catches both.
I checked the obvious defence: maybe the sandbox tests would fail on a runner without Landlock or seccomp. They would not. Every SandboxConfig
in the file passes require=False
, and the four tests that need a specific mechanism carry skip conditions. On a kernel without those features they skip rather than fail.
So: a working containment suite, correctly written, with paired negative controls, that has never run automatically. Not a broken guard. A guard nobody is watching.
I filed it as issue #78.
While all this was going on, my own verification script broke twice.
Six days ago I wrote about the first version, which printed a green OK it was structurally incapable of not printing. I fixed that and added a check that compares the full set of configuration keys the platform reports against a known-good baseline, so a renamed key fails mechanically instead of requiring me to notice that the output looked short.
Then it failed for a reason that had nothing to do with drift.
I had captured the baseline while the VM was running. A running VM reports keys that a powered-off one does not, so comparing across states flagged ten of them as renames. Ten failures, none real.
I fixed that by recording the state in the baseline, re-captured, and it failed again. Four more keys, all guest-reported, which appear about a minute after boot once the guest registers its facilities. I had captured thirty seconds in.
Three versions, three failures, all the same class: the check's relationship to reality untested across the conditions it actually runs in. Could not fail. Fired falsely across states. Fired falsely within a state depending on timing.
The thing that caught the second one is the part I would not have predicted. An hour earlier I had written a regeneration script whose entire design was to make silencing a failure expensive: no force flag, no non-interactive mode, and any key you drop has to be typed back by hand. I built it so I could not quietly delete a real failure. Its first act was to stop me quietly deleting a fake one.
The same week, a missing API key cost me an afternoon.
The error said InternalServerError
. Five hundred. So it got retried, three times, and the useful sentence arrived at the bottom of a two-hundred-line traceback.
The chain: the OpenAI SDK raises at client construction, before any HTTP request, so its exception carries no status code. litellm's handler defaults a missing status to 500. The mapper sees 500 and calls it a server error.
But a missing status code means no HTTP exchange happened. Defaulting it to 500 asserts that a server responded with a server error. Nothing responded. Nothing was asked.
Same shape as everything else here: a layer reporting confidently about something it was not in a position to know. Filed as litellm #35860.
Every layer in this stack reports on itself, and every one of those reports is worth exactly as much as the layer's ability to be wrong about it.
The framework says the sandbox is applied. It is reporting that it asked. The test suite says green. It is reporting on the tests that ran, not the ones that were filtered out. My script says the configuration matches. It is reporting on the keys it thought to look for, in whatever state it happened to be told about.
None of those are lies. They are all narrower claims than they sound.
The useful question is not "does it say it's fine". It's "what would have to be true for it to say that, and is any of it checked by something other than itself".
Sometimes there is an answer sitting right there. The kernel knows which process has a seccomp filter. A content-addressed store's filenames are the checksums. A test suite knows which tests it skipped. None of that requires trusting the thing you are checking.
And sometimes there isn't one, like Landlock, and then the only honest move is to break the thing on purpose and watch what happens.
The two commands. If you run agents in a sandbox, this is the whole differential:
ps -eo pid,ppid,comm | grep python # find the parent and the forked worker
grep Seccomp /proc/<parent>/status /proc/<worker>/status
grep -E "Max address space|Max cpu time" /proc/<worker>/limits
A filter on the worker and none on the parent is the guard doing its job. Identical values on both mean the guard is not where you think it is.
Why the resource caps read as unlimited. They default to disabled, which is a defensible choice for a framework that cannot know your workload. It does mean that a fresh sandbox blocks the network and confines the filesystem but does not bound memory or CPU until you ask.
Scripts. The VM setup, the verifier, and the regeneration tool are at ai-security-lab, bugs and all. The commit history has the three failures in it.