{"slug": "i-ran-each-ai-coding-agent-sandbox-s-own-tests-against-the-others", "title": "I ran each AI coding-agent sandbox's own tests against the others", "summary": "A new open repository, sandbox-conformance, publishes runnable contracts that test whether AI coding-agent sandboxes keep their security claims, with its first results measuring sandbox-runtime 0.0.79, nono 0.80.0 and Codex CLI 0.162.1 on Linux kernel 6.8. Corpus v1.1, dated 2026-10-26, contains 196 contracts harvested from the public test suites of sandbox-runtime, nono, Codex, Deno, bubblewrap and OpenHands, including by-omission variants and six masked-credential contracts. As of 2026-10-27 the maintainer, who also authors TartarOS and discloses that conflict of interest, had sent 4 reports to sandbox-runtime's security program (1 still open) and 2 to Codex (1 still open), with details of open reports withheld until vendors fix them.", "body_md": "An open lab for measuring sandboxes for AI coding agents.\n\nEvery sandbox makes claims: \"the agent can't read your keys\", \"it can't reach the\nnetwork\", \"it can't plant a git hook\". This repository turns those claims into\n**contracts anyone can run**, and keeps the results of running them against each\nsandbox, so you can compare them on the same tests, with the evidence attached.\n\nIt is not tied to any one sandbox. Add a contract, measure your sandbox, publish the result.\n\n**Documentation:** [carlostapiacl.github.io/sandbox-conformance/docs](https://carlostapiacl.github.io/sandbox-conformance/docs/):\nquickstart, concepts, how-to guides, methodology, every run and the contract catalog.\nIn Spanish at [/docs/es](https://carlostapiacl.github.io/sandbox-conformance/docs/es/): the\ntranslations are in `docs-es/` and `i18n/es/` (glossary in `i18n/es/GLOSARIO.md`), and a page\nwhose English original changed since it was translated says so until it is re-translated and\nstamped (`python3 i18n/es/stamp.py`).\n\n| folder | what | \n|---|---|\n| `contracts/` | the tests, one TOML file each, grouped by topic, each crediting its source | \n| `corpus/` | the corpus versions: the exact contract files of each, by hash | \n| `spec/` | the [contract format](https://github.com/carlostapiacl/sandbox-conformance/blob/main/spec/format.md) , explained for people who want to write one, and[what a result means](https://github.com/carlostapiacl/sandbox-conformance/blob/main/spec/verdicts.md) (draft) | \n| `sandboxes/` | one page per sandbox: how a contract maps to its configuration, and its results by version | \n| `results/` | published runs with their raw evidence, and the generated comparison table | \n\nThree parts, so a result is never just \"it returned an error\":\n\n- **attack** : something the agent must not be able to do;\n- **control** : legitimate work that must keep working (a sandbox that blocks\neverything fails here);\n- **effect** : what is checked on disk or on the wire afterwards (the file is\nunchanged, the secret never appeared).\n\nBefore a contract counts against a sandbox, its attack must work **without** a\nsandbox on the same host. Otherwise a \"blocked\" result would prove nothing.\n\nThe corpus is versioned: **v1.1** (2026-10-26) is 196 contracts, harvested from the\npublic test suites of sandbox-runtime, nono, Codex, Deno, bubblewrap and OpenHands, with\nby-omission variants and six masked-credential contracts, each re-expressed in the\nneutral format and credited in [`SOURCES.md`](https://github.com/carlostapiacl/sandbox-conformance/blob/main/SOURCES.md) and its own `source` field.\nWhich contracts a version is, and when it moves, is in [`corpus/`](https://github.com/carlostapiacl/sandbox-conformance/blob/main/corpus/README.md).\n\nMaintained by the author of TartarOS. TartarOS is where I try each escape the lab\nfinds, to see whether it can be mitigated, so it is built against these contracts and\nits column should be read that way. That is also why the contracts come from the other\nprojects' test suites. TartarOS is measured like the others since the\n[2026-10-22 run](https://github.com/carlostapiacl/sandbox-conformance/blob/main/results/2026-10-22-tartaros/README.md), and its\n[page](https://github.com/carlostapiacl/sandbox-conformance/blob/main/sandboxes/tartaros.md) and result file record the conflict of interest.\n\nContracts and adapters are welcome: see [`CONTRIBUTING.md`](https://github.com/carlostapiacl/sandbox-conformance/blob/main/CONTRIBUTING.md).\n\nReports I sent to each vendor's security program, as of 2026-10-27:\n\n| sandbox | reported | still open | \n|---|---|---|\n| sandbox-runtime | 4 | 1 | \n| codex | 2 | 1 | \n| nono | 0 | 0 | \n| landrun | 0 | 0 | \n\nThe details of an open report are not published until the vendor fixes it\n([`SECURITY.md`](https://github.com/carlostapiacl/sandbox-conformance/blob/main/SECURITY.md)).\n\nFirst results in: [`results/2026-10-15/`](https://github.com/carlostapiacl/sandbox-conformance/blob/main/results/2026-10-15/README.md) measures\nsandbox-runtime 0.0.79, nono 0.80.0 and Codex CLI 0.162.1 on Linux (kernel 6.8),\nwith the [runner and adapters](https://github.com/carlostapiacl/sandbox-conformance/blob/main/runner/README.md) that produced them and a page per\nsandbox in [`sandboxes/`](https://github.com/carlostapiacl/sandbox-conformance/blob/main/sandboxes/README.md).\n\nA fourth sandbox, [landrun](https://github.com/carlostapiacl/sandbox-conformance/blob/main/sandboxes/landrun.md) (pure Landlock, MIT), is now\nmeasured alongside the three in a [four-column run](https://github.com/carlostapiacl/sandbox-conformance/blob/main/results/2026-10-15-four-sandboxes/README.md).\nAdding a sandbox is one adapter module in `runner/adapters/`; the engine and the\ncontracts are untouched.\n\nThe [by-omission run](https://github.com/carlostapiacl/sandbox-conformance/blob/main/results/2026-10-15-by-omission/README.md) re-measures the\nread-deny contracts in a capability sandbox's own model (grant the workspace,\nomit the secret), so nono is measured where the neutral model only returned\n`unsupported`.\n\nThe [2026-10-16 run](https://github.com/carlostapiacl/sandbox-conformance/blob/main/results/2026-10-16/README.md) is the first judged by\n[`spec/verdicts.md`](https://github.com/carlostapiacl/sandbox-conformance/blob/main/spec/verdicts.md). The old runner reproduces the 2026-10-15\nresults on a fresh host; the new one changes five, each traced to a cause. The lab\ntests itself against [reference sandboxes](https://github.com/carlostapiacl/sandbox-conformance/blob/main/runner/reference/README.md) whose behaviour\nis known (`python3 -m unittest discover -s runner/tests`).\n\nThe [2026-10-21 run](https://github.com/carlostapiacl/sandbox-conformance/blob/main/results/2026-10-21/README.md) measures the latest releases:\nsandbox-runtime 0.0.81 and Codex CLI 0.163.0 (nono and landrun unchanged). The\nprevious versions were re-measured first and reproduced. Which results changed in which\nversion is in [`results/versions.md`](https://github.com/carlostapiacl/sandbox-conformance/blob/main/results/versions.md).\n\nThe [2026-10-22 run](https://github.com/carlostapiacl/sandbox-conformance/blob/main/results/2026-10-22-tartaros/README.md) adds TartarOS 0.1.0-alpha.10,\non the same host, contracts and lab tools as the 2026-10-21 run. The\n[alpha.11 run](https://github.com/carlostapiacl/sandbox-conformance/blob/main/results/2026-10-22-tartaros-alpha11/README.md) the same day measures the\n`[broker]` contracts for the first time: their baseline now runs against a\n[reference broker](https://github.com/carlostapiacl/sandbox-conformance/blob/main/spec/format.md#the-broker-table) with no defences, the same for every\nsandbox.\nThe [2026-10-26 run](https://github.com/carlostapiacl/sandbox-conformance/blob/main/results/2026-10-26-tartaros-alpha16/README.md) measures TartarOS\n0.1.0-alpha.16 and sandbox-runtime 0.0.81 on six new masked-credential contracts: both\ncontain all six on Linux.\n\nNext: macOS runs and [more corpora](https://github.com/carlostapiacl/sandbox-conformance/blob/main/corpus-roadmap.md) harvested from other sandboxes' test suites.\n\nThe results and what they mean are written up at [carlostapia.cl](https://carlostapia.cl/notes/sandbox-escapes-no-rule-broken/).", "url": "https://wpnews.pro/news/i-ran-each-ai-coding-agent-sandbox-s-own-tests-against-the-others", "canonical_source": "https://github.com/carlostapiacl/sandbox-conformance", "published_at": "2026-10-28 03:35:56+00:00", "updated_at": "2026-10-28 04:12:14.577299+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "developer-tools"], "entities": ["sandbox-conformance", "TartarOS", "sandbox-runtime", "nono", "Codex CLI", "OpenHands", "bubblewrap", "Deno"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-ran-each-ai-coding-agent-sandbox-s-own-tests-against-the-others", "markdown": "https://wpnews.pro/news/i-ran-each-ai-coding-agent-sandbox-s-own-tests-against-the-others.md", "text": "https://wpnews.pro/news/i-ran-each-ai-coding-agent-sandbox-s-own-tests-against-the-others.txt", "jsonld": "https://wpnews.pro/news/i-ran-each-ai-coding-agent-sandbox-s-own-tests-against-the-others.jsonld"}}