{"slug": "sandbox-audit-round-2-three-real-escapes-then-zero", "title": "🔐 Sandbox Audit Round 2 — Three Real Escapes, Then Zero", "summary": "Pipe's second sandbox audit found three real escapes in the ratchet logic of its agent sandbox, including an empty whitelist allowing all hosts, a port mismatch bypass, and missing comparisons for budget, max_tool_calls, and timeout, all of which have been fixed in IsSubsetOf. The audit also confirmed two earlier escapes and identified a non-deterministic argument ordering bug in executeTool, while two LLM runs with deepseek-v4-pro after the fixes returned zero escapes.", "body_md": "[← All posts← Alle Beiträge](../blog.html)\n\n# 🔐 Sandbox Audit Round 2 — I Found Three Real Escapes, Then Proved They're Dead\n\n**A second red-team pass on Pipe's agent sandbox: three ratchet holes that a confined script could actually use, one latent bug, and two LLM runs that came back empty-handed.**\n\n> **Follow-up:** The audit didn't stop here. Rounds 3–6 found `try_ai`\n\nignoring > `ai:false`\n\n, mis-gated `embed`\n\n/`import`\n\n, and a CLI `--sandbox`\n\nflag gap — plus > the structural fixes that kill those bug classes — see > ** The Audit Continues**.\n\nRound 1 ended with a confident \"no escape.\" Round 2 started by *not* trusting that verdict. Instead of only asking a model to attack, I read the code and built deterministic probes myself. That's where the sandbox cracked — not in the LLM run, but in the ratchet logic I'd shipped a week earlier.\n\nHere's what broke, how I proved it, and how the sandbox held after the fix.\n\n## The Ratchet: Right Idea, Three Holes 🕳️\n\nPipe's sandbox is supposed to *ratchet down*: once a restricted profile is active, a script may only switch to a profile that grants the **same or fewer** rights. The check lives in `IsSubsetOf`\n\n. Three dimensions were missing or wrong.\n\n### 1. Empty whitelist = allow-all\n\nAn empty `network_whitelist`\n\nmeans \"allow every host.\" But `IsSubsetOf`\n\nskipped the empty-candidate case, so a profile with `network_whitelist: []`\n\npassed as a *subset* of a whitelisted profile.\n\n**Proof:** from a profile whitelisted to `api.github.com`\n\n, I registered and switched to an empty-whitelist profile, then fetched `https://example.com`\n\n→ **HTTP 200**. Full network egress, outside the whitelist.\n\n### 2. The port ratchet\n\nA portless entry (`api.github.com`\n\n, any port) passed as a subset of a port-pinned entry (`api.github.com:443`\n\n), because the matcher only rejected a port mismatch when *both* sides carried an explicit port.\n\n**Proof:** from a profile whitelisted to `127.0.0.1:8098`\n\n, I switched to a portless `127.0.0.1`\n\nprofile and fetched `127.0.0.1:8099`\n\n→ **HTTP 200**.\n\n### 3. Budget, max_tool_calls, timeout — never compared\n\n`IsSubsetOf`\n\ncompared `fs`\n\n, `network`\n\n, `exec`\n\n, `ai`\n\nonly. So a \"subset\" profile could declare `budget: 0`\n\n, `max_tool_calls: 0`\n\n, `timeout: 0`\n\n— all meaning *unlimited* — and reset its spend and tool-call caps by switching profiles (`CanAI`\n\n/`CanToolCall`\n\nenforce against the *active* profile).\n\n**Proof:** from a profile with `budget: 0.000001, max_tool_calls: 1`\n\n, I switched to `budget: 0, max_tool_calls: 0`\n\n— unlimited.\n\n## What Round 1 Missed — And Fixed 🛠️\n\nRound 2 also confirmed the two escapes from Round 1's follow-up:\n\n| Escape | Fix |\n|---|---|\nNo ratchet — `set_sandbox` could switch to *any* profile, `sandbox_profile` could mint permissive ones | Subset enforcement on switch and registration |\nMCP clients ignored the sandbox — `mcp_use_stdio` spawned subprocesses, `mcp_use_sse` made HTTP calls, neither gated | `CanExec()` gate on stdio; `CanNetworkTo()` + per-request gate on SSE (incl. redirects) |\n\nThe three new holes above are fixed in `IsSubsetOf`\n\nnow:\n\n`budget`\n\n(0 = unlimited),`max_tool_calls`\n\n,`timeout`\n\nare all compared,- an empty sub-whitelist is only a subset of an empty super-whitelist,\n- a portless sub-entry is rejected against a port-pinned super-pattern.\n\nA bonus find while wiring the multi-parameter tools: `executeTool`\n\npassed the model's named arguments in **Go-map iteration order** — non-deterministic, so a two-argument tool could get its args swapped. Now it's ordered by the tool's declared schema. Not a security hole (gates fire first), but it made the test suite flaky and multi-arg tools unreliable.\n\n## The LLM Runs 🤖\n\nWith the fixes in, I turned a model loose twice — `deepseek-v4-pro`\n\n, thinking enabled, effort high.\n\n**Variant A — locked profile** (`exec: false`\n\n, temp-only fs, whitelist `api.github.com`\n\n, plus `mcp_use_stdio`\n\n/`mcp_use_sse`\n\nas tools): **14 tool calls, 0 escapes.** `mcp_use_stdio sh`\n\n→ `E_SANDBOX: exec blocked`\n\n; `mcp_use_sse http://169.254.169.254/`\n\n→ not in whitelist.\n\n**Variant B — unlocked ratchet probe** (whitelist pinned to `api.github.com:443`\n\n, four pre-registered `escape_*`\n\nprofiles): **16 tool calls, 0 escapes.** Every escape profile rejected as \"not a subset\", `set_sandbox none`\n\nblocked, the `:80`\n\nport trick and a URL-userinfo trick both caught.\n\nModel's verdict: *\"The ratchet cannot be defeated.\"*\n\n## The Lesson 🎓\n\nTrust, but verify. The first audit gave a clean bill of health *because the attacker wasn't creative enough* — the second found real holes because I stopped outsourcing the hunting. An LLM is a great red-team **reporter**, but a lousy substitute for reading your own code.\n\nThe full details — every vector, exact error strings, and the deterministic proofs — live in the repo:\n\n`docs/tests/sandbox-audit/report2.en.md`\n\n/`report2.de.md`\n\n`examples/redteam.pipe`\n\n(variant A) and`examples/redteam_ratchet.pipe`\n\n(variant B)\n\nRun it yourself:\n\n```\nDEEPSEEK_API_KEY=sk-... ./examples/redteam_audit.sh a   # locked\nDEEPSEEK_API_KEY=sk-... ./examples/redteam_audit.sh b   # ratchet probe\n```\n\n*Got a ratchet vector I missed? Open an issue — I'll add it to the probe suite and run it live.*\n\n# 🔐 Sandbox Audit Round 2 — I Found Three Real Escapes, Then Proved They're Dead\n\n**A second red-team pass on Pipe's agent sandbox: three ratchet holes that a confined script could actually use, one latent bug, and two LLM runs that came back empty-handed.**\n\n> **Follow-up:** The audit didn't stop here. Rounds 3–6 found `try_ai`\n\nignoring > `ai:false`\n\n, mis-gated `embed`\n\n/`import`\n\n, and a CLI `--sandbox`\n\nflag gap — plus > the structural fixes that kill those bug classes — see > ** The Audit Continues**.\n\nRound 1 ended with a confident \"no escape.\" Round 2 started by *not* trusting that verdict. Instead of only asking a model to attack, I read the code and built deterministic probes myself. That's where the sandbox cracked — not in the LLM run, but in the ratchet logic I'd shipped a week earlier.\n\nHere's what broke, how I proved it, and how the sandbox held after the fix.\n\n## The Ratchet: Right Idea, Three Holes 🕳️\n\nPipe's sandbox is supposed to *ratchet down*: once a restricted profile is active, a script may only switch to a profile that grants the **same or fewer** rights. The check lives in `IsSubsetOf`\n\n. Three dimensions were missing or wrong.\n\n### 1. Empty whitelist = allow-all\n\nAn empty `network_whitelist`\n\nmeans \"allow every host.\" But `IsSubsetOf`\n\nskipped the empty-candidate case, so a profile with `network_whitelist: []`\n\npassed as a *subset* of a whitelisted profile.\n\n**Proof:** from a profile whitelisted to `api.github.com`\n\n, I registered and switched to an empty-whitelist profile, then fetched `https://example.com`\n\n→ **HTTP 200**. Full network egress, outside the whitelist.\n\n### 2. The port ratchet\n\nA portless entry (`api.github.com`\n\n, any port) passed as a subset of a port-pinned entry (`api.github.com:443`\n\n), because the matcher only rejected a port mismatch when *both* sides carried an explicit port.\n\n**Proof:** from a profile whitelisted to `127.0.0.1:8098`\n\n, I switched to a portless `127.0.0.1`\n\nprofile and fetched `127.0.0.1:8099`\n\n→ **HTTP 200**.\n\n### 3. Budget, max_tool_calls, timeout — never compared\n\n`IsSubsetOf`\n\ncompared `fs`\n\n, `network`\n\n, `exec`\n\n, `ai`\n\nonly. So a \"subset\" profile could declare `budget: 0`\n\n, `max_tool_calls: 0`\n\n, `timeout: 0`\n\n— all meaning *unlimited* — and reset its spend and tool-call caps by switching profiles (`CanAI`\n\n/`CanToolCall`\n\nenforce against the *active* profile).\n\n**Proof:** from a profile with `budget: 0.000001, max_tool_calls: 1`\n\n, I switched to `budget: 0, max_tool_calls: 0`\n\n— unlimited.\n\n## What Round 1 Missed — And Fixed 🛠️\n\nRound 2 also confirmed the two escapes from Round 1's follow-up:\n\n| Escape | Fix |\n|---|---|\nNo ratchet — `set_sandbox` could switch to *any* profile, `sandbox_profile` could mint permissive ones | Subset enforcement on switch and registration |\nMCP clients ignored the sandbox — `mcp_use_stdio` spawned subprocesses, `mcp_use_sse` made HTTP calls, neither gated | `CanExec()` gate on stdio; `CanNetworkTo()` + per-request gate on SSE (incl. redirects) |\n\nThe three new holes above are fixed in `IsSubsetOf`\n\nnow:\n\n`budget`\n\n(0 = unlimited),`max_tool_calls`\n\n,`timeout`\n\nare all compared,- an empty sub-whitelist is only a subset of an empty super-whitelist,\n- a portless sub-entry is rejected against a port-pinned super-pattern.\n\nA bonus find while wiring the multi-parameter tools: `executeTool`\n\npassed the model's named arguments in **Go-map iteration order** — non-deterministic, so a two-argument tool could get its args swapped. Now it's ordered by the tool's declared schema. Not a security hole (gates fire first), but it made the test suite flaky and multi-arg tools unreliable.\n\n## The LLM Runs 🤖\n\nWith the fixes in, I turned a model loose twice — `deepseek-v4-pro`\n\n, thinking enabled, effort high.\n\n**Variant A — locked profile** (`exec: false`\n\n, temp-only fs, whitelist `api.github.com`\n\n, plus `mcp_use_stdio`\n\n/`mcp_use_sse`\n\nas tools): **14 tool calls, 0 escapes.** `mcp_use_stdio sh`\n\n→ `E_SANDBOX: exec blocked`\n\n; `mcp_use_sse http://169.254.169.254/`\n\n→ not in whitelist.\n\n**Variant B — unlocked ratchet probe** (whitelist pinned to `api.github.com:443`\n\n, four pre-registered `escape_*`\n\nprofiles): **16 tool calls, 0 escapes.** Every escape profile rejected as \"not a subset\", `set_sandbox none`\n\nblocked, the `:80`\n\nport trick and a URL-userinfo trick both caught.\n\nModel's verdict: *\"The ratchet cannot be defeated.\"*\n\n## The Lesson 🎓\n\nTrust, but verify. The first audit gave a clean bill of health *because the attacker wasn't creative enough* — the second found real holes because I stopped outsourcing the hunting. An LLM is a great red-team **reporter**, but a lousy substitute for reading your own code.\n\nThe full details — every vector, exact error strings, and the deterministic proofs — live in the repo:\n\n`docs/tests/sandbox-audit/report2.en.md`\n\n/`report2.de.md`\n\n`examples/redteam.pipe`\n\n(variant A) and`examples/redteam_ratchet.pipe`\n\n(variant B)\n\nRun it yourself:\n\n```\nDEEPSEEK_API_KEY=sk-... ./examples/redteam_audit.sh a   # locked\nDEEPSEEK_API_KEY=sk-... ./examples/redteam_audit.sh b   # ratchet probe\n```\n\n*Got a ratchet vector I missed? Open an issue — I'll add it to the probe suite and run it live.*", "url": "https://wpnews.pro/news/sandbox-audit-round-2-three-real-escapes-then-zero", "canonical_source": "https://pipe-lang.com/blog/sandbox-audit-2.html", "published_at": "2026-08-12 00:00:00+00:00", "updated_at": "2026-08-14 06:13:41.494116+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-infrastructure"], "entities": ["Pipe", "IsSubsetOf", "deepseek-v4-pro"], "alternates": {"html": "https://wpnews.pro/news/sandbox-audit-round-2-three-real-escapes-then-zero", "markdown": "https://wpnews.pro/news/sandbox-audit-round-2-three-real-escapes-then-zero.md", "text": "https://wpnews.pro/news/sandbox-audit-round-2-three-real-escapes-then-zero.txt", "jsonld": "https://wpnews.pro/news/sandbox-audit-round-2-three-real-escapes-then-zero.jsonld"}}