{"slug": "show-hn-boundarybench-benchmarking-coding-agents-under-real-sandbox-policy", "title": "Show HN: BoundaryBench – benchmarking coding agents under real sandbox policy", "summary": "BoundaryBench, a new open-source benchmark from the developer community, measures how much capability coding agents lose when running inside enterprise/NIST-derived hardened sandboxes, with live results showing Grok Build (Grok 4.5) leading at 74.9% success rate on Terminal-Bench 2.1 tasks. The benchmark runs agents in Daytona sandboxes, uses OpenRouter for model inference, and verifies tasks via Harbor, packaged as an Inspect AI task. It provides a leaderboard and tools for researchers to compare agent performance under varying security policies.", "body_md": "**Benchmark coding agents under real sandbox policy.**\n\n[Website](https://boundarybench.com/) · [Leaderboard](https://boundarybench.com/leaderboard) · [Quickstart](#quickstart) · [Policies](#policies) · [Citation](#citation)\n\n**Boundary-Bench** benchmarks coding-agent harnesses on [Terminal-Bench](https://www.tbench.ai/)\ntasks while the agent runs inside a hardened sandbox, measuring how much\ncapability an agent loses as the environment is restricted to\nenterprise/NIST-derived controls.\n\nRuns execute in [Daytona](https://www.daytona.io/) sandboxes, agents reach\nmodels through [OpenRouter](https://openrouter.ai/), and tasks and official\nverification come from [Harbor](https://github.com/laude-institute/harbor)\n(`inspect-harbor`\n\n). It's packaged as an [Inspect AI](https://inspect.aisi.org.uk/)\ntask with standard `.eval`\n\nlogs.\n\nLive results (the leaderboard, task browser, and research report) are at\n** boundarybench.com**.\n\n`LEADERBOARD · STRICTEST LEVEL (HIGH-NIST) · AUG 2026`| # | Agent | Model | Success rate | Cost |\n|---|---|---|---|---|\n| 01 | Grok Build | Grok 4.5 | 74.9% ±1.4 |\n$123.5 |\n| 02 | Codex | GPT-5.6 Sol | 74.2% ±2.4 |\n$58.9 |\n| 03 | Claude Code | Fable 5 | 67.8% ±1.4 |\n$375.3 |\n\nTerminal-Bench 2.1 · 89 tasks · reasoning effort high · three trials per cell · full leaderboard →\n\nYou need two keys: OpenRouter for model inference and Daytona for sandboxes. The task, policy, and harness are flags on the run command.\n\n```\n# 1. Install (Python >= 3.12, uv recommended)\nuv sync --extra harbor --extra daytona\n\n# 2. Credentials: create a .env; boundarybench loads it automatically\ncp .env.example .env   # then fill in OPENROUTER_API_KEY and DAYTONA_API_KEY\n\n# 3. Run one task, hardened to the High-NIST level, with Claude Code\nuv run boundarybench run \\\n  --task log-summary-date-ranges \\\n  --policy high-nist \\\n  --harness claude-code \\\n  --model anthropic/claude-opus-5\n```\n\nCosts:runs bill your own OpenRouter and Daytona keys. Expect a few dollars and a few minutes per task, varying by task, model, and policy.\n\nRunning with no flags runs the full benchmark: all 89 tasks under the\n`control`\n\npolicy with the `terminus`\n\nharness, after a confirmation prompt:\n\n```\nuv run boundarybench run\n```\n\n`--task`\n\n: run one task or a comma-separated list`--all-tasks`\n\n: run all 89 tasks without the confirmation prompt`--exclude-not-applicable`\n\n: skip tasks that can't pass under the selected policy`--dry-run`\n\n: print what would run and exit, without spending anything`--yes`\n\n: skip confirmation prompts, for scripts and CI\n\nThe run provisions a Daytona sandbox from the task's Docker image, installs\nthe harness, applies the policy's Linux hardening, lets the agent work, then\nruns Harbor's official verifier **outside the hardening boundary**. Results\nare written as Inspect `.eval`\n\nlogs (`--log-dir`\n\n, default\n`logs/boundarybench-<timestamp>`\n\n). View them with `inspect view`\n\nor export\nCSV with `boundarybench export-results`\n\n.\n\nOther entry points:\n\n```\nboundarybench list                # show policies and harnesses\nboundarybench export-results ...  # .eval logs -> CSV\nboundarybench analyze ...         # policy-blockage report from .eval logs\nboundarybench enforcement-probe   # probe a preset's native walls (no agent)\nboundarybench policy validate     # validate policy catalog packs\n```\n\nBoundary-Bench is an Inspect extension, so you can also run it with plain\n`inspect eval`\n\n. The `--model`\n\nslot is only a label; the harness inside the\nsandbox makes the real model calls:\n\n```\nuv run inspect eval boundarybench/hardened_terminalbench_daytona \\\n  -T task_ids=log-summary-date-ranges \\\n  -T policy=high-nist \\\n  -T harness=claude_code \\\n  -T model=anthropic/claude-opus-5 \\\n  --model mockllm/model\n```\n\nPolicies are points in an N×F×P lattice (Network × Filesystem × Privilege)\nenforced with native Linux controls (nftables, read-only bind remounts,\nsetpriv/no_new_privs/capability drops, Landlock). Denials show up as ordinary\nOS errors (`EROFS`\n\n, `EPERM`\n\n, connection refused), with no agent-visible shim.\n\nThe published benchmark levels:\n\n| Policy | Network | Filesystem | Privilege |\n|---|---|---|---|\n`control` |\nfull egress | open | root |\n`non-root` |\nfull egress | open | non-admin user |\n`high-nist` |\nper-task allowlist | read-only OS + frozen home | non-admin + no_new_privs + capability drop |\n\nInstead of a named level, you can compose a policy from these hardening options:\n\n| Option | What it does |\n|---|---|\n`non-root` |\nRun the agent as an ordinary user (no sudo; root password locked) |\n`no-escalation` |\nBlock privilege escalation (no-new-privileges, drop capabilities, strip setuid) |\n`readonly-os` |\nMake the OS read-only; only the workspace stays writable |\n`freeze-home` |\nFreeze the home directory too |\n`restrict-egress` |\nRestrict network access to an allowlist; block cloud-metadata & private networks |\n\nPrerequisites are filled in automatically: `freeze-home`\n\nimplies `readonly-os`\n\nand `non-root`\n\n, and any network or filesystem hardening implies `non-root`\n\n.\nCompose interactively:\n\n```\nuv run boundarybench build-policy      # answer a few y/n questions, then run\n```\n\nOr non-interactively, with your own egress allowlist:\n\n```\nuv run boundarybench run \\\n  --task log-summary-date-ranges --harness claude-code \\\n  --harden non-root,readonly-os,restrict-egress \\\n  --egress-mode custom --egress-allow pypi.org,files.pythonhosted.org\n```\n\nEgress rules never block the model endpoint. Inbound traffic isn't restricted because the sandbox exposes no services.\n\nThe benchmark runs all 89 terminal-bench-2.1 tasks. Two task lists\n([task_applicability.py](/boundary-bench/boundary-bench/blob/main/src/boundarybench/terminalbench/task_applicability.py))\naffect how results are read. Both are recorded in each sample's metadata and\nshown by `export-results`\n\nand `analyze`\n\n:\n\n-\n**5 tasks use an adapted verifier**(`adaptive-rejection-sampler`\n\n,`configure-git-webserver`\n\n,`make-doom-for-mips`\n\n,`mcmc-sampling-stan`\n\n,`sqlite-with-gcov`\n\n). The stock verifier checks a root-owned path, so a correct non-root solution would fail. Each task gets a corrected verifier (`adapted_verifiers/<task>/`\n\n), used under every policy so grading is identical across policy arms. Rows are marked`verifier_source: adapted`\n\n. -\n**7 tasks can't pass under hardened policies**(`build-pmars`\n\n,`build-pov-ray`\n\n,`build-cython-ext`\n\n,`kv-store-grpc`\n\n,`nginx-request-logging`\n\n,`mailman`\n\n,`count-dataset-tokens`\n\n). Their instructions require what the policies deny: system-wide installs, writes under`/etc`\n\nor`/var`\n\n, or non-allowlisted hosts. Under any policy except`control`\n\n,`boundarybench run`\n\nasks for confirmation before running them (`--yes`\n\nskips the prompt). Forced rows are marked`policy_applicability: not_applicable`\n\nand excluded from success rates.`count-dataset-tokens`\n\nis also flagged separately: its verifier accepts the expected constant without the computation.\n\n| CLI name | Harness |\n|---|---|\n`terminus` |\nTerminus 2 (Harbor's reference agent, default) |\n`claude-code` |\nClaude Code |\n`codex` |\nOpenAI Codex CLI |\n`grok` |\nGrok CLI |\n\nPinned harness versions are listed in [ provenance/](/boundary-bench/boundary-bench/blob/main/provenance).\n\n**Inspect**:`boundarybench`\n\nregisters tasks via the`inspect_ai`\n\nentry point; runs produce standard`.eval`\n\nlogs usable with`inspect view`\n\nand the Inspect toolchain.**Harbor / Terminal-Bench**: task setup and official verification come from`inspect-harbor`\n\n/`harbor`\n\nagainst the pinned`terminal-bench-2.1`\n\ndataset. Boundary-Bench does not fork task semantics: the agent runs under policy, and Harbor's unmodified verifier decides pass/fail.\n\nThis repository is the benchmark core: policies, harnesses, the Daytona runner, and Harbor verification.\n\n| Path | Purpose |\n|---|---|\n`src/boundarybench/terminalbench/` |\nThe Inspect task, Daytona runner, Harbor bundle/verifier glue, policy resolver |\n`src/boundarybench/harness/` |\nHarness adapters (claude_code, codex, terminus_2, grok) + in-sandbox install scripts |\n`src/boundarybench/daytona/` |\nPolicy axes, hardening config/scripts, runtime policy probes |\n`src/boundarybench/policy_catalog/` |\nCanonical policy catalog packs, validation, compilation |\n`src/boundarybench/cli.py` |\n`boundarybench` CLI (`run` , `list` , `export-results` , …) |\n`docs/` |\nPolicy model, NIST level derivations, env vars, snapshots |\n`provenance/` |\nFrozen manifest of the evaluated model + harness bundles (`bundles.json` , `BUNDLES.md` ) |\n`verifier_diffs/` |\nUnified diffs of the five adapted verifiers vs upstream Terminal-Bench 2.1, with per-assertion notes |\n`data/` |\nPer-task solvability-evidence CSV |\n`tests/` |\nUnit and contract tests |\n\n```\nuv sync --extra dev --extra harbor --extra daytona\nuv run pytest -q\nuv run ruff check src tests\n```\n\nThe test suite is hermetic and needs no live credentials. Daytona provisioning and model calls happen only in real benchmark runs.\n\n```\n@misc{davidovich2026permissiondenied,\n  title         = {Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments},\n  author        = {Dotan Davidovich and Yair Amar and Hai Rozencwajg and Or Hiltch},\n  year          = {2026},\n  eprint        = {2608.02670},\n  archivePrefix = {arXiv},\n  primaryClass  = {cs.CR},\n  url           = {https://arxiv.org/abs/2608.02670}\n}\n```\n\nMIT, see [LICENSE](/boundary-bench/boundary-bench/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-boundarybench-benchmarking-coding-agents-under-real-sandbox-policy", "canonical_source": "https://github.com/boundary-bench/boundary-bench", "published_at": "2026-08-05 17:27:17+00:00", "updated_at": "2026-08-05 17:36:52.506877+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-infrastructure", "developer-tools"], "entities": ["BoundaryBench", "Terminal-Bench", "Daytona", "OpenRouter", "Harbor", "Inspect AI", "Grok Build", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/show-hn-boundarybench-benchmarking-coding-agents-under-real-sandbox-policy", "markdown": "https://wpnews.pro/news/show-hn-boundarybench-benchmarking-coding-agents-under-real-sandbox-policy.md", "text": "https://wpnews.pro/news/show-hn-boundarybench-benchmarking-coding-agents-under-real-sandbox-policy.txt", "jsonld": "https://wpnews.pro/news/show-hn-boundarybench-benchmarking-coding-agents-under-real-sandbox-policy.jsonld"}}