Show HN: BoundaryBench – benchmarking coding agents under real sandbox policy

BoundaryBench, a new open-source benchmark from the developer community, measures how much capability coding agents lose when running inside enterprise/NIST-derived hardened sandboxes, with live results showing Grok Build (Grok 4.5) leading at 74.9% success rate on Terminal-Bench 2.1 tasks. The benchmark runs agents in Daytona sandboxes, uses OpenRouter for model inference, and verifies tasks via Harbor, packaged as an Inspect AI task. It provides a leaderboard and tools for researchers to compare agent performance under varying security policies.

Benchmark coding agents under real sandbox policy. Website https://boundarybench.com/ · Leaderboard https://boundarybench.com/leaderboard · Quickstart quickstart · Policies policies · Citation citation Boundary-Bench benchmarks coding-agent harnesses on Terminal-Bench https://www.tbench.ai/ tasks while the agent runs inside a hardened sandbox, measuring how much capability an agent loses as the environment is restricted to enterprise/NIST-derived controls. Runs execute in Daytona https://www.daytona.io/ sandboxes, agents reach models through OpenRouter https://openrouter.ai/ , and tasks and official verification come from Harbor https://github.com/laude-institute/harbor inspect-harbor . It's packaged as an Inspect AI https://inspect.aisi.org.uk/ task with standard .eval logs. Live results the leaderboard, task browser, and research report are at boundarybench.com . LEADERBOARD · STRICTEST LEVEL HIGH-NIST · AUG 2026 | | Agent | Model | Success rate | Cost | |---|---|---|---|---| | 01 | Grok Build | Grok 4.5 | 74.9% ±1.4 | $123.5 | | 02 | Codex | GPT-5.6 Sol | 74.2% ±2.4 | $58.9 | | 03 | Claude Code | Fable 5 | 67.8% ±1.4 | $375.3 | Terminal-Bench 2.1 · 89 tasks · reasoning effort high · three trials per cell · full leaderboard → You need two keys: OpenRouter for model inference and Daytona for sandboxes. The task, policy, and harness are flags on the run command. 1. Install Python = 3.12, uv recommended uv sync --extra harbor --extra daytona 2. Credentials: create a .env; boundarybench loads it automatically cp .env.example .env then fill in OPENROUTER API KEY and DAYTONA API KEY 3. Run one task, hardened to the High-NIST level, with Claude Code uv run boundarybench run \ --task log-summary-date-ranges \ --policy high-nist \ --harness claude-code \ --model anthropic/claude-opus-5 Costs:runs bill your own OpenRouter and Daytona keys. Expect a few dollars and a few minutes per task, varying by task, model, and policy. Running with no flags runs the full benchmark: all 89 tasks under the control policy with the terminus harness, after a confirmation prompt: uv run boundarybench run --task : run one task or a comma-separated list --all-tasks : run all 89 tasks without the confirmation prompt --exclude-not-applicable : skip tasks that can't pass under the selected policy --dry-run : print what would run and exit, without spending anything --yes : skip confirmation prompts, for scripts and CI The run provisions a Daytona sandbox from the task's Docker image, installs the harness, applies the policy's Linux hardening, lets the agent work, then runs Harbor's official verifier outside the hardening boundary . Results are written as Inspect .eval logs --log-dir , default logs/boundarybench-<timestamp . View them with inspect view or export CSV with boundarybench export-results . Other entry points: boundarybench list show policies and harnesses boundarybench export-results ... .eval logs - CSV boundarybench analyze ... policy-blockage report from .eval logs boundarybench enforcement-probe probe a preset's native walls no agent boundarybench policy validate validate policy catalog packs Boundary-Bench is an Inspect extension, so you can also run it with plain inspect eval . The --model slot is only a label; the harness inside the sandbox makes the real model calls: uv run inspect eval boundarybench/hardened terminalbench daytona \ -T task ids=log-summary-date-ranges \ -T policy=high-nist \ -T harness=claude code \ -T model=anthropic/claude-opus-5 \ --model mockllm/model Policies are points in an N×F×P lattice Network × Filesystem × Privilege enforced with native Linux controls nftables, read-only bind remounts, setpriv/no new privs/capability drops, Landlock . Denials show up as ordinary OS errors EROFS , EPERM , connection refused , with no agent-visible shim. The published benchmark levels: | Policy | Network | Filesystem | Privilege | |---|---|---|---| control | full egress | open | root | non-root | full egress | open | non-admin user | high-nist | per-task allowlist | read-only OS + frozen home | non-admin + no new privs + capability drop | Instead of a named level, you can compose a policy from these hardening options: | Option | What it does | |---|---| non-root | Run the agent as an ordinary user no sudo; root password locked | no-escalation | Block privilege escalation no-new-privileges, drop capabilities, strip setuid | readonly-os | Make the OS read-only; only the workspace stays writable | freeze-home | Freeze the home directory too | restrict-egress | Restrict network access to an allowlist; block cloud-metadata & private networks | Prerequisites are filled in automatically: freeze-home implies readonly-os and non-root , and any network or filesystem hardening implies non-root . Compose interactively: uv run boundarybench build-policy answer a few y/n questions, then run Or non-interactively, with your own egress allowlist: uv run boundarybench run \ --task log-summary-date-ranges --harness claude-code \ --harden non-root,readonly-os,restrict-egress \ --egress-mode custom --egress-allow pypi.org,files.pythonhosted.org Egress rules never block the model endpoint. Inbound traffic isn't restricted because the sandbox exposes no services. The benchmark runs all 89 terminal-bench-2.1 tasks. Two task lists task applicability.py /boundary-bench/boundary-bench/blob/main/src/boundarybench/terminalbench/task applicability.py affect how results are read. Both are recorded in each sample's metadata and shown by export-results and analyze : - 5 tasks use an adapted verifier adaptive-rejection-sampler , configure-git-webserver , make-doom-for-mips , mcmc-sampling-stan , sqlite-with-gcov . The stock verifier checks a root-owned path, so a correct non-root solution would fail. Each task gets a corrected verifier adapted verifiers/<task / , used under every policy so grading is identical across policy arms. Rows are marked verifier source: adapted . - 7 tasks can't pass under hardened policies build-pmars , build-pov-ray , build-cython-ext , kv-store-grpc , nginx-request-logging , mailman , count-dataset-tokens . Their instructions require what the policies deny: system-wide installs, writes under /etc or /var , or non-allowlisted hosts. Under any policy except control , boundarybench run asks for confirmation before running them --yes skips the prompt . Forced rows are marked policy applicability: not applicable and excluded from success rates. count-dataset-tokens is also flagged separately: its verifier accepts the expected constant without the computation. | CLI name | Harness | |---|---| terminus | Terminus 2 Harbor's reference agent, default | claude-code | Claude Code | codex | OpenAI Codex CLI | grok | Grok CLI | Pinned harness versions are listed in provenance/ /boundary-bench/boundary-bench/blob/main/provenance . Inspect : boundarybench registers tasks via the inspect ai entry point; runs produce standard .eval logs usable with inspect view and the Inspect toolchain. Harbor / Terminal-Bench : task setup and official verification come from inspect-harbor / harbor against the pinned terminal-bench-2.1 dataset. Boundary-Bench does not fork task semantics: the agent runs under policy, and Harbor's unmodified verifier decides pass/fail. This repository is the benchmark core: policies, harnesses, the Daytona runner, and Harbor verification. | Path | Purpose | |---|---| src/boundarybench/terminalbench/ | The Inspect task, Daytona runner, Harbor bundle/verifier glue, policy resolver | src/boundarybench/harness/ | Harness adapters claude code, codex, terminus 2, grok + in-sandbox install scripts | src/boundarybench/daytona/ | Policy axes, hardening config/scripts, runtime policy probes | src/boundarybench/policy catalog/ | Canonical policy catalog packs, validation, compilation | src/boundarybench/cli.py | boundarybench CLI run , list , export-results , … | docs/ | Policy model, NIST level derivations, env vars, snapshots | provenance/ | Frozen manifest of the evaluated model + harness bundles bundles.json , BUNDLES.md | verifier diffs/ | Unified diffs of the five adapted verifiers vs upstream Terminal-Bench 2.1, with per-assertion notes | data/ | Per-task solvability-evidence CSV | tests/ | Unit and contract tests | uv sync --extra dev --extra harbor --extra daytona uv run pytest -q uv run ruff check src tests The test suite is hermetic and needs no live credentials. Daytona provisioning and model calls happen only in real benchmark runs. @misc{davidovich2026permissiondenied, title = {Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments}, author = {Dotan Davidovich and Yair Amar and Hai Rozencwajg and Or Hiltch}, year = {2026}, eprint = {2608.02670}, archivePrefix = {arXiv}, primaryClass = {cs.CR}, url = {https://arxiv.org/abs/2608.02670} } MIT, see LICENSE /boundary-bench/boundary-bench/blob/main/LICENSE .