Benchmark coding agents under real sandbox policy.
Website · Leaderboard · Quickstart · Policies · Citation
Boundary-Bench benchmarks coding-agent harnesses on Terminal-Bench tasks while the agent runs inside a hardened sandbox, measuring how much capability an agent loses as the environment is restricted to enterprise/NIST-derived controls.
Runs execute in Daytona sandboxes, agents reach
models through OpenRouter, and tasks and official
verification come from Harbor
(inspect-harbor
). It's packaged as an Inspect AI
task with standard .eval
logs.
Live results (the leaderboard, task browser, and research report) are at ** boundarybench.com**.
LEADERBOARD · STRICTEST LEVEL (HIGH-NIST) · AUG 2026| # | Agent | Model | Success rate | Cost |
|---|---|---|---|---|
| 01 | Grok Build | Grok 4.5 | 74.9% ±1.4 |
$123.5 |
| 02 | Codex | GPT-5.6 Sol | 74.2% ±2.4 |
$58.9 |
| 03 | Claude Code | Fable 5 | 67.8% ±1.4 |
$375.3 |
Terminal-Bench 2.1 · 89 tasks · reasoning effort high · three trials per cell · full leaderboard →
You need two keys: OpenRouter for model inference and Daytona for sandboxes. The task, policy, and harness are flags on the run command.
uv sync --extra harbor --extra daytona
cp .env.example .env # then fill in OPENROUTER_API_KEY and DAYTONA_API_KEY
uv run boundarybench run \
--task log-summary-date-ranges \
--policy high-nist \
--harness claude-code \
--model anthropic/claude-opus-5
Costs:runs bill your own OpenRouter and Daytona keys. Expect a few dollars and a few minutes per task, varying by task, model, and policy.
Running with no flags runs the full benchmark: all 89 tasks under the
control
policy with the terminus
harness, after a confirmation prompt:
uv run boundarybench run
--task
: run one task or a comma-separated list--all-tasks
: run all 89 tasks without the confirmation prompt--exclude-not-applicable
: skip tasks that can't pass under the selected policy--dry-run
: print what would run and exit, without spending anything--yes
: skip confirmation prompts, for scripts and CI
The run provisions a Daytona sandbox from the task's Docker image, installs
the harness, applies the policy's Linux hardening, lets the agent work, then
runs Harbor's official verifier outside the hardening boundary. Results
are written as Inspect .eval
logs (--log-dir
, default
logs/boundarybench-<timestamp>
). View them with inspect view
or export
CSV with boundarybench export-results
.
Other entry points:
boundarybench list # show policies and harnesses
boundarybench export-results ... # .eval logs -> CSV
boundarybench analyze ... # policy-blockage report from .eval logs
boundarybench enforcement-probe # probe a preset's native walls (no agent)
boundarybench policy validate # validate policy catalog packs
Boundary-Bench is an Inspect extension, so you can also run it with plain
inspect eval
. The --model
slot is only a label; the harness inside the sandbox makes the real model calls:
uv run inspect eval boundarybench/hardened_terminalbench_daytona \
-T task_ids=log-summary-date-ranges \
-T policy=high-nist \
-T harness=claude_code \
-T model=anthropic/claude-opus-5 \
--model mockllm/model
Policies are points in an N×F×P lattice (Network × Filesystem × Privilege)
enforced with native Linux controls (nftables, read-only bind remounts,
setpriv/no_new_privs/capability drops, Landlock). Denials show up as ordinary
OS errors (EROFS
, EPERM
, connection refused), with no agent-visible shim.
The published benchmark levels:
| Policy | Network | Filesystem | Privilege |
|---|---|---|---|
control |
|||
| full egress | open | root | |
non-root |
|||
| full egress | open | non-admin user | |
high-nist |
|||
| per-task allowlist | read-only OS + frozen home | non-admin + no_new_privs + capability drop |
Instead of a named level, you can compose a policy from these hardening options:
| Option | What it does |
|---|---|
non-root |
|
| Run the agent as an ordinary user (no sudo; root password locked) | |
no-escalation |
|
| Block privilege escalation (no-new-privileges, drop capabilities, strip setuid) | |
readonly-os |
|
| Make the OS read-only; only the workspace stays writable | |
freeze-home |
|
| Freeze the home directory too | |
restrict-egress |
|
| Restrict network access to an allowlist; block cloud-metadata & private networks |
Prerequisites are filled in automatically: freeze-home
implies readonly-os
and non-root
, and any network or filesystem hardening implies non-root
. Compose interactively:
uv run boundarybench build-policy # answer a few y/n questions, then run
Or non-interactively, with your own egress allowlist:
uv run boundarybench run \
--task log-summary-date-ranges --harness claude-code \
--harden non-root,readonly-os,restrict-egress \
--egress-mode custom --egress-allow pypi.org,files.pythonhosted.org
Egress rules never block the model endpoint. Inbound traffic isn't restricted because the sandbox exposes no services.
The benchmark runs all 89 terminal-bench-2.1 tasks. Two task lists
(task_applicability.py)
affect how results are read. Both are recorded in each sample's metadata and
shown by export-results
and analyze
:
5 tasks use an adapted verifier(adaptive-rejection-sampler
,configure-git-webserver
,make-doom-for-mips
,mcmc-sampling-stan
,sqlite-with-gcov
). The stock verifier checks a root-owned path, so a correct non-root solution would fail. Each task gets a corrected verifier (adapted_verifiers/<task>/
), used under every policy so grading is identical across policy arms. Rows are markedverifier_source: adapted
. -
7 tasks can't pass under hardened policies(build-pmars
,build-pov-ray
,build-cython-ext
,kv-store-grpc
,nginx-request-logging
,mailman
,count-dataset-tokens
). Their instructions require what the policies deny: system-wide installs, writes under/etc
or/var
, or non-allowlisted hosts. Under any policy exceptcontrol
,boundarybench run
asks for confirmation before running them (--yes
skips the prompt). Forced rows are markedpolicy_applicability: not_applicable
and excluded from success rates.count-dataset-tokens
is also flagged separately: its verifier accepts the expected constant without the computation.
| CLI name | Harness |
|---|---|
terminus |
|
| Terminus 2 (Harbor's reference agent, default) | |
claude-code |
|
| Claude Code | |
codex |
|
| OpenAI Codex CLI | |
grok |
|
| Grok CLI |
Pinned harness versions are listed in provenance/.
Inspect:boundarybench
registers tasks via theinspect_ai
entry point; runs produce standard.eval
logs usable withinspect view
and the Inspect toolchain.Harbor / Terminal-Bench: task setup and official verification come frominspect-harbor
/harbor
against the pinnedterminal-bench-2.1
dataset. Boundary-Bench does not fork task semantics: the agent runs under policy, and Harbor's unmodified verifier decides pass/fail.
This repository is the benchmark core: policies, harnesses, the Daytona runner, and Harbor verification.
| Path | Purpose |
|---|---|
src/boundarybench/terminalbench/ |
|
| The Inspect task, Daytona runner, Harbor bundle/verifier glue, policy resolver | |
src/boundarybench/harness/ |
|
| Harness adapters (claude_code, codex, terminus_2, grok) + in-sandbox install scripts | |
src/boundarybench/daytona/ |
|
| Policy axes, hardening config/scripts, runtime policy probes | |
src/boundarybench/policy_catalog/ |
|
| Canonical policy catalog packs, validation, compilation | |
src/boundarybench/cli.py |
|
boundarybench CLI (run , list , export-results , …) |
|
docs/ |
|
| Policy model, NIST level derivations, env vars, snapshots | |
provenance/ |
|
Frozen manifest of the evaluated model + harness bundles (bundles.json , BUNDLES.md ) |
|
verifier_diffs/ |
|
| Unified diffs of the five adapted verifiers vs upstream Terminal-Bench 2.1, with per-assertion notes | |
data/ |
|
| Per-task solvability-evidence CSV | |
tests/ |
|
| Unit and contract tests |
uv sync --extra dev --extra harbor --extra daytona
uv run pytest -q
uv run ruff check src tests
The test suite is hermetic and needs no live credentials. Daytona provisioning and model calls happen only in real benchmark runs.
@misc{davidovich2026permissiondenied,
title = {Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments},
author = {Dotan Davidovich and Yair Amar and Hai Rozencwajg and Or Hiltch},
year = {2026},
eprint = {2608.02670},
archivePrefix = {arXiv},
primaryClass = {cs.CR},
url = {https://arxiv.org/abs/2608.02670}
}
MIT, see LICENSE.