# Show HN: A benchmark for AI agent guardrails that caught my own plugin

> Source: <https://github.com/couldbeme/holdline>
> Published: 2026-08-17 23:11:48+00:00

A guard's job is to hold the line. **holdline** measures whether it does — a neutral benchmark for AI-agent write-guards. It scores **any guard** — expressed as a `(commitments, action) → block?`

function — over a labeled corpus, and reports the metrics that matter for a gate: catch rate, false-block rate, and **class-balanced Cohen's kappa** (raw kappa lies under class imbalance). The corpus includes an **injection-attack class**: actions whose content tries to talk the guard out of its verdict.

Why: the DeepSeek Harness ecosystem has 20+ guard/policy plugins and no shared way to measure whether any of them works. A guard's README saying "blocks dangerous commands" is not evidence. This harness is the evidence.

```
pnpm install
node run.mjs                    # scores every built-in guard over the 42-case corpus
node run.mjs --model <id>       # point the judge guard at a different local model
node odcv-run.mjs               # score the judge on REAL agent trajectories (ODCV-Bench)
```

Two result sets: [RESULTS.md](/couldbeme/holdline/blob/main/RESULTS.md) (authored corpus, incl. a real named guard and an injection class) and [RESULTS-ODCV.md](/couldbeme/holdline/blob/main/RESULTS-ODCV.md) (the harder number: agreement with a 4-model judge panel on real agent trajectories we did not write, balanced kappa **0.82**). Scorer is tested (`pnpm test`

); it dogfoods the published [ dsh-write-gate](https://www.npmjs.com/package/dsh-write-gate) core for the judge guard.

Implement the `Guard`

interface in `src/guards.ts`

(name, kind, note, `block(case)`

), add it to the list in `run.mjs`

, and open a PR with your results. A guard that mounts an actual published plugin (rather than a strategy archetype) is especially welcome.

v0, honest limits stated in [RESULTS.md](/couldbeme/holdline/blob/main/RESULTS.md): the corpus is small and hand-authored, the deny-list is a strategy archetype (not a specific plugin), and the numbers are one model / one run. The value is the *shape* it exposes and that anyone can re-run it. MIT.