A guard's job is to hold the line. holdline measures whether it does — a neutral benchmark for AI-agent write-guards. It scores any guard — expressed as a (commitments, action) → block?
function — over a labeled corpus, and reports the metrics that matter for a gate: catch rate, false-block rate, and class-balanced Cohen's kappa (raw kappa lies under class imbalance). The corpus includes an injection-attack class: actions whose content tries to talk the guard out of its verdict.
Why: the DeepSeek Harness ecosystem has 20+ guard/policy plugins and no shared way to measure whether any of them works. A guard's README saying "blocks dangerous commands" is not evidence. This harness is the evidence.
pnpm install
node run.mjs # scores every built-in guard over the 42-case corpus
node run.mjs --model <id> # point the judge guard at a different local model
node odcv-run.mjs # score the judge on REAL agent trajectories (ODCV-Bench)
Two result sets: RESULTS.md (authored corpus, incl. a real named guard and an injection class) and RESULTS-ODCV.md (the harder number: agreement with a 4-model judge panel on real agent trajectories we did not write, balanced kappa 0.82). Scorer is tested (pnpm test
); it dogfoods the published dsh-write-gate core for the judge guard.
Implement the Guard
interface in src/guards.ts
(name, kind, note, block(case)
), add it to the list in run.mjs
, and open a PR with your results. A guard that mounts an actual published plugin (rather than a strategy archetype) is especially welcome.
v0, honest limits stated in RESULTS.md: the corpus is small and hand-authored, the deny-list is a strategy archetype (not a specific plugin), and the numbers are one model / one run. The value is the shape it exposes and that anyone can re-run it. MIT.