{"slug": "show-hn-a-benchmark-for-ai-agent-guardrails-that-caught-my-own-plugin", "title": "Show HN: A benchmark for AI agent guardrails that caught my own plugin", "summary": "A new open-source benchmark, holdline, measures the effectiveness of AI-agent write-guards, reporting a class-balanced Cohen's kappa of 0.82 for agreement with a 4-model judge panel on real agent trajectories from ODCV-Bench. Created by a developer who found their own plugin lacking, holdline scores any guard expressed as a (commitments, action) → block? function over a 42-case corpus that includes an injection-attack class, and is designed to provide evidence for the DeepSeek Harness ecosystem's 20+ guard/policy plugins.", "body_md": "A guard's job is to hold the line. **holdline** measures whether it does — a neutral benchmark for AI-agent write-guards. It scores **any guard** — expressed as a `(commitments, action) → block?`\n\nfunction — over a labeled corpus, and reports the metrics that matter for a gate: catch rate, false-block rate, and **class-balanced Cohen's kappa** (raw kappa lies under class imbalance). The corpus includes an **injection-attack class**: actions whose content tries to talk the guard out of its verdict.\n\nWhy: the DeepSeek Harness ecosystem has 20+ guard/policy plugins and no shared way to measure whether any of them works. A guard's README saying \"blocks dangerous commands\" is not evidence. This harness is the evidence.\n\n```\npnpm install\nnode run.mjs                    # scores every built-in guard over the 42-case corpus\nnode run.mjs --model <id>       # point the judge guard at a different local model\nnode odcv-run.mjs               # score the judge on REAL agent trajectories (ODCV-Bench)\n```\n\nTwo result sets: [RESULTS.md](/couldbeme/holdline/blob/main/RESULTS.md) (authored corpus, incl. a real named guard and an injection class) and [RESULTS-ODCV.md](/couldbeme/holdline/blob/main/RESULTS-ODCV.md) (the harder number: agreement with a 4-model judge panel on real agent trajectories we did not write, balanced kappa **0.82**). Scorer is tested (`pnpm test`\n\n); it dogfoods the published [ dsh-write-gate](https://www.npmjs.com/package/dsh-write-gate) core for the judge guard.\n\nImplement the `Guard`\n\ninterface in `src/guards.ts`\n\n(name, kind, note, `block(case)`\n\n), add it to the list in `run.mjs`\n\n, and open a PR with your results. A guard that mounts an actual published plugin (rather than a strategy archetype) is especially welcome.\n\nv0, honest limits stated in [RESULTS.md](/couldbeme/holdline/blob/main/RESULTS.md): the corpus is small and hand-authored, the deny-list is a strategy archetype (not a specific plugin), and the numbers are one model / one run. The value is the *shape* it exposes and that anyone can re-run it. MIT.", "url": "https://wpnews.pro/news/show-hn-a-benchmark-for-ai-agent-guardrails-that-caught-my-own-plugin", "canonical_source": "https://github.com/couldbeme/holdline", "published_at": "2026-08-17 23:11:48+00:00", "updated_at": "2026-08-17 23:41:20.880009+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-tools", "ai-research"], "entities": ["holdline", "DeepSeek Harness", "ODCV-Bench", "dsh-write-gate"], "alternates": {"html": "https://wpnews.pro/news/show-hn-a-benchmark-for-ai-agent-guardrails-that-caught-my-own-plugin", "markdown": "https://wpnews.pro/news/show-hn-a-benchmark-for-ai-agent-guardrails-that-caught-my-own-plugin.md", "text": "https://wpnews.pro/news/show-hn-a-benchmark-for-ai-agent-guardrails-that-caught-my-own-plugin.txt", "jsonld": "https://wpnews.pro/news/show-hn-a-benchmark-for-ai-agent-guardrails-that-caught-my-own-plugin.jsonld"}}