cd /news/ai-agents/runtime-verification-of-multiple-nat… · home › topics › ai-agents › article
[ARTICLE · art-172361] src=machinebrief.com ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

Runtime Verification of Multiple Natural Language Criteria for Agent Governance

A new arXiv paper (2611.23074v1) introduces the VFM, a tree-attention verifier that scores N natural language criteria against an agent's output in parallel and returns independent per-criterion ternary verdicts of satisfied, violated, or inapplicable. The verifier is pretrained on CriteriaBank, a corpus of 394K (context, target, criterion, verdict) tuples spanning over 321K distinct criterion strings; a finetuned VFM-4B outperforms GPT-5.4 on a synthetic governance benchmark and beats DynaGuard-4B on trace-level failure detection in the DynaBench multi-criterion benchmark. On a single H100, VFM-4B evaluates 1,000 criteria against a 4K-token context in under 2 seconds at under 11 GB of VRAM, which the authors present as making runtime agent governance practical.

recommendation

For runtime agent governance, pretrain a tree-attention verifier on criterion-verdict tuples like CriteriaBank's 394K examples and score criteria in parallel; VFM-4B handled 1,000 criteria per 4K-token context in under 2s on one H100.

more in playbook: Evals and testing for LLM apps
by read1 min views4 publishedNov 19, 2026

arXiv:2611.23074v1 Announce Type: new Abstract: AI agents are governed by hundreds of natural language criteria, yet existing verifiers cannot efficiently evaluate an agent's output against a large set of such criteria and report which are satisfied, violated, or inapplicable. We introduce the VFM, a tree-attention verifier that encodes the shared context and target text, and scores N natural language criteria in parallel, returning independent per-criterion ternary verdicts. The VFM is pretrained on CriteriaBank, a corpus of 394K (context, target, criterion, verdict) tuples spanning over 321K distinct criterion strings. A finetuned VFM-4B outperforms GPT-5.4 on a synthetic governance benchmark. On the DynaBench multi-criterion benchmark, it outperforms DynaGuard-4B on trace-level failure detection, and its per-criterion verdicts localize the violated rule and support focused escalation to a larger chain-of-thought judge. We show that CriteriaBank pretraining improves cross-domain transfer, particularly in the low-data regime, while matching a strong direct-finetuning control at full data. On a single H100, VFM-4B evaluates 1,000 criteria against a 4K-token context in under 2s at <11 GB VRAM, making it suitable for runtime verification.

── more in #ai-agents 4 stories · sorted by recency
── more on @vfm 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/runtime-verification…] indexed:0 read:1min 2026-11-19 · —