# Runtime Verification of Multiple Natural Language Criteria for Agent Governance

> Source: <https://www.machinebrief.com/news/runtime-verification-of-multiple-natural-language-criteria-f-1ht3>
> Published: 2026-11-19 05:00:00+00:00

**Recommendation:** For runtime agent governance, pretrain a tree-attention verifier on criterion-verdict tuples like CriteriaBank's 394K examples and score criteria in parallel; VFM-4B handled 1,000 criteria per 4K-token context in under 2s on one H100.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

arXiv:2611.23074v1 Announce Type: new 
Abstract: AI agents are governed by hundreds of natural language criteria, yet existing verifiers cannot efficiently evaluate an agent's output against a large set of such criteria and report which are satisfied, violated, or inapplicable. We introduce the VFM, a tree-attention verifier that encodes the shared context and target text, and scores N natural language criteria in parallel, returning independent per-criterion ternary verdicts. The VFM is pretrained on CriteriaBank, a corpus of 394K (context, target, criterion, verdict) tuples spanning over 321K distinct criterion strings. A finetuned VFM-4B outperforms GPT-5.4 on a synthetic governance benchmark. On the DynaBench multi-criterion benchmark, it outperforms DynaGuard-4B on trace-level failure detection, and its per-criterion verdicts localize the violated rule and support focused escalation to a larger chain-of-thought judge. We show that CriteriaBank pretraining improves cross-domain transfer, particularly in the low-data regime, while matching a strong direct-finetuning control at full data. On a single H100, VFM-4B evaluates 1,000 criteria against a 4K-token context in under 2s at <11 GB VRAM, making it suitable for runtime verification.
