Making LLM security verdicts verifiable: the evidence gate pattern

A developer built USAP, an open-source system that enforces a strict evidence gate for LLM security verdicts, requiring every verdict to cite a resolvable source. The system rejects hand-wavy output by checking evidence references against a contract of 11 typed JSON fields, and it ships with a held-out corpus of real incidents for evaluation. USAP runs as an MCP server or as system prompts, aiming to make AI security analyst outputs verifiable and trustworthy.

Every "AI security analyst" I tried had the same flaw: a correct verdict and a confident-but-wrong one are indistinguishable on screen. In security that's not a UX nit — it's the whole problem. So I built USAP around a single rule, and this post is about that rule and three things that fell out of it. USAP's output contract is 11 typed JSON fields. The uncompromising one is evidence references : every verdict must carry at least one source that resolves . Four accepted forms: mcp:<logical :<tool :<call id — evidence fetched live from a connected MCP https://... — a canonical external source CVE, EPSS feed, MITRE s3://... — an operator artifact local://<repo-path — an in-repo standard, for advisory verdicts the path must exist A source like "scanner" or "the SIEM showed it" is rejected at the contract boundary. That one check removes most hand-wavy output, because the model can't satisfy the contract without pointing at something real. If a verdict cites mcp:siem:search , that can't hard-code Splunk — not everyone runs Splunk. So agents declare logical capabilities and a registry resolves them to whatever the operator connected Splunk, Elastic, Sentinel . If nothing implements a capability, the agent degrades gracefully and marks that axis UNKNOWN rather than inventing telemetry. Once you demand resolvable evidence, narrated numbers feel wrong too. CVSS is computed from the published vector. EPSS is pulled live from the FIRST feed. Confidence comes from a written rubric. If a number can't be computed, the tool returns "qualitative" instead of fabricating one — and the contract rejects a cvss score that disagrees with its cited vector. A system that enforces evidence shouldn't grade itself against its own answer key. USAP ships a held-out corpus of real public incidents Log4Shell, xz, Capital One, MOVEit plus benign false-positive traps, and a stdlib harness that reports precision, recall, FPR, and MTTD. USAP is open-source Apache-2.0 , stdlib-only, and runs as an MCP server or as paste-in system prompts in any model. Every example in the repo is a real command you can re-run. I'd value feedback on the gate design.