ASSERT: A Measurement Pipeline for GenAI Audits

Researchers introduced ASSERT, a specification-driven measurement pipeline for generative AI audits that ties each reported compliance rate to a written specification of measurement choices. In a case study on conversational deception, the reported rate varied substantially with dialogue setup, simulated user, judge, and evidence bar, and could reorder GenAI system rankings. The pipeline helps draft behavioral rubrics and test cases, then runs audits and returns reported rates, making differences across audits easier to attribute and interpret.

arXiv:2608.13840v1 Announce Type: new Abstract: Audits of generative AI GenAI systems often summarize behavior as a reported rate: how often the audited system complies with policy. Researchers and stakeholders use that rate to compare systems, track regressions, and gate deployment. A reported rate reflects both the system under audit and the measurement choices behind it, so a change in the rate can leave it unclear whether the system or those choices moved. We introduce ASSERT, a specification-driven measurement pipeline for GenAI audits that ties each reported rate to a written specification of the measurement choices used to produce it. ASSERT helps draft a behavioral rubric and test cases, then runs the audit against a GenAI system and returns a reported rate. In a case study on conversational deception, we observe that the reported rate moves substantially with the dialogue setup, the simulated user, the judge, and the evidence bar for non-compliance. These measurement choices substantially change the reported rate and can reorder GenAI system rankings. Because each reported rate is tied to an explicit specification, differences across audits are easier to attribute and interpret.