Source-balanced Capped Fisher: Bounded Evidence Aggregation for User Inference in Language Models

Researchers proposed source-balanced capped Fisher (SCF), an aggregation method for user inference that determines whether a source contributed documents to a language-model update using only fresh documents from that source. SCF uses known nonparticipating sources to gauge how unusual each document score is, converts reference-tail frequencies into negative-log evidence, caps individual contributions, and averages them within each source, paired with a p-value decision rule calibrated against separate known nonparticipating sources that provides finite-sample Type-I error control under source exchangeability. Across four real-world datasets, SCF achieved the highest AUROC among evaluated aggregation methods in four domains, with empirical false-positive rates generally close to or below nominal levels.

arXiv:2611.07228v1 Announce Type: new Abstract: User inference asks whether a source contributed documents to a language-model update, even when the auditor observes only fresh documents from that source. Document-level scores can provide evidence of participation, but combining this evidence is challenging: documents differ in baseline predictability and in how strongly they reflect information learned from other documents of the same source. We propose source-balanced capped Fisher SCF , an aggregation method that uses known nonparticipating sources to assess how unusual each document score is. It transforms the resulting reference-tail frequencies into negative-log evidence, caps individual contributions, and averages them within each source. This limits the influence of isolated extreme scores while retaining evidence across multiple documents. We complete the inference procedure with a decision rule that tests source participation using a p-value calibrated against a separate set of known nonparticipating sources. The rule provides finite-sample Type-I error control under source exchangeability. Experiments across four real-world datasets show that SCF achieves the highest AUROC among the evaluated aggregation methods in four domains. With the calibration-based decision rule, SCF delivers consistently competitive detection power across the tested settings, achieving the highest power in many cases, while empirical false-positive rates remain generally close to or below the nominal levels.