{"slug": "leak-it-a-probabilistic-approach-to-training-data-extraction-from-black-box", "title": "Leak It: A Probabilistic Approach to Training-Data Extraction from Black-Box Language Models", "summary": "A new study from arXiv (2608.00144v1) finds that aggregate ROC-AUC metrics hide per-document training-data leakage in black-box language models, with Pythia-6.9B reproducing exact identifiers for 16.6% of 500 Pile documents (21.3% of those with email addresses), a rate that grows with model capacity from 5.6% at 410M to 16.6% at 6.9B. The authors release leakit, a black-box extraction-audit tool, and recommend privacy audits report per-document extraction decomposed by domain rather than a single AUC.", "body_md": "arXiv:2608.00144v1 Announce Type: new\nAbstract: Membership inference (MIA) on language models is usually summarised by an aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines separate members from non-members from surface text alone. We study black-box, sampling-based training-data leakage through a probabilistic lens, treating N samples from p(.|x) as an estimate of the output distribution and casting leakage signals as functionals of it. We extend the blind-baseline critique into the sampling regime: on WikiMIA a blind bag-of-words classifier reaches AUC 0.97 (TPR 0.90 at 5% FPR) and sampling adds nothing, while on an IID Pile split (MIMIR) neither self-concentration nor gold-continuation recovery significantly beats a blind baseline (incremental AUC 95% CI includes zero). Aggregate metrics hide the real harm. The same sampling verbatim-extracts training data for a tail of documents no blind attack can reach. On Pythia-6.9B, 83 of 500 Pile documents bearing a real identifier (16.6%; 21.3% of those bearing an email address) have that exact identifier reproduced AND not reproduced under a mismatched-prefix control, so each leak is attributable to that document, not to a globally common string. This per-document disclosure is invisible to aggregate AUC and grows with capacity (5.6% to 16.6% from 410M to 6.9B). The risk is uneven: identifier leakage is ~3x stronger in code than prose, though prose stays clearly positive and also grows with capacity (4.0% to 12.1%), while recovery of arbitrary held-out continuations is confined to code (+0.44 member gap on GitHub vs at most +0.014 on prose). Temperature and nucleus sampling matter little, a 16-token prefix suffices, and we detect no reduction from corpus deduplication. Privacy audits should report per-document extraction, decomposed by domain, not a single AUC. We release leakit, a black-box extraction-audit tool.", "url": "https://wpnews.pro/news/leak-it-a-probabilistic-approach-to-training-data-extraction-from-black-box", "canonical_source": "https://arxiv.org/abs/2608.00144", "published_at": "2026-08-04 04:00:00+00:00", "updated_at": "2026-08-04 04:33:51.410161+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research", "ai-tools"], "entities": ["arXiv", "WikiMIA", "MIMIR", "Pile", "Pythia-6.9B", "GitHub", "leakit"], "alternates": {"html": "https://wpnews.pro/news/leak-it-a-probabilistic-approach-to-training-data-extraction-from-black-box", "markdown": "https://wpnews.pro/news/leak-it-a-probabilistic-approach-to-training-data-extraction-from-black-box.md", "text": "https://wpnews.pro/news/leak-it-a-probabilistic-approach-to-training-data-extraction-from-black-box.txt", "jsonld": "https://wpnews.pro/news/leak-it-a-probabilistic-approach-to-training-data-extraction-from-black-box.jsonld"}}