Controlled benchmark clarifies failures in language-model data attribution

The paper studies why language-model data-attribution methods fail, noting evaluations often vary several dataset properties at once. It introduces CATT, a controlled fact-tracing benchmark using fine-tuning on entirely fictitious knowledge with unambiguous ground truth.

#

Why it matters

Reliable attribution is needed to trace model outputs to training data, audit sources, and diagnose failures.

#

Analysis

Attribution research may adopt controlled synthetic benchmarks before claiming progress on real, confounded training data.

Source: arXiv AI disclosure: this article was produced with AI assistance and reviewed for accuracy.