Reevaluating Weight Spectra in Language Models: A Missed Shortcut?
Researchers analyzing GPT-2 checkpoints found recurring spectral patterns in weight matrices, but reusing these spectra as initialization for new models failed to improve performance. The study suggests that while pretra…