Challenge: Hand coding weights for efficient sequence memorisation
A research team hand-coded weights for one-layer MLPs that memorize labels for input token sequences of length two, finding that the number of facts memorized with 90% accuracy scales roughly linearly…