An Induction Head in Disguise: Chasing Grammar in a Character-Level Transformer
A researcher trained a character-level transformer on Nietzsche texts and identified a copying head (B5H0) using eigenvalue analysis, but found its QK circuit insufficient for bracket completion. The β¦