A Mathematical Framework for Transformer Circuits (2021)
Researchers published "A Mathematical Framework for Transformer Circuits," a paper that reverse engineers toy attention-only transformers with two layers or fewer and identifies "induction heads" as t…