A Controlled Study of Attention-Only Transformers
A controlled study pretraining attention-only decoder transformers (Simple Attention Networks) against standard transformers matched for parameters, compute, and depth found that reallocating the feed-forward budget into…