SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

A new research paper introduces SMELT, a study of scaling laws for compute-matched Mixture-of-Experts (MoE) looped Transformers, finding that looping provides architectural advantages beyond mere FLOPs. The authors compare looped and non-looped models at matched per-token FLOPs and total non-embedding parameters, demonstrating improved performance for looped MoE Transformers.

Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding p