A new research paper introduces SMELT, a study of scaling laws for compute-matched Mixture-of-Experts (MoE) looped Transformers, finding that looping provides architectural advantages beyond mere FLOPs. The authors compare looped and non-looped models at matched per-token FLOPs and total non-embedding parameters, demonstrating improved performance for looped MoE Transformers.
Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding p
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.