Switch Transformer

A new analysis from the Journal of Machine Learning Research explains that Mixture-of-Experts (MoE) routing can cause large language models to produce different outputs across runs due to discrete exp…

Mixture of Experts (MoE) is a machine learning architecture that divides a model into specialized subnetworks called experts, with a gating network routing each input token to only the most relevant s…