Gregdiamos (auto-discovered)

A new paper and model from researcher George Diamos show that a block-routed mixture-of-experts with 128 experts and 3.65 million active parameters achieves 93% on in-context induction and 90% on posi…