The Ultimate MoE Training Handbook: training Mixture-of-Experts models with Megatron-LM, setting by setting [P]

Hugging Face released a Mixture-of-Experts training handbook aimed at engineers who have already trained a dense model and want to train an MoE next, according to author Nouamane, who worked on the Ultra-Scale and Smol Training Playbooks. The handbook covers training MoE models with Megatron-LM, setting by setting, and is hosted on a Hugging Face Space.

Handbook: https://huggingface.co/spaces/huggingface/moe-blogpost Hello, I'm Nouamane, one of the authors. I'm at Hugging Face and also worked on the Ultra-Scale and Smol Training Playbooks, if you've read those. This handbook is for people who have trained a dense model and want to train an MoE next