Hugging Face released a Mixture-of-Experts training handbook aimed at engineers who have already trained a dense model and want to train an MoE next, according to author Nouamane, who worked on the Ultra-Scale and Smol Training Playbooks. The handbook covers training MoE models with Megatron-LM, setting by setting, and is hosted on a Hugging Face Space.
Handbook: https://huggingface.co/spaces/huggingface/moe-blogpost Hello, I'm Nouamane, one of the authors. I'm at Hugging Face and also worked on the Ultra-Scale and Smol Training Playbooks, if you've read those. This handbook is for people who have trained a dense model and want to train an MoE next
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.