Hey everyone,
I’ve been experimenting with fine-grained routing in Mixture of Experts (MoE) architectures and wanted to share SplitMoE, an alternative take on modular expert routing.
GitHub Repo: https://github.com/Priyanshu-5257/SplitMoE Standard MoEs (like Mixtral) use fewer, larger experts with Top-K routing. While effective, large monolithic experts can suffer from parameter redundancy and coarse specialization.
SplitMoE builds on the philosophy of breaking down wide feed-forward layers into smaller, specialized sub-components, making token representation much more flexible and computationally modular.
| Feature | DeepSeekMoE | SplitMoE |
|---|---|---|
| Fine-grained Experts | Splits standard FFNs into m smaller sub-experts (higher expert count, lower cost per expert). | Also adopts smaller, fine-grained expert partitions for finer expressivity. |
| Shared vs. Routed | Explicitly isolates a fixed subset of experts as permanently active shared experts alongside routed ones. | Explores splitting and grouping mechanisms across intermediate dimensions/routing pools without strictly forcing standard fixed shared pathways. |
| Routing / Gating | Standard sigmoid/softmax-based Top-K over a very large pool with device/expert balancing loss. | Focuses on a streamlined partitioning strategy to keep routing lightweight and avoid heavy dispatch overhead. |
Take a look at the repo, test it out, and let me know your thoughts or feedback on routing stability and scaling behavior!
source & further reading
discuss.huggingface.co — original article
Rakarrack-0.6.1 port making progress! ( AI assisted )
Cloud Storage Poll
Welcome to Haiku basic(Haiku Docs, Haiku slide and Haiku sheets)