# SplitMOE- similar to deepseek-MOE `same same but different`

> Source: <https://discuss.huggingface.co/t/splitmoe-similar-to-deepseek-moe-same-same-but-different/180682#post_1>
> Published: 2026-09-22 11:40:13+00:00

Hey everyone,

I’ve been experimenting with fine-grained routing in Mixture of Experts (MoE) architectures and wanted to share **SplitMoE**, an alternative take on modular expert routing.

 **GitHub Repo:** [https://github.com/Priyanshu-5257/SplitMoE](https://github.com/Priyanshu-5257/SplitMoE?utm_source=gemini)

Standard MoEs (like Mixtral) use fewer, larger experts with Top-K routing. While effective, large monolithic experts can suffer from parameter redundancy and coarse specialization.

**SplitMoE** builds on the philosophy of breaking down wide feed-forward layers into smaller, specialized sub-components, making token representation much more flexible and computationally modular.

| **Feature** | **DeepSeekMoE** | **SplitMoE** | 
|---|---|---|
| **Fine-grained Experts** | Splits standard FFNs into m smaller sub-experts (higher expert count, lower cost per expert). | Also adopts smaller, fine-grained expert partitions for finer expressivity. | 
| **Shared vs. Routed** | Explicitly isolates a fixed subset of experts as permanently active **shared experts** alongside routed ones. | Explores splitting and grouping mechanisms across intermediate dimensions/routing pools without strictly forcing standard fixed shared pathways. | 
| **Routing / Gating** | Standard sigmoid/softmax-based Top-K over a very large pool with device/expert balancing loss. | Focuses on a streamlined partitioning strategy to keep routing lightweight and avoid heavy dispatch overhead. | 

Take a look at the repo, test it out, and let me know your thoughts or feedback on routing stability and scaling behavior!