{"slug": "splitmoe-similar-to-deepseek-moe-same-same-but-different", "title": "SplitMOE- similar to deepseek-MOE `same same but different`", "summary": "Developer Priyanshu-5257 released SplitMoE, an open-source Mixture of Experts routing architecture that splits wide feed-forward layers into smaller, fine-grained expert partitions, published on GitHub at github.com/Priyanshu-5257/SplitMoE. SplitMoE differs from DeepSeekMoE by exploring splitting and grouping mechanisms across intermediate dimensions and routing pools without forcing fixed shared expert pathways, aiming to keep routing lightweight and avoid heavy dispatch overhead. The project targets routing stability and scaling behavior for modular expert routing in large language models.", "body_md": "Hey everyone,\n\nI’ve been experimenting with fine-grained routing in Mixture of Experts (MoE) architectures and wanted to share **SplitMoE**, an alternative take on modular expert routing.\n\n **GitHub Repo:** [https://github.com/Priyanshu-5257/SplitMoE](https://github.com/Priyanshu-5257/SplitMoE?utm_source=gemini)\n\nStandard MoEs (like Mixtral) use fewer, larger experts with Top-K routing. While effective, large monolithic experts can suffer from parameter redundancy and coarse specialization.\n\n**SplitMoE** builds on the philosophy of breaking down wide feed-forward layers into smaller, specialized sub-components, making token representation much more flexible and computationally modular.\n\n| **Feature** | **DeepSeekMoE** | **SplitMoE** | \n|---|---|---|\n| **Fine-grained Experts** | Splits standard FFNs into m smaller sub-experts (higher expert count, lower cost per expert). | Also adopts smaller, fine-grained expert partitions for finer expressivity. | \n| **Shared vs. Routed** | Explicitly isolates a fixed subset of experts as permanently active **shared experts** alongside routed ones. | Explores splitting and grouping mechanisms across intermediate dimensions/routing pools without strictly forcing standard fixed shared pathways. | \n| **Routing / Gating** | Standard sigmoid/softmax-based Top-K over a very large pool with device/expert balancing loss. | Focuses on a streamlined partitioning strategy to keep routing lightweight and avoid heavy dispatch overhead. | \n\nTake a look at the repo, test it out, and let me know your thoughts or feedback on routing stability and scaling behavior!", "url": "https://wpnews.pro/news/splitmoe-similar-to-deepseek-moe-same-same-but-different", "canonical_source": "https://discuss.huggingface.co/t/splitmoe-similar-to-deepseek-moe-same-same-but-different/180682#post_1", "published_at": "2026-09-22 11:40:13+00:00", "updated_at": "2026-09-22 11:55:17.739029+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["SplitMoE", "DeepSeekMoE", "Mixtral", "Priyanshu-5257", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/splitmoe-similar-to-deepseek-moe-same-same-but-different", "markdown": "https://wpnews.pro/news/splitmoe-similar-to-deepseek-moe-same-same-but-different.md", "text": "https://wpnews.pro/news/splitmoe-similar-to-deepseek-moe-same-same-but-different.txt", "jsonld": "https://wpnews.pro/news/splitmoe-similar-to-deepseek-moe-same-same-but-different.jsonld"}}