Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers

A new arXiv paper (arXiv:2609.22199v1) proposes CS-MoE, a Transformer architecture that shares neural experts across layers via a centralized global expert pool, achieving lower perplexity than equal-scale dense Transformers while activating only 55% of parameters. The authors report that CS-MoE's performance scales monotonically with the number of activated experts and approaches Mixture-of-Experts counterparts that consume more FLOPs, establishing a flexible Pareto frontier between computational cost and model capacity for computation-constrained environments.

arXiv:2609.22199v1 Announce Type: new Abstract: Transformer-based large language models often suffer from inter-layer parameter redundancy, where functional transformations are redundantly learned across network depths. We propose CS-MoE, a novel Transformer architecture featuring cross-layer expert sharing to address this inefficiency. Deviating from the widely used Mixture-of-Experts MoE architecture that terminates each Transformer block with layer-isolated experts, CS-MoE combines layer-independent experts with concurrent access to a centralized, globally shared expert pool. This \textit{Global Experts Sharing} mechanism enables elastic control over token-level parameter activation and computational consumption FLOPs . Experiments demonstrate that CS-MoE achieves lower perplexity than equal-scale dense Transformers while activating only 55\% of parameters. Furthermore, its performance scales monotonically with an increased number of activated experts and approaches MoE counterparts that consume more FLOPs by expanding the shared pool with a fixed FLOPs budget. CS-MoE also establishes a flexible Pareto frontier between computational cost and model capacity, offering an efficient alternative for computation-constrained environments.