{"slug": "improving-parameter-utilization-by-sharing-neural-experts-across-layers-in", "title": "Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers", "summary": "A new arXiv paper (arXiv:2609.22199v1) proposes CS-MoE, a Transformer architecture that shares neural experts across layers via a centralized global expert pool, achieving lower perplexity than equal-scale dense Transformers while activating only 55% of parameters. The authors report that CS-MoE's performance scales monotonically with the number of activated experts and approaches Mixture-of-Experts counterparts that consume more FLOPs, establishing a flexible Pareto frontier between computational cost and model capacity for computation-constrained environments.", "body_md": "arXiv:2609.22199v1 Announce Type: new \nAbstract: Transformer-based large language models often suffer from inter-layer parameter redundancy, where functional transformations are redundantly learned across network depths. We propose CS-MoE, a novel Transformer architecture featuring cross-layer expert sharing to address this inefficiency. Deviating from the widely used Mixture-of-Experts (MoE) architecture that terminates each Transformer block with layer-isolated experts, CS-MoE combines layer-independent experts with concurrent access to a centralized, globally shared expert pool. This \\textit{Global Experts Sharing} mechanism enables elastic control over token-level parameter activation and computational consumption (FLOPs). Experiments demonstrate that CS-MoE achieves lower perplexity than equal-scale dense Transformers while activating only 55\\% of parameters. Furthermore, its performance scales monotonically with an increased number of activated experts and approaches MoE counterparts that consume more FLOPs by expanding the shared pool with a fixed FLOPs budget. CS-MoE also establishes a flexible Pareto frontier between computational cost and model capacity, offering an efficient alternative for computation-constrained environments.", "url": "https://wpnews.pro/news/improving-parameter-utilization-by-sharing-neural-experts-across-layers-in", "canonical_source": "https://www.machinebrief.com/news/improving-parameter-utilization-by-sharing-neural-experts-ac-gtjp", "published_at": "2026-09-22 04:00:00+00:00", "updated_at": "2026-09-22 05:24:45.169940+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["CS-MoE", "arXiv", "Mixture-of-Experts"], "alternates": {"html": "https://wpnews.pro/news/improving-parameter-utilization-by-sharing-neural-experts-across-layers-in", "markdown": "https://wpnews.pro/news/improving-parameter-utilization-by-sharing-neural-experts-across-layers-in.md", "text": "https://wpnews.pro/news/improving-parameter-utilization-by-sharing-neural-experts-across-layers-in.txt", "jsonld": "https://wpnews.pro/news/improving-parameter-utilization-by-sharing-neural-experts-across-layers-in.jsonld"}}