{"slug": "mixture-of-channel-experts-static-sparse-supports-with-input-adaptive-mixing-for", "title": "Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections", "summary": "Researchers introduce Mixture of Channel Experts (MoCE), a structured sparse channel-mixing layer that replaces pointwise projections in convolutional networks, achieving a 16.7% reduction in MACs and lower end-to-end latency while matching or exceeding dense baselines on ImageNet-1K and CIFAR-100. The method, detailed in arXiv:2608.23794v1, moves the expert axis from operator duplication to channel selection, with each expert being a single output channel with a learned sparse support of k << C input channels.", "body_md": "arXiv:2608.23794v1 Announce Type: new\nAbstract: Mixture-of-Experts (MoE) scales language models by routing each input through a small set of independently parameterized experts. We show that copying this design into convolutional networks fails for a structural reason: parallel convolutional experts that read the same input channels learn nearly identical filters. We therefore move the expert axis from operator duplication to channel selection. We introduce Mixture of Channel Experts (MoCE), a structured sparse channel-mixing layer, inspired by MoE, that replaces pointwise (1x1) channel-reduction projections. In MoCE, an expert is a single output channel with a learned sparse support of k << C input channels. The selected channels are combined by a softmax whose temperature is predicted per input, so each expert can move between mean-like and max-like aggregation. A residual expert summarizes the unselected channels, and a load-balancing loss keeps channel coverage complete. MoCE replaces a dense projection whose cost is quadratic in C with a mechanism whose relative cost scales as k/C, and the predicted savings hold in measured wall-clock time. Across ResNet backbones on ImageNet-1K and CIFAR-100, transfer learning, EfficientViT, and a strong modern training recipe, MoCE matches or exceeds dense baselines and prior channel-selection methods while reducing MACs by 16.7% and end-to-end latency.", "url": "https://wpnews.pro/news/mixture-of-channel-experts-static-sparse-supports-with-input-adaptive-mixing-for", "canonical_source": "https://arxiv.org/abs/2608.23794", "published_at": "2026-08-26 04:00:00+00:00", "updated_at": "2026-08-26 04:12:59.423048+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks"], "entities": ["Mixture of Channel Experts", "arXiv", "ImageNet-1K", "CIFAR-100", "ResNet", "EfficientViT"], "alternates": {"html": "https://wpnews.pro/news/mixture-of-channel-experts-static-sparse-supports-with-input-adaptive-mixing-for", "markdown": "https://wpnews.pro/news/mixture-of-channel-experts-static-sparse-supports-with-input-adaptive-mixing-for.md", "text": "https://wpnews.pro/news/mixture-of-channel-experts-static-sparse-supports-with-input-adaptive-mixing-for.txt", "jsonld": "https://wpnews.pro/news/mixture-of-channel-experts-static-sparse-supports-with-input-adaptive-mixing-for.jsonld"}}