Mixture-of-Experts

A router-augmented membership inference attack that combines output-side signals with aggregated MoE routing features increases true-positive rate at 1% false-positive rate by 2.7 to 9.4 percentage po…

A paper submitted to arXiv on 30 Sep 2026 introduces Loop Scaling Laws, which its authors describe as the first scaling law to jointly model recurrence and sparsity alongside model size and data. The …

E-MoE, a new method from researchers behind arXiv paper 2609.37533v1, builds the reverse process of masked diffusion models as a mixture of factorized distributions over a discrete shared latent set b…

AWS published an architecture that combines Amazon Elastic Kubernetes Service (Amazon EKS), Elastic Fabric Adapter (EFA), and DeepEP to scale Mixture-of-Experts reinforcement learning, reporting 40% m…

Researchers proposed NSFT (Neural Sub-expert Fine-Tuning), a fine-grained parameter-efficient fine-tuning framework that refines Mixture-of-Experts (MoE) large language model adaptation from experts t…

Researchers introduced UniGIO, a generative framework that models global in-situ weather dynamics directly from spatiotemporally incomplete station observations, according to an arXiv paper (2609.2221…

A new arXiv paper (arXiv:2609.22199v1) proposes CS-MoE, a Transformer architecture that shares neural experts across layers via a centralized global expert pool, achieving lower perplexity than equal-…

Researchers introduced MoME, a Mixture-of-Memory Embeddings method for context-aware sparse lookup that addresses a limitation in existing memory-embedding approaches, which retrieve via a determinist…

Kuaishou deployed OneBid, a unified auto-bidding foundation model built on Decision Transformer, reporting an overall +2.2% ADVV gain on oCPX Ads and a peak +13.1% gain in the ROAS scenario in online …

NVIDIA's Nemotron 3.5 Lightning 30B-A3B model demonstrates how a Mixture-of-Experts (MoE) architecture can activate only 3B of its 30B parameters per token by routing each token through a selected sub…

Researchers at arXiv introduced a heterogeneous Mixture-of-Experts (MoE) framework that combines decision trees, linear support vector machines, and quadratic discriminant analysis under a unified pro…

Researchers developed a memory-efficient, single-GPU training and inference architecture for autoformalizing natural language mathematics into Lean 4 statement code, using the Qwen3-Coder-30B-A3B spar…

A new analysis from the Journal of Machine Learning Research explains that Mixture-of-Experts (MoE) routing can cause large language models to produce different outputs across runs due to discrete exp…

A new arXiv paper (arXiv:2607.28633v1) proposes a topology-aware transfer orchestrator for disaggregated GPU inference, claiming existing systems like DistServe, Splitwise, and Mooncake ignore that ba…

Moonshot AI's Kimi K3 (Max), a 2.8-trillion-parameter open-weight model, achieved a +9.75% net-improvement score to rank third overall in Agent Arena, trailing only Claude Fable 5 (High) and GPT-5.6 S…

Researchers have introduced the Loopie series, two Mixture-of-Experts models (20B parameters with 2B active and 6B with 0.6B active) that outperform vanilla Transformer baselines trained with the same…

Researchers propose Multi-level Context Fusion MOE (MCF-MOE), a framework that improves expert selection consistency in Mixture-of-Experts Transformer models by integrating cross-layer semantic aggreg…

Stanford researchers introduced TRACE, an open-source system that diagnoses recurring agent failures and trains targeted capabilities via synthetic reinforcement learning environments, achieving a hig…

Researchers introduced TriRoute, a unified controller that jointly optimizes attention resolution, expert selection, and KV-cache bit-width for language models, achieving Pareto-dominant performance o…

Google DeepMind released Gemma 4, a new generation of open-weight multimodal language models featuring dense and Mixture-of-Experts architectures from 2.3B to 31B parameters. The models introduce a th…