Researchers at arXiv introduced a heterogeneous Mixture-of-Experts (MoE) framework that combines decision trees, linear support vector machines, and quadratic discriminant analysis under a unified pro…
Researchers developed a memory-efficient, single-GPU training and inference architecture for autoformalizing natural language mathematics into Lean 4 statement code, using the Qwen3-Coder-30B-A3B spar…
A new analysis from the Journal of Machine Learning Research explains that Mixture-of-Experts (MoE) routing can cause large language models to produce different outputs across runs due to discrete exp…
A new arXiv paper (arXiv:2607.28633v1) proposes a topology-aware transfer orchestrator for disaggregated GPU inference, claiming existing systems like DistServe, Splitwise, and Mooncake ignore that ba…
Moonshot AI's Kimi K3 (Max), a 2.8-trillion-parameter open-weight model, achieved a +9.75% net-improvement score to rank third overall in Agent Arena, trailing only Claude Fable 5 (High) and GPT-5.6 S…
Researchers have introduced the Loopie series, two Mixture-of-Experts models (20B parameters with 2B active and 6B with 0.6B active) that outperform vanilla Transformer baselines trained with the same…
Researchers propose Multi-level Context Fusion MOE (MCF-MOE), a framework that improves expert selection consistency in Mixture-of-Experts Transformer models by integrating cross-layer semantic aggreg…
Stanford researchers introduced TRACE, an open-source system that diagnoses recurring agent failures and trains targeted capabilities via synthetic reinforcement learning environments, achieving a hig…
Researchers introduced TriRoute, a unified controller that jointly optimizes attention resolution, expert selection, and KV-cache bit-width for language models, achieving Pareto-dominant performance o…
Google DeepMind released Gemma 4, a new generation of open-weight multimodal language models featuring dense and Mixture-of-Experts architectures from 2.3B to 31B parameters. The models introduce a th…
NVIDIA released NeMo AutoModel, an open-source library that accelerates fine-tuning of Mixture-of-Experts (MoE) transformer models by 3.4-3.7x in training throughput and reduces GPU memory usage by 29…
MiniMax released MSA (MiniMax Sparse Attention), a block-sparse attention method that reduces the quadratic cost of softmax attention at long contexts. The method was trained on a 109B-parameter Mixtu…
Researchers found that repetition loops in Gemma 4 instruction-tuned LLMs can be fixed by editing as few as one neuron, but the fix does not resolve deeper 'doom loops' caused by missing knowledge. Th…
Mixture-of-Experts (MoE) architecture allows large language models to use only a fraction of their total parameters for each token, enabling models with 671 billion total parameters to run at speeds c…
JetBrains released Mellum2, a 12-billion-parameter Mixture-of-Experts model with 2.5 billion active parameters per token, under the Apache 2.0 license. The model is specialized for software engineerin…
StepFun released Step 3.7 Flash, a 198-billion parameter Mixture-of-Experts vision-language model with native image understanding and improved tool-use reliability for agentic coding and search workfl…