GLM 5.2, a 744-billion-parameter model, runs inference on consumer-grade hardware by activating only 40 billion parameters per token using Mixture of Experts (MoE) architecture. The MoE routing mechan…
LongCat-2.0, a 1.6 trillion parameter Mixture of Experts (MoE) architecture, introduces a hierarchical routing mechanism and hybrid parallelism to scale model capacity while maintaining deployment fea…
Researchers propose GeMoE, a dynamic routing method for Mixture-of-Experts-based large vision-language models that uses gating entropy to adaptively select the number of experts per token, achieving 9…
A developer built a novel triple-hybrid LLM combining Mamba, Attention, and a 32-expert Mixture of Experts architecture from scratch for approximately $50, completing Titan v1 and the first training c…
A comprehensive review published on arXiv examines fine-tuning techniques for Large Language Models (LLMs), covering methodologies from supervised and unsupervised learning to parameter-efficient meth…
Mixture of Experts (MoE) improves model inference speed by optimizing token routing, enabling higher request volume scaling.…