Nvidia Developer Relations Manager Igor Dmochowski said at Infobip Shift 2026 that the compute required to process an LLM's context grows quadratically with context length, making it unscalable for pr…
MoonshotAI researchers submitted a paper on 18 Feb 2025 introducing Mixture of Block Attention (MoBA), an attention mechanism that applies Mixture of Experts principles to let long-context large langu…
GLM 5.2, a 744-billion-parameter model, runs inference on consumer-grade hardware by activating only 40 billion parameters per token using Mixture of Experts (MoE) architecture. The MoE routing mechan…
LongCat-2.0, a 1.6 trillion parameter Mixture of Experts (MoE) architecture, introduces a hierarchical routing mechanism and hybrid parallelism to scale model capacity while maintaining deployment fea…
Researchers propose GeMoE, a dynamic routing method for Mixture-of-Experts-based large vision-language models that uses gating entropy to adaptively select the number of experts per token, achieving 9…
A developer built a novel triple-hybrid LLM combining Mamba, Attention, and a 32-expert Mixture of Experts architecture from scratch for approximately $50, completing Titan v1 and the first training c…
A comprehensive review published on arXiv examines fine-tuning techniques for Large Language Models (LLMs), covering methodologies from supervised and unsupervised learning to parameter-efficient meth…
Mixture of Experts (MoE) improves model inference speed by optimizing token routing, enabling higher request volume scaling.…