MOBA: Mixture of Block Attention for Long-Context LLMs
MoonshotAI researchers submitted a paper on 18 Feb 2025 introducing Mixture of Block Attention (MoBA), an attention mechanism that applies Mixture of Experts principles to let long-context large langu…