Grouped Query Attention

MiniMax introduced sparse key-value caches that reduce attention scaling from quadratic to near-linear, enabling practical multi-hundred-kilobyte context windows on a single GPU. The method cuts per-t…

MiniMax released MSA (MiniMax Sparse Attention), a block-sparse attention method that reduces the quadratic cost of softmax attention at long contexts. The method was trained on a 109B-parameter Mixtu…

Researchers have introduced CosmicFish-HRM, a compact language model that uses a Hierarchical Reasoning Module (HRM) to dynamically allocate computational effort based on input complexity rather than …