Gaussian Mixture Attention: Linear-Time Sequence Mixing via Probabilistic Latent Routing
Researchers introduced Gaussian Mixture Attention (GMA), a probabilistic attention mechanism that replaces pairwise query-key comparisons with routing through K learned Gaussian mixture components, achieving linear-time …