Breaking Down Long-Context Transformer Bottlenecks
Researchers have developed a new approach to overcome the quadratic cost of causal self-attention in long-context transformers, using state update design and structural interventions like sink tokens and short convolutio…