SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
Researchers introduced Simple Attention Sparsification (SAS), a gated sparse attention mechanism that optimizes context ranking end-to-end with the language modeling loss by injecting the selector's c…