SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization

A method called SpectralShift extends the effective context window of Gated DeltaNet, a linear attention architecture used in place of softmax attention for long-context modeling, by reparameterizing the model's spectral properties. The approach addresses a gap in existing context extension techniques, which apply continued pretraining directly to linear attention layers without modifying them and therefore overlook those layers' spectral properties.

Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typically apply continued pretraining directly without modifying these layers, overlooking the spectral properties of linea