A method called SpectralShift extends the effective context window of Gated DeltaNet, a linear attention architecture used in place of softmax attention for long-context modeling, by reparameterizing the model's spectral properties. The approach addresses a gap in existing context extension techniques, which apply continued pretraining directly to linear attention layers without modifying them and therefore overlook those layers' spectral properties.
Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typically apply continued pretraining directly without modifying these layers, overlooking the spectral properties of linea
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.