ResonatorLM: Causal Resonant Field Mixing for Efficient Long-Context Language Modelin

Researchers introduced ResonatorLM, a new language model mechanism that replaces transformer self-attention with causal functions of damped resonators derived from physics. In a 6M parameter setting, ResonatorLM achieved up to 6.47x faster decoding at 32K tokens and 61.31% accuracy on WikiText, outperforming standard transformers in long-context efficiency.

arXiv:2607.05583v1 Announce Type: new Abstract: Contemporary language models are dominated by the transformer architecture, which leverages self-attention mechanisms to enable more efficient, parallelized training across a wide set of documents and corpora. This has allowed transformers to effectively model data across a wide range of modalities and contexts. However, transformers, along with their conventional counterparts such as recurrent neural networks RNNs and convolutional neural networks CNNs , often struggle to maintain efficiency when processing long contexts. We introduce ResonatorLM, a new mechanism that replaces attention with a physics-derived alternative. ResonatorLM treats token sequences as a single, driven one-dimensional latent field and replaces attention dot products with causal functions of damped resonators. We implement ResonatorLM on a traditional network architecture and test it on standard long-context modeling tasks. We find that in a small, 6M matched setting, training and prefill speedups increase with sequence length, decode speed reaches 6.47x compared to that of a standard, optimized transformer at 32K tokens, and accuracy reaches 61.31 percent compared to 55.32 percent on WikiText.