BeamGPT: A new paradigm for attention

An unaffiliated researcher has developed BeamGPT, a new attention mechanism that achieves 73x lower training loss with nearly 4x parameter reduction compared to standard transformers. The hybrid model uses a linear-complexity field operator alongside attention, offering 2.3x savings at long context. The researcher is withholding exact details to handle the release carefully and is seeking a research environment to develop the mechanism further.

I have found an operator that achieves striking results in learning curves when used alongside standard attention in a nanoGPT-style character-level language model. It finds structure in the sequence that attention misses. The model learns a mix ratio of around 45% attention to 55% of the field operator. This ratio seems consistent across layers. This operator is linear in sequence length. Standard attention is quadratic. The hybrid scaling model gives roughly 2.3 savings at long context. As you can see this model goes from slightly more expensive than standard quadratic attention to better than it at long context. This was my first try of integrating this operator into the attention mechanism deriving from the theory. When we replace the standard MLP layers in the transformer architecture with Beam, we achieve 73x lower training loss with almost a 4x parameter reduction. Due to the significance of this mechanism, I have decided against publicly disclosing the exact notation of this field operator. I believe this architectural improvement represents a step change in pretraining efficiency, and I want to handle its release carefully. I am currently unaffiliated. I am looking for the right research environment to develop this mechanism further. I am willing to share more details upon contact.