BCMT: Blockwise Causal Memory Transformer - Research Feedback Welcome

A researcher introduced BCMT (Blockwise Causal Memory Transformer), a novel architecture for long-context language modeling that uses block-level memory representations to achieve O(TL) complexity instead of O(T²). The model achieves validation perplexities close to dense Transformers on WikiText-103 while offering higher training throughput and lower GPU memory usage. The project is open source and the paper is available via DOI.

Hi everyone, I’d like to share a recent research project that I’ve been working on: BCMT Blockwise Causal Memory Transformer . BCMT explores an alternative architecture for long-context language modeling. Instead of relying on dense global self-attention, the model combines: The main idea is to investigate whether long-range dependencies can be modeled efficiently through compact block-level memory representations rather than explicit global token-to-token attention. For a fixed block size, the resulting computational complexity is O TL , compared to O T² for standard dense self-attention. The repository currently includes: The accompanying paper presents the architectural design, mathematical formulation, and an initial experimental evaluation on WikiText-103. In the current experiments, BCMT achieves validation perplexities close to a dense Transformer baseline while providing higher training throughput and lower GPU memory usage. I’m particularly interested in technical feedback on: The project is fully open source: Paper DOI : https://doi.org/10.20944/preprints202607.0333.v1 https://doi.org/10.20944/preprints202607.0333.v1 Thank you very much for taking the time to read it. Any constructive comments or suggestions would be greatly appreciated.