Hi everyone,
I’d like to share a recent research project that I’ve been working on: BCMT (Blockwise Causal Memory Transformer).
BCMT explores an alternative architecture for long-context language modeling. Instead of relying on dense global self-attention, the model combines:
The main idea is to investigate whether long-range dependencies can be modeled efficiently through compact block-level memory representations rather than explicit global token-to-token attention.
For a fixed block size, the resulting computational complexity is O(TL), compared to O(T²) for standard dense self-attention. The repository currently includes:
The accompanying paper presents the architectural design, mathematical formulation, and an initial experimental evaluation on WikiText-103.
In the current experiments, BCMT achieves validation perplexities close to a dense Transformer baseline while providing higher training throughput and lower GPU memory usage.
I’m particularly interested in technical feedback on:
The project is fully open source:
Paper (DOI): https://doi.org/10.20944/preprints202607.0333.v1 Thank you very much for taking the time to read it. Any constructive comments or suggestions would be greatly appreciated.