# BCMT: Blockwise Causal Memory Transformer - Research Feedback Welcome

> Source: <https://discuss.huggingface.co/t/bcmt-blockwise-causal-memory-transformer-research-feedback-welcome/177851#post_1>
> Published: 2026-07-15 13:30:22+00:00

Hi everyone,

I’d like to share a recent research project that I’ve been working on: **BCMT (Blockwise Causal Memory Transformer).**

**BCMT** explores an alternative architecture for long-context language modeling. Instead of relying on dense global self-attention, the model combines:

The main idea is to investigate whether long-range dependencies can be modeled efficiently through compact block-level memory representations rather than explicit global token-to-token attention.

For a fixed block size, the resulting computational complexity is **O(TL)**, compared to **O(T²)** for standard dense self-attention.

The repository currently includes:

The accompanying paper presents the architectural design, mathematical formulation, and an initial experimental evaluation on WikiText-103.

In the current experiments, BCMT achieves validation perplexities close to a dense Transformer baseline while providing higher training throughput and lower GPU memory usage.

I’m particularly interested in technical feedback on:

The project is fully open source:

**Paper (DOI)**: [https://doi.org/10.20944/preprints202607.0333.v1](https://doi.org/10.20944/preprints202607.0333.v1)

Thank you very much for taking the time to read it. Any constructive comments or suggestions would be greatly appreciated.