{"slug": "bcmt-blockwise-causal-memory-transformer-research-feedback-welcome", "title": "BCMT: Blockwise Causal Memory Transformer - Research Feedback Welcome", "summary": "A researcher introduced BCMT (Blockwise Causal Memory Transformer), a novel architecture for long-context language modeling that uses block-level memory representations to achieve O(TL) complexity instead of O(T²). The model achieves validation perplexities close to dense Transformers on WikiText-103 while offering higher training throughput and lower GPU memory usage. The project is open source and the paper is available via DOI.", "body_md": "Hi everyone,\n\nI’d like to share a recent research project that I’ve been working on: **BCMT (Blockwise Causal Memory Transformer).**\n\n**BCMT** explores an alternative architecture for long-context language modeling. Instead of relying on dense global self-attention, the model combines:\n\nThe main idea is to investigate whether long-range dependencies can be modeled efficiently through compact block-level memory representations rather than explicit global token-to-token attention.\n\nFor a fixed block size, the resulting computational complexity is **O(TL)**, compared to **O(T²)** for standard dense self-attention.\n\nThe repository currently includes:\n\nThe accompanying paper presents the architectural design, mathematical formulation, and an initial experimental evaluation on WikiText-103.\n\nIn the current experiments, BCMT achieves validation perplexities close to a dense Transformer baseline while providing higher training throughput and lower GPU memory usage.\n\nI’m particularly interested in technical feedback on:\n\nThe project is fully open source:\n\n**Paper (DOI)**: [https://doi.org/10.20944/preprints202607.0333.v1](https://doi.org/10.20944/preprints202607.0333.v1)\n\nThank you very much for taking the time to read it. Any constructive comments or suggestions would be greatly appreciated.", "url": "https://wpnews.pro/news/bcmt-blockwise-causal-memory-transformer-research-feedback-welcome", "canonical_source": "https://discuss.huggingface.co/t/bcmt-blockwise-causal-memory-transformer-research-feedback-welcome/177851#post_1", "published_at": "2026-07-15 13:30:22+00:00", "updated_at": "2026-07-15 13:30:56.704321+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["BCMT", "WikiText-103"], "alternates": {"html": "https://wpnews.pro/news/bcmt-blockwise-causal-memory-transformer-research-feedback-welcome", "markdown": "https://wpnews.pro/news/bcmt-blockwise-causal-memory-transformer-research-feedback-welcome.md", "text": "https://wpnews.pro/news/bcmt-blockwise-causal-memory-transformer-research-feedback-welcome.txt", "jsonld": "https://wpnews.pro/news/bcmt-blockwise-causal-memory-transformer-research-feedback-welcome.jsonld"}}