Beyond the Transformer FFN: How CellularFlow Solves Catastrophic Forgetting

By Celcilin C S (@celcilin)

A deep dive into replacing dense feed-forward networks with addressable DNA memory banks, achieving zero-backpropagation streaming learning, and preserving 83.9% domain retention.

If you take any state-of-the-art Large Language Model (LLaMA, Mistral, GPT-4) and train it sequentially on new domains—say, Medical notes, then Legal contracts, then Rust code—something catastrophic happens.

It suffers from Catastrophic Forgetting. By the time the model masters Rust, its diagnostic medical reasoning has degraded significantly.

In a standard Transformer block, sequence reasoning is handled by Multi-Head Self-Attention, but all the model's factual knowledge, vocabulary associations, and world facts are packed into dense Feed-Forward Networks (FFN / SwiGLU / MLP).

Standard Transformer Block:
Input Token ──→ [ Self-Attention ] ──→ [ Dense FFN / MLP ] ──→ Output
                                             ▲
                                             │
               All world knowledge, facts, and syntax are
               entangled across monolithic dense matrices!

Because an MLP is a dense matrix multiplication ( W2⋅act(W1x) ), every single weight participates in every single token. There are no "folders", no "slots", and no isolated boundaries. When you backpropagate gradients on a new domain, you rewrite the same weights that held the old domain's knowledge.

To add insult to injury:

What if an LLM didn't store its factual knowledge inside dense, monolithic transform matrices?

What if, instead:

This is the architectural thesis behind CellularFlow.

                       Input Sequence: X (B, T, d)
                                   │
                    ┌──────────────┴──────────────┐
                    ▼                             ▼
       ┌─────────────────────────┐   ┌─────────────────────────┐
       │   Multi-Head DNA Memory │   │   Episodic Memory Slot  │
       │   Associative Banks     │   │   Buffer (Fast-Write)   │
       └────────────┬────────────┘   └────────────┬────────────┘
                    │                             │
                    └──────────────┬──────────────┘
                                   │ (Gated Memory Enrichment)
                                   ▼
       ┌───────────────────────────────────────────────────────┐
       │  Causal Multi-Head Self-Attention with RoPE (FlashAttn)│
       └───────────────────────────┬───────────────────────────┘
                                   │
                                   ▼
                       Output Sequence: Y (B, T, d)

CellularFlow fuses two computational engines into a unified Hybrid CMC Layer:

CMCLayer) Instead of an MLP, each layer contains learned memory banks split across multiple independent heads ( H ).

Each head maintains:

When a token arrives, it computes its cosine similarity against the keys in each head subspace:

Sparse Top-K routing has a famous failure mode: dead slots. A few initially lucky keys monopolize all the routing, while 70% of the memory bank never learns. CellularFlow injects small Gaussian exploration noise during training, ensuring that every single memory slot receives gradient updates over time.

Following memory enrichment, sequence tokens are routed through multi-head causal self-attention powered by FlashAttention-2 kernel dispatch (F.scaled_dot_product_attention).

To handle sequences longer than training length without breaking, CellularFlow uses Dynamic NTK-Aware RoPE scaling:

When sequence length exceeds the pretraining threshold ( T>2048 ), the base frequency is stretched dynamically:

This enables zero-shot context length extrapolation without fine-tuning.

On the final layer, an explicit key-value buffer (EpisodicMemory) acts as a "working memory" scratchpad:

CellularFlow introduces a principled, 3-tier memory hierarchy:

               Continual Learning Inputs
                           │
                           ▼
                    [ Select Mode ]
                     │      │      └────────────────────────────────┐
                     ▼      ▼                                       ▼
         Mode 1: Live Learn    Mode 2: Selective Fine-Tune   Mode 3: Episodic Buffer
         (Streaming EMA)       (Freeze 85% Backbone)         (Fast-Write Slot Buffer)
         [0 Backpropagation]   [Train DNA Banks Only]        [Post-Epoch Consolidation]
trainer.live_learn("Streaming real-time log telemetry...")
trainer.selective_finetune("Technical medical notes on oncology...", epochs=10)
trainer.inject_fact("The capital of Mars colony is Bradbury Landing.")

We benchmarked CellularFlow v4 against a standard autoregressive Transformer (GPT-mini) trained under identical conditions on a standardized multi-domain corpus.

Despite having less than half the parameters, CellularFlow achieved a dramatic reduction in perplexity and doubled prediction accuracy, demonstrating the high parametric density of associative memory banks compared to dense MLPs.

Models were trained sequentially across 5 disparate domains (Literature ➔ Science ➔ History ➔ Technical ➔ Poetry). After completing the final domain, retention accuracy was measured across all initial domains:

Domain Retention after 5 Sequential Tasks:
┌─────────────────────────────────────────────────────────────┐
│ Baseline Full Fine-Tuning:   61.8% [████████████░░░░░░░░]   │
│ Mode 2 Selective Fine-Tune:  83.9% [████████████████░░░░]   │
└─────────────────────────────────────────────────────────────┘
                Advantage: +22.1 percentage points!

CellularFlow comes with an interactive glassmorphic web dashboard powered by a FastAPI backend and WebSockets.

uvicorn server.app:app --host 0.0.0.0 --port 8000
git clone https://github.com/celcilin/cellularflow.git
cd cellularflow
pip install -e .
python
import torch
from cellularflow import CellularFlowLM, CellularFlowTrainer, BPEDataset

dataset = BPEDataset("Alice was beginning to get very tired...", context_len=256)
model = CellularFlowLM(
    vocab_size=dataset.vocab,
    dim=512,
    n_layers=6,
    n_heads=8,
    n_entries=128,
    context_len=256
)

trainer = CellularFlowTrainer(model, dataset, device="cuda" if torch.cuda.is_available() else "cpu")
trainer.pretrain(epochs=100, seed_dna=True)

print(trainer.generate("Alice saw a", max_new=100))

trainer.inject_fact("The White Rabbit's pocket watch is made of titanium.")

trainer.selective_finetune("Technical medical notes...", epochs=10)

trainer.live_learn("Streaming user inputs...")

CellularFlow proves that language models do not have to be rigid, monolithic black boxes that forget their past whenever they learn something new.

By replacing dense FFNs with multi-head associative memory banks, we can build models that:

The entire codebase, training pipelines, interactive dashboard, and IEEE research paper are open source under the MIT License.

CONTRIBUTING.md to get involved!