# Beyond the Transformer FFN: How CellularFlow Solves Catastrophic Forgetting > Source: > Published: 2026-09-06 23:41:33+00:00 *By Celcilin C S ([@celcilin](https://github.com/celcilin))* *A deep dive into replacing dense feed-forward networks with addressable DNA memory banks, achieving zero-backpropagation streaming learning, and preserving 83.9% domain retention.* If you take any state-of-the-art Large Language Model (LLaMA, Mistral, GPT-4) and train it sequentially on new domains—say, Medical notes, then Legal contracts, then Rust code—something catastrophic happens. It suffers from **Catastrophic Forgetting**. By the time the model masters Rust, its diagnostic medical reasoning has degraded significantly. In a standard Transformer block, sequence reasoning is handled by **Multi-Head Self-Attention**, but all the model's factual knowledge, vocabulary associations, and world facts are packed into dense **Feed-Forward Networks (FFN / SwiGLU / MLP)**. ``` Standard Transformer Block: Input Token ──→ [ Self-Attention ] ──→ [ Dense FFN / MLP ] ──→ Output ▲ │ All world knowledge, facts, and syntax are entangled across monolithic dense matrices! ``` Because an MLP is a dense matrix multiplication ( W2⋅act(W1x) ), **every single weight participates in every single token**. There are no "folders", no "slots", and no isolated boundaries. When you backpropagate gradients on a new domain, you rewrite the same weights that held the old domain's knowledge. To add insult to injury: What if an LLM didn't store its factual knowledge inside dense, monolithic transform matrices? What if, instead: This is the architectural thesis behind **CellularFlow**. ``` Input Sequence: X (B, T, d) │ ┌──────────────┴──────────────┐ ▼ ▼ ┌─────────────────────────┐ ┌─────────────────────────┐ │ Multi-Head DNA Memory │ │ Episodic Memory Slot │ │ Associative Banks │ │ Buffer (Fast-Write) │ └────────────┬────────────┘ └────────────┬────────────┘ │ │ └──────────────┬──────────────┘ │ (Gated Memory Enrichment) ▼ ┌───────────────────────────────────────────────────────┐ │ Causal Multi-Head Self-Attention with RoPE (FlashAttn)│ └───────────────────────────┬───────────────────────────┘ │ ▼ Output Sequence: Y (B, T, d) ``` CellularFlow fuses two computational engines into a unified **Hybrid CMC Layer**: `CMCLayer`) Instead of an MLP, each layer contains learned memory banks split across multiple independent heads ( H ). Each head maintains: When a token arrives, it computes its cosine similarity against the keys in each head subspace: Sparse Top-K routing has a famous failure mode: **dead slots**. A few initially lucky keys monopolize all the routing, while 70% of the memory bank never learns. CellularFlow injects small Gaussian exploration noise during training, ensuring that **every single memory slot receives gradient updates over time**. Following memory enrichment, sequence tokens are routed through multi-head causal self-attention powered by **FlashAttention-2** kernel dispatch (`F.scaled_dot_product_attention`). To handle sequences longer than training length without breaking, CellularFlow uses **Dynamic NTK-Aware RoPE scaling**: When sequence length exceeds the pretraining threshold ( T>2048 ), the base frequency is stretched dynamically: This enables zero-shot context length extrapolation without fine-tuning. On the final layer, an explicit key-value buffer (`EpisodicMemory`) acts as a "working memory" scratchpad: CellularFlow introduces a principled, 3-tier memory hierarchy: ``` Continual Learning Inputs │ ▼ [ Select Mode ] │ │ └────────────────────────────────┐ ▼ ▼ ▼ Mode 1: Live Learn Mode 2: Selective Fine-Tune Mode 3: Episodic Buffer (Streaming EMA) (Freeze 85% Backbone) (Fast-Write Slot Buffer) [0 Backpropagation] [Train DNA Banks Only] [Post-Epoch Consolidation] trainer.live_learn("Streaming real-time log telemetry...") trainer.selective_finetune("Technical medical notes on oncology...", epochs=10) trainer.inject_fact("The capital of Mars colony is Bradbury Landing.") ``` We benchmarked CellularFlow v4 against a standard autoregressive Transformer (GPT-mini) trained under identical conditions on a standardized multi-domain corpus. | Metric | GPT-mini (Baseline) | CellularFlow v4 (Hybrid CMC) | Advantage | |---|---|---|---| | **Parameters** | 810K | **379K** | **2.1× smaller** | | **Final Perplexity** | 8.51 | **2.54** | **−70.3% reduction** | | **Top-1 Accuracy** | 36.4% | **73.7%** | **+37.3 pp** | | **Training Steps** | 150 epochs | 150 epochs | Same compute budget | Despite having **less than half the parameters**, CellularFlow achieved a dramatic reduction in perplexity and doubled prediction accuracy, demonstrating the high parametric density of associative memory banks compared to dense MLPs. Models were trained sequentially across 5 disparate domains (*Literature* ➔ *Science* ➔ *History* ➔ *Technical* ➔ *Poetry*). After completing the final domain, retention accuracy was measured across all initial domains: ``` Domain Retention after 5 Sequential Tasks: ┌─────────────────────────────────────────────────────────────┐ │ Baseline Full Fine-Tuning: 61.8% [████████████░░░░░░░░] │ │ Mode 2 Selective Fine-Tune: 83.9% [████████████████░░░░] │ └─────────────────────────────────────────────────────────────┘ Advantage: +22.1 percentage points! ``` CellularFlow comes with an interactive glassmorphic web dashboard powered by a FastAPI backend and WebSockets. ``` uvicorn server.app:app --host 0.0.0.0 --port 8000 # Open http://localhost:8000 in your browser git clone https://github.com/celcilin/cellularflow.git cd cellularflow pip install -e . python import torch from cellularflow import CellularFlowLM, CellularFlowTrainer, BPEDataset # 1. Dataset & Model dataset = BPEDataset("Alice was beginning to get very tired...", context_len=256) model = CellularFlowLM( vocab_size=dataset.vocab, dim=512, n_layers=6, n_heads=8, n_entries=128, context_len=256 ) # 2. Pretraining trainer = CellularFlowTrainer(model, dataset, device="cuda" if torch.cuda.is_available() else "cpu") trainer.pretrain(epochs=100, seed_dna=True) # 3. Fast Incremental Generation (KV-Cache) print(trainer.generate("Alice saw a", max_new=100)) # 4. Mode 3: Instant Fact Injection trainer.inject_fact("The White Rabbit's pocket watch is made of titanium.") # 5. Mode 2: Domain Adaptation (Backbone Frozen) trainer.selective_finetune("Technical medical notes...", epochs=10) # 6. Mode 1: Forward-Pass Streaming Learning (0 Backprop) trainer.live_learn("Streaming user inputs...") ``` CellularFlow proves that language models do not have to be rigid, monolithic black boxes that forget their past whenever they learn something new. By replacing dense FFNs with multi-head associative memory banks, we can build models that: The entire codebase, training pipelines, interactive dashboard, and IEEE research paper are open source under the MIT License. `CONTRIBUTING.md` to get involved!