Beyond the Transformer FFN: How CellularFlow Solves Catastrophic Forgetting

Engineer Celcilin C S introduced CellularFlow, a novel architecture that replaces dense feed-forward networks in transformers with addressable DNA memory banks to solve catastrophic forgetting. The design achieves zero-backpropagation streaming learning and preserves 83.9% domain retention, offering a potential breakthrough for continual learning in large language models.

By Celcilin C S @celcilin https://github.com/celcilin A deep dive into replacing dense feed-forward networks with addressable DNA memory banks, achieving zero-backpropagation streaming learning, and preserving 83.9% domain retention. If you take any state-of-the-art Large Language Model LLaMA, Mistral, GPT-4 and train it sequentially on new domains—say, Medical notes, then Legal contracts, then Rust code—something catastrophic happens. It suffers from Catastrophic Forgetting . By the time the model masters Rust, its diagnostic medical reasoning has degraded significantly. In a standard Transformer block, sequence reasoning is handled by Multi-Head Self-Attention , but all the model's factual knowledge, vocabulary associations, and world facts are packed into dense Feed-Forward Networks FFN / SwiGLU / MLP . Standard Transformer Block: Input Token ──→ Self-Attention ──→ Dense FFN / MLP ──→ Output ▲ │ All world knowledge, facts, and syntax are entangled across monolithic dense matrices Because an MLP is a dense matrix multiplication W2⋅act W1x , every single weight participates in every single token . There are no "folders", no "slots", and no isolated boundaries. When you backpropagate gradients on a new domain, you rewrite the same weights that held the old domain's knowledge. To add insult to injury: What if an LLM didn't store its factual knowledge inside dense, monolithic transform matrices? What if, instead: This is the architectural thesis behind CellularFlow . Input Sequence: X B, T, d │ ┌──────────────┴──────────────┐ ▼ ▼ ┌─────────────────────────┐ ┌─────────────────────────┐ │ Multi-Head DNA Memory │ │ Episodic Memory Slot │ │ Associative Banks │ │ Buffer Fast-Write │ └────────────┬────────────┘ └────────────┬────────────┘ │ │ └──────────────┬──────────────┘ │ Gated Memory Enrichment ▼ ┌───────────────────────────────────────────────────────┐ │ Causal Multi-Head Self-Attention with RoPE FlashAttn │ └───────────────────────────┬───────────────────────────┘ │ ▼ Output Sequence: Y B, T, d CellularFlow fuses two computational engines into a unified Hybrid CMC Layer : CMCLayer Instead of an MLP, each layer contains learned memory banks split across multiple independent heads H . Each head maintains: When a token arrives, it computes its cosine similarity against the keys in each head subspace: Sparse Top-K routing has a famous failure mode: dead slots . A few initially lucky keys monopolize all the routing, while 70% of the memory bank never learns. CellularFlow injects small Gaussian exploration noise during training, ensuring that every single memory slot receives gradient updates over time . Following memory enrichment, sequence tokens are routed through multi-head causal self-attention powered by FlashAttention-2 kernel dispatch F.scaled dot product attention . To handle sequences longer than training length without breaking, CellularFlow uses Dynamic NTK-Aware RoPE scaling : When sequence length exceeds the pretraining threshold T 2048 , the base frequency is stretched dynamically: This enables zero-shot context length extrapolation without fine-tuning. On the final layer, an explicit key-value buffer EpisodicMemory acts as a "working memory" scratchpad: CellularFlow introduces a principled, 3-tier memory hierarchy: Continual Learning Inputs │ ▼ Select Mode │ │ └────────────────────────────────┐ ▼ ▼ ▼ Mode 1: Live Learn Mode 2: Selective Fine-Tune Mode 3: Episodic Buffer Streaming EMA Freeze 85% Backbone Fast-Write Slot Buffer 0 Backpropagation Train DNA Banks Only Post-Epoch Consolidation trainer.live learn "Streaming real-time log telemetry..." trainer.selective finetune "Technical medical notes on oncology...", epochs=10 trainer.inject fact "The capital of Mars colony is Bradbury Landing." We benchmarked CellularFlow v4 against a standard autoregressive Transformer GPT-mini trained under identical conditions on a standardized multi-domain corpus. | Metric | GPT-mini Baseline | CellularFlow v4 Hybrid CMC | Advantage | |---|---|---|---| | Parameters | 810K | 379K | 2.1× smaller | | Final Perplexity | 8.51 | 2.54 | −70.3% reduction | | Top-1 Accuracy | 36.4% | 73.7% | +37.3 pp | | Training Steps | 150 epochs | 150 epochs | Same compute budget | Despite having less than half the parameters , CellularFlow achieved a dramatic reduction in perplexity and doubled prediction accuracy, demonstrating the high parametric density of associative memory banks compared to dense MLPs. Models were trained sequentially across 5 disparate domains Literature ➔ Science ➔ History ➔ Technical ➔ Poetry . After completing the final domain, retention accuracy was measured across all initial domains: Domain Retention after 5 Sequential Tasks: ┌─────────────────────────────────────────────────────────────┐ │ Baseline Full Fine-Tuning: 61.8% ████████████░░░░░░░░ │ │ Mode 2 Selective Fine-Tune: 83.9% ████████████████░░░░ │ └─────────────────────────────────────────────────────────────┘ Advantage: +22.1 percentage points CellularFlow comes with an interactive glassmorphic web dashboard powered by a FastAPI backend and WebSockets. uvicorn server.app:app --host 0.0.0.0 --port 8000 Open http://localhost:8000 in your browser git clone https://github.com/celcilin/cellularflow.git cd cellularflow pip install -e . python import torch from cellularflow import CellularFlowLM, CellularFlowTrainer, BPEDataset 1. Dataset & Model dataset = BPEDataset "Alice was beginning to get very tired...", context len=256 model = CellularFlowLM vocab size=dataset.vocab, dim=512, n layers=6, n heads=8, n entries=128, context len=256 2. Pretraining trainer = CellularFlowTrainer model, dataset, device="cuda" if torch.cuda.is available else "cpu" trainer.pretrain epochs=100, seed dna=True 3. Fast Incremental Generation KV-Cache print trainer.generate "Alice saw a", max new=100 4. Mode 3: Instant Fact Injection trainer.inject fact "The White Rabbit's pocket watch is made of titanium." 5. Mode 2: Domain Adaptation Backbone Frozen trainer.selective finetune "Technical medical notes...", epochs=10 6. Mode 1: Forward-Pass Streaming Learning 0 Backprop trainer.live learn "Streaming user inputs..." CellularFlow proves that language models do not have to be rigid, monolithic black boxes that forget their past whenever they learn something new. By replacing dense FFNs with multi-head associative memory banks, we can build models that: The entire codebase, training pipelines, interactive dashboard, and IEEE research paper are open source under the MIT License. CONTRIBUTING.md to get involved