{"slug": "show-hn-cellularflow-continual-learning-llm-using-associative-memory", "title": "Show HN: CellularFlow – Continual-learning LLM using associative memory", "summary": "CellularFlow, a continual-learning LLM architecture from developer celcilin, replaces dense feed-forward networks with Multi-Head Associative DNA Memory Banks and an Episodic Memory Slot Buffer, achieving 83.9% retention across sequential domains versus 61.8% for standard Transformers, and enabling zero-backprop streaming learning during inference. The open-source project, available on GitHub, supports three modes: live learning via EMA, selective fine-tuning that freezes ~85% of the backbone, and episodic fact injection with temporal decay.", "body_md": "**CellularFlow** is a memory-augmented neural architecture designed as a continual-learning alternative to standard Transformers. By replacing dense Feed-Forward Networks (FFN/MLP) with **Multi-Head Associative DNA Memory Banks** and an **Episodic Memory Slot Buffer**, CellularFlow decouples factual knowledge storage from sequence reasoning.\n\nIt achieves **state-of-the-art catastrophic forgetting mitigation (83.9% retention across sequential domains)** and enables **zero-backprop streaming learning** during inference.\n\n| Feature | Standard Transformer (LLaMA/GPT) | CellularFlow v4 | \n|---|---|---|\n| **Parametric Architecture** | Dense FFN / SwiGLU | **Multi-Head Associative DNA Memory (`CMCLayer`)** | \n| **Sequential Adaptation** | Severe catastrophic forgetting (61.8% retention) | **83.9% retention via Selective Fine-Tuning (Mode 2)** | \n| **Real-time Live Learning** | ❌ Impossible without retraining | ✅ **Mode 1: EMA streaming forward update (0 backprop)** | \n| **Instant Fact Injection** | ❌ Requires finetuning or external RAG | ✅ **Mode 3: Episodic slot buffer with decay & consolidation** | \n| **Sequence Attention** |  |  | \n| **Inference Efficiency** | Full recompute or dense KV cache | **Decoupled memory lookup + incremental KV-cache** | \n| **Knowledge Inspectability** | Diffuse, entangled weights | **Discrete, addressable, and prunable memory slots** | \n\nCellularFlow fuses two computational pathways into a unified **Hybrid CMC Layer**:\n\n```\n                       Input Sequence: X (B, T, d)\n                                   │\n                    ┌──────────────┴──────────────┐\n                    ▼                             ▼\n       ┌─────────────────────────┐   ┌─────────────────────────┐\n       │   Multi-Head DNA Memory │   │   Episodic Memory Slot  │\n       │   Associative Banks     │   │   Buffer (Fast-Write)   │\n       └────────────┬────────────┘   └────────────┬────────────┘\n                    │                             │\n                    └──────────────┬──────────────┘\n                                   │ (Gated Memory Enrichment)\n                                   ▼\n       ┌───────────────────────────────────────────────────────┐\n       │  Causal Multi-Head Self-Attention with RoPE (FlashAttn)│\n       └───────────────────────────┬───────────────────────────┘\n                                   │\n                                   ▼\n                       Output Sequence: Y (B, T, d)\n```\n\nEach head (\n\n- **Specialized Subspaces:** Heads specialize independently across syntax, semantics, and domain knowledge.\n- **Exploration Noise:** Gaussian perturbation prevents dead memory slots during Top-K sparse routing.\n\n- **Mode 1 — Live Learning (`inference_write=True`):** Updates DNA memory values on the fly during inference via Exponential Moving Average (EMA) with zero backward pass. Protected by**Spherical Anisotropy Regularization** to prevent vector collapse.\n- **Mode 2 — Selective Fine-Tuning (`set_mode(\"selective\")`):** Freezes ~85% of the backbone (projections, embeddings, LayerNorms) and trains only the DNA banks. Retains foundational knowledge while rapidly absorbing new domains.\n- **Mode 3 — Episodic Fact Injection (`inject_fact`):** Writes facts into slot-based episodic memory with temporal age decay (`exp(-0.005 * age)` ) and consolidates top facts into DNA banks post-epoch.\n\nRequires **Python  $\\ge$ 3.11** and \n\n**PyTorch$\\ge$ 2.4.0**.\n\n```\n# Clone the repository\ngit clone https://github.com/celcilin/cellularflow.git\ncd cellularflow\n\n# Install dependencies using UV (recommended) or pip\npip install -e .\n\n# For GPU acceleration (CUDA 12.4+):\npip install torch --index-url https://download.pytorch.org/whl/cu124\npython\nimport torch\nfrom cellularflow import CellularFlowLM, CellularFlowTrainer, BPEDataset\n\n# 1. Initialize tokenizer & dataset\ndataset = BPEDataset(\"Alice was beginning to get very tired of sitting by her sister...\", context_len=256)\n\n# 2. Instantiate CellularFlow LM\nmodel = CellularFlowLM(\n    vocab_size   = dataset.vocab,\n    dim          = 512,\n    n_layers     = 6,\n    n_heads      = 8,\n    n_entries    = 128,\n    context_len  = 256,\n    use_episodic = True\n)\n\n# 3. Pretraining\ntrainer = CellularFlowTrainer(model, dataset, device=\"cuda\" if torch.cuda.is_available() else \"cpu\")\ntrainer.pretrain(epochs=100, seed_dna=True)\n\n# 4. Fast Generation (with KV-Cache)\nprompt = \"The journey into\"\nprint(trainer.generate(prompt, max_new=100, temperature=0.8))\n\n# 5. Continual Learning: Mode 3 Fact Injection\ntrainer.inject_fact(\"The hidden archives are kept inside Vault 42.\")\n\n# 6. Continual Learning: Mode 2 Selective Fine-Tuning (Backbone Frozen)\ntrainer.selective_finetune(\"Technical medical notes on neurology...\", epochs=10)\n\n# 7. Continual Learning: Mode 1 Live Streaming Learning (0 Backprop)\ntrainer.live_learn(\"Streaming log telemetry received in real time...\")\n```\n\nCellularFlow includes an interactive glassmorphic web dashboard for real-time inference, fact injection, and memory inspection:\n\n```\n# Start the FastAPI server\nuvicorn server.app:app --host 0.0.0.0 --port 8000\n```\n\nOpen **`http://localhost:8000`** in your browser to interactively generate text, inspect layer-wise episodic slot utilization, and test live fact injections.\n\n```\n# Interactive CLI Playground\npython analysis.py --checkpoint checkpoint/CMC_BaseModel.pt --interactive\n```\n\nEvaluated on a standardized 62KB multi-domain corpus:\n\n| Architecture | Parameters | Perplexity | Accuracy | \n|---|---|---|---|\n| **GPT-mini (Vanilla Transformer)** | 810K | 8.51 | 36.4% | \n| **CellularFlow v4 (Hybrid CMC)** | **379K (2.1× fewer)** | **2.54 (−70.3%)** | **73.7%** | \n\nTrained sequentially across *Literature*, *Science*, *History*, *Technical*, and *Poetry*:\n\n| Fine-Tuning Strategy | Overall Domain Retention | \n|---|---|\n| Full Fine-Tuning (All Weights) | 61.8% | \n| **Mode 2: Selective DNA Fine-Tuning** | **83.9% (+22.1 pp)** | \n\n```\ncellularflow/\n├── cellularflow/\n│   ├── core.py               # CMCLayer, HybridCMCLayer, EpisodicMemory, CellularFlowLM\n│   ├── trainer.py            # Pretraining, selective fine-tuning, live learning, mixed precision\n│   ├── extensions.py         # Blockwise Attention, Compressed KV (CKV), MTP, Beaconing\n│   ├── corpus.py             # Multi-domain benchmark corpora\n│   └── swarm.py              # DNASwarm evolutionary optimizer\n├── benchmarks/\n│   └── evaluate_checkpoint.py# Evaluation harness for perplexity, accuracy, and memory norms\n├── server/\n│   └── app.py                # FastAPI server + WebSocket endpoint\n├── dashboard/\n│   ├── index.html            # Web dashboard UI\n│   ├── app.js                # Frontend WebSocket and API client\n│   └── styles.css            # Dark glassmorphic design system\n├── sft/\n│   ├── sft_dataset.py        # ChatML templates and target loss masking\n│   └── sft_trainer.py        # Supervised fine-tuning curriculum engine\n├── scripts/\n│   ├── train_tokenizer.py    # ByteLevelBPE tokenizer builder\n│   └── test_extensions.py    # Architecture extension verification\n├── analysis.py               # CLI exploration & interactive REPL\n├── pyproject.toml            # Project build & dependency definitions\n└── CONTRIBUTING.md           # Contribution guidelines & developer standards\n```\n\nWe welcome contributions from researchers, engineers, and developers worldwide! Please review [CONTRIBUTING.md](/celcilin/cellularflow/blob/main/CONTRIBUTING.md) for instructions on setting up your environment, adhering to XLA/TPU graph rules, and submitting pull requests.\n\n**Celcilin C S**\n\n- GitHub: [@celcilin](https://github.com/celcilin)\n- Email: [celcilin204@gmail.com](mailto:celcilin204@gmail.com)\n\nThis project is licensed under the MIT License — see the [LICENSE](/celcilin/cellularflow/blob/main/LICENSE) file for details.", "url": "https://wpnews.pro/news/show-hn-cellularflow-continual-learning-llm-using-associative-memory", "canonical_source": "https://github.com/celcilin/cellularflow", "published_at": "2026-09-06 23:16:18+00:00", "updated_at": "2026-09-06 23:32:09.557783+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence"], "entities": ["CellularFlow", "celcilin", "GitHub", "PyTorch", "Python"], "alternates": {"html": "https://wpnews.pro/news/show-hn-cellularflow-continual-learning-llm-using-associative-memory", "markdown": "https://wpnews.pro/news/show-hn-cellularflow-continual-learning-llm-using-associative-memory.md", "text": "https://wpnews.pro/news/show-hn-cellularflow-continual-learning-llm-using-associative-memory.txt", "jsonld": "https://wpnews.pro/news/show-hn-cellularflow-continual-learning-llm-using-associative-memory.jsonld"}}