{"slug": "my-latest-ablation-run-integrating-engram-onto-two-backbones", "title": "My latest ablation run: integrating Engram onto two backbones", "summary": "A researcher at OLMo-core ran a 200-step, ~1.7B-parameter ablation comparing Engram on a standard Transformer versus a GDN hybrid backbone, finding that the Transformer achieved slightly lower loss and higher MFU. After fixing indexing bugs in the Engram implementation, the performance gap between the two architectures narrowed significantly, revealing that the initial differences were partly execution artifacts rather than pure architectural behavior.", "body_md": "Engram + Transformer reached slightly lower loss than Engram + GDN hybrid — but the more important result was how much of the original gap disappeared after I fixed an indexing edge case.\n\nI ran a 200-step, ~1.7B-parameter ablation in OLMo-core comparing Engram on two backbones:\n\n• a standard attention Transformer\n\n• a 3 GDN-layer + 1 attention-layer hybrid\n\nMain observations:\n\n• Transformer + Engram reached slightly lower CE loss and perplexity\n\n• Transformer showed larger isolated gradient spikes; the hybrid had a choppier overall gradient profile\n\n• Transformer achieved higher MFU; the hybrid processed more tokens per second\n\n• GPU memory stayed flat for both runs, even when the hybrid produced elevated CUDA allocation retries\n\nThe central systems tradeoff came from parameter parity.\n\nMatching the two models pushed the hybrid toward `d_model=1280`, which was less friendly to the FLA/Triton/compiler path than `1024`. The result was lower MFU, lower TPS, and more allocator churn.\n\nThen the Engram implementation surfaced two indexing failures:\n\n• signed overflow in the hash path produced negative lookup indices\n\n• padded token IDs exceeded the real Engram lookup range\n\nFixing those paths — along with moving static tensors into registered buffers — lowered loss for both backbones and substantially narrowed the apparent architecture gap.\n\nThat changed the scientific interpretation:\n\nWhat initially looked like architecture behavior was partly execution artifact.\n\nWorking inside OLMo-core made the engineering bar very visible. It pushed me to treat throughput, memory behavior, compiler constraints, and clean integration as part of the research itself — not implementation plumbing after the math.\n\nI wrote up the full training-dynamics and engineering story here:\n\n[“When Memory Meets Memory: Engram on Transformers vs GDN Hybrids”](https://medium.com/@jenwei0312/when-memory-meets-memory-engram-on-transformers-vs-gdn-hybrids-6dbefa1b6d23?sharedUserId=jenwei0312)\n\nRepo: [GitHub - JenWei0312/OLMo-core at feature/engram-poc · GitHub](https://github.com/JenWei0312/OLMo-core/tree/feature/engram-poc)\n\nRuns/W&B link: [Weights & Biases](https://wandb.ai/jenwei0312/olmo3-engram-experiments)", "url": "https://wpnews.pro/news/my-latest-ablation-run-integrating-engram-onto-two-backbones", "canonical_source": "https://discuss.huggingface.co/t/my-latest-ablation-run-integrating-engram-onto-two-backbones/177873#post_1", "published_at": "2026-07-16 02:58:05+00:00", "updated_at": "2026-07-16 03:07:25.925883+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks", "ai-research", "developer-tools"], "entities": ["Engram", "OLMo-core", "Transformer", "GDN", "FLA", "Triton", "JenWei0312", "Weights & Biases"], "alternates": {"html": "https://wpnews.pro/news/my-latest-ablation-run-integrating-engram-onto-two-backbones", "markdown": "https://wpnews.pro/news/my-latest-ablation-run-integrating-engram-onto-two-backbones.md", "text": "https://wpnews.pro/news/my-latest-ablation-run-integrating-engram-onto-two-backbones.txt", "jsonld": "https://wpnews.pro/news/my-latest-ablation-run-integrating-engram-onto-two-backbones.jsonld"}}