Engram + Transformer reached slightly lower loss than Engram + GDN hybrid — but the more important result was how much of the original gap disappeared after I fixed an indexing edge case.
I ran a 200-step, ~1.7B-parameter ablation in OLMo-core comparing Engram on two backbones:
• a standard attention Transformer
• a 3 GDN-layer + 1 attention-layer hybrid
Main observations:
• Transformer + Engram reached slightly lower CE loss and perplexity
• Transformer showed larger isolated gradient spikes; the hybrid had a choppier overall gradient profile
• Transformer achieved higher MFU; the hybrid processed more tokens per second
• GPU memory stayed flat for both runs, even when the hybrid produced elevated CUDA allocation retries
The central systems tradeoff came from parameter parity.
Matching the two models pushed the hybrid toward d_model=1280, which was less friendly to the FLA/Triton/compiler path than 1024. The result was lower MFU, lower TPS, and more allocator churn.
Then the Engram implementation surfaced two indexing failures:
• signed overflow in the hash path produced negative lookup indices
• padded token IDs exceeded the real Engram lookup range
Fixing those paths — along with moving static tensors into registered buffers — lowered loss for both backbones and substantially narrowed the apparent architecture gap.
That changed the scientific interpretation:
What initially looked like architecture behavior was partly execution artifact.
Working inside OLMo-core made the engineering bar very visible. It pushed me to treat throughput, memory behavior, compiler constraints, and clean integration as part of the research itself — not implementation plumbing after the math.
I wrote up the full training-dynamics and engineering story here:
[“When Memory Meets Memory: Engram on Transformers vs GDN Hybrids”](https://medium.com/@jenwei0312/when-memory-meets-memory-engram-on-transformers-vs-gdn-hybrids-6dbefa1b6d23?sharedUserId=jenwei0312)
Repo: [GitHub - JenWei0312/OLMo-core at feature/engram-poc · GitHub](https://github.com/JenWei0312/OLMo-core/tree/feature/engram-poc)
Runs/W&B link: [Weights & Biases](https://wandb.ai/jenwei0312/olmo3-engram-experiments)