{"slug": "olmo-core-engram-graft-2b-600m-a-debug-comparison", "title": "OLMo-core + Engram graft: 2B/600M-A debug comparison", "summary": "A researcher ran a 200-step debug comparison between a base OLMo3 600M model and a DeepSeek-style Engram memory graft variant, finding the graft stable and showing improved early learning behavior. The Engram variant added ~1B parameters but only 40k more active parameters per token, and the integration required careful handling of FSDP/HSDP wrapping and optimizer policies.", "body_md": "I ran a `200-step`\n\n, with `global_batch _size =32`\n\ndebug comparison between a base OLMo3 600M model and the same dense backbone with a DeepSeek-style Engram memory graft.\n\nThe goal was to check whether the custom module was wired correctly, whether FSDP/HSDP wrapping and optimizer handling were stable, and whether the training/eval curves looked coherent.\n\nSetup\n\nBase model:\n\n- ~676M trainable parameters\n\nEngram variant:\n\n-\n~1.7B trainable parameters\n\n-\nEngram injected into layers 1 and 5\n\n-\nMost added parameters come from sparse/hash-memory capacity, but active param per token with engram was only 40k more than the dense backbone.\n\nBoth are trained with [Dion](https://github.com/microsoft/dion/)[ ](https://github.com/microsoft/dion/)optimizer.\n\nWhat I observed\n\nUnder the same short debug setup, the Engram variant showed:\n\nThe early signal is encouraging: the Engram graft is training-shaped, stable, and appears to improve early learning behavior in this setup.\n\nMain systems lesson\n\nCustom architecture work is not just “does the forward pass run?”\n\nFor this integration, the parameter hierarchy, wrapping policy, optimizer handling, memory profile, and training curves all had to line up. Earlier versions trained mathematically, but had poor memory behavior because the custom modules were not placed inside the wrapped block hierarchy.\n\nW&B logs: [Weights & Biases](https://wandb.ai/jenwei0312/olmo3-engram-experiments)\n\nI wrote an additional tradeoff analysis from the ablation design and eval metric point of view. Cross-posting here for completeness.\n\n[Post here](https://www.linkedin.com/posts/jenweiprofile_share-machinelearning-deeplearning-ugcPost-7475318099727003649-eHys/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAXh_gcBauwhR3xIRxe6bS1hQWshSOirhVI)", "url": "https://wpnews.pro/news/olmo-core-engram-graft-2b-600m-a-debug-comparison", "canonical_source": "https://discuss.huggingface.co/t/olmo-core-engram-graft-2b-600m-a-debug-comparison/177045#post_2", "published_at": "2026-06-25 16:14:24+00:00", "updated_at": "2026-06-25 16:22:41.319219+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-infrastructure"], "entities": ["OLMo3", "DeepSeek", "Engram", "Dion", "Weights & Biases", "Microsoft"], "alternates": {"html": "https://wpnews.pro/news/olmo-core-engram-graft-2b-600m-a-debug-comparison", "markdown": "https://wpnews.pro/news/olmo-core-engram-graft-2b-600m-a-debug-comparison.md", "text": "https://wpnews.pro/news/olmo-core-engram-graft-2b-600m-a-debug-comparison.txt", "jsonld": "https://wpnews.pro/news/olmo-core-engram-graft-2b-600m-a-debug-comparison.jsonld"}}