{"slug": "widendepth-a-separation-of-knowledge-storage-from-iterative-reasoning", "title": "WideNDepth — A separation of knowledge storage from iterative reasoning", "summary": "A researcher introduced WideNDepth (WND), a neural architecture that separates knowledge storage from iterative reasoning, outperforming a 600K-parameter Transformer on graph reasoning tasks with only 100K parameters. The architecture uses a Wide Layer and Encoder for knowledge and a Depth Layer for reasoning, but the researcher faces challenges in preventing the knowledge components from taking over reasoning tasks.", "body_md": "Hello, this is my first research paper that I have been working on, and I would greatly appreciate any feedback.\n\nI’ve been working on a neural architecture that I call WND (WideNDepth). The purpose of this architecture is to separate “Reasoning” from “Knowledge.”\n\nBut why?\n\nAs we see in today’s language models, the parameters are expected to both contain knowledge and be able to reason. I consider this a waste of parameters. For example, imagine a 1M parameter Transformer:\n\n```\n###\n###\n###\n```\n\nThis is a static multi-layer stack of parameters that is expected to both reason and store knowledge at the same time.\n\nNow let’s look at how WND separates this idea:\n\n``` php\n#\n# -> ###\n#\n```\n\nIn other words, one part is mainly responsible for knowledge, while another part is mainly responsible for reasoning.\n\n```\nInput\n\n  │\n  ▼\n\nWide Layer\n\n  │\n  ▼\n\nEncoder ───────────────► Feature Bank\n                             ▲\n  │                          │\n  │                          │\n  ▼                          │\n                             │ \nCompressor                Attention\n                             ▲\n  │                          │\n  ▼                          │\n                             │\nDepth Layer (×N iterations) ─┘\n  \n  │        \n  ▼\n\nOutput\n```\n\nTo explain it simply:\n\nThe goal is for the Depth Layer to handle reasoning step-by-step.\n\nFrom what I have tested so far, on graph reasoning tasks **(long chains, medium difficulty)**, a **~100K parameter WND** model is currently outperforming a **~600K Transformer** baseline. It also trains faster, is more sample-efficient, and has a smaller footprint.\n\nThat said, **I should mention that WND has not yet been tested on other tasks**, so at this stage I do not know where it truly excels and where its weaknesses are.\n\nThe main **issue I am facing right now is that the Wide Layer and the Encoder keep taking over most of the work**, leaving the Depth Layer almost idle. I tried adding auxiliary weights, but that did not really solve the issue.\n\nHow can I encourage, constrain, or force the\n\nWide LayerandEncoderto focus only on producing rich and useful information for theDepth Layer, while making sure that the actual reasoning is done by theDepth Layeritself?", "url": "https://wpnews.pro/news/widendepth-a-separation-of-knowledge-storage-from-iterative-reasoning", "canonical_source": "https://discuss.huggingface.co/t/widendepth-a-separation-of-knowledge-storage-from-iterative-reasoning/177809#post_1", "published_at": "2026-07-14 09:30:18+00:00", "updated_at": "2026-07-14 11:00:11.710802+00:00", "lang": "en", "topics": ["neural-networks", "machine-learning", "ai-research"], "entities": ["WideNDepth", "WND", "Transformer"], "alternates": {"html": "https://wpnews.pro/news/widendepth-a-separation-of-knowledge-storage-from-iterative-reasoning", "markdown": "https://wpnews.pro/news/widendepth-a-separation-of-knowledge-storage-from-iterative-reasoning.md", "text": "https://wpnews.pro/news/widendepth-a-separation-of-knowledge-storage-from-iterative-reasoning.txt", "jsonld": "https://wpnews.pro/news/widendepth-a-separation-of-knowledge-storage-from-iterative-reasoning.jsonld"}}