Hello, this is my first research paper that I have been working on, and I would greatly appreciate any feedback.
I’ve been working on a neural architecture that I call WND (WideNDepth). The purpose of this architecture is to separate “Reasoning” from “Knowledge.”
But why?
As we see in today’s language models, the parameters are expected to both contain knowledge and be able to reason. I consider this a waste of parameters. For example, imagine a 1M parameter Transformer:
###
###
###
This is a static multi-layer stack of parameters that is expected to both reason and store knowledge at the same time.
Now let’s look at how WND separates this idea:
#
#
In other words, one part is mainly responsible for knowledge, while another part is mainly responsible for reasoning.
Input
│
▼
Wide Layer
│
▼
Encoder ───────────────► Feature Bank
▲
│ │
│ │
▼ │
│
Compressor Attention
▲
│ │
▼ │
│
Depth Layer (×N iterations) ─┘
│
▼
Output
To explain it simply:
The goal is for the Depth Layer to handle reasoning step-by-step.
From what I have tested so far, on graph reasoning tasks (long chains, medium difficulty), a ~100K parameter WND model is currently outperforming a ~600K Transformer baseline. It also trains faster, is more sample-efficient, and has a smaller footprint.
That said, I should mention that WND has not yet been tested on other tasks, so at this stage I do not know where it truly excels and where its weaknesses are.
The main issue I am facing right now is that the Wide Layer and the Encoder keep taking over most of the work, leaving the Depth Layer almost idle. I tried adding auxiliary weights, but that did not really solve the issue.
How can I encourage, constrain, or force the
Wide LayerandEncoderto focus only on producing rich and useful information for theDepth Layer, while making sure that the actual reasoning is done by theDepth Layeritself?