A trainable input embedding table assigns each vocabulary item an independently adjustable vector. We investigate whether this token-specific parameterization is required for substantial language-modeling capability, or whether a shared Transformer can learn from fixed token identities. We compare t
Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale
A study at 1.7B-class scale found that a shared Transformer can learn language modeling from fixed token identities rather than a trainable input embedding table, according to the paper's headline claim. The research compares fixed minimal token codes against token-specific adjustable vectors to test whether per-token parameterization is required for substantial capability.
At 1.7B-class scale, replacing the trainable input embedding table with fixed minimal token codes still lets a shared Transformer learn language modeling, suggesting token-specific embedding parameters are not required for substantial capability.
Run your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain β perfect for shipping the agent you just read about.