WeChat AI Team Details WeLM Models Scaling to 617B Parameters

Tencent’s WeChat AI team has detailed a new scaling approach for its WeLM model family. The team trained WeLM-HD4-80B and WeLM-HD4-617B models using a method called Hidden Decoding, which expands each token into multiple internal computation streams without increasing the main Transformer backbone.

The 80B model activates 3 billion parameters, while the 617B model activates 23 billion. Both models outperformed their matched autoregressive baselines across nine shared benchmarks in the team’s tests. The reported training cost was 5.1 times the baseline for the 80B model and 4.4 times for the 617B model. [WeLM official blog]