EDWARD-NMT: Elastic Dimensions With Adaptive Routing and Dual-consistency for Neural Machine Translation

Researchers posted arXiv paper 2610.32115v1 introducing EDWARD-NMT, a sparse Mixture-of-Experts encoder-decoder architecture for neural machine translation that exposes a family of deployable sub-networks from a single trained checkpoint. EDWARD-NMT combines Matryoshka-style elastic dimensions, an adaptive sigmoid router with a learned task-conditioned threshold that lets each token set its own active expert count, and a dual-consistency objective; experiments on PhoMT (En-Vi), WMT-14 (En-De), multilingual OPUS-16 and a multi-domain De-En benchmark show the full-capacity model matches or exceeds strong MoE baselines with improved parameter efficiency, and extracted sub-networks consistently outperform comparably sized dense Transformers without retraining.

arXiv:2610.32115v1 Announce Type: new Abstract: Sparse Mixture-of-Experts MoE is a powerful paradigm for scaling Neural Machine Translation NMT and managing language heterogeneity. However, practical limitations hinder its deployment on resource-constrained devices: the entire expert pool must remain in memory, and the per-token compute budget is statically fixed during training by rigid top-k routing. We introduce EDWARD-NMT Elastic Dimensions With Adaptive Routing and Dual-consistency for NMT , a novel MoE encoder-decoder architecture designed for a train once, deploy anywhere paradigm that exposes a family of deployable sub-networks at inference time. EDWARD-NMT integrates three key innovations: i Matryoshka-style elastic dimensions to flexibly reduce the memory footprint; ii an adaptive sigmoid router with a learned, task-conditioned threshold, allowing each token to dynamically determine its active expert count; and iii a dual-consistency objective to maximize the performance of the extracted sub-networks. Extensive experiments on PhoMT En-Vi , WMT-14 En-De , multilingual OPUS-16, and a multi-domain De-En benchmark validate our approach. The full-capacity architecture matches or exceeds strong MoE baselines with improved parameter efficiency. Furthermore, the sub-networks seamlessly extracted from a single checkpoint consistently outperform comparably sized dense Transformers without any retraining.