What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning

A study comparing replacement operators for training-free MoE expert pruning, holding the pruning mask fixed, evaluated three operator families on OLMoE, DeepSeek-V2-Lite and Mo (a third model whose name is truncated in the source). The work isolates the second of the two decisions training-free expert pruning makes — what a layer computes when its router still selects a removed expert — since published methods change the mask and the replacement operator at once. The comparison covers OLMoE, DeepSeek-V2-Lite and Mo, but the source excerpt does not report results.

Training-free expert pruning makes two decisions: which experts to remove, and what the layer computes when its router still selects a removed expert. Published methods change both at once. Holding the mask fixed, we compare replacement operators from three families on OLMoE, DeepSeek-V2-Lite and Mo