TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates
Researchers behind the arXiv paper 2611.23026v1 introduced TPHC, a method that routes a separate mixture of residual streams to each tensor-parallel shard in a Transformer, letting each hyper-connection stream receive a …