{"slug": "tphc-improving-hyper-connections-with-tensor-parallel-grouped-updates", "title": "TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates", "summary": "Researchers behind the arXiv paper 2611.23026v1 introduced TPHC, a method that routes a separate mixture of residual streams to each tensor-parallel shard in a Transformer, letting each hyper-connection stream receive a different mixture of shard outputs without adding parameters or FLOPs to the attention or FFN sublayers. On a 1.1B model trained on 100B tokens, TPHC cut validation loss by 0.7-1.0% and raised average zero-shot accuracy across six commonsense reasoning benchmarks by 0.3-0.7 percentage points versus manifold-constrained hyper-connections (mHC) at matched stream counts N in {4,8,16}. TPHC cost only 9% more training time than mHC on an 8B model at four streams, and going from four to sixteen streams raised training time by just 19%; code is available at github.com/wuyou2002/TPHC.", "body_md": "arXiv:2611.23026v1 Announce Type: new \nAbstract: Hyper-Connections (HC) expand the residual stream of a Transformer into $N$ parallel streams, and manifold-constrained HC (mHC) stabilizes them at scale. However, mHC benefits little from more than four streams, partly because each sublayer writes a single output to all streams, limiting how differently they can be updated. In this work, we propose TPHC, which integrates tensor-parallel (TP) shards with HC streams to promote greater independence across streams. Specifically, we route a separate mixture of the residual streams to each TP shard and keep the individual shard outputs separate during write-back, so each stream can receive a different mixture of those outputs. This design makes the residual-stream updates more expressive without adding parameters or FLOPs to the attention or FFN sublayers. With native support for stream sharding under TP, TPHC requires only 9% more training time than mHC on an 8B model when both use four streams. Increasing the number of streams from four to sixteen raises its training time by only 19%. On a 1.1B model trained on 100B tokens, TPHC reduces validation loss by 0.7-1.0% and raises average zero-shot accuracy across six commonsense reasoning benchmarks by 0.3-0.7 percentage points relative to mHC at matched stream counts $N\\in\\{4,8,16\\}$. Our code is available at https://github.com/wuyou2002/TPHC.", "url": "https://wpnews.pro/news/tphc-improving-hyper-connections-with-tensor-parallel-grouped-updates", "canonical_source": "https://www.machinebrief.com/news/tphc-improving-hyper-connections-with-tensor-parallel-groupe-1hlt", "published_at": "2026-11-19 05:00:00+00:00", "updated_at": "2026-11-19 05:43:01.103276+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "large-language-models", "neural-networks"], "entities": ["TPHC", "Hyper-Connections", "manifold-constrained HC", "arXiv", "github.com/wuyou2002/TPHC"], "also_reported_by": [], "takeaway": "Routing a separate residual-stream mixture to each tensor-parallel shard and keeping shard outputs separate on write-back cut validation loss 0.7-1.0% and lifted six-benchmark zero-shot accuracy 0.3-0.7 points over mHC at N in {4,8,16}, for 9% more training time at four streams.", "recommendation": {"advice": "When training with hyper-connections under tensor parallelism, route per-shard residual mixtures and keep shard outputs separate on write-back; the source measured 0.7-1.0% lower validation loss for 9% more training time.", "playbook": "training-data", "playbook_url": "https://wpnews.pro/playbooks/training-data"}, "alternates": {"html": "https://wpnews.pro/news/tphc-improving-hyper-connections-with-tensor-parallel-grouped-updates", "markdown": "https://wpnews.pro/news/tphc-improving-hyper-connections-with-tensor-parallel-grouped-updates.md", "text": "https://wpnews.pro/news/tphc-improving-hyper-connections-with-tensor-parallel-grouped-updates.txt", "jsonld": "https://wpnews.pro/news/tphc-improving-hyper-connections-with-tensor-parallel-grouped-updates.jsonld"}}