xHC: Expanded Hyper-Connections

arXiv cs.CL · 6 hours ago

Researchers introduced xHC (Expanded Hyper-Connections), a method that extends Transformer models by scaling residual streams beyond the previous limit of N=4 parallel streams to N=16 while maintaining computational efficiency. On 18B and 28B MoE models, xHC achieved a 4.0 point improvement in downstream scores over the prior mHC method while requiring only 1.5x the compute of baseline approaches to reach equivalent loss levels. The approach uses temporal feature augmentation and sparse stream updates to overcome previous bottlenecks, enabling residual-stream expansion as a practical scaling axis for large language model pre-training.