xHC: Expanded Hyper-Connections Enable Large-Scale Residual Stream Expansion for LLMs
Researchers introduce xHC (Expanded Hyper-Connections), a method that enables Transformer models to expand their residual streams beyond the previous limit of N=4. By combining temporal feature augmentation and a sparse residual-stream architecture, xHC achieves strong and consistent downstream improvements in 18B and 28B MoE models. The xHC-Flash variant further reduces memory traffic, making large-N residual-stream expansion practical for large language model pre-training.
Why it matters: This work establishes a new, practical scaling axis for large language models, enabling more efficient pre-training and consistent performance gains beyond traditional width and depth scaling.
Full story at: arXiv Machine Learning ↗