Gate-Zero Growth: A Geometric Framework for Function-Preserving Continual Learning
Researchers propose gate-zero growth, a function-preserving operator for continual learning that adds new residual blocks to neural networks via zero-initialized gates. In experiments with a 300M-to-857M parameter Transformer transitioning from WikiText-103 to BookCorpus, gate-zero growth achieves near-zero forgetting, while a non-function-preserving control exhibits significantly higher forgetting. The framework also unifies the geometric analysis of related methods such as LoRA, ReZero, and zero-init adapters.
Why it matters: This work offers a principled approach to expanding neural network capacity without catastrophic forgetting, which is important for scalable continual learning in large models.
Full story at: arXiv Machine Learning ↗