Dual Attention Residuals Enable Cross-Stream Depth Selection in Transformers
A new method called Dual Attention Residuals (DAR) introduces reciprocal cross-stream interaction for historical retrieval in Transformer models. DAR computes depth weights from one stream to select information from another stream's history, improving validation loss across dense models (0.1B–1B parameters) and a 7B sparse-MoE model. The approach preserves depth-wise diversity and avoids redundancy seen in other two-stream architectures.
Why it matters: DAR provides a simple architectural change that consistently improves Transformer performance without increasing parameter count, potentially benefiting a broad range of language models.
Full story at: arXiv Computation and Language ↗