Formal Complexity Limits of Structural Generalization in Transformers
A new arXiv preprint formally defines 'structural generalization' and uses computational complexity theory to argue that pure Transformer architectures cannot learn this property under standard assumptions (TC0 ≠ NC1). The authors further contend that neuro-symbolic systems only succeed at structural generalization by hard-coding part of the solution, and that current benchmarks do not distinguish between genuinely learned and pre-specified rules.
Why it matters: This work challenges the prevailing assumption that Transformers can achieve structural generalization, raising fundamental questions about the capabilities and evaluation of modern neural language models.
Full story at: arXiv Computation and Language ↗