Frontier Language Models Struggle to Copy: 2D Text Representation Enables Dramatic Improvements
A new preprint demonstrates that even state-of-the-art large language models (LLMs) struggle to exactly copy input strings within their context windows, a task much simpler than many they routinely solve. The authors attribute this limitation to the inductive biases of standard positional encodings in Transformer architectures. They introduce 2D-RoPE, a method that organizes text into a 2D grid with separate row and column IDs, enabling shallow Transformers to achieve perfect copying at input lengths hundreds of times longer than those seen during training. Experiments, including pretraining models up to 1.4B parameters, confirm the advantage of this approach over standard methods.
Why it matters: This work reveals a fundamental limitation in current Transformer-based LLMs and proposes a simple architectural change that could improve their reliability on basic tasks like copying.
Full story at: arXiv Computation and Language ↗