Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
Researchers introduce Looped Latent Attention (LLA), a post-training cache codec that compresses the key/value (KV) cache in looped (weight-tied) transformers by leveraging low-rank structure across recurrence steps. LLA achieves up to 21.3x compression on an H200 GPU, increasing batch capacity from 32 to 768 sequences at 4k context length, and improves MATH-500 accuracy from 0.43 to 0.66 at 4x compression. The method outperforms other cache reduction techniques and remains near-lossless up to 32x compression in decoder-independent evaluation.
Why it matters: This work enables much larger batch sizes and longer rollouts for reasoning models by demonstrating that the recurrent KV cache in looped transformers can be aggressively compressed without loss.
Full story at: arXiv Computation and Language ↗