VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
VarRate is a training-free key-value (KV) cache compression method for long-context large language models (LLMs) that assigns each token a variable low-rank budget based on query salience, ensuring all tokens are retained at nonzero rank. On the LongBench benchmark, VarRate achieves accuracy within 0.8 points of uncompressed models at a 20% memory budget, outperforming token-selection methods that suffer from irreversible token eviction and matching or exceeding other compression baselines without requiring retraining.
Why it matters: VarRate offers a practical, training-free solution to the memory bottleneck in long-context LLM inference, enabling more efficient deployment of large models on limited hardware.
Full story at: arXiv Computation and Language ↗