SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation
SelKV is a training-free framework for compressing key-value (KV) caches in large language models (LLMs). It uses a soft cosine gate to adaptively merge or drop tokens based on value-vector similarity, and introduces an attention-ratio compensation mechanism to correct softmax imbalances caused by merging. On LongBench, SelKV retains only 25% of the KV cache while achieving strong performance, near-lossless quality on grouped-query attention (GQA) models, and a 3.3x decoding speedup at 100k tokens.
Why it matters: SelKV addresses the memory and speed bottlenecks of KV caches in LLMs, enabling longer context lengths and faster decoding without significant quality loss.
Full story at: arXiv AI/ML ↗