C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
C²KV is a unified framework for non-prefix key-value (KV) cache reuse in large language models that jointly optimizes KV extraction and inference-time concatenation. The method learns a composable and compressed KV cache manifold that is position-agnostic, enabling modular KV representations to be reused without modifying the base model. Experiments show that C²KV can achieve up to 17× inference speedup under long-context scenarios while preserving generation quality.
Why it matters: This work addresses the bottleneck of storing and accessing large KV caches in long-context LLM serving, offering significant speedups without loss of accuracy.
Full story at: arXiv Computation and Language ↗