SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

SeKV introduces a resolution-adaptive semantic KV cache for long-context inference. It partitions context into entropy-guided semantic spans, keeps lightweight summary vectors on the GPU, and stores low-rank SVD bases on the CPU for on-demand token-level reconstruction. During decoding, a trained zoom-in mechanism expands only query-relevant spans, while the base LLM remains frozen. Across four benchmarks, the paper reports a 5.9% average improvement over its strongest semantic compression baseline and a 53.3% reduction in GPU memory versus full KV caching at 128K context, with fewer than 0.05% trainable parameters.