量子位一水
Hong Kong UST, CMU, and Rockchip Release Open-Source Model with 99% KV Cache Reduction
Original title:KV Cache砍掉99%，单卡吞吐飙6倍，港科CMU联手岩山科技开源新模型
Open Source88
A single card, running 512 concurrent long inference streams
Why it's worth reading
As memory constraints remain the primary bottleneck for long-context inference, this 99% KV Cache reduction and 6x throughput improvement provide essential engineering insights for high-performance model deployment.
Tags
LLMKV CacheInferenceOptimizationOpen Source