KV-cache compression
Technique · Architecture & Efficiency · 3 items since Apr 2026
Shrinking the attention key-value cache, by evicting or compressing entries, to serve long contexts cheaply.
Related
- Organizations
Trends
Items
- Sep 17
DeepSeek's V4.1-Flash technical report
The report details a Causal Encoder-Decoder that activates 16B parameters at decode but only 8B at prefill, and a CSA2-plus-FP4 KV-cache design that cuts the in-HBM footprint to 890 bytes per token, roughly a quarter of V4-Flash's.
- Sep 4
Salesforce's Random Attention evicts reasoning tokens randomly
Keeping the prompt and evicting reasoning tokens from the KV cache uniformly at random matches the strongest scored evictors while serving 32-43% more throughput in vLLM, because reasoning traces protect themselves through redundancy.
- Apr 6
TriAttention compresses the KV cache 10.7x
A training-free method from MIT, NVIDIA, and ZJU scores KV importance from the stable trigonometric structure of pre-RoPE queries and keys, matching full attention on AIME25 at 2.5x throughput or 10.7x less KV memory and fitting a 32B reasoning model on a 24GB RTX 4090.