Sparse attention

Technique · Architecture & Efficiency · 3 items since Jun 2026

Computing attention over selected tokens or blocks instead of all pairs, cutting long-context cost.

Trends

Aug 15
Tencent's FlashPrefill V2 sparse prefill backend
A production-ready block-sparse prefill-attention backend with FlashAttention-3/4-aligned kernels, FP8, and paged-KV continuous batching reaches up to 47.3x speedup over FlashAttention-2 at 128K context and 30.5x over a dense FP8 baseline on H20 GPUs.
Jun 22
Baidu's Unlimited OCR Works
Reference Sliding Window Attention holds the decoder's KV cache constant, letting an open-weight DeepSeek-OCR-based model transcribe dozens of pages in a single 32K forward pass, a parsing-attention mechanism extensible to speech recognition and translation.
Jun 1
MiniMax M3
An open-weight model combining frontier coding (vendor-reported 59.0% on SWE-Bench Pro, 66.0% on Terminal-Bench 2.1), native multimodality, and a 1M-token context via the new MiniMax Sparse Attention, which reports ~9x prefill and ~15x decode speedups at 1M tokens; weights are pledged within ~10 days.