HiLS Attention: How Tencent Built a Sparse Attention Mechanism That Extrapolates to 4 Million Tokens

HiLS Attention: How Tencent Built a Sparse Attention Mechanism That Extrapolates to 4 Million Tokens Long-context modeling has a persistent tension at its core: full attention is accurate but scales quadratically with sequence length, while sparse attention is fast but tends to degrade badly when you push it beyond its training window. A new paper from Tencent's Hunyuan team, HiLS Attention (arXiv:2607.02980) , proposes a way to resolve that tension — not by approximating attention more…