
Read the paper
HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs
on GitHub
Imagine you're grading a stack of 500 essays. For each one, you compare it to every other essay across 20 rubric criteria. Standard attention does the full comparison—all 20 criteria, every pair. HiSA's insight is that most of those 20 criteria are noise for any given pair of essays: their scores cancel out or contribute nothing. So you skip the irrelevant criteria based on which essays you're comparing (their position in the stack), and you get better grades faster because you stopped averaging in noise. The committed claim: you can permanently drop roughly two-thirds of the query-key dot product computation in RoPE-based attention by exploiting the structure of rotary position embeddings, and models trained this way from scratch perform equal to or better than dense-attention baselines. This is not approximate attention or token pruning. It is a static, parameter-free rule that decides which hidden dimensions participate in each attention score based solely on the relative positions of the tokens involved. The mechanism rides on how RoPE works. RoPE encodes position by rotating pairs of hidden dimensions at different frequencies. High-frequency rotation pairs oscillate so fast between nearby tokens that their contribution to the dot product is essentially noise—it averages out. HiSA identifies which frequency bands matter for which relative distances and zeroes out the rest. The decomposition is entirely deterministic: no learned gates, no data-dependent routing. The sparsity pattern is fixed before training begins. On the ladder: the authors train 1.5B-parameter LLMs from scratch on 100B tokens and compare against dense-attention models with identical architecture and data. HiSA matches or slightly beats the dense baseline on standard LM benchmarks while computing only about a third of the QK inner products. They also integrate with FlashAttention-3 kernels, which is where the wall-clock savings materialize—decoding speed improves and memory footprint shrinks. The comparison is apples-to-apples: same model size, same data, same training budget. Integrity is reasonable for an architecture paper. The baselines are the right ones (dense attention, same model), benchmarks are standard community evals, and code is released. The main limitation is scale: 1.5B parameters and 100B tokens is a solid proof point but far from frontier. The authors do not claim SOTA on any downstream task—they claim architectural equivalence at lower compute, which is the right framing. No pre-registration, but that's standard for this type of work. The deeper contribution is empirical confirmation of something suspected but not proven: dense attention learns to suppress exactly the score components that HiSA drops. The authors show this by analyzing trained dense models and finding that the high-frequency RoPE dimensions contribute negligible attention weight. HiSA just hardcodes what the model would learn to ignore anyway, freeing capacity and cutting compute. The open question is scaling. Does the 1/3 compute ratio hold at 7B, 70B, 405B? Does the frequency-band structure of RoPE remain stable as context lengths push past 128K? The authors stop at 1.5B/100B tokens—likely a compute budget constraint rather than a negative result. The next paper almost certainly scales this up, and the real test is whether frontier labs integrate HiSA-style sparsity into production training runs.
The analogy
Imagine grading 500 essays against each other on 20 rubric criteria. For most pairs, only 6-7 criteria actually differentiate them—the rest are noise. HiSA skips the noise criteria based on which pair you're comparing, determined entirely by their position in the stack. You get better grades with a third of the work because you stopped averaging in irrelevant dimensions.
Essay rubric
Skip rubric criteria that produce noise for a given pair of essays based on their position in the grading stack
HiSA attention
Skip RoPE frequency bands whose dot-product contributions cancel out at a given relative token distance
Honest assessment
HiSA is a clean, well-motivated architectural contribution that exploits RoPE structure to cut QK compute by ~2/3 with no quality loss at 1.5B scale. The mechanism is analytically grounded and the code is released. The load-bearing question is whether the sparsity ratio and quality parity hold at frontier scale—until someone trains a 7B+ model with HiSA, this remains a promising proof-of-concept rather than a production method. Engineers working on inference optimization and attention kernels should read it now; everyone else should watch for the scaling follow-up.
What Is CLAIMS?
73
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
4/5
Architecture
4/5
Integrity
3/5
Milestone
4/5
Successor
4/5
What is this paper actually committing to?
Real but incremental. Hidden-dimension sparsity in attention is genuinely underexplored, and the RoPE frequency-band mechanism is clean. But it's an efficiency trick, not a paradigm shift.
The field fight
Should attention efficiency come from sparsifying the token dimension or the hidden dimension?
Side A
Token-level sparsity (top-k, sliding window, sparse patterns) is the natural axis—skip irrelevant tokens
Held by: Longformer, BigBird, StreamingLLM, local attention
Side B
Hidden-dimension sparsity is cheaper and more principled—skip irrelevant feature interactions, not tokens
Held by: This paper (HiSA)
Milestone tracker
Unlocks: Integration into frontier training pipelines; compounding savings with token-level sparsity
Current
1.5B
params
this paper
Next
7B+
params
production-relevant scale
~1-2 years if a compute-rich group picks up the method
Verified Facts (6)
HiSA computes only about a third of the query-key products
Models trained from scratch with HiSA, partial dot product does not hurt but improves performance compared with the complete product
HiSA extends to FlashAttention-3 kernels for training and inference
Dense attention learns to suppress the attention score parts that HiSA drops
Code is available at https://github.com/ambroise-b/hisa
Authors: Ambroise Borbely, Mathieu Salzmann, Olivier Verscheure
Practical Implications (4)
Inference optimization teams can integrate HiSA into FlashAttention-3 kernels for immediate decoding speedups at 1.5B scale
The static sparsity pattern requires zero architectural changes to existing RoPE-based models—it's a drop-in modification to the attention kernel
If the 1/3 compute ratio holds at scale, training cost for attention computation drops by ~67% with no quality tradeoff
HiSA and token-level sparsity are orthogonal—potential for compounding efficiency gains if combined
More latest news
Read all