BTHA: Block-Then-Hash Attention for Efficient Long Context

Abstract

Long-context large language models incur substantial computational overhead during autoregressive decoding. Existing sparse attention methods can improve inference efficiency, but they typically rely on fixed sparse patterns, historical attention statistics, or coarse-grained proxy representations to estimate important KV positions, making it difficult to accurately capture query-dependent fine-grained relevance for dynamic KV retrieval. In this paper, we propose Block-then-Hash Attention (BTHA), a two-stage KV retrieval method: it first performs block-level routing with mean key representations to rapidly reduce the candidate search space, and then applies a learnable orthogonal hash network within the routed KV candidates for fine-grained token-level position retrieval. The hash network is trained offline to learn the hash mapping between queries and keys, with a low training cost: on Llama-3.1-8B-Instruct, training can be completed in approximately two hours using a single NVIDIA A100 GPU. During inference, BTHA implements block-level routing, hash-based retrieval, and sparse attention computation with dedicated operators, and further employs CPU–GPU collaborative scheduling to reduce memory access, synchronization, and candidate selection overhead, thereby achieving end-to-end decoding acceleration. Extensive experiments on LongBench-E show that BTHA consistently outperforms state-of-the-art top-K attention methods in both accuracy and efficiency; under a 512-position budget, it achieves the best average accuracy on both Llama-2-7B-32K-Instruct and Llama-3.1-8B-Instruct, while delivering up to 7.0× speedup over vanilla full attention.

1. Introduction

Large language models (LLMs) have shown strong capabilities in long-context understanding, including multi-document question answering, long-document summarization, code understanding, and retrieval-augmented generation [1]. As the context length increases, however, autoregressive decoding becomes increasingly expensive due to the growing key–value (KV) cache. At each decoding step, the current query needs to access historical cached keys and values. For a sequence with length S and head dimension d, the decoding cost of full attention for each query head is $O\left(Sd\right)$. This cost becomes a major bottleneck in long-context inference, especially when the context length reaches tens or hundreds of thousands of tokens. Recent fault-tolerant intelligent control studies have also emphasized the importance of robustness and deployment reliability under actuator faults and performance constraints [2].

A large body of work has attempted to reduce the cost of long-context attention. Sparse attention methods reduce the quadratic complexity of full attention by imposing local windows, global tokens, random connections, low-rank structures, or routing-based sparse patterns [3,4,5,6,7]. These methods have demonstrated that long-context modeling does not always require dense attention over all positions. However, many sparse attention patterns are either predefined or mainly designed for training-time attention computation, while long-context LLM inference requires efficient and query-dependent KV cache access during decoding.

Another line of research focuses on KV cache compression and selection. Methods such as H₂O [8], StreamingLLM [9], SnapKV [10], Quest [11], and related approaches reduce the memory and bandwidth cost by preserving only important cached positions. These methods exploit attention sinks, heavy hitters, observation windows, or query-aware KV selection to reduce the number of cached entries used during inference. Although effective, many of them rely on accumulated attention statistics, heuristic cache policies, or coarse-grained selection rules. Such designs may be insufficient for fine-grained, query-dependent retrieval when the relevant information is sparsely distributed across a long context.

Block-level sparse attention further exploits the structural regularity of long-context attention. By partitioning the sequence into contiguous blocks, block-sparse methods can reduce the retrieval or attention space at a coarse granularity [12,13,14]. Block-level selection is attractive because it is computationally cheaper than token-level selection. Nevertheless, block-level routing alone may retain many irrelevant positions inside selected blocks, since all positions within a selected block are usually treated uniformly. Therefore, a purely block-level method may still introduce redundant KV access, while an overly aggressive block selection may discard important fine-grained positions.

Hash-based attention and retrieval provide another efficient mechanism for approximate similarity search. For example, LSH-based attention methods use randomized projections to assign similar representations into the same buckets [5,15,16]. However, random-projection-based hashing usually requires many hash bits or multiple hash rounds to maintain retrieval accuracy. Moreover, if hash retrieval is applied globally to all cached positions, it still needs to compute Hamming distances over the full KV cache, which limits the practical speedup in long-context decoding. This motivates a coarse-to-fine design: first reduce the candidate search space at the block level, and then apply fine-grained hash retrieval only inside the routed candidates. We propose BTHA, a Block-then-Hash Attention method for efficient long-context decoding. As illustrated in Figure 1, BTHA decomposes KV position selection into two consecutive stages. First, BTHA partitions the KV cache into fixed-size blocks and scores each block according to the similarity between the current query and the block-level mean key representation, thereby performing coarse-grained block routing. This stage rapidly identifies historical regions that are likely to be relevant to the current query and reduces the search space from the entire KV cache to a compact candidate set. Second, BTHA applies a learnable orthogonal hashing network within the selected candidate blocks to conduct fine-grained KV position retrieval. Unlike random LSH, the hashing projection matrices in BTHA are learned from data and regularized with orthogonality constraints, encouraging non-redundant hash dimensions and preserving the geometric structure of query–key representations before binarization. Finally, BTHA computes standard dot-product softmax attention only over the selected key–value entries. Therefore, approximation is introduced only in the KV retrieval stage, while the final attention aggregation remains exact. Unlike recent trainable retrieval or neural routing methods [16,17], BTHA does not learn a new attention mechanism, fine-tune the backbone LLM, or directly learn an end-to-end token/block selection policy. Its learnable component is restricted to an offline-trained orthogonal hash mapping, whose objective is to make query–key pairs with high dense-attention relevance have smaller distances in the hash space. In this way, the hash distance better approximates the dense-attention top-k query–key relevance. Meanwhile, the orthogonality constraint encourages non-redundant hash dimensions and helps preserve the geometric structure of the original query–key representation space before binarization, avoiding excessive distortion of the representation space. Therefore, BTHA learns a similarity-preserving KV retrieval space rather than a new language-modeling representation, an attention-weight predictor, or an end-to-end sparse-attention policy.Experimental results show that BTHA achieves a superior accuracy–speed trade-off and delivers end-to-end decoding acceleration, as shown in Figure 2.

The main contributions of this work are summarized as follows:

• We propose Block-then-Hash Attention, an efficient sparse attention mechanism tailored for long-context decoding. By combining coarse-grained block routing with fine-grained hash-based retrieval, BTHA substantially reduces the KV search space while preserving high attention fidelity.
• We design a learnable orthogonal hashing network for KV cache retrieval. The network is trained offline to learn query–key hashing mappings, improving fine-grained position retrieval within candidate KV blocks compared with random or purely heuristic hashing schemes.
• We implement an end-to-end acceleration framework for long-context decoding. Compared with existing top-K attention methods, BTHA achieves better accuracy and efficiency, and obtains up to 7.0× end-to-end decoding speedup over standard full attention.

2. Related Works

2.1. Long-Context Sparse Attention

The quadratic complexity of full self-attention limits the scalability of Transformer models to long-context scenarios [18]. Early long-context architectures explored recurrence, compression, locality, and sparsity to reduce the attention cost. Transformer-XL introduces segment-level recurrence to reuse hidden states across segments and extend the effective context length [19]. Compressive Transformer further compresses old memories into a smaller representation, enabling longer-range dependency modeling under limited memory [20]. Reformer replaces full attention with locality-sensitive hashing attention and reversible layers, reducing the memory and computation cost for long sequences [5]. Longformer adopts sliding-window attention with task-motivated global attention, making attention cost scale linearly with sequence length for long-document tasks [3]. BigBird combines local, global, and random attention patterns and provides theoretical support for sparse attention while maintaining strong performance on long-sequence tasks [4]. Performer approximates softmax attention with positive random features and offers a linear-attention alternative for long sequences [6]. The Routing Transformer clusters tokens and performs content-based sparse attention to avoid attending to all positions [7]. ETC designs global–local sparse attention for long and structured inputs [21]. Nyströmformer approximates self-attention using the Nyström method to reduce complexity while preserving long-range modeling ability [22]. Long Range Arena provides a unified benchmark for evaluating efficient Transformer variants under long-context tasks [23]. Memorizing Transformers extend context access by retrieving key–value memories from a non-differentiable memory bank [24]. These studies demonstrate that sparse or approximate attention can substantially reduce long-context computation. Different from these methods, our work focuses on decoding-time KV position selection: block-level routing first narrows the historical search space, and a learnable hash module further retrieves fine-grained KV positions under an explicit budget, while the final attention aggregation remains exact.

2.2. KV Cache Compression and Selection

KV cache has become a major memory and bandwidth bottleneck in long-context LLM inference. Recent studies exploit the sparsity and structural regularity of attention to compress, evict, or selectively load cached key–value entries. H₂O observes that a small set of heavy-hitter tokens contributes most of the attention mass and formulates KV eviction as a dynamic submodular optimization problem [8]. Scissorhands exploits the persistence of token importance and maintains a fixed-size KV cache by preferentially preserving pivotal tokens [25]. StreamingLLM identifies attention sinks and shows that preserving initial sink positions together with recent positions enables stable streaming inference over very long sequences [9]. FastGen profiles attention heads and applies adaptive KV cache policies according to their structural patterns [26]. SnapKV uses an observation window to identify head-specific important prompt positions and compresses the KV cache without fine-tuning [10]. Quest introduces query-aware sparsity and selects critical KV pages according to the current query, reducing memory movement during long-context inference [11]. SparQ Attention selectively fetches cached history to improve bandwidth efficiency during LLM inference [27]. CacheGen compresses and streams KV caches using quantization and entropy coding to reduce serving bandwidth [28]. InfiniGen dynamically manages KV cache offloading and prefetching, fetching only essential KV entries from CPU memory [29]. LazyLLM dynamically prunes prompt tokens and selectively computes KV states for positions relevant to the next-token prediction [30]. DuoAttention separates retrieval heads from streaming heads and applies different KV cache strategies to reduce memory and latency [31]. Ada-KV further studies adaptive head-wise budget allocation for KV cache eviction [32]. Compared with these KV cache compression and selection methods, our method does not rely solely on accumulated attention scores, fixed windows, or head-type heuristics. Instead, it combines block-level candidate routing with learnable hash-based fine retrieval, where the block stage is controlled by a block selection ratio and the hash stage is controlled by an absolute KV position budget.

2.3. Block-Sparse Attention

Block-sparse attention exploits the observation that long-context attention often exhibits structured sparsity at the level of contiguous token blocks. Sparse Transformer is an early attempt to reduce attention complexity using structured sparse attention patterns [33]. Longformer and BigBird use local windows, global tokens, and random connections, which can be interpreted as structured sparse attention patterns over long sequences [3,4]. ETC also relies on global–local sparse attention to encode long and structured inputs [21]. The Routing Transformer performs content-based sparse routing, grouping similar tokens so that attention is computed within routed subsets [7]. The Sinkhorn Transformer learns differentiable sorting-based sparse attention patterns for efficient sequence modeling [34]. Nyströmformer reduces attention cost by approximating the global attention matrix with landmark points [22]. The Blockwise Parallel Transformer performs blockwise computation to support large-context training with reduced memory usage [35]. Landmark Attention represents each block with a landmark token and retrieves relevant blocks through the attention mechanism itself [36]. XAttention introduces antidiagonal scoring as an efficient proxy for block importance and prunes unimportant attention blocks during long-context inference [12]. Native Sparse Attention combines coarse-grained token compression, selective token attention, and local attention in a hardware-aligned sparse architecture [13]. MoBA applies mixture-of-experts-style routing to block attention, allowing the model to dynamically select relevant historical blocks [14]. These block-sparse methods mainly reduce attention computation by selecting or approximating attention blocks. In contrast, our method uses block selection only as a coarse routing stage. The selected blocks are expanded into candidate KV positions, and a learnable orthogonal hash network further selects fine-grained positions under an absolute budget before exact softmax attention is computed.

3. Method

3.1. Problem Formulation

Consider a decoder-only Transformer at layer l. Let the current query of head h be denoted as

$${\mathbf{q}}_{l,h}\in {\mathbb{R}}^d,$$

(1)

where d is the head dimension. The cached keys and values are denoted as

$${\mathbf{K}}_l={\left\{{\mathbf{k}}_{l,t,g}\right\}}_{t=1}^S,{\mathbf{V}}_l={\left\{{\mathbf{v}}_{l,t,g}\right\}}_{t=1}^S,$$

(2)

where S is the current KV cache length, t denotes the cached position index, and g is the key–value head index. For models using grouped-query attention (GQA), multiple query heads share one key–value head. We denote the key–value head corresponding to query head h as $g\left(h\right)$.

In standard full attention, the output of head h is computed over all cached positions:

$${\mathbf{o}}_{l,h}={\displaystyle \sum _{t=1}^S}{\alpha }_{l,h,t}{\mathbf{v}}_{l,t,g\left(h\right)},$$

(3)

where

$${\alpha }_{l,h,t}={\displaystyle \frac{exp\left({\mathbf{q}}_{l,h}^{\top }{\mathbf{k}}_{l,t,g\left(h\right)}/\sqrt{d}\right)}{{\sum }_{j=1}^{S}exp\left({\mathbf{q}}_{l,h}^{\top }{\mathbf{k}}_{l,j,g\left(h\right)}/\sqrt{d}\right)}}.$$

(4)

When S is large, full attention introduces substantial memory access and computational overhead. Therefore, our goal is to construct a compact subset of cached KV positions

$${\mathcal{T}}_{l,h}\subseteq \{1,\dots ,S\},$$

(5)

and compute exact attention only over the selected positions. Each selected position corresponds to one cached key–value entry. The keys are used for routing and retrieval, while the values are used only for final attention aggregation.

3.2. Block-Level Coarse Routing

Under the squared reconstruction error, the mean vector is the optimal single-vector representative for a given block. Specifically, for a block containing key vectors ${\left\{k_i\right\}}_{i=1}^B$, the vector that minimizes ${\sum }_{i=1}^B{\parallel k_i-c\parallel }_2^2$ is the block mean $c^{\ast }={\displaystyle \frac{1}{B}}{\sum }_{i=1}^B{k}_i$. Motivated by this property, we first partition the KV cache along the position dimension into fixed-size blocks and use the mean key of each block as its coarse-grained representation.

We first partition the KV cache along the position dimension into fixed-size blocks. Let the block size be B, and the number of blocks be

$$N=⌈{\displaystyle \frac{S}{B}}⌉.$$

(6)

The position index set of the b-th block is denoted as ${\mathcal{I}}_b$. For each block and key–value head, we compute a mean key representation:

$${\overline{\mathbf{k}}}_{l,b,g}={\displaystyle \frac{1}{|{\mathcal{I}}_b|}}\sum _{t\in {\mathcal{I}}_b}{\mathbf{k}}_{l,t,g}.$$

(7)

For the last incomplete block, the mean is computed using the actual number of valid positions, avoiding the dilution caused by padding.

The relevance score between query head h and block b is computed by the scaled dot product between the query and the block-level mean key:

$$s_{l,h,b}={\displaystyle \frac{{\mathbf{q}}_{l,h}^{\top }{\overline{\mathbf{k}}}_{l,b,g\left(h\right)}}{\sqrt{d}}}.$$

(8)

For each query head, we select the top-ranked blocks according to a block selection ratio ${\rho }_b$:

$${\mathcal{B}}_{l,h}={TopK}_b\left(s_{l,h,b},⌈{\rho }_{b}N⌉\right),$$

(9)

where ${\rho }_b\in (0,1]$ controls the fraction of selected blocks. The selected blocks are expanded into a candidate KV position set:

$${\mathcal{C}}_{l,h}=\left(\bigcup _{b\in {\mathcal{B}}_{l,h}}{\mathcal{I}}_b\right)\cup {\mathcal{S}}_{\mathrm{sink}},$$

(10)

where ${\mathcal{S}}_{\mathrm{sink}}$ denotes the sink positions that are always retained for stable access to global context.

The block-level routing stage serves as a coarse retrieval module. It reduces the search space before fine-grained retrieval while keeping the selection process aligned with the key vectors in the KV cache.

3.3. Shared Learnable Orthogonal Hash Network

After block-level routing, BTHA applies a shared learnable orthogonal hash network to perform fine-grained position-level retrieval within the candidate KV position set. Unlike asymmetric query–key hashing, the query and key are projected by the same hash matrix. For layer l and query head h, the continuous hash projections are defined as

$${\mathbf{u}}_{l,h}^q={\mathbf{W}}_{l,h}{\mathbf{q}}_{l,h},{\mathbf{u}}_{l,t,h}^k={\mathbf{W}}_{l,h}{\mathbf{k}}_{l,t,g\left(h\right)},$$

(11)

where ${\mathbf{W}}_{l,h}\in {\mathbb{R}}^{r\times d}$ is a learnable hash projection matrix, r is the number of hash bits, and d is the head dimension.

When $r=d$, the hash matrix is constrained as a square orthogonal matrix:

$${\mathbf{W}}_{l,h}^{\top }{\mathbf{W}}_{l,h}=\mathbf{I}.$$

(12)

When $r<d$, the hash matrix is constrained to have orthonormal rows:

$${\mathbf{W}}_{l,h}{\mathbf{W}}_{l,h}^{\top }={\mathbf{I}}_r.$$

(13)

This constraint encourages different hash dimensions to capture non-redundant directions in the representation space.

During inference, binary hash codes are obtained by the sign function:

$${\mathbf{b}}_{l,h}^q=sign\left({\mathbf{W}}_{l,h}{\mathbf{q}}_{l,h}\right),{\mathbf{b}}_{l,t,h}^k=sign\left({\mathbf{W}}_{l,h}{\mathbf{k}}_{l,t,g\left(h\right)}\right),$$

(14)

where each element of the binary code belongs to $\{-1,+1\}$. The Hamming distance between the query code and the cached key code is defined as

$$d_H\left({\mathbf{q}}_{l,h},{\mathbf{k}}_{l,t,g\left(h\right)}\right)={\displaystyle \sum _{i=1}^r}\mathbb{I}\left[b_{l,h,i}^q\ne b_{l,t,h,i}^k\right].$$

(15)

A smaller Hamming distance indicates that the cached key at position t is more similar to the current query in the learned hash space.

During decoding, the key hash codes are maintained in a key hash cache. When a new key is appended to the KV cache, only the hash code of the newly appended key needs to be computed and cached. Similarly, the block mean cache is updated incrementally: completed blocks are reused, while only the current tail block is updated.

3.4. Orthogonality Analysis

The orthogonal constraint is introduced to preserve the geometry of hidden representations before binarization. We first consider the square case where $\mathbf{W}\in {\mathbb{R}}^{d\times d}$ and

$${\mathbf{W}}^{\top }\mathbf{W}=\mathbf{I}.$$

(16)

For any vector $\mathbf{x}\in {\mathbb{R}}^d$, the projected representation is

$$\mathbf{u}=\mathbf{W}\mathbf{x}.$$

(17)

Then its squared norm is

$${\parallel \mathbf{u}\parallel }_2^2={\left(\mathbf{W}\mathbf{x}\right)}^{\top }\left(\mathbf{W}\mathbf{x}\right)={\mathbf{x}}^{\top }{\mathbf{W}}^{\top }\mathbf{W}\mathbf{x}={\mathbf{x}}^{\top }\mathbf{x}={\parallel \mathbf{x}\parallel }_2^2.$$

(18)

Therefore, an orthogonal projection preserves the vector norm.

Similarly, for any two vectors $\mathbf{x},\mathbf{y}\in {\mathbb{R}}^d$, we have

$${\left(\mathbf{W}\mathbf{x}\right)}^{\top }\left(\mathbf{W}\mathbf{y}\right)={\mathbf{x}}^{\top }{\mathbf{W}}^{\top }\mathbf{W}\mathbf{y}={\mathbf{x}}^{\top }\mathbf{y}.$$

(19)

Thus, the dot product between two vectors is also preserved. Since cosine similarity is defined by the normalized dot product, we further have

$$cos\left(\mathbf{W}\mathbf{x},\mathbf{W}\mathbf{y}\right)={\displaystyle \frac{{\left(\mathbf{W}\mathbf{x}\right)}^{\top }\left(\mathbf{W}\mathbf{y}\right)}{{\parallel \mathbf{W}\mathbf{x}\parallel }_2{\parallel \mathbf{W}\mathbf{y}\parallel }_2}}={\displaystyle \frac{{\mathbf{x}}^{\top }\mathbf{y}}{{\parallel \mathbf{x}\parallel }_2{\parallel \mathbf{y}\parallel }_2}}=cos\left(\mathbf{x},\mathbf{y}\right).$$

(20)

Therefore, before binarization, the shared orthogonal hash projection preserves the norm, dot product, and angular relation of the original query–key representations. This property is important because attention relevance is originally measured by the scaled dot product between the query and cached keys.

For the rectangular case $\mathbf{W}\in {\mathbb{R}}^{r\times d}$ with $r<d$, we constrain the rows of $\mathbf{W}$ to be orthonormal:

$$\mathbf{W}{\mathbf{W}}^{\top }={\mathbf{I}}_r.$$

(21)

Although the full norm and dot product in ${\mathbb{R}}^d$ are not exactly preserved after dimensionality reduction, the orthogonality constraint still ensures that the projected hash dimensions are non-redundant and have equal scale. This prevents the learned hash matrix from collapsing into correlated or degenerate projection directions.

3.5. Training Objective

The hash network is trained to approximate the KV position ranking induced by full attention. For each query, we first compute the full attention logits over all cached positions:

$$a_{l,h,t}={\displaystyle \frac{{\mathbf{q}}_{l,h}^{\top }{\mathbf{k}}_{l,t,g\left(h\right)}}{\sqrt{d}}}.$$

(22)

The positions with the largest attention logits are treated as positive cached positions:

$${\mathcal{P}}_{l,h}={TopK}_t\left(a_{l,h,t},K_{\mathrm{oracle}}\right),$$

(23)

where $K_{\mathrm{oracle}}$ is the oracle positive-position budget. Negative positions are sampled from the remaining cached positions:

$${\mathcal{N}}_{l,h}\subseteq \{1,\dots ,S\}\setminus {\mathcal{P}}_{l,h}.$$

(24)

Since the sign function is non-differentiable, directly optimizing the binary hash codes is difficult. Therefore, during training, we replace the hard sign function with a smooth continuous relaxation:

$$\tilde{\mathbf{b}}=2·Sigmoid\left(\gamma \mathbf{u}\right)-1,$$

(25)

where $\gamma$ is a temperature coefficient. When $\gamma$ becomes larger, the relaxed output becomes closer to the hard binary code produced by the sign function:

$$\underset{\gamma \to \infty }{lim}\left(2·Sigmoid\left(\gamma u\right)-1\right)=sign\left(u\right).$$

(26)

We define the relaxed hash-space similarity between a query and a cached key as

$${\tilde{s}}_H\left(\mathbf{q},\mathbf{k}\right)={\displaystyle \frac{1}{r}}{\tilde{\mathbf{b}}}^q{\left({\tilde{\mathbf{b}}}^k\right)}^{\top }.$$

(27)

This relaxed similarity is differentiable and is consistent with Hamming distance in the binary case. Specifically, when ${\mathbf{b}}^q,{\mathbf{b}}^k\in {\{-1,+1\}}^r$, the Hamming distance satisfies

$$d_H\left({\mathbf{b}}^q,{\mathbf{b}}^k\right)={\displaystyle \frac{1}{2}}\left(r-{\left({\mathbf{b}}^q\right)}^{\top }{\mathbf{b}}^k\right).$$

(28)

Therefore, maximizing the binary inner product is equivalent to minimizing the Hamming distance.

The pairwise ranking objective encourages positive cached positions to have larger relaxed hash similarity than negative cached positions:

$${\mathcal{L}}_{\mathrm{rank}}={\displaystyle \frac{1}{\left|\mathcal{P}\right|\left|\mathcal{N}\right|}}\sum _{p\in \mathcal{P}}\sum _{n\in \mathcal{N}}max\left(0,m-{\tilde{s}}_H\left(\mathbf{q},{\mathbf{k}}_p\right)+{\tilde{s}}_H\left(\mathbf{q},{\mathbf{k}}_n\right)\right),$$

(29)

where m is the ranking margin.

To prevent hash bit collapse, we introduce a bit balance loss:

$${\mathcal{L}}_{\mathrm{bal}}={∥{\displaystyle \frac{1}{M}}{\displaystyle \sum _{i=1}^M}{\tilde{\mathbf{b}}}_i∥}_2^2,$$

(30)

where M is the number of relaxed hash representations in a mini-batch.

Furthermore, to explicitly encourage the learned hash matrix to be orthogonal, we introduce an orthogonality regularization term:

$${\mathcal{L}}_{\mathrm{orth}}={∥{\mathbf{W}}_{l,h}{\mathbf{W}}_{l,h}^{\top }-{\mathbf{I}}_r∥}_F^2.$$

(31)

The final training objective is

$$\mathcal{L}={\mathcal{L}}_{\mathrm{rank}}+{\lambda }_{\mathrm{bal}}{\mathcal{L}}_{\mathrm{bal}}+{\lambda }_{\mathrm{orth}}{\mathcal{L}}_{\mathrm{orth}},$$

(32)

where ${\lambda }_{\mathrm{bal}}$ and ${\lambda }_{\mathrm{orth}}$ control the strengths of the balance and orthogonality regularization terms, respectively.

3.6. Block-Then-Hash Inference

During inference, BTHA first performs block-level routing and obtains the candidate KV position set ${\mathcal{C}}_{l,h}$. The block stage is controlled by the block selection ratio ${\rho }_b$, while the hash stage is controlled by an absolute KV position budget $K_{\mathrm{hash}}$. The final selected position set is defined as

$${\mathcal{T}}_{l,h}=\left\{\begin{array}{cc}{\mathcal{C}}_{l,h},\hfill & |{\mathcal{C}}_{l,h}|\le K_{\mathrm{hash}},\hfill \\ {TopK}_{t\in {\mathcal{C}}_{l,h}}^{\mathrm{smallest}}\left(d_H\left({\mathbf{q}}_{l,h},{\mathbf{k}}_{l,t,g\left(h\right)}\right),K_{\mathrm{hash}}\right),\hfill & |{\mathcal{C}}_{l,h}|>K_{\mathrm{hash}}.\hfill \end{array}\right.$$

(33)

After the final KV position subset ${\mathcal{T}}_{l,h}$ is obtained, the output is computed using exact dot-product attention. Specifically, for $t\in {\mathcal{T}}_{l,h}$,

$${\alpha }_{l,h,t}={\displaystyle \frac{exp\left({\mathbf{q}}_{l,h}^{\top }{\mathbf{k}}_{l,t,g\left(h\right)}/\sqrt{d}\right)}{{\sum }_{j\in {\mathcal{T}}_{l,h}}exp\left({\mathbf{q}}_{l,h}^{\top }{\mathbf{k}}_{l,j,g\left(h\right)}/\sqrt{d}\right)}}.$$

(34)

The attention output is then

$${\mathbf{o}}_{l,h}=\sum _{t\in {\mathcal{T}}_{l,h}}{\alpha }_{l,h,t}{\mathbf{v}}_{l,t,g\left(h\right)}.$$

(35)

Therefore, BTHA approximates only the KV position retrieval stage, while preserving exact softmax attention over the selected key–value entries.

The overall Block-then-Hash inference procedure is summarized in Algorithm 1.

Algorithm 1 Block-then-Hash Attention (BTHA)

Require: Query ${\mathbf{q}}_{l,h}$, key cache ${\mathbf{K}}_l$, value cache ${\mathbf{V}}_l$, block mean cache ${\overline{\mathbf{K}}}_l$, key hash cache ${\mathbf{B}}_l^k$, shared hash projection ${\mathbf{W}}_{l,h}$, block size B, block selection ratio ${\rho }_b$, hash budget $K_{\mathrm{hash}}$, sink positions ${\mathcal{S}}_{\mathrm{sink}}$ Ensure: Attention output ${\mathbf{o}}_{l,h}$ 1: $g\leftarrow g\left(h\right)$                                       ▹ Map query head h to its KV head under GQA 2: $S\leftarrow$ current KV cache length 3: $N\leftarrow \lceil S/B\rceil$    Stage 1: Block-level coarse routing 4: for $b=1$ to N do 5:      ${\mathcal{I}}_b\leftarrow \{(b-1)B+1,\dots ,min(bB,S)\}$ 6:      $s_b\leftarrow {\mathbf{q}}_{l,h}^{\top }{\overline{\mathbf{k}}}_{l,b,g}/\sqrt{d}$ 7: end for 8: $M_b\leftarrow \lceil {\rho }_{b}N\rceil$ 9: ${\mathcal{B}}_{l,h}\leftarrow {TopK}_b(s_b,M_b)$ 10: ${\mathcal{C}}_{l,h}\leftarrow \left({\bigcup }_{b\in {\mathcal{B}}_{l,h}}{\mathcal{I}}_b\right)\cup {\mathcal{S}}_{\mathrm{sink}}$      Stage 2: hash retrieval 11: if $|{\mathcal{C}}_{l,h}|\le K_{\mathrm{hash}}$ then 12:      ${\mathcal{T}}_{l,h}\leftarrow {\mathcal{C}}_{l,h}$ 13: else 14:       ${\mathbf{b}}_{l,h}^q\leftarrow sign\left({\mathbf{W}}_{l,h}{\mathbf{q}}_{l,h}\right)$ 15:       for all $t\in {\mathcal{C}}_{l,h}$ do 16:             $d_t\leftarrow d_H\left({\mathbf{b}}_{l,h}^q,{\mathbf{b}}_{l,t,h}^k\right)$ 17:       end for 18:       ${\mathcal{T}}_{l,h}\leftarrow {TopK}_{t\in {\mathcal{C}}_{l,h}}^{\mathrm{smallest}}(d_t,K_{\mathrm{hash}})$ 19: end if       Stage 3: Exact attention over selected positions 20: for all $t\in {\mathcal{T}}_{l,h}$ do 21:       $a_t\leftarrow {\mathbf{q}}_{l,h}^{\top }{\mathbf{k}}_{l,t,g}/\sqrt{d}$ 22: end for 23: ${\alpha }_t\leftarrow {Softmax}_{t\in {\mathcal{T}}_{l,h}}\left(a_t\right)$ 24: ${\mathbf{o}}_{l,h}\leftarrow {\sum }_{t\in {\mathcal{T}}_{l,h}}{\alpha }_t{\mathbf{v}}_{l,t,g}$ 25: return ${\mathbf{o}}_{l,h}$

3.7. Complexity Analysis

We analyze the per-step decoding complexity of BTHA for one layer and one query head. Let S be the current KV cache length, d be the head dimension, B be the block size, and

$$N=⌈{\displaystyle \frac{S}{B}}⌉$$

(36)

be the number of KV blocks. Let ${\rho }_b$ denote the block selection ratio, r denote the number of hash bits, and $K_{\mathrm{hash}}$ denote the hash retrieval budget. The number of routed candidate positions is approximately

$$|{\mathcal{C}}_{l,h}|\approx {\rho }_{b}S+\left|{\mathcal{S}}_{\mathrm{sink}}\right|.$$

(37)

For standard full attention, the dominant decoding cost per head is

$$\mathcal{O}\left(Sd\right),$$

(38)

which comes from computing query–key scores over all cached positions and aggregating the corresponding values.

In BTHA, the block-level routing stage uses cached block mean keys. Since each block is represented by one mean key vector, the block scoring cost is

$$\mathcal{O}\left(Nd\right)=\mathcal{O}\left({\displaystyle \frac{S}{B}}d\right).$$

(39)

After the top-ranked blocks are selected, hash-based retrieval is performed only inside the routed candidate set. The query hash computation costs

$$\mathcal{O}\left(rd\right),$$

(40)

and the dense Hamming comparison cost is

$$\mathcal{O}\left(\right|{\mathcal{C}}_{l,h}\left|r\right).$$

(41)

When binary hash codes are packed into machine words of width w, the Hamming distance computation is reduced to

$$\mathcal{O}\left(|{\mathcal{C}}_{l,h}|⌈{\displaystyle \frac{r}{w}}⌉\right),$$

(42)

where bitwise XOR produces the mismatch mask and population-count operations compute the Hamming distance.

Finally, BTHA computes exact dot-product attention only over the selected KV positions. Let

$$K_{\mathrm{eff}}=min\left(|{\mathcal{C}}_{l,h}|,K_{\mathrm{hash}}\right)$$

(43)

be the effective number of selected positions. The final exact attention cost is

$$\mathcal{O}\left(K_{\mathrm{eff}}d\right).$$

(44)

Therefore, the overall per-step decoding complexity is

$$\mathcal{O}\left({\displaystyle \frac{S}{B}}d+rd+\left|{\mathcal{C}}_{l,h}\right|⌈{\displaystyle \frac{r}{w}}⌉+K_{\mathrm{eff}}d\right).$$

(45)

Since $B>1$, ${\rho }_b<1$, and $K_{\mathrm{eff}}\ll S$ in long-context decoding, BTHA substantially reduces both KV memory access and exact attention computation compared with full attention.

BTHA is further optimized at the system level. We implement the hash encoding, candidate filtering, XOR-based Hamming computation, and sparse KV gathering with low-level C/CUDA kernels [37,38]. These kernels operate directly on cached block summaries and packed hash codes, enabling BTHA to be integrated into existing decoding pipelines without changing model weights or the high-level Transformer architecture [39]. The block mean cache and key hash cache are maintained incrementally: when a new key is appended, only the current tail block mean and the hash code of the new key are updated.

The additional storage overhead is small. The block mean cache requires

$$\mathcal{O}\left({\displaystyle \frac{S}{B}}d\right)$$

(46)

storage per key–value head, while the bit-packed key hash cache requires storage.

$$\mathcal{O}\left(S⌈{\displaystyle \frac{r}{w}}⌉\right)$$

(47)

In our implementation, the packed hash cache accounts for only about $5\%$ of the original KV cache size. Thus, BTHA avoids full-cache recomputation while introducing limited auxiliary memory overhead.

3.8. Hash Training Data and Configuration

We train the orthogonal hash projection using query and key representations sampled from real-world long-context data. The training data contains 100 sampled sequences, evenly drawn from the Book and ArXiv subsets of RedPajama [40]. For each sampled sequence, we first perform a dense prefill pass with the backbone LLM and collect the corresponding query and key representations from each layer and attention head. To construct a training instance, we randomly sample a query position from the second half of the sequence so that the selected query has access to a sufficiently long historical context. Following the causal constraint of decoder-only attention, only keys whose positions are not later than the sampled query position are used as candidate keys.

For a sampled query at position i, we compute its dense query–key relevance scores against all causally visible keys using the standard scaled dot product. The visible keys are then ranked according to these dense attention scores. In our implementation, the oracle positive budget is set to $K_{\mathrm{oracle}}=\lceil 0.1i\rceil$, corresponding to the top 10% highest-scoring visible keys. These keys are treated as positive samples, while negative samples are drawn from the remaining visible keys. Each training unit is represented as a labeled query–key triplet $(\mathbf{q},\mathbf{k},y)$, where $y=1$ denotes a high-relevance key and $y=0$ denotes a low-relevance key. The triplets are randomly shuffled across mini-batches during optimization. This construction directly aligns the hash learning objective with dense-attention-induced token relevance, allowing the learned Hamming distance to better approximate the dense top-k retrieval order.

The hash projection is optimized with stochastic gradient descent (SGD). We use momentum and weight decay for stable optimization and train the hash network with repeated representation sampling to improve robustness across different query–key distributions. The main training configuration is summarized in

Table 1. Training configuration of the BTHA orthogonal hash projection.

Hyperparameter	Value
Sampled sequences	100
Data source	Book and ArXiv [40]
Positive-key ratio	Top 10%
Optimizer	SGD
Training epochs	20
Training iterations per epoch	20
Repeated representation-sampling iterations	10
Learning rate	0.08
Momentum	0.9
Weight decay	${10}^{-6}$
Balance loss weight ${\lambda }_{\mathrm{bal}}$	0.5
Orthogonality loss weight ${\lambda }_{\mathrm{orth}}$	1

.

4. Experimental

4.1. Experimental Setup

(1) 

Experiment platform.

We conduct all experiments on a server equipped with an NVIDIA A100 80 GB GPU and a 96-core CPU. The system runs Ubuntu 24.04 with CUDA 12.1. Our implementation is based on PyTorch 2.4 [41] and FlashInfer [42].

(2) 

Baselines and configurations.

We compare BTHA with representative top-k attention baselines, including Loki [43] and Quest [11]. Loki accelerates top-k attention through low-rank approximation, while Quest performs query-aware block-level KV selection. We further compare BTHA with MagicPIG [44], which accelerates top-k attention using LSH [15]. LSH is a randomized hashing method that typically relies on random projections to generate hash codes. Unlike learning-to-hash methods, LSH usually requires a large number of hash bits to maintain retrieval accuracy.

In addition to top-k attention baselines, we compare BTHA with several KV cache compression methods, including StreamingLLM [9], H₂O [8], and SnapKV [10]. For all baselines, we adopt the recommended configurations from the original papers, including hyperparameters such as channel numbers, block sizes, and KV cache budgets when applicable. For BTHA, we set the number of hash bits to $\mathtt{rbit}=128$ and the block selection ratio to ${\rho }_b=0.5$, which provide a robust configuration across most tasks. Following Quest [11], we use vanilla full attention in the first two layers, since these layers are commonly observed to be outlier layers in top-k attention methods. We also include the vanilla Transformer with full attention, denoted as dense, as a reference baseline to evaluate both the effectiveness and efficiency of BTHA.

(3) 

Models and datasets.

We evaluate BTHA on two mainstream large language models, Llama-2 [45] and Llama-3.1 [46]. The evaluation is conducted on two widely used long-context benchmarks, LongBench-E [47] and RULER [48]. LongBench-E is a multi-task benchmark covering question answering, document summarization, few-shot learning, synthetic tasks, and code understanding, which provides a comprehensive evaluation of long-context language modeling ability. RULER focuses on controlled long-context evaluation, especially retrieval-oriented tasks over extremely long contexts. These benchmarks allow us to evaluate both the task-level accuracy and long-range retrieval capability of BTHA under different context lengths.

4.2. Accuracy Evaluation

(1) 

Evaluation on LongBench-E.

We first evaluate all methods on LongBench-E, which covers question answering, document summarization, synthetic tasks, few-shot learning, and code understanding. As shown in

Table 2. Transposed accuracy results on LongBench-E with 512 KV position budget. MP denotes MagicPIG, SL denotes StreamingLLM, and S-KV denotes SnapKV. The best result in each column is shown in bold.

Model	Method	LCC	PRetr	HQA	TQA	Repo	Sam	Trec	MQA	2Wiki	Gov	PCnt	MltN	Qaspr	AVG.
Llama-2-7B-32K-Instruct	Dense	67.53	11.89	15.30	85.03	55.03	39.32	69.00	22.44	13.13	32.01	1.17	24.51	11.76	34.47
	Loki	58.68	11.97	14.91	85.30	44.41	38.95	69.00	22.11	13.09	30.51	0.52	23.82	12.82	32.78
	Quest	65.14	15.53	13.64	85.18	52.57	39.24	67.57	19.33	12.51	24.83	1.20	16.61	10.93	32.64
	MP	66.43	10.01	14.69	86.17	55.81	38.94	69.00	21.70	13.29	31.29	1.08	23.74	11.06	34.09
	SL	46.91	4.84	8.22	60.67	35.43	19.50	28.00	15.51	7.54	19.39	0.00	18.09	5.43	20.73
	H2O	27.42	2.41	4.19	20.07	16.33	6.74	24.33	3.06	2.19	9.61	0.00	7.83	0.23	9.57
	S-KV	52.74	9.44	11.78	66.90	45.13	37.56	37.00	17.44	12.65	13.54	0.08	12.98	7.24	24.96
	BTHA	68.50	9.73	14.96	86.30	56.34	39.26	69.14	23.10	13.34	31.49	0.44	24.91	11.85	34.57
Llama-3.1-8B-Instruct	Dense	67.24	99.67	60.21	91.64	52.36	42.55	71.66	54.82	44.08	35.03	13.19	26.19	44.68	54.10
	Loki	61.29	99.67	59.48	91.45	48.47	41.99	72.33	54.47	44.33	34.74	12.74	25.85	45.15	53.23
	Quest	58.81	99.67	60.03	90.79	46.72	39.75	71.33	51.50	43.90	33.64	13.16	25.69	43.52	52.19
	MP	53.39	98.83	56.28	77.90	42.35	34.28	63.67	49.10	37.84	32.58	9.96	24.57	38.20	47.61
	SL	64.90	94.33	48.52	79.24	45.58	40.10	51.67	34.13	37.81	22.73	13.15	21.51	24.91	44.51
	H2O	64.99	94.00	57.89	92.06	46.70	41.23	65.00	45.12	40.91	29.40	12.82	24.64	33.75	49.89
	S-KV	66.49	99.67	59.93	91.98	48.60	40.58	60.33	52.82	43.63	26.29	13.04	23.20	36.42	51.00
	BTHA	68.52	99.67	60.12	91.96	52.51	42.23	71.67	54.63	43.26	34.36	12.48	25.78	45.03	54.02

, BTHA achieves strong accuracy under the same 512 KV position budget. On Llama-2-7B-32K-Instruct, BTHA obtains an average score of $34.57$, slightly surpassing the dense full-attention baseline with $34.47$ and outperforming all compared sparse attention and KV cache compression baselines. On Llama-3.1-8B-Instruct, BTHA achieves an average score of $54.02$, which is very close to the dense full-attention baseline of $54.10$, with only a $0.08$ point gap. Compared with non-dense baselines, BTHA also achieves the best average score, outperforming Loki, Quest, MagicPIG, StreamingLLM, H₂O, and SnapKV. These results indicate that BTHA can effectively preserve task-level accuracy while attending to only a limited number of KV positions.

(2) 

Evaluation on RULER.

We further evaluate all methods on RULER, a controlled long-context benchmark designed to test retrieval, tracing, aggregation, and question-answering capabilities under extended context lengths. As shown in

Table 3. Transposed accuracy results on RULER. MP denotes MagicPIG, SL denotes StreamingLLM, and S-KV denotes SnapKV. The best result in each column is shown in bold.

Model	Method	NS1	NS2	NS3	NMK1	NMK2	NMV	NMQ	VT	FWE	QA1	QA2	AVG.
Llama-2-7B-32K-Instruct	Dense	93.75	100.00	91.67	93.75	81.25	66.67	52.08	21.04	48.61	30.21	36.46	65.04
	Loki	25.00	2.08	0.00	0.00	0.00	0.00	0.00	1.04	19.44	14.58	16.67	7.16
	Quest	100.00	95.83	52.08	87.50	54.17	52.34	56.51	26.87	36.36	23.96	34.38	56.37
	MP	97.92	93.75	54.17	83.33	71.88	59.38	43.49	16.04	51.39	30.21	35.42	57.91
	SL	1.04	5.21	1.04	6.25	1.04	6.25	4.17	0.21	54.86	22.92	27.08	11.82
	H2O	0.00	0.00	0.00	3.12	0.00	0.78	0.00	0.00	20.49	15.62	16.67	5.15
	S-KV	25.00	1.04	0.00	2.08	0.00	0.26	0.00	3.96	20.49	22.92	26.04	9.25
	BTHA	99.42	99.31	82.76	94.18	78.69	64.97	54.83	20.64	42.88	28.74	36.91	63.94
Llama-3.1-8B-Instruct	Dense	100.00	98.96	100.00	97.92	77.08	94.27	96.09	51.04	75.35	78.12	40.62	82.68
	Loki	98.96	97.92	96.88	96.88	59.38	91.67	94.79	50.00	57.99	76.04	35.29	77.80
	Quest	100.00	93.75	47.92	97.35	53.12	78.91	90.10	61.25	63.19	73.96	38.54	72.55
	MP	94.79	69.79	51.04	65.62	20.83	44.79	57.81	41.67	57.99	67.71	38.54	55.51
	SL	1.04	4.17	3.12	5.21	3.12	4.69	5.73	0.62	67.70	51.04	34.38	16.44
	H2O	36.46	2.08	3.12	4.17	2.08	1.82	1.56	23.33	48.61	48.96	33.33	18.68
	S-KV	98.96	88.54	8.33	89.58	3.12	13.28	52.34	52.29	38.89	78.12	39.58	51.18
	BTHA	100.00	98.92	100.00	98.08	70.99	90.26	96.81	51.41	72.38	77.24	41.73	81.62

, BTHA consistently maintains accuracy close to the dense full-attention baseline on both evaluated models. On Llama-2-7B-32K-Instruct, BTHA achieves an average score of $63.94$, only $1.10$ points lower than dense full attention, while clearly outperforming all non-dense baselines. On Llama-3.1-8B-Instruct, BTHA obtains an average score of $81.62$, again only $1.10$ points below dense full attention and substantially higher than Loki, Quest, MagicPIG, StreamingLLM, H₂O, and SnapKV. These results demonstrate that BTHA can preserve long-range retrieval ability through block-level coarse routing and hash-based fine-grained retrieval, providing a favorable accuracy-efficiency trade-off for long-context inference.

(3) 

Task-level observations.

Although BTHA achieves the best or near-best average accuracy across the evaluated benchmarks, its gains are not uniform across all tasks. On LongBench-E, BTHA obtains larger improvements on tasks where the useful evidence is relatively sparse or concentrated, such as LCC, RepoBench, TQA, MultiFieldQA, and MultiNews. These tasks benefit from the Block-then-Hash design because block-level routing first narrows the search space to relevant context regions, while hash-based retrieval further selects fine-grained KV entries within the routed candidates. On RULER, BTHA performs particularly well on retrieval-oriented tasks such as needle search and multi-query retrieval, indicating that the learned hash codes can effectively preserve salient long-range evidence under a strict KV budget. Meanwhile, BTHA still shows small gaps compared with dense attention on several tasks, especially those requiring broad-context aggregation, highly diffuse evidence, or fine-grained counting and verification. For example, on LongBench-E, the gaps are more visible on tasks such as Passage Count and GovReport, while on RULER, performance drops are observed on some aggregation-oriented or full-context matching tasks such as FWE, NMK2, and NMV. This suggests that when the answer depends on many weakly relevant tokens distributed across the context, a fixed KV selection budget may discard information that is individually low-ranked but collectively useful. Overall, these task-level results show that BTHA is most effective when the attention mass is concentrated on a small subset of informative tokens, while further improvement is still possible for tasks requiring more exhaustive context coverage.

(4) 

End-to-end efficiency.

We further evaluate the end-to-end inference efficiency under the same configuration, with a prefill length of 36K tokens and a decode length of 3.6K tokens. In addition to the decoding latency, we also report the prefill time to provide a comprehensive comparison of the overall inference cost. As shown in Figure 3, BTHA consistently reduces the decoding time compared with dense full attention. On Llama-2-7B-32K-Instruct, BTHA achieves the lowest decoding cost among the compared methods, reducing the decode time from about 189 s for dense attention to about 95 s. On Llama-3.1-8B-Instruct, BTHA also substantially decreases the decoding cost, reducing the decode time from about 292 s to about 196 s. The prefill overhead of BTHA remains comparable to other methods, indicating that the proposed Block-then-Hash selection mainly accelerates the decoding stage without introducing significant prefill overhead. These results demonstrate that BTHA improves end-to-end inference efficiency while maintaining competitive accuracy under a fixed token selection budget.

(5) 

Layer-wise latency under different sequence lengths.

To further analyze the scalability of BTHA, we compare the latency of a single Transformer layer under different sequence lengths. As shown in Figure 4, the latency of dense attention increases rapidly as the sequence length grows, since it needs to access and compute attention scores over all cached KV positions. In contrast, BTHA maintains a much slower latency growth under the same 1.56% token selection ratio. For Llama-2, BTHA achieves the lowest latency across long sequence lengths and shows a clear advantage over Dense, Loki, and Quest when the sequence length increases to 128K and 256K. For Llama-3.1 with GQA, BTHA also keeps the latency nearly stable and remains substantially faster than Dense and Loki at long contexts. These results indicate that the Block-then-Hash selection mechanism effectively reduces the per-layer decoding cost and scales better with increasing context length.

(6) 

Model-scale generalization analysis.

To further examine whether the sparse retrieval mechanism remains effective across different model scales and context settings, we additionally evaluate BTHA on Qwen2.5-14B-Instruct-1M and Qwen2.5-32B-Instruct. As shown in

Table 4. Accuracy results on LongBench-E for Qwen2.5-14B-Instruct-1M [49] with sparse token budget = 512.

Methods	LCC	PRetr	HQA	TQA	Repo	Sam	Trec	MQA	2Wiki	Gov	PCnt	MltN	Qaspr	AVG.
Dense	44.32	100.00	65.96	88.41	36.25	45.52	76.34	53.73	60.68	31.93	22.83	22.14	41.41	53.04
BTHA	44.46	100.00	64.90	88.20	36.41	45.33	76.37	53.55	59.70	31.56	20.72	22.12	41.37	52.67

,

Table 5. Accuracy results on LongBench-E for Qwen2.5-32B-Instruct [49] with sparse token budget = 512.

Methods	LCC	PRetr	HQA	TQA	Repo	Sam	Trec	MQA	2Wiki	Gov	PCnt	MltN	Qaspr	AVG.
Dense	54.04	99.83	69.27	86.26	36.03	43.60	75.67	52.28	60.69	30.14	22.00	21.91	44.08	53.52
BTHA	53.84	99.83	68.88	86.32	35.82	43.14	75.27	52.09	60.21	29.02	21.90	21.84	43.90	53.24

and

Table 6. Accuracy results on RULER (256K) for Qwen2.5-14B-Instruct-1M [50] with sparse token budget = 4096 (1.56%).

Methods	NS1	NS2	NS3	NMK1	NMK2	NMV	NMQ	VT	FWE	QA1	QA2	AVG.
Dense	100.00	100.00	100.00	100.00	90.00	85.00	97.50	100.00	95.00	60.00	40.00	87.95
top-k	100.00	100.00	100.00	100.00	90.00	81.25	98.75	100.00	88.33	60.00	40.00	87.12
BTHA	98.00	100.00	99.00	100.00	96.50	88.00	96.50	95.00	86.50	61.00	46.50	87.91

, BTHA achieves accuracy close to dense attention under strict sparse token budgets. On LongBench-E, BTHA obtains comparable average performance to dense attention on both the 14B and 32B models with only 512 selected KV tokens, with average gaps of 0.37 and 0.28 points, respectively. On RULER with 256K context length, BTHA nearly matches dense attention under a 4096-token budget, with only a 0.04-point average gap, while outperforming the token-level top-k baseline. These results suggest that the proposed sparse retrieval strategy can generalize to larger models and longer contexts.

4.3. Ablation Study

We conduct ablation experiments on Llama-3.1-8B-Instruct to examine the effectiveness of the main design choices in BTHA. The evaluation focuses on the block selection ratio, the necessity of the Block-then-Hash retrieval pipeline, and the contribution of offline hash training. Unless otherwise specified, all experiments are conducted on LongBench-E under the same KV position budget.

(1) 

Effect of block selection ratio ${\rho }_b$.

The block selection ratio ${\rho }_b$ controls how many KV blocks are routed into the candidate set before hash-based token-level retrieval. A smaller ${\rho }_b$ reduces the candidate search space and improves efficiency, but it may discard relevant blocks before fine-grained retrieval. In contrast, a larger ${\rho }_b$ preserves more historical information but introduces more redundant candidates, increasing the overhead of hash comparison and sparse KV gathering.

Figure 5 and

Table 7. LCC results under different block selection ratios ${\rho }_b$ on Llama-3.1-8B-Instruct.

${\rho }_b$	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
LCC	67.85	67.77	67.80	68.62	67.92	68.21	68.04	67.78

show the LCC results under different values of ${\rho }_b$. The results indicate that increasing ${\rho }_b$ does not monotonically improve performance. When ${\rho }_b$ is too small, the routed candidate set may miss useful KV positions; when ${\rho }_b$ is too large, excessive irrelevant tokens are passed to the hash retrieval stage, which weakens fine-grained selection. BTHA achieves the best LCC score of 68.62 when ${\rho }_b=0.5$, suggesting that a moderate block selection ratio provides a better balance between candidate recall and redundancy reduction.

(2) 

Effect of the Block-then-Hash retrieval pipeline.

We further evaluate whether both retrieval stages are necessary. To this end, we compare BTHA with two simplified variants. The first variant removes block-level routing and directly performs hash retrieval, denoted as Hash Only. The second variant removes hash-based fine-grained retrieval and only uses block-level routing, denoted as Block Only.

As shown in

Table 8. Component ablation of block-level routing and hash-based retrieval on Llama-3.1-8B-Instruct.

Method	Block Routing	Hash Retrieval	LCC
Hash Only	No	Yes	67.46
Block Only	Yes	No	66.34
BTHA	Yes	Yes	68.62

, Hash Only achieves an LCC score of 67.46. Although it preserves token-level retrieval, it lacks the coarse block routing stage and therefore suffers from a larger search space. Block Only obtains a lower LCC score of 66.34, indicating that coarse block selection alone is insufficient because all tokens inside the selected blocks are treated uniformly. By combining block-level routing and hash-based token retrieval, BTHA achieves the highest LCC score of 68.62. These results show that the two stages play different and complementary roles. Block routing acts as a coarse-grained recall stage: by selecting query-relevant blocks, it increases the chance of covering dense-attention top-k KV positions while reducing the global search space. Hash retrieval then acts as a fine-grained filtering stage, removing redundant tokens inside the routed blocks and selecting more relevant token-level KV entries. Consequently, the full BTHA pipeline achieves the best LCC score by combining coarse top-k candidate recall with fine-grained token selection.

(3) 

Effect of offline hash training.

We also evaluate the effect of offline training on the hash network. Before training, the hash projection is randomly initialized and behaves similarly to random projection-based hashing. Although this provides a simple approximation for similarity search, it is not optimized for the query–key relevance pattern of LLM attention. After offline training, the hash network learns to approximate the KV position ranking induced by full attention, making the learned Hamming distance more consistent with the attention-based top-k positions.

Table 9. Effect of offline hash training on Top-k IoU using Llama-3.1-8B-Instruct.

Hash Variant	Training Status	Top-k IoU
Random Hash	Before Training	41.32%
Learned Hash	After Training	59.74%

reports the Top-k IoU before and after hash training. On Llama-3.1-8B-Instruct, the Top-k IoU improves from 41.32% before training to 59.74% after training. This significant improvement shows that offline training effectively aligns the hash space with full-attention retrieval targets, thereby improving the quality of fine-grained KV position selection.

(4) 

End-to-end efficiency.

The above ablation results show that BTHA benefits from both coarse-grained and fine-grained retrieval. Compared with Hash Only, BTHA avoids global hash comparison over the full KV cache by first routing a compact set of candidate blocks. Compared with Block Only, BTHA further removes redundant tokens inside the selected blocks through learned hash retrieval. Moreover, the trained hash network provides a more attention-aligned retrieval space than random hashing. These results demonstrate that the block routing stage, the hash retrieval stage, and offline hash training jointly contribute to the final accuracy–efficiency trade-off of BTHA.

(5) 

Token budget ablation.

First, we investigate the impact of token budget on the inference performance of BTHA. As shown in Figure 6, BTHA consistently achieves higher accuracy than Quest and Loki across different token-budget settings. More importantly, BTHA exhibits only mild performance degradation as the token budget decreases. Even under an extremely constrained setting where only about 0.4% of tokens are retained, BTHA still maintains competitive model accuracy. These results indicate that the learned hash-based retrieval mechanism enables BTHA to effectively preserve critical contextual information under strict KV selection budgets, highlighting the potential of hash learning for efficient long-context decoding.

We further investigate the effect of hash length on BTHA by varying the number of hash bits from 32 to 256. As shown in Figure 7, using too few hash bits leads to noticeable accuracy degradation, especially under the 32-bit setting. This is because overly compact hash codes provide limited discrimination capability for distinguishing important KV entries. As the number of hash bits increases, the accuracy improves rapidly and becomes stable when the hash length reaches 128 bits. On both LCC and GovReport, BTHA with 128 or 256 hash bits achieves performance close to the dense-attention baseline, and in some cases even slightly surpasses it. These results indicate that BTHA does not require excessively long hash codes to maintain retrieval quality; a moderate hash length is sufficient to preserve critical contextual information while keeping the hash representation compact and efficient.

5. Conclusions

In this paper, we presented BTHA, a budget-aware Block-then-Hash Attention framework for efficient long-context inference. BTHA decomposes KV position selection into two consecutive stages: block-level coarse routing and hash-based fine-grained retrieval. The block-level stage first identifies query-relevant historical regions using cached block mean keys, thereby reducing the search space before token-level selection. The hash stage then performs efficient position-level retrieval within the routed candidate blocks using shared learnable orthogonal hash projections and cached binary key codes. Finally, exact dot-product attention is computed only over the selected KV positions, preserving the standard attention computation on the retained tokens.

The proposed design offers both algorithmic and system-level benefits. Algorithmically, the two-stage selection mechanism combines coarse-grained context localization with fine-grained token retrieval, enabling BTHA to retain important KV positions under a fixed position budget. System-wise, BTHA maintains block mean caches and key hash caches incrementally, and uses low-level C/CUDA kernels for hash encoding, XOR-based Hamming computation, candidate filtering, and sparse KV gathering. These optimizations allow BTHA to be integrated into existing Transformer decoding pipelines without modifying model weights or the high-level model architecture.

Extensive experiments on LongBench-E and RULER demonstrate that BTHA consistently preserves accuracy close to dense full attention while substantially reducing the number of attended KV positions. Compared with representative sparse attention and KV cache compression baselines, BTHA achieves better average accuracy under the same KV position budget across both Llama-2-7B-32K-Instruct and Llama-3.1-8B-Instruct. These results indicate that hierarchical KV retrieval is an effective direction for scalable long-context inference. Future work may explore more expressive block representations and adaptive budget allocation across layers and heads.