Multi-Context Concatenation Across Requests for LLMs

Abstract

Reusing separate, pre-filled Key-Value (KV) Caches for multiple contexts has become a common practice in handling multi-context scenarios with Large Language Models. However, this leads to a lack of cross-attention mechanisms between contexts. To address this, we propose CatLLM, the first method that concatenates multiple contexts across requests offline to compensate for this deficiency. Specifically, during offline processing, CatLLM identifies contexts that severely lack cross-attention by incorporating the weighted inner products of Q and K vectors from tokens in an un-concatenated context into an equivalently transformed weighted formulation for concatenated Q and K inner products. This yields a weighting $w_i^{A+B}$ corresponding to the output vector difference, which can then be used to identify contexts with severe cross-attention deficiencies and concatenate them into a single context for KV Cache computation. Experimental results show that, compared to the baseline of separate caching (i.e., no concatenation), fully concatenating all contexts improves the F1 score by 6%. Meanwhile, the proposed method reduces the number of contexts requiring caching from 10 to 7 while achieving a 3% F1 score, thereby maximizing performance improvement while minimizing the degree of context compression.

1. Introduction

Large Language Models (LLMs) [1,2,3] demonstrate remarkable performance in foundational tasks such as text generation and code writing. However, as tasks grow increasingly complex, Retrieval-Augmented Generation (RAG) [4,5] systems frequently rely on substantial volumes of fixed system prompts, instructions, or in-context examples to achieve effective few-shot learning. A significant portion of this pre-loaded, tokenized information is repeated across multiple queries and inference sessions. Consequently, the efficient reuse of such repetitive content emerges as a critical challenge for enhancing the overall service efficiency and scalability of LLM-powered systems.

In current LLMs, the attention mechanism [6,7] operates based on the principle of Query-Key-Value (QKV) computation. Each token i is first converted from a word to a vector representation, denoted as $x_i$. This vector is then mapped to query, key, and value vectors through linear transformations:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_i=W_Q{x}_i,K_i=W_K{x}_i,V_i=W_V{x}_i,\end{array}\end{array}$$

(1)

where $W_Q$, $W_K$, and $W_V$ are weight matrices. Essentially, these transformations are function mappings that project the original vector $x_i$ into different vector spaces [8,9]. The product of query and key, $Q_i·K_j$ [10], represents a form of non-linear self-fusion of the vector itself, capturing the interaction between tokens i and j [11,12]. The softmax function then concretizes this fusion result into a probability distribution, where the sum of probabilities over all tokens equals 1.

Specifically, for token i, the attention scores with respect to all tokens j are computed as: ${\mathrm{s}}_{ij}=Q_i·K_j$ and the softmax [13,14,15] over these scores yields:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\alpha }_{ij}=\mathrm{softmax}\left({\mathrm{s}}_{ij}\right)={\displaystyle \frac{e^{{\mathrm{s}}_{ij}}}{{\sum }_k{e}^{{\mathrm{s}}_{ik}}}}.\end{array}\end{array}$$

(2)

The output for token i is then the weighted sum of value vectors:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill o_i=\sum _j{\alpha }_{ij}{V}_j.\end{array}\end{array}$$

(3)

This output $o_i$ serves as the input to the next layer in the LLM. It is important to note that LLMs employ a mask mechanism to prevent attending to future tokens. Specifically, the $QK$ product for certain token positions is artificially set to an infinitesimally small value (e.g., $-\infty$) before the softmax computation [16]. This ensures that when exponentiated, these values approach zero, and after applying softmax, the corresponding probabilities also tend to zero.

Based on the aforementioned $O\left(N^2\right)$ complexity, caching the Key and Value (KV) [17] of contexts has become a crucial strategy to address the efficiency issues in large model serving [18]. Existing methods typically perform prefill operations for multiple contexts separately to enable KV Cache reuse across requests. However, this approach inherently lacks cross-attention interactions between different contexts [19]. For example, the upper part of Figure 1 illustrates the process of computing KV Caches for two separate contexts, while the lower part shows the process of concatenating these two contexts into a longer one. In Figure 1, the purple lines indicate that, due to the mask mechanism, subsequent tokens rely on the previously computed KV Cache to obtain the output of the current layer, which also serves as the input to the next layer. The red lines highlight the differences in attention computation before and after concatenation. Here, we focus specifically on the attention computation for the first token of the second context, while the connections for other tokens are only vaguely indicated. As can be seen from the figure, for layers 1 and 2, the red lines intuitively represent the influence of attention from the original first context. However, a similar phenomenon exists between every pair of adjacent layers, causing the differences observed in layer 1 to be further amplified in the following layers. This means that the cross-attention between documents before and after concatenation gradually intensifies as the layers go deeper.

Figure 1. The difference in computing cross-attention with and without concatenation (the red lines indicate the source of the difference).

To mitigate this limitation, current research [20,21,22] primarily focuses on recomputing the KV states of certain important tokens, aiming to compensate for missing attention at critical positions. While this method alleviates the issue to some extent, it still faces a significant drawback: when the content of one context changes—such as the first context—the KV Cache of all subsequent contexts must still be recomputed [23]. This substantially increases the overhead of cache reuse and constrains further improvements in the overall service efficiency of LLMs.

In fact, when LLMs process different requests, certain pieces of contextual information are often retrieved together or eliminated together from the candidate set. This “same-in-same-out” property of contexts allows them to be concatenated into a continuous contextual unit. Such concatenation does not affect cross-request serving yet significantly mitigates the missing cross-attention problem caused by traditional separate prefill. However, concatenating all contexts into a single whole loses the ability to reuse KV Cache across requests, whereas insufficient concatenation provides limited mitigation for the missing cross-attention problem. To address this, we propose CatLLM, a multi-context concatenation strategy designed for cross-request scenarios. Directly concatenating contexts [24,25] can most effectively mitigate the absence of cross-attention.

Specifically, the objective of CatLLM is to maximize the compensation for missing cross-attention with minimal context concatenation. Therefore, we first need to identify the contexts that require concatenation the most, i.e., the contexts where the absence of cross-attention is most severe. To this end, when multiple contexts are provided, we analyze the missing cross-attention for both scenarios—concatenating two contexts or not concatenating them—and further discuss the difference between the two cases. Unlike using the original attention formula [26,27,28], we extend it to compute the outputs for both cases and calculate the difference between them. In this process, we find that whether two contexts are concatenated is influenced by a softmax-like weighting factor. Thus, by reducing the value of this weighting factor, we can mitigate the impact caused by missing cross-attention. Extending this idea further, by selecting contexts for concatenation based on this weight, we can obtain the most reasonable context concatenation strategy.

Our contributions are primarily as follows:

• We propose the first solution for concatenating multiple contexts across requests in LLMs, providing an important reference for question-answering tasks.
• Our concatenation scheme fundamentally mitigates the lack of cross-context attention caused by prefilling multiple contexts separately.
• Experiments show that our method effectively improves inference performance while enabling the reuse of multi-context KV Cache across requests.

In the subsequent sections of this paper, Section 2 provides a detailed derivation of the concatenation principle for CatLLM and proposes a greedy algorithm logic for multiple concatenations. Section 3 describes the experimental setup for CatLLM, while Section 4 presents the experimental results along with corresponding analysis and discussion. Finally, Section 5 concludes the paper by summarizing the contributions of the concatenation strategy and offering prospects for future research.

2. Our Proposed CatLLM

In LLM inference where KV Cache is prefilled based on multiple contexts separately, there is a lack of cross-attention mechanisms among different Caches. Compared to recalculating some tokens in real-time based on the required contexts when a request arrives, directly concatenating contexts for prefilling can effectively avoid attention deficiency. However, if all contexts are concatenated and prefilled at once, the ability to reuse across requests will be lost. To address this, this paper proposes the CatLLM method, which maximizes the mitigation of cross-attention deficiency by prefilling a small number of concatenated contexts while preserving cross-request reuse capabilities.

2.1. Derivation of Equations

To simplify the scenario, we focus only on two contexts here: context A and context B (lengths of $n^A$ and $n^B$, respectively). In the case of multiple contexts, their KV Caches are denoted as $K{V}^A$ and $K{V}^B$, respectively. The KV Cache for the concatenated context is referred to as the $K{V}^{A+B}$. This section primarily discusses the impact on the output (o) of the first layer for context B under these two conditions.

According to the original formulation of attention, it can be deduced that for token i in context B at layer 1, only the KV within context B itself needs to be considered in the multiple contexts, denoted as $o_i^B$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill o_i^B={\displaystyle \frac{\begin{array}{c}\hfill {\sum }_{j=1}^i{e}^{Q_i^B{K}_j^B/\sqrt{d}}{V}_j^B\end{array}}{\begin{array}{c}\hfill {\sum }_{j=1}^i{e}^{Q_i^B{K}_j^B/\sqrt{d}}\end{array}}}.\end{array}\end{array}$$

(4)

Here, d denotes the dimension of $QKV$. Similarly, the representation of token i in the concatenated context is computed. At this point, token i needs to consider the KV from both context A and context B, denoted as $o_i^{A+B}$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill o_i^{A+B}={\displaystyle \frac{{\sum }_{j=1}^{n^A}{e}^{Q_i^B{K}_j^A/\sqrt{d}}{V}_j^A+\begin{array}{c}\hfill {\sum }_{j=1}^i{e}^{Q_i^B{K}_j^B/\sqrt{d}}{V}_j^B\end{array}}{{\sum }_{j=1}^{n^A}{e}^{Q_i^B{K}_j^A/\sqrt{d}}+\begin{array}{c}\hfill {\sum }_{j=1}^i{e}^{Q_i^B{K}_j^B/\sqrt{d}}\end{array}}}.\end{array}\end{array}$$

(5)

By comprehensively observing the blue parts of Equations (4) and (5), we can roughly see that their forms are completely identical. The following will prove that they are also numerically identical: although the position of token i in Equation (4) is i, while in Equation (5) it becomes $n^A+i$, the blue parts both represent the computation of i within context B. Therefore, the relative position between token i and other tokens in context B remains unchanged. Since RoPE’s positional encoding [29] has the property that the inner product of Q and K remains unchanged when the relative position is constant, although $Q_i^B$ and $K_j^B$ have changed, their inner product $Q_i^B{K}_j^B$ remains unchanged, and thus they are also numerically identical. By substituting Equation (4) into Equation (5) and rearranging, we obtain:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill o_i^{A+B}-o_i^B={\displaystyle \frac{{\sum }_{j=1}^{n^A}{e}^{Q_i^B{K}_j^A/\sqrt{d}}(V_j^A-o_i^{A+B})}{{\sum }_{j=1}^i{e}^{Q_i^B{K}_j^B/\sqrt{d}}}}.\end{array}\end{array}$$

(6)

As can be seen from Equation (6), this structure resembles the result of a softmax similar to attention weights multiplied by the Value (V). If we aim to reduce the missing cross-attention between contexts to zero without recomputing tokens—that is, make $o_i^{A+B}-o_i^B$ approach zero, the most straightforward approach is to decrease the weight $w_i^{A+B}$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill w_i^{A+B}={\displaystyle \frac{{\sum }_{j=1}^{n^A}{e}^{Q_i^B{K}_j^A/\sqrt{d}}}{{\sum }_{j=1}^i{e}^{Q_i^B{K}_j^B/\sqrt{d}}}}.\end{array}\end{array}$$

(7)

It should be noted that in Equation (7), the position of $Q_i^B$ in the numerator is $n^A+i$, while in the denominator, the position of $Q_i^B$ is i. These two are not the same, and positional encoding should be assigned accordingly based on this requirement.

From Equation (7), it can be seen that the goal is for $w_i^{A+B}$ to be 0, which represents a traditional sparsity structure. However, since traditional sparsity is only suitable for analyzing the importance of the KV itself to sparsify attention, and is not applicable to multi-context concatenation tasks, we extend the original attention mechanism accordingly.

2.2. Interpretation of Equations

Through the analysis of Equation (7), the influencing factors of $w_i^{A+B}$ can be decomposed as follows:

• $n^A$: If context B and $K_j^A$ remain unchanged, when $n^A$ increases, the denominator of $w_i^{A+B}$ remains constant while the numerator increases, leading to a larger value. This indicates that when $K{V}^A$ is placed directly before $K{V}^B$, the longer context A is, the greater its impact on the cross-attention deficiency of B.
• i: If context A and $Q_i^B$ remain unchanged, a smaller position index i results in an unchanged numerator for $w_i^{A+B}$ and fewer accumulated terms in the denominator, leading to a larger value. This means that when $K{V}^A$ is placed before $K{V}^B$, the effect is most significant for earlier tokens (e.g., the first token) in context B and weakest for later tokens (e.g., the last token).
• $K_j^A$: If context B and $n^A$ remain unchanged, and the token j in context A has a stronger correlation with $Q_i^B$ (i.e., a larger inner product), then the numerator of $w_i^{A+B}$ increases while the denominator stays the same, thus $w_i^{A+B}$ increases accordingly.
• $K_j^B$: If context A and $Q_i^B$ remain unchanged, and the token j in context B has a stronger correlation with $Q_i^B$ (i.e., a larger inner product), then the numerator of $w_i^{A+B}$ stays the same while the denominator increases, thus $w_i^{A+B}$ decreases accordingly.
• $Q_i^B$: If $n^A$, $K_j^A$, and $K_j^B$ all remain unchanged, the effect of changes in $Q_i^B$ on $w_i^{A+B}$ is difficult to judge intuitively.

In fact, the magnitude of $w_i^{A+B}$ is jointly influenced by multiple factors, making it difficult to effectively control its value through direct adjustment of a single factor. Therefore, it is necessary to conduct further analysis of $w_i^{A+B}$ in order to more clearly reveal its inherent characteristics.

To clearly express the overall influence of context A on B and avoid confusion with the notation for the combined tokens A+B, we define $w^{A\to B}$ as the aggregated influence weight of all tokens in A on all tokens in B:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill w^{A\to B}={\displaystyle \frac{1}{n^B}}\sum _i^{n^B}{w}_i^{A+B}.\end{array}\end{array}$$

(8)

By substituting different contexts for A and B in the equation, we obtain Figure 2a, which illustrates the $w^{A\to B}$ values corresponding to all attention heads across various contexts. It can be observed from the figure that $w^{5\to B}$ for context 5 is generally large across most attention heads, whereas the values for context 10 are mostly small. This size pattern is consistent across different attention heads—either generally high or generally low. To simplify the representation and better observe the overall trend, we average the values across all attention heads and reduce them to a single representation. The result is shown in Figure 2b (top).

Figure 2. $w^{A\to B}$ of the Llama-3.1-8B-Instruct model across 10 contexts on the 2WikiMQA sample: (a) per-head $w^{A\to B}$; (b) top: average $w^{A\to B}$ over all heads, bottom: column-wise mean as average aggregation weight $w^{·\to B}$.

Combining Figure 2b (top) and Equation (8), it can be observed that the value of $w^{5\to 10}$ is the largest, indicating that whether context 5 is concatenated before context 10 has the most significant impact on the absence of its cross-attention. Further examination of Figure 2b as a whole reveals that context 5 has a relatively significant influence on all other contexts, while the influence of other contexts on context 5 is comparatively smaller. This suggests that the impact reflects more of the intrinsic characteristics of the context itself. Therefore, we simplify Figure 2b (bottom) by computing the mean of each column, obtaining the average influence degree of any other context on context B, denoted as $w^{·\to B}$, as shown in Figure 2b (bottom):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill w^{·\to B}={\displaystyle \frac{1}{\left|Contexts\right|}}\sum _A^{Contexts}{w}^{A\to B}.\end{array}\end{array}$$

(9)

Here, $Contexts$ denotes all contexts, and $\left|Contexts\right|$ represents the total number of contexts, which also includes context B itself. In this case, when context B is concatenated before context B, the position encoding of the latter context B needs to be reset based on the premise that there is a preceding context B. This results in $w^{B\to B}$ not being equal to 1 [16].

As can be seen from Figure 2b (bottom), the value of $w^{·\to 5}$ is the smallest, and the number of tokens in context 5 is the largest (1607); while the value of $w^{·\to 10}$ is the largest, and the number of tokens in context 10 is the smallest. Considering the relationship between $w^{·\to B}$ and the corresponding token count $\left|tokens\right|$ in the remaining contexts, a negative correlation trend is observed between the two.

2.3. Concatenate Contexts

Prefilling multiple contexts separately leads to a lack of cross-attention between them. Therefore, concatenating a small portion of contexts not only allows reuse of KV Cache across requests, but also effectively mitigates the missing cross-attention. However, Figure 2a,b only present a single sample, which only indicate that $w^{·\to B}$ is negatively correlated with $\left|tokens\right|$. In fact, across all samples, $w^{·\to B}$ does not strictly decrease monotonically with $\left|tokens\right|$ but shows a negative correlation trend. To derive a more rigorous concatenation strategy, we choose to follow the pattern shown in Figure 2b (bottom), i.e., concatenating the two contexts with the largest $w^{·\to B}$ values.

In addition to determining which two contexts to splice, another key issue is the number of splices. We compare the effects of splicing from 1 to 5 times for two reasons: first, as the first systematic exploration of context splicing strategies, there is no established baseline for improvement; second, in real-world industrial scenarios, meeting diverse cross-request needs [30] and performance requirements involves flexible KV Cache pre-storage strategies without a one-size-fits-all solution. For example, the number of concatenations can be influenced by factors such as device requirements. In server applications, thanks to the strong performance of the server itself, it is assumed that it can simultaneously load the Top-1000 KV Cache. In this case, even if a small amount of erroneous information is concatenated into one of the contexts across other rounds of requests, the presence of 1000 contexts ensures that a few errors are not enough to dilute the vast amount of relevant information. Under such circumstances, multiple concatenations are recommended. However, for devices with limited computational performance, such as mobile phones, it is assumed that their performance can only support loading the Top-10 KV Cache. In this scenario, the insertion of irrelevant context can lead to overall information confusion, so the number of concatenations should be reduced. Therefore, the decision depends on the model’s performance and the practical conditions of the industrial environment. By enumerating various concatenation results, it becomes easier to select the appropriate number of concatenations.

In multiple concatenation rounds, adding short contexts increases their token count and reduces their weight $w^{·\to B}$, which may eventually remove them from having the lowest weight in later rounds and thus halt further concatenation. Consequently, candidate contexts vary dynamically across rounds. We employ a greedy strategy that selects the context with the smallest $w^{·\to B}$ within the current context for concatenation in each round. The detailed procedure is outlined in Algorithm 1.

In Algorithm 1, $\mathrm{Concatenate}(B_1,B_2)$ denotes appending the text of $B_1$ to the front of the text of $B_2$. The variable $Num$ specifies the minimum number of contexts to retain after concatenation, effectively setting a lower bound on the number of concatenated contexts. Meanwhile, we limit the concatenation process to at most K iterations to control its extent. These two variables are introduced to constrain the concatenation process because only a very small number of samples in the dataset deviate from having 10 contexts, resulting in an overall average of 9.93 contexts per sample—very close to 10. For the sake of readability, we uniformly denote the number of contexts as 10 throughout this paper. Although both K and $Num$ serve to constrain the concatenation process, K is primarily used to facilitate implementation, while $Num$ is designed to simplify the comparison and analysis of the resulting number of contexts. Thus, they serve different purposes.

Algorithm 1 Greedy Selection of Context for Concatenation

Require:  Contexts to be concatenated $Contexts={\{Contex{t}_1,Contex{t}_2,\cdots \}}_N$ Ensure:  Concatenated contexts $Context{s}^{\prime }={\{Contex{t}_1^{\prime },Contex{t}_2^{\prime },\cdots \}}_{N^{\prime }}$ ($N^{\prime }<N$)   1: $k=0$   2: while  $N>Num$  and  $k<K$  do   3:       $k=k+1$   4:       $w{s}^{·\to B}=⌀$   5:       for all $B\in Contexts$ do   6:             compute $w^{·\to B}$   7:             $w{s}^{·\to B}=w{s}^{·\to B}\cup \left\{w^{·\to B}\right\}$   8:       end for   9:       Identify $B_1$, $B_2$ with the largest $w^{·\to B}$ 10:       $Contexts=Contexts\setminus \{B_1,B_2\}$ 11:       $Contex{t}^{\prime }=\mathrm{Concatenate}(B_1,B_2)$ 12:       $Contexts=Contexts\cup \left\{Contex{t}^{\prime }\right\}$ 13: end while 14: $Context{s}^{\prime }=Contexts$

Note that the concatenation between contexts can be completed offline, rather than in response to online requests, so it will not affect real-time responsiveness. To ensure the rigor of our approach, we also provide a theoretical derivation of the offline runtime for practical reference. In fact, the algorithm undergoes further optimization during implementation, and the optimized procedure differs slightly from Algorithm 1, as detailed below:

• When some contexts are not concatenated, their weights do not need to be recomputed, thereby reducing a portion of the computation time. For example, in a given concatenation operation, if the algorithm concatenates contexts A and B, while contexts C and D remain un-concatenated, then $w^{C\to D}$ does not need to be recomputed. Consequently, for the computation of $w^{·\to D}$, some weight values can be skipped.
• In implementation, the numerator and denominator of $w^{A\to B}$ are obtained by directly decomposing the softmax rather than by a second computation. The original softmax computation is already part of the regular inference within the model, so only the ratio of the decomposed terms needs to be calculated, incurring no additional weight computation complexity.

According to our calculations, the offline time consumption of this concatenation operation satisfies $O\left(\min \right(K,N-\mathrm{Num})\times L)$, where L denotes the total number of tokens for all contexts relevant to a given question. The length is linear because we only use the first layer, independent of the number of layers in the LLM. As for the specific numerical value of $O\left(L\right)$ time consumption, it depends on the model type and device computation bandwidth. This time complexity is provided here for reference, to facilitate selection based on K, Num, and L. However, this time consumption occurs entirely before the user raises the question, and thus it is not perceived during the user’s questioning and response processes.

3. Experiments

Implementation. We use the original Huggingface [31] Transformers in PyTorch 2.1.0 [32], and extract the first layer of the model for independent computation to serve as the basis for concatenation.

Datasets. In this experiment, we select two commonly used question-answering datasets from LongBench [33]—2WikiMQA and HotpotQA—for evaluation. Each dataset contains 200 samples, with each sample consisting of a question, ten context passages, and one or more reference answers. It is worth noting that in these datasets, all ten context passages are essential for answering the corresponding question; none of them can be omitted. If only a subset of the context passages is used (e.g., extracting five of them for the experiment), any resulting performance degradation cannot be attributed solely to the absence of necessary information or the failure of the algorithm. Therefore, we are unable to experiment with subsets of context and must instead use the full set of context passages. Additionally, in the evaluation, we adopt the F1 score as the metric.

Models. We select the Qwen [34,35,36] and Llama [37] series models for evaluation, a decision motivated primarily by their widespread adoption and significant influence within the research community. This choice ensures that our findings are relevant to popular and commonly used architectures. In terms of model scale, we aim to balance representational capacity with practical hardware constraints. Consequently, we opt for the 3B and 8B parameter variants—specifically, the Qwen2.5-3B-Instruct (https://huggingface.co/Qwen/Qwen2.5-3B-Instruct, accessed on 10 August 2025) and Llama-3.1-8B-Instruct (https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct, accessed on 17 August 2025) models. This selection allows us to examine performance across different model sizes while remaining feasible under our computational resources.

Methods. To intuitively illustrate the impact of context concatenation (cat), we compare three methods: NoCat (no concatenation; prefill KV Cache separately then concatenate for inference; baseline), AllCat (full concatenation into a single context for inference), and CatLLM (concatenating context K times, with K ranging from 1 to 5). In real-world industrial settings, the value of K depends on practical requirements. Here it is used only for testing and demonstration. Figure 3 provides a visual comparison of these different approaches.

Figure 3. Schematic diagram of different text concatenation methods.

Environment. We conduct experiments on a server with a 64 GB 910B3 NPU, a 256-core Kunpeng 920 CPU (2.60 GHz, hyper-threading disabled), 2 TB DRAM, running EulerOS 2.0 (kernel 5.10.0).

Details. Experiments reveal that better results can be achieved by removing the position encoding from all KV Cache items, concatenating them, and then reassigning position encodings based on their respective locations. Therefore, this approach is applied to all methods in this paper, including the NoCat method where no context is concatenated—each KV Cache is concatenated and then reassigned position information.

4. Results and Discussion

Before quantitatively analyzing the F1 scores of specific models on the dataset, we first conduct a qualitative analysis of two extreme scenarios:

• NoCat: No contexts are concatenated, which results in the maximum absence of cross-attention between contexts and should yield the poorest performance.
• AllCat: All contexts are concatenated into a single context, ensuring no loss of cross-attention, which should lead to the best performance.

Given that CatLLM lies between these two extremes, its performance is expected to fall between them. For instance, compared to NoCat, CatLLM achieves better results but reduces the number of contexts. However, a decrease in the number of contexts may diminish its cross-request applicability.

While higher F1 scores and larger numbers of contexts are both desirable, they exhibit a negative correlation. To represent both aspects simultaneously with a single metric, we define $\Delta \mathrm{F}1/\Delta \#$ (where # denotes the number of contexts):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\displaystyle \frac{\Delta \mathrm{F}1}{\Delta \#}}={\displaystyle \frac{{\mathrm{F}1}_{\mathrm{Method}}-{\mathrm{F}1}_{\mathrm{NoCat}}}{{\#}_{\mathrm{NoCat}}-{\#}_{\mathrm{Method}}}}.\end{array}\end{array}$$

(10)

From Equation (10), it can be observed that, on one hand, a larger numerator indicates a higher F1 score of the method, meaning better performance; on the other hand, a smaller denominator suggests that the amount of concatenated context is closer to NoCat scenario, implying stronger cross-request capability. Overall, a larger value of $\Delta \mathrm{F}1/\Delta \#$ is desirable. It should be noted that Equation (10) is not a strictly derived evaluation formula and is only suitable for roughly analyzing the relative trade-off between F1 improvement and cross-request capability degradation from a qualitative perspective. To synthesize performance across multiple datasets, we compute the mean F1 score (Avg.) across all datasets and finally calculate $\Delta \mathrm{Avg}./\Delta \#$. Therefore, a larger $\Delta \mathbf{Avg}\mathbf{.}/\Delta \#$ indicates a greater improvement in F1 but a smaller decrease in #, that is, a higher cost-effectiveness of a splicing method.

4.1. Overall Improvement

After experimentation, we present the effectiveness of NoCat and AllCat, along with the best-performing results from CatLLM, in Table 1. As shown in Table 1, compared to the NoCat method, CatLLM consistently improves the performance of different LLMs while only slightly reducing the number of contexts. Specifically, on the 2WikiMQA dataset, the F1 scores of both models increase by an average of approximately 5%, approaching the performance of AllCat, whereas on HotpotQA, the improvement is smaller, ranging from 0.7% to 0.9%. Despite the differences in model scale and architecture, both exhibit a consistent trend. This discrepancy may stem from the inherent characteristics of the datasets, making the method more suitable for 2WikiMQA. This paper focuses on the context concatenation mechanism and does not explore this aspect further.

Table 1. Comparison of the effects of different concatenation methods, where CatLLM demonstrates the best F1 score here. # represents the number of concatenated contexts.

Methods	2WikiMQA		HotpotQA		Avg.	$\Delta \mathbf{Avg}./\Delta \#$
	#	F1	#	F1
Qwen2.5-3B-Instruct
NoCat	10	26.98	10	32.81	29.90	-
AllCat	$1_{-9}$	${33.34}_{+6.36}$	$1_{-9}$	${40.11}_{+7.30}$	${36.73}_{+6.83}$	0.76
CatLLM	$7_{-3}$	${32.70}_{+5.72}$	$7_{-3}$	${33.58}_{+0.77}$	${33.14}_{+3.24}$	1.08
Llama-3.1-8B-Instruct
NoCat	10	13.01	10	20.75	16.88	-
AllCat	$1_{-9}$	${21.48}_{+8.47}$	$1_{-9}$	${24.12}_{+3.37}$	${22.80}_{+5.92}$	0.66
CatLLM	$6_{-4}$	${18.30}_{+5.29}$	$6_{-4}$	${21.69}_{+0.94}$	${20.00}_{+3.12}$	0.78

A further analysis of $\Delta \mathrm{Avg}./\Delta \#$ reveals that although CatLLM does not perform as well as AllCat, the total number of concatenated contexts it produces is significantly larger than that of AllCat, thereby granting it stronger cross-request capability, which also leads to a greater $\Delta \mathrm{Avg}./\Delta \#$. Since $\Delta \mathrm{Avg}./\Delta \#$ can only be used for qualitatively assessing the cost-effectiveness of a concatenation method rather than for quantitative comparison, we do not analyze the differences in specific numerical values here.

4.2. Detailed Experiments

In addition to comparisons with other methods, we conduct experiments with different numbers of splicing operations, ranging from 1 to 5 times, as shown in Table 2. From the table, it can be observed that as the number of splicing operations increases, the # gradually decreases, while the F1 score generally shows an upward trend. However, this increase is not strictly monotonic—the overall trend is upward, but occasional performance drops also occur. This indicates that some splicing operations do not effectively improve performance and may even have negative effects, a phenomenon observed in both models. This suggests that sacrificing certain cross-request capabilities does not necessarily lead to a corresponding improvement in performance. Of course, when such negative effects occur, their impact on the F1 score is relatively minor, approximately 1%. However, on the other hand, the negative impact on the reduction in the number of contexts is inevitable, inevitably leading to a decrease of 1 in the number of contexts, which directly affects the practicality of cross-request operations. This also highlights the necessity of using $\Delta \mathrm{Avg}./\Delta \#$ for qualitative comparisons.

Table 2. Comparison of the effects after concatenating different times in CatLLM. # represents the number of contexts after concatenation.

2WikiMQA		HotpotQA		Avg.	$\Delta \mathbf{Avg}./\Delta \#$
#	F1	#	F1
Qwen2.5-3B-Instruct
$9_{-1}$	${27.12}_{+0.14}$	$9_{-1}$	${32.79}_{-0.02}$	${29.96}_{+0.06}$	0.06
$8_{-2}$	${30.29}_{+3.31}$	$8_{-2}$	${33.11}_{+0.30}$	${31.70}_{+1.80}$	0.90
$7_{-3}$	${32.70}_{+5.72}$	$7_{-3}$	${33.58}_{+0.77}$	${33.14}_{+3.24}$	1.08
$6_{-4}$	${30.13}_{+3.15}$	$6_{-4}$	${32.58}_{+0.23}$	${31.36}_{+1.46}$	0.37
$5_{-5}$	${31.73}_{+4.75}$	$5_{-5}$	${33.42}_{+0.61}$	${32.58}_{+2.68}$	0.54
Llama-3.1-8B-Instruct
$9_{-1}$	${13.92}_{+0.91}$	$9_{-1}$	${20.94}_{+0.19}$	${17.43}_{+0.55}$	0.55
$8_{-2}$	${14.05}_{+1.04}$	$8_{-2}$	${20.83}_{+0.08}$	${17.44}_{+0.56}$	0.28
$7_{-3}$	${17.12}_{+4.11}$	$7_{-3}$	${21.07}_{+0.32}$	${19.10}_{+2.22}$	0.74
$6_{-4}$	${18.30}_{+5.29}$	$6_{-4}$	${21.69}_{+0.94}$	${20.00}_{+3.12}$	0.78
$5_{-5}$	${17.96}_{+4.95}$	$5_{-5}$	${21.66}_{+0.91}$	${19.81}_{+2.93}$	0.59

As shown in Table 2, as concatenation progresses, the number of contexts gradually decreases, and the overall F1 shows an upward trend, though not strictly monotonic. For example, Qwen achieves a better F1 when the number of contexts is 7 and performs well across both datasets, while in the Llama model, the best F1 is achieved when the number of contexts is 6. These observations indicate that a certain degree of concatenation benefits model performance. In practice, it can be observed that when contexts are concatenated 6 or 7 times, most contexts become relatively long and of similar length, resulting in a smaller variance in the number of $\left|tokens\right|$. In this state, the influence among similarly long contexts is minimized. However, as concatenation continues, the length difference among contexts—which initially converged—increases again, widening the gap between the maximum and minimum values of $w^{·\to B}$ and leading to a slight decline in performance. Due to the current limitation of 10 contexts, we set an upper limit of 5 concatenations. We hypothesize that if the initial number of contexts were unlimited, the lengths of contexts would repeatedly undergo cycles of convergence and divergence during iterative concatenation, and the F1 score would exhibit a spiral pattern of rising, gradually declining, and then rising again.

From Table 2, a simple conclusion applicable to real production environments can be drawn: appropriately concatenating contexts can not only effectively compensate for the lack of cross-attention between contexts and further enhance model performance; however, once the lengths of the contexts have become similar, further increasing the number of concatenations will enlarge the length differences among the contexts, which may lead to a slight decline in model effectiveness.

4.3. Online Process Analysis

Although CatLLM requires pairwise computations across multiple documents, which can be time-consuming, these computations are performed offline and do not affect online processing. Therefore, they introduce no additional latency. However, in similar multi-context scenarios, recomputation-based methods such as CacheBlend and EPIC operate by first caching the KV Cache and then, during online inference, loading the corresponding KV Cache based on the required context and performing necessary recomputation. This process inevitably introduces latency. In contrast, CatLLM performs its computations offline before the KV Cache is stored. As a result, these recomputation-based methods and CatLLM belong to different stages of KV Cache management. Theoretically, by using CatLLM to compute the KV Cache before storage and then leveraging CacheBlend and EPIC during runtime, the benefits of both approaches could be combined. However, since such integration is more of an engineering extension and adaptation rather than a focus of this paper, it will not be explored in detail here.

Additionally, since CatLLM operates entirely offline, there is no increase in GPU memory consumption or any change in throughput. This is because the total number of tokens in the KV Cache remains the same and unchanged. In summary, CatLLM behaves identically to NoCat or AllCat during the inference process.

4.4. Concatenation Basis and Layer Selection

In current LLMs, the shallower layers tend to focus more on syntactic processing, while the deeper layers emphasize semantic understanding. However, in this paper, we choose to utilize the first layer rather than deeper ones, and we provide an analysis and explanation for this choice.

First, we present a straightforward qualitative analysis: according to conventional reasoning, if there is a significant difference in the weights before and after concatenation at the shallow syntactic processing stage, this difference will only be amplified, rather than diminished, in deeper layers. This is because, as the layers go deeper, the lack of attention caused by not concatenating the preceding context becomes increasingly severe. Therefore, if the difference is already substantial in the first layer, it will continue to widen in subsequent layers, making deeper analysis unnecessary. This idea is somewhat similar to CacheBlend’s approach of computing attention differences only for the first layer. However, both the formulas used for computation and the application directions differ—CacheBlend is designed for recomputation, while CatLLM is intended for concatenation. These reasons explain our preference for selecting the first layer.

In fact, during the preliminary experiments of this paper with 3B and 8B models, we design experiments targeting different layers to verify the feasibility of this theory. The experimental results showed that, compared to other layers, selecting concatenation objects based on the first layer yields better concatenation outcomes. This finding aligns with the qualitative analysis mentioned above and ultimately led us to choose the first layer.

5. Conclusions

This paper proposes CatLLM, the first multi-context concatenation framework designed for cross-request scenarios. It systematically compensates for the missing cross-attention caused by independent pre-filling without recalculating the KV Cache. By deriving an upper bound for cross-attention weight differences, we demonstrate that offline concatenation of only a small number of “most influential” contexts can significantly enhance the model’s reasoning quality while incurring almost no loss in cross-request reuse capability. Experiments on 2WikiMQA and HotpotQA show that by sacrificing only 2–3 independent cache units, CatLLM improves the average F1 score of Qwen2.5-3B-Instruct and Llama-3.1-8B-Instruct by 3%.

More importantly, CatLLM provides a “plug-and-play” trade-off solution for industrial long-text services: it preserves the RAG system’s ability to flexibly reuse arbitrary combinations of multiple contexts while closely approximating the full attention upper bound with minimal concatenation. Future work will proceed in three directions: (i) extending the weight computation from the first layer to multiple layers for more precise “layer-head”-level concatenation; (ii) integrating dynamic sparsification techniques to further compress KV Cache memory usage after concatenation; and (iii) validating its generalizability across more long-context reasoning tasks and larger model scales and releasing a unified benchmark to promote ongoing community exploration of the “cross-request multi-context reasoning” paradigm.