Characterizing KV-Cache Performance in Large Language Model Inference: Context Length, Attention Architecture, and GPU Memory Residency

Abstract

Large language models (LLMs) commonly employ key–value (KV) caching to avoid recomputing previous attention states during autoregressive inference. However, its measured performance benefit depends on workload characteristics and the memory environment. This study presents a controlled systems characterization across input context length, model-specific grouped-query attention configurations, and GPU-memory residency. Experiments use Qwen2.5-3B and SmolLM3-3B under verified fully resident conditions with fixed 128-token generation, prompt lengths from 128 to 2048 tokens, randomized cache-enabled/cache-disabled execution, GPU telemetry, and repeated paired validation. End-to-end generation speedup increases from 1.34× to 8.31× for Qwen2.5-3B and from 1.28× to 8.62× for SmolLM3-3B. At 2048 prompt tokens, 30 randomized paired trials confirm mean speedups of 8.10× and 8.51×, corresponding to runtime reductions of 87.65% and 88.25%, while physical GPU-memory utilization remains below 61%. SmolLM3-3B requires twice the theoretical KV-cache storage per token, yet both models show comparable context-dependent scaling in the fully resident regime. Because the models differ in characteristics beyond KV-head count, this cross-model comparison is descriptive and does not attribute the observed performance difference causally to GQA configuration. A separate Mistral-7B case study shows that apparent end-to-end speedup changes substantially under memory pressure; the underlying memory-system mechanism is not isolated. These results establish cache-related end-to-end generation speedup as workload- and system-dependent and demonstrate that meaningful benchmarking requires reporting context length, cache configuration, model and cache footprints, physical accelerator-memory residency, runtime, throughput, and relevant GPU-memory conditions during each inference experiment.

1. Introduction

Large language models (LLMs) based on the Transformer architecture have become a fundamental computational component of modern artificial intelligence systems. Their applications span natural language processing, code generation, information retrieval, decision support, and interactive intelligent systems. Despite their capabilities, deploying LLMs remains computationally demanding, particularly during autoregressive inference. Unlike training, where many tokens can be processed in parallel, autoregressive generation produces tokens sequentially. Each newly generated token becomes part of the context used to predict the next token, making inference latency and resource utilization increasingly important as sequence lengths grow.

A major optimization used in decoder-based Transformer inference is the key–value (KV) cache. During self-attention, each Transformer layer computes query, key, and value representations. Without caching, the key and value representations associated with previously processed tokens must be recomputed during subsequent decoding steps. KV caching avoids this repeated computation by retaining previously calculated key and value tensors and reusing them as generation proceeds. The optimization therefore exchanges additional memory consumption for reduced computation.

The memory cost of KV caching is closely related to attention architecture. Conventional multi-head attention (MHA) maintains independent key and value representations for each attention head. Multi-query attention (MQA) shares a single pair of key and value heads across query heads [1], while grouped-query attention (GQA) provides an intermediate design in which groups of query heads share key–value heads [2]. These choices create systems-level tradeoffs among memory footprint, bandwidth, and computational reuse.

Efficient management of KV state has received substantial attention. FlashAttention demonstrated the importance of reducing data movement through the GPU memory hierarchy [3]. PagedAttention and vLLM addressed fragmentation and allocation of dynamically growing KV caches [4], while FlexGen explored LLM inference under constrained GPU memory using coordinated GPU, CPU, and storage resources [5]. Cache compression, quantization, selection, and offloading have also become active research areas [6,7,8,9,10].

However, a methodological issue remains when interpreting KV-cache benchmarks. Measured speedup is not necessarily an intrinsic constant of a model. It can depend on prompt length, generation length, attention configuration, numerical precision, software implementation, and the memory regime in which inference executes. When a model and runtime state fit entirely within physical GPU memory, the cache-enabled/cache-disabled comparison provides a relatively clean characterization of computational reuse. When the working set approaches or exceeds GPU capacity, paging, migration, offloading, allocation pressure, or other memory-system effects may materially alter wall-clock time.

This study therefore characterizes conventional KV caching while explicitly separating fully GPU-resident and memory-constrained execution. Qwen2.5-3B and SmolLM3-3B are evaluated as the primary fully resident models. Both contain 36 Transformer layers, hidden dimension 2048, and 16 query heads, but Qwen2.5-3B uses two KV heads whereas SmolLM3-3B uses four. Their theoretical FP16 KV-cache requirements are therefore 36 and 72 KiB/token, respectively. A larger Mistral-7B configuration is used separately to investigate memory-constrained behavior.

The study addresses five research questions: (RQ1) how KV caching affects end-to-end generation latency and throughput under fully resident conditions; (RQ2) how this benefit changes with input context length; (RQ3) how the model-specific GQA configurations and their corresponding theoretical KV footprints compare descriptively with observed memory–performance behavior, without treating KV-head count as an isolated causal factor; (RQ4) how GPU-memory residency affects measured and interpreted end-to-end generation speedup; and (RQ5) how GPU utilization, memory consumption, and power behavior differ between cache-enabled and cache-disabled inference.

The principal contribution is not the observation that KV caching accelerates autoregressive inference, which is well established. Rather, this work provides a controlled residency-aware systems characterization of the magnitude and interpretation of end-to-end generation speedup through systematic context-length variation, randomized paired measurements, descriptive cross-model comparison, accelerator telemetry, and a separate memory-constrained case study. The cross-model results are not used as a causal ablation of KV-head count.

The remainder of this paper is organized as follows. Section 2 develops the background needed to connect autoregressive generation, attention-state reuse, KV-cache storage, and GPU-memory residency. Section 3 positions the study relative to efficient-attention, KV-cache management, and LLM-serving research. Section 4 describes the experimental platform, models, workload controls, execution pipeline, telemetry, and statistical analysis. Section 5 reports the fully resident results, followed by the memory-constrained Mistral-7B case study in Section 6. Section 7 discusses the interpretation and systems implications of the measurements, Section 8 summarizes threats to validity and limitations, and Section 9 concludes the paper.

2. Background and Motivation

2.1. Autoregressive Transformer Inference

Let $x_i$ denote the token at sequence position i, let N denote the total sequence length, and let i index the current autoregressive position. Autoregressive generation factorizes the sequence probability as

$$P(x_1,\dots ,x_N)={\displaystyle \prod _{i=1}^N}P(x_i\mid x_1,\dots ,x_{i-1}),$$

(1)

where $P(·)$ denotes a probability distribution and $x_1,\dots ,x_{i-1}$ form the prefix available when predicting $x_i$. Equation (1) motivates KV caching because each newly generated token is conditioned on an increasingly long prefix; without reuse, intermediate states associated with already processed tokens may be reconstructed repeatedly.

Inference consists of prompt prefill followed by sequential decoding. If P denotes prompt length and G the number of generated tokens, the effective sequence lengths during successive decoding steps grow approximately as

$$P,P+1,P+2,\dots ,P+G-1.$$

(2)

Equation (2) makes the system’s consequence explicit: the reusable history grows at every decoding step. KV caching targets this repeated work by retaining the key and value states associated with that history.

For an input representation matrix X, self-attention forms

$$Q=X{W}_Q,K=X{W}_K,V=X{W}_V,$$

(3)

where $W_Q$, $W_K$, and $W_V$ are the learned query, key, and value projection matrices. Scaled dot-product attention is

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_h}}}\right)V,$$

(4)

where $d_h$ is the attention-head dimension and ⊤ denotes matrix transpose. During autoregressive decoding, the key and value representations associated with previously processed positions remain reusable; KV caching stores these tensors so that they need not be regenerated at every subsequent step.

2.2. Key–Value Caching

Let $K_{1:t-1}$ and $V_{1:t-1}$ denote cached states. At step t, only

$$K_t=x_t{W}_K,V_t=x_t{W}_V$$

(5)

must be produced and appended:

$$K_{1:t}=[K_{1:t-1};K_t],V_{1:t}=[V_{1:t-1};V_t].$$

(6)

For a decoder-only Transformer with L layers, batch size B, sequence length T, $N_{\mathrm{KV}}$ KV heads, head dimension $d_h$, and s bytes per stored scalar, theoretical KV-cache storage is

$$M_{\mathrm{KV}}=2LBT{N}_{\mathrm{KV}}{d}_{h}s,$$

(7)

where $M_{\mathrm{KV}}$ is the total cache capacity in bytes and the factor two accounts for keys and values. Thus,

$$M_{\mathrm{KV/token}}=2LB{N}_{\mathrm{KV}}{d}_{h}s,$$

(8)

and, for a fixed model and precision, $M_{\mathrm{KV}}\propto T$.

2.3. Attention Architecture and KV-Cache Footprint

For MHA, $N_{\mathrm{KV}}=N_Q$. MQA reduces the cache requirement to $N_{\mathrm{KV}}=1$ [1]. GQA lies between these cases, with $1\le N_{\mathrm{KV}}<N_Q$ [2]. Table 1 summarizes the architectures relevant to this study. For FP16, batch size one, Qwen2.5-3B requires

$$2\left(36\right)\left(1\right)\left(2\right)\left(128\right)\left(2\right)=36,864; \mathrm{bytes}=36; \mathrm{KiB/token},$$

(9)

whereas SmolLM3-3B requires

$$2\left(36\right)\left(1\right)\left(4\right)\left(128\right)\left(2\right)=73,728; \mathrm{bytes}=72; \mathrm{KiB/token}.$$

(10)

and 128 KiB/token for Mistral-7B.

Table 1. Architectural parameters relevant to theoretical FP16 KV-cache storage at batch size one.

Model	L	${\mathbf{N}}_{\mathbf{Q}}$	$\mathbf{N\_KV}$	${\mathbf{d}}_{\mathbf{h}}$	KV/Token
Qwen2.5-3B	36	16	2	128	36 KiB
SmolLM3-3B	36	16	4	128	72 KiB
Mistral-7B	32	32	8	128	128 KiB

At 2048 prompt tokens, the theoretical prompt KV–cache footprints are 72 MiB for Qwen2.5-3B and 144 MiB for SmolLM3-3B. After 128 generated tokens, they increase to 76.5 and 153 MiB, respectively.

The reduction in KV heads provided by GQA decreases not only KV-cache capacity requirements but also the memory-bandwidth demands associated with autoregressive decoding, making attention architecture an important systems-level consideration for LLM inference [11].

2.4. Computational Effect of KV Caching

Without cache reuse, successive decoding passes repeatedly process a growing prefix. Conceptually,

$$C_{\mathrm{nocache}}\propto {\displaystyle \sum _{g=0}^{G-1}}(P+g)=GP+{\displaystyle \frac{G(G-1)}{2}}.$$

(11)

With caching, historical KV states are retained, and the sequence state newly projected and propagated is conceptually closer to

$$C_{\mathrm{cache},\mathrm{new}}\propto P+G.$$

(12)

These expressions are not exact FLOP models; they motivate the expectation that cache benefit grows with context length.

2.5. GPU Memory Residency

Three memory regimes are useful for interpretation. In a fully resident regime, model parameters and runtime state fit comfortably within physical GPU memory. In a memory-pressure regime, the working set approaches physical capacity. In a memory-constrained or oversubscribed regime, logical memory requirements exceed accelerator capacity and execution may depend on host memory, paging, migration, offloading, or related mechanisms. Recent benchmarking of LLM inference on resource-constrained systems similarly demonstrates that insufficient memory can cause execution to become dominated by paging or storage I/O, while memory bandwidth remains a major determinant of autoregressive inference performance [12].

The commonly reported KV-cache speedup is

$$S_{\mathrm{KV}}={\displaystyle \frac{t_{\mathrm{cacheoff}}}{t_{\mathrm{cacheon}}}}.$$

(13)

When both conditions remain fully resident, this ratio is more readily interpreted in terms of computational reuse. Under severe memory pressure, however, the observed ratio may also include memory-system effects. This motivates treating physical accelerator-memory residency as an explicit experimental control.

3. Related Work

3.1. Attention Architectures and Efficient Attention

MQA reduces key–value storage and memory bandwidth by sharing one KV head across query heads [1]. GQA generalizes this design and provides an intermediate tradeoff between MHA and MQA [2]. FlashAttention approaches attention efficiency from the memory-hierarchy perspective by reducing data movement between high-bandwidth memory and on-chip SRAM [3]. These studies establish that attention performance depends on both arithmetic computation and memory movement.

3.2. KV-Cache Compression, Quantization, and Selection

H2O retains heavy-hitter tokens under a restricted cache budget [6]. Scissorhands exploits persistence in token importance to reduce cache storage [7]. KIVI applies tuning-free asymmetric 2-bit KV-cache quantization [8]. Quest uses query-aware sparsity to load only cache pages predicted to be relevant to the current query [9]. These techniques primarily seek to make KV caching more memory-efficient rather than characterize the baseline benefit of conventional cache reuse. Recent work has explored cross-modal, variance-aware KV-cache optimization to improve the efficiency of multimodal long-context inference [13].

3.3. Serving and Memory Management

Orca introduced iteration-level scheduling for generative model serving [14]. PagedAttention, implemented in vLLM, uses paging-inspired allocation to reduce KV-cache fragmentation and improve serving memory utilization [4]. FlexGen coordinates GPU, CPU, and storage resources for inference beyond GPU capacity [5]. InfiniGen reduces KV-cache transfer overhead in offloading-based inference through dynamic selection and prefetching [10].

Although entropy-guided KV-cache allocation improves memory efficiency by using prefill-stage attention distributions to prioritize salient tokens, its reliance on mean attention entropy as the primary measure of layer importance may not fully capture variations in cross-modal attention behavior [15].

Recent characterization studies further emphasize that cache performance is system dependent. Ye et al. examine KV caching under concurrent Transformer inference workloads [16], while Mamo et al. compare alternative KV-cache management strategies [17]. The present work complements these studies by focusing on controlled batch-one cache-enabled/cache-disabled inference while explicitly verifying physical GPU-memory residency. Table 2 provides a comparative positioning of the methodology adopted in this study relative to representative research on KV caching and LLM inference.

Table 2. Methodological positioning relative to representative KV-cache and LLM-inference research. Entries summarize the primary emphasis of each study rather than implying absence of every secondary experiment.

Study	Context Variation	ON/OFF	KV Optimization	Residency	Memory Constrained
MQA [1]	–	–	Yes	–	–
PagedAttention [4]	Yes	–	Yes	Yes	–
H2O [6]	Yes	–	Yes	–	–
KIVI [8]	Yes	–	Yes	–	–
InfiniGen [10]	Yes	–	Yes	Yes	Yes
Ye et al. [16]	Yes	–	–	Yes	Yes
Mamo et al. [17]	Yes	–	Yes	Yes	Yes
This work	Yes	Yes	–	Yes	Yes

4. Materials and Methods

4.1. Experimental Design

Three complementary stages were conducted. First, a context-length sweep evaluated

$$P\in \{128,256,512,1024,2048\}$$

(14)

with fixed generation length

$$G=128.$$

(15)

Second, the 2048-token condition was repeated using 30 randomized paired cache-enabled/cache-disabled measurements for each primary model. Third, Mistral-7B was evaluated separately under FP16 and INT8 configurations to examine memory-constrained behavior.

4.2. Experimental Pipeline and Reproducibility

Figure 1 provides the system-level view of the experimental workflow, whereas Algorithm 1 specifies the corresponding execution procedure. Together, they separate the experimental configuration from the sequence of operations used to collect the measurements. For every comparison, the model, prompt length, generated-token count, batch size, numerical precision, and generation settings are held fixed, while the KV-cache state is varied through use_cache. This controlled comparison is important because changes in workload configuration could otherwise be incorrectly attributed to the effect of KV caching.

Figure 1. Experimental measurement pipeline. Cache-enabled and cache-disabled executions use the same workload and model configuration; only the cache condition is changed within a comparison. The timing boundary surrounds the complete model.generate() call, while GPU telemetry provides complementary system-level observations.

The timing boundary is also kept identical across the cache-enabled and cache-disabled conditions. GPU synchronization is performed before the timer is started and again before it is stopped, and the measured interval encloses the complete model.generate() call. The reported runtime therefore represents end-to-end generation under the specified workload rather than an isolated kernel or individual inference phase. Warm-up executions are excluded from the measured samples, and a cooldown interval is applied between repeated comparisons. GPU telemetry is collected as complementary system-level evidence and is interpreted together with physical GPU-memory occupancy when assessing whether the workload remains resident in device memory.

The complete benchmarking and validation procedure is summarized in Algorithm 1. Phase A performs the controlled context-length sweep over prompt lengths from 128 to 2048 tokens, with three measured repetitions at each prompt length after warm-up. Within each comparison pair, the execution order of the cache-enabled and cache-disabled conditions is randomized to reduce systematic run-order effects. Phase B provides a stronger endpoint validation at $P=2048$ using 30 paired measurements and evaluates the resulting differences using both the paired t test and the Wilcoxon signed-rank test, together with run-order diagnostics. Phase C then places the performance measurements in their memory-residency context by comparing physical GPU-memory occupancy with available device capacity. The Mistral-7B experiment is treated separately as a memory-constrained case study rather than being pooled with the fully resident primary-model measurements.

Algorithm 1 Controlled KV-cache benchmarking and validation pipeline.

Input: model m; prompt lengths $\mathcal{P}=\{128,256,512,1024,2048\}$; generated length $G=128$; cache states $\mathcal{C}=\{\mathrm{OFF},\mathrm{ON}\}$. Output: end-to-end runtimes, throughput, paired speedups, runtime reductions, statistical-test results, run-order diagnostics, and GPU telemetry. Phase A: Context-length sweep    for each $P\in \mathcal{P}$ do       Construct a token-level prompt containing exactly P tokens.       Execute two warm-up runs for each cache state; exclude warm-ups from analysis.       for repetition $r=1,\dots ,3$ do          Randomize the order of $\mathcal{C}$ within the comparison pair.          for each cache state c in the randomized order do             Set use_cache according to c and retain all other workload settings.             Synchronize the GPU and start the end-to-end timer.             Execute model.generate() for exactly G new tokens.             Synchronize the GPU; stop the timer; record runtime and sampled GPU telemetry.          end for          Apply the prescribed 60-s cooldown.       end for    end for Phase B: Repeated endpoint validation at $P=2048$    Execute three warm-up runs before measured validation.    for pair $i=1,\dots ,30$ do       Randomize cache OFF/ON execution order within pair i.       Execute both cache states using the same synchronized timing boundary and telemetry procedure.       Store $t_{\mathrm{off},i}$, $t_{\mathrm{on},i}$, and $S_i=t_{\mathrm{off},i}/t_{\mathrm{on},i}$.    end for    Compute mean/median paired speedup, runtime reduction, paired t test, Wilcoxon signed-rank test, and run-order drift diagnostics. Phase C: Memory-regime interpretation    Compare physical GPU-memory occupancy with physical capacity to establish residency for the primary models.    Evaluate Mistral-7B separately as the memory-constrained case study; do not pool its FP16/INT8 ratios with the fully resident primary-model estimates.

4.3. Hardware and Software Environment

All experiments were conducted on the same NVIDIA GeForce RTX 4080 Laptop GPU with 12,282 MiB (11.994 GiB) physical GPU memory. The hardware and software specifications are provided in Table 3. The experiments ran under Linux in WSL2. The principal software stack was Python 3.14.4, PyTorch 2.14.0+cu132, Transformers 5.16.1, CUDA 13.2, and NVIDIA driver 610.47.

Table 3. Experimental hardware and software environment.

Component	Configuration
GPU	NVIDIA GeForce RTX 4080 Laptop GPU
Physical GPU memory	12,282 MiB (11.994 GiB)
Operating environment	Linux under WSL2
Linux kernel	6.6.87.2-microsoft-standard-WSL2
Python	3.14.4
PyTorch	2.14.0+cu132
Transformers	5.16.1
CUDA runtime	13.2
NVIDIA driver	610.47
Primary precision	FP16
Batch size	1

4.4. Evaluated Models

As indicated in Table 4, Qwen2.5-3B and SmolLM3-3B constitute the primary fully resident models. Mistral-7B was selected deliberately as the memory-constrained case study because its runtime-reported FP16 model footprint (13.489 GiB) exceeds the 11.994 GiB physical capacity of the experimental RTX 4080 Laptop GPU. It therefore provides a contrasting execution regime on the same hardware rather than a third fully resident replication model. The INT8 configuration is included only to show how changing the memory regime alters the observed end-to-end ratio; because quantization also changes numerical representation and kernel behavior, it is not a precision-controlled ablation.

Table 4. Models and architectural parameters relevant to the experiments.

Model	Layers	Hidden	Q Heads	KV Heads	Head Dim.	FP16 Footprint
Qwen2.5-3B	36	2048	16	2	128	5.748 GiB
SmolLM3-3B	36	2048	16	4	128	5.728 GiB
Mistral-7B	32	–	32	8	128	13.489 GiB

4.5. Cache Conditions and Workload Control

Cache ON used use_cache=True; cache OFF used use_cache=False. The default Hugging Face Transformers cache implementation was retained. No custom cache compression, quantization, eviction, sparse selection, or paged-cache mechanism was introduced in the primary experiments.

Prompts were constructed at token level to produce the target lengths. Generation was constrained to exactly 128 new tokens in every primary measured run. The timer surrounded the complete model.generate() operation; reported latency therefore includes both prefill and autoregressive decoding and should not be interpreted as decode-only latency. The original instrumentation did not place separate timing boundaries around time to first token (TTFT), prefill, or per-token decoding. Consequently, TTFT, isolated prefill time, decode time per token, and decode-only throughput cannot be reconstructed reliably from the recorded aggregate generation times and are not reported. The revised analysis therefore consistently interprets the measured quantity as end-to-end generation latency, throughput, or speedup rather than attributing the full ratio specifically to decoding.

4.6. Context Sweep and Repeated Validation

For each of five context lengths, both cache conditions were measured three times after two warm-up runs. A 60-s cooldown was used, and within-pair condition order was randomized. Because this sweep contains only three measured repetitions per condition, it is treated as a descriptive scaling experiment rather than as a basis for narrow inferential confidence intervals. The endpoint at 2048 tokens is therefore supported separately by the larger 30-pair randomized validation described below.

At 2048 prompt tokens, each primary model was subsequently evaluated using 30 paired observations (60 measured runs). Three warm-up runs preceded measured execution, and cache-condition ordering was randomized within each pair to reduce systematic order effects. The available experimental record specifies randomized within-pair ordering but does not provide a reproducible randomization seed; this is therefore reported as a reproducibility limitation rather than reconstructed retrospectively.

For pair i,

$$S_i={\displaystyle \frac{t_{\mathrm{off},i}}{t_{\mathrm{on},i}}},$$

(16)

with mean paired speedup

$$\overline{S}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{S}_i.$$

(17)

Generation throughput is

$$R={\displaystyle \frac{G}{t}}.$$

(18)

4.7. KV-Cache Measurement and GPU Telemetry

Theoretical cache storage was calculated using Equation (7). Instrumentation was validated using explicit token-by-token Mistral-7B decoding. The measured cache increased from 4.875 MiB at 39 cached tokens to 20.875 MiB at 167 tokens, exactly matching the theoretical 128 KiB/token growth.

Physical GPU telemetry was sampled at approximately 1-s intervals and included physical GPU memory usage, GPU utilization, memory utilization, temperature, power draw, SM clock, memory clock, and performance state where available. Physical memory utilization was computed as

$$U_{\mathrm{VRAM}}={\displaystyle \frac{M_{\mathrm{used}}}{M_{\mathrm{physical}}}}\times 100\%.$$

(19)

4.8. Statistical Analysis

The 30-pair experiments were analyzed using paired-samples t tests and Wilcoxon signed-rank tests at $\alpha =0.05$. Spearman rank correlation between run order and runtime was used to examine potential experimental drift. Engineering effect sizes—runtime, throughput, absolute time saved, runtime reduction, and paired speedup—were emphasized because very low timing variance can produce extremely large standardized effects.

5. Results

5.1. Effect of Context Length

Both primary models exhibit a strong increase in KV-cache benefit with context length. Figure 2 shows that Qwen2.5-3B speedup increases from 1.337× at 128 prompt tokens to 8.314× at 2048 tokens. SmolLM3-3B in Figure 2 increases from 1.278× to 8.619× over the same range. The exact speedup for each experiment is given in Table 5.

Figure 2. KV-cache speedup as a function of input context length for Qwen2.5-3B and SmolLM3-3B. Each point is based on mean cache-disabled and cache-enabled end-to-end generation time for a fixed 128-token continuation.

Table 5. Effect of context length on end-to-end generation performance. Each request generates exactly 128 new tokens.

Model	Prompt	OFF (s)	ON (s)	Speedup	Reduction
Qwen2.5-3B	128	4.891	3.655	1.337×	25.23
	256	6.053	3.589	1.687×	40.72
	512	9.229	3.541	2.607×	61.64
	1024	16.176	3.723	4.343×	76.98
	2048	31.302	3.764	8.314×	87.97
SmolLM3-3B	128	4.492	3.517	1.278×	21.71
	256	5.846	3.383	1.728×	42.14
	512	9.255	3.496	2.648×	62.23
	1024	15.833	3.403	4.653×	78.51
	2048	30.704	3.563	8.619×	88.40

Cache-disabled runtime rises from approximately 4–5 s at 128 prompt tokens to more than 30 s at 2048 tokens. Cache-enabled runtime remains in a much narrower range of approximately 3.4–3.8 s. This does not imply constant-time cached attention; each new query still attends over accumulated KV state. Rather, over the tested range, that growing cost is substantially smaller than the repeated historical computation avoided by caching.

5.2. Repeated Validation at 2048 Tokens

The inferential analysis is reported separately from the descriptive context-length sweep. Table 6 summarizes the 30 randomized paired trials at the 2048-token condition and reports the complete paired t-test and Wilcoxon signed-rank p-values for both primary models.

Table 6. Statistical validation for the 2048-token prompt condition using 30 randomized paired trials per model.

Model	OFF Mean	ON Mean	Paired Speedup	Reduction	Paired t-Test p	Wilcoxon p
Qwen2.5-3B	30.471 s	3.762 s	8.101×	87.65%	$1.786\times {10}^{-73}$	$9.31\times {10}^{-10}$
SmolLM3-3B	30.703 s	3.609 s	8.511×	88.25%	$2.808\times {10}^{-79}$	$9.31\times {10}^{-10}$

For Qwen2.5-3B, the mean paired speedup was 8.101×, and the median paired speedup was 8.084×, saving approximately 26.71 s per request. For SmolLM3-3B, the mean paired speedup was 8.511×, and the median was 8.533×, saving approximately 27.09 s. All 30 measured pairs for both models favored cache-enabled inference.

As illustrated in Figure 3, timing variability was low: Qwen standard deviations were approximately 0.061 s (OFF) and 0.051 s (ON); Smol values were approximately 0.055 s and 0.064 s.

Figure 3. Mean end-to-end generation runtime across input context lengths with KV caching enabled and disabled. Cache-disabled runtime grows strongly with context length, whereas cache-enabled runtime varies comparatively little over the tested range.

Statistical significance was assessed using paired t-tests, which yielded $p=1.786\times {10}^{-73}$ for Qwen2.5-3B and $p=2.808\times {10}^{-79}$ for SmolLM3-3B. Wilcoxon signed-rank tests yielded $p=9.31\times {10}^{-10}$ for both models, as fully reported in Table 6. These values were re-verified against the recorded statistical results during revision. No meaningful run-order drift was observed. The consistency between the three-run sweep and 30-pair validation indicates that the large endpoint result is not driven by isolated measurements.

5.3. KV-Cache Footprint and Cross-Model Comparison

Table 7 shows that despite a twofold difference in theoretical KV-cache storage, the models exhibit similar context-dependent scaling while both remain comfortably within physical GPU memory. This result does not isolate KV-head count as a causal variable because the models differ in other architectural and implementation details. The experiment also does not separately identify the contributions of memory bandwidth, cache-allocation overhead, kernel implementation, or GPU scheduling. These factors are therefore treated as possible components of the observed system-level behavior rather than as independently measured causes.

Table 7. KV-cache architecture, theoretical storage, and observed 2048-token performance.

Model	KV Heads	KV/Token	KV2048	KV2176	Speedup	Max VRAM
Qwen2.5-3B	2	36 KiB	72 MiB	76.5 MiB	8.10×	60.53
SmolLM3-3B	4	72 KiB	144 MiB	153 MiB	8.51×	60.73

5.4. GPU Resource Behavior

The GPU telemetry supporting the residency assessment is reported in Table 8. The table provides a complete entry for each model–condition combination for the telemetry metrics summarized in the repeated 2048-token experiment: mean GPU utilization, sampled power, and temperature. Maximum physical GPU-memory occupancy used for the residency classification is reported separately in Table 7.

Table 8. Mean GPU telemetry during the 2048-token repeated experiments.

Model	Condition	GPU Utilization	Power	Temperature
Qwen2.5-3B	Cache OFF	95.6%	158.5 W	64.7 °C
Qwen2.5-3B	Cache ON	57.5%	66.7 W	46.4 °C
SmolLM3-3B	Cache OFF	95.6%	159.7 W	68.0 °C
SmolLM3-3B	Cache ON	61.2%	69.9 W	49.1 °C

SmolLM3-3B requires twice the theoretical KV storage per token as Qwen2.5-3B, yet both exhibit closely similar context-dependent scaling while remaining well within physical GPU memory. At 2048 prompt tokens, theoretical cache sizes are 72 and 144 MiB; after generation they are 76.5 and 153 MiB. This comparison does not isolate KV-head count causally because the models differ in other architectural and implementation details. It does show that a twofold cache-footprint difference did not prevent similar scaling in the tested resident regime.

The telemetry values in Table 8 were re-verified against the recorded 2048-token repeated-experiment summaries; all four model–condition rows report GPU utilization, sampled power, and temperature. GPU telemetry reinforces the system’s interpretation. With caching disabled, mean GPU utilization was approximately 95.6% for both models, with sampled power near 159 W. With caching enabled, utilization fell to 57.5% for Qwen and 61.2% for Smol, while sampled power fell to 66.7 and 69.9 W. The lower utilization should not be interpreted as inferior performance: the same fixed-generation request completes about eight times faster, so redundant work occupies the accelerator for far less time. Most importantly, maximum physical GPU-memory occupancy remained below 61% for both models, supporting their classification as fully resident.

6. Memory-Constrained Inference Case Study

6.1. Mistral-7B FP16

The experimental GPU provides 11.994 GiB physical memory, whereas the Mistral-7B FP16 runtime-reported model footprint was approximately 13.489 GiB. After model loading, approximately 11,927 MiB of 12,282 MiB was occupied, or about 97.1%. This configuration is therefore memory constrained.

The FP16 experiment used a 25-token prompt, generated exactly 128 tokens, and collected 30 measured runs per cache condition after warm-up. The memory-constraint experiment is illustrated in Table 9.

Table 9. Mistral-7B FP16 performance under the memory-constrained regime.

Metric	Value
Physical GPU memory	11.994 GiB
FP16 model footprint	13.489 GiB
GPU memory after load	11,927 MiB
Prompt length	25 tokens
Generated tokens	128
Runs per condition	30
Cache OFF mean runtime	142.214 s
Cache ON mean runtime	60.570 s
Mean paired speedup	2.343×
Runtime reduction	57.41
Throughput improvement	133.1

The measured 2.343× ratio is an end-to-end generation result under memory pressure, not a clean estimate of the arithmetic benefit of KV reuse. The experiment establishes that the configuration is memory constrained, but the collected telemetry does not identify whether paging, host–device migration, explicit or implicit offloading, allocator behavior, or another memory-system mechanism dominates the additional time. No mechanism-specific causal claim is therefore made.

6.2. Mistral-7B INT8

Using bitsandbytes LLM.int8 weight quantization reduced the runtime-reported model footprint to approximately 6.989 GiB. After loading, physical GPU-memory usage was approximately 8355 MiB. Table 10 provides the output of this experiment.

Table 10. Mistral-7B INT8 performance. The configuration is not a precision-controlled comparison with FP16.

Metric	Value
INT8 model footprint	6.989 GiB
GPU memory after load	8355 MiB
Cache OFF mean runtime	273.35 s
Cache ON mean runtime	19.64 s
Cache OFF throughput	0.484 token/s
Cache ON throughput	6.52 token/s
Apparent speedup	13.92×
Cache ON peak reserved	∼7.48 GiB
Cache OFF peak reserved	∼21.23 GiB

The 13.92× ratio should be treated only as an apparent end-to-end generation speedup under a substantially different memory regime. Weight quantization changes model footprint, numerical representation, kernel selection, and memory behavior simultaneously; the INT8 result is therefore not a precision-controlled comparison with the FP16 case and is not used to attribute the ratio to a specific memory mechanism.

As indicated in Table 11 these cases demonstrate that observed speedup

$$S_{\mathrm{observed}}={\displaystyle \frac{t_{\mathrm{off}}}{t_{\mathrm{on}}}}$$

(20)

measures the ratio of complete system execution times. When memory behavior differs materially between conditions, the ratio cannot be attributed solely to avoided computation.

Table 11. Comparison of principal memory regimes. Speedups should be interpreted within, rather than directly across, these regimes.

Configuration	Model Footprint	Physical VRAM	Memory Indicator	Speedup
Qwen2.5-3B FP16	5.748 GiB	11.994 GiB	Max 60.53
SmolLM3-3B FP16	5.728 GiB	11.994 GiB	Max 60.73
Mistral-7B FP16	13.489 GiB	11.994 GiB	97.1
Mistral-7B INT8	6.989 GiB	11.994 GiB	8355 MiB after load	13.92×

7. Discussion

7.1. Context-Dependent Computational Reuse

The clearest result is the strong relationship between input context length and measured cache benefit. Cache-disabled runtime rises rapidly as the growing sequence prefix is repeatedly processed, while cache-enabled decoding retains historical KV states and computes new states only for newly generated positions. The experiments do not demonstrate constant-time cached decoding; attention still accesses an accumulated KV state whose size grows with context.

7.2. Attention Architecture and Cache Footprint

SmolLM3-3B requires twice the theoretical KV-cache storage per token as Qwen2.5-3B, yet both exhibit similar context-dependent end-to-end generation speedup over the tested range. This is a descriptive cross-model observation, not a GQA ablation: the models differ in architectural and implementation characteristics beyond KV-head count. The result therefore supports only the narrower conclusion that a twofold difference in theoretical KV-cache footprint coexisted with similar scaling while both models remained comfortably resident. Isolating the causal effect of KV-head count would require a same-model intervention or controlled architectural ablation.

7.3. Physical Residency as an Experimental Control

Physical accelerator-memory residency is a central methodological variable. Framework allocator statistics alone are insufficient to establish residency because allocated or reserved quantities do not necessarily equal physical accelerator occupancy. System-level telemetry provides complementary evidence that the primary comparisons are not being driven by GPU-memory exhaustion.

7.4. Memory Pressure and Apparent Speedup

Mistral-7B illustrates how memory pressure can either suppress or amplify apparent cache benefit. A useful conceptual decomposition is

$$t=t_{\mathrm{compute}}+t_{\mathrm{attention}}+t_{\mathrm{memory}}+t_{\mathrm{runtime}}.$$

(21)

When both cache conditions remain resident, the memory component is comparatively controlled. When one or both approach physical capacity, differences in memory-system behavior can become large enough to materially change the observed speedup.

Recent work on pipeline-parallel LLM inference similarly shows that increasing KV-cache pressure can trigger cache eviction, migration, or offloading, introducing additional recomputation and data-transfer overheads that affect end-to-end inference performance [18,19].

7.5. GPU Utilization and Power

Cache-disabled execution produced approximately 95.6% mean GPU utilization and about 159 W sampled power for both primary models, whereas cache-enabled execution operated at approximately 57–61% utilization and 67–70 W while completing the workload roughly eight times faster. Higher utilization is therefore not inherently preferable when it reflects redundant computation. The power samples are supporting GPU telemetry and should not be interpreted as dedicated whole-system energy measurements.

7.6. Implications for Inference-System Design and Benchmarking

The measured benefit of KV caching grows with prompt length for the fixed 128-token continuation used in this study, making effective reuse particularly important for the tested long-context conditions. At the same time, the cache grows linearly with sequence length, batch size, number of layers, KV heads, head dimension, and representation size. Because G is fixed at 128 in the primary experiments, the present data do not establish how end-to-end speedup changes as output length increases. Longer continuations increase both sequential decoding duration and accumulated KV state and therefore require a separate controlled output-length sweep. The appropriate systems strategy consequently depends on workload shape as well as the memory regime.

KV-cache benchmarks should report model architecture and precision, query and KV-head counts, batch size, prompt length, generated-token length, cache implementation, model footprint, physical accelerator-memory capacity, peak physical memory occupancy, offloading or oversubscription behavior, and the exact timing boundary. Without these details, a reported speedup can be difficult to interpret or reproduce.

8. Threats to Validity and Limitations

8.1. Internal Validity

Cache-enabled and cache-disabled runs used the same model, prompt length, generation length, numerical precision, batch size, software stack, and hardware. Condition order was randomized, warm-up runs were excluded, cooldown intervals were used, and generation length was fixed at exactly 128 tokens. Residual variability from GPU frequency scaling, thermal state, power variation, operating-system activity, CUDA behavior, allocation state, and WSL2 scheduling cannot be eliminated completely. GPU clocks were not fixed during the experiments; the recorded telemetry is therefore used as supporting context rather than as evidence of clock-controlled execution.

The Qwen2.5-3B/SmolLM3-3B comparison does not isolate KV-head count as an independent causal variable because the models differ in other architectural and implementation characteristics. Nor does the design independently separate attention architecture from memory bandwidth, cache-allocation overhead, kernel implementation, or GPU scheduling. The study therefore reports associations at the complete-system level and avoids assigning the observed scaling to any one of these mechanisms.

8.2. External Validity

Experiments were performed on one RTX 4080 Laptop GPU. Data-center accelerators may provide different memory bandwidth, compute throughput, cache hierarchy, capacity, kernel implementations, and power behavior. The evaluated model set is limited, and all primary experiments use batch size one. Production serving systems may employ continuous batching, request scheduling, prefix sharing, speculative decoding, or other optimizations.

8.3. Context and Timing Scope

The fully resident sweep covers prompt lengths from 128 to 2048 tokens and fixed generation of 128 tokens. Results should not be directly extrapolated to much longer contexts or longer generated outputs without measurement. Because generation length is held constant, the experiment characterizes prompt-length scaling at $G=128$ rather than the joint effect of prompt and output length. The timer surrounds the complete generation call and therefore includes both prefill and decoding; the reported speedups are end-to-end generation speedups rather than pure decode-kernel acceleration. Because the original experiment did not record separate TTFT, prefill, or decode timing boundaries, those components cannot be recovered faithfully after the fact. A future phase-separated benchmark is required to quantify precisely how much of the observed end-to-end gain arises during decoding.

8.4. Software and Measurement Limitations

The results depend on PyTorch, Hugging Face Transformers, CUDA, the NVIDIA driver, the default cache implementation, and WSL2. Alternative runtimes and optimized serving systems may yield different absolute performance.

GPU telemetry was sampled at approximately one-second intervals. This cadence may miss short-lived transient peaks, frequency changes, or thermal excursions, and instrumentation itself introduces small overhead. The experiment used randomized within-pair cache-condition order, but the available record does not provide a reproducible randomization seed; exact replay of the historical order is therefore not claimed. The same procedure was used for both cache conditions, reducing its impact on comparative conclusions, but the telemetry is not equivalent to fixed-clock execution. The three-run context sweep also provides limited information about the distribution at intermediate context lengths; the separate 30-pair experiment supplies the stronger repeated evidence only for the 2048-token endpoint.

Theoretical KV-cache storage represents key and value tensors only; total runtime memory additionally includes model parameters, activations, temporary tensors, CUDA contexts, workspaces, fragmentation, and allocator-reserved regions. The Mistral case establishes a memory-constrained regime but does not identify the precise paging, migration, host–device transfer, offloading, or allocator mechanism responsible for the observed behavior. Direct instrumentation of those mechanisms is required before assigning the observed ratios to a specific cause.

9. Conclusions

This study presented a controlled systems characterization of KV-cache performance in autoregressive LLM inference with emphasis on context length, attention architecture, and physical GPU-memory residency. Under fully resident conditions, Qwen2.5-3B speedup increased from approximately 1.34× at a 128-token prompt to 8.31× at 2048 tokens, while SmolLM3-3B increased from approximately 1.28× to 8.62×. Thirty randomized paired trials at 2048 tokens confirmed mean paired speedups of 8.10× and 8.51× and runtime reductions of 87.65% and 88.25%, respectively. Physical GPU-memory occupancy remained below 61%.

Comparable context-dependent scaling occurred despite SmolLM3-3B requiring twice the theoretical KV-cache storage per token. This does not establish a causal relationship between KV-head count and performance, but demonstrates that cache footprint should be interpreted together with available physical memory.

The Mistral-7B case study shows why residency matters. Its FP16 footprint exceeds the physical GPU capacity and produced a 2.34× speedup under memory pressure. An INT8 configuration produced an apparent 13.92× speedup under a substantially different memory regime. Thus, memory-system effects can suppress or amplify the observed wall-clock ratio.

KV-cache speedup should therefore not be treated as an intrinsic scalar property of an LLM. Meaningful benchmarking requires reporting workload configuration, cache architecture, model and cache footprints, physical accelerator capacity and occupancy, and the execution regime alongside runtime and throughput.

Future work should extend the characterization to longer contexts, additional GPU architectures and model families, concurrent workloads, alternative cache implementations, and explicit profiling of host–device transfers and memory migration.