Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This manuscript characterizes KV-cache performance in autoregressive LLM inference under fully GPU-resident and memory-constrained regimes. It varies prompt length, compares cache-on/off execution, examines GQA-related cache footprints, and uses randomized paired trials plus GPU telemetry for Qwen2.5-3B and SmolLM3-3B, with Mistral-7B as a memory-pressure case study. The main contribution is a residency-aware benchmarking argument: KV-cache speedup is workload- and system-dependent and should not be reported as a single intrinsic value. My comments is listed as follows:

1. RQ3 asks how different GQA configurations relate to the memory–performance tradeoff, but the Qwen2.5-3B vs. SmolLM3-3B comparison cannot support a causal claim because the models differ in multiple architectural and implementation details beyond KV-head count. The paper acknowledges this limitation, but the abstract and conclusions should avoid implying a GQA-based explanation; a same-model KV-head ablation or consistently non-causal framing is needed.

2. The reported speedup is measured over the full model.generate() call, so it includes prefill and decoding, whereas KV caching mainly benefits decoding. The term “KV-cache speedup” should be replaced or consistently qualified as end-to-end generation speedup, and the paper should report TTFT, prefill time, decode time per token, and decode throughput to show where the gain actually comes from.

3. The Mistral-7B case study shows that memory pressure changes apparent speedup, but it does not identify the mechanism, such as paging, offloading, host–device transfer, or allocator behavior. Since FP16 exceeds physical GPU memory and INT8 is not precision-controlled, the 2.34× and 13.92× ratios should be interpreted cautiously unless memory-system behavior is directly instrumented.

4. The 2048-token paired validation is strong, but the context-length sweep uses only three runs per condition and lacks confidence intervals, paired-difference distributions, and run-order drift diagnostics. GPU frequency scaling, thermal state, power variation, and WSL2 scheduling could also affect the very low timing variance, so fixed-clock runs or additional telemetry would strengthen the scaling claims.

 

Comments on the Quality of English Language

 

 

Author Response

Thank you very much for taking the time to review this manuscript. We appreciate the detailed and constructive comments. The manuscript has been revised to strengthen the non-causal interpretation of the cross-model GQA comparison, qualify the performance ratios consistently as end-to-end generation measurements, limit the interpretation of the Mistral-7B memory-pressure experiments to what the instrumentation supports, and clarify the statistical and telemetry limitations of the three-run context sweep. No unmeasured TTFT, prefill, decode-only, paging, transfer, or allocator results have been introduced.
Please see the attached file for more information about the respected reviewer comments.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Authors,

The manuscript addresses a useful systems question, but the current version does not yet build a complete chain from claims to evidence. The introduction should end with a short paragraph outlining the remaining sections. The background section reads as isolated pieces rather than a logical argument, and Equations 1 and 2 are not clearly tied to the motivation in Section 2.1. In addition, T is used for both a time step and a matrix transpose, which creates confusion. The methodology needs a system diagram and pseudocode for the full pipeline, with all symbols in Equation 1 defined. Without these, independent reproduction is difficult.

A deeper issue is how the reported speedups are explained. The paper states that KV-cache speedup is workload- and system-dependent, but it does not separate the effect of attention architecture from memory bandwidth, cache allocation overhead, or GPU scheduling. Both models show similar scaling even though SmolLM3-3B needs twice the theoretical cache storage per token; the paper does not explain why the physical GPU memory utilization stays below 61% or what limits the cache benefit. The fixed 128-token generation also leaves open whether the speedup changes for longer outputs. Section 5.1 reports mean speedups from 30 paired trials, but there is no confidence interval, significance test, or discussion of random seeds and run order. The Mistral-7B case study is used to show memory pressure effects, yet the selection of that model and the memory conditions are not fully described. A major revision should address these gaps in reproducibility, statistical support, and causal explanation.

Sincerely,

Reviewer

Author Response

Thank you very much for the detailed and constructive review. We revised the manuscript to strengthen the chain from claims to evidence and improve reproducibility. The revision adds a paper-organization paragraph, restructures the background motivation and notation, defines mathematical symbols explicitly, adds an experimental system diagram and full-pipeline pseudocode, clarifies the limits of causal interpretation, scopes the results to the fixed 128-token output length, makes the existing statistical tests and run-order controls more explicit, and expands the rationale and memory conditions for the Mistral-7B case study. Where evidence was not retained in the completed experimental record—such as the full paired-speedup series required to reconstruct a confidence interval or a reproducible randomization seed—we state the limitation rather than fabricate retrospective measurements.

Please see the attached file for point to point clarification.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Authors

The manuscript needs a stronger and more complete evidence chain. Tables 6 and 8 contain missing values at key trial positions, including paired t and Wilcoxon p-values and several GPU telemetry entries; these must be re-verified and fully reported. Missing statistical values weaken the validation of the 2048-token condition, and incomplete telemetry makes the memory-residency claims harder to check. The pseudocode should be rewritten in a more professional style to improve clarity and reproducibility. A clearer pseudocode structure would help readers follow the experimental procedure and would better support the reported results. Addressing these points would make the performance characterization more credible and easier to reproduce.

Author Response

Dear Reviewers,

Thank you very much for your careful evaluation of our manuscript, “Characterizing KV-Cache Performance in Large Language Model Inference: Context Length, Attention Architecture, and GPU Memory Residency,” and for your constructive comments and suggestions.

We have carefully considered all comments and have revised the manuscript accordingly. The reviewers’ feedback has helped us improve the clarity, methodological description, reproducibility, statistical reporting, presentation of the experimental results, and discussion of the study’s scope and limitations.

Please find attached our detailed point-by-point responses to the reviewer comments. Each comment has been addressed individually, and the corresponding revisions made to the manuscript are described in the response documents.

We sincerely appreciate the time and effort devoted to reviewing our work and hope that the revisions satisfactorily address the concerns raised.

Kind regards,

Prof. Abed Al Raoof K. Bsoul
Corresponding Author

Author Response File: Author Response.pdf