Large Language Models in Peer Review: Decision Alignment, Review-Text Characteristics, and Human–AI Aggregation at ICLR 2025

Abstract

Large language models (LLMs) are increasingly employed in scholarly peer review, yet their suitability as autonomous evaluators remains uncertain. Using the ICLR 2025 review process, this study compares 2401 human reviews with 7203 reviews produced in separate, context-isolated API runs using Claude Sonnet 4.5, GPT-5.2 Thinking, and Gemini 3 Pro Preview across decision agreement, review-text characteristics, inter-model consistency, and human–AI aggregation. Raw LLM scores showed systematic leniency and score compression. A 0.1-point grid search identified thresholds of 6.2, 6.3, and 6.7 for Claude, GPT, and Gemini, respectively; repeated stratified cross-validation reproduced these thresholds. When applied without retuning to a stratified balanced sample of 300 ICLR 2024 papers, decision-agreement accuracy was 0.927, 0.913, and 0.930. Independent human coding of research type and primary field showed substantial pre-adjudication agreement (Cohen’s kappa = 0.774 and 0.714), and the recalculated analyses did not support H3. Review-text indicators showed similar structural completeness across sources but uneven critical-section length; these descriptive measures do not establish review quality. Human-containing aggregation rules showed higher agreement with conference decisions than corresponding AI-only rules, without establishing independent review quality or causal complementarity. A textual-overlap check found very low exact eight-gram containment, and manual inspection of the highest-similarity 1% found shared manuscript content or domain terminology rather than reviewer-specific evaluative language; possible prior exposure nevertheless could not be excluded.

1. Introduction

Peer review has long served as a cornerstone of quality control in scholarly publishing, playing a central role in filtering innovative research, upholding scientific norms, and maintaining academic credibility. However, the continued surge in global research output has placed this system under increasing structural strain, manifested in the slow growth of the available reviewer pool, rising workloads for individual reviewers, and longer review turnaround times. A report by the National Center for Science and Engineering Statistics (NCSES) of the U.S. National Science Foundation (NSF) indicates that the global output of science and engineering publications indexed in Scopus reached 3.3 million in 2022 (National Center for Science and Engineering Statistics, 2023). According to the latest DOI registration records available from Crossref, the global volume of journal article records had reached approximately six million by 2024 (Crossref, 2026). Although Crossref and bibliographic databases differ in coverage, the Crossref data similarly point to the rapid expansion of scholarly output (Crossref, 2025).

This growth has been particularly pronounced in artificial intelligence. The International Conference on Learning Representations (ICLR), which is the focus of this study, provides a representative example: the number of valid submissions increased from 2594 in 2020 to 18,949 in 2026 (OpenAccept, 2026), an increase of more than 630%. Reviewer resources, however, have not expanded at a comparable pace, leaving the scholarly publishing system under mounting strain. As Nature has reported, peer review is increasingly operating under overload, as securing reviewers has become more difficult and editors often need to issue multiple rounds of invitations to recruit a sufficient number of referees (Adam, 2025). Editorial data from Biological Invasions offer further evidence of this trend: records from 2003 to 2024 show a sustained decline in reviewer acceptance rates and a corresponding increase in refusal rates (Meyerson et al., 2025). Consistent with this journal-level evidence, IOP Publishing’s global survey found that 50% of respondents had experienced an increase in review requests over the preceding three years, while lack of availability was the most commonly reported reason for declining an invitation (IOP Publishing, 2024). Taken together, the widening gap between submission growth and reviewer availability has become a critical bottleneck affecting both the efficiency of scholarly communication and the quality of peer review.

Against this backdrop, the use of large language models (LLMs) in peer review has been proposed as a potential means of alleviating reviewer burden. The release of GPT-4 in 2023 marked a new stage in the general capabilities of LLMs, and its performance in text understanding, generation, and evaluation has prompted extensive discussion regarding the feasibility of AI-assisted reviewing. LLMs have already begun to enter real-world review workflows. Liang et al. (2024a), for example, developed an AI-content detection method based on maximum likelihood estimation and applied it to review corpora from ICLR 2024, NeurIPS 2023, CoRL 2023, and EMNLP 2023. They estimate that 6.5% to 16.9% of review texts underwent substantive AI-assisted revision beyond mere spelling correction, with higher rates of use observed among reviews submitted close to the deadline, reviews written by low-confidence reviewers, and reviews from reviewers who did not participate in author discussion. Latona et al. (2024) further quantified the causal effects of AI-assisted reviewing. Using GPTZero (https://gptzero.me/), they detected AI involvement in at least 15.8% of ICLR 2024 reviews and found that, among borderline submissions, papers receiving AI-assisted reviews were approximately 4.9 percentage points more likely to be accepted, providing evidence that AI-assisted reviewing may introduce distortions in fairness.

Existing evidence suggests that LLMs may hold some potential for improving the clarity and actionability of review reports. However, current applications have largely entered the review process through spontaneous and decentralized use by individual reviewers, without standardized procedural integration, quality control, or mechanisms for constraining bias. As a result, the reliability and fairness of such practices remain difficult to ensure. To address this gap, the present study draws on papers and review data from ICLR 2025 to simulate review scenarios and constructs a parallel comparative dataset comprising human reviews and reviews generated by three mainstream LLMs. Through empirical analysis, calibration strategies for LLM-based reviewing, disclosure mechanisms, and human-containing aggregation rules are examined, with the aim of providing evidence to inform future policy design.

2. Literature Review

Existing research on whether large language models (LLMs) possess effective peer-review capabilities has reported a range of empirical findings, from cautious skepticism to relatively optimistic assessments. It is worth noting that the consistency of human reviewers is itself far from ideal. Consistency studies by Cortes and Lawrence (2021) and Beygelzimer et al. (2023), based on NeurIPS 2014 and 2021, show that even within the same conference, substantial discrepancies remain in the scores assigned by different reviewers to the same paper. This observation provides an important human baseline for interpreting the review performance of LLMs.

In the early exploratory stage, studies were generally conservative due to limited sample sizes. R. Liu and Shah (2023), in their evaluation of ReviewerGPT, found that GPT-4 was useful for identifying basic flaws, but exhibited clear limitations in deeper methodological critique and domain-specific judgment, making it unsuitable for assuming full reviewing responsibilities. As research has scaled up, some studies have begun to report more encouraging results. Liang et al. (2024b) found that the overlap between GPT-4-generated review feedback and the concerns raised by human experts reached 30–40%, approaching the average level of agreement between two human reviewers. However, that study also noted that LLMs tended to focus on specific improvement suggestions, such as dataset expansion, while remaining relatively weak in critically evaluating the underlying methodological design. Earlier work by members of the present author team on PeerJ reviews identified complementary orientations: human reviewers provided more integrative and socially contextualized judgments, whereas GPT-4o emphasized structural verification and internal consistency (Yang et al., 2026). Building on that foundation, the present study advances this line of research by extending the analysis from a single-model journal setting to a three-model conference setting and by adding explicit tests of decision alignment, model-specific threshold calibration, and prespecified multi-source aggregation rules.

In terms of cross-model comparison, Pataranutaporn et al. (2025) evaluated four models, GPT-4o, Claude 3.5, Gemma 3, and Llama 3.3—using more than 29,000 assessments of 1220 anonymized economics papers, together with a second experiment comprising 8910 bias-focused evaluations of 330 papers. Their results indicate that all models could distinguish high-, medium-, and low-quality papers. Claude and Gemma showed more stable performance in identifying quality gradients, whereas GPT-4o demonstrated a comparative advantage in detecting AI-generated content. At the methodological level, Zhu et al. (2025) proposed DeepReview, a multi-stage framework that combines structured analysis, literature retrieval, and evidence-based argumentation. Using the DeepReview-13K dataset, its DeepReviewer-14B model outperformed CycleReviewer-70B and achieved reported win rates of 88.21% and 80.20% against GPT-o1 and DeepSeek-R1, respectively, illustrating a shift from direct prompting toward structured review generation.

The use of LLMs for scoring has further given rise to the “LLM-as-a-Judge” paradigm, whose systematic biases have increasingly been supported by empirical evidence. Zheng et al. (2023) found that LLM judges exhibit a pronounced position bias, tending to assign higher scores to candidate responses presented first; this pattern was consistently observed across multiple models. Koo et al. (2024) systematically measured six categories of cognitive bias in LLM evaluation and showed that, across 16 tested models, approximately 40% of pairwise comparisons displayed biased tendencies, while human-model preference agreement was only 44%. These findings call into question the reliability of directly using LLM scoring for selection decisions. In the context of academic peer review, such biases also exhibit domain-specific characteristics. In a single-blind experiment, Pataranutaporn et al. (2025) found that when author names and institutional affiliations were visible, multiple LLMs tended to assign higher scores to submissions from top institutions, male authors, and well-known scholars; under anonymous conditions, scores for the same papers declined significantly. This suggests that part of the bias may stem from social priors internalized during training. Li et al. (2024) conducted a systematic review of LLM evaluation methods and categorized scoring bias into forms such as leniency bias and central tendency bias, thereby providing a conceptual framework for understanding LLM scoring behavior. With respect to scale calibration, several methods have been proposed to mitigate LLM bias, including the pairwise preference search proposed by Y. Liu et al. (2024) and the prompt-optimization-based review generation framework developed by Gao et al. (2024). However, most of these methods were designed for general evaluation tasks, and systematic validation in the structured setting of academic conference acceptance decisions remains limited.

To address the limitations of a single LLM instance, such as context window constraints, bias propagation, and limited generalizability, multi-model collaboration and multi-agent methods have rapidly emerged. The MARG framework proposed by D’Arcy et al. (2024) distributes paper content across multiple specialized GPT-4 instances coordinated by a leader agent. Compared with a single-instance baseline, MARG reduced the proportion of generic comments from 60% to 29% and increased the number of high-quality comments per paper from 1.7 to 3.7. Jin et al. (2024) proposed the Agent Review framework, which applies a multi-agent system to the analysis of peer-review mechanisms. Their study showed that variation in reviewer role settings alone—responsible, irresponsible, or malicious—could produce a 37.1% difference in final acceptance decisions, thereby quantifying the effects of social influence and authority bias in peer review. Related work on research assessment has also proposed a novelty-first intelligent review framework in which AI and scientometric evidence augment, rather than displace, expert judgment (Lathabai et al., 2026). Although developed for research-proposal evaluation, this design reinforces the need to assess AI reviewing as a structured decision-support system rather than as an autonomous substitute.

At the governance level, the application of LLMs in the review process has raised concerns regarding review fairness and research integrity. Submitting unpublished manuscripts to AI service platforms may violate the confidentiality principle of peer review, and in most cases, neither journals nor authors can determine whether reviewers have used AI tools or to what extent. Most journals explicitly prohibit reviewers from using AI to generate the main body of review reports, while permitting limited language polishing. For example, major publishers such as Springer Nature (2026), Wiley (2026), and Frontiers (2026) adopt different but consistently human-accountable approaches: Springer Nature prohibits manuscript uploads and requires disclosure of AI-supported evaluation; Wiley states that peer review should not be delegated to AI; and Frontiers restricts external tools while providing a secure internal assistant that cannot participate in editorial decisions. Elsevier (2026) and Cell Press (2026), by contrast, explicitly prohibit the use of generative AI or AI-assisted technologies in manuscript review to avoid potential confidentiality and copyright risks. Academic conferences, however, tend to adopt more permissive and less standardized policies regarding AI use (Zhuang et al., 2025). These policy differences reflect the lack of consensus within academia concerning the risks of LLM involvement in peer review, and they highlight the urgent need for unified standards of AI use, disclosure, and accountability. An audit of 1119 Latin American journals found that only 27.2% had explicit AI-use guidelines, with significant variation across journal quality strata (Zahn-Muñoz et al., 2026). This uneven regulatory landscape supports the need for disclosure, confidentiality, and accountability mechanisms specific to AI-assisted review.

Against this background, the innovations and contributions of this study are reflected in four main aspects.

First, in terms of data construction, a parallel review corpus is built from ICLR 2025 conference data. The corpus includes 600 papers, 2401 human reviews, and 7203 AI-generated reviews, and incorporates a four-level decision hierarchy: Oral, Spotlight, Poster, and Rejected, thus providing a data foundation for hierarchy-sensitive analysis.

Second, in terms of model selection and methodological design, three current commercial LLMs are examined: Claude Sonnet 4.5, GPT-5.2 Thinking, and Gemini 3 Pro Preview. For each paper and model, the number of separate, context-isolated review generations is matched to the number of available human reviews, enabling cross-model comparison and human-containing aggregation analysis. Standardized prompts, common generation parameters, and a dual-threshold framework support consistent comparisons across models.

Third, in terms of the analytical framework, a unified analysis system is constructed that examines not only what is reviewed but also how the review is conducted, thereby addressing the limitations of prior studies that either focus on a single domain or emphasize feedback quality without analyzing decision alignment.

Fourth, in terms of empirical value, this study quantifies systematic high-score bias and hierarchical compression in raw LLM ratings and examines whether threshold calibration and prespecified aggregation rules increase agreement with conference decisions. The findings characterize the behavior of these rules in the evaluated ICLR cohorts; they do not by themselves establish real-world workflow effectiveness, independent review quality, or causal human–AI complementarity.

3. Research Methodology and Hypotheses

3.1. Research Design and Technical Workflow

This study adopts an empirical research paradigm and takes the ICLR 2025 review process as its research context. It systematically examines decision alignment, review-text characteristics, and multi-source aggregation patterns across several mainstream LLMs in academic peer review. The overall technical framework is illustrated in Figure 1.

Figure 1. Technical roadmap.

3.2. Multidimensional Analysis Framework

Centered on the core question of whether LLMs can perform paper review tasks, this study systematically evaluates their performance in academic peer review across multiple dimensions. The proposed analytical framework comprises the following aspects:

• Decision alignment and threshold calibration

A dual-threshold framework (τ = 5 and τ = 6) is adopted to map continuous AI scores into binary decisions. Accuracy, macro-F1, and Cohen’s κ are computed, supplemented by confusion-matrix-based structural analysis. The full threshold range τ ∈ (1, 10) is also systematically traversed to identify the optimal decision threshold. This design serves to reveal the systematic high-score bias and hierarchical compression in raw LLM ratings, and to examine whether threshold calibration improves decision alignment. The ICLR 2025 cohort was used for threshold development, after which the selected thresholds were locked and applied without retuning to the stratified balanced ICLR 2024 validation sample.

• Review-text characteristics analysis

Review-text characteristics are described using review length, structural completeness, critical-section elaboration, and TF-IDF lexical features. These indicators characterize output form and lexical emphasis; they do not measure technical correctness, usefulness, methodological depth, major-error detection, or hallucinated criticism.

• Inter-model consistency

Fleiss’ κ is employed to measure the overall agreement among the three LLMs and human reviewers in review decisions, and Spearman’s ρ is used to quantify monotonic consistency at the scoring level. Building upon this, dissent-contribution analysis is applied to identify reviewers who systematically deviate from the majority opinion in collective review settings, thereby revealing the structural characteristics of agreement both across models and between humans and models.

• Sensitivity to paper attributes

The sample is stratified by research type and primary field, and Kruskal–Wallis tests assess differences in LLM decision alignment across nominal categories. In addition, 95% confidence intervals are constructed through 1000 paper-level bootstrap resamples to characterize uncertainty and observed subgroup heterogeneity.

• Alignment with human confidence

Human reviewers are divided into three confidence groups: high (4–5), medium (3), and low (1–2). The point-biserial correlation between AI scores and each confidence group is computed. Concurrently, the median absolute deviation (MAD) is used to characterize the distribution of papers that are controversial among human reviewers. On this basis, the relationship between human–AI alignment and the degree of disagreement within human review is examined, testing whether high-confidence human reviewers exhibit stronger agreement with LLM scores.

• Multi-model and human-containing aggregation analysis

For paper $p$ and review source $a$, let ${\overline{s}}_{pa}$ denote the mean score across the available reviews from that source. For a set $A$ of review sources, the equal-weight mean rule predicts acceptance when

$${\displaystyle \frac{1}{\left|A\right|}}\sum _{a\in A}{\overline{s}}_{pa}>\tau .$$

Voting first converts each source mean to

$$b_{pa}=\mathbb{I}\left({\overline{s}}_{pa}>\tau \right)$$

and predicts acceptance only when

$$\sum _{a\in A}{b}_{pa}>{\displaystyle \frac{\left|A\right|}{2}};$$

ties are classified as rejected. Thus, each source-level mean, rather than each individual review, receives equal weight.

These are retrospective aggregation rules rather than interactive human–AI collaboration. The two configurations selected on the 2025 development cohort were the Human + GPT joint-acceptance rule (two-source voting with ties rejected) at τ = 5 and the Human + Claude + GPT equal-weight mean at τ = 6. Both rules were applied unchanged to the 2024 cross-year robustness cohort. Paired accuracy differences were evaluated using 5000 paper-level bootstrap resamples and exact McNemar tests.

3.3. Research Hypotheses

Based on the analytical framework above, this study proposes four research hypotheses (H1–H4), which will be tested in the subsequent analysis. The hypotheses and their theoretical foundations are summarized in Table 1.

Table 1. Core hypotheses and their theoretical foundations.

No.	Hypothesis	Analytical Focus	Theoretical Basis
H1	Raw LLM scores exhibit systematic high-score bias (leniency bias). Under the standard decision threshold (τ = 5), their acceptance rate is significantly higher than the actual ICLR 2025 acceptance benchmark. Through optimal threshold calibration, decision alignment, as measured by Cohen’s κ, can be significantly improved.	Scoring bias	Research on LLM-as-a-Judge suggests that LLM scoring may be affected by systematic biases such as position bias and verbosity preference (Li et al., 2024), indicating that raw scores do not constitute neutral measurement. Automation bias theory further suggests that automated recommendations may induce overtrust and amplify bias (Parasuraman & Riley, 1997), thereby increasing the risk of over-acceptance (Latona et al., 2024). Methodologically, model calibration and post hoc threshold tuning have been shown to improve probabilistic outputs and decision metrics (Guo et al., 2017), providing a basis for threshold calibration.
H2	At the paper level, the three LLMs exhibit nontrivial convergence in binary decisions and score rankings.	Inter-model convergence	Fleiss’ and Cohen’s kappa quantify agreement among raters on a common unit of analysis (Fleiss, 1971). Prior work also indicates that different LLMs may reproduce shared scoring tendencies despite architectural differences (Li et al., 2024).
H3	LLM decision alignment differs across research types. The expectation is stronger alignment for Empirical papers centered on experimental validation and greater disagreement for Theoretical papers dominated by formal derivation.	Type sensitivity	Empirical studies indicate that LLMs are better suited to verifiable and highly structured subtasks, while remaining limited in holistic quality judgment (R. Liu & Shah, 2023). Existing work also shows that LLM-generated reviews partially overlap with human concerns but tend to focus more heavily on experiments and validation evidence (Liang et al., 2024b). In addition, surveys of mathematical and formal reasoning indicate that LLMs still face structural limitations in rigorous derivation and formal reasoning (Wang et al., 2025).
H4	Under prespecified aggregation rules, configurations that include the human-review mean show higher agreement with ICLR 2025 conference decisions than the evaluated single-model aggregates.	Human-containing aggregation	Ensemble learning theory holds that when individual judgments possess some degree of accuracy, and their errors are not perfectly homogeneous, voting or weighted aggregation can produce better overall performance than any single agent (Dietterich, 2000). The Condorcet Jury Theorem explains, from a group decision-making perspective, why majority voting can improve correctness under certain conditions (Y. Liu et al., 2024). Hybrid intelligence research further emphasizes complementarity and collaborative optimization between humans and machines (Kamar, 2016), while cooperative game-theoretic methods such as the Shapley value can be used to quantify the marginal contribution of each participant to ensemble performance (Shapley, 1953).

4. Original Data Collection and Preprocessing

4.1. Data Collection

This study uses the official ICLR 2025 review platform, OpenReview, as the data source for the ICLR-Review-600 dataset. Stratified balanced sampling yielded 300 accepted and 300 rejected papers, including 100 Oral, 100 Spotlight, and 100 Poster papers, together with 2401 human review reports. For cross-year validation, a separate stratified balanced sample of 300 ICLR 2024 papers was constructed, comprising 50 Oral, 50 Spotlight, 50 Poster, and 150 Rejected papers. The validation sample was excluded from threshold development. The structure of the peer-review content is illustrated in Figure 2.

Figure 2. OpenReview peer review interface structure diagram. Note: Figure 2 is adapted from Paleka et al. (2024), Consistency checks for language model forecasters, accepted at ICLR 2025.

To enable efficient and accurate data acquisition, a two-stage automated collection pipeline based on Python 3 is developed. In the first stage, Requests and BeautifulSoup are used to perform structured parsing of the OpenReview platform, batch-extracting paper metadata and associated human review reports, including fields such as Score and Confidence. In the second stage, Selenium combined with Edge WebDriver is employed to collect content from dynamically rendered pages, ensuring complete retrieval of initial manuscript PDFs. This division of labor between the two stages not only improves collection efficiency but also enhances adaptability to different rendering mechanisms across web pages.

The entire collection workflow was executed in a containerized environment. Task progress and collected entries were persistently recorded through status files to support resumption from interruption, thereby ensuring robustness and data integrity in large-scale acquisition. Data collection was strictly limited to publicly accessible, non-confidential content available on OpenReview. According to OpenReview’s terms, user-submitted comment content, including reviews, discussions, and responses, is publicly released under the CC BY 4.0 license upon submission, while metadata associated with the works is provided under CC0 1.0. Accordingly, this study conducts data access, storage, and analysis strictly within the scope permitted by the platform’s terms and open licenses (OpenReview, 2024), without involving any non-public content, personal private information, or materials requiring additional authorization. This ensures the transparency, traceability, reproducibility, and compliance of the data collection process, and provides a reliable data foundation for subsequent corpus analysis and hypothesis testing.

4.2. Data Preprocessing and Feature Annotation

During data preprocessing, the initial-submission PDFs underwent structural parsing and content extraction. The AI systems evaluated these initial-submission PDFs. Human review texts and scores were retrieved from the public OpenReview records after the conference decisions had been released and may therefore include post-rebuttal or post-discussion updates. Accordingly, the analysis does not assume that the AI and human assessments were produced at the same review stage. Human review reports were standardized by extracting score and confidence fields, and paper-level decision categories were assigned from the official OpenReview records. Research type (Theoretical, Empirical, or Hybrid) and primary field (D1–D8) were independently assigned to all 600 papers by two human coders using a predefined coding scheme; no AI system participated in this coding. The first complete coding was used for full-sample stratified analyses, while the second independent coding was used to quantify pre-adjudication agreement. Agreement-only subsets were examined as a sensitivity analysis.

The 2024 and 2025 data were processed using the same paper-level aggregation and binary-decision rules. Scores outside the permitted ICLR categories were harmonized before aggregation (4 to 5 and 7 to 8). For each paper and model, the first N non-missing AI scores were retained, where N equaled the number of corresponding human reviews; unavailable reviewer slots were excluded. A paper was classified as accepted only when its source-level mean score was strictly greater than the specified threshold.

ICLR 2025 served as the development cohort. Thresholds were searched from 1.0 to 10.0 in 0.1-point increments by maximizing macro-F1, with ties resolved by selecting the lowest threshold. Stability was assessed using 20 repetitions of stratified five-fold cross-validation. The selected thresholds and the two prespecified aggregation rules taken forward were then locked and applied without retuning to the ICLR 2024 cross-year robustness cohort; no analytical setting was changed after inspecting the 2024 outcomes. Uncertainty was estimated using 5000 paper-level class-stratified bootstrap resamples, and paired classification changes were examined with exact McNemar tests.

Three current commercial LLMs, Claude Sonnet 4.5, GPT-5.2 Thinking, and Gemini 3 Pro Preview, were used to generate review reports and are abbreviated as Claude, GPT, and Gemini, respectively, in the main text and figures. Their differing architectures and scoring tendencies permit cross-model and aggregation analyses. For each paper, each model completed the same number of separate, context-isolated review generations as the number of corresponding human reviewers. This procedure produced 7203 AI-generated reviews, or approximately 12 reviews per paper across the three models.

Each paper–model-review instance was produced in a separate API run with a fresh, isolated context containing only the manuscript and fixed prompt. The leader agent served solely as an orchestration layer: it dispatched requests and collected completed outputs, but did not generate review content, enable communication among runs, discuss results, or synthesize reviews during generation. No run could access or influence another AI output or any human review. The runs were therefore procedurally independent at the request level, although outputs from the same model were not treated as statistically equivalent to judgments from independent human reviewers.

Potential textual overlap was assessed at the paper–model level across 1800 concatenated review documents. After removal of metadata, ratings, confidence fields, URLs, section headings, and boilerplate, each AI document was compared with the corresponding human-review text and a fixed-seed within-stratum mismatch. Normalized token eight-gram containment, TF-IDF cosine similarity, and sentence-level token-sequence similarity were calculated. Paired differences were estimated at the paper level using 5000 bootstrap resamples and 10,000 sign-flip permutations. Because corresponding human and AI reviews addressed the same manuscript, positive differences were treated as a diagnostic of shared topical content rather than evidence of prior model exposure.

A unified structured prompting framework was applied across all models and generation tasks. The prompt assigned the model the role of an “ICLR 2025 Reviewer” to establish a consistent evaluation context. It also required each model to assess the complete manuscript using the standard ICLR review structure, including Summary, Presentation, Contribution, Strengths, Weaknesses, and Questions. Each review concluded with an overall rating selected from the official ICLR score categories (1, 3, 5, 6, 8, or 10) and a confidence score ranging from 1 to 5. Applying the same role definition, evaluation dimensions, scoring scale, and output structure across all batch-generation tasks enhanced structural comparability among models and maintained consistency in the experimental conditions. The complete fixed prompt and generation procedure are provided in Appendix A.

The three models were configured with identical generation parameters: Temperature = 1.0, Top-p = 1.0, and max_tokens = 8192. These settings were applied consistently across all separate, context-isolated review generations. The maximum token limit was set to provide sufficient capacity for complete review reports and minimize the risk of output truncation; alternative decoding settings were not evaluated in the present analysis.

Variation across repeated, context-isolated generations was evaluated for each paper–model combination using the within-paper standard deviation of review scores, binary-decision unanimity at tau = 5 and at each model’s locked threshold and mean pairwise agreement at the locked threshold. Summary statistics were aggregated at the paper level.

To distinguish variability in individual generated reviews from the stability of the aggregated paper-level result, a leave-one-review-out analysis recomputed each source-level mean after omitting one generated review at a time and compared the resulting decision with the decision based on the full set of generated reviews.

The final ICLR decision served as the outcome criterion for decision-agreement analyses. These decisions incorporate reviewer reports and scores, meta-review, discussion, and author response.

5. Analysis and Validation

5.1. Descriptive Statistics and Distributional Characteristics of the Dataset

The ICLR-Review-600 dataset contains 600 papers, including 300 accepted and 300 rejected papers. The accepted papers comprise 100 Oral, 100 Spotlight, and 100 Poster papers. Under the primary human coding, 515 papers were Empirical (85.8%), 48 Hybrid (8.0%), and 37 Theoretical (6.2%). By primary field, D1 was the largest group with 151 papers (25.2%), whereas D7 was the smallest with 22 papers (3.7%).

Regarding the human-review setup, each paper is assigned an average of 4.00 reviewers, ranging from 3 to 6, and the number of separate, context-isolated generations per model is strictly matched to the number of human reviewers. The distribution of human reviewers’ confidence is right-skewed: high confidence (scores 4–5) accounts for 56.6%, medium confidence (score 3) for 35.7%, and low confidence (scores 1–2) for only 7.7%, suggesting that most reviewers have relatively high confidence in their judgments. Table 2 reports the mean scores from different review sources across acceptance tiers.

Table 2. Average score by review source based on acceptance tier.

Acceptance Tier	Human	Claude	GPT	Gemini
Oral	7.75	6.88	6.83	7.39
Spotlight	7.32	6.71	6.84	7.22
Poster	6.28	6.66	7.00	7.20
Rejected	4.62	5.46	5.55	5.92
Overall range	3.13	1.42	1.28	1.47

Note: Score range: 1–10. Range = (mean Oral score − mean Rejected score), reflecting how well a review source distinguishes quality gradients.

From Table 2, human review scores can clearly differentiate between quality tiers: the mean scores for Oral, Spotlight, Poster, and Rejected decrease in order, forming a stable gradient. In contrast, although the three LLMs also show an overall trend of decreasing scores as tier decreases, their tier-separation ability is noticeably weaker than that of human reviewers.

Specifically, LLM scoring shows two prominent characteristics. First, there is an overall high-score bias. For rejected papers, Claude, GPT, and Gemini have mean scores of 5.46, 5.55, and 5.92, respectively, each higher than the human mean of 4.62. Gemini’s mean score for rejected papers is already close to the standard acceptance threshold (τ = 5), reflecting a strong tendency toward over-acceptance. Second, there is tier compression. Using the score range between Oral and Rejected to measure quality-gradient separability, the human range is 3.13, whereas Claude, GPT, and Gemini are only 1.42, 1.28, and 1.47, respectively. This indicates that LLMs fail to spread scores sufficiently between high-and-low-quality papers and thus struggle to capture quality differences within acceptance tiers.

Figure 3 uses empirical cumulative distribution functions (ECDFs), supplemented with compact interquartile summaries, to compare score distributions across review sources and acceptance tiers.

Figure 3. Empirical cumulative score distributions by review source and acceptance tier.

Figure 3 shows that human reviewers provide the clearest separation among acceptance tiers, with the Rejected tier shifted substantially toward lower scores and Oral papers concentrated at the upper end. The LLM curves are more compressed and overlap more strongly between Poster and Rejected papers, indicating weaker discrimination for borderline and low-quality submissions.

5.2. Decision Alignment and Threshold Calibration

To examine how well the three LLMs’ acceptance decisions align with ICLR 2025 conference decisions under different thresholds, the following analysis is conducted. (Pairwise Cohen’s κ among human reviewers is 0.392 at τ = 5 and 0.328 at τ = 6, indicating that even within the same conference, inter-reviewer agreement among humans is only moderate to low.) A dual-threshold framework is adopted: a standard threshold τ = 5 (mean score > 5 indicates acceptance) and a stricter threshold τ = 6 (mean score > 6 indicates acceptance). Models are compared across five dimensions: Accuracy, macro-averaged F1 (F1-macro), Cohen’s κ, false positive rate (FPR), and false negative rate (FNR). The results are shown in Table 3.

Table 3. Decision-alignment metrics under dual thresholds.

Model	Threshold	Accuracy	F1-Macro	Cohen’s κ	FPR	FNR
Claude	τ = 5	0.645	0.595	0.290	0.707	0.003
Claude	τ = 6	0.917	0.916	0.833	0.150	0.017
GPT	τ = 5	0.692	0.660	0.383	0.613	0.003
GPT	τ = 6	0.787	0.779	0.573	0.397	0.030
Gemini	τ = 5	0.603	0.529	0.207	0.793	0.000
Gemini	τ = 6	0.682	0.647	0.363	0.630	0.007

Note: FPR is the false positive rate (rejects misclassified as accepts), and FNR is the false negative rate (accepts misclassified as rejects). Human inter-reviewer κ: 0.392 at τ = 5 and 0.328 at τ = 6.

As shown in Table 3, under the standard threshold τ = 5, all three LLMs exhibit pronounced alignment imbalance. Specifically, Claude, GPT, and Gemini have FPRs of 0.707, 0.613, and 0.793, meaning that more than 60% of rejected papers are incorrectly judged as accepted, while FNR is nearly zero (≤0.003), i.e., there are almost no missed acceptances. This asymmetric error structure indicates that LLM raw scores generally have a strong leniency bias: Models tend to assign high scores, producing systematic over-acceptance under the standard threshold.

When the threshold is increased to tau = 6, decision agreement improves for all three models, although the magnitude differs. Claude improves most: accuracy increases from 0.645 to 0.917 and Cohen’s kappa from 0.290 to 0.833. GPT and Gemini show smaller gains. Raising the threshold therefore reduces over-acceptance and improves agreement with the ICLR 2025 decisions.

Model-specific thresholds were identified using a 0.1-point grid search over tau in [1.0, 10.0], with macro-F1 as the optimization criterion. Results are shown in Table 4.

Table 4. Thresholds and performance estimated on the ICLR 2025 sample by F1-macro search.

Model	Threshold, τ*	Accuracy	F1-Macro	Cohen’s Kappa	ΔAccuracy vs. τ = 5
Claude	6.2	0.927	0.926	0.853	+0.282
GPT	6.3	0.913	0.913	0.827	+0.221
Gemini	6.7	0.932	0.932	0.863	+0.329

Note: Thresholds were selected on the ICLR 2025 development cohort using a 0.1-point grid and were locked before cross-year validation. Values are model- and sample-specific calibration parameters τ* denotes the model-specific threshold that maximized macro-F1 on the ICLR 2025 development cohort in a 0.1-point grid search. Thresholds were locked before cross-year validation. ΔAccuracy vs. τ = 5 denotes the change in accuracy relative to the τ = 5 baseline. Values are model- and sample-specific calibration parameters.

The development-cohort thresholds were 6.2 for Claude, 6.3 for GPT, and 6.7 for Gemini. Corresponding accuracy values were 0.927, 0.913, and 0.932, with Cohen’s kappa values of 0.853, 0.827, and 0.863. Each optimum occupied a single grid point. These estimates describe calibration within the development cohort and are evaluated separately below through repeated cross-validation and cross-year validation.

Figure 4 presents the decision-outcome composition for each model under the standard threshold (τ = 5) and the stricter threshold (τ = 6).

Figure 4. Decision-outcome composition under standard and strict thresholds.

Figure 4 shows that, under τ = 5, all three models generate substantial false positives, with many rejected papers classified as accepted. Raising the threshold to τ = 6 markedly reduces false positives, especially for Claude, and produces a more conservative decision profile. Figure 5 further examines whether these classification patterns are associated with systematic score bias.

Figure 5. Model-specific density plots of score bias for Claude, GPT, and Gemini. Note: Hexagons show the joint distribution of mean human scores and score bias, with darker shading indicating more papers per hexagon. The colored line summarizes the trend in score bias across mean human scores, and the surrounding translucent band indicates the uncertainty around that trend. The horizontal dashed line denotes zero bias, and the vertical dotted line denotes the standard threshold τ = 5.

Figure 5 plots LLM score bias (LLM mean score minus human mean score) against the mean human review score for each model. The density patterns indicate that LLMs tend to overestimate lower-scored papers and that the magnitude of overestimation declines as human scores increase, revealing a systematic calibration bias rather than random scoring noise.

Based on these analyses: (i) LLM raw scores show pronounced leniency and cannot be transferred directly into conference-style binary decisions; (ii) threshold calibration substantially improves agreement with ICLR 2025 decisions and tier-related separation; and (iii) at τ = 5, all three models have FPR > 0.60, indicating systematic over-acceptance relative to the conference outcomes. H1 is therefore supported within the development cohort and is further examined through cross-validation and locked cross-year testing.

5.2.1. Repeated Cross-Validation and Cross-Year Validation

Across 100 training-fold fits from repeated stratified five-fold cross-validation, the median thresholds were 6.2 for Claude, 6.3 for GPT, and 6.7 for Gemini, with zero-width interquartile ranges at the reported precision. Mean out-of-fold accuracy was 0.927 (95% CI: 0.905–0.947), 0.913 (0.890–0.935), and 0.932 (0.912–0.952), respectively; corresponding Cohen’s kappa values were 0.853, 0.827, and 0.863.

The locked thresholds were then applied without retuning to the stratified balanced ICLR 2024 sample. Claude achieved accuracy 0.927 (95% CI: 0.897–0.953), macro-F1 0.926, and kappa 0.853; GPT achieved 0.913 (0.880–0.943), 0.913, and 0.827; and Gemini achieved 0.930 (0.900–0.957), 0.930, and 0.860. Sensitivity/specificity were 0.987/0.867, 0.967/0.860, and 0.940/0.920, respectively. The calibration pattern therefore transferred across conference years without threshold retuning.

A like-for-like aggregate comparison used the same conference-decision outcome and the same 0.1-point threshold-search rule. In 2025, aggregate human scores achieved accuracy 0.940 and kappa 0.880 at tau = 5.9, compared with 0.927/0.853 for Claude, 0.913/0.827 for GPT, and 0.932/0.863 for Gemini. With these thresholds locked, the corresponding 2024 values were 0.903/0.807, 0.927/0.853, 0.913/0.827, and 0.930/0.860. This comparison replaces the nonequivalent contrast between model-to-decision kappa and pairwise kappa among individual human reviewers. The complete like-for-like comparison is reported in Appendix B (Table A3).

5.2.2. Prevalence and Cost-Sensitivity Scenarios

Under a 30% acceptance-rate scenario, projected accuracy was 0.903 for Claude, 0.892 for GPT, and 0.926 for Gemini; PPV was 0.760, 0.747, and 0.834, and NPV was 0.993, 0.984, and 0.973. Per 1000 submissions, the corresponding estimates were 93.3, 98.0, and 56.0 false acceptances and 4.0, 10.0, and 18.0 false rejections. These projections retain the class-conditional error rates observed in the 2024 validation sample. Detailed projected results are provided in Appendix B (Table A1).

At a 30% prevalence, cost-sensitive optimization on the development cohort yielded preferred thresholds ranging from 6.2 to 6.7 for Claude, 6.3 to 7.0 for GPT, and 6.7 to 7.0 for Gemini across false-acceptance:false-rejection cost ratios of 1:1, 2:1, 1:2, 5:1, and 1:5. A single threshold should therefore not be interpreted as universally optimal. The full set of cost-sensitive thresholds is reported in Appendix B (Table A2).

The projected 30% acceptance-prevalence results also show that the value of a human–AI combination depends on the aggregation rule. The Human + GPT joint-acceptance rule achieved expected agreement of 0.828, below Claude (0.903), GPT (0.892), and Gemini (0.926) under their locked thresholds. Because ties are rejected in this two-source rule, both Human and GPT must recommend acceptance. In contrast, the Human + Claude + GPT equal-weight mean achieved expected agreement of 0.965. The thresholds and rules were selected using the 2025 development cohort, whereas these 30% projections use the class-conditional error rates observed after locked application to the separate 2024 cohort.

5.3. Review-Text Characteristics

5.3.1. Review Length and Structural Completeness

Review length provides a descriptive measure of output volume rather than substantive quality. As shown in Table 5, Claude produces the longest reviews (weighted mean 1214.5 tokens), compared with 971.3 tokens for human reviewers. GPT (882.5 tokens) and Gemini (828.3 tokens) are shorter than the human reviews. This pattern is stable across acceptance tiers and primarily characterizes source-specific generation length.

Table 5. Statistics on the length of review comments by source.

Source	Mean Tokens	Token Ratio	Mean Characters in Weaknesses	Length Characteristic
Human	971.3	1.00	1240	Human reference
Claude	1214.5	1.25	1394	Longest output; largest Weaknesses section
GPT	882.5	0.91	686	Shorter output; smaller Weaknesses section
Gemini	828.3	0.85	555	Shortest output; smallest Weaknesses section

Note: Token ratio uses the human mean as the denominator.

Table 5 summarizes length statistics of review comments by source. In terms of critical content, Claude’s Weaknesses section has an average of 1394 characters, also higher than the human average of 1240 characters; GPT and Gemini are 686 and 555 characters, respectively, noticeably shorter than human reviews. Overall, Claude is the closest to (or even exceeds) human reviewers in textual elaboration, while GPT and Gemini show a more concise tendency.

It should be noted that longer length does not necessarily imply higher-quality criticism. Although Claude produces the most text in the Weaknesses section, it contains many structured numbering and formatted expressions, which can inflate surface length without necessarily increasing the density of substantive critique. Therefore, the length advantage reflects an expansive generation style rather than directly equating to review depth.

5.3.2. Repeated-Generation Stability

Repeated, context-isolated generations showed non-negligible within-paper variation under the fixed prompting protocol. Table 6 summarizes score dispersion and binary-decision consistency across the 3–6 reviews generated for each paper–model combination.

Table 6. Stability across repeated isolated generations.

Model	Mean Within-Paper SD	Median SD (IQR)	Unanimity at tau = 5	Unanimity at tau*	Pairwise Agreement at tau*
Claude	1.154	1.258 (0.996–1.500)	0.132	0.395	0.645
GPT	1.488	1.500 (1.258–1.528)	0.092	0.248	0.545
Gemini	1.257	1.258 (1.095–1.500)	0.400	0.170	0.534

Note: SD is calculated across repeated review scores within each paper. Unanimity is the proportion of papers for which all repeated reviews yielded the same binary decision. tau* denotes the locked model-specific threshold.

Mean within-paper score SD was 1.154 for Claude, 1.488 for GPT, and 1.257 for Gemini. Decision unanimity at tau = 5 was 13.2%, 9.2%, and 40.0%, respectively; At the locked thresholds, it was 39.5%, 24.8%, and 17.0%. The generation runs were procedurally independent because no run could observe or influence another. Nevertheless, outputs from the same model shared the model architecture, training background, prompt, manuscript, and hyperparameters; they were therefore interpreted as repeated, context-isolated generations from one model rather than as statistically independent human judgments.

The leave-one-review-out results distinguish the instability of a single generated review from the stability of the averaged paper-level decision. At the locked thresholds, 92.7% of Claude, 85.8% of GPT, and 86.9% of Gemini leave-one-out decisions matched the corresponding full-mean decision. The decision remained unchanged under every possible one-review omission for 74.5%, 68.7%, and 73.5% of papers, respectively. Averaging therefore reduced the influence of any single generation, although it did not eliminate stochastic variability.

The differing Gemini pattern reflects the position of the threshold relative to its score distribution. At τ = 5, Gemini produced 234 unanimous-accept and five unanimous-reject cases; at its locked threshold of 6.7, these counts changed to 18 and 84, respectively. The higher threshold moved into a dense region of Gemini scores, so repeated scores more often fell on opposite sides of the cutoff. For Claude and GPT, by contrast, the locked thresholds produced 237 and 149 unanimous-reject cases, respectively, which increased their overall unanimity. Thus, threshold-specific unanimity should not be interpreted as a model-invariant measure of generation stability.

5.3.3. Inter-Coder Reliability of Paper Attributes

The two human coders independently classified all 600 papers before any reconciliation. Table 7 reports pre-adjudication agreement for the paper attributes used in the stratified analyses.

Table 7. Pre-adjudication reliability of independent human coding.

Coding Dimension	N	Observed Agreement	Cohen’s Kappa	Bootstrap 95% CI
Research type	600	0.947	0.774	0.695–0.843
Primary field	600	0.755	0.714	0.673–0.753
Secondary field	600	0.630	0.409	0.357–0.461

Note: Confidence intervals were obtained from 20,000 paper-level bootstrap resamples. Kappa is unweighted because all coding dimensions are nominal. Secondary-field agreement treats the absence of a secondary field as a valid category.

Observed agreement was 94.7% for research type and 75.5% for primary field, with Cohen’s kappa values of 0.774 and 0.714, respectively. Secondary-field coding showed lower agreement (63.0%; kappa = 0.409). The H3 reanalysis therefore reports the complete primary coding and an agreement-only sensitivity analysis rather than treating unresolved coder differences as reconciled labels.

5.3.4. TF-IDF Lexical Features and Content Analysis

Lexical feature analysis can further reveal differences in what different review sources focus on. Table 8 lists the top eight high-weight unigram terms (by TF-IDF) for each source.

Table 8. Top 8 high-TF-IDF unigram terms by review source.

Rank	Human	Claude	GPT	Gemini
1	Propose (0.032)	Theoretical (0.030)	Baseline (0.025)	Performance (0.027)
2	Performance (0.028)	Performance (0.029)	Across (0.024)	Like (0.027)
3	Task (0.027)	Evaluation (0.027)	Task (0.024)	Task (0.026)
4	LLM (0.025)	Improvement (0.026)	Evaluation (0.023)	High (0.025)
5	Different (0.024)	Task (0.026)	Propose (0.022)	Specific (0.024)
6	Fair (0.023)	Baseline (0.026)	Train (0.022)	Training (0.024)
7	Learn (0.023)	Across (0.025)	Empirical (0.021)	Standard (0.023)
8	Compare (0.023)	Multiple (0.025)	Claim (0.021)	LLM (0.023)

From Table 8, high-weight terms in human reviews include ‘propose’, ‘performance,’ ‘task,’ ‘different,’ and ‘fair,’ indicating stronger attention to methodological novelty, task setup, performance, and comparative fairness. Claude’s high-weight terms, including ‘theoretical, ’evaluation,’ ‘improvement,’ and ‘baseline,’ suggest greater emphasis on theoretical rigor, experimental evaluation, and baseline design. GPT more frequently uses terms such as ‘baseline,’ ‘claim,’ ‘empirical,’ and ‘train,’ indicating greater focus on empirical support and the credibility of claims. Gemini’s high-weight terms include ‘performance,’ ‘like,’ ‘high,’ ‘specific,’ and ‘standard,’ reflecting greater use of general evaluative vocabulary and a more summary-oriented lexical profile.

Overall, review sources differ in both length and lexical focus. Human reviewers place more emphasis on contributions and comparative fairness; Claude emphasizes methodological and theoretical scrutiny; GPT focuses on empirical support and verifiability; and Gemini uses more general evaluative vocabulary.

5.3.5. Text-Overlap Robustness Check

Across 1800 paper–model documents, exact eight-gram containment was 0.000400–0.001223 for corresponding human reviews and 0–0.00000287 for within-stratum mismatches. TF-IDF similarity was 0.413–0.446 versus 0.076–0.101, and sentence-sequence similarity was 0.179–0.212 versus 0.122–0.146. All paper-level paired differences were significant (p < 0.001). Manual inspection of the highest-similarity 1% (18 documents) attributed the observed overlap to shared manuscript content or domain terminology rather than reviewer-specific evaluative language. The extremely low exact-phrase containment and higher semantic similarity are consistent with both review sources addressing the same manuscript; they neither establish nor exclude training-data exposure.

5.4. Inter-Model Consistency

To evaluate inter-model convergence consistently, all binary-decision and score-ranking statistics were calculated at the paper level across the same 600 manuscripts. Table 9, Table 10 and Table 11 report pairwise Cohen’s kappa, pairwise Spearman correlation, and three-model Fleiss’ kappa. No comparison with reviewer-level human agreement is used to test H2.

Table 9. Paper-level pairwise agreement among LLM binary decisions.

Model Pair	N Papers	Cohen’s κ	Bootstrap 95% CI
Claude–GPT	600	0.650	0.563–0.727
Claude–Gemini	600	0.495	0.387–0.595
GPT–Gemini	600	0.554	0.462–0.642

Note: Binary decisions use τ = 5. Confidence intervals were obtained from 5000 paper-level bootstrap resamples.

Table 10. Paper-level pairwise Spearman correlations among LLM mean scores.

Model Pair	N Papers	Spearman’s ρ	Bootstrap 95% CI
Claude–GPT	600	0.768	0.731–0.801
Claude–Gemini	600	0.799	0.769–0.827
GPT–Gemini	600	0.705	0.659–0.747

Note: Correlations use source-level mean scores across the same 600 papers. Confidence intervals were obtained from 5000 paper-level bootstrap resamples; all correlations were significant at p < 0.001.

Table 11. Consistency statistics for the three models.

Metric	Value
All-three agreement rate (τ = 5)	83.7%
All predicted “accept”	77.2%
All predicted “reject”	6.5%
2-vs-1 disagreement rate	16.3%
Fleiss’ κ (τ = 5)	0.570
Fleiss’ κ (τ = 6)	0.512

At the paper level, binary-decision agreement was κ = 0.650 (95% CI: 0.563–0.727) for Claude–GPT, 0.495 (0.387–0.595) for Claude–Gemini, and 0.554 (0.462–0.642) for GPT–Gemini. The Claude–GPT versus Claude–Gemini difference was significant (Δκ = 0.155, 95% CI: 0.048–0.265; bootstrap p = 0.005), whereas the other pairwise differences were not significant (p = 0.084 and 0.226). Score-ranking correlations were ρ = 0.768 (0.731–0.801), 0.799 (0.769–0.827), and 0.705 (0.659–0.747), respectively. Fleiss’ κ across all three models was 0.570 at τ = 5 and 0.512 at τ = 6. H2 is therefore supported as nontrivial paper-level inter-model convergence, without comparison to reviewer-level human agreement.

In Table 11, the three-model agreement shows an asymmetric structure: among the 83.7% agreement cases, as many as 77.2% are unanimous acceptance, while only 6.5% are unanimous rejection, again confirming the systematic leniency bias of LLMs under the standard threshold.

Dissenter analysis revealed a directional asymmetry. GPT was the minority source in 31 cases: in 28 cases, GPT recommended rejection while Claude and Gemini recommended acceptance, and all 28 corresponded to a final conference rejection; in the remaining three cases, GPT recommended acceptance while the other two models recommended rejection, and none corresponded to a final acceptance. GPT’s overall minority-decision agreement of 90.3% was therefore driven by its more conservative rejection direction in these disputed cases. This descriptive result may reflect differences in model-specific acceptance tendencies and does not support assigning greater weight to GPT minority opinions in general.

Figure 6 represents inter-model agreement as weighted networks using Cohen’s κ for threshold-based decisions and Spearman’s ρ for score rankings.

Figure 6. Weighted inter-model agreement networks based on Cohen’s κ and Spearman’s ρ.

Figure 6 indicates that agreement among the three LLMs is nontrivial but uneven. GPT and Claude exhibit the strongest decision-level agreement (κ = 0.650), while the rank-order correlations are generally higher, suggesting that the models are more similar in relative scoring tendencies than in binary accept/reject decisions.

Together, Table 9, Table 10 and Table 11 show nontrivial convergence among the three LLMs in both binary decisions and score rankings. H2 is therefore supported with respect to inter-model convergence; no inference is drawn by comparing calibrated model-to-decision kappa with pairwise kappa among individual human reviewers.

5.5. Paper Attribute Sensitivity

5.5.1. Sensitivity of Research Type

Table 12 reports decision-agreement accuracy by independently human-coded research type at tau = 5.

Table 12. Decision accuracy by study type (τ = 5).

Research Type	N	Accept/Reject	Claude (tau = 5)	GPT (tau = 5)	Gemini (tau = 5)
Theoretical (T)	37	19/18	0.595	0.676	0.622
Empirical (E)	515	252/263	0.633	0.683	0.590
Hybrid (M)	48	29/19	0.812	0.792	0.729

Note: N is the number of papers in the first complete human coding. Accuracy denotes agreement with the conference decision at tau = 5.

At tau = 5, Hybrid papers showed the highest decision-agreement accuracy for Claude (0.812), GPT (0.792), and Gemini (0.729). Research-type differences were significant only for Claude (Kruskal–Wallis H = 6.60, p = 0.037), but not for GPT (H = 2.45, p = 0.293) or Gemini (H = 3.59, p = 0.166). The predicted advantage for Empirical papers and disadvantage for Theoretical papers was not observed.

At the locked model-specific thresholds, decision-agreement accuracy was 0.892/0.932/0.896 for Claude across Theoretical/Empirical/Hybrid papers, 0.892/0.913/0.938 for GPT, and 0.865/0.940/0.896 for Gemini. The differences were non-significant for all three models (p = 0.462, 0.751, and 0.129, respectively).

Thus, any research-type differences were limited, model-specific, and further attenuated after threshold calibration. An agreement-only sensitivity analysis produced the same substantive conclusion: calibrated differences were non-significant for Claude (p = 0.608), GPT (p = 0.697), and Gemini (p = 0.419).

Figure 7 uses forest plots with bootstrap 95% confidence intervals to compare decision accuracy across study types.

Figure 7. Decision-agreement accuracy by independently human-coded research type with bootstrap 95% confidence intervals.

Based on the independently human-coded analysis, Hybrid papers showed the highest raw decision-agreement accuracy, whereas calibrated differences across research types were non-significant for all three models. The agreement-only sensitivity analysis led to the same conclusion. Hypothesis H3, which predicted stronger performance on Empirical papers and more disagreement for Theoretical papers, is therefore not supported.

The Theoretical subgroup contained only 37 papers. Estimates for this subgroup are therefore less precise, and the analysis has limited power to detect modest differences across research types. The non-significant calibrated results for H3 should consequently be interpreted as insufficient evidence for the predicted pattern rather than evidence that research-type differences are absent.

5.5.2. Sensitivity to Research Fields

Table 13 presents decision-agreement accuracy by the primary field assigned in the first complete human coding. At tau = 5, field differences were not significant for Claude (H = 11.13, p = 0.133), GPT (H = 8.93, p = 0.258), or Gemini (H = 7.02, p = 0.427). The agreement-only sensitivity analysis indicated a field difference for Claude (p = 0.039) but not for GPT (p = 0.154) or Gemini (p = 0.221), showing that field patterns were not consistent across models or coding subsets.

Table 13. Accuracy of decisions by main area.

Primary Field	N	Claude	GPT	Gemini
D1 LLM/NLP	151	0.682	0.742	0.649
D2 CV/Multimodal	58	0.552	0.655	0.603
D3 RL/Decision-Making	69	0.696	0.681	0.580
D4 ML Theory	92	0.728	0.761	0.685
D5 AI4Sci	42	0.524	0.595	0.548
D6 System/Efficiency	90	0.622	0.678	0.556
D7 Graph Network	22	0.682	0.727	0.545
D8 Security/Alignment	76	0.579	0.605	0.539

Note: Kruskal–Wallis results are Claude H = 11.13, p = 0.133; GPT H = 8.93, p = 0.258; and Gemini H = 7.02, p = 0.427.

Overall, the manually coded paper attribute analyses do not show a stable advantage for Empirical papers or a consistent field-specific pattern across models. Apparent subgroup differences depend on the model, threshold, and treatment of coder disagreement; they should therefore be interpreted as descriptive heterogeneity rather than stable capability boundaries.

5.6. Confidence Alignment

To examine whether self-reported reviewer confidence is associated with human–AI agreement, the 2401 human reviews were grouped as high confidence (scores 4–5; N = 1358), medium confidence (score 3; N = 857), and low confidence (scores 1–2; N = 186). Agreement rates and Cohen’s kappa were then calculated within each confidence group. The results are reported in Table 14.

Table 14. Agreement between LLM decisions and human reviewers by confidence group (τ = 5).

Confidence Grouping	N	Claude Agreement Rate	Claude κ	GPT Agreement Rate	GPT κ	Gemini Agreement Rate	Gemini κ
High (4–5)	1358	0.638	0.225	0.660	0.279	0.613	0.165
Medium (3)	857	0.728	0.289	0.727	0.312	0.694	0.165
Low (1–2)	186	0.731	0.333	0.742	0.376	0.704	0.215

Note: Agreement is the proportion of matching binary decisions between an LLM and an individual human reviewer. Kappa is calculated at the reviewer level. A paper-clustered bootstrap with 5000 resamples indicates significant high- versus low-confidence differences at τ = 5 for all three models (p < 0.05).

Figure 8 shows the comparison of agreement rates between LLMs and humans across different confidence levels.

Figure 8. Agreement between LLM and human decisions across reviewer-confidence levels.

At τ = 5, agreement was higher in the low-confidence group than in the high-confidence group for all three models. For Claude, the agreement rate increased from 0.638 to 0.731 and kappa increased from 0.225 to 0.333 (Δ = 0.108). For GPT, the agreement rate increased from 0.660 to 0.742 and kappa from 0.279 to 0.376 (Δ = 0.097). For Gemini, the agreement rate increased from 0.613 to 0.704 and kappa from 0.165 to 0.215 (Δ = 0.050). Paper-clustered bootstrap tests with 5000 resamples indicated significant high–low differences for all three models (p < 0.05).

This association is descriptive and should not be interpreted as evidence that confidence identifies reviewer seniority or expertise. At τ = 6, the difference was attenuated for Claude (κ = 0.611 versus 0.571) but remained for GPT and Gemini, indicating that the pattern depends on both model and threshold choice.

5.7. Multi-Model and Human-Containing Aggregation Strategies

To compare decision agreement across aggregation strategies, pure-human, pure-AI, and human-containing rules were evaluated using voting, equal-weight means, and prespecified tie handling. These retrospective rules combine source-level scores and do not represent interactive human–AI collaboration. The results are shown in Table 15.

Table 15. Performance comparison of key integration strategies.

Integration Strategy	Threshold	Category	Accuracy	Cohen’s κ	Comparison with the Best Single Model
Human + Claude + GPT (Equal-Weight Mean)	τ = 6	Human-containing	0.970	0.940	+0.053
Human + Claude (Equal-Weight Mean)	τ = 6	Human-containing	0.967	0.933	+0.050
Human + GPT (Joint Acceptance; Tie Rejected)	τ = 5	Human-containing	0.872	0.743	+0.180
Human + Claude (Voting, Tie Rejected)	τ = 5	Human-containing	0.847	0.693	+0.155
Human Only	τ = 6	Human	0.923	0.847	—
Claude Only	τ = 6	AI	0.917	0.833	—
GPT Only	τ = 5	AI	0.692	0.383	—
Claude + GPT (Voting; Tie Rejected)	τ = 6	AI	0.932	0.863	—
Claude + GPT + Gemini (Majority Voting)	τ = 6	AI	0.793	0.587	—

Table 15 shows that the human-containing rules achieved the highest observed decision agreement. At τ = 6, the Human + Claude + GPT equal-weight mean achieved accuracy 0.970 and κ = 0.940, 0.053 above the best same-threshold single model, Claude (0.917; κ = 0.833). The Human + Claude equal-weight mean achieved accuracy 0.967 and κ = 0.933. At τ = 5, the Human + GPT joint-acceptance rule (two-source voting with ties rejected) achieved accuracy 0.872 and κ = 0.743, 0.180 above GPT alone; the corresponding Human + Claude rule also showed a positive difference. Pure-AI combinations remained below the best human-containing rules under the evaluated settings.

Paired comparisons confirmed that the four human-containing rules in Table 15 exceeded both the same-threshold human-only baseline and the best same-threshold single-LLM baseline. For the Human + GPT joint-acceptance rule at τ = 5, the accuracy difference was +0.180 versus GPT (95% CI: 0.148–0.213) and +0.052 versus Human (0.033–0.072). For the Human + Claude + GPT equal-weight mean at τ = 6, the corresponding differences were +0.053 versus Claude (0.035–0.073) and +0.047 versus Human (0.023–0.070). The Human + Claude mean and voting rules also retained positive paired differences against both baselines. All eight exact McNemar comparisons were significant at p < 0.001.

As shown in Table 16, the Shapley marginal contribution decomposition quantifies the differentiated contribution of each model within the evaluated ensembles. At τ = 5, GPT has the largest Shapley value (0.070), whereas at τ = 6, Claude has the largest value (0.200). Gemini contributes less under both thresholds. Figure 9 separately compares the best pure-AI and human-containing rules at the same number of review sources.

Table 16. Shapley’s marginal contribution decomposition.

Model	Shapley Value (τ = 5)	Shapley Value (τ = 6)	Role Positioning
Claude	0.033	0.200	Maximum contribution after calibration
GPT	0.070	0.094	Maximum contribution under standard threshold
Gemini	0.002	0.018	Limited marginal contribution

Note: The Shapley value measures the marginal contribution of each model to the overall accuracy of the ensemble, calculated based on an exhaustive enumeration of all possible subsets.

Figure 9. Accuracy differences between the best pure-AI and human-containing rules by number of review sources.

Figure 9 compares rules with the same number of review sources. At one source, the comparison is between the human-only rule and the best single-model aggregate; at two and three sources, it compares the best AI-only rule with the best human-containing aggregation rule. Here, a source is a paper-level mean and should not be interpreted as one statistically independent reviewer or as an equal information unit. The largest observed accuracy difference is 0.158 for the three-source rule at τ = 6.

In summary, the Human + GPT joint-acceptance rule achieved accuracy 0.872 at tau = 5, 18.0 percentage points above GPT alone; the Human + Claude + GPT mean-score rule achieved 0.970 at tau = 6, 5.3 percentage points above Claude alone. H4 is supported only in the limited sense of higher agreement with the conference decisions under the specified retrospective aggregation rules, not as independent evidence of superior review quality, interactive collaboration, or causal complementarity.

5.8. Cross-Year Robustness of the Prespecified Aggregation Rules

Section 5.7 reports the development-cohort results used to select the two aggregation configurations in ICLR 2025. The present section reports their locked application to the separate ICLR 2024 cohort; no rule, threshold, or tie-handling setting was changed after the 2024 outcomes were examined.

The two prespecified 2025 configurations were applied unchanged to the stratified balanced ICLR 2024 sample. The Human + GPT joint-acceptance rule achieved accuracy 0.873 and kappa 0.747, compared with 0.697 for GPT alone. The equal-weight Human + Claude + GPT mean-score rule achieved accuracy 0.973 and kappa 0.947, compared with 0.927 for Claude alone. Both gains retained the direction observed in 2025.

6. Conclusions and Prospect

6.1. Research Conclusions

This study constructed a parallel evaluation framework comprising 600 ICLR 2025 papers, 2401 human reviews, and 7203 model-specific generated reviews, and evaluated Claude, GPT, and Gemini across decision agreement, review-text characteristics, inter-model consistency, paper attribute sensitivity, and human–AI aggregation. Thresholds and prespecified aggregation rules developed in 2025 were additionally evaluated on a stratified balanced sample of 300 ICLR 2024 papers.

Scoring bias and threshold calibration (H1: Supported). All three LLMs exhibited leniency under tau = 5. Development-cohort thresholds of 6.2, 6.3, and 6.7 yielded accuracy 0.927, 0.913, and 0.932 for Claude, GPT, and Gemini. Repeated stratified cross-validation reproduced these thresholds, and locked-threshold accuracy in the ICLR 2024 sample was 0.927, 0.913, and 0.930. The preferred thresholds nevertheless varied under different prevalence and error-cost assumptions.

Inter-model convergence (H2: Supported). Pairwise Cohen’s kappa among the three LLMs ranged from 0.495 to 0.650, and Spearman correlations ranged from 0.705 to 0.799. These results support nontrivial convergence in binary decisions and score rankings without equating model-to-decision agreement with pairwise agreement among individual human reviewers.

Paper attribute sensitivity (H3: Not supported). Under the independent human coding, Hybrid papers showed the highest raw decision-agreement accuracy, rather than the predicted advantage for Empirical papers. After applying the locked thresholds, research-type differences were non-significant for all three models. Agreement-only sensitivity analyses produced the same conclusion, and primary-field patterns were not consistent across models or coding subsets.

Human-containing aggregation (H4: Supported in the specified sense). In 2025, the Human + GPT joint-acceptance rule achieved decision-agreement accuracy of 0.872 at tau = 5, whereas the Human + Claude + GPT mean-score rule achieved 0.970 at tau = 6. The same configurations achieved 0.873 and 0.973 in the 2024 cross-year robustness sample, preserving the direction of the observed differences. These results concern retrospective agreement with conference decisions and do not demonstrate interactive collaboration, independent review quality, or causal complementarity.

Review-text characteristics. LLM-generated reviews approximated human output in structural completeness, but critical-section elaboration remained uneven. Claude produced a longer Weaknesses section than the human average (1394 vs. 1240 characters), whereas GPT and Gemini produced substantially shorter critiques (686 and 555 characters). TF-IDF analysis also showed different lexical emphases: human reviews prioritized methodological novelty and comparative fairness, Claude emphasized theoretical rigor and baseline scrutiny, GPT focused on empirical verifiability, and Gemini used more general evaluative vocabulary.

6.2. Theoretical Contributions

The theoretical contributions of this study unfold across three interconnected dimensions: methodological, interpretive, and mechanism-level contributions. Together, these dimensions provide an integrated analytical framework for understanding LLM-based peer review.

At the methodological level, this study integrates decision alignment, descriptive review-text characteristics, inter-model consistency, and multi-source aggregation analysis within a common empirical framework. The threshold procedure separates score-scale calibration from paper ranking and locks settings estimated on ICLR 2025 before application to the stratified balanced ICLR 2024 cross-year robustness cohort.

At the interpretive level, the independently human-coded results show that apparent research-type differences are limited and model-specific under the standard threshold and become non-significant after calibration. This finding cautions against interpreting raw subgroup disparities as stable differences in model comprehension across research paradigms and redirects attention toward threshold choice, coding uncertainty, and measurement design.

At the mechanism level, the observed moderate convergence among the three LLMs indicates shared scoring tendencies at the paper level. Because the study does not identify the causal source of this convergence, it should not be attributed directly to common training data or interpreted as evidence that model agreement exceeds human agreement. The result instead motivates further investigation of when multi-model combinations provide complementary rather than redundant judgments.

6.3. Practical Implications

Based on the above empirical findings, this study offers the following practical recommendations for the academic publishing community:

Positioning LLMs as review assistants, not autonomous decision-makers. Although single-model agreement improves after threshold calibration, standalone LLM results remain below the strongest human-containing configurations, and persistent variation across repeated generations, uneven critical-section elaboration, and domain-sensitive judgment preclude delegating final evaluative authority to LLMs at the current stage of development.

Implementing model- and context-specific threshold calibration. The three models required different score cutoffs, and the cost-sensitive analysis showed that preferred thresholds changed with outcome prevalence and the relative costs of false acceptance and false rejection. Calibration should therefore use a designated development cohort, prespecified error trade-offs, and separate validation rather than a universal cutoff.

Evaluating a three-source human-containing aggregation rule. Within the present decision-alignment setting, the Human + Claude + GPT equal-weight mean achieved accuracy 0.970 and kappa 0.940. This result should be treated as evidence about a retrospectively evaluated aggregation rule rather than as a validated workflow or general optimum, because the conference decision is not an independent quality criterion and transfer beyond the two balanced ICLR cohorts remains untested.

Establishing quality audit mechanisms for LLM-generated reviews. The present review-text indicators describe output length, structure, critical-section length, and lexical emphasis, but they do not establish technical correctness, usefulness, or domain-specific methodological insight. Journals and conferences considering such tools should therefore institute systematic sampling, manual verification, and periodic quality audits of substantive review content. Such audits should remain under active editorial oversight, because technical interventions cannot by themselves ensure that reviews are rigorous, constructive, and professionally appropriate (Bustin, 2025).

6.4. Limitations

Several limitations warrant acknowledgment. The analysis covers two annual cohorts from a single machine-learning venue, which limits disciplinary generalizability, and the evaluated systems represent a time-specific snapshot of commercial models. The AI systems reviewed initial-submission PDFs, whereas human review texts and scores were collected from the publicly visible records after conference decisions and may include post-rebuttal updates; same-version equivalence is therefore not established. In addition, review length, structural completeness, critical-section length, and lexical features are descriptive indicators rather than validated measures of technical correctness, usefulness, methodological depth, error detection, or hallucinated criticism. No independent expert rating of review quality was conducted.

Because proprietary training and post-training corpora are not auditable, prior exposure to manuscripts, public reviews, or decisions cannot be excluded. The textual-overlap check detects direct lexical reuse but may miss paraphrasing or latent memorization; it therefore narrows but does not eliminate the contamination concern.

The publicly available ICLR 2024 review records are attributed to human reviewers, but prior research suggests that AI assistance may have been used in preparing some reviews from that conference. It cannot therefore be established with certainty that every review in the 2024 sample was produced entirely without AI assistance. This is treated as a possibility and a limitation rather than as an established property of the sample.

Thresholds were developed in ICLR 2025 and evaluated through repeated cross-validation and locked application to the stratified balanced ICLR 2024 sample. Both analytical samples contain equal numbers of accepted and rejected papers. The 30% prevalence and cost-sensitive scenarios demonstrate that predictive values, expected error counts, and preferred thresholds depend on deployment conditions.

Finally, human reviews contributed to the final ICLR decisions and also entered the human-containing aggregation rules. This criterion incorporation may increase apparent H4 gains. Moreover, the analyses combine source-level scores retrospectively and do not represent an interactive human–AI review process. The results therefore concern agreement with conference decisions under the specified rules and do not establish performance against an independent expert assessment, real-world workflow effectiveness, or causal complementarity.

6.5. Future Research Directions

Several avenues for future inquiry emerge from the present findings. First, cross-disciplinary replication studies are needed to determine the extent to which the calibration parameters and aggregation rules identified herein transfer to review contexts beyond machine learning, particularly to fields characterized by distinct epistemological norms, such as interpretive social sciences or clinical medicine. Second, as LLM architectures continue to evolve, longitudinal tracking of scoring bias trajectories across successive model generations would clarify whether leniency bias is a transient artifact of current training practices or a more persistent structural property. Third, the integration of interactive review protocols, wherein LLMs engage in multi-turn dialog with authors or other reviewers, represents a promising direction for enhancing the depth and contextual sensitivity of AI-generated feedback. Fourth, developing domain-adapted fine-tuning or retrieval-augmented generation strategies specifically optimized for peer review may address the current limitations in methodological critique and field-specific judgment. Fifth, the governance dimension warrants further attention: empirical research on disclosure mechanisms, confidentiality safeguards, and accountability frameworks is essential for informing the institutional policies that will shape the responsible integration of LLMs into scholarly evaluation systems. Finally, the multidimensional evaluation framework and empirical baselines established in this study provide a continuous comparative reference against which the review capabilities of future AI systems can be benchmarked, facilitating cumulative scientific understanding of the evolving relationship between human expertise and machine intelligence in academic gatekeeping.