Abstract
Background: Large language models (LLMs) have shown potential for clinical decision support. However, the diagnostic performance of newly released frontier models—including GPT-5 and Gemini 2.5 Pro—in gastroenterology remains insufficiently evaluated. This study evaluated the diagnostic performance and repeated-run consistency of four frontier LLMs—DeepSeek-V3.1, GPT-5, GPT-4.1, and Gemini 2.5 Pro—in challenging gastroenterology cases. Methods: This cross-sectional comparative study evaluated four LLMs (DeepSeek-V3.1, GPT-5, GPT-4.1, and Gemini 2.5 Pro) using 50 challenging gastroenterology cases retrieved from PubMed. Cases published after February 2025 were selected to minimize the potential risk of training-data contamination and prior model exposure. Each case was structured into a standardized prompt comprising clinical history, examination findings, laboratory results, and imaging data. Three independent runs were performed for each case. First-run outputs were used to assess ordinal primary diagnostic scores and differential-diagnosis quality, whereas all three runs were used to assess repeated-run consistency. Outputs were anonymized and scored by a gastroenterologist blinded to model identity. Results: Gemini 2.5 Pro achieved significantly higher ordinal primary diagnostic scores than the other three models (all Holm-adjusted pairwise p < 0.01). Exact-match rates were 46% for Gemini 2.5 Pro, 22% for GPT-5, and 18% each for GPT-4.1 and DeepSeek-V3.1. All four models had a median differential-diagnosis quality score of 4; the Friedman test did not detect a statistically significant difference among the models (p = 0.725). Complete agreement in the primary diagnosis across all three runs was observed in 68–78% of cases, depending on the model. No statistically significant differences in repeated-run consistency scores were detected among the models. Conclusions: In this investigator-selected benchmark of 50 published gastroenterology cases, Gemini 2.5 Pro achieved higher ordinal primary diagnostic scores than the other three models. No statistically significant differences were detected in differential-diagnosis quality or repeated-run consistency. These nonsignificant findings do not establish equivalence. Prospective evaluation in clinical settings is required before drawing conclusions about clinical effectiveness.
1. Introduction
Large language models (LLMs) are being investigated for diagnostic assistance, medical education, and clinical decision support in research and experimental clinical settings. These applications should be distinguished from routine clinical implementation, which requires validation of effectiveness and safety in the intended clinical setting [1,2,3]. Especially in the field of gastroenterology, the manifestations of diseases are complex and diverse, and the scope of differential diagnosis is extensive, which places extremely high requirements on doctors’ clinical reasoning abilities. Therefore, developing intelligent tools that can quickly integrate multi-source information and provide accurate differential diagnosis is of great significance for improving the efficiency and accuracy of diagnosis and treatment [4].
Recent research has explored LLM-based diagnostic support in gastroenterology and hepatology [5]. For example, a comparative study evaluated diagnostic suggestions generated by multiple LLMs and gastroenterologists using challenging cases with gastrointestinal symptoms [1]. Such diagnostic reasoning tasks involve generating and prioritizing possible diagnoses from clinical information and should be distinguished from NLP applications focused on extracting information from medical records.
Although existing studies have preliminarily verified the feasibility of LLMs in medical diagnosis, there are obvious limitations in the existing literature. Most studies use older model versions or lack direct comparisons of different LLMs under the same standard [6]. With the release of new-generation models such as GPT-5, Gemini 2.5 Pro, and DeepSeek-V3.1, their performance in complex specialized fields is still unclear. In particular, there is still a lack of systematic evaluation of the differences among these models in three key dimensions: the accuracy of primary diagnosis, the comprehensiveness of differential diagnosis, and the repeated-run consistency [7,8].
The relative diagnostic performance of these models in challenging gastroenterology cases cannot be reliably inferred from general descriptions of their capabilities and therefore warrants direct comparison using a common case set, prompt, and evaluation criteria. Accordingly, this study aimed to evaluate whether current frontier LLMs can provide accurate, comprehensive, and consistent AI-assisted diagnostic support in challenging gastroenterology cases. Four contemporary models—DeepSeek-V3.1, GPT-5, GPT-4.1, and Gemini 2.5 Pro—were assessed using a standardized set of 50 recently published clinical cases. The ordinal primary diagnostic score was the main comparative outcome. Differential-diagnosis quality and repeated-run consistency were assessed as additional outcomes. Exact-match accuracy was reported separately as a descriptive measure. By examining both diagnostic capability and response consistency, this study sought to characterize the potential and current limitations of frontier LLMs as adjunctive clinical decision-support tools in gastroenterology and to provide evidence relevant to their responsible translation into clinical practice.
2. Methods
We conducted this cross-sectional comparative study in October 2025. The study utilized published case reports and did not involve patient privacy data; therefore, ethical committee approval was not required. The study workflow is summarized in Figure 1.
Figure 1.
Study workflow. Fifty challenging gastroenterology cases published after February 2025 were selected from published case reports indexed in PubMed and structured into standardized clinical summaries (Step 1). Each case was independently processed by four LLMs (GPT-5, GPT-4.1, DeepSeek-V3.1, and Gemini 2.5 Pro) using standardized prompts with three runs per case; outputs were assigned fixed anonymized labels (Answer 1–4) to conceal model identities during evaluation. (Step 2). Blinded evaluation assessed three dimensions: ordinal primary diagnostic score (three-category scale scored 0, 1, and 2), differential diagnosis quality (5-point scale), and repeated-run consistency (0–2 ordinal scale) (Step 3).
2.1. Case Selection
To construct a diagnostic challenge benchmark, we selected published gastroenterology case reports according to the clinical criteria described below. The resulting sample was investigator-selected and was not intended to represent the case mix encountered in routine gastroenterology practice.
All cases were retrieved from authoritative academic databases. We searched PubMed between 11 September and 7 October 2025 for gastroenterology-related case reports published after February 2025, using the following search strategy: (“gastroenterology” OR “digestive system” OR “IBD” OR “Crohn” OR “ulcerative colitis” OR “celiac disease” OR “liver disease” OR “hepatitis” OR “pancreatitis” OR “GI bleeding”) AND (“case report” OR “case series” OR “clinical case” OR “diagnostic challenge”). The search expression was entered directly into PubMed without additional field restrictions. The target benchmark size of 50 cases was chosen with reference to the study sizes used in previous comparable LLM evaluation studies [9]. Two gastroenterology residents, who are the co-first authors of this study, screened the search results sequentially after sorting them using PubMed’s “Most Recent” option. The two residents jointly screened the reports and discussed uncertain cases before making the final selection. Screening stopped when 50 cases meeting the selection criteria had been identified. By that point, 1160 unique case reports had been examined. The case-selection process is summarized in Figure 2. The resulting benchmark was investigator-selected and was not intended to represent the case mix of routine gastroenterology practice.
Figure 2.
Case-selection flow diagram. Gastroenterology case reports published after February 2025 were retrieved from PubMed between 11 September and 7 October 2025, and screened sequentially using PubMed’s ‘Most Recent’ sort order. Screening stopped when 50 eligible cases had been selected, by which point 1160 reports had been examined. Report-level exclusion reasons were not systematically recorded; therefore, counts by exclusion reason are unavailable.
Priority was given to clinically challenging cases exhibiting multi-system involvement, divergent pathological manifestations, atypical presentations, or broad differential diagnostic requirements. Reports were excluded if the final primary diagnosis was not a gastroenterological disease, no definitive diagnosis was reported, the clinical information was insufficient, or diagnostic assessment depended extensively on images or imaging data that could not be adequately represented in the standardized text summaries. Other exclusion criteria included limited diagnostic possibilities, extremely rare conditions, management-focused rather than diagnostic-focused scenarios, and non-critical presentations. The judgments of “extremely rare conditions” and “non-critical presentations” were based on the screening physicians’ clinical judgment; no quantitative incidence or severity thresholds were specified. Report-level exclusion reasons were not systematically recorded, and counts by exclusion reason are therefore unavailable.
One gastroenterology resident extracted clinical information from each selected case report and prepared a standardized plain-text summary using a common template. The template covered the chief complaint and history of present illness, past medical history, key physical examination findings, laboratory results, imaging and endoscopy descriptions, and initial treatment and response. A second gastroenterology resident checked the extracted information and the resulting summaries against the original case reports. The final diagnosis reported in each source case report was used as the reference diagnosis for evaluation and was excluded from the model inputs.
2.2. LLMs and Prompts
The four models—GPT-5, GPT-4.1, DeepSeek-V3.1, and Gemini 2.5 Pro—were accessed through their respective web interfaces in October 2025. Reasoning mode was enabled where supported by the interface. Web browsing and search were disabled, memory and personalization features were disabled where available, and a separate new chat session was initiated for every run of each case. These are the testing conditions that the authors can confirm. Exact day-level access dates, backend model identifiers, detailed model-specific reasoning-mode labels, and numerical generation settings, such as temperature and top-p, were not systematically recorded and are unavailable. These undocumented details were not inferred from published API specifications or assumed default values.
We selected DeepSeek models for their open-source nature and widespread adoption; for comparison, we included widely recognized frontier proprietary ChatGPT and Gemini models. All models were accessed through their web-based interfaces.
All 50 clinical cases were presented to the four LLMs using standardized prompts to ensure consistency. The standardized prompt was developed with reference to prompting approaches described in previous studies [9]. No preliminary testing or comparison of alternative prompt variants was performed. The prompt was not modified based on the model outputs or diagnostic scores of the 50 evaluation cases. The prompt was as follows: “Act as an attending gastroenterologist. A summary of the patient’s clinical information will be presented, and you will use this information to predict the diagnosis. Describe the differential diagnoses and the rationale for each, listing the most likely diagnosis at the top: [case information].” Model outputs were assigned anonymized labels (Answer 1–4) in a fixed order by Jiaxuan Ji. The same model-to-label mapping was used across all cases and runs. The evaluator was not informed of the mapping between these labels and the model identities during scoring. Whether the evaluator inferred model identity from response style was not formally assessed.
2.3. Outcomes
The main comparative outcome was the ordinal primary diagnostic score, rated on a 0–2 scale. Additional outcomes were differential-diagnosis quality, rated on a 1–5 scale, and repeated-run consistency of the primary diagnosis, rated on a 0–2 scale. Exact-match accuracy was reported separately as a descriptive measure. Three independent runs were performed for each of the 50 cases with each model. Primary diagnostic performance and differential-diagnosis quality were evaluated using only the output from the first run. The second and third runs were used exclusively for the consistency analysis, which compared the primary diagnoses across all three runs. Thus, each model–case combination contributed one primary diagnostic score and one differential-diagnosis quality score to the corresponding performance analyses. Model outputs were evaluated by a single attending physician in the Department of Gastroenterology at a tertiary Grade A hospital, with more than 10 years of clinical experience. The evaluator was blinded to model identity. The three outcomes were assessed using investigator-defined ordinal scoring rubrics. The scoring rubrics were initially developed by the investigators with reference to similar published studies [9,10] and subsequently reviewed by a senior gastroenterologist before use. The review focused on the clarity and clinical relevance of the category definitions. The criteria distinguished the degree of agreement between the primary diagnosis and the reference diagnosis, the clinical relevance and completeness of differential-diagnosis lists, and agreement among primary diagnoses across three runs. No pilot assessment or formal validation of these rubrics was conducted. Differential-diagnosis quality was rated by a single evaluator, and inter-rater agreement could not be assessed.
- Ordinal Primary Diagnostic Score: Each LLM-generated primary diagnosis was compared with the reference diagnosis reported in the source case report using a three-category ordinal scoring rubric: 0 points, unrelated to the reference diagnosis; 1 point, representing a superordinate concept or core component of the reference diagnosis; and 2 points, an exact match. Exact-match accuracy was calculated separately as the proportion of first-run primary diagnoses assigned a score of 2 and was reported descriptively. Exact matching was based on diagnostic meaning rather than identical wording; synonymous terms referring to the same diagnosis were accepted.
- Quality of Differential Diagnosis: The first-run differential-diagnosis lists were assessed by the same blinded physician using an investigator-defined five-category ordinal scoring rubric scored 1–5. These scores represent the evaluator’s assessment of the clinical relevance, coverage, and explanatory detail of the generated lists. The specific scoring criteria were as follows: 1 point—differential diagnosis entirely irrelevant; 2 points—incomplete differential diagnosis or containing obvious errors; 3 points—basically reasonable differential diagnosis but with omissions; 4 points—reasonable differential diagnosis covering major possibilities; 5 points—comprehensive differential diagnosis with clear logic and inclusion of key differential points.
- Repeated-Run Consistency: For each model–case combination, the primary diagnoses generated across all three runs were compared, yielding a single consistency score. Scores were assigned as follows: 0 points, all three primary diagnoses differed; 1 point, exactly two primary diagnoses matched; and 2 points, all three primary diagnoses matched. This outcome measured agreement in the primary diagnosis across repeated runs. Agreement across repeated runs does not indicate diagnostic correctness, because a model may repeatedly generate the same incorrect diagnosis.
2.4. Statistical Analysis
Because the same 50 cases were assessed by all four models, comparisons between models accounted for pairing at the case level. The main comparative analysis used the ordinal primary diagnostic score (0–2). Scores were summarized by category frequencies and percentages, medians and interquartile ranges (IQRs); means were also reported descriptively. The Friedman test was used to compare the four models, with case identity as the blocking factor. Kendall’s W was calculated as an effect size. Following a significant overall test, the six pairwise comparisons were performed using paired Wilcoxon signed-rank tests, with Holm adjustment for multiple comparisons. Normal-approximation p-values with continuity correction were used for these comparisons. Exact-match accuracy was defined as the proportion of first-run primary diagnoses assigned a score of 2 and was summarized descriptively as the number of exact matches out of 50 and the corresponding percentage. No separate hypothesis test was performed for this measure. Wilson 95% confidence intervals (CIs) were calculated for first-run exact-match accuracy and complete three-run agreement proportions. These intervals describe uncertainty conditional on the selected benchmark and do not address case-selection bias.
Differential-diagnosis quality and repeated-run consistency scores were also compared across models using Friedman tests with case identity as the blocking factor. Their score distributions, medians, and IQRs were reported descriptively. Pairwise comparisons were performed only when the corresponding overall test was significant.
All statistical analyses were performed using R software (version 4.4.3), with the significance level set at p < 0.05.
3. Results
3.1. Ordinal Primary Diagnostic Score
The first-run primary diagnoses generated by the four LLMs across 50 cases were compared against reference standards and scored using a three-category ordinal scoring rubric with scores of 0, 1, and 2 (0: no match; 1: partial match; 2: exact match). Table 1 summarizes the ordinal primary diagnostic score distributions and first-run exact-match accuracy for each model.
Table 1.
Distribution of Ordinal Primary Diagnostic Scores and First-Run Exact-Match Accuracy *.
As shown in Table 1, the mean ordinal primary diagnostic scores were 0.88 for GPT-5, 0.88 for GPT-4.1, 0.86 for DeepSeek-V3.1, and 1.28 for Gemini 2.5 Pro. These means are reported descriptively; the paired statistical comparisons are presented below.
To compare ordinal primary diagnostic scores (0–2) among the four LLMs across the same 50 cases, we used the Friedman test, followed by paired Wilcoxon signed-rank tests with Holm adjustment for multiple comparisons.
The primary diagnostic score distributions and pairwise comparisons are presented in Figure 3. The Friedman test showed a significant difference in ordinal primary diagnostic scores among the four models across the same 50 cases (χ2(3) = 23.45, p < 0.001; Kendall’s W = 0.156). Paired Wilcoxon signed-rank tests with Holm adjustment showed that Gemini 2.5 Pro had higher scores than GPT-5 (adjusted p = 0.0063), GPT-4.1 (adjusted p = 0.0063), and DeepSeek-V3.1 (adjusted p = 0.0011). Scores did not differ significantly among GPT-5, GPT-4.1, and DeepSeek-V3.1 (all adjusted p > 0.05). The median score (25th–75th percentile) was 1 (1–2) for Gemini 2.5 Pro and 1 (0–1) for each of the other three models. First-run exact-match accuracy was 46% (23/50) for Gemini 2.5 Pro, 22% (11/50) for GPT-5, and 18% (9/50) each for GPT-4.1 and DeepSeek-V3.1. The corresponding Wilson 95% CIs are reported in Table 1.
Figure 3.
Primary diagnostic assessment. (A) Comparison of ordinal primary diagnostic scores across LLMs: Stacked bar chart showing the distribution of primary diagnosis scores across four LLMs (GPT-5, GPT-4.1, DeepSeek-V3.1, Gemini 2.5 Pro) based on 50 complex gastroenterology cases. Scoring criteria: 0 = Irrelevant, 1 = Related concept, 2 = Exact match. (B) Box plots and jittered points show the distribution of primary diagnostic scores; red diamonds indicate median scores. The Friedman test showed an overall difference among models (χ2(3) = 23.45, p < 0.001). Brackets show significant paired Wilcoxon signed-rank comparisons, with Holm-adjusted p-values: Gemini 2.5 Pro versus GPT-5 (p = 0.0063), GPT-4.1 (p = 0.0063), and DeepSeek-V3.1 (p = 0.0011). No other pairwise comparisons were significant.
3.2. Differential-Diagnosis Quality
For the differential-diagnosis quality scores assigned by the single blinded evaluator, the Friedman test did not detect a statistically significant difference among the four models (χ2(3) = 1.32, p = 0.725; Kendall’s W = 0.009). As shown in Table 2 and Figure 4, all four models had a median score of 4. The 25th–75th percentile range was 3–5 for GPT-5 and Gemini 2.5 Pro and 3–4 for GPT-4.1 and DeepSeek-V3.1. Mean ± SD scores were 4.02 ± 0.82 for GPT-5, 3.94 ± 0.74 for GPT-4.1, 3.88 ± 0.98 for Gemini 2.5 Pro, and 3.86 ± 0.83 for DeepSeek-V3.1. These numerical differences are reported descriptively and do not establish superiority of any model in differential-diagnosis quality.
Table 2.
Quality assessment of differential diagnosis.
Figure 4.
Differential diagnostic assessment. (A) Violin plots show the differential diagnosis scores of each LLM. The width of the violin represents the distribution density, the box plot shows the interquartile range, and the white dot represents the median. (B) Comparison of differential-diagnosis quality scores across LLMs: Stacked bar chart showing the distribution of differential-diagnosis scores across four LLMs (GPT-5, GPT-4.1, DeepSeek-V3.1, Gemini 2.5 Pro) based on 50 complex gastroenterology cases. Scoring criteria: 1 = Completely Irrelevant, 2 = Incomplete or Incorrect, 3 = Adequate but Incomplete, 4 = Reasonable differential diagnosis covering major possibilities; 5 = Comprehensive differential diagnosis with clear logic and inclusion of key differential points.
Regarding score distributions (Figure 4B), GPT-5 and Gemini 2.5 Pro achieved the highest proportions of top scores (5 points) at 34% each, whereas GPT-4.1 demonstrated the highest proportion of “reasonable differential diagnosis covering major possibilities” (4 points) at 46%. DeepSeek-V3.1 and Gemini 2.5 Pro received a score of 2 (“incomplete differential diagnosis or containing obvious errors”) in 4% and 8% of cases, respectively; no responses from GPT-5 or GPT-4.1 received this score. The proportions of responses receiving scores of 4 or 5 were 68% for GPT-5, 70% for GPT-4.1, 66% for DeepSeek-V3.1, and 62% for Gemini 2.5 Pro. The Friedman test showed no statistically significant difference in differential-diagnosis scores among the four models across the same 50 cases (χ2(3) = 1.32, p = 0.725; Kendall’s W = 0.009). Therefore, no post hoc pairwise comparisons were performed.
3.3. Repeated-Run Consistency of Primary Diagnoses
We assessed repeated-run consistency by comparing the primary diagnoses generated across three runs for each model–case combination. The distributions of consistency scores and complete agreement rates are presented in Table 3 and Figure 5.
Table 3.
Repeated-run consistency of primary diagnoses *.
Figure 5.
Repeated-run consistency of primary diagnoses. Comparison of repeated-run consistency of primary diagnoses across LLMs: Stacked bar chart showing the distribution of repeated-run consistency scores across four LLMs (GPT-5, GPT-4.1, DeepSeek-V3.1, Gemini 2.5 Pro) based on 50 complex gastroenterology cases.
The Friedman test did not detect a statistically significant difference in repeated-run consistency scores among the four models (χ2(3) = 3.14, p = 0.37).
All models had a median repeated-run consistency score of 2.0. Complete agreement in the primary diagnosis across all three runs was observed in 70% of cases for GPT-5, 68% for GPT-4.1, 74% for DeepSeek-V3.1, and 78% for Gemini 2.5 Pro. The corresponding Wilson 95% CIs are reported in Table 3. A consistency score of 0, indicating that all three primary diagnoses differed, was observed in 4% of cases for each of GPT-5, GPT-4.1, and DeepSeek-V3.1, and in no cases for Gemini 2.5 Pro.
4. Discussion
This study systematically compared the performance of four large language models in differential diagnosis of complex gastroenterological diseases. Gemini 2.5 Pro achieved significantly higher ordinal primary diagnostic scores than the other three models, whereas no statistically significant differences were detected among the models in differential-diagnosis quality or repeated-run consistency. These findings provide a comparative assessment of diagnostic performance and repeated-run consistency across the four models within the evaluated 50-case benchmark.
Gemini 2.5 Pro had the highest observed exact-match rate (46%), compared with 22% for GPT-5 and 18% each for GPT-4.1 and DeepSeek-V3.1. These rates were reported descriptively; the statistically significant differences reported above concern ordinal primary diagnostic scores. Even the highest observed first-run exact-match rate was below 50% in this selected challenging case set, indicating that higher ordinal scores should not be equated with sufficiently accurate standalone diagnosis. A meta-analysis of 50 studies reported pooled relative diagnostic accuracies of 0.89 (95% CI, 0.79–1.00) for top-1, 1.04 (95% CI, 0.89–1.22) for top-5, and 1.17 (95% CI, 0.87–1.57) for top-10 diagnoses when comparing LLMs with healthcare professionals [11]. The top-5 and top-10 comparisons did not show statistically significant differences. These findings do not establish superiority or equivalence, and substantial heterogeneity limits broad conclusions across models and clinical settings.
Cross-study differences in model performance require cautious interpretation. In a study using script concordance testing (SCT) to assess clinical reasoning under uncertainty, OpenAI o3 achieved a mean score of 67.8%, compared with 52.1% for Gemini 2.5 Pro Preview [12]. SCT assesses how additional information changes diagnostic or therapeutic judgments, whereas our study evaluated diagnoses generated from standardized case summaries. Differences in model versions, clinical domains, prompts, and evaluation methods prevent attributing the contrasting findings to task type alone or establishing a stable ranking of model families.
All four models achieved a median differential-diagnosis score of 4, indicating that responses were generally rated as covering the major diagnostic possibilities in the evaluated cases. In contrast to the significant differences in ordinal primary diagnostic scores, no statistically significant differences were detected in differential-diagnosis quality scores. A pediatric radiology study also reported no statistically significant differences in top-3 differential-diagnosis accuracy among the evaluated models under text-only conditions; however, its outcome measure and model versions differed from those used here [13].
The absence of a statistically significant difference does not establish clinical equivalence. GPT-5 and Gemini 2.5 Pro achieved the highest proportions of perfect scores (5 points) at 34% each, whereas GPT-4.1 showed the highest proportion in the “reasonable differential diagnosis covering major possibilities” (4-point) category at 46%. These differences in the proportions of 4- and 5-point ratings are descriptive and do not establish differences in reasoning depth. Whether these descriptive differences translate into improvements in clinical decision-making requires further investigation.
In a benchmark of 125 patient cases, Sandmann et al. reported higher diagnostic ratings for DeepSeek-R1 and GPT-4o than for Gemini 2.0 Flash Thinking Experimental [7]. These findings concern different model versions and evaluation conditions from those used in the present study. Differences in clinical domain, prompt design, input modality, and scoring criteria limit direct comparisons across studies. These comparisons provide context but do not establish consistent strengths of particular model families or reveal the mechanisms underlying their performance.
For clinical workflows, the AMIE study found that clinicians using model assistance generated more comprehensive differential-diagnosis lists than those using standard resources alone in an evaluation based on published case reports [14]. Our study evaluated model-generated outputs but did not test their use by clinicians. The observed scores therefore do not establish benefits for clinical decision-making or patient outcomes.
Complete agreement in the primary diagnosis across all three runs was observed in 68–78% of cases, depending on the model. These results describe repeated-run consistency in primary diagnostic outputs under the evaluated conditions. Agreement across repeated runs does not establish diagnostic correctness or clinical safety. The “hallucination” problem in medical LLMs—generating seemingly plausible reasoning lacking clinical evidence—remains a significant concern. A study on medical image multiple-choice questions found that even when predictions were correct, models often relied on “superficial patterns or hallucinated logic,” with substantial disconnect between answer selection and clinical reasoning [15]. More concerning is the implicit risk of temporally sensitive knowledge: LLMs have inherent knowledge cutoff limitations, and when handling cases involving the latest clinical guidelines, drug indications, or epidemiological data, models may unconsciously cite outdated information [16]. For instance, when queried about blood lead level intervention thresholds, Llama 3.1 cited obsolete OSHA standards rather than the updated EU biological limit values from February 2024, displaying no uncertainty [17]. In gastroenterology, this risk is particularly pronounced: Helicobacter pylori eradication regimens, biologic indications for inflammatory bowel disease, and hepatocellular carcinoma immunotherapy combinations are all rapidly evolving. Our use of cases published after February 2025 aimed to reduce the possibility of prior model exposure, although such exposure could not be excluded.
The present study evaluated diagnostic outputs generated from standardized text-based case summaries. It did not assess multimodal inputs, explainability methods, structured model debates, or hallucination-mitigation strategies, and therefore provides no evidence regarding their effects. Clinical implications should remain limited to the observed text-based diagnostic performance and its measurement limitations. Prospective evaluation using representative cases and clinically relevant outcomes is needed before conclusions can be drawn about clinical effectiveness or safety.
This study has several limitations. First, the study was based on published text-based case reports, lacking multidimensional information from real-world clinical environments, such as physician-patient communication and dynamic vital signs—consistent with conclusions from recent research indicating that “LLM performance declines significantly in dynamic consultation scenarios” [18]. Our static text paradigm may overestimate models’ actual clinical capabilities, as real-world diagnosis involves progressive information disclosure, follow-up history-taking, and observation of treatment responses, whereas model performance in dynamic uncertainty reasoning may be inferior to that demonstrated in this study [11].
Second, cases were deliberately selected for diagnostic complexity, atypical presentations, and broad differential-diagnosis requirements. This selection produced a diagnostic challenge benchmark that does not represent the case mix encountered in routine gastroenterology practice. Screening followed PubMed’s “Most Recent” order and stopped once 50 eligible cases had been selected; therefore, other eligible reports may have been missed. Some exclusion criteria relied on clinical judgment without quantitative thresholds, and the absence of report-level exclusion records limits the reproducibility and auditability of case selection. Selection may influence both the observed scores and relative model performance, but the direction and magnitude of this bias cannot be determined from the present study.
Third, this study employed text-only cases, whereas real-world gastroenterology practice heavily relies on multimodal data integration. Recent studies demonstrate that current vision-language models (VLMs) exhibit a “visual paradox” in medical image processing—diagnostic accuracy declined from 80% to 58% when endoscopic or pathological images were incorporated [4]. Because the present study used text-only inputs, it cannot assess multimodal diagnostic performance or determine the mechanisms underlying the performance differences reported in previous studies.
Additional limitations include the small, selected sample of 50 cases and evaluation by a single physician. Several differential-diagnosis scoring categories require subjective clinical judgment, and the ratings may therefore be influenced by the evaluator’s interpretation. Blinding to model identity does not eliminate this potential bias, and inter-rater agreement could not be assessed. Accordingly, the differential-diagnosis quality scores reflect one evaluator’s judgments and should not be interpreted as independently validated measurements of diagnostic quality. The small sample size and coarse ordinal scales may have limited the ability to detect meaningful between-model differences in differential-diagnosis quality and repeated-run consistency. Outputs were anonymized using a fixed model-to-label mapping rather than randomized labels. This procedure concealed the declared model identities but could not exclude inference from recurring response styles or potential label-related effects. Future studies should involve multiple independent gastroenterologists and quantify agreement between their ratings. Primary diagnostic performance and differential-diagnosis quality were assessed using only first-run outputs, so these results do not characterize variation in diagnostic performance across repeated runs. Only one standardized prompt was evaluated, and the sensitivity of model performance to alternative prompt formulations was not assessed. Taken together, investigator-selected cases, subjective ratings from a single evaluator, and use of a single prompt may influence relative model rankings, but the direction and magnitude of their combined effects cannot be determined from the present data. The findings are specific to the evaluated model versions and testing conditions in October 2025 and may not generalize to subsequent model updates or different interface settings. Although the available interface settings are reported, the absence of exact day-level access dates, backend model identifiers, detailed model-specific reasoning-mode labels, and generation-parameter records prevents exact replication of the original testing conditions. The findings should therefore be interpreted as specific to the tested web-interface conditions, with limited reproducibility under equivalent conditions. The absence of clinician comparators and clinical outcome measures also limits conclusions about clinical usefulness.
5. Conclusions
In an investigator-selected diagnostic challenge benchmark comprising 50 published gastroenterology case reports, Gemini 2.5 Pro achieved higher ordinal primary diagnostic scores than GPT-5, GPT-4.1, and DeepSeek-V3.1. Exact-match accuracy was reported descriptively. No statistically significant differences were detected in differential-diagnosis quality or agreement among primary diagnoses across three runs; these findings do not establish equivalence. The results characterize performance on selected published cases rather than diagnostic accuracy in prospective patient encounters. Prospective evaluation using representative clinical cases, multiple independent evaluators, and direct clinician comparisons is needed to assess clinical usefulness.
Author Contributions
J.J.: Writing—review & editing, Writing—original draft, Visualization, Methodology, Data curation, Conceptualization. H.Z.: Writing—review & editing, Writing—original draft, Methodology, Data curation. W.L.: Writing—review & editing, Validation, Supervision. J.Z.: Writing—review & editing, Supervision, Resources, Project administration. X.C.: Writing—review & editing, Supervision, Project administration, Methodology, Conceptualization. X.F.: Writing—review & editing, Supervision, Resources, Conceptualization. All authors have read and agreed to the published version of the manuscript.
Funding
This work was supported by the Tianjin Key Medical Discipline (Specialty) Construction Project (grant number TJYXZDXK-002A; project numbers 2023074 and 2023075), the Natural Science Foundation of Tianjin (grant number 24JCYBJC00340), and the Tianjin Special Fund for Guiding Technological Innovation (grant number 25YDTPJC00160).
Institutional Review Board Statement
Ethics committee approval was not required for this study, which used information exclusively from published case reports and involved no patient recruitment, clinical interventions, or collection of identifiable private patient data.
Informed Consent Statement
Not applicable. This study used information from previously published case reports and did not involve direct recruitment of or interaction with patients.
Data Availability Statement
The selected case list, standardized case summaries, model-generated outputs, case-level diagnostic ratings, and R scripts used for the statistical analyses are available from the corresponding author upon reasonable request. The original case reports are available from their respective publishers, subject to applicable access restrictions.
Acknowledgments
During the preparation of this work, the authors used AI-based language tools to assist with translation and language refinement. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article.
Conflicts of Interest
The authors declare no conflicts of interest.
References
- Yang, X.; Li, T.; Wang, H.; Zhang, R.; Ni, Z.; Liu, N.; Zhai, H.; Zhao, J.; Meng, F.; Zhou, Z.; et al. Multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal symptoms. npj Digit. Med. 2025, 8, 85. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Zhang, R.-Y.; Qiang, P.-P.; Hao, Y.-X.; Tan, H.-Y.; Zhao, K.; Cai, L.-J.; Wang, J.-P. GutGPT: A multidimensional knowledge-enhanced large language model for gastrointestinal medicine. J. Biomed. Inf. 2025, 169, 104885. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Gravina, A.G.; Pellegrino, R.; Palladino, G.; Imperio, G.; Ventura, A.; Federico, A. Charting new AI education in gastroenterology: Cross-sectional evaluation of ChatGPT and perplexity AI in medical residency exam. Dig. Liver Dis. 2024, 56, 1304–1311. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Safavi-Naini, S.A.A.; Ali, S.; Shahab, O.; Shahhoseini, Z.; Savage, T.; Rafiee, S.; Samaan, J.S.; Al Shabeeb, R.; Ladak, F.; Yang, J.O.; et al. Benchmarking proprietary and open-source language and vision-language models for gastroenterology clinical reasoning. npj Digit. Med. 2025, 8, 797. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Omar, M.; Nassar, S.; SharIf, K.; Glicksberg, B.S.; Nadkarni, G.N.; Klang, E. Emerging applications of NLP and large language models in gastroenterology and hepatology: A systematic review. Front. Med. 2024, 11, 1512824. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Lee, S.; Jung, S.; Park, J.-H.; Cho, H.; Moon, S.; Ahn, S. Performance of ChatGPT, gemini and DeepSeek for non-critical triage support using real-world conversations in emergency department. BMC Emerg. Med. 2025, 25, 176. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Sandmann, S.; Hegselmann, S.; Fujarski, M.; Bickmann, L.; Wild, B.; Eils, R.; Varghese, J. Benchmark evaluation of DeepSeek large language models in clinical decision-making. Nat. Med. 2025, 31, 2546–2549. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Miao, B.; Sun, Q.; Wang, P.; Shao, R.; Ding, Y.; Chen, Y.; Ying, R. Exploring the use of large language models for classification, clinical interpretation, and treatment recommendation in breast tumor patient records. Sci. Rep. 2025, 15, 31450. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Wu, X.; Huang, Y.; He, Q. Diagnostic performance of newly developed large language models in critical illness cases: A comparative study. Int. J. Med. Inf. 2025, 204, 106088. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Patel, A.; Ruoff, C.; Helgeson, S.A.; Carvalho, D.Z.; Castillo, P.R.; Cheung, J. Diagnostic performance of large language models (LLMs) compared with physicians in sleep medicine. Sleep Med. 2025, 134, 106677. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Chen, M.; Wu, Y.; Ma, J.; Jia, X.; Gao, C.; Zhao, F.; Qiao, Y. Independent and collaborative performance of large language models and healthcare professionals in diagnosis and triage. npj Digit. Med. 2026, 9, 222. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- McCoy, L.G.; Swamy, R.; Sagar, N.; Wang, M.; Bacchi, S.; Fong, J.M.N.; Tan, N.C.; Tan, K.; Buckley, T.A.; Brodeur, P.; et al. Assessment of large language models in clinical reasoning: A novel benchmarking study. NEJM AI 2025, 2, AIdbp2500120. [Google Scholar] [CrossRef] [Scilit]
- Jung, J.; Phillipi, M.; Tran, B.; Chen, K.; Chan, N.; Ho, E.; Sun, S.; Houshyar, R. Accuracy of large language models in generating differential diagnosis from clinical presentation and imaging findings in pediatric cases. Pediatr. Radiol. 2025, 55, 1927–1933. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- McDuff, D.; Schaekermann, M.; Tu, T.; Palepu, A.; Wang, A.; Garrison, J.; Singhal, K.; Sharma, Y.; Azizi, S.; Kulkarni, K.; et al. Towards accurate differential diagnosis with large language models. Nature 2025, 642, 451–457. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Hazra, D.; Mukherjee, S.; Kumar, S.; Chatterjee, S.; Khan, P.W.; Abbas, K. Evaluating hallucination and diagnostic reliability of LLMs on medical image-based multiple choice tasks. IEEE J. Biomed. Health Inform. 2025. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Zhang, Y.; Yang, Y.; Shu, J.; Wen, X.; Sang, J. Agent models: Internalizing chain-of-action generation into reasoning models. arXiv 2025. [Google Scholar] [CrossRef] [Scilit]
- Pęzik, P.; Kaczyński, K.; Szymańska, M.; Żarnecki, F.; Deckert, Z.; Kwiatkowski, K.; Janowski, W. LLMLagBench: Identifying temporal training boundaries in large language models. arXiv 2025. [Google Scholar] [CrossRef] [Scilit]
- Qiu, P.; Wu, C.; Liu, S.; Fan, Y.; Zhao, W.; Chen, Z.; Gu, H.; Peng, C.; Zhang, Y.; Wang, Y.; et al. Quantifying the reasoning abilities of LLMs on clinical cases. Nat. Commun. 2025, 16, 9799. [Google Scholar] [CrossRef] [Scilit] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.