Reliability and Performance Stability of Large Language Models in Medical Knowledge Assessment: Evidence from the European Board of Nuclear Medicine Examination

Abstract

Background: Large language models (LLMs) have demonstrated strong performance on general medical examinations. Whether this performance translates to highly specialized, subspecialty-level board examinations remains unclear. This study evaluates the accuracy and inter-run stability of contemporary LLMs using authentic European Board of Nuclear Medicine (EBNM) Fellowship Examination material. Methods: Ten LLMs (five proprietary, five open-source) completed 50 EBNM multiple-choice questions across five independent zero-shot runs, resulting in 2500 total inferences. Accuracy was calculated per model across runs. Inter-run reliability was assessed using pairwise Cohen’s kappa coefficients. Pairwise model differences were analyzed using McNemar’s test with Bonferroni correction (α = 0.0011). Results: Mean accuracy ranged from 53.6% to 100.0%, with all models exceeding an illustrative 50% pass threshold. Inter-run reliability varied substantially (κ = 0.370–1.000; mean κ = 0.716). High accuracy did not consistently correspond to high reproducibility. Gemini 2.5 Pro achieved high accuracy (93.6%) but showed the lowest reliability (κ = 0.370), whereas DeepSeek V3.2 demonstrated perfect accuracy and agreement across all runs. No significant correlation between accuracy and reliability was observed (Spearman ρ = 0.394, p = 0.26). Conclusions: LLMs demonstrate strong but heterogeneous performance on high-stakes medical knowledge assessments. Differences in reproducibility highlight the need for multi-run evaluation when considering LLMs for educational or clinical knowledge-support applications and for continued validation using non-disclosed examination material.

1. Introduction

Large language models (LLMs) are increasingly used in medical education and assessment, with growing evidence of strong performance across clinical domains [1,2]. Recent evaluations have shown that advanced AI systems can achieve passing scores on standardized medical licensing examinations, including the United States Medical Licensing Examination (USMLE) and various specialty certification tests [3,4]. These findings have prompted widespread interest in understanding AI performance boundaries across increasingly specialized medical fields. Nuclear medicine represents a particularly challenging domain for AI evaluation, integrating complex physics principles, radiopharmaceutical chemistry, and subspecialty clinical expertise spanning oncology, cardiology, neurology, and therapeutic applications [5]. The multimodal nature of nuclear medicine knowledge, requiring simultaneous understanding of technical instrumentation, tracer kinetics, and clinical interpretation, provides a demanding test of AI medical knowledge.

The European Board of Nuclear Medicine (EBNM), established in 1993 under the European Union of Medical Specialists (UEMS), introduced the Fellowship Examination in 1996 to certify nuclear medicine specialists according to European standards [6]. The examination comprises 140 multiple-choice questions covering sixteen curriculum domains, including oncology, cardiology, endocrinology, physics and quality control, neurology, therapeutic procedures, radiopharmacy, and nephro-urology [6,7]. The two-step certification process, written examination followed by oral assessment, has awarded the Fellow of the EBNM (FEBNM) title to hundreds of nuclear medicine physicians from countries worldwide, with the written examination part transitioning to an online format in 2019 [6]. The EBNM Fellowship Examination thus provides an established, internationally recognized benchmark for evaluating nuclear medicine competence, making it an ideal instrument for assessing AI capabilities in this specialized domain.

Despite growing interest in AI performance on medical examinations, nuclear medicine remains notably underexplored in the AI evaluation literature. Previous studies have predominantly focused on general medical licensing examinations and broader specialty assessments, with limited attention to imaging subspecialties requiring integrated technical and clinical knowledge [8,9]. Furthermore, most existing evaluations employ single-run methodologies that cannot assess response consistency, a critical consideration for potential clinical deployment where reproducible outputs are essential [10]. The newest generation of LLMs, released throughout 2024–2025, has not been systematically evaluated in nuclear medicine examination contexts. Additionally, comparative analyses between proprietary commercial models and increasingly capable open-source alternatives remain sparse, despite significant implications for accessibility, cost, and deployment in resource-limited healthcare settings.

This study had four objectives:

(1)

Evaluate the performance of ten state-of-the-art LLMs on EBNM Fellowship Examination questions, including five proprietary and five open-source systems.

(2)

Assess inter-run reliability using Cohen’s kappa coefficient across five independent evaluations per model, establishing the first reliability benchmarks for AI in nuclear medicine examinations.

(3)

Identify statistically significant performance differences through rigorous pairwise comparisons with Bonferroni correction for multiple testing.

(4)

Compare proprietary versus open-source model capabilities to inform accessibility considerations for educational and clinical applications. We hypothesized that leading models would approach human first-time candidate performance levels while demonstrating high inter-run consistency, with significant performance stratification emerging between model tiers.

2. Materials and Methods

2.1. Study Design

We conducted a multi-run evaluation study of ten state-of-the-art LLMs on 50 publicly available questions from the MCQ paper of 2010 European Board of Nuclear Medicine (EBNM) Fellowship Examination [11]. Each model completed five independent runs on 28–29 October 2025, yielding 2500 total observations (50 questions × 10 models × 5 runs). The study employed a zero-shot evaluation approach without training examples or feedback.

2.2. AI Models

Ten contemporary LLMs were evaluated, comprising five proprietary and five open-source systems:

Proprietary models: Claude Sonnet 4.5 (Anthropic, San Francisco, CA, USA), Gemini 2.5 Pro (Google LLC, Mountain View, CA, USA), GPT-5 Pro (OpenAI, San Francisco, CA, USA), Grok-4 (xAI, San Francisco, CA, USA), and ERNIE 4.5 Turbo (Baidu, Beijing, China).

Open-source models: DeepSeek V3.2 (DeepSeek, Hangzhou, China), Falcon H1-34B (Technology Innovation Institute, Abu Dhabi, United Arab Emirates), Llama 3.3 70B (Meta AI, CA, USA), Mistral Medium 3.1 (Mistral AI, Paris, France), and Qwen3 Max (Alibaba Cloud, Hangzhou, China).

For category balancing, Mistral Medium 3.1 was assigned to the open-source group because it represents the developers openly released model family and maintains architectural continuity with prior open-weight models.

Model inclusion and exclusion criteria were defined a priori. LLMs were included if they were accessible via a browser-based interface at the time of evaluation, capable of answering English-language multiple-choice medical examination questions. The systems selected for evaluation represented the most recent or manufacturer-designated flagship versions available at the time of study.

LLMs were excluded if custom fine-tuning, local deployment, or API-only access was required, if non-default parameter adjustments were necessary, or if technical or usage restrictions prevented repeated independent evaluations under identical conditions. The use of browser-based interfaces reflects real-world user-facing deployment conditions rather than optimized laboratory settings.

Evaluation was conducted during October 2025 via the respective web interfaces, and all assessments were performed under identical conditions using the same set of 50 publicly available EBNM examination questions.

2.3. Examination

The evaluation used all 50 available multiple-choice questions in EBNM Fellowship Examination format; no questions were excluded. Each question presented five answer options (A through E) with exactly one correct answer. The Questions covered 15 of 16 clinical domains as specified in the UEMS/EBNM training requirements [6], with the following distribution: oncology (n = 6, 12%), endocrinology (n = 5, 10%), cardiology (n = 5, 10%), therapy (n = 4, 8%), skeletal imaging (n = 4, 8%), radiopharmacy (n = 4, 8%), physics and quality control (n = 4, 8%), neurology (n = 4, 8%), nephro-urology (n = 4, 8%), infection and inflammation (n = 3, 6%), gastroenterology (n = 2, 4%), and radiobiology, radiation protection, pulmonology, hematology, and biostatistics (n = 1 each, 5 items total, 10%). The paediatrics domain was not represented in the available question set.

The complete EBNM examination comprises 140 questions with fixed proportional domain weights [7]. The 50-item subset used in this study therefore reflects an opportunistic domain distribution based solely on publicly available material. Scoring followed standard EBNM criteria with +1 point per correct response and no negative marking, yielding a maximum score of 50.

2.4. Prompt Engineering

We employed a standardized prompt following the CO-STAR framework (Context, Objective, Style, Tone, Audience, Response). The prompt specified [12]:

• Context: Expert nuclear medicine physician with comprehensive subspecialty knowledge;
• Objective: Answer 50 EBNM examination questions selecting exactly one option per question;
• Style: Clinical expertise with definitive single-best-answer selections;
• Tone: Professional and evidence-based;
• Audience: European subspecialty certification level;
• Response: Strict table format (Question|Answer) with no explanatory text.

Models received all questions in a single prompt with instructions to select exactly one letter (A–E) per question. No training examples, hints, or feedback were provided (zero-shot approach).

2.5. Data Collection

Two investigators conducted data collection with defined roles: one investigator queried all models and recorded responses, while a second investigator independently verified responses and performed scoring against the answer key. An additional reviewer supervised the process and resolved any discrepancies.

Each run was conducted as an independent evaluation session. For all ten web-based models, new browser sessions with cleared cache were initiated between runs. Responses were validated for completeness (all 50 questions answered) and correct format (single letter A–E per question) before scoring.

2.6. Statistical Analysis

Descriptive performance metrics included mean accuracy (%) across five independent runs per model. Variability of overall model performance across runs was quantified using the standard deviation (SD) and coefficient of variation (CV), calculated from raw total scores on the 0–50 scale, with CV used as a measure of inter-run score consistency. 95% confidence intervals were calculated using Student’s t-distribution (df = 4).

Inter-run reliability at the item-response level was assessed using Cohen’s kappa coefficient for each of the ten pairwise run combinations per model [C(5,2) = 10 pairs]. Mean kappa with range (minimum–maximum) across all ten pairs was reported for each model, interpreted according to Landis and Koch criteria [13]: <0.20 slight, 0.20–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and >0.80 almost perfect agreement. Cohen’s kappa was selected over Fleiss’ kappa because our objective was to quantify pairwise test–retest reliability at the item level rather than overall concordance among all five runs simultaneously. The pairwise approach also permitted identification of run pairs with unusually low agreement, as observed with Gemini 2.5 Pro (range: 0.000–1.000).

Pairwise model comparisons employed McNemar’s test [14] for paired binary data. For each of the 50 questions, we compared the consensus response (majority vote from five runs) for Model A against the consensus response for Model B, yielding n = 50 paired binary outcomes per comparison. The majority-vote consensus was selected for pairwise comparisons to filter out stochastic inter-run variability and represent the models’ most stable convergent answer for statistical testing. This approach prioritizes stable response tendencies over run-specific variability and may therefore underestimate session-level fluctuations. To control for multiple comparisons across 45 tests, we applied Bonferroni correction with an adjusted significance threshold of α = 0.0011 (0.05/45).

Given the sample size of 50 items, the study had limited power to detect moderate accuracy differences.

2.7. Ethical Considerations

This study did not involve human participants, patient data, or identifiable personal information and therefore did not require approval by an ethics committee.

3. Results

3.1. Overall Performance

All ten LLMs successfully completed the 50-question evaluation across five independent runs, yielding 2500 observations. Mean accuracy ranged from 53.6% (95% CI: 47.4–59.8%) for Falcon H1-34B to 100.0% (95% CI: 100.0–100.0%) for DeepSeek V3.2, representing a 46.4-percentage-point difference. All ten models exceeded the 50% threshold corresponding to minimum passing performance, with only Falcon H1-34B marginally above this benchmark. Complete performance metrics are presented in Table 1.

Table 1. Model Performance Summary.

Rank	Model	Mean Score	Mean Accuracy (%)	SD	95% CI (%)	CV (%)	Cohen’s κ
1	DeepSeek V3.2	50.0	100	0	100.0–100.0	0	1.000
2	Gemini 2.5 Pro	46.8	93.6	4.44	82.6–100.0	9.5	0.370
3	Grok-4	43.6	87.2	0.55	85.8–88.6	1.3	0.676
4	Mistral Medium 3.1	41.8	83.6	0.45	82.4–84.8	1.1	0.972
5	Claude Sonnet 4.5	40.8	81.6	1.79	77.2–86.0	4.4	0.802
6	Qwen3 Max	40.4	80.8	0.55	79.4–82.2	1.4	0.947
7	GPT-5 Pro	36.8	73.6	1.64	69.6–77.6	4.5	0.684
8	ERNIE 4.5 Turbo	33.6	67.2	1.14	64.4–70.0	3.4	0.500
9	Llama 3.3 70B	32.0	64.0	0.71	62.2–65.8	2.2	0.670
10	Falcon H1-34B	26.8	53.6	2.49	47.4–59.8	9.3	0.543

SD and CV were calculated from raw scores (0–50 scale). The 95% confidence intervals were calculated using the t-distribution (df = 4) across five independent runs. Cohen’s κ denotes mean inter-run agreement.

3.2. Top Performers

DeepSeek V3.2 achieved the highest mean accuracy of 100.0% (95% CI: 100.0–100.0%, SD = 0.00), demonstrating perfect consistency across all five runs with zero variability (CV = 0.0%). Gemini 2.5 Pro ranked second with 93.6% mean accuracy (95% CI: 82.6–100.0%), though showing moderate variability across runs (CV = 9.5%). Confidence intervals were truncated at 0% and 100% where necessary. Grok-4 achieved 87.2% accuracy (95% CI: 85.8–88.6%) with low variability (CV = 1.3%), indicating highly consistent performance. The top three models all substantially exceeded 80% accuracy, far surpassing the illustrative benchmark of 50%. Performance rankings are visualized in Figure 1.

Figure 1. Model performance with 95% confidence intervals across five independent runs. Error bars indicate 95% confidence intervals.

3.3. Proprietary Versus Open-Source Models

Proprietary models (n = 5) demonstrated a mean accuracy of 80.6% (range: 67.2–93.6%), while open-source models (n = 5) achieved 76.4% (range: 53.6–100.0%). Despite the slightly lower mean accuracy for open-source models, this category exhibited greater performance variability, including both the highest-performing model overall and the lowest performer. DeepSeek V3.2 (open-source) was the highest-performing model at 100.0%, achieving rank 1 overall and substantially outperforming all proprietary alternatives.

3.4. Inter-Run Reliability

Inter-run reliability ranged from κ = 0.370 for Gemini 2.5 Pro to κ = 1.000 for DeepSeek V3.2 (Figure 2 and Table 2). Four models achieved “almost perfect” agreement (κ > 0.80), three achieved “substantial” agreement (κ = 0.61–0.80), two showed “moderate” agreement (κ = 0.41–0.60), and one demonstrated “fair” agreement, with a mean κ = 0.716 across all models. DeepSeek V3.2 demonstrated the highest consistency (κ = 1.000, range: 1.000–1.000), reflecting identical responses across all five runs. Notably, Gemini 2.5 Pro showed the lowest reliability (κ = 0.370, range: 0.000–1.000) despite achieving the second-highest accuracy (93.6%), indicating substantial response variability across runs. No significant correlation was observed between mean accuracy and inter-run reliability (Spearman ρ = 0.394, p = 0.26, two-tailed), indicating that high performance does not automatically imply high consistency.

Figure 2. Inter-run score consistency measured by accuracy across five independent runs. Orange dotted line represents the illustrative pass threshold of 50%.

Table 2. Inter-run reliability metrics.

Model	Mean κ	Range	Interpretation
DeepSeek V3.2	1.000	1.000–1.000	Almost Perfect
Mistral Medium 3.1	0.972	0.929–1.000	Almost Perfect
Qwen3 Max	0.947	0.864–1.000	Almost Perfect
Claude Sonnet 4.5	0.802	0.732–0.922	Almost Perfect
GPT-5 Pro	0.684	0.508–0.896	Substantial
Grok-4	0.676	0.558–1.000	Substantial
Llama 3.3 70B	0.670	0.566–0.826	Substantial
Falcon H1-34B	0.543	0.356–0.878	Moderate
ERNIE 4.5 Turbo	0.500	0.288–0.955	Moderate
Gemini 2.5 Pro	0.370	0.000–1.000	Fair

Cohen’s κ values are reported as mean with range across ten pairwise run comparisons per model. Interpretation follows Landis and Koch criteria [13].

3.5. Pairwise Model Comparisons

Of 45 pairwise comparisons, 11 achieved statistical significance after Bonferroni correction (p < 0.0011; Table 3). The largest performance difference was observed between DeepSeek V3.2 and Falcon H1-34B (χ² = 21.00, p < 0.000005), representing a 46.4-percentage-point accuracy advantage. Significant differences were primarily detected between higher-performing and lower-performing models, with an average rank difference of 6.6 positions among significant pairs. Notably, the top three models (DeepSeek V3.2, Gemini 2.5 Pro, and Grok-4) showed no statistically significant differences from each other (all p ≥ 0.0011), despite accuracy differences ranging from 6.4 to 12.8 percentage points. Most significant comparisons (7 of 11) involved Falcon H1-34B, the lowest-performing model, compared against mid-tier or top-tier models.

Table 3. Significant pairwise model comparisons.

Model A	Model B	χ2	p-Value
DeepSeek V3.2	Falcon H1-34B	21.00	p < 0.0011
Falcon H1-34B	Gemini 2.5 Pro	21.00	p < 0.0011
DeepSeek V3.2	ERNIE 4.5 Turbo	17.00	p < 0.0011
DeepSeek V3.2	Llama 3.3 70B	17.00	p < 0.0011
ERNIE 4.5 Turbo	Gemini 2.5 Pro	17.00	p < 0.0011
Gemini 2.5 Pro	Llama 3.3 70B	17.00	p < 0.0011
Falcon H1-34B	Grok-4	14.22	p < 0.0011
DeepSeek V3.2	GPT-5 Pro	14.00	p < 0.0011
GPT-5 Pro	Gemini 2.5 Pro	14.00	p < 0.0011
Falcon H1-34B	Mistral Medium 3.1	11.27	p < 0.0011
Falcon H1-34B	Qwen3 Max	11.00	p < 0.0011

Pairwise comparisons were performed using McNemar’s test on majority-vote consensus answers. Statistical significance was assessed using Bonferroni correction (α = 0.0011).

3.6. Non-Significant Clinical Differences

Several relevant absolute accuracy differences exceeding 10 percentage points did not reach statistical significance after Bonferroni correction, reflecting the limited statistical power associated with a 50-item evaluation. These findings underscore that large absolute performance differences may remain statistically non-significant in small paired samples despite clear descriptive separation.

4. Discussion

In line with our predefined objectives, this study first evaluated the overall performance of ten state-of-the-art LLMs on 50 EBNM Fellowship Examination questions (Objective 1), revealing a 46.4-percentage-point performance range (53.6–100.0%). We then assessed inter-run reliability (Objective 2), conducted Bonferroni-adjusted pairwise comparisons (Objective 3), and compared proprietary versus open-source systems (Objective 4). DeepSeek V3.2 achieved perfect accuracy (100%, κ = 1.000), followed by Gemini 2.5 Pro (93.6%) and Grok-4 (87.2%), all substantially exceeding the illustrative pass threshold of 50%. All models except Falcon H1-34B achieved >60% accuracy, demonstrating baseline performance on released nuclear medicine examination items. The performance distribution suggests that model architecture and scale, rather than proprietary versus open-source status, may be the primary determinants of EBNM examination performance.

Inter-run reliability varied considerably (κ = 0.370–1.000, mean = 0.716), with four models achieving “almost perfect” agreement but others showing only moderate consistency [13]. Statistical comparisons identified 11 significant differences (p < 0.0011) among 45 pairwise tests, primarily distinguishing top-tier from lower-tier models. These findings establish preliminary benchmarks for AI performance on released examination material, while highlighting substantial variability in both accuracy and consistency across contemporary LLMs.

DeepSeek V3.2’s reported 100% accuracy across five runs requires critical contextualization. While this could theoretically imply domain mastery, the combination of zero error and perfect inter-run stability (κ = 1.000) across 250 observations is statistically indistinguishable from memorization. We interpret this apparent perfection cautiously, as it may represent a diagnostic signal of benchmark contamination, although alternative explanations cannot be excluded [15]. Without access to the model’s training corpus, we cannot definitively distinguish these scenarios. Unlike human experts, who exhibit stochastic variability even at high competence levels, DeepSeek’s absolute determinism suggests it may be retrieving “known” sequences from its pre-training corpus rather than solving novel problems. Consequently, this result highlights a systemic vulnerability in evaluating LLMs on publicly available datasets: extreme high performance may serve as an indicator of training data leakage rather than generalizable intelligence. To definitively separate reasoning from recall, future evaluations must strictly employ withheld, newly authored examination material that has no digital footprint. At the same time, alternative explanations must be considered. Perfect accuracy with full inter-run agreement may also arise from deterministic decoding configurations, low-temperature sampling, or stable inference behavior within the model architecture. Under such conditions, identical outputs across runs do not necessarily indicate prior exposure to the specific items but may reflect consistent inference pathways applied to structured multiple-choice questions. Without transparency regarding inference configuration and training data composition, it is not possible to conclusively attribute this result to contamination alone. The finding therefore highlights both the potential strength of advanced LLMs in structured knowledge retrieval and the methodological difficulty of disentangling memorization from genuine reasoning in publicly available benchmark settings.

Gemini 2.5 Pro presented a contrasting anomaly: high mean accuracy (93.6%, rank 2) coupled with the lowest reliability (κ = 0.370). This combination of strong average performance and substantial response variability indicates inconsistent behavior across runs rather than stable, reproducible competence. Gemini’s variability showed a structured rather than random pattern. Runs 2 to 4 exhibited perfect internal agreement (κ = 1.000), whereas runs 1 and 5 diverged on specific items. This suggests session-dependent variation instead of true uncertainty in underlying knowledge. Such instability raises concerns for clinical deployment, where consistent decision-making is essential. These findings underline the importance of multi-run evaluation protocols [16,17], since single-run assessments would have obscured Gemini’s reliability issues and likely overestimated its practical utility. Future studies should analyze the specific items that diverge and investigate factors contributing to variability, including temperature settings, prompt sensitivity, deterministic versus nondeterministic decoding, and potential backend or model-version effects.

Contrary to expectations that proprietary models would dominate, open-source alternatives showed competitive performance and in one case were superior. Proprietary models averaged 80.6% accuracy (range 67.2–93.6%) while open-source models averaged 76.4% (range 53.6–100.0%). The single best performer overall was the open-source system DeepSeek V3.2 with 100% accuracy. These results challenge the assumption that commercial status alone confers an advantage in specialized medical domains. The wide performance spread within each category, and especially within the open-source group, indicates that factors such as model architecture, parameter count and the quality of training data are the main determinants of exam performance. From an accessibility and cost perspective, strong open-source performance is important for resource-limited settings and research applications. At the same time the extreme variability among open-source models, for example Falcon H1-34B at 53.6% versus DeepSeek at 100%, implies that careful selection and rigorous validation are more critical for open-source systems than for proprietary systems, which tended to show more consistent mid-to-high performance.

The observed variability in inter-run reliability (κ = 0.370 to 1.000, mean κ = 0.716) shows that accuracy and consistency are distinct performance dimensions. Accuracy did not significantly predict reliability (Spearman ρ = 0.394, p = 0.26). Gemini 2.5 Pro, for example, reached the second-highest accuracy at 93.6 percent but showed the lowest stability with κ = 0.370, whereas several mid-range models demonstrated near-perfect reproducibility such as Mistral Medium 3.1 with κ = 0.972 and Qwen3 Max with κ = 0.947. This pattern carries important consequences for applied use. In educational contexts, learners depend on predictable responses and inconsistent answers undermine trust and instructional value. In clinical decision support, variability raises safety concerns because identical scenarios may lead to different recommendations across interactions. These results indicate that evaluation protocols for AI models should include multi-run assessments rather than relying on single-run metrics. Incorporating minimum reliability thresholds such as κ above 0.80 alongside accuracy criteria would improve the assessment of practical utility. Future work should examine the drivers of response variability, including prompt formulation, decoding temperature and architectural factors associated with stable behavior.

Our findings also fit within the evolving context of nuclear medicine education and examination practices. The EBNM Fellowship Examination was created to standardize specialist training across Europe and relies on a comprehensive written assessment that defines the expected breadth of nuclear medicine knowledge. Subsequent commentaries have emphasized its role in maintaining consistent professional standards and ensuring clinical quality across countries. The transition to online delivery during the COVID-19 period further highlighted both the opportunities and vulnerabilities of remote examination formats, including issues related to exam integrity and candidate authentication. Early evaluations of LLMs in this setting demonstrated marked inconsistency and frequent factual errors, indicating that model performance still falls short of the standards expected of certified specialists. Against this backdrop, our multi-run analysis provides complementary evidence that reliable and reproducible model behavior will be essential before LLMs can meaningfully support nuclear medicine training, exam preparation, or digital assessment workflows [18,19,20,21].

This study has several limitations:

• From an applied sciences perspective, this work should be interpreted as a methodological benchmark of model performance and reproducibility rather than an assessment of clinical competence.
• We evaluated 50 publicly available items rather than the full 140-question EBNM examination, which reduces representativeness and statistical power. Power calculations based on the observed data indicated limited ability to detect moderate differences in accuracy, and some differences in the 10 to 20 percentage point range may therefore have gone undetected.
• DeepSeek V3.2’s perfect performance raises concerns about possible test-set contamination or memorization, which cannot be excluded without access to the model’s training corpus [17]. This ambiguity necessitates replication on entirely withheld material before claims of true generalisable superiority can be made.
• Models were accessed via web interfaces using default settings, so sampling and decoding parameters were not controlled and may have introduced variability across runs.
• The evaluation focused on multiple-choice items and did not assess clinical reasoning, image interpretation, or hands-on skills that are central to nuclear medicine practice.
• Items were presented in English only and results may not generalize to other languages. Finally, LLM capabilities change rapidly and our results reflect models available in October 2025, so findings may not apply to future versions. Despite these limitations, this proof-of-concept study provides baseline benchmarks and underscores the importance of multi-run reliability assessment in medical AI evaluation.

Although this study focuses on nuclear medicine, the proposed multi-run evaluation framework is transferable to other specialized medical and technical certification contexts.

Future research priorities:

• Establish reproducibility and inter-run stability as primary performance dimensions in the evaluation of large language models, alongside conventional accuracy metrics.
• Investigate drivers of inter-run variability such as prompt design, decoding temperature, deterministic decoding and backend model-version differences. Develop standardized multi-run benchmarking protocols for high-stakes knowledge assessments, including predefined reliability thresholds alongside accuracy metrics.
• Evaluate performance on the complete 140-question EBNM exam to increase power and representativeness.
• Replicate the evaluation on entirely withheld and newly authored exam items to investigate potential data contamination and to confirm generalisability.
• Audit or request training-data provenance where feasible to clarify the origin of unusually strong performance.
• Expand assessment to include image-based tasks using SPECT, PET and hybrid cases and to evaluate clinical reasoning beyond multiple-choice formats.
• Compare model outputs directly with expert nuclear medicine physicians to establish clinical performance benchmarks.
• Develop and test workflows for safe human–AI collaboration in education and clinical practice.
• Evaluate model behavior on multiple-choice questions in which the number of correct response options is not specified, to assess both knowledge representation and response strategies under uncertainty, including tendencies toward risk-averse or risk-seeking answer selection.

5. Conclusions

Contemporary LLMs can achieve high accuracy on domain-specific, high-stakes multiple-choice assessments, demonstrating substantial knowledge retrieval capabilities in specialized medical contexts. However, this study shows that accuracy alone is insufficient to characterize model performance. Inter-run reliability varies markedly between models and does not consistently correlate with accuracy, indicating that single-run evaluations may substantially overestimate practical utility. Multi-run assessment and explicit reporting of reproducibility metrics, such as Cohen’s κ, are therefore essential components of robust LLM benchmarking. From an applied sciences perspective, the presented multi-run evaluation framework provides a transferable methodology for assessing both performance and stability of LLMs across specialized knowledge domains. Future validation should prioritize withheld test material, controlled inference settings, and task formats beyond text-only multiple-choice questions to better reflect real-world deployment conditions.