Comparative Evaluation of Large Language Models in Clinical Diagnostics for Real-World Medical Cases

Abstract

Background/Aim: Although large language models (LLMs) achieve >90% accuracy on medical multiple-choice questions, their actual diagnostic utility remains unproven. This study compared four approaches to general medical diagnostics using real clinical data and LLMs: standalone LLM, LLM with Retrieval-Augmented Generation (RAG), LLM with expert system, and full integration of LLM, RAG, and expert system. Methods: Twenty LLMs were tested on 1655 unpublished clinical cases (paediatrics and internal medicine) in Polish and English, yielding 264,800 diagnostic evaluations. TOP-1, TOP-3, and grouped International Classification of Diseases, 10th Revision (ICD-10) accuracy were measured. Results: Standalone LLMs achieved only 16–20% TOP-1 accuracy. RAG added minimal benefit (+2–5 percentage points). The expert system improved performance 2.5-fold (55% paediatrics, 39% internal medicine). Unexpectedly, combining all components reduced results compared to the expert system alone, revealing an “integration paradox”. Conclusions: LLMs alone are insufficient for clinical diagnostics in practical applications with incomplete data. Developers of clinical decision support systems should not treat LLMs as standalone diagnostic engines. Expert systems based on machine learning algorithms provide better support and should serve as the primary component in hybrid architectures. Combining LLM, RAG, and expert systems without deliberate output weighting paradoxically reduces performance. Hybrid systems should implement dynamic source selection or prediction weighting mechanisms rather than simple integration.

1. Introduction

Large language models have generated considerable interest in medical applications. Multiple studies report high accuracy across various diagnostic tasks. In an analysis of 1181 multiple-choice questions from intensive care medicine, GPT-4o achieved 93.3% accuracy, while LLaMA 3.1 70B reached 87.5%, Mistral Large 2407 achieved 87.9%, GPT-4o-mini scored 83%, and GPT-3.5 Turbo achieved 72.7%. All models substantially outperformed random guessing (41.5%) and physician responses (61.9%) [1]. In paediatric radiology diagnostics, GPT-4 Turbo achieved 88.6% accuracy on question-answering tasks, with results reaching 100% in selected clinical subsections. LLaMA 3.1 achieved 77.2% accuracy in the same domain [2]. These results suggest potential for LLMs to support clinical diagnostics. However, high accuracy on standardised multiple-choice questions does not necessarily translate into effectiveness in diagnosing real clinical cases with greater complexity.

A comprehensive meta-analysis of 83 studies from 2018 to 2024 revealed markedly divergent results. Average diagnostic accuracy of generative AI was approximately 52.1%, comparable to non-expert physicians, who showed only a 0.6 percentage-point advantage. Expert physicians substantially outperformed LLMs by 15.8 percentage points [3]. A systematic review of 19 LLMs in medical diagnostics found that clinicians typically achieve higher diagnostic accuracy than LLMs, though some studies showed models performing comparably to physicians [4]. These studies relied primarily on curated, well-defined clinical cases (case reports, vignettes) as opposed to the complex situations typical of hospital practice [3,4]. The disparity between encouraging results and the findings of meta-analysis indicates methodological problems in evaluating diagnostic systems.

A key finding is that GPT-4 results may be distorted by data leakage in LLM research. The spectacular TOP-1 (95.6%) and TOP-3 (100%) accuracy of GPT-4 was attributed to the fact that cases had been previously published and may have been included in training data. With unpublished data, model accuracy dropped substantially to 61% at TOP-6, barely better than physician accuracy of 49.1% [5]. Some studies deliberately used cases published after the model training cutoff to avoid memorisation effects and to obtain more reliable diagnostic accuracy assessments. Using this approach, LLaMA 3.1 405B achieved 45% at TOP-1 and 73% when providing the correct diagnosis among several proposed options [6].

Beyond methodological issues, LLMs show practical limitations in clinical settings. A paediatric study using ChatGPT 3.5 revealed an 83% rate of incorrect diagnoses. The chatbot failed to identify certain clinical relationships, such as the connection between autism and vitamin deficiencies. The authors emphasise the irreplaceable value of clinical experience [7]. A comprehensive analysis of real clinical data covering 2400 cases across four abdominal pathologies showed that current LLMs cannot accurately diagnose patients across all pathologies. They perform poorly, fail to follow diagnostic or therapeutic guidelines, and cannot interpret laboratory results, posing serious risks to patient health [8].

Traditional machine learning-based expert systems are often overlooked in discussions of modern diagnostic solutions. This omission seems unjustified given the documented clinical value. Expert systems are characterised by deterministic reasoning. A fuzzy logic-based expert system achieved 94% accuracy in identifying pregnancy complications [9]. The study’s system using machine learning (ML) algorithms with TOP-1 and TOP-3 mechanisms demonstrated ability to function effectively with incomplete clinical data [10]. These results indicate that classical expert systems still hold diagnostic value.

A limitation of some LLMs and chatbots is lack of real-time access to medical information [7]. Retrieval-Augmented Generation offers a promising approach to improving LLM effectiveness in medicine through integration of external knowledge sources. RAG extends basic language model capabilities through dynamic retrieval and delivery of context from medical databases, clinical guidelines, and published cases. A systematic review of RAG applications in healthcare showed this approach finds use in multiple areas including clinical guideline interpretation, diagnostic support, and selection of information from scientific literature. Studies confirm that RAG-based systems achieve higher diagnostic accuracy than baseline LLMs, particularly in tasks requiring current and contextual medical knowledge. The main challenges in RAG implementation include ensuring source knowledge currency and quality, reducing hallucinations, and validation requirements in clinical settings [11,12].

An important consideration for medical LLM applications is comparing the effectiveness of general-purpose models with that of medically specialised models. Research on Polish medical examinations showed that general models like LLaMA 3.1:70B and GPT-4o achieve higher accuracy than medical models such as Meditron:70B [13]. The authors suggest a possible cause: medical models were trained primarily on English medical data, limiting their effectiveness in other languages. These studies also confirm the impact of parameter count on model accuracy. As parameter count increases, accuracy improves and the gap between language results decreases. For LLaMA 3.1 8B, the difference between Polish and English versions is 13.44 percentage points, while for LLaMA 3.1 70B it drops to 1.66 percentage points [13].

The motivation for this research was an identified deficit in comprehensive comparative studies of different diagnostic approaches under controlled conditions. Most previous work focused on isolated evaluation of single LLMs or expert systems. Systematic analyses comparing these different technologies on the same set of clinical cases with uniform evaluation methodology are lacking. Particularly scarce are studies assessing LLM effectiveness under real medical practice conditions characterised by incomplete diagnostic data. A comprehensive scoping review covering 398 LLM studies in disease diagnostics indicates the need to develop more rigorous evaluation methodologies and to integrate different diagnostic approaches [14].

Given these limitations, alternative diagnostic approaches merit consideration. The most promising results come from systems combining different diagnostic technologies. In a study on predicting heart failure decompensation, an ML XGBoost system enhanced with data from the Medly expert system and electronic health records (EHR) achieved 98.08% accuracy. For comparison, the rule-based system alone recorded 93.6% accuracy. The positive predictive value (PPV) of alerts from that system was only 55.8%, while the XGBoost model increased it to 88.18%. This substantial PPV improvement of 32.38 percentage points with modest accuracy increase demonstrates complementarity between rule-based and ML approaches [15]. The authors of some studies explicitly recommend developing hybrid approaches that combine LLM capabilities in language analysis and explanation with the deterministic and explanatory capabilities of traditional clinical decision support systems [16]. A direct comparison of the DXplain expert system with GPT-4 and Gemini 1.5 on 36 unpublished clinical cases confirmed the complementarity of these tools. After the results of laboratory tests were taken into account, DXplain placed the correct diagnosis in TOP-25 in 72% of cases, compared to 64% for GPT-4 and 58% for Gemini 1.5 [16]. In the area of RAG systems, augmentation with external knowledge has been shown to improve accuracy by 1–18 pp relative to chain-of-thought prompting. The authors also noted that MEDRAG-assisted GPT-3.5 achieved results comparable to GPT-4 without RAG [17]. However, the benefits of RAG are not uniform for all models. In radiology examination tasks, RAG significantly improved GPT-4 scores, but no significant improvement was shown for Claude Opus or Mixtral [18]. The integration of knowledge graphs offers a complementary approach: for KG-RAG based on the SPOKE graph, a 71% improvement in Llama-2-13B scores was recorded [19]. Neuro-symbolic systems combining GPT-4 with rule-based expert components, in turn, achieved F1 = 1.00 versus F1 = 0.63 for standalone GPT-4 [20]. Hybrid systems have the potential to combine the advantages of three complementary approaches: LLMs enable natural language reasoning and the processing of complex clinical descriptions, expert systems offer deterministic reasoning and resistance to data leakage, whilst RAG enriches the diagnostic process with the context of similar cases and access to up-to-date medical knowledge from external sources. An example of such a hybrid architecture is the PARROT AI system [21], which integrates speech recognition, NLP-based semantic analysis, an ML expert system and contact centre components to automate the entire workflow associated with primary care appointments. However, there is a lack of comprehensive studies systematically evaluating the comparative effectiveness of different configurations of hybrid systems under controlled conditions. This paper fills this gap by comparing four integration scenarios on 1655 real clinical cases and documents the integration paradox.

An important consideration when selecting LLMs for medical applications concerns accessibility and cost issues. Commercial models like GPT-4 and Claude 3 Opus achieve accuracy comparable to non-expert physicians [3], but require paid API access, generating substantial operational costs. Research on Me-LLaMA shows that appropriately fine-tuned open-source models can achieve results comparable to or better than ChatGPT and GPT-4 on many test sets. Due to privacy concerns, the authors also emphasise that patient data cannot be sent to external services, adding another advantage for open-source models [22].

The aim of this study was to conduct a comprehensive comparative analysis of diagnostic effectiveness across four different approaches:

• Standalone LLM without additional support;
• LLM enhanced with a RAG system providing similar clinical cases;
• LLM enhanced with a machine learning-based expert system;
• Full integration of all three components: LLM + RAG + expert system.

The study was conducted on a dataset of 1655 clinical cases from internal medicine (N = 616) and paediatrics (N = 1039). All models were tested in two languages: Polish and English. Twenty different open-source LLMs were evaluated.

Previous research indicates substantial discrepancies between standardised test results and real-world clinical effectiveness. In response to identified literature gaps, the following research questions were formulated:

• Is standalone LLM effectiveness in real clinical cases substantially lower than in standardised multiple-choice tests? This study uses unpublished cases from actual medical documentation characterised by data incompleteness, which should reveal real limitations of standalone LLMs in clinical diagnostics.
• Will introducing the RAG mechanism produce substantial improvement in diagnostic effectiveness compared to standalone LLMs?
• Will full integration of three components (LLM + RAG + expert system) achieve the highest diagnostic effectiveness through synergistic use of the complementary advantages of each approach?
• Do general-purpose language models achieve higher diagnostic effectiveness than medical models in multilingual tasks? Do models with more parameters show greater effectiveness and smaller differences between languages, indicating higher cross-linguistic knowledge-transfer capability?

This study continues and extends earlier work on a diagnostic decision support system using incomplete clinical data [10]. The earlier study developed an expert system using 19 machine learning algorithms and TOP-1 and TOP-3 methodology generating rankings of one and three most probable diagnoses. The system was tested on a dataset of 1655 clinical cases from internal medicine and paediatrics characterised by varying degrees of data completeness. Identification of different optimal algorithms for each specialisation confirmed the necessity of adapting modelling strategies to medical domain specifics. This study uses the same test dataset and expert system as one component of the evaluated diagnostic approaches, enabling direct comparison of traditional and modern method effectiveness.

Organisation of the article is as follows: Section 2 describes the clinical dataset, tested scenarios, 20 LLMs used in the study, expert system, RAG mechanism, prompt structure, and evaluation metrics. Section 3 presents comparative analysis results for all four diagnostic scenarios across different models, languages, and specialisations, discussing results in the context of the existing literature. Section 4 contains the main conclusions and proposed avenues for future research.

2. Materials and Methods

2.1. Dataset

The clinical dataset used in this study was described in detail in an earlier publication [10]. The dataset comprises 1655 clinical cases from actual medical documentation across two specialisations: 616 cases from internal medicine and 1039 from paediatrics. Data was collected with the consent of patients. Each case underwent a process of anonymisation before inclusion in the dataset. As a result of anonymisation, each case was given a unique identifier replacing any personal data. Each case contains the following diagnostic elements: subjective symptoms reported by patients or caregivers, objective signs from physical examination, limited laboratory test results, demographic data (age, sex), comorbidity information, and final diagnosis coded according to ICD-10. The dataset is characterised by varying degrees of diagnostic data completeness, reflecting real clinical practice conditions [23]. In the context of this study, an incomplete case is defined as a case in which the lack of a complete clinical picture prevents a clear diagnosis. At least one such element was absent in about 70% of internal medicine cases and about 80% of paediatric cases. Average symptom count per case was 11.12 ± 5.59 in paediatrics and 10.24 ± 5.10 in internal medicine [10]. The heterogeneous representation of symptom is a deliberate dataset feature enabling assessment of the effectiveness of the diagnostic system under incomplete clinical data conditions.

For this study, two language versions of the dataset were compiled. The first version contained original Polish descriptions. The second version was created by machine-translating all textual descriptions into English. The whole thing has been verified by a native speaker for terminological correctness. ICD-10 codes and numerical values remained unchanged in both versions.

2.2. Diagnostic System Architecture

2.2.1. Diagnostic Scenarios

The study was implemented through systematic testing of each clinical case across defined diagnostic scenarios:

• S1—the LLM receives only clinical case data without additional context. This represents baseline LLM diagnostic capability.
• S2—the LLM is enhanced with similar case context from the RAG system.
• S3—the LLM uses expert system knowledge for prediction.
• S4—The LLM, RAG and expert system are fully integrated. This represents the most advanced configuration, using complementarity of different methods.

Each case was independently evaluated by 20 LLMs in two language versions: Polish and English, yielding a total of 264,800 diagnostic processes.

2.2.2. Large Language Models

Twenty LLMs with diverse architectural characteristics and sizes were selected for the study. The key selection criterion was open-source availability, enabling local deployment without licencing costs and ensuring compliance with medical data protection regulations. In an intensive care medicine study, processing 1181 questions cost EUR 3.60 for GPT-4o (93.3% accuracy), while GPT-4o-mini (83%) cost EUR 0.14; a more than 25-fold difference [1]. Commercial models like ChatGPT and GPT-4 offer advanced capabilities, but their closed nature limits flexible adaptation and medical application. Model selection also considered the degree of parameter diversity and presence of medically specialised models. Table 1 presents the complete list of models used along with producer information, profile (general, medical, with advanced reasoning mechanism), and parameter count. This compilation provides a baseline for further analysis of the impact of a model’s scale and characteristics on experimental results.

Table 1. List of models used in the study.

Producer	Model	Parameters	Description
OpenAI (San Francisco, CA, USA) [24]	GPT-OSS	20B	Medium general-purpose model.
	GPT-OSS	120B	Largest model in the study; general-purpose model.
	GPT-OSS-Med	20B	Medium model optimised for medical applications.
Google (Mountain View, CA, USA) [25,26]	Gemma 3	12B	Small general-purpose model.
	Gemma 3	27B	Medium general-purpose model.
	MedGemma	4B	Small model optimised for medical applications.
	MedGemma	27B	Medium model optimised for medical applications.
Meta Platforms, Inc. (Menlo Park, CA, USA) [27]	LLaMA 3.1	70B	Large general-purpose model.
	LLaMA 3.3	70B	Large general-purpose model.
DeepSeek (Hangzhou, China) [28]	DeepSeek-r1	14B	Small general-purpose model with extended reasoning.
	DeepSeek-r1	32B	Medium general-purpose model with extended reasoning.
	DeepSeek-r1	70B	Large general-purpose model with extended reasoning.
Mistral AI (Paris, France) [29,30]	Mistral NeMo	12B	Medium general-purpose model.
	Mistral Small 3.2	24B	Large general-purpose model.
Deep Cogito (San Francisco, CA, USA) [31]	Cogito	14B	Small general-purpose model.
	Cogito	32B	Medium general-purpose model.
	Cogito	70B	Large general-purpose model.
Alibaba Cloud (Hangzhou, China) [32]	Qwen 3	14B	Small general-purpose model.
	Qwen 3	32B	Medium general-purpose model.
PLLuM Consortium (Wroclaw, Poland) [33]	PLLuM	12B	Small general-purpose model with 8-bit quantisation, optimised for memory and performance.

All models were run locally using the Ollama platform (Ollama, Inc., San Francisco, CA, USA), which ensured control over the inference process and enabled precise measurement of performance parameters.

2.2.3. Expert System

The expert system was trained on a synthetic dataset completely separate from the test dataset. The training set consisted of 100% synthetically generated cases, while the test set exclusively contained real clinical cases from medical documentation. This differentiation ensures no data leakage between sets and enables assessment of the expert system’s generalisation capability under real clinical conditions. It serves as a reference point for comparisons of diagnostic effectiveness. Its architecture and parameterisation were described in detail in an earlier publication by the same authors [10]. The system used 19 machine learning algorithms: Decision Tree, Extra Tree, Random Forest, Extra Trees, k-Nearest Neighbors, Support Vector Machine, Multilayer Perceptron, Naive Bayes with Multinomial NB and Bernoulli NB variants, Logistic Regression, Ridge Classifier, Stochastic Gradient Descent (SGD) Classifier, Linear Discriminant Analysis, Quadratic Discriminant Analysis, Gradient Boosting, Histogram-based GB, AdaBoost, and Dynamic Ensemble. The system applies a minimum probability threshold of 0.3. In scenario S4, the absence of an expert system result was recorded in 5980 cases, which directly affected the availability of input data for the LLM in this scenario.

The system generates rankings of most probable diagnoses (TOP-1 and TOP-3) along with prediction confidence values. The ranking mechanism uses probability aggregation from individual models. The system is optimised separately for each medical specialisation and shows high effectiveness under incomplete input data conditions through native handling of missing values by most of the algorithms implemented [10].

2.2.4. RAG Mechanism

The RAG system extends the basic diagnostic process by providing LLMs with additional context in the form of similar clinical cases from the training database. The system uses the Intfloat Multilingual-E5-Large model (Microsoft Research, Redmond, WA, USA) [34] to generate vector representations of clinical cases in 1024-dimensional space. This model was selected due to Polish and English language support and the highest scores on the multilingual MTEB benchmark.

Figure 1 illustrates the RAG pipeline. The first step in the similar case retrieval process is query preparation. This involves transforming clinical case data into a standardised format containing all available information: confirmed and excluded symptoms, demographic data, ICD-10 codes of comorbidities, medical history, and physical examination. The generated query vector is used by the ChromaDB system (Chroma, Inc., San Francisco, CA, USA) for a k-nearest neighbour search using the Euclidean distance metric. Efficient similar case retrieval requires optimised computational algorithms, a comparative analysis of which was presented for different processing models in [35]. The value k = 10 was selected empirically as a compromise between providing sufficient diagnostic context and avoiding informational noise from excessive reference cases. A sensitivity analysis was performed for k ∈ {3, 5, 10, 15, 20} and similarity thresholds from 0.0 to 0.6 on all test cases in both languages and specialisations. The results are presented in Table 2 in Section 3.1. For each case found, a similarity value is calculated in the range [0, 1], where higher values indicate a greater match. The system uses a minimum similarity threshold of 0.3, which was established to eliminate cases with an insufficient degree of similarity that could mislead LLMs. The RAG system operates as an independent module providing a REST API interface.

Figure 1. RAG pipeline.

Table 2. RAG system sensitivity analysis—retrieval accuracy for different values of k and similarity thresholds.

Lang	Specialisation	Exact Hit Rate (%)					Grouped Hit Rate (%)					AVG
		k = 3	k = 5	k = 10	k = 15	k = 20	k = 3	k = 5	k = 10	k = 15	k = 20	Sim
PL	Paediatrics	9.82	11.65	13.28	14.05	14.73	18.86	21.56	24.45	25.70	26.76	0.803
PL	Internal Med.	2.76	3.90	4.87	4.87	5.19	8.28	9.90	12.01	12.82	13.31	0.789
PL	All	7.19	8.76	10.15	10.63	11.18	14.92	17.22	19.82	20.91	21.75	0.798
EN	Paediatrics	10.11	11.16	12.03	12.51	12.80	17.52	19.25	20.69	21.75	22.14	0.785
EN	Internal Med.	3.41	4.22	4.71	5.36	5.36	9.74	11.20	12.82	13.47	13.96	0.778
EN	All	7.61	8.58	9.31	9.85	10.03	14.62	16.25	17.76	18.67	19.09	0.788

2.2.5. Prompt Structure

Interaction with LLMs is based on a defined prompt structure consisting of two components. The first is the system prompt, which defines the model’s role and required response format. Listing 1 presents the system prompt structure in English for the basic S1 scenario. It defines the model as an advanced medical diagnostic system required to generate responses exclusively in the JSON format with a specified structure. Each response must contain a brief description of the reasoning process, exactly three potential diagnoses ranked by probability, and a reasoning summary. Each proposed diagnosis must also have an ICD-10 code, diagnosis name in the required language, confidence level (high, medium, low), and symptom-based justification. In scenarios using additional context sources (S2, S3, S4), the system prompt instructs the model to consider this information in the diagnostic process. Accordingly, the required JSON structure contains additional fields for summarising the utilised data.

Listing 1. System prompt structure for scenario S1 in English version.

You are an advanced medical diagnostic system. You MUST respond ONLY in JSON format. REQUIREMENTS: 1. Respond ONLY with valid JSON 2. DO NOT add any text before or after the JSON 3. USE EXACTLY the specified structure 4. ALWAYS provide 3 diagnoses in order of probability REQUIRED JSON STRUCTURE: {   “thinking_process”: “short description of the thinking and analysis process”,   “diagnoses”: [   {   “rank”: 1,   “icd10_code”: “ICD-10 code”,   “diagnosis_name_en”: “name of the diagnosis in English”,   “confidence”: “high”,   “justification”: “justification based on symptoms”   },   {   “rank”: 2,   “icd10_code”: “ICD-10 code”,   “diagnosis_name_en”: “name of the diagnosis in English”,   “confidence”: “medium”,   “justification”: “justification based on symptoms”   },   {   “rank”: 3,   “icd10_code”: “ICD-10 code”,   “diagnosis_name_en”: “name of the diagnosis in English”,   “confidence”: “low”,   “justification”: “justification based on symptoms”   }   ],   “clinical_reasoning”: “clinical thinking process” } REMEMBER: Respond ONLY with this JSON, without any additional text before or after.

The second component is the user prompt, which contains clinical case data and diagnostic instructions. For scenario S1 (Listing 2) the prompt directs systematic analysis of patient clinical data. The model receives a structured case description and instructions for a hierarchical assessment of symptoms that distinguishes between confirming symptoms and excluding ones. Consideration of comorbidities in differential diagnosis is also required, along with the application of Occam’s razor to select the diagnosis that best explains the complete clinical picture. Additionally, the model was required to generate responses in the JSON format according to the required structure, to enable further processing and analysis of results. For extended scenarios S2, S3, and S4, the user prompt provides additional information from the RAG system, expert system, or both. The instruction list is also supplemented with further data to be considered, taken from supporting systems in the diagnostic process.

Structured patient data includes basic information (age, sex), numerical data (symptom count, comorbidity count), lists of confirmed and excluded symptoms, physical examination description, and medical history.

Listing 2. User prompt structure for scenario S1 in English version.

Analyse patient data systematically. PATIENT DATA: {formatted_patient_data} INSTRUCTIONS: 1. Conduct a systematic assessment of all clinical data provided. 2. Consider the hierarchy of symptom importance (confirming vs. excluding symptoms). 3. Consider comorbidities in the context of differential diagnosis. 4. Apply Occam’s razor—prefer diagnoses that explain the most symptoms. 5. Respond ONLY with clean JSON that complies with the required structure. RESPOND ONLY WITH JSON!

The English prompt is a direct translation of the Polish version, maintaining identical structure and requirements. Generation parameters were standardised for all models as a methodological requirement to ensure the comparability of results. A temperature of 0.3 favours deterministic responses. This choice is clinically justified, as LLM responses are non-deterministic and may vary with an identical prompt. This presents a challenge for diagnostic applications requiring reproducible results [36]. The values top_p = 0.8 and repeat_penalty = 1.1 are standard settings to limit hallucinations and repetitions. In cases where the model returned a response that did not conform to the required JSON structure, the case was subjected to one re-iteration. Of the 264,800 diagnostic processes, re-iteration was necessary in 1588 cases (0.6%).

2.3. Evaluation Metrics

Diagnostic effectiveness was assessed using four metrics reflecting different aspects of the diagnostic process in clinical practice.

TOP-1 Accuracy measures the proportion of cases where the correct diagnosis was ranked first. High values indicate the system’s ability to precisely identify the most probable diagnosis.

TOP-3 Accuracy measures the proportion of correct diagnoses appearing in the TOP-3 ranking, regardless of position. This metric reflects differential diagnosis practice where physicians consider several best-matched diagnoses before making a final decision.

Grouped TOP-1 Accuracy checks whether the first-position prediction belongs to the same ICD-10 code group as the correct diagnosis, even if not identical. Grouping is based on clinical similarity of disease entities and was performed by a physician expert. For example, if the correct diagnosis is M05.8 (other seropositive rheumatoid arthritis) and the system returned M05 (seropositive rheumatoid arthritis), both codes belong to the same clinical group covering rheumatoid arthritis. Such a prediction is considered partially correct in the context of grouped TOP-1 because it directs the physician to the proper diagnostic category.

Grouped TOP-3 Accuracy checks whether any of the three best codes belong to the same clinical group as the correct diagnosis. This metric reflects the system’s ability to narrow differential diagnosis to the proper clinical category even without precisely identifying the disease subtype.

3. Results

This section presents a systematic evaluation of the diagnostic effectiveness results for 20 LLMs across the four scenarios defined in Section 2.2.1. Each model was independently tested in both language versions on the dataset of 1655 clinical cases (1039 paediatrics, 616 internal medicine). Experiments were conducted on a Supermicro AS-4124GO-NART (Super Micro Computer, Inc., San Jose, CA, USA) server equipped with two AMD EPYC 7702 processors (Advanced Micro Devices, Inc., Santa Clara, CA, USA) (2 × 64 cores, 2.0 GHz), 1 TB DDR4 3200 MHz RAM, and eight NVIDIA A100 SXM4 40 GB accelerators (NVIDIA Corporation, Santa Clara, CA, USA). LLMs were run locally using the Ollama platform.

3.1. Sensitivity Analysis of RAG System Parameters

The conducted sensitivity analysis showed that the similarity threshold did not affect the results (Table 2). All returned cases received a similarity score above 0.6. The embedding space does not effectively distinguish between diagnostically accurate and inaccurate cases. Increasing k from 3 to 20 resulted in only a marginal improvement: +4.0 pp for Polish and +2.4 pp for English in exact ICD-10 code matching. Even with k = 20, a correct diagnosis emerged among the returned cases in only 10% to 11% of paediatric cases and about 5% of internal medicine cases. In approximately 80% to 95% of cases, the LLM received reference cases that did not contain a correct diagnosis. The limited impact of RAG is therefore not an artefact of suboptimal parameter selection. It stems from the fundamental difference between a synthetic training base and real clinical cases with incomplete data.

3.2. Diagnostic Effectiveness by Scenario

Table 3 presents diagnostic effectiveness measured by the TOP-1, TOP-3, and grouped TOP-1/TOP-3 metrics broken down by specialisation and language. All values represent the average effectiveness for the 20 models in each configuration.

Table 3. Diagnostic effectiveness by scenario, language, and specialisation.

Scenario	Lang	Paediatrics		Internal Med.		Paediatrics Grouped		Internal Med. Grouped
		TOP-1 [%]	TOP-3 [%]	TOP-1 [%]	TOP-3 [%]	TOP-1 [%]	TOP-3 [%]	TOP-1 [%]	TOP-3 [%]
S1	PL	20.71	31.3	16.75	24.81	24.54	37.13	28.23	41.46
	EN	19.67	29.92	15.4	22.99	23.45	35.72	25.7	37.95
S2	PL	23.08	36.15	17.65	28.52	27.02	42.4	29.63	43.84
	EN	24.83	36.79	19.98	32.05	29.31	42.54	34.47	49.82
S3	PL	53.45	65.73	39.08	60.08	64.28	73.27	73.93	81.57
	EN	56.18	67.59	39.66	59.02	67.37	76.49	73.25	80.93
S4	PL	45.9	61.74	37.45	52.98	53.7	71.67	66.77	79.51
	EN	48.63	63.54	37.31	53.7	57.03	73.56	64.82	78.62

Sample size N = 3310.

Scenario S1 revealed substantial limitations of standalone LLMs in diagnosing real clinical cases. Average TOP-1 effectiveness was 20.19% (PL: 20.71%, EN: 19.67%) in paediatrics and 16.08% (PL: 16.75%, EN: 15.4%) in internal medicine. Models generated correct diagnoses as the first proposal in only one in five paediatric cases and one in six internal medicine cases. The TOP-3 metric increased effectiveness to 31.3% (PL) and 29.92% (EN) in paediatrics, and 24.81% (PL) and 22.99% (EN) in internal medicine. Applying ICD-10 grouping for TOP-1 improved effectiveness by an average of 3.81 percentage points in paediatrics and 10.89 percentage points in internal medicine, indicating the value of differential diagnosis in the context of related disease entities. Differences between language versions were minimal. The Polish version outperformed English by an average of 1.2% in TOP-1 and 1.6% in TOP-3, suggesting that the models’ diagnostic limitations do not systematically stem from language barriers but from fundamental clinical reasoning deficits.

Introducing the RAG system (S2) resulted in modest improvement in diagnostic effectiveness. In paediatrics, TOP-1 effectiveness increased by 2.37 percentage points in PL and 5.16 percentage points in EN, reaching an average of 23.96%. In internal medicine, the improvement was 0.9 percentage points (PL) and 4.58 percentage points (EN), reaching an average of 18.82%. The minimal RAG system effectiveness suggests that under conditions of very low baseline LLM model effectiveness, the RAG mechanism does not provide sufficient context for substantial improvement in the diagnostic process. ICD-10 TOP-3 grouping increased RAG effectiveness to 42.4% (PL) and 42.54% (EN) in paediatrics, and 43.84% (PL) and 49.82% (EN) in internal medicine. This remains substantially below clinically useful levels.

Scenario S3 showed major improvement in diagnostic effectiveness relative to S1. In paediatrics, TOP-1 effectiveness reached 53.45% PL and 56.18% EN, representing a 2.7-fold increase relative to S1. In internal medicine, similar behaviour was observed: the Polish version showed an improvement of 22.33 percentage points in TOP-1 and 35.27 percentage points in TOP-3. The English version surpassed these results, achieving 39.66% in TOP-1 and 59.02% in TOP-3. Applying ICD-10 grouping further increased effectiveness: grouped TOP-1 reached an average of 65.83% in paediatrics and 73.6% in internal medicine, while grouped TOP-3 reached 74.88% and 81.25%, respectively. These results demonstrate that the deterministic approach of expert systems and ML algorithms substantially outperforms the effectiveness of generative language models under real clinical practice conditions characterised by data incompleteness.

The full integration scenario (S4) showed an unexpected decrease in effectiveness relative to S3 in every configuration. In both the PL and EN datasets, paediatrics TOP-1 declined by 7.55 percentage points relative to S3 while increasing by 25.19 percentage points (PL) and 28.96 percentage points (EN) relative to S1. Similarly in internal medicine, TOP-1 PL showed a fall from 39.08% to 37.45%, and EN from 39.66% to 37.31%. The most pronounced decline was recorded in paediatrics EN grouped TOP-1, from 67.37% to 57.03%.

The observed decrease in effectiveness in S4 is a key finding of the study. In order to quantify the mechanism of this phenomenon, all S4 cases in which the expert system and the RAG proposed different ICD-10 codes were identified (conflict cases n = 37,942). In these cases, the LLM followed the expert system in 65.4% of cases (S4 effectiveness: 64.3%), RAG in 24.3% of cases (S4 effectiveness: 2.7%) and no component in 10.2% of cases (S4 effectiveness: 3.6%). The detailed results are presented in Table 4. These data indicate that a key mechanism for the degradation of S4 results is the fact that LLM selectively followed RAG recommendations in cases of conflict, even though the expert system indicated the correct diagnosis in 82.6% to 87.2% of these cases.

Table 4. Distribution of LLM decisions in S4 conflict cases (n = 37,942).

LLM Followed	n	% of Conflict Cases	S4 Effectiveness [%]	Expert System Correctly [%]	RAG Correctly [%]
Expert system	24,833	65.5	64.3	87.2	1.4
RAG	9225	24.3	2.7	82.6	4.8
None	3884	10.2	3.6	80.7	1.1

The statistical significance of differences between scenarios was assessed using McNemar’s test. Scenario S3 achieved significantly higher grouped TOP-3 effectiveness than S1 (Δ = +41.5 pp, p < 0.001) and S2 (Δ = +34.4 pp, p < 0.001). The difference between S3 and S4 fell below statistical significance (Δ = −2.2 pp, p > 0.05), though a trend of declining effectiveness was observed after adding the RAG component. Comparison of TOP-5 models in scenario S3 showed no statistically significant differences between all pairs (p > 0.21), indicating equivalent diagnostic effectiveness of the LLaMA3.1 70B, LLaMA3.3 70B, Cogito 70B, Mistral-Small3.2 24B, and GPT-OSS 20B models. Given the number of conducted comparisons, the risk of Type-I error due to multiple testing was taken into account. Key results, however, remain immune to this concern. Comparisons between scenarios (S1 vs. S3 and S2 vs. S3) remain relevant even after a conservative Bonferroni correction. Statistically insignificant results would retain their interpretation with any adjustment for multiple comparisons because these adjustments can only raise the significance threshold, not lower it.

3.3. Analysis of TOP-5 Model Effectiveness Across Scenarios by Specialisation

Table 3 clearly shows substantial differences in results between specialisations. Figure 1, Figure 2, Figure 3 and Figure 4 present the TOP-5 results of models distinguishing between specialisation and scenario. Results are presented for single TOP-3 and grouped TOP-3. This methodology reflects the actual practice of differential diagnosis.

Figure 2. Effectiveness of TOP-5 models in S1 for internal medicine (a) and paediatrics (b) for single TOP-3 and grouped TOP-3 metrics.

Figure 3. Effectiveness of TOP-5 models in S2 for internal medicine (a) and paediatrics (b) for single TOP-3 and grouped TOP-3 metrics.

Figure 4. Effectiveness of TOP-5 models in S3 for internal medicine (a) and paediatrics (b) for single TOP-3 and grouped TOP-3 metrics.

Figure 2a presents results for S1 in internal medicine, where Cogito 70B showed the highest effectiveness, with scores of 36.93% for single TOP-3 and 52.03% for grouped TOP-3. The same model ranked fourth in paediatrics (Figure 2b) with a score of 46.44%. The 9.51-percentage-point increase between specialisations for single TOP-3 illustrates the universality of this model’s architecture. It does not require specialised domain knowledge to maintain competitive effectiveness. In paediatrics, Mistral-Small3.2 24B proved most effective, achieving 54.28% single TOP-3 and 61.07% grouped TOP-3. The same model ranked second in internal medicine with a score 18.48 percentage points lower in single TOP-3. The larger drop for single scores suggests difficulty in precisely indicating a single diagnosis in internal medicine. The medically dedicated model (MedGemma 27B) showed the second-best result in paediatrics, achieving 52.98% in single TOP-3 and 59.29% in grouped TOP-3. However, in internal medicine, this model came last, recording an effectiveness 22.03 percentage points lower relative to paediatrics in single codes and 12.05 percentage points lower in grouped codes. This substantial difference indicates the limitations of narrow medical model specialisation. Model MedGemma 27B may be optimised for certain case types, resulting in high effectiveness for paediatrics with reduced effectiveness in internal medicine in more heterogeneous cases. The average difference between the single and grouped score is 16.57 percentage points in internal medicine and 6.81 percentage points in paediatrics. This twofold higher value in internal medicine confirms the effectiveness of semantic grouping with higher diagnostic complexity. In internal medicine, symptoms are less specific, and precisely indicating a single ICD-10 code is more difficult. In paediatrics, the smaller difference between single and grouped results suggests less ambiguous symptomatology, with symptoms more often leading to a specific diagnosis without requiring broad differential diagnosis.

A confidence interval analysis (95% CI) reveals substantial heterogeneity between models and differences between metrics. In paediatrics, intervals for single TOP-3 range from [36.7%, 40.9%] for LLaMa3.3 70B to [52.1%, 56.5%] for Mistral-Small3.2 24B, giving a spread of 15.4 percentage points. For grouped TOP-3, the range extends from [44.7%, 48.8%] for LLaMa3.3 70B to [59%, 63.1%] for Mistral-Small3.2 24B. In internal medicine, intervals for single TOP-3 are more similar, from [27.4%, 32.6%] for MedGemma 27B to [34.2%, 39.6%] for Cogito 70B. The maximum spread is only 7 percentage points. For grouped TOP-3, the range extends from [43.9%, 49.6%] for Gemma3 27B to [49.2%, 54.7%] for Cogito 70B and Mistral-Small3.2 24B. This greater homogeneity in internal medicine suggests that standalone models reach a similar performance ceiling regardless of architecture.

In S2 (Figure 3a), Cogito 70B maintained its first position with a slight improvement of 1.3 percentage points in single TOP-3 and 3.25 percentage points in grouped TOP-3. Second place went to GPT-OSS 120B achieving 37.01% in single and 56.66% in grouped TOP-3. The Mistral-Small3.2 24B model dropped from second to fourth position, noting minimal improvement of 0.73 percentage points for single TOP-3 and 1.43 percentage points for grouped TOP-3 relative to S1. Significant improvement was achieved by MedGemma 27B reaching 36.69% and 56.90% for single and grouped TOP-3, respectively. This is the highest effectiveness level in the grouped labels metric. RAG allowed medical models to utilise additional domain knowledge. The improvement between internal medicine models is minimal relative to S1, confirming limited RAG value under conditions of very low baseline model effectiveness. In paediatrics (Figure 3b), Gemma3 27B took the lead with a minimal difference of 2.02 percentage points in single and 2.41 percentage points in grouped TOP-3 relative to S1. Similarly to internal medicine, GPT-OSS 120B placed second with 48.56% in single TOP-3 and a slightly lower grouped TOP-3 score than the top-ranked model at 55.29%. The medical model ranked third with a drop of 7.65 percentage points in single and 6.88 percentage points in grouped TOP-3 relative to S1. A substantial decline was also shown by Mistral-Small3.2 24B: 11.11 percentage points (single TOP-3) and 12.71 percentage points (grouped TOP-3). The average difference between metrics remained at a similar level as in S1. On the paediatric dataset, the models achieve results on average 9.2% higher in single TOP-3 than in internal medicine. However, in the grouped codes metric, the models performed slightly better in internal medicine than in paediatrics, by an average of 2.14 percentage points. This unexpected inverse relationship in grouped TOP-3 results suggests that RAG introduces information noise that interferes with simple paediatric cases more than complex internal medicine cases.

Confidence intervals for S2 show characteristic patterns differing between metrics. In paediatrics, the range for single TOP-3 extends from [41%, 45.1%] to [46.6%, 50.6%], giving a spread of 5.5 percentage points. For grouped TOP-3, intervals range from [46.2%, 50.5%] for Mistral-Small3.2 24B to [53.8%, 58%] for Gemma3 27B, giving a spread of 7.5 percentage points. In internal medicine, intervals for single TOP-3 extend from [32.8%, 38%] to [35.5%, 40.9%]. For grouped TOP-3, the range is from [47.5%, 53.2%] for Mistral-Small3.2 24B to [54.1%, 59.6%] for MedGemma 27B.

Scenario S3 (Figure 4) showed a substantial improvement of result in both specialisations. In internal medicine, LLaMA3.3 70B ranked first with 69.89% and 84.58% for single and grouped TOP-3, respectively. It achieved improvement relative to S1 of 36.77 percentage points in single codes and 33.61 percentage points in grouped. Second place went to GPT-OSS 120B noting a substantial improvement of 30.28 percentage points (single TOP-3) and 28.32 percentage points (grouped TOP-3). The Cogito 70B model dropped from first to third place achieving 66.96% in single and 84.74% in grouped. For paediatrics, once again Mistral-Small3.2 24B proved best, achieving results 19.2 percentage points higher in single and 19.39 percentage points in grouped TOP-3 relative to S1. Medical model MedGemma 27B achieved the highest value in grouped TOP-3 at 82.39%. In last place in both specialisations was GPT-OSS 20B. In paediatrics, it achieved 72.67% in single and 80.41% in grouped. In internal medicine, it achieved 66.64% in single and 84.17% in grouped. The difference between specialisations is 6.03 percentage points in favour of paediatrics in single TOP-3. For grouped TOP-3, the difference is 3.76 percentage points in favour of internal medicine. In the single TOP-3 metric, paediatrics achieved an advantage with an average result of 73.05%, while internal medicine averaged 67.52%. However, in the grouped TOP-3 metric, internal medicine averaged 84.66%, gaining an advantage of 3.9%. This inverse relationship in grouped TOP-3 indicates that the expert system more effectively identifies groups of related entities in internal medicine. In paediatrics, precise indication of a single code is easier, while in internal medicine, semantic grouping compensates for greater diagnostic complexity. S3 is characterised by the narrowest confidence intervals in both metrics, indicating high determinism. In paediatrics, intervals for single TOP-3 fall within the range of [70.6%, 74.8%] to [71.6%, 75.5%]. The maximum spread between models is only 0.7 percentage points on the lower bound, while in S1 it was 15.4 percentage points. For grouped TOP-3, intervals extend from [78.4%, 81.8%] for LLaMA3.1 70B to [80.7%, 83.9%] for MedGemma 27B, giving a spread of 2.1 percentage points. This reduced heterogeneity confirms that the expert system levels out the architectural differences between models. In internal medicine, intervals for single TOP-3 range from [64.0%, 69.3%] to [67.4%, 72.3%], with a spread of 3 percentage points. For grouped TOP-3, the range is from [82.1%, 86.2%] for GPT-OSS 20B to [83%, 86.9%] for GPT-OSS 120B. It is worth noting that in internal medicine, all models exceed 82% on the lower interval bound for grouped TOP-3, while in paediatrics values start from 78.4%. This indicates higher predictability of the expert system in internal medicine, where ML algorithms compensate for greater diagnostic complexity.

In S4 (Figure 5), Mistral-Small 3.2 24B ranked first in both specialisations. The single TOP-3 metric showed higher value for paediatrics, by 6.52%. However, the grouped TOP-3 result was better in internal medicine, by 6.16%. In internal medicine, LLaMA3.3 70B achieved second place with values of 58.93% for single and 84.58% for grouped. The improvement relative to S1 was 25.81 percentage points in single and 33.61 percentage points in grouped. Fourth position went to the medical model from the GPT-OSS family, achieving 59.93% and 81.98% for single TOP-3 and grouped TOP-3, respectively. In paediatrics, GPT-OSS 120B performed best in the grouped TOP-3 metric, achieving 79.6%. A similar result was achieved by Cogito 70B at 79.5%. The average difference between metrics is 22.39 percentage points in internal medicine and 9.12 percentage points in paediatrics. As with S3, the single TOP-3 metric shows better results on the paediatric dataset, by an average of 8.24 percentage points, while for grouped TOP-3, the average is 5.02 percentage points higher in internal medicine. Full system integration results in wider confidence intervals relative to S3 in both metrics. This confirms the observed decrease in determinism. In paediatrics, intervals for single TOP-3 extend from [65.4%, 69.4%] for DeepSeek-R1 70B to [69.1%, 73.1%] for GPT-OSS 120B. The DeepSeek-R1 70B model shows the lowest lower interval bound in the entire S4 scenario, indicating problems with integrating extended thinking with the expert system. For grouped TOP-3, intervals range from [74.2%, 77.8%] for DeepSeek-R1 70B to [77.9%, 81.1%] for Cogito 70B. In internal medicine, intervals for single TOP-3 extend from [56.2%, 61.7%] for LLaMa3.3 70B to [62.1%, 67.5%] for Mistral-Small3.2 24B. For grouped TOP-3, the range is from [79.9%, 84.3%] for GPT-OSS-Med 20B to [82.5%, 86.5%] for LLaMA3.3 70B.

Figure 5. Effectiveness of TOP-5 models in S4 for internal medicine (a) and paediatrics (b) for single TOP-3 and grouped TOP-3 metrics.

3.4. Impact of Parameter Count and Model Specialisation on Diagnostic Effectiveness and Computational Efficiency

Table 5 presents single TOP-3 diagnostic effectiveness and average response time for 13 models grouped into four families: GPT-OSS (3 variants), Gemma (4 variants), DeepSeek-R1 (3 variants), and Cogito (3 variants). An analysis of the impact of parameter count on effectiveness reveals a nonlinear relationship between model scale and diagnostic results.

Table 5. Diagnostic effectiveness (single TOP-3) and average response time for 13 LLMs in four diagnostic scenarios (S1–S4). Models grouped into four families: GPT-OSS, Gemma, DeepSeek-R1, and Cogito. Medical variants marked in italics. Values represent averages for both languages and medical specialisations.

Model	S1		S2		S3		S4
	Single TOP-3 [%]	Average Time [s]	Single TOP-3 [%]	Average Time [s]	Single TOP-3 [%]	Average Time [s]	Single TOP-3 [%]	Average Time [s]
GPT-OSS 20B	26.37	8.13	37.25	12.3	70.42	13.84	66.07	13.16
GPT-OSS 120B	31.33	11.63	44.26	19.09	70.85	14.58	67.34	20.3
GPT-OSS-Med 20B	25.98	9.73	36.53	13.82	68.97	10.56	64.35	14.47
Gemma3 12B	34.2	8.64	37.64	13.54	59.64	11.16	53.87	14.24
Gemma3 27B	40.48	15.2	43.69	22.89	61.18	18.45	57.13	24.1
MedGemma 4B	11.3	7.54	23.34	43.61	63.91	10.99	53.09	50.11
MedGemma 27B	44.41	24.59	42.11	72.21	67.07	32.2	60.36	81.14
DeepSeek-R1 14B	9.55	14.61	22.81	17.66	48.82	14.91	45.05	17.16
DeepSeek-R1 32B	13.38	28.99	26.8	33.53	56.86	29.25	57.98	33.32
DeepSeek-R1 70B	27.98	48.82	34.98	52.96	65.2	48.91	63.6	55.02
Cogito 14B	10.51	6.94	27.4	11.17	51.63	8.36	50.85	11.96
Cogito 32B	22.87	12.35	32.87	18.9	59.12	14.72	56.5	20.19
Cogito 70B	42.9	22.1	42.05	30.42	70.15	24.97	64.32	31.32

Sample size N = 3310.

In the GPT-OSS family, increasing parameters from 20B to 120B in S1 results in an improvement in effectiveness of 4.96 percentage points. This increase requires a minimal response time increase of 3.5 s. In S3, the difference in effectiveness between the 20B and 120B variants shrinks to 0.43 percentage points (70.42% vs. 70.85%). The expert system balances out the differences in model capacity. Response time increases by only 0.74 s. S4 shows a minimal advantage of 1.27 percentage points in favour of the 120B variant with response time extended by 7.14 s. This suggests diminishing returns on investment in larger models when using advanced prompting techniques. The gemma family shows similar characteristics. Increasing from 12B to 27B in S1 yields an improvement of 6.28 percentage points with a time cost of 6.56 s (from 8.64 s to 15.20 s). In S3, the difference in effectiveness is only 1.54 percentage points with a response-time increase of 7.29 s. The efficiency of parameter scaling decreases in the context of S3. The Cogito family shows the highest benefits from parameter scaling. Increasing from 14B to 70B in S1 results in an improvement of 32.39 percentage points (from 10.51% to 42.90%). However, inference time increases more than threefold (from 6.94 s to 22.10 s). In S3, the difference in effectiveness remains at 18.52 percentage points (51.63% vs. 70.15%) with a time increase of 16.61 s. The differences between 32B and 70B are also substantial. In S2, the 70B model gains an effectiveness advantage of 9.18 percentage points with the response time extended by 11.52 s. The Cogito architecture shows stronger dependence between parameter count and diagnostic effectiveness than other model families, maintained even in advanced prompting scenarios.

The DeepSeek-R1 family implements an extended reasoning mechanism. In S1, the 70B variant achieves 27.98% effectiveness with a response time of 48.82 s. For comparison, Cogito 70B achieves 42.9% in 22.10 s. DeepSeek-R1 70B is 26.72 s slower and 14.92 percentage points less effective. In S4, DeepSeek-R1 70B achieves 63.6% in 55.02 s, while Cogito 70B achieves 64.32% in 31.32 s. The difference in effectiveness is only 0.72 percentage points against DeepSeek-R1, but its response time is 23.7 s longer. The extended reasoning mechanism provides no benefit in diagnostic tasks with incomplete clinical data. Additional computational time does not translate into improved effectiveness. In S3, the expert system provides a diagnostic reasoning structure. The additional knowledge is better utilised in Cogito 70B, surpassing DeepSeek-R1 70B by 4.95 percentage points in effectiveness and 23.94 s in response speed. The extended thinking mechanism becomes redundant, and the computational overhead does is not justified by the minimal diagnostic benefits.

Medically dedicated models show varied effectiveness relative to their general counterparts. In the GPT-OSS family, the medical model achieves 25.98% effectiveness in S1, while the general GPT-OSS 20B achieves 26.37%. The difference is 0.39 percentage points against the medical model. In S3, GPT-OSS-Med 20B achieves 68.97%, while GPT-OSS 20B achieves 70.42%, a degradation of 1.45 percentage points. The medical model is slightly faster (10.56 s vs. 13.84 s in S3) but this does not compensate for its lower diagnostic effectiveness. In the Gemma family, strong result heterogeneity is observed. The smaller MedGemma 4B achieves only 11.3% effectiveness in S1, while the general Gemma3 12B achieves 34.2%, a degradation of 22.9 percentage points with similar inference time (7.54 s vs. 8.64 s). The larger MedGemma 27B achieves 44.41% effectiveness in S1, while the general-purpose Gemma3 27B achieves 40.48%; an advantage of 3.93 percentage points for the medical model. The time cost is substantial: MedGemma 27B requires 24.59 s, while Gemma3 27B needs only 15.20 s. In S2, general models perform better than medical models in terms of both accuracy and time taken. Gemma3 12B achieves a result 14.3 percentage points higher in 30.07 s less time than MedGemma 4B. Between larger models (Gemma3 27B vs. MedGemma 27B), the difference in effectiveness is only 1.58 percentage points in favour of the general model, with execution time shorter by 49.32 s. RAG introduces substantial computational overhead for medical models. In S4, MedGemma 27B achieves 60.36% in 81.14 s, while Gemma3 27B achieves 57.13% in 24.1 s. The medical model’s advantage is 3.23 percentage points with the inference time more than three times as long.

Figure 6 presents the percentage of correct diagnoses (single TOP-3) for each model for both internal medicine and paediatrics. Paediatrics consistently shows higher correct diagnosis counts across all scenarios. In S1, MedGemma 27B achieves the largest paediatrics advantage with a difference of 22.1 percentage points. Large differences can also be seen between Gemma3 27B (17.9 pp) and Cogito 70B (15.4 pp). In S2, differences between specialisations decrease. In internal medicine, the MedGemma 4B model performs more poorly, by 7.3 percentage points, and DeepSeek-R1 14B by 7.8 percentage points. RAG reduces the disproportion between specialisations.

Figure 6. Number of correct diagnoses (single TOP-3) for individual models in four diagnostic scenarios for internal medicine and paediatrics.

The DeepSeek-R1 family exhibits the lowest effectiveness in both specialisations. In S1, DeepSeek-R1 14B achieves only 4.4% correct diagnoses in internal medicine and 5.11% in paediatrics. However, in S3 it achieves 22.4% in internal medicine and 29.5% in paediatrics. The expert system compensates for the inefficiency of the extended reasoning mechanism more effectively in paediatrics than in internal medicine. Compared to general models from the Cogito family, no particular differences were noted in the results of DeepSeek-R1. For S4, Cogito 14B achieved a result 5.3 percentage points higher than DeepSeek-R1 14B. In the 32b and 70b variants, this difference decreased to 0.5 percentage points.

Medical models show different characteristics between specialisations. In S1, the MedGemma 4B model achieves only 5.3% correct diagnoses in internal medicine and 6% in paediatrics. A substantial improvement was recorded by MedGemma 27B with results of 11.1% for internal medicine and 33.3% for paediatrics. In S3, for MedGemma 27B the difference reaches 24.2 percentage points in favour of paediatrics. For the MedGemma 4B and GPT-OSS-Med 20B models, similar behaviour can be noted with differences of 20 percentage points and 19.8 percentage points, respectively. The expert system increases medical model effectiveness in paediatrics more than in internal medicine.

3.5. Energy Efficiency of Models

To quantify the trade-off between diagnostic accuracy and computational cost, we propose the Energy Efficiency Score (Formula (1)).

$$S_{energy}(m,s)={\displaystyle \frac{A_{m,s}}{\left(\frac{{\overline{T}}_{m,s}}{\widetilde{T_s}}\right)\ast \left(\frac{{\overline{L}}_{m,s}}{\widetilde{L_s}}\right)}}$$

(1)

where

• $A_{m,s}$—diagnostic accuracy of model m in scenario s (range [0, 1]);
• ${\overline{T}}_{m,s}$—mean total token count for model m in scenario s;
• ${\overline{L}}_{m,s}$—mean inference latency (seconds) for m in scenario s;
• $\widetilde{T_s}$—median of per-model mean token counts across all models within scenario s;
• $\widetilde{L_s}$—median per-model mean inference times across all models within scenario s;

Normalisation within the scenario ensures a fair comparison. Models in token-intensive scenarios (e.g., S4) are compared with the typical cost in that scenario rather than the lower base of simpler scenarios (e.g., S1). A score of S > 1 indicates that the model provides higher diagnostic accuracy per unit of normalised computational cost than the median model in a given scenario. Bootstrap confidence intervals were calculated for all S_energy values (10,000 iterations, 95% CI). Table 6 shows the aggregated energy efficiency results for all models, aggregated across all scenarios.

Table 6. Aggregated across all scenarios. $\tilde{T}$ mean total tokens; $\tilde{L}$ mean inference time; A(g,k) grouped Top-k accuracy; S(g,k) energy efficiency score; In/Out mean input/output tokens. Sorted by S(g,1) descending.

Model	N	$\tilde{\mathit{T}}$	$\tilde{\mathit{L}}(\mathit{s})$	A(g,1)	A(g,3)	S(g,1)	S(g,3)	In/Out
Mistral-NeMo 12B	13,240	2181	6.5	0.425	0.541	1.317	1.667	1715/466
PLLuM-12B-q8-ic	13,240	2210	8.7	0.487	0.564	1.079	1.254	1715/496
Mistral-Small3.2 24B	13,240	2347	13.0	0.521	0.673	0.739	0.963	1717/630
Cogito 14B	13,240	2277	9.6	0.354	0.487	0.693	0.955	1737/540
Gemma3 12B	13,240	2448	12.0	0.427	0.584	0.624	0.859	1744/705
GPT-OSS 20B	13,240	2846	11.8	0.497	0.621	0.612	0.778	1676/1170
GPT-OSS-Med 20B	13,240	2952	12.3	0.488	0.601	0.575	0.709	1679/1273
Cogito 32B	13,240	2271	16.6	0.428	0.571	0.490	0.656	1737/534
GPT-OSS 120B	13,240	2854	16.6	0.506	0.652	0.455	0.589	1677/1177
Gemma3 27B	13,240	2461	20.2	0.469	0.638	0.408	0.557	1744/718
Qwen3 14B	13,240	2913	18.1	0.475	0.580	0.392	0.479	1739/1174
Cogito 70B	13,240	2219	27.0	0.521	0.666	0.372	0.478	1702/517
LLaMA3.1:70B	13,240	2215	26.8	0.511	0.619	0.363	0.444	1702/513
MedGemma 4B	13,219	2813	27.3	0.395	0.459	0.360	0.411	2061/752
LLaMA3.3 70B	13,240	2276	29.7	0.527	0.645	0.331	0.408	1702/574
DeepSeek-R1 14B	13,240	2710	16.2	0.311	0.436	0.314	0.442	1726/984
Qwen3 32B	13,240	2834	32.6	0.455	0.579	0.216	0.274	1739/1095
DeepSeek-R1 32B	13,240	2827	31.7	0.404	0.516	0.201	0.257	1726/1101
MedGemma 27B	13,240	3022	52.5	0.463	0.667	0.156	0.224	2061/961
DeepSeek-R1 70B	13,240	2736	51.6	0.471	0.599	0.145	0.184	1690/1045

3.6. Impact of Prompt Language on Diagnostic Effectiveness and Error Correction Capability

Table 7 presents an analysis of diagnostic errors by scenario. The sum of errors in S1 serves as the starting point for assessing compensation capability in the remaining scenarios. The DeepSeek-R1 14B model exhibited the highest error count (2994). The English version was marginally better with 20 fewer errors. Model Cogito 14B recorded 2962 errors with an English version advantage of 26 cases. The MedGemma 4B model made 2936 errors with minimal language difference (8 errors favouring EN). The Mistral-Small3.2 24B model demonstrated the lowest error count (1741), with the Polish version marginally better (19 error advantage). Medgemma:27B recorded 1840 errors with a Polish version advantage of 56 cases. Cogito 70B scored 1890 errors favouring PL by 106 cases. The highest Polish version advantage in the entire set was achieved by Gemma3 27B at 108 cases, with a total S1 error count of 1970.

Table 7. Analysis of diagnostic errors in S1 and correction capability in S2–S4. Columns present: sum of errors in S1, number of S1 cases rectified in subsequent scenarios, and difference in error count between language versions in S1 (EN-PL; positive values indicate more errors in EN version, negative values in PL version).

Model	S1 Errors *	Fixed in S2	Fixed in S3	Fixed in S4	EN-PL Diff in S1
DeepSeek-R1 14B	2994	594	1362	1260	−20
Cogito 14B	2962	722	1439	1454	−26
MedGemma 4B	2936	655	1842	1549	−8
DeepSeek-R1 32B	2867	674	1542	1607	−13
Qwen3 14B	2626	661	1640	1542	22
Cogito 32B	2553	622	1313	1251	−89
Mistral-NeMo 12B	2504	538	1424	1185	−40
GPT-OSS-Med 20B	2450	711	1612	1477	32
GPT-OSS 20B	2437	650	1614	1484	35
DeepSeek-R1 70B	2384	655	1416	1384	36
Qwen3 32B	2365	603	1138	1081	−11
LLaMA3.1 70B	2315	603	1559	1404	95
GPT-OSS 120B	2273	678	1473	1366	53
PLLuM-12B-q8-ic	2210	411	1290	1009	66
Gemma3 12B	2178	433	954	826	44
LLaMA3.3 70B	2095	521	1309	1141	47
Gemma3 27B	1970	412	813	738	108
Cogito 70B	1890	476	1114	986	106
MedGemma 27B	1840	398	884	776	56
Mistral-Small3.2 24B	1741	419	924	946	19

* Sample size N = 3310.

Introducing scenarios S2, S3, and S4 reveals varied capability of model to rectify S1 errors. Despite a high S1 error count (2936), Cogito 14B showed the highest correction capability in S2 (722 errors rectified). Second place went to GPT-OSS-Med 20B with 711 corrections. The lowest rectification counts were recorded by MedGemma 27B (398 cases), PLLuM-12b-q8-ic (411), and Gemma3 27B (412). The highest correction capability in S3 was achieved by MedGemma 4B with 1842 rectifications, the highest compensation rate in the entire set. Gemma3 27B recorded the lowest correction count in both S3 (813) and S4 (738). The highest correction capability in S4 was demonstrated by DeepSeek-R1 32B.

Figure 7 presents differences between language versions (EN-PL) in rectified error counts for each scenario. Positive values indicate an English-version advantage, negative values indicate a Polish-version advantage. Analysis reveals systematic language patterns dependent on model architecture and scenario type.

Figure 7. Language effect in diagnostic error compensation (EN-PL) for individual models in S2, S3, S4. Positive values indicate that the EN version rectified more S1 errors than the PL version. Negative values indicate an advantage for the PL version.

The Cogito family shows the most heterogeneous language profile. In S2, Cogito 70B demonstrated an EN advantage (+78). In S3, the EN advantage increased to 128 fixed errors, and in S4 it stood at 112, the highest English-version advantage in the entire set for scenarios S3 and S4. Cogito 32B showed the opposite pattern: in S2, the Polish version surpassed English by 22 corrected errors, in S3 the PL advantage was 63 errors, and in S4 the PL advantage increased to 87 errors. Cogito 14B also showed a PL advantage in S4 of 20 repairs, while in S2 and S3 EN had an advantage of 28 and 47 repairs, respectively. Similarly to Cogito 32B, Mistral-NeMo 12B rectified more errors on the PL dataset: 138 in S3 and 23 in S4. The LLaMA family shows a consistent advantage for the English version. In S2m LLaMA 3.1 70B rectified 55 more errors in the EN version. In S3, the advantage was 93 errors, and in S4 it stood at +106 errors. The LLaMA3.3 70B model exhibited a similar pattern: +57 (S2), +57 (S3), +67 (S4). A stable EN advantage across all scenarios indicates that models trained on the LLaMA architecture are more strongly aligned to English.

Despite Polish specialisation, PLLuM-12B-q8-ic showed a strong English-version advantage in error correction. In S2, it fixed 107 more errors in EN. In S3, the advantage was +72 errors, and in S4 it reached +103 errors. This behaviour indicates fundamental limitations of narrow linguistic specialisation. A model optimised for Polish made 66 fewer errors than for EN (Table 6) but showed lower error correction capability in the Polish version when using advanced prompting techniques. This suggests that the linguistic specialisation of models does not automatically translate into higher diagnostic effectiveness in that language. Prompting mechanisms and the expert system work more effectively in English regardless of the model’s native specialisation.

4. Conclusions

This study provided systematic evaluation of the diagnostic effectiveness of LLMs, RAG, expert systems, and their hybrid configurations under real clinical practice conditions. The experiments conducted covered four diagnostic scenarios tested on 1655 clinical cases from two medical specialisations. Each case was independently analysed by 20 LLMs in two language versions, yielding a total of 264,800 diagnostic processes.

The results revealed a major discrepancy between the effectiveness of standalone LLMs and results reported in standardised tests. The average TOP-1 effectiveness was only 20.19% in paediatrics and 16.08% in internal medicine in S1, representing a drastic drop relative to results reported in multiple-choice tests where GPT-4o achieved 93.3% and LLaMA 3.1 70B scored 87.5% [1]. This discrepancy confirms the findings of a meta-analysis of 83 studies showing an average LLM effectiveness of 52.1% under real clinical conditions [3]. Using unpublished cases from actual medical documentation characterised by data incompleteness revealed fundamental limitations of standalone models. The results are comparable to the LLaMA 3.1 405B study on unpublished data (45% TOP-1) [6]. A further decrease in performance is due to the incomplete information typical of the diagnostic process.

Introducing the RAG mechanism resulted in a minimal improvement in effectiveness. In paediatrics, the increase was 2.37 percentage points in Polish and 5.16 percentage points in English. In internal medicine, the improvement was limited to 0.9 and 4.58 percentage points, respectively. A systematic review of RAG applications in healthcare indicated potential for this approach [11,12], but the effectiveness of RAG proved dependent on the model’s baseline capability for clinical reasoning. Under conditions of very low standalone model effectiveness, the RAG mechanism did not provide sufficient context for substantial improvement in the diagnostic process.

The scenario using the expert system showed an average 2.5-fold increase in effectiveness relative to S1. The average TOP-1 effectiveness reached 54.82% in paediatrics and 39.37% in internal medicine. Applying ICD-10 grouping further increased effectiveness to an average of 65.83% in paediatrics and 76.6% in internal medicine. These results exceed the average non-expert physician effectiveness reported in the meta-analysis [3]. The deterministic nature of reasoning and native handling of missing values by machine learning algorithms provided an advantage under real medical practice conditions.

The most unexpected result was reduced effectiveness in the full component integration scenario. Instead of the predicted synergy, effectiveness dropped relative to the expert system alone. Detailed analysis revealed a fundamental architectural conflict between the deterministic expert system and the probabilistic inference of LLMs. The models excessively prioritised RAG results, which showed minimal standalone diagnostic value, at the expense of substantially more effective expert system recommendations. The RAG system, rather than supporting the decision process, introduced informational noise that disrupted effective inference. This result indicates that simply combining components does not guarantee collaboration and may lead to reduced effectiveness.

An analysis of secondary factors revealed several additional patterns, which are discussed in detail in Section 3. The ability to utilise external context does not correlate linearly with the performance of the base model; some weaker models demonstrated greater flexibility in integrating the expert system’s recommendations. The specialisation of a medical model does not guarantee better diagnostic performance, general-purpose models matched or outperformed the results of specialised variants in most configurations, which is consistent with the results of Polish comparative tests in medicine [13]. The effect of the number of parameters showed diminishing returns with the support of the expert system: in the GPT-OSS family, the difference in performance between the 20B and 120B variants decreased from 4.96 pp (S1) to 0.43 pp (S3), indicating that investment in larger models may be unjustified when appropriate support techniques are applied. Extended inference mechanisms (DeepSeek-R1) generated a computational load disproportionate to the diagnostic benefits. Linguistic analysis revealed heterogeneous patterns across different model families, with no consistent advantage for input data in Polish or English.

The study’s findings are of significant importance for the design of systems to support diagnostic decision-making in clinical practice. Standalone LLMs do not achieve a level of accuracy sufficient for direct diagnostic use. An accuracy rate of 16–20% poses a serious threat to patient safety, confirming the need for a doctor to retain control over the diagnostic process when using LLM models as support tools. Clinicians and system architects should therefore avoid implementing LLM models as the primary diagnostic component in real-world clinical tools, particularly when patient data is incomplete. Expert systems based on machine learning algorithms represent a more reliable solution for diagnostics based on incomplete data and should be treated as the foundation of any hybrid clinical system, rather than as an additional auxiliary tool. The RAG mechanism requires further development and integration with diverse sources of medical knowledge to achieve significant clinical value. Minimal improvements in effectiveness do not justify the additional computational burden.

Despite their limited effectiveness in terms of precise differential diagnosis, LLMs demonstrate significant clinical value in tasks related to day-to-day patient care. The PARROT AI system [21] illustrates this principle: it combines semantic analysis based on natural language processing (NLP) with a machine learning (ML)-based expert system to automate consultation documentation from symptom recognition to diagnostic suggestions based on real-world clinical records. Rather than relying on an LLM to integrate competing sources (as in the S4 scenario), PARROT AI employs a sequential pipeline in which each module operates independently, with the clinician serving as the final decision-making layer. In such applications, the model’s diagnostic imperfection does not constitute a critical limitation, as the final decision remains with the clinician.

Designing hybrid systems requires advanced mechanisms for cooperation between individual components. The observed decline in effectiveness in S4 indicates a need to investigate methods for weighting predictions from different sources and dynamically selecting the most reliable recommendations. In practice, future hybrid systems should implement forecast weighting mechanisms capable of dynamically selecting the most reliable source, rather than relying on an LLM model to integrate competing recommendations.

This study is subject to several limitations requiring attention when interpreting the results. The dataset covered only two medical specialisations with specific symptomatological patterns. Twenty open-source models were used, while commercial models may show different effectiveness. Selection was determined by medical data protection requirements preventing transmission of patient information to external services [22]. The RAG system used only similar clinical cases from the training database with no integration with clinical guidelines or the scientific literature. The S4 integration mechanism used a simple combination of components with no advanced methods. Each clinical case was tested once. A temperature above zero introduces probabilistic sampling, which may cause slight variance in the result across repeated trials. Full assessment of the stability of results would require multiple testing of each case, which with 264,800 diagnostic processes would present a substantial computational challenge. LLM generation parameters were standardised for all models, which may have limited the full potential of some architectures. The sensitivity analysis of generation parameters for selected models is a valuable direction for future research. Future studies should include variance analysis for a representative case sample.

The results indicate several key future research opportunities. Three approaches are identified as specific fusion mechanisms to be explored in future work. The first is confidence-gated Fusion, where the RAG context is only transferred to the LLM when the confidence of the expert system is low. The second approach is retrieval quality gating, i.e., RAG cases are only included in the context if they meet increased diagnostic accuracy criteria. The third is learned late fusion with a meta-classifier dynamically weighting sources based on their certainty and retrieval quality. The key verification indicator for these mechanisms will be the improvement of TOP-1 and TOP-3 relative to the S3 scenario. The RAG mechanism requires extension to diverse sources of medical knowledge. Future studies should integrate RAG with clinical guidelines, pharmacological databases, and scientific literature. Extension of research to a broader range of medical specialisations with different diagnostic complexity and symptomatological patterns is required. Future studies should cover surgical specialisations, diagnostic imaging, and emergency medicine to more fully assess the capabilities of diagnostic decision support systems. Future research should also investigate the impact of alternative explanatory field structures in the prompt on the quality of diagnostic responses. For example, fields indicating the clinical evidence used, missing data or the main rationale leading to each of the proposed diagnoses.