Abstract
Background and Objectives: To compare the diagnostic and therapeutic performance of a multimodal large language model (GPT) with that of two human evaluators in proximal femoral fracture (PFF) assessment, focusing on correctness and completeness. Materials and Methods: A consecutive series of 250 PFFs was reconstructed into standardized clinical scenarios including radiographs and structured clinical data. Each case was independently evaluated by a trauma fellow, a second-year orthopaedic resident, and GPT. Responses were scored for diagnostic and treatment correctness and completeness. Severe treatment errors were predefined as category-changing mismatches, failure to recognize an existing implant, or incorrect laterality. Results: Fully correct diagnoses were achieved in 97.2% of cases by the fellow, 44.8% by the resident, and 14.0% by GPT (p < 0.001). GPT showed lower odds of correct diagnosis compared with the resident (matched OR 0.23, 95% CI 0.14–0.37) and fellow (OR 0.005, 95% CI 0.0001–0.03), with clustering of errors in extracapsular fractures. Diagnostic completeness also differed significantly (p < 0.001), with GPT providing fully complete assessments in 60.4% of the cases (151/250). Fully correct treatment recommendations were provided in 98.8%, 81.2%, and 36.4% of cases, respectively (p < 0.001). Severe treatment errors occurred in 58.8% of GPT outputs versus 2.0% and 0% for the resident and fellow (OR vs. resident 36.5, 95% CI 13.95–135.75). Although treatment completeness was high (93.2%), 58.8% (147/250) of GPT responses were complete but incorrect. Conclusions: GPT demonstrated substantial limitations in both diagnostic and therapeutic reasoning for PFF, with frequent pattern-dependent errors and a high rate of clinically consequential treatment mismatches.
1. Introduction
Proximal femoral fractures (PFF) represent a major public health problem worldwide, particularly in the elderly population, and are associated with substantial morbidity, functional decline, and mortality [1]. One-year mortality rates after hip fracture range between 20% and 30%, with early postoperative mortality exceeding 9% at 30 days in large national registries [1].
Timely and accurate diagnosis is therefore critical: delays in fracture recognition and surgical treatment are associated with increased complications, prolonged hospitalization, and worse survival [2]. In recent years, artificial intelligence has been increasingly applied to fracture detection and classification: convolutional neural networks have demonstrated accuracies exceeding 90% for femoral neck fracture detection and classification on plain radiographs, with performance comparable to computed tomography [2,3].
These systems, however, are task-specific, require dedicated training datasets, and are primarily limited to image-based classification rather than integrated clinical decision-making [4].
Large language models (LLMs) based on generative pretrained transformers (GPT) represent a fundamentally different approach, as they can integrate textual and visual information and generate diagnostic reasoning and treatment recommendations [5]. Early exploratory studies evaluating GPT-based models in orthopaedics have reported heterogeneous diagnostic performance, ranging from moderate agreement to clearly insufficient accuracy in trauma radiograph interpretation [6]. In a comparative study on traumatic bone pathologies, GPT-4o achieved an overall diagnostic accuracy of only 11%, markedly inferior to both emergency physicians and orthopaedic specialists [6]. Conversely, other investigations have reported higher accuracy in PFF, with overall accuracies exceeding 80% but reduced sensitivity for nondisplaced fractures [7]. Studies focusing on PFF classification rather than binary detection have shown fair-to-moderate agreement (κ = 0.420) between GPT-based models and CT-based reference standards [8].
Due to these preliminary contradictory findings, the reliability of GPT-based systems in providing both accurate diagnoses and clinically appropriate treatment recommendations for PFF remains insufficiently characterized [9]. To date, no large-scale study has systematically evaluated the correctness and completeness of GPT-based diagnostic and therapeutic reasoning in PFF using real-world clinical scenarios. The use of artificial intelligence may be particularly relevant for smaller or peripheral hospitals, where on-site orthopaedic expertise may be limited and decision-support tools could play a supportive role in early fracture management [2].
Thus, we retrospectively selected a consecutive series of PFF in adults aged 65 or more treated at two participating institutions and reconstructed standardized clinical scenarios from real-world data including key clinical information and representative radiographic images. Each case was independently evaluated by a trauma fellow and a second-year orthopaedic resident and subsequently submitted to a GPT-based model using a standardized prompt.
The aims of this study were (1) to compare the diagnostic correctness and completeness of a GPT-based multimodal large language model with those of human evaluators with different levels of orthopaedic experience in PFF assessment; (2) to compare the correctness and completeness of treatment recommendations provided by a GPT-based multimodal large language model and human evaluators in PFF management.
We hypothesized that GPT-based models would achieve correct and clinically appropriate diagnoses and treatment recommendations in at least 60% of cases, an exploratory threshold informed by the heterogeneous previous studies [6,7,8], with performance comparable to that of a junior orthopaedic resident.
2. Materials and Methods
2.1. Study Design and Setting
Two institutional trauma surgery databases were retrospectively queried to identify a consecutive series of PFF (AO/OTA 31-A or 31-B) in adults aged 65 or more, treated at tertiary trauma centers in Italy between December 2019 and August 2023.
Eligible cases were required to have complete medical records, adequate radiographic documentation, a clearly defined diagnostic work-up, and a well-documented treatment pathway, with patients treated and followed within our departments. Cases with incomplete documentation, pathological fractures of oncological origin, periprosthetic fractures, or AO/OTA 31-C fractures were excluded.
Overall, 293 consecutive cases were screened (197 from Ortopedia dell’Appennino, center A, and 96 from IRCCS Istituto Ortopedico Rizzoli, center B). Of these, 43 were excluded due to incomplete documentation (n = 11), AO/OTA 31-C fractures (n = 23), periprosthetic fractures (n = 3), or pathological fractures (n = 6). The final cohort comprised 250 cases (188 from center A and 62 from center B), independently assessed by all three evaluators.
Evaluators were blinded to each other’s responses, as well as to the final diagnosis and treatment, were not involved in the patients’ care and analyzed the same data.
2.2. Clinical Scenario Construction and Imaging Data
Each PFF case was summarized in a concise yet representative clinical vignette. This introductory section was followed by a structured narrative including patient age, sex, Charlson Comorbidity Index, time from traumatic event to presentation, clinical symptoms and signs at admission, and any additional information relevant to diagnostic reasoning. This format was deliberately chosen to exploit the ability of GPT-based models to integrate textual and visual information within a single reasoning process, thereby approximating real-world clinical decision-making during initial patient assessment. For each patient, an anteroposterior pelvic radiograph and a lateral hip radiograph were provided. Images were captured from the institutional Carestream Vue PACS (Rochester, NY, USA) at native screen resolution and saved in JPEG format (approximately 180–250 kB per image, 380–600 × 250–500 pixels, width × height) with complete removal of any alphanumeric identifiers or metadata traceable to individual patients. This process resulted in 250 complete, anonymized clinical scenarios derived from real-world data.
2.3. Evaluations and Outcomes
All cases were independently evaluated by a trauma fellow and a second-year orthopaedic resident. Evaluators were blinded to each other’s responses, as well as to the final diagnosis and treatment. Each participant was asked to provide a diagnosis, including fracture classification according to the AO/OTA system (type and subgroups), and a corresponding treatment recommendation [10]. Subsequently, each case was submitted to ChatGPT (OpenAI; San Francisco, CA, USA) (https://chatgpt.com/, accessed on Monday 2 Febraury 2026). The individual case summary and the corresponding radiographic images were then entered. Images were uploaded through the standard web-based interface of GPT, without additional preprocessing or external computational enhancement. A new session was initiated for each case to ensure that responses were generated independently and were not influenced by previous interactions. Each case was queried once and a single response was retained for analysis. Because GPT was accessed through the standard ChatGPT web interface, temperature, random seed, and other sampling parameters were not available for investigator control. Each interaction began with the single standardized, unoptimized zero-shot prompt:
“As an orthopedic surgeon, I intend to use your assistance for research purposes. Assuming you are a hypothetical orthopedic surgeon, please provide the most likely diagnosis and the most appropriate treatment for each case based on the patient information I will present.”
No few-shot examples, chain-of-thought instruction, AO/OTA reference criteria, or additional prompt engineering were provided. Custom instructions and account memory were disabled during all evaluations. This approach was selected to assess performance under a simple free-form interaction. All evaluations were performed during February 2026 using the standard ChatGPT web interface under a ChatGPT Pro subscription, with automatic model selection enabled. The specific model variant assigned to each query was not recorded.
2.4. Reference Standard and Coding System
Two senior reviewers (a consultant surgeon and the department head) independently assessed all responses generated by the human evaluators and by GPT. The two senior reviewers were blinded to the source of each response during the assessment process. The scoring system was developed ad hoc for this study to reflect clinically meaningful differences in diagnostic and therapeutic reasoning. Responses were rated for: correctness (0 = incorrect, 1 = imprecise, 2 = correct); completeness (0 = incomplete, 1 = partially complete, 2 = complete). Correctness and completeness were assessed independently across four domains, using a three-point scale (0–2). Diagnostic correctness was scored as 0 for incorrect identification of the main fracture type, 1 for correct identification of the main type but incorrect subgroup classification, and 2 for correct fracture classification. For diagnosis, a correctness score of 1 was assigned when the main AO/OTA fracture type was correctly identified but the specific subgroup was incorrect (e.g., 31-A2 instead of 31-A1). Diagnostic completeness was scored as 0 for absent or insufficient diagnostic information, 1 for partially complete descriptions, and 2 for complete diagnostic descriptions, regardless of accuracy. A completeness score of 1 was assigned when the main fracture type was correctly reported but the AO/OTA subgroup was omitted. Treatment correctness was scored as 0 for inappropriate recommendations, 1 for imprecise or partially appropriate recommendations, and 2 for clinically appropriate treatment strategies. Treatment completeness was scored as 0 for absent or insufficient recommendations, 1 for partially specified therapeutic strategies, and 2 for complete treatment recommendations, independently of correctness. Multiple clinically acceptable treatment options were considered correct. All PFF were classified according to the AO/OTA system into type 31-A (extracapsular) and type 31-B (intracapsular); no type 31-C fractures were included in the present cohort [10]. Fracture displacement was recorded as a binary variable (displaced vs. nondisplaced). Intracapsular fractures were considered displaced when classified as Garden III or IV, whereas extracapsular fractures were considered displaced when surgical reduction was required prior to internal fixation [10,11]. Treatment correctness recommendations were assessed with a binary severity scale. Severe treatment errors were defined a priori as non-discretionary, potentially clinically consequential mismatches, including (A) category-changing errors (e.g., arthroplasty versus internal fixation mismatch), (B) failure to recognize a clearly existing implant or prosthesis, and (C) incorrect laterality. All other incorrect recommendations were classified as non-severe errors. Complete-but-wrong treatment recommendations were defined as responses receiving a treatment completeness score of 2 and a treatment correctness score of 0. All assessments were blinded. To ensure consistency, both senior reviewers repeated the evaluation process after a one-month interval. Discrepancies were resolved through consensus discussion, yielding a final reference evaluation. The reference standard responses were further cross-checked against the official radiology reports and the actual surgical management performed for each case, to support the clinical adjudication process. Treatment recommendations were also qualitatively reviewed for consistency with established AAOS recommendations, providing an external benchmark beyond institutional practice [12]. Although this process does not represent a formal multi-observer consensus, it provided a robust and clinically grounded reference standard, minimizing interpretative bias. Treatment appropriateness was assessed considering patient age, Charlson Comorbidity Index, radiographic findings, and the available clinical information. For displaced intracapsular fractures, both hemiarthroplasty and total hip arthroplasty were considered correct when the available information did not clearly favor either option, regardless of the procedure actually performed. Differences between fixation methods within the same therapeutic category were not classified as severe errors. Comparative analyses were then performed between machine-generated responses and those provided by the two human evaluators.
2.5. Statistical Analysis
All 250 consecutive eligible clinical scenarios were included. Based on prior orthopaedic studies, a sample-size sensitivity calculation using the exact two-sided McNemar test (α = 0.05, 80% power), assuming a 40% absolute difference in paired proportions and a conservative 100% discordance rate, indicated a minimum requirement of 49 paired cases [6,7,8]. Interobserver agreement between the two senior reviewers was assessed before consensus using unweighted Cohen’s kappa (κ), pooling all 750 responses (250 cases × three evaluators) within each scoring domain. Intraobserver agreement was assessed separately for each senior reviewer by comparing evaluations repeated after one month. Ninety-five percent confidence intervals were calculated using patient-level bootstrap resampling (20,000 replicates).
Fully correct or complete responses (score = 2 vs. 0–1) were compared using Cochran’s Q test, followed by exact McNemar tests for paired post hoc comparisons. Ordinal distributions (0–2 scale) were analyzed with the Friedman test, followed by Wilcoxon post hoc tests with Benjamini–Hochberg adjustment applied within each scoring domain. Binary and exploratory comparisons were not adjusted for multiple testing.
Prespecified subgroup analyses by fracture type and displacement were exploratory, as the sample-size calculation addressed the overall comparison rather than within-subgroup differences. Severe treatment errors and “complete-but-wrong” outputs were analyzed as paired binary outcomes using Cochran’s Q with McNemar post hoc testing. Effect sizes for paired binary comparisons were reported as matched odds ratios (OR), calculated from discordant pairs, with exact conditional 95% confidence intervals. Ninety-five percent confidence intervals for individual proportions were calculated using the Wilson score method.
Statistical analyses were performed in R (version 4.5.2) (R Foundation for Statistical Computing, Vienna, Austria).
3. Results
Pre-consensus interobserver agreement was excellent across all 750 responses, with Cohen’s κ values of 0.991 (95% CI 0.982–0.998) for diagnostic correctness, 0.988 (95% CI 0.970–1.000) for diagnostic completeness, 0.994 (95% CI 0.984–1.000) for treatment correctness, and 0.957 (95% CI 0.897–1.000) for treatment completeness.
One-month intraobserver agreement was similarly high. For the first senior reviewer, κ values were 0.993 (95% CI 0.985–1.000), 0.977 (0.952–0.995), 0.994 (0.984–1.000), and 0.957 (0.899–1.000), respectively. Corresponding values for the second senior reviewer were 0.989 (0.978–0.998), 0.988 (0.970–1.000), 0.991 (0.978–1.000), and 0.943 (0.874–0.988).
3.1. Diagnostic Performance
Fully correct diagnoses (score = 2) were achieved in 97.2% of cases by the trauma fellow (243/250), 44.8% by the resident (112/250), and 14.0% by GPT (35/250) (Table 1). Overall differences were significant (Friedman and Cochran’s Q, both p < 0.001; all post hoc p < 0.001). GPT showed significantly lower correctness (14.0%; 95% CI 10.2–18.8%) and markedly reduced odds versus both the resident (matched OR 0.23, 95% CI 0.14–0.37) and the fellow (matched OR 0.005, 95% CI 0.0001–0.03) (Table 2). Completely incorrect diagnoses occurred in 57.6% of GPT assessments versus 2.0% and 0% for the resident and fellow, respectively (Table 3). Diagnostic correctness was particularly low in extracapsular fractures (31-A), with only 4/149 (2.7%) fully correct classifications, compared with 31/101 (30.7%) in intracapsular fractures (31-B; p < 0.001).
Table 1.
Percentages of score distribution across the different measured outcomes achieved by the three evaluators.
Table 2.
Matched odds ratios (95% confidence intervals) for fully correct or complete responses. Odds ratios compare GPT with each human evaluator and were calculated from discordant case pairs (GPT-only success/comparator-only success). Success was defined as score 2, and exact conditional 95% confidence intervals are reported.
Table 3.
Diagnostic correctness, diagnostic completeness, and severe treatment errors stratified by fracture type and displacement status.
Diagnostic correctness was 14.7% (34/231) in displaced fractures and 5.3% (1/19) in nondisplaced fractures, with no significant association between displacement status and diagnostic correctness (OR 3.11, 95% CI 0.40–24.04; p = 0.488).
Diagnostic completeness also differed significantly (Friedman and Cochran’s Q, both, p < 0.001). GPT provided fully complete assessments in 60.4% of cases (151/250), compared with 100% for both human evaluators. Completeness was significantly lower (95% CI 54.2–66.3%), with reduced odds versus both the resident and fellow (matched OR 0, 95% CI 0–0.04 for both comparisons). No statistically significant association with fracture type was detected (p = 0.087) (Table 3).
3.2. Treatment Recommendation Performance
Fully correct treatment recommendations were provided in 98.8% of cases by the fellow (247/250), 81.2% by the resident (203/250), and 36.4% by GPT (91/250) (Table 1). Differences were significant (Friedman and Cochran’s Q, both p < 0.001; all post hoc p < 0.001). GPT showed significantly lower correctness (36.4%; 95% CI 30.7–42.5%) and reduced odds versus the resident (matched OR 0.21, 95% CI 0.13–0.31) and the fellow (matched OR 0.02, 95% CI 0.004–0.06) (Table 2). Severe treatment errors occurred in 58.8% of GPT cases versus 2.0% for the resident and 0% for the fellow (Cochran’s Q p < 0.001), with a matched OR of 36.5 (95% CI 13.95–135.75; exact McNemar p < 0.001) compared with the resident (Table 3). Among nondisplaced fractures, severe treatment errors occurred in 10/11 extracapsular (90.9%) and 5/8 intracapsular cases (62.5%). Severe treatment errors were predominantly category-changing mismatches (144/147, 98.0%), rather than differences between fixation methods. Three cases involved incorrect laterality (2.0%), while no severe errors were attributed to failure to recognize existing implants. Severe errors were more frequent in extracapsular fractures (135/149, 90.6%) than in intracapsular fractures (12/101, 11.9%). All five severe errors made by the resident were category-changing mismatches, whereas no severe errors occurred for the trauma fellow. All 147 severe treatment errors were also classified as complete-but-wrong, indicating that most potentially consequential treatment mismatches occurred in apparently complete recommendations.
In contrast, treatment completeness was high across evaluators (GPT: 93.2%; resident: 93.2%; fellow: 99.6%). Although overall differences were significant (Friedman and Cochran’s Q, both p < 0.001), GPT did not differ from the resident (OR 1.00, 95% CI 0.41–2.43, exact McNemar p = 1.000) but showed lower completeness than the fellow (matched OR 0, 95% CI 0–0.26; exact McNemar p < 0.001). Notably, complete-but-wrong outputs occurred in 58.8% (147/250) of GPT cases versus 6.8% (17/250) for the resident (matched OR 19.57, 95% CI 9.25–49.59; exact McNemar p < 0.001) and 1.2% (3/250) for the fellow.
4. Discussion
The study demonstrated remarkable differences between GPT and human evaluators in both diagnostic and therapeutic performance for PFF in free-form reasoning. GPT achieved fully correct diagnoses in only 14% of cases and fully correct treatment recommendations in 36.4%, with large effect sizes compared with both the resident and the trauma fellow. Diagnostic errors clustered in extracapsular fractures, with lower diagnostic correctness in extracapsular fractures. Although treatment completeness was high (93.2%), more than half of GPT outputs were either incorrect or severely erroneous, and 58.8% were “complete but wrong,” highlighting a clinically relevant dissociation between verbosity and correctness. In contrast, human evaluators consistently demonstrated high correctness with minimal severe errors. Therefore, the exploratory 60% accuracy benchmark was not reached for either diagnosis or treatment recommendations, and GPT performed substantially worse than the junior resident.
In the present study, GPT demonstrated lower diagnostic correctness (14%) than both human evaluators (97.2–44.8%), and a clear clustering of errors in extracapsular (31-A) fracture patterns. Dedicated convolutional neural networks reported accuracies exceeding 90% in femoral fracture detection and classification under controlled imaging conditions [2,3]. However, direct comparisons are limited by differences in tasks, datasets, and performance criteria, as GPT models provide broader multimodal assessments rather than task-specific image classification. Prior studies showed highly variable diagnostic performance in orthopaedic trauma [6,7,8]. Öztürk et al. reported an overall accuracy of only 11% for GPT-4o in traumatic bone pathologies, underscoring the instability of LLM performance in radiographic interpretation [6]. Conversely, Erdem et al. described overall acceptable accuracy in PFF (82.5%), but noted subtype-dependent limitations (non-displaced fractures) [7]. Similarly, Noda et al. found only moderate agreement (κ = 0.420) between GPT outputs and CT-based reference standards in lateral PFF fracture classification, highlighting inconsistency in pattern recognition [8]. Differences in classification requirements, particularly the need for accurate AO/OTA subgroup identification, imaging conditions, prompt design, and definitions of diagnostic correctness may partly explain the heterogeneous results reported across previous GPT-based studies. However, our findings extend this literature by demonstrating a pronounced, pattern-specific decline in diagnostic correctness with standard pelvic X-rays, particularly in extracapsular fractures, even when compared with a junior resident. Although diagnostic correctness differed markedly between 31-A and 31-B fractures, this association should be interpreted cautiously, as the potential contributions of case mix, image preparation, model routing, and scoring criteria cannot be independently determined. Notably, diagnostic completeness (60.4%) was substantially higher than diagnostic correctness, indicating that GPT frequently generated structurally complete but clinically inaccurate assessments. This dissociation reinforces concerns that syntactic coherence may not equate to diagnostic reliability in LLMs applied to trauma imaging.
Beyond diagnosis, our data show that GPT is not currently reliable as a treatment decision-support tool for PFF, with fully correct recommendations in only 36.4% of cases and a high rate of clinically consequential failures. Severe treatment errors occurred in 58.8% of scenarios and were overwhelmingly category-changing mismatches (98%), indicating that the dominant failure mode was not “fine-tuning” of fixation strategy but fundamental misclassification of the appropriate therapeutic pathway. This is particularly concerning because treatment recommendations represent the step at which errors translate most directly into potential harm.
Importantly, treatment completeness was high (93.2%), yet 58.8% of GPT outputs were “complete but wrong,” highlighting a clinically hazardous dissociation between apparent plausibility and correctness. This pattern is consistent with the known tendency of LLMs to generate fluent, structured responses despite insufficient grounding (“hallucination”) [4,6,7,8]. Notably, severe treatment errors occurred in 78.9% (15/19) of nondisplaced fractures, involving both extracapsular and intracapsular patterns. This finding should be interpreted cautiously given the small subgroup size. Prior orthopaedic evaluations have already shown that GPT performance in trauma contexts can be heterogeneous and, in some settings, clearly insufficient, supporting the need for extreme caution in downstream clinical tasks [6].
Compared with earlier PFF-focused reports describing overall acceptable accuracy but reduced sensitivity in specific subtypes, our findings suggest that even when diagnostic performance is not uniformly poor, therapeutic reasoning may remain unreliable and may fail [7]. This extends classification-oriented work showing only moderate agreement between GPT outputs and reference standards, by demonstrating that treatment generation adds an additional layer of risk beyond pattern recognition alone [8]. Collectively, these results reinforce that general-purpose multimodal LLMs are not equivalent to dedicated clinical decision systems and require task- and pattern-aware validation that explicitly captures error severity and “complete-but-wrong” behavior rather than relying on completeness or narrative quality as proxies for safety [4].
This study has several limitations. First, although all cases were reviewed according to routine clinical practice and adjudicated by two independent senior surgeons through consensus, the reference standard was not derived from a formal multi-institutional or blinded consensus panel. Therefore, center-specific treatment philosophies and a residual degree of interpretative variability cannot be entirely excluded. Second, GPT was evaluated using a single unoptimized zero-shot prompt. As LLM performance may vary according to prompt formulation, our findings should be interpreted as reflecting performance under this specific interaction strategy rather than the model’s maximal capability. However, the use of a simple standardized prompt was consistent with the pragmatic aim of evaluating a readily applicable decision-support approach for settings with limited specialist expertise, where complex prompt engineering would reduce usability and scalability. Furthermore, the underlying model version was not recorded for individual queries, potentially allowing variation in model routing across cases and precluding version-specific conclusions. Third, each case was queried only once, and sampling parameters could not be controlled through the standard ChatGPT web interface. Therefore, the study does not characterize test–retest variability, and individual outputs may partly reflect the stochastic nature of the model. Repeated-query studies under controlled and prespecified model conditions are needed to assess output reproducibility. Fourth, radiographic images were converted from DICOM format to JPEG at screen resolution prior to upload. This process inevitably reduced image fidelity and may not reflect optimal radiographic diagnostic conditions. However, this choice enhances the pragmatic nature of the study, as it mirrors real-world teleconsultation or decision-support scenarios rather than idealized AI training environments. Fifth, the predominance of extracapsular (31-A) fractures in the cohort may have influenced subgroup analyses. Therefore, the observed differences in performance across fracture types cannot be attributed exclusively to fracture morphology. Sixth, only GPT-based LLMs were evaluated. Performance may vary substantially across GPT model versions and evaluation settings [5]; therefore, the present findings cannot be generalized to other generative AI systems. Seventh, only two human evaluators participated in the comparative analysis. Although both represented clinically realistic levels of orthopaedic training, their performance may not fully reflect the variability expected across broader practitioner populations. Although assessors were blinded to response sources, distinctive linguistic or structural characteristics may have allowed identification of machine-generated responses, potentially introducing assessment bias. Finally, the retrospective reconstruction of clinical scenarios may not fully reproduce the information and contextual factors available during real-world clinical decision-making, potentially limiting the generalizability of our findings. Despite these limitations, this study represents, to our knowledge, the largest structured comparison between human evaluators and a multimodal LLM in PFF diagnosis and treatment planning in free-form reasoning. Importantly, it incorporates fracture pattern stratification and a severity-based analysis of treatment errors, providing clinically meaningful insight beyond simple accuracy metrics.
5. Conclusions
In conclusion, under these specific prompting and imaging conditions, GPT demonstrated substantial limitations in both diagnostic and therapeutic reasoning for PFF in free-form, with marked differences in diagnostic performance across fracture types and a high proportion of severe, category-changing treatment mismatches. The dissociation between completeness and correctness underscores the risk of relying on generative plausibility as a surrogate for clinical reliability.
Under the evaluated conditions, GPT did not demonstrate sufficient reliability for independent PFF diagnosis and treatment planning. Its potential role as a supplementary tool under specialist supervision requires further clinical validation, including fracture pattern–aware assessment and higher-fidelity imaging.
Supplementary Materials
The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/medicina62112106/s1, Supplementary Material: Example of a standard vignette submitted to the evaluators.
Author Contributions
Conceptualization, M.M.; data curation, N.V.C. and G.B.S.; formal analysis, F.C., S.S. and R.V.; investigation, N.V.C.; methodology, F.C. and F.T.; project administration, F.C. and F.T.; visualization, F.C. and R.V.; writing—original draft preparation, M.M. and F.C.; writing—review and editing, M.M., N.V.C., F.C., G.B.S., S.S., R.V. and F.T. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
This study was approved by the Institutional Review Board CE AVEC: 74/2026/OSS/IOR, date: 18 January 2026. The approved study protocol explicitly authorized the external processing of anonymized clinical information and radiographic images through the ChatGPT service. The study was conducted in accordance with the Declaration of Helsinki.
Informed Consent Statement
Patient consent was waived due to the retrospective nature of the study.
Data Availability Statement
The data presented in this study are available on reasonable request from the corresponding author.
Acknowledgments
During the preparation and revision of this manuscript, the authors used ChatGPT (OpenAI; San Francisco, CA, USA) to assist with language editing, textual restructuring, and the preparation of responses to reviewers. All AI-assisted content was critically reviewed and edited by the authors, who take full responsibility for the scientific content and conclusions of the manuscript.
Conflicts of Interest
The authors declare no conflicts of interest.
Abbreviations
The following abbreviations are used in this manuscript:
| AAOS | American Academy of Orthopaedic Surgeons |
| AI | Artificial Intelligence |
| AO/OTA | Arbeitsgemeinschaft für Osteosynthesefragen/Orthopaedic Trauma Association |
| CI | Confidence Interval |
| CT | Computed Tomography |
| DICOM | Digital Imaging and Communications in Medicine |
| GPT | Generative Pre-trained Transformer |
| JPEG | Joint Photographic Experts Group |
| LLM | Large Language Model |
| OR | Odds Ratio |
| PACS | Picture Archiving and Communication System |
| PFF | Proximal Femoral Fractures |
References
- Dijkstra, H.; Parsons, C.S.; Bremen, H.V.; Willems, H.C.; De Hond, A.A.H.; Van Munster, B.C.; Doornberg, J.N.; Oosterhoff, J.H.F. Machine learning-based prediction of short- and long-term mortality for shared decision-making in older hip fracture patients: The Dutch Hip Fracture Audit algorithms in 74,396 cases. Acta Orthop. 2025, 96, 521–528. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Sevinç, H.F.; Üreten, K.; Karadeniz, T.; Gültekin, G.K. Detection and classification of femoral neck fractures from plain pelvic X-rays using deep learning and machine learning methods. Ulus. Travma Acil Cerrahi Derg. 2025, 31, 783–788. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Yang, J.; Park, J.; Park, K.; Lim, E.J.; Kim, J.W.; Kweon, J.; Kim, C.H. Garden classification of femoral neck fracture using deep-learning algorithm. Sci. Rep. 2025, 15, 43926. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Mu, L.; Liu, Y.; Xie, Y.; Liu, H.; Liu, K.; Miao, Z.; Xue, H.; Li, M.; Dong, D.; Zhang, H. From Imaging to Intervention: A Multicenter-Validated Radiomics Pipeline for Guiding Femoral Neck Fracture Surgical Management. Acad. Radiol. 2025, 33, 1049–1059. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Goto, H.; Shiraishi, Y.; Okada, S. Performance Evaluation of GPT-4o and o1-Preview Using the Certification Examination for the Japanese ‘Operations Chief of Radiography with X-rays’. Cureus 2024, 16, e74262. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Öztürk, A.; Günay, S.; Ateş, S.; Yiğit, Y. Can Gpt-4o Accurately Diagnose Trauma X-Rays? A Comparative Study with Expert Evaluations. J. Emerg. Med. 2025, 73, 71–79. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Erdem, T.E.; Kirilmaz, A.; Kekec, A.F. Diagnostic Performance of ChatGPT-4o in Detecting Hip Fractures on Pelvic X-rays. Cureus 2025, 17, e86654. [Google Scholar] [CrossRef] [Scilit] [PubMed] [PubMed Central]
- Noda, M.; Takahara, S.; Hayashi, S.; Inui, A.; Oe, K.; Matsushita, T. Evaluating ChatGPT’s Performance in Classifying Pertrochanteric Fractures Based on Arbeitsgemeinschaft für Osteosynthesefragen/Orthopedic Trauma Association (AO/OTA) Standards. Cureus 2025, 17, e78068. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Nietsch, K.S.; Shrestha, N.; Mazudie Ndjonko, L.C.; Ahmed, W.; Mejia, M.R.; Zaidat, B.; Ren, R.; Duey, A.H.; Li, S.Q.; Kim, J.S.; et al. Can Large Language Models (LLMs) Predict the Appropriate Treatment of Acute Hip Fractures in Older Adults? Comparing Appropriate Use Criteria with Recommendations from ChatGPT. J. Am. Acad. Orthop. Surg. Glob. Res. Rev. 2024, 8, e24.00206. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Meinberg, E.G.; Agel, J.; Roberts, C.S.; Karam, M.D.; Kellam, J.F. Fracture and Dislocation Classification Compendium—2018. J. Orthop. Trauma 2018, 32, S1–S10. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Garden, R.S. Low-angle fixation in fractures of the femoral neck. J. Bone Jt. Surg. Br. 1961, 43-B, 647–663. [Google Scholar] [CrossRef] [Scilit]
- American Academy of Orthopaedic Surgeons. Appropriate Use Criteria: Hip Fractures in Older Adults: Acute Treatment; American Academy of Orthopaedic Surgeons: Rosemont, IL, USA, 2023; Available online: https://aaos.webauthor.com/go/auc/auc.cfm?auc_id=224954 (accessed on 31 August 2026).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Published by MDPI on behalf of the Lithuanian University of Health Sciences. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.