Evaluating the Reliability of ChatGPT for Health-Related Questions: A Systematic Review

Abstract

The rapid advancement of large language models like ChatGPT has significantly impacted natural language processing, expanding its applications across various fields, including healthcare. However, there remains a significant gap in understanding the consistency and reliability of ChatGPT’s performance across different medical domains. We conducted this systematic review according to an LLM-assisted PRISMA setup. The high-recall search term “ChatGPT” yielded 1101 articles from 2023 onwards. Through a dual-phase screening process, initially automated via ChatGPT and subsequently manually by human reviewers, 128 studies were included. The studies covered a range of medical specialties, focusing on diagnosis, disease management, and patient education. The assessment metrics varied, but most studies compared ChatGPT’s accuracy against evaluations by clinicians or reliable references. In several areas, ChatGPT demonstrated high accuracy, underscoring its effectiveness. However, performance varied, and some contexts revealed lower accuracy. The mixed outcomes across different medical domains emphasize the challenges and opportunities of integrating AI like ChatGPT into healthcare. The high accuracy in certain areas suggests that ChatGPT has substantial utility, yet the inconsistent performance across all applications indicates a need for ongoing evaluation and refinement. This review highlights ChatGPT’s potential to improve healthcare delivery alongside the necessity for continued research to ensure its reliability.

1. Introduction

In recent years, the emergence of large language models [1,2,3,4] has revolutionized the landscape of artificial intelligence (AI) for natural language processing (NLP) tasks among others [5]. These models, fueled by advancements in deep learning techniques and access to vast amounts of data, have demonstrated remarkable capabilities in understanding and generating human-like text [6]. Among these models, ChatGPT [1,2] has emerged as a prominent example, capturing widespread attention and adoption due to its ability to engage in coherent and contextually relevant conversations. ChatGPT was released to the public by OpenAI, an artificial intelligence research laboratory, in late 2022. It is based on a variant of the GPT (Generative Pre-trained Transformer) model, specifically fine-tuned to be capable of generating human-like responses given a prompt or query. The release of ChatGPT marked a significant milestone in the development of conversational AI, providing a powerful tool for natural language understanding and generation [7]. Since its launch, ChatGPT has gained immense popularity, becoming one of the fastest-growing consumer internet apps to date. Within just two months, it attracted an estimated 100 million monthly users, showcasing its widespread adoption. Currently, over 2 million developers are actively utilizing the company’s API, including the majority of Fortune 500 companies, highlighting its significance across various industries.

In healthcare, ChatGPT offers a wide range of opportunities, including patient education, academic research, triage, diagnosis, decision-making, clinical documentation, and trial enrollment [8,9,10].

However, there are significant concerns surrounding its potential misuse, hallucination, data privacy, and authenticity of the information it generates [11]. Particularly given the delicate and highly regulated nature of healthcare, ensuring the accuracy and reliability of the information it delivers is a primary concern among healthcare workers [12]. Inaccurate or misleading information in healthcare can have severe consequences, including misdiagnoses, improper treatments, and potential harm to patients’ well-being and safety.

Although prior research has explored ChatGPT’s potential applications and associated concerns in specific healthcare problems, there remains a notable gap in the literature concerning its actual performance in addressing broader healthcare-related inquiries. This gap underscores the need for further investigation to comprehensively evaluate ChatGPT’s effectiveness in this domain. Therefore, this study aimed to delve deeper into the literature to provide insights into the strengths, limitations, and future directions of leveraging ChatGPT in healthcare contexts.

By systematically synthesizing existing literature, this review addresses the following research questions:

• What is the overall accuracy and reliability of ChatGPT in providing health-related information across various medical fields?
• How does the context of inquiries impact the performance of ChatGPT?
• What is the perception of researchers about the use of ChatGPT in healthcare?

Ultimately, such a review holds the potential to inform stakeholders, including healthcare practitioners, researchers, policymakers, and technology developers, about the opportunities and challenges associated with integrating ChatGPT into the healthcare ecosystem.

2. Materials and Methods

This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [13]. The PRISMA framework provided a structured approach to ensure transparency, rigor, and reproducibility throughout the review process.

2.1. Search Strategy

We utilized PubMed as the sole database for identifying relevant publications. Given the focus of our systematic review on the healthcare applications of ChatGPT, we prioritized a database that would most likely include studies of high relevance to our topic. We used only the term “ChatGPT” to ensure a comprehensive and unbiased search without applying any additional search restrictions. The search, conducted on 15 December 2023, targeted publications from 2023 onwards with “ChatGPT” in the title or abstract, resulting in 1940 articles. Out of these articles, we excluded 839 papers with missing abstracts, resulting in a total of 1101. These articles underwent a rigorous screening process based on their titles and abstracts, aimed at excluding studies that did not evaluate ChatGPT answers, used unquantifiable metrics, or engaged only in theoretical discussions about the potential of chatbots for healthcare. The PRISMA diagram in Figure 1 illustrates the overall process.

Figure 1. PRISMA diagram showing the stages from initial search to final inclusion, starting with 1101 articles identified on PubMed and concluding with 128 studies included in the review after detailed screening and eligibility checks.

2.2. Study Selection and Screening

The study selection process began with an initial screening of titles and abstracts using an innovative methodology that leveraged ChatGPT. A specific prompt was engineered to guide ChatGPT in identifying studies relevant to healthcare applications of ChatGPT, ensuring comprehensive coverage without missing any relevant studies (Figure 2). This innovative method emphasized achieving 100% recall, ensuring comprehensive coverage without falsely excluding any relevant studies. To validate this, two reviewers independently assessed a random sample of 100 papers confirming the recall/sensitivity of ChatGPT’s selections and ensuring that all relevant studies were accurately identified. This automated screening (part-A) excluded 625 articles, leaving 476 for further evaluation.

Figure 2. Prompt used to utilize ChatGPT for reviewing the titles and abstracts of studies retrieved from PubMed. This prompt was carefully engineered to maximize the inclusion of eligible studies, even those with a low likelihood of relevance, while minimizing the risk of false exclusion. It included detailed inclusion and exclusion criteria, along with step-by-step instructions, to guide ChatGPT through the screening process, ensuring a thorough and consistent review of the studies.

Next, eight human reviewers conducted a secondary screening of the remaining articles (part-B). The articles were systematically divided among four pairs of reviewers, with each pair independently reviewing their assigned articles. This process involved a detailed assessment of titles and abstracts and, where necessary, a review of the full texts. Standardized assessment forms were employed to evaluate each study, and inter-rater reliability checks were performed to ensure consistency between reviewers. Discrepancies within each pair were resolved through discussion and consensus, providing a robust and unbiased selection process. This screening resulted in the exclusion of 263 articles, leaving 213 for further evaluation.

2.2.1. Inclusion and Exclusion Criteria

The inclusion and exclusion criteria ensured that only empirical and validated studies were included, maintaining the scientific rigor of the review. Refer to Table 1 for a detailed breakdown of the criteria.

Table 1. A complete breakdown of inclusion and exclusion criteria used to screen papers related to the study research topic.

Inclusion Criteria

Studies were included if they met the following criteria: Utilized ChatGPT to answer health-related questions. Applied ChatGPT to healthcare problems. Were peer-reviewed journal articles.

Exclusion Criteria

Studies were excluded if they met the following criteria: Did not validate chatbot answers through healthcare professionals or authoritative reference materials. Used unquantifiable metrics to evaluate chatbots (e.g., vague claims such as “ChatGPT is harmful” or “potentially useful”). Used only statistical significance. Utilized ChatGPT to take medical exams like USMLE. Were theoretical papers without empirical data. Were non-article types, including notes, letters, reviews, short surveys, book chapters, conference papers, conference reviews, editorials, errata, research support, and news articles. Were not published in English.

Out of the 213 articles assessed for eligibility, 85 were excluded, leaving 128 studies for inclusion in the systematic review. These studies were then analyzed to evaluate how ChatGPT was queried and assessed. The analysis included a thematic synthesis to identify common themes and patterns, as well as statistical methods to quantitatively summarize the findings where applicable. This comprehensive approach aimed to offer a nuanced understanding of the current landscape and potential future directions for ChatGPT’s use in healthcare.

2.2.2. Data Extraction

Data extraction was conducted independently by eight reviewers, with the included studies divided among them. Due to the large number of studies, data extraction was not performed in pairs but was instead individually assigned to each reviewer using a pre-designed data extraction form to ensure consistency and accuracy. The following data were extracted from each included study:

• Authors: The names of the authors of the study.
• Publication Year: The year in which the study was published.
• Medical Domain: The specific health/medical areas that the study inquires about or investigates.
• Type of Inquiries: The type of data used in the study to make the inquiry (e.g., questions, case scenarios/vignette).
• Number of Entries: The total number of inquiries asked from ChatGPT to evaluate its accuracy.
• Context of Inquiries: The context in which the questions were asked (e.g., diagnosis, education).
• Authors’ Perception: The authors’ perspective or perception about their evaluation results.
• ChatGPT Version: The version of ChatGPT that was used or evaluated in the study.
• Benchmark Reference: The benchmark or standard against which the ChatGPT-generated answers were compared (e.g., clinicians, guidelines).
• Evaluation Metrics: The metrics used to quantify the performance of ChatGPT in answering the inquiries (e.g., precision, recall).

Most studies used Likert-like scales to measure the accuracy of ChatGPT responses. However, the scales varied significantly—some used three-point, four-point, or five-point scales, among others, with starting points being either 0 or 1. Other studies utilized standard performance metrics such as accuracy, recall, precision, and specificity. This variability influenced the significance of each Likert point, making direct comparisons across the studies challenging. To address this issue, we introduced a new measure that we named adjusted accuracy, which normalizes all evaluation metrics to a uniform 0–100% scale. This adjustment accounts for differences in scale range and starting points, enabling a meaningful synthesis of results across diverse evaluation strategies. Adjusted accuracy is defined in Equation (1).

$$\mathit{Adjusted} \mathit{Accuracy}={\displaystyle \frac{L-S_{min}}{S_{max}-S_{min}}}$$

(1)

where L is the reported average Likert value associated with accuracy by a given study, and $S_{min}$ and $S_{max}$ are the minimum and maximum possible scores on the Likert scale. For studies that used traditional accuracy metrics, we simply took the accuracy percentage reported. To avoid bias, we considered 0 as the minimum value during standardization, as some studies used scales starting from 1, which would not equate to 0% accuracy when standardized.

Due to the division of studies among reviewers, any discrepancies within a reviewer’s extracted data were addressed individually, and additional reviewers were consulted if needed.

2.3. Synthesis of Data

Given the significant heterogeneity in the data, we decided not to perform a meta-analysis. Some categories had sparse data—with only one or two entries—and pooling such data could lead to biased or misleading results. Instead, we opted for a boxplot-based approach to provide a transparent and descriptive synthesis of the data. This method allowed us to visualize central tendencies and variability across categories while respecting the dataset’s limitations. It effectively highlights trends without imposing assumptions that might not hold under these conditions. Additionally, boxplots serve as an excellent exploratory tool for analyzing heterogeneous data and identifying outliers or patterns.

3. Results

3.1. Study Selection

In the initial PubMed search, a total of 1101 articles met the inclusion criteria based on their titles. Following the screening process described in the PRISMA diagram (Figure 1), 625 articles were excluded through automated screening using ChatGPT. These were identified as irrelevant, unquantifiable in their metrics, or solely theoretical in nature.

Subsequently, the remaining 476 articles underwent a secondary screening by human reviewers. This rigorous process, involving detailed assessments of titles, abstracts, and, where necessary, full texts, led to the exclusion of 263 additional articles. The most common reasons for exclusion at this stage were the lack of empirical data and failure to validate ChatGPT’s responses against a recognized standard. Thus, 213 articles were assessed for full-text eligibility, and 83 of these were further excluded based on the pre-defined inclusion and exclusion criteria (Table 1). Ultimately, 128 studies met all eligibility criteria and were included in the final systematic review.

3.2. Characteristics of Included Studies

A total of 128 studies was included in this systematic review, covering a wide range of medical specialties and inquiry types. The studies were published between 2023 and 2024 and focused on evaluating ChatGPT’s performance in healthcare-related tasks. Table 2 summarizes the key characteristics of the included studies, detailing the year, medical domain, type of inquiries, number of entries, context of inquiries, authors’ perception of ChatGPT’s performance, version of ChatGPT used, and the comparison benchmarks.

Table 2. A summary of the 128 studies included in the systematic review. It shows the authors, publication year, medical field, type of inquiries addressed by ChatGPT, the number of entries analyzed, context of the questions, authors’ perception of ChatGPT’s performance, version of ChatGPT used, and the comparison benchmark for the answers provided.

Author	Year	Medical Domain	Type of Inquiries	No. of Entries	Context of Inquiries	Author’s Perception	Version	Benchmark Reference
Yeo, Yee Hui et al. [14]	2023	Cancer and Oncology	Questions	164	Diagnosis	Positive	4	Clinicians
Moise, Alexander et al. [15]	2023	Otolaryngology (ENT) and Head & Neck Surgery	Questions	23	Disease management	Positive	3.5	Authoritative Reference
Fuchs, Alexander et al. [16]	2023	Dental and Oral Health	Questions	60	Education	Negative	4	Authoritative Reference
Cinar, Cigdem et al. [17]	2023	Orthopedics and Musculoskeletal Disorders	Questions	72	Disease management	Positive	Not Reported	Authoritative Reference
Geetha, Saroja D et al. [18]	2023	Cancer and Oncology	Questions	258	Education	Negative	4	Clinicians
Barlas, Tugba et al. [19]	2023	Endocrinology and Metabolic Diseases	Questions	20	Disease management	Negative	3.5	Authoritative Reference
Gan, Rick Kye et al. [20]	2023	Emergency Medicine and Trauma	Case Scenarios/Vignette	15	Public health/epidemiology	Positive	3.5	Clinicians
Lechien, Jerome R. et al. [21]	2023	Otolaryngology (ENT) and Head & Neck Surgery	Case Scenarios/Vignette	40	Diagnosis	Positive	Not Reported	Clinicians
Barash, Yiftach et al. [22]	2023	Radiology and Imaging	Case Scenarios/Vignette	40	Medication management	Positive	4	Clinicians
Suárez, Ana et al. [23]	2024	Dental and Oral Health	Questions	60	Disease management	Neutral	4	Clinicians
Mago, Jyoti et al. [24]	2023	Dental and Oral Health	Questions	80	Education	Positive	3.5	Other
Antaki, Fares et al. [25]	2023	Ophthalmology and Vision Science	Questions	260	Education, Disease management	Positive	3.5	Clinicians, Authoritative Reference
Schulte, Brian et al. [26]	2023	Cancer and Oncology	Questions	51	Diagnosis	Positive	Not Reported	Authoritative Reference
Bellinger, Jeffrey R. et al. [27]	2023	Otolaryngology (ENT) and Head & Neck Surgery	Questions	30	Education	Negative	3.5	Other
Zhou, Zeyu et al. [28]	2023	Digestive System Diseases	Questions	12	Diagnosis, Disease management	Positive	4	Authoritative Reference
Rogasch, Julian M.M. et al. [29]	2023	Radiology and Imaging	Questions	25	Disease management	Positive	4	Clinicians
Buhr, Christoph Raphael et al. [30]	2023	Otolaryngology (ENT) and Head & Neck Surgery	Questions	41	Education	Positive	4	Clinicians
Duey, Akiro H. et al. [31]	2023	Orthopedics and Musculoskeletal Disorders	Questions	12	Disease management	Positive	4	Clinicians, Authoritative Reference
Sultan, Iyad et al. [32]	2023	Cancer and Oncology	EHR Data	53	Diagnosis	Positive	3.5	Authoritative Reference
Wei, Kimberly et al. [33]	2024	Cancer and Oncology	Questions	49	Education	Negative	4	Authoritative Reference
Franco D’Souza, Russell et al. [34]	2023	Psychiatry and Mental Health	Questions	100	Diagnosis	Positive	3.5	Clinicians
Kao, Hsu-Ju et al. [35]	2023	Miscellaneous	Case Scenarios/Vignette	300	Diagnosis	Positive	Not Reported	Clinicians
Alan, Raif et al. [36]	2023	Dental and Oral Health	Questions	16	Diagnosis	Neutral	4	Clinicians
Mondal, Himel et al. [37]	2023	Miscellaneous	Case Scenarios/Vignette	20	Education, Diagnosis, Disease management	Positive	3.5	Clinicians
Kirk, Daniel et al. [38]	2023	Nutrition and Dietetics	Questions	8	Education	Positive	3.5	Clinicians
Köroğlu, Ekin Y et al. [39]	2023	Endocrinology and Metabolic Diseases	Questions	55	Disease management	Positive	Not Reported	Clinicians
Ayoub, Marc et al. [40]	2023	Emergency Medicine and Trauma	Questions	9	Diagnosis	Positive	4	Clinicians
Hirosawa, Takanobu et al. [41]	2023	Miscellaneous	Questions	30	Diagnosis	Positive	3.5	Clinicians
Cakir, Hakan et al.  [42]	2023	Urology and Nephrology	Questions	93	Diagnosis	Positive	Not Reported	Authoritative Reference
Qu, Roy W. et al. [43]	2023	Otolaryngology (ENT) and Head & Neck Surgery	Questions	20	Diagnosis	Positive	4	Clinicians
Al-Dujaili, Zahraa et al. [44]	2023	Clinical Pharmacy and Pharmacology	Questions	20	Diagnosis	Positive	Not Reported	Authoritative Reference
Hsu, Hsing-Yu et al. [45]	2023	Clinical Pharmacy and Pharmacology	Questions	80	Medication management	Negative	3.5	Clinicians
Kaarre, Janina et al. [46]	2023	Orthopedics and Musculoskeletal Disorders	Questions	20	Education	Positive	4	Clinicians
Choo, Jeong Min et al. [47]	2024	Cancer and Oncology	EHR Data	30	Disease management	Positive	3.5	Clinicians
Roosan, Don et al. [48]	2023	Clinical Pharmacy and Pharmacology	Case Scenarios/Vignette	39	Medication management	Positive	4	Authoritative Reference
Sütcüoğlu, Bengü Mutlu et al. [49]	2023	Endocrinology and Metabolic Diseases	Questions	25	Disease management	Neutral	3.5	Clinicians
Cocci, Andrea et al. [50]	2023	Urology and Nephrology	EHR Data	100	Diagnosis	Negative	3.5	Clinicians
Wagner, Matthias W. et al. [51]	2023	Radiology and Imaging	Questions	88	Other	Negative	3.5	Authoritative Reference
Daher, Mohammad et al. [52]	2023	Orthopedics and Musculoskeletal Disorders	Case Scenarios/Vignette	29	Diagnosis, Disease management	Neutral	Not Reported	Clinicians
Wang, Guoyong et al. [53]	2023	Infectious Diseases and Immunology	Questions	487	Public health/epidemiology	Neutral	4	Clinicians
Sarbay, İbrahim et al. [54]	2023	Miscellaneous	Case Scenarios/Vignette	50	Other	Positive	Not Reported	Clinicians
Kim, Hyun-Woo et al. [55]	2023	Neurological and Neurosurgical Conditions	Questions	57	Education, Diagnosis, Disease management, Medication management	Positive	4	Clinicians
Høj, Simon et al. [56]	2023	Otolaryngology (ENT) and Head & Neck Surgery	Questions	20	Education, Diagnosis, Disease management, Medication management	Neutral	Not Reported	Clinicians
Rosen, Shani et al. [57]	2024	Radiology and Imaging	Case Scenarios/Vignette	97	Disease management	Positive	3.5	Authoritative Reference
Liu, Jiayu et al. [58]	2023	Neurological and Neurosurgical Conditions	Questions	50	Other	Positive	4	Authoritative Reference
Mishra, Akash et al. [59]	2023	Neurological and Neurosurgical Conditions	Questions	40	Education	Neutral	3.5	Other
Biswas, Sayantan et al. [60]	2023	Ophthalmology and Vision Science	Questions	11	Other	Positive	3.5	Clinicians
Chen, Jingfang et al. [61]	2023	Miscellaneous	Case Scenarios/Vignette	147	Diagnosis	Neutral	4	Clinicians
Walker, Harriet Louise et al. [62]	2023	Digestive System Diseases	Questions	36	Other	Positive	4	Authoritative Reference
Rizwan, Ayesha et al. [63]	2023	Cardiovascular Diseases	Case Scenarios/Vignette	10	Diagnosis, Disease management	Positive	Not Reported	Clinicians
Gabriel, Joseph et al. [64]	2023	Urology and Nephrology	Questions	14	Education	Positive	3.5	Clinicians
Huang, Xiaoru et al. [65]	2024	Clinical Pharmacy and Pharmacology	Questions	30	Education, Medication management	Neutral	Not Reported	Clinicians
Liu, Hilary Y. et al. [66]	2023	Gynecology and Obstetrics	Questions	20	Education	Positive	3.5	Clinicians
Dhanvijay, Anup Kumar D et al.  [67]	2023	Physiology	Case Scenarios/Vignette	77	Education, Diagnosis	Neutral	3.5	Clinicians
Kusunose, Kenya et al. [68]	2023	Cardiovascular Diseases	Questions	31	Disease management	Positive	3.5	Clinicians
Krusche, Martin et al. [69]	2023	Orthopedics and Musculoskeletal Disorders	Case Scenarios/Vignette	600	Diagnosis	Positive	4	Clinicians
Babayiğit, Osman et al. [70]	2023	Dental and Oral Health	Questions	70	Education	Positive	3.5	Clinicians
Cankurtaran, Rasim Eren et al. [71]	2023	Digestive System Diseases	Questions	20	Education, Diagnosis, Disease management	Neutral	4	Clinicians
Alessandri-Bonetti, Mario et al. [72]	2023	Surgery	Questions	15	Education	Positive	3.5	Clinicians
Janopaul-Naylor, James R. et al. [73]	2023	Cancer and Oncology	Questions	117	Education	Positive	3.5	Other
Al-Ashwal, Fahmi Y et al. [74]	2023	Clinical Pharmacy and Pharmacology	Case Scenarios/Vignette	255	Medication management	Positive	4	Authoritative Reference
Lim, Zhi Wei et al. [75]	2023	Ophthalmology and Vision Science	Questions	31	Disease management	Positive	4	Clinicians
Barclay, Kayson S. et al. [76]	2023	Ophthalmology and Vision Science	Questions	10	Education	Neutral	4	Clinicians
Durairaj, K. Kay et al. [77]	2023	Otolaryngology (ENT) and Head & Neck Surgery	Questions	6	Education	Positive	3.5	Clinicians
Levartovsky, Asaf et al. [78]	2023	Digestive System Diseases	Case Scenarios/Vignette	20	Disease management	Positive	4	Clinicians
Lyu, Qing et al. [79]	2023	Radiology and Imaging	EHR Data	138	Disease management	Positive	4	Clinicians
O’Hagan, Ross et al. [80]	2023	Dermatology and Skin Conditions	Questions	25	Education	Positive	4	Clinicians
Bushuven, Stefan et al. [81]	2023	Emergency Medicine and Trauma	Case Scenarios/Vignette	22	Disease management	Negative	4	Authoritative Reference
Hermann, Catherine E. et al. [82]	2023	Cancer and Oncology	Questions	64	Disease management	Positive	Not Reported	Clinicians
Gebrael, Georges et al. [83]	2023	Cancer and Oncology	EHR Data	56	Diagnosis	Positive	4	Clinicians
Cheong, Ryan Chin Taw et al. [84]	2023	Sleep Medicine	Questions	46	Education	Positive	3.5	Authoritative Reference
Dallari, Virginia et al. [85]	2023	Otolaryngology (ENT) and Head & Neck Surgery	Questions	20	Disease management	Negative	Not Reported	Clinicians
Athavale, Anand et al. [86]	2023	Cardiovascular Diseases	Questions	40	Disease management	Positive	4	Clinicians
Nikdel, Mojgan et al. [87]	2023	Ophthalmology and Vision Science	Questions	28	Education	Positive	4	Clinicians
Stevenson, Emma et al. [88]	2023	Endocrinology and Metabolic Diseases	Case Scenarios/Vignette	15	Disease management	Neutral	3.5	Clinicians
Nazario-Johnson, Lleayem et al. [89]	2023	Neurological and Neurosurgical Conditions	Case Scenarios/Vignette	147	Diagnosis	Positive	3.5	Authoritative Reference
Gorelik, Yuri et al. [90]	2023	Digestive System Diseases	Case Scenarios/Vignette	20	Disease management	Positive	4	Clinicians, Authoritative Reference
Christy, Michele et al. [91]	2023	Orthopedics and Musculoskeletal Disorders	Questions	30	Disease management	Neutral	3.5	Clinicians
Xie, Yi et al. [92]	2023	Gynecology and Obstetrics	Case Scenarios/Vignette	6	Disease management	Neutral	Not Reported	Clinicians
Campbell, Daniel J. et al. [93]	2023	Endocrinology and Metabolic Diseases	Questions	30	Education	Positive	Not Reported	Authoritative Reference
Whiles, Bristol B. et al. [94]	2023	Urology and Nephrology	Case Scenarios/Vignette	13	Disease management	Neutral	4	Clinicians
Rahsepar, Amir Ali et al. [95]	2023	Cancer and Oncology	Questions	40	Disease management	Neutral	3.5	Clinicians
Sorin, Vera et al. [96]	2023	Cancer and Oncology	Case Scenarios/Vignette	10	Disease management	Positive	3.5	Clinicians
Mira, Felipe Ahumada et al. [97]	2023	Sleep Medicine	Questions	10	Diagnosis, Disease management	Positive	3.5	Clinicians
Gan, Rick Kye et al. [20]	2024	Emergency Medicine and Trauma	Case Scenarios/Vignette	15	Public health/epidemiology	Neutral	3.5	Clinicians
Morath, Benedict et al. [98]	2023	Clinical Pharmacy and Pharmacology	Questions	50	Medication management	Negative	3.5	Authoritative Reference
Balel, Yunus et al. [99]	2023	Dental and Oral Health	Questions	120	Education, Disease management	Positive	Not Reported	Clinicians
Anastasio, Albert Thomas et al. [100]	2023	Orthopedics and Musculoskeletal Disorders	Questions	5	Education, Disease management	Neutral	Not Reported	Other
Scquizzato, Tommaso et al. [101]	2023	Cardiovascular Diseases	Questions	40	Education	Positive	Not Reported	Clinicians
Horiuchi, Daisuke et al. [102]	2023	Neurological and Neurosurgical Conditions	Case Scenarios/Vignette	100	Diagnosis	Positive	4	Authoritative Reference
Hurley, Eoghan T. et al. [103]	2023	Orthopedics and Musculoskeletal Disorders	Questions	Not Reported	Education, Disease management	Positive	Not Reported	Authoritative Reference
Potapenko, Ivan et al. [104]	2023	Ophthalmology and Vision Science	Questions	20	Education	Neutral	4	Clinicians
Abi-Rafeh, Jad et al. [105]	2023	Orthopedics and Musculoskeletal Disorders	Case Scenarios/Vignette	16	Disease Management	Neutral	Not Reported	Other
Suthar, Pokhraj P. et al. [106]	2023	Infectious Diseases and Immunology	Case Scenarios/Vignette	44	Diagnosis, Disease management	Negative	4	Clinicians
Abi-Rafeh, Jad et al. [107]	2023	Surgery	Case Scenarios/Vignette	16	Disease management	Neutral	Not Reported	Clinicians
Suthar, Pokhraj P. et al. [106]	2023	Neurological and Neurosurgical Conditions	Case Scenarios/Vignette	140	Diagnosis	Positive	4	Authoritative Reference
Hernandez, Carlos A et al. [108]	2023	Endocrinology and Metabolic Diseases	Questions	70	Education	Positive	3.5	Clinicians
Juhi, Ayesha et al. [109]	2023	Clinical Pharmacy and Pharmacology	Questions	40	Medication management	Neutral	Not Reported	Authoritative Reference
Deiana, Giovanna et al. [110]	2023	Vaccinology	Questions	11	Public health/epidemiology	Positive	4	Authoritative Reference
Copeland-Halperin, Libby R. et al. [111]	2023	Gynecology and Obstetrics	Questions	20	Education	Neutral	Not Reported	Clinicians
Braun, Eva-Marie et al. [112]	2023	Gynecology and Obstetrics	Case Scenarios/Vignette	10	Diagnosis, Disease management	Positive	Not Reported	Authoritative Reference
Pagano, Stefano et al. [113]	2023	Orthopedics and Musculoskeletal Disorders	EHR Data	100	Diagnosis, Disease management	Positive	4	Clinicians
Caglar, Ufuk et al. [114]	2023	Urology and Nephrology	Questions	137	Education	Positive	3.5	Authoritative Reference
Coskun, Burhan et al. [115]	2023	Cancer and Oncology	Questions	59	Education	Neutral	Not Reported	Authoritative Reference
Wang, G. et al. [116]	2023	Surgery	Questions	6	Diagnosis	Positive	4	Authoritative Reference
Shao, Chen-ye et al. [117]	2023	Surgery	Questions	37	Education	Positive	Not Reported	Other
Pushpanathan, Krithi et al. [118]	2023	Ophthalmology and Vision Science	Questions	37	Diagnosis, Disease management	Positive	4	Clinicians
Kuroiwa, Tomoyuki et al. [119]	2023	Orthopedics and Musculoskeletal Disorders	Questions	5	Diagnosis	Neutral	3.5	Clinicians
Gordon, Emile B. et al. [120]	2023	Radiology and Imaging	Questions	22	Education	Positive	3.5	Clinicians
Lyons, Riley J. et al. [121]	2023	Ophthalmology and Vision Science	Case Scenarios/Vignette	44	Diagnosis, Disease management	Positive	4	Other
Haemmerli, Julien et al. [122]	2023	Cancer and Oncology	Case Scenarios/Vignette	10	Diagnosis, Disease management	Positive	3.5	Clinicians
Jazi, Amir Hossein Davarpanah et al. [123]	2023	Endocrinology and Metabolic Diseases	Case Scenarios/Vignette	10	Disease management	Positive	4	Clinicians
Zúñiga Salazar, Gabriel et al. [124]	2023	Emergency Medicine and Trauma	Questions	176	Public health/epidemiology	Positive	3.5	Clinicians, Authoritative Reference
Vaira, Luigi Angelo et al. [125]	2023	Dental and Oral Health	Questions	159	Diagnosis, Disease management	Positive	4	Clinicians
Ferro Desideri, Lorenzo et al. [126]	2023	Ophthalmology and Vision Science	Questions	28	Education, Disease management	Positive	3.5	Clinicians
TANAKA, Orlando Motohiro et al. [127]	2023	Dental and Oral Health	Questions	45	Education	Positive	4	Clinicians
Balas, Michael et al. [128]	2023	Dental and Oral Health	Questions	20	Education, Disease management	Positive	3.5	Clinicians
Crook, Bryan S. et al. [129]	2023	Orthopedics and Musculoskeletal Disorders	Questions	Not Reported	Disease management	Neutral	3.5	Authoritative Reference
Banerjee, Arijita et al. [130]	2023	Physiology	Questions	82	Education	Negative	3.5	Authoritative Reference
Pugliese, Nicola et al. [131]	2023	Digestive System Diseases	Questions	15	Disease management	Neutral	3.5	Authoritative Reference
Davis, Ryan J. et al. [132]	2023	Cancer and Oncology	Case Scenarios/Vignette	15	Disease management	Neutral	3.5	Clinicians
Jiao, Cheng et al. [133]	2023	Ophthalmology and Vision Science	Case Scenarios/Vignette	52	Education, Diagnosis	Positive	4	Authoritative Reference
Draschl, Alexander et al. [134]	2023	Orthopedics and Musculoskeletal Disorders	Questions	27	Education, Diagnosis, Disease management	Positive	3.5	Authoritative Reference
Davis, Ryan et al. [135]	2023	Urology and Nephrology	Questions	18	Diagnosis, Disease management	Positive	3.5	Clinicians
Bernstein, Isaac A. et al. [136]	2023	Ophthalmology and Vision Science	Questions	200	Other	Positive	3.5	Clinicians
Benary, Manuela et al. [137]	2023	Cancer and Oncology	Case Scenarios/Vignette	10	Disease management, Other	Neutral	4	Clinicians
Wilhelm, Theresa Isabelle et al. [138]	2023	Miscellaneous	Case Scenarios/Vignette	60	Disease management	Neutral	4	Clinicians
Pan, Alexander et al. [139]	2023	Cancer and Oncology	Questions	100	Public health/epidemiology	Neutral	3.5	Authoritative Reference

3.3. Thematic Synthesis and Authors’ Perceptions

The thematic synthesis of the studies shows that the highest frequency of research was conducted in the fields of cancer and oncology [14,26,47], orthopedics and musculoskeletal disorders [31,113], and ophthalmology and vision science [75,87,118,121] (Figure 3). In these fields, the majority of authors had a positive perception of ChatGPT’s performance. Other medical fields, such as nutrition and dietetics [38], psychiatry and mental health [34], dermatology and skin conditions [80], and vaccinology [110], were studied less frequently, with all reported studies indicating positive perceptions.

Figure 3. A stacked bar chart showing the frequency of studies across medical fields and authors’ perceptions of ChatGPT’s performance.

In contrast, fields such as otolaryngology (ENT) and head and neck surgery [15,21,27,30,43,56,77,85], clinical pharmacy and pharmacology [44,45,48,65,98,109], and emergency medicine and trauma [20,40,81,140] displayed a more varied range of author perceptions, with studies reporting positive, neutral, and negative outcomes (Figure 3). Overall, the data suggest a general trend of positive author perceptions across a wide range of medical specialties, though the extent of research and perception varies by field.

3.4. Performance Across Medical Domains

ChatGPT’s performance across different medical domains was evaluated using the adjusted accuracy metric (Equation (1)). The overall mean accuracy of ChatGPT was 73.4%, with a 95% confidence interval ranging between 70.3% and 76.5%. The results, depicted in Figure 4, reveal a wide variation in performance depending on the medical specialty. Psychiatry and mental health [34], along with dermatology and skin conditions [80], emerge as the areas where ChatGPT exhibits the highest accuracy, with relatively consistent results and minimal outliers.

Figure 4. A boxplot showing the adjusted accuracy of ChatGPT responses across various medical domains. In addition to the median, the mean is highlighted using a star for each medical domain box.

In contrast, specialties such as cardiovascular diseases [63,68,86,101], neurological and neurosurgical conditions [59,89,102,106], and cancer and oncology [18,33,95,115,139] present more variability in accuracy. These fields display a broader range of performance, with several outliers indicating instances where ChatGPT’s responses are less reliable. This suggests that, while ChatGPT can perform effectively in some medical domains, its accuracy is less consistent in more complex or specialized fields, reflecting the challenges in achieving uniform performance across diverse medical topics.

3.5. Performance Across Inquiry Contexts

ChatGPT’s performance was further analyzed across different contexts of inquiries, such as diagnosis, disease management, education, and medication management. The boxplot in Figure 5 illustrates the adjusted accuracy of ChatGPT responses across these contexts. Educational inquiries show relatively higher accuracy compared to more complex contexts like disease management and diagnosis. Specific outliers are observed in contexts such as diagnosis and public health/epidemiology, indicating areas where ChatGPT’s performance is less consistent. This variability highlights the need for the continued refinement of large language models to address the challenges in more intricate medical fields.

Figure 5. Adjusted accuracy of ChatGPT responses across various inquiry contexts, with the mean highlighted using a star for each context box.

3.6. Performance Across ChatGPT Versions

The analysis of ChatGPT versions revealed a relatively balanced representation, with version 3.5 being utilized in 40.63% of the cases, while version 4.0 was employed in 38.28% of the studies, and 21.09% of the cases did not report the version used. To determine whether there was a significant difference in performance between these versions, we conducted a Mann–Whitney U test on the evaluated accuracy scores. The test yielded a p-value of 0.96, indicating no statistically significant difference in performance between the two versions overall.

3.7. Benchmark References and Inquiry Types

The studies used various benchmarks to evaluate ChatGPT’s performance, with clinicians serving as the primary reference in 62.5% of cases (Figure 6). Authoritative references were used in 28.1% of the studies, while other references accounted for 6.2%. The types of inquiries posed to ChatGPT predominantly consisted of direct questions (68.0%), followed by case scenarios/vignettes (27.3%), and electronic health record (EHR) data (4.7%).

Figure 6. Distribution of benchmark references and inquiry types used in performance evaluation of ChatGPT. Note that outliers in the boxplot have been hidden to better show the distribution differences.

The boxplot on the right side of (Figure 6) provides further insight into the distribution of the number of entries by each type of inquiry. This plot reveals significant variability across different inquiry types. Questions exhibit the widest range, with most studies containing between 50 and 100 entries. Case scenarios/vignettes display a slightly narrower range, with the majority of studies containing between 40 and 80 entries. The number of entries for EHR data inquiries is generally more consistent, clustering between 60 and 100 entries, indicating less variability in the volume of these inquiries.

The larger number of entries associated with direct questions suggests a broader application of this inquiry type, while the more focused range for case scenarios/vignettes and EHR data may reflect the specialized and often more detailed nature of these inquiries. These distributions highlight the diverse methods employed to assess ChatGPT’s effectiveness and its practical application in healthcare.

4. Discussion

This systematic review included a total of 128 studies that evaluated the performance of ChatGPT in various healthcare-related tasks across various medical domains. Through a detailed analysis of these studies, we aimed to assess the current effectiveness of ChatGPT in healthcare and identify areas that require further improvement.

One of the most notable observations is the generally positive perceptions among authors across a wide range of medical specialties. This suggests a growing confidence in the utility of ChatGPT for specific healthcare-related tasks. However, the variability in perceptions across different fields, particularly in more complex specialties such as diagnosis, underscores the need for cautious optimism. These differences highlight that, while ChatGPT has shown promise, its performance is not uniformly reliable across all medical domains. The performance analysis of ChatGPT further supports this nuanced view. The overall mean accuracy of 73.4% is encouraging, yet the variability in accuracy across specialties points to significant room for improvement. For instance, the high accuracy observed in psychiatry and dermatology contrasts sharply with the more variable results in endocrinology and metabolic diseases, and clinical pharmacy and pharmacology. This disparity likely reflects the inherent complexity and specialized knowledge required in different medical fields, indicating that ChatGPT may be better suited for certain types of inquiries than others.

The context in which inquiries are made also plays a crucial role in determining ChatGPT’s effectiveness. Educational inquiries, which tend to be more straightforward and less nuanced, showed higher accuracy compared to more complex tasks like diagnosis and disease management. This finding suggests that, while ChatGPT can be a useful tool for education and patient engagement, its application in more critical and complex medical decisions should be approached with caution. The presence of outliers in contexts such as diagnosis further emphasizes the need for ongoing refinement of the model to enhance its consistency and reliability.

Interestingly, the analysis of two versions of ChatGPT (3.5 vs. 4.0) did not reveal any significant differences in performance overall. In contrast to our findings, individual studies have reported significant improvements with GPT-4.0 over version 3.5 in specific domains [16,75,133]. For example, in ophthalmology, Jiao et al. [133] reported that GPT-4.0 significantly outperformed GPT-3.5, with an accuracy rate of 75% compared to 46% for GPT-3.5 in answering multiple-choice ophthalmic case challenges [133]. This suggests that the improvements observed in GPT-4.0 might be context-dependent, indicating that its enhanced performance may be more pronounced in certain specialized applications than in others.

The diversity in the performance metrics employed to evaluate the accuracy of ChatGPT’s responses poses a significant challenge in drawing consistent and comparable conclusions across the studies. The wide range of metrics used, from basic accuracy to more detailed measures, makes it difficult to assess the model’s performance uniformly. This lack of standardization complicates the synthesis of findings and limits the ability to benchmark ChatGPT’s reliably. Moreover, only six studies (Supplementary Table S1) reported specific metrics such as precision, recall, and specificity, which are crucial for a detailed understanding of the model’s performance. These metrics provide deeper insights into the model’s ability to correctly identify relevant responses (precision), capture all relevant instances (recall), and accurately distinguish between different classes or conditions (specificity). The absence of these critical metrics in the majority of studies highlights a significant gap in the current evaluation framework, suggesting that the reported performance may not fully capture the nuances of ChatGPT’s capabilities. To advance the field, future research should prioritize the use of comprehensive and standardized performance metrics to enable more accurate and meaningful comparisons.

4.1. Study Limitations

Despite these promising applications, our review identified several challenges and limitations that must be addressed while discussing our results. First, the search strategy was limited to a single database, PubMed, due to the focus on studies utilizing ChatGPT in healthcare settings. Although this approach ensured the inclusion of studies relevant to the objective, it may have excluded pertinent research published in non-PubMed-indexed journals or other databases. While acknowledging that including additional databases may yield a broader range of results, we believe our focus on PubMed allowed us to maintain scientific rigor while effectively addressing our research questions.

In addition, an innovative approach was adopted to automate the title and abstract screening process using ChatGPT, with a prompt specifically engineered to ensure the inclusion of all eligible studies. Although this method was validated with a sample of 100 studies, achieving 100% recall, there remains a slight possibility of error. Furthermore, the data extraction process encountered several challenges. Given the large number of included studies, data extraction was not performed in pairs. Nevertheless, efforts were made to ensure accuracy by openly communicating any uncertainties during the extraction process and reaching a consensus when needed. Additionally, the heterogeneity of the included studies posed significant challenges in data synthesis. The studies varied widely in the medical fields they covered and in the performance metrics used to evaluate ChatGPT. As a result, this variability required the development of an adjusted accuracy metric to effectively synthesize the data. While this approach was necessary to manage these differences, it may not fully capture the specific nuances of each study’s findings, thereby limiting the interpretability and comparability of the synthesized results.

Another important consideration is the risk of bias among the included studies, which stems from the variability in how ChatGPT’s performance was assessed. Each study used a different set of inquiries, and performance evaluations were often based on the subjective opinions of judges. Although many studies employed clinicians or authoritative references for these evaluations, the inherent subjectivity of human judgment introduces a risk of bias that could affect the validity of the findings.

Finally, the generalizability of the findings is limited by the specific contexts in which the included studies were conducted. The results may not be fully applicable to other populations, settings, or contexts, particularly given the diversity of medical fields and evaluation methods represented in the reviewed studies. Therefore, caution should be exercised when extending these findings to broader applications of ChatGPT in healthcare.

4.2. Future Directions

The findings of this systematic review underscore several key areas for future research and development in the application of ChatGPT in healthcare. Firstly, there is a clear need for further exploration of the model’s performance across a broader range of medical specialties, particularly those that involve complex diagnostic and treatment decisions. Future studies should aim to evaluate ChatGPT’s capabilities in more nuanced and specialized contexts, where the accuracy and reliability of the model are critical.

Secondly, a crucial area for future research is the standardization of performance metrics. The review highlighted a significant lack of uniformity in the evaluation frameworks across the studies, underscoring the necessity for a consistent set of metrics that can accurately assess ChatGPT’s capabilities. Metrics such as precision, recall, and specificity, among others, should be universally adopted to provide a comprehensive understanding of the model’s performance. Establishing standardized benchmarks will not only facilitate more meaningful comparisons across different studies but also enable a more precise and reliable assessment of the model’s strengths and weaknesses, ultimately advancing the field of AI in healthcare.

Additionally, the potential of GPT-4.0 and future iterations of the model warrants further investigation, particularly in specialized applications where preliminary findings suggest significant improvements over earlier versions. Research should continue to explore the context-dependent nature of these improvements to better understand where and how new versions of ChatGPT can be most effectively utilized in healthcare.

Finally, large language models like ChatGPT are highly sensitive to the prompts they receive, which can significantly influence their output. This sensitivity necessitates further research into prompt engineering—a process that involves crafting prompts in ways that optimize the model’s performance. Future work should focus on developing and utilizing standardized methodologies, such as Retrieval-Augmented Generation (RAG), to construct better prompts and improve accuracy. By enhancing the way prompts are designed and interpreted, researchers can help ensure that ChatGPT provides more reliable and contextually appropriate responses, thereby increasing its utility in clinical settings.

5. Conclusions

This systematic review highlights the potential and limitations of ChatGPT in healthcare, based on the analysis of 128 studies across various medical domains. The findings indicate a generally positive perception of ChatGPT’s utility, particularly in simpler tasks such as educational inquiries. However, significant variability in performance was observed across different specialties, with higher accuracy in fields like psychiatry and dermatology, and more inconsistent results in complex domains such as endocrinology, clinical pharmacy, and pharmacology. This suggests that, while ChatGPT shows promise, its accuracy is not uniform across all medical applications, necessitating cautious implementation in critical healthcare decisions. The comparison between ChatGPT versions 3.5 and 4.0 revealed that, while there may be domain-specific improvements in ChatGPT 4.0, overall performance differences were not statistically significant. This indicates that advancements in AI models might be context-dependent, further emphasizing the need for tailored application and continuous refinement.