Abstract
Background: Vaccines have eradicated deadly diseases, yet vaccine hesitancy persists, leading to reduced uptake. Some individuals, mistrustful of healthcare providers, may turn to tools like ChatGPT for information. We evaluated ChatGPT’s responses to common vaccine hesitancy questions. Methods: Infectious disease physicians compiled 15 frequently encountered questions relating to vaccine hesitancy, focusing on concerns about efficacy, adverse effects, and cultural or religious issues, and submitted them to ChatGPT on 18 October 2023. Two independent physicians assessed the responses for factual accuracy and alignment with recommendations from the U.S. Centers for Disease Control and Prevention’s Advisory Committee on Immunization Practices (ACIP). Results: A representative selection of responses is shown. ChatGPT consistently provided fact-based, reassuring responses. For instance, it accurately addressed the benefits of male HPV vaccination; however, it failed to consider ACIP’s age-specific recommendations and individual sexual health factors. While correcting mRNA vaccine misconceptions, it did not mention the availability of non-mRNA COVID-19 vaccines. It also lacked depth in discussing religious objections, deferring users to faith leaders and providers. Conclusions: ChatGPT provides generally accurate information and may be a useful adjunct in addressing vaccine hesitancy. With refinement, it could complement public health efforts to improve vaccine confidence and counter misinformation.
1. Introduction
Vaccine hesitancy is a phenomenon of global concern, particularly in the context of the recent coronavirus disease 2019 (COVID-19) pandemic [1]. It covers a broad range of factors that contribute to an individual’s behavioural decision to accept a vaccine [2]. From a public health perspective, this leads to excess morbidity and mortality that could have been prevented with improved uptake of vaccines. One study estimated that non-vaccination against COVID-19 contributed towards greater than 200,000 deaths from COVID-19 in the United States alone [3]. Willingness to vaccinate is influenced by several factors that contribute to an individual’s health beliefs. Easy access to reliable and accurate information, as well as trust in these sources, contributes significantly to an individual’s overall knowledge and attitudes towards vaccines [4].
Individuals who are sceptical of data and information provided by scientific publications on vaccinations may turn to alternate sources for information on the safety and efficacy of vaccines [5]. Several unreliable or overly commercial sources provide inaccurate and sensationalised myths and misinformation on vaccinations, which leads to increased vaccine hesitancy and reduced uptake of important life-saving vaccines [6].
In addition to these sources, individuals may also turn to artificial intelligence (AI) natural language processing chatbots such as ChatGPT for information on vaccines [7]. ChatGPT is widely accessible and provides easy access and responses to a wide range of queries posed by individuals. Its use has been studied in several settings, such as medical education [8,9]. Recently, ChatGPT has been examined in various clinical settings, including pre-travel advice and counselling on safe living practices for solid organ transplant recipients [10,11]. However, its role in providing accurate information and addressing concerns pertaining to vaccine hesitancy remains to be understood and examined. Therefore, from the end-user perspective, we sought to assess ChatGPT’s responses to common vaccine hesitancy concerns, in accordance with factual accuracy and alignment with U.S. CDC Advisory Committee on Immunization Practices (ACIP) guidelines.
2. Methods
To do so, we instructed ChatGPT (GPT-3.5) to answer a series of frequently asked queries related to commonly used vaccines that may lead to vaccine hesitancy on 18 October 2023. The date cut-off for ChatGPT’s training data was January 2022 at that time. Fifteen questions were drawn from clinician experience in infectious diseases and vaccine counselling and represented commonly encountered concerns among the general public. We divided these questions into the domains of (i) doubting vaccine efficacy, (ii) concerns about adverse effects, and (iii) cultural concerns pertaining to vaccines. In this exploratory analysis, a qualitative assessment of ChatGPT’s responses was performed independently by two physicians (MCYK and JNN). The responses were checked for factual accuracy and appropriateness against information provided by the ACIP guidelines [12].
3. Results
ChatGPT answered all the questions posed on vaccine hesitancy (Table A1). An appraisal of specific questions asked, along with summarized responses given by ChatGPT, is shown in Table 1. Overall, we found that ChatGPT was able to provide highly accurate and up-to-date information when asked about specific vaccines. It was able to address vaccine hesitancy in a balanced manner, providing the pros and cons of vaccination as well as encouraging further discussion with a healthcare professional, to allow for shared decision-making between the individual and their provider.
Table 1.
Evaluation of ChatGPT responses to commonly asked questions on vaccine hesitancy.
4. Discussion
In the domain of vaccine efficacy, the information provided by ChatGPT was accurate. With regards to measles vaccination, ChatGPT was able to accurately highlight that the benefits of vaccination extend beyond the individual and confer herd immunity in the community. In addition, an individual living in low-prevalence settings for measles may travel to higher endemicity areas in their lifetime, or encounter imported cases of measles, giving an important reason for vaccination at an early age to confer lifelong protection. Indeed, with falling rates of measles vaccination, outbreaks have been reported even in historically low-prevalence areas [13]. When discussing the HPV vaccine for men, ChatGPT was able to explain that the vaccine conferred protection against anal, penile, and oropharyngeal cancers, as well as some protection against non-cancerous conditions like genital warts in addition to cervical cancers. Increasing HPV vaccination in men also confers herd immunity, which helps protect women in the community [14].
However, the responses provided by ChatGPT were generic and not tailored to a specific individual. For example, in trying to convince an individual to get vaccinated against measles, it was important to consider and emphasize specific factors such as the likelihood of an individual to travel to areas of higher endemicity or whether the vaccine itself was mandated by regulatory agencies. There were also minor lapses in the factual accuracy on HPV vaccination. The ACIP recommends routine vaccination in men up to 26 years of age [12], while for those between the ages of 27 and 45, the ACIP recommends shared decision-making between the individual and their healthcare providers. This discussion considers an individual’s risk in the form of sexual health and practices as well as individual preferences. ChatGPT did not consider an individual’s age when making a recommendation for HPV vaccination, but it did direct an individual to discuss further with their healthcare provider for more specific and detailed advice.
In the domain of vaccine adverse effects, the advice provided by ChatGPT was also highly accurate. With regards to the influenza vaccine and egg allergies, the advice provided was up-to-date, highlighting that newer influenza vaccines no longer contain egg proteins, which were previously a concern for patients with severe egg allergies [15]. Rather than being dismissive of the concern, ChatGPT gave the historical context for these initial concerns and then provided reassurance with the latest evidence. ChatGPT was also able to firmly dispel other common myths pertaining to vaccines, such as the permanent alteration to deoxyribonucleic acid (DNA) caused by messenger ribonucleic acid (mRNA) vaccines or the association of the measles, mumps, and rubella (MMR) vaccine with the development of autism in children. In these responses, ChatGPT provided a measured view and explained the mechanism by which vaccines work, with a focus on their safety profiles. When asked about specific vaccine concerns, it took the opportunity to encourage and emphasize the importance of vaccination and prompted the individual to discuss further with a healthcare provider.
ChatGPT was also able to address parental concerns with regards to COVID-19 vaccinations, such as the long-term safety of vaccines. It acknowledged the uncertainty of the vaccine’s long-term side effect profile but reassured the individual that the benefits likely outweigh the risks of vaccination, along with the added reassurance that comes from continued monitoring and safeguards in place. However, although the mRNA vaccines are safe in children, ChatGPT did not provide alternatives such as inactivated COVID-19 vaccines, which may still be an option for parents concerned about the mRNA vaccine [16], or sources of information in the event that the individual or parent is eager to find out more about each specific vaccine.
Finally, when addressing cultural concerns with regards to vaccination, ChatGPT was able to provide balanced views. It demonstrated respect for a person’s religious beliefs while simultaneously encouraging open discussion with religious leaders on this topic. It also provided the scientific facts on the benefits of vaccination for the individual to consider, as these are understandably important concerns to address to improve vaccine uptake in certain religious communities across the world [17]. However, given a more specific context, it may be able to address issues specific to an individual’s religion, rather than keeping its responses vague.
When discussing ethical concerns relating to vaccine manufacturing and development processes, ChatGPT’s answers were sensible and measured. It firmly denied the argument that actual human embryos are used in the testing of mRNA vaccines, without going into the details of which vaccines use foetal cell lines [18]. However, it acknowledged that animal testing does occur in the context of vaccine development, with clear explanations that these studies were conducted with care to protect the welfare of the animals involved and cause the least amount of harm or distress. Animal testing is important to determine the safety and efficacy of vaccines, and contributes to the development of vaccines, which would translate to improving human health. Thus, the responses acknowledged an individual’s concerns but also aimed to provide alternative views to address the individual’s vaccine hesitancy. A further important factor pertaining to COVID-19 is the notion of ‘vaccine fatigue’. Where an individual may have initially accepted vaccines, over time, there may be an increasing inertia towards repeated follow-up vaccination. These may reflect changes in health beliefs, and the increasing misinformation propagated pertaining to vaccines [19]. The measured approach that ChatGPT takes in explaining mRNA vaccines may help to allay some of these fears. However, an important limitation is that this model of ChatGPT also did not discuss the possibility of non-mRNA vaccines, which may be considered for individuals who are sceptical about mRNA vaccines. Additionally, ChatGPT did not cite references for its responses, which may be helpful if individuals were keen to check the veracity of the information by referring to the original source.
Overall, our findings highlight the potential role of AI chatbots like ChatGPT as intermediary tools in public health communication, particularly in the context of addressing vaccine hesitancy in individuals who may distrust traditional sources. There remains a tension between the improved accessibility and personalization of responses from the large language model. For example, GPT-3.5 is easy to access at any time and by anyone, but has trade-offs in lacking specificity to the local context and tends to give generic responses. Additionally, it would be important to consider real-world implementation and evaluate the integration of AI tools into clinical or public health workflows. However, this would still need careful human oversight and regulatory guidance. This is because ChatGPT’s responses still remain limited in terms of the specificity of the advice and response provided. The quality of responses depends on how much information is provided by the individual asking the question. For example, unless details of the specific religious or ethical concerns are given, the responses could only remain generic and not tailored to address an individual’s specific beliefs.
We have previously described the use of ChatGPT as an adjunctive tool for pre-travel advice and counselling. Although it helped to enhance the consultation experience for the traveller, it still could not be used in isolation and was merely an adjunctive tool that required follow-up with further counselling from a human healthcare provider [20]. Furthermore, another important consideration is that the large language model algorithms may also be altered without warning, which may inadvertently affect the performance of these models.
5. Limitations
Our study examined a single natural language processing chatbot that was the most widely available. Other chatbots that are designed to answer healthcare-related questions more specifically may be able to provide better responses. As this was a preliminary, hypothesis-generating study, we opted for a qualitative assessment by two independent physicians. We did not employ a formal rubric but checked for factual accuracy and alignment with ACIP recommendations. As such, we were not able to meaningfully provide a numerical percentage of ChatGPT’s responses that adequately met pre-defined standards. Future work could formally assess these responses with a structured rubric and measure inter-rater reliability metrics such as Cohen’s kappa. Also, we did not replicate our study at different timepoints, although with more information over time, artificial intelligence chatbots may also improve on the responses provided. Lastly, the topic of vaccine hesitancy is very broad, so the questions examined could not be fully exhaustive. Also, the questions were selected by healthcare professionals, which lends itself to a degree of bias and may not fully reflect the concerns of the general public. Thus, our findings were meant to be hypothesis-generating, demonstrating that with increasing global vaccine hesitancy and mistrust in science, AI chatbots may help to mediate and correct misinformation on vaccines among the general public.
6. Conclusions
ChatGPT’s responses to vaccine hesitancy were mostly accurate and may help to correct misinformation and knowledge in individuals who harbour vaccine misconceptions that lead to a reduced willingness to vaccinate. AI chatbots may be a helpful tool as an adjunct to public health campaigns in eliminating vaccine hesitancy. Future work should include the development of standardized evaluation frameworks for AI tools in health communication. Performance in real-world settings, including evaluating its performance in other languages, would also be an important subject of future study. As newer models are developed, their performance also should be constantly re-evaluated, as these models may be able to overcome the limitations of older models and may be able to provide more personalised and accurate responses.
Author Contributions
J.N.N. and M.C.Y.K. contributed to the conception, data collection, analysis, and writing of the manuscript. B.M.A.S., P.A.T., S.A. and J.E.L.O. contributed to the conception, data analysis, and critical review of the manuscript. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
Not applicable. This short report does not directly involve any patients or confidential information and, therefore, did not require any ethics review or approval for its conduct.
Informed Consent Statement
Not applicable.
Data Availability Statement
Data may be made available on reasonable request from the corresponding author.
Conflicts of Interest
The authors declare no conflicts of interest.
Abbreviations
The following abbreviations are used in this manuscript:
| ACIP | Advisory Committee on Immunization Practices |
| AI | Artificial intelligence |
| COVID-19 | Coronavirus disease 2019 |
| DNA | Deoxyribonucleic acid |
| HPV | Human Papillomavirus |
| MMR | Measles, mumps, and rubella |
| mRNA | Messenger ribonucleic acid |
Appendix A
Table A1.
ChatGPT responses to questions pertaining to vaccine hesitancy.
References
- Troiano, G.; Nardi, A. Vaccine hesitancy in the era of COVID-19. Public Health 2021, 194, 245–251. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- MacDonald, N.E.; SAGE Working Group on Vaccine Hesitancy. Vaccine hesitancy: Definition, scope and determinants. Vaccine 2015, 33, 4161–4164. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Jia, K.M.; Hanage, W.P.; Lipsitch, M.; Johnson, A.G.; Amin, A.B.; Ali, A.R.; Scobie, H.M.; Swerdlow, D.L. Estimated preventable COVID-19-associated deaths due to non-vaccination in the United States. Eur. J. Epidemiol. 2023, 38, 1125–1128. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Dubé, E.; Laberge, C.; Guay, M.; Bramadat, P.; Roy, R.; Bettinger, J.A. Vaccine hesitancy: An overview. Human. Vaccines Immunother. 2013, 9, 1763–1773. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Yuan, S.; Rui, J.; Peng, X. Trust in scientists on COVID-19 vaccine hesitancy and vaccine intention in China and the US. Int. J. Disaster Risk Reduct. 2023, 86, 103539. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Ullah, I.; Khan, K.S.; Tahir, M.J.; Ahmed, A.; Harapan, H. Myths and conspiracy theories on vaccines and COVID-19: Potential effect on global vaccine refusals. Vacunas 2021, 22, 93–97. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Adamopoulou, E.; Moussiades, L. An Overview of Chatbot Technology. In Artificial Intelligence Applications and Innovations; Maglogiannis, I., Iliadis, L., Pimenidis, E., Eds.; IFIP Advances in Information and Communication Technology; Springer International Publishing: Cham, Switzerland, 2020; Volume 584, pp. 373–383. ISBN 978-3-030-49185-7. [Google Scholar]
- Khan, R.A.; Jawaid, M.; Khan, A.R.; Sajjad, M. ChatGPT—Reshaping medical education and clinical management. Pak. J. Med. Sci. 2023, 39, 605–607. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Sharma, S.; Pajai, S.; Prasad, R.; Wanjari, M.B.; Munjewar, P.K.; Sharma, R.; Pathade, A. A Critical Review of ChatGPT as a Potential Substitute for Diabetes Educators. Cureus 2023, 15, e38380. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Ngiam, J.N.; Koh, M.C.Y.; Lye, P.; Liong, T.S.; Salada, B.M.A.; Tambyah, P.A.; Ling, J.O.E. Artificial intelligence models for pre-travel consultation and advice: Yea or nay? J. Travel. Med. 2023, 31, taad124. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Koh, M.C.Y.; Ngiam, J.N.; Lum, L.H. ChatGPT may help in counseling on strategies for safe living following solid organ transplantation. Transplant. Infect. Dis. 2024, 26, e14222. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Murthy, N.; Wodi, A.P.; McNally, V.; Cineas, S.; Ault, K. Advisory Committee on Immunization Practices Recommended Immunization Schedule for Adults Aged 19 Years or Older—United States, 2023. MMWR Morb. Mortal. Wkly. Rep. 2023, 72, 141–144. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Patel, M.; Lee, A.D.; Clemmons, N.S.; Redd, S.B.; Poser, S.; Blog, D.; Zucker, J.R.; Leung, J.; Link-Gelles, R.; Pham, H.; et al. National Update on Measles Cases and Outbreaks—United States, January 1–October 1, 2019. MMWR Morb. Mortal. Wkly. Rep. 2019, 68, 893–896. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Stanley, M. HPV vaccination in boys and men. Hum. Vaccin. Immunother. 2014, 10, 2109–2111. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Zeiger, R.S. Current issues with influenza vaccination in egg allergy. J. Allergy Clin. Immunol. 2002, 110, 834–840. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Watanabe, A.; Kani, R.; Iwagami, M.; Takagi, H.; Yasuhara, J.; Kuno, T. Assessment of Efficacy and Safety of mRNA COVID-19 Vaccines in Children Aged 5 to 11 Years: A Systematic Review and Meta-analysis. JAMA Pediatr. 2023, 177, 384–394. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- López-Cepero, A.; Rodríguez, M.; Joseph, V.; Suglia, S.F.; Colón-López, V.; Toro-Garay, Y.G.; Archevald-Cansobre, M.D.; Fernández-Repollet, E.; Pérez, C.M. Religiosity and Beliefs toward COVID-19 Vaccination among Adults in Puerto Rico. Int. J. Environ. Res. Public Health 2022, 19, 11729. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Zimmerman, R.K. Helping patients with ethical concerns about COVID-19 vaccines in light of fetal cell lines used in some COVID-19 vaccines. Vaccine 2021, 39, 4242–4244. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Su, Z.; Cheshmehzangi, A.; McDonnell, D.; da Veiga, C.P.; Xiang, Y.-T. Mind the “Vaccine Fatigue”. Front. Immunol. 2022, 13, 839433. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Koh, M.C.Y.; Ngiam, J.N.; Chan, N.J.H.; Goh, W.; Salada, B.M.A.; Lum, L.H.; Smitasin, N.; Tambyah, P.A.; Archuleta, S.; Ling, J.O.E. Implementation of ChatGPT to enhance pre-travel consultation in a specialist tertiary centre in Singapore. J. Travel. Med. 2024, taae099. [Google Scholar] [CrossRef] [Scilit] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).