Next Article in Journal
Improving Multi-Label Learning by Correlation Embedding
Previous Article in Journal
Constant-Current, Constant-Voltage Operation of a Dual-Bridge Resonant Converter: Modulation, Design and Experimental Results
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 11
Issue 24
10.3390/app112412142
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Development and Evaluation of a Child Vaccination Chatbot Real-Time Consultation Messenger Service during the COVID-19 Pandemic

by
Yeong-Joo Hong
1,
Meihua Piao
2,*,
Jeongeun Kim
3 and
Jae-Ho Lee
4
1
Korea Health Industry Development Institute, 187 Osongsaengmyeong 2-ro, Osong-eup, Heungdeok-gu, Cheongju-si 28159, Korea
2
Office of Hospital Information, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul 03080, Korea
3
College of Nursing, Seoul National University, 103 Daehak-ro, Jongno-gu, Seoul 03080, Korea
4
Department of Emergency Medicine, Asan Medical Center, University of Ulsan College of Medicine, 88, Olympic-ro 43-gil, Songpa-gu, Seoul 138736, Korea
*
Author to whom correspondence should be addressed.
Appl. Sci. 2021, 11(24), 12142; https://doi.org/10.3390/app112412142
Submission received: 30 September 2021 / Revised: 30 October 2021 / Accepted: 10 December 2021 / Published: 20 December 2021
(This article belongs to the Section Robotics and Automation)
Browse Figures
Review Reports Versions Notes

Abstract

:

Featured Application

Increase vaccination motivation, self-efficacy, and vaccination behavioral intention by using Child Vaccination Chatbot.

Abstract

The decreased rate of children’s vaccination has resulted in outbreaks of vaccine-preventable diseases, and vaccination hesitancy is being brought about by the uncertainty caused by the continuing COVID-19 pandemic. With this study, we aimed to assess the efficacy of a child vaccination chatbot based on changes in variables such as vaccination information, motivation, self-efficacy, and vaccination behavioral intention. From 30 January to 15 February 2020, 65 parents raising children ≤35 months old who were expected to be vaccinated within three months participated in the trial through online recruitment. Participants were randomly assigned to either the experimental group (n = 34) or the control group (n = 31) and were followed up with over a period of 12 weeks. During this period, both groups of participants were provided with vaccination schedule reminders. The experimental group were additionally provided with vaccination-related information and motivation boosters by a chatbot (real-time consultation messenger service); the control group was provided the same information by brochure. Comparing both groups, the experimental group that used the chatbot scored higher on vaccination information, motivation, self-efficacy, and vaccination behavioral intention than the control group. This suggests that the chatbot provided useful and timely information to parents, increasing vaccination motivation, self-efficacy, and vaccination rates. This study provides evidence that chatbots are useful tools to encourage immunization through the provision of reminders and real-time consultation messenger services during the global health crisis and beyond.

1. Introduction

Vaccination is considered the safest and most effective way to protect children from contagious diseases. According to the Korean National Health Data report “2018 Children’s Regular Vaccination Rate”, released in 2019, 96.8% of Korean children over 12 months old were vaccinated. However, there was a decrease in vaccination rates as children grew older, with rates of 94.7% for children aged 24 months and 88.3% for children aged 72 months [1].
Various factors affect the reduction in vaccination rates, including several sociodemographic [2] and parental factors. Parents’ health beliefs have a direct impact on their children’s vaccination status, as children are unable to direct their own health behavior. Furthermore, the American Centers for Disease Control (CDC) reported that the ongoing COVID-19 pandemic has contributed to a decrease in overall child vaccination rates [3]. Parents concerned about exposing their children to COVID-19 might contribute to the observed decline [3]. Additionally, according to national data from the Korea Disease Control and Prevention Agency, between the first quarters of 2019 and 2020, there was a decrease in pediatric vaccine administration, likely influenced by COVID-19 [4]. To prevent such declines, it is essential to provide timely vaccination-related information to parents, reminding them of the vital need to protect their children against serious vaccine-preventable diseases, even as the COVID-19 pandemic continues.
Previous research emphasizes the importance of providing adequate information regarding child vaccinations. According to Kauffman et al. (2018)’s research, providing parents with child vaccination information can improve parental awareness of vaccinations, which can raise vaccination rates. Moreover, it showed that providing educational intervention for parents or child immunization-related information can have a positive influence on vaccination rates. As shown in the previous research, one of the important factors in vaccination implementation is the provision of vaccination information to parents.
In Korea, vaccination-related information is provided to parents by health centers, hospitals, and clinics. Additionally, parents use analog child vaccination notebooks to record their children’s vaccination information. The weakness of traditional methods includes mistaken records and loss of vaccination notebooks, which may lead to missed vaccinations [5].
Improvements to existing vaccination management practices are needed. A chatbot could supplement existing methods, as it can provide vaccination-related information in real time. According to Nadarzynski et al. (2019), people tend to use a chatbot for minor health problems. Public health organizations and pediatric clinics provide repeated trivial vaccination-related information. The Q&A (question and answer) method of chatbots is similar to FAQs. However, it is different in the speed of providing answers because it usually provides information in real time with appropriate answers (Wang, Kraut, and Levine, 2015). Considering these points, using mutual communication techniques to encourage child vaccination has a positive effect on vaccination decision making [6,7,8].
However, the importance of vaccination management for children is easily neglected and research on vaccination has been limited to studies mainly on vaccination awareness and satisfaction, as well as on vaccination status and related factors [9,10,11,12,13]. Few studies have been conducted on chatbots in this context. Awareness and dissemination of information could potentially improve if parents used technical tools, such as chatbots, when raising their children [14]. Therefore, receiving vaccination information through a chatbot channel is expected to contribute to boosting vaccination rates because relevant information can be provided more effectively than when using traditional methods. The purpose of this study is to develop a real-time consultation messenger service chatbot for vaccination questions and answers. We made the chatbot available to child-raising parents and evaluated its effect on the score changes for vaccination information, motivation, self-efficacy, and vaccination behavioral intention.

2. Theoretical Background

This study is based on the theory that information, motivation, and behavioral skills relate to one’s health behavior and can be crucial in the performance of health activities. Through this study, the first step was to develop a child vaccination guidance chatbot, and the second was to evaluate its effectiveness, and ultimately induce changes in users’ behaviors. Based on the information–motivation–behavioral skills (IMB) theory, this study focused on changes in providing vaccination knowledge and improving vaccination behavior motivation, self-efficacy, and vaccination behavior skills of parents. The information–motivation–behavioral skills (IMB) model developed by Fisher et al. (2009) shows that providing appropriate health behavior information results in increased motivation and improved self-efficacy, which can ultimately lead to changes in health behaviors. The IMB theory conceptualizes action-related elements to maintain self-care behaviors (Fisher, J. D., and Fisher, W. A., 1992; Fisher, J. D., Fisher, W. A., and Shuper, 2009).

3. Materials and Methods

3.1. Research Design

A child vaccination chatbot to answer parents’ frequently asked questions regarding their children’s vaccinations was employed according to the system development life cycle. To investigate the effects of potential changes in vaccination motivation, self-efficacy, and behavioral intention caused by receiving parental vaccination information, a quasi-experimental research design was employed. The Institutional Review Board of Seoul National University (IRB no. 2002/001-008) approved this study.
In order to remove the possibility of test effects, we used a study design without a pre-test according to a previous study by DeCarlo [14], since exposure to a research questionnaire could affect the results. The study states that the effectiveness of online-based education is highest after the first month of use. The effect decreased after 1, 3, 6, and 12 months [15]. Thus, in this study, data were collected 4, 8, and 12 weeks after using the chatbot for both the control and experimental groups (Table 1).

3.2. Recruitment

The sample size was calculated using a power analysis formula [16]. With a significance level α = 0.05, power 1-β = 0.80, and medium effect size = 0.3, based on the analysis of variance (ANOVA) test, the minimum sample size for each group was 21. After assuming a 25% dropout rate, each group needed 35 participants for a total of 70 research participants. According to previous research, the median target effect size was 0.3 in the publicly funded Health Technology Assessment trial [17].
All participants were parents raising 0–35-month-old children who required vaccination in accordance with Korean law. The research participants were provided with information regarding the study. The inclusion and exclusion criteria for study participants were as follows.
Inclusion criteria:
  • Parents raising 0–35-month-old children who required vaccination;
  • Parents raising children who were required to be immunized within three months;
  • Parents who could understand the survey and were able to respond;
  • Parents who were raising their first born child;
  • Parents who understood the purpose of this study and who provided a written consent;
  • Kakao Talk app users.
Exclusion criteria:
  • Parents who have career related to the medical field;
  • Parents raising more than one child;
  • Parents who cannot read or communicate in Korean;
  • Parents who refused to participate.
The convenience sampling method was used in this study. The participants were randomly recruited online community via “Mom Café” (http://cafe.naver.com/ttokks accessed on 3 January 2020) by announcing recruitment on the bulletin board for parents who met the inclusion criteria. The participants were recruited online and through social networking services (SNS). The recruitment process lasted from 30 January to 15 February 2020. After the parents provided their consent, they were asked to provide basic information, such as the age of their child, that could be used to develop appropriate groups.
To retain homogeneity between the control group and the experimental group, parents raising their first child who was younger than 35 months were recruited. To distribute parents evenly in each group, children of a similar age were placed in pairs in the experimental and control groups. The two groups were otherwise randomly assigned. Block randomization was performed using R software (R Foundation for Statistical Computing).
The initial survey was sent to participants via Kakao Talk using their mobile numbers. A chatbot link was provided via Kakao Talk to the experimental group, while a child vaccination paper brochure was sent to the control group. After 4, 8, and 12 weeks of using the chatbot, the experimental group was asked to evaluate the chatbot using a mobile survey (http://forms.gle/byhoq725xYkWBNcn8 accessed on 1 July 2020).

3.3. Chatbot Intervention Program Development

In this study, we used the Kakao Plus Friend platform, which contains 94.4% of the Korean population [18], to deliver a chatbot intervention program to provide real-time child vaccination consultation. The chatbot developed for this study provides information on child vaccination with 1:1 customization, a feature enabled by building an algorithm appropriate for each child by considering their age without the need to expose the counseling content to others.
During the first testing stage, the dialogue content of the prototype chatbot was constructed using Dialogflow, supported by Google. Dialogflow is a convenient program which can apply natural language learning to chatbot implementation and can be linked to Kakao i Open Builder [19]. Through the Dialogflow program, keywords for 17 infectious diseases and 17 types of vaccines suggested by the KCDC website were included. By entering the keywords can lead to finding nearby health centers, vaccination schedules, nationally designated vaccinations, other vaccinations, and information regarding vaccination medical institutions. Representations of the child vaccination chatbot based on Kakao Plus Friend are shown in Figure 1, Figure 2, Figure 3 and Figure 4. If the candidates agreed to participate, they were registered, and a survey was sent to a 1:1 chat room. The structure of the chatbot is shown in Figure 5. The child vaccination chatbot service scenario is presented in Table 2.
The conversation below shows the questions raised by parents to the chatbot. The conversation occurred in real time.
Real-time Conversation 1.
Parent: Hello:) Today, my two-year-old baby was vaccinated with the 3rd round of Japanese encephalitis and meningococcus (Menbio).
Chatbot: Yes, understood
Parent: However, the doctor told me that the meningococcal vaccination requires another vaccination two months later. Which is confusing because I thought only one round was enough.
Chatbot: So what is your main question?
Parent: How many vaccinations do I need in total? I wonder if one-time vaccination is enough.
Chatbot: There are two types of meningococcal vaccines available in Korea.
First, Menveo: From 2 months to 55 years of age, 1 dose. If vaccination is started at 2~6 months, 3 doses at 2-month intervals 4th vaccination is 6 months at intervals. Inoculation after 1 year of age 2 doses if the vaccination is started between 7 and 23 months of age
Second, Menactra: 9~23 months 2 doses at 3-month intervals. 24 months~55 years of age 1 dose. The frequency of vaccination varies depending on when your child is vaccinated and the type of medication.
* For details such as the inoculation period and frequency of inoculation, you need to consult with your assigned doctor.
Parent: I am worried about the vaccination my child will receive today whether it requires two times of vaccination to generate antibodies.
Chatbot: The age and frequency of vaccination may vary, so consult with a specialist before vaccination.
Parent: Thank you
Real-time Conversation 2.
Parent: After child vaccination my child has a mild fever. What shall I do?
Chatbot: Check the child’s temperature regularly.
Parent: The temperature is 37.3 degrees
Chatbot: Apply a lukewarm temperature wet towel and gently massage the child’s body avoiding the vaccination area.
Parent: When should I see the pediatrician?
Chatbot: If the temperature goes above 38 degrees and doesn’t drop consider visiting the hospital.
The reason for using the chatbot based on an SNS platform, Kakao Plus Friend, is because this is the most widely used domestic SNS platform in Korea. “Kakao talk” is a widely used messenger service similar to WhatsApp, WeChat, etc. Comparing methods of information provision by conventional documentation, websites, and chatbot information update speed can be performed in real time and provides information related to the context, while conventional documentation and websites do not. The chatbot can also provide a two-way interaction and customized information. There are also no time/space constraints, as the chatbot is mobile-based and can be ubiquitous. Most of all, unlike traditional applications, it does not require users to install software separately. Therefore, users do not have to figure out how to us a chatbot but can directly obtain information by communicating daily. Below, several figures present the chatbot scenarios, which were originally written in Korean. These figures were simplified and translated to English.

3.4. Chatbot Usability

The duration of using the chatbot is 12 weeks. The average number of questions per user is twice a day and usually the questions were concentrated as the vaccination schedule was imminent or after the vaccination. The frequency of visiting the chatbot has a pattern corresponding to the child vaccination schedule.
Starting from 1 February 2020, the experimental parent group used the chatbot for 12 weeks. The function of the chatbot included notifications of the child vaccination schedule one week prior to the vaccination. It also regularly provided articles related to vaccinations, webtoons, and newsletters. In order to encourage child vaccination, encouraging text messages, coupons, and information were sent via chatbot. Moreover, the chatbot provided directions to nearby vaccination centers and pediatric clinics. Automatic answers were provided related to vaccination questions in real time. The control group were provided with a vaccination information brochure just once during the first week of the study and also were allowed to freely search the internet. During the 12-week trial, every 3 weeks, both groups underwent a usability test. To analyze the chatbot users’ perception of its performance in each phase, a repeated measures ANOVA was performed. The results showed that the mean value gradually increased from the first (M = 4.01, SD = 0.63) to the second (M = 4.08, SD = 0.71) and the third phase (M = 4.34, SD = 0.69). This was statistically significant (F = 3304.685, p = 0.000). The post-test results indicate a statistically significant difference between the first and third phases, showing that the users’ awareness of the usefulness of the chatbot increased as time progressed. The results of the usability test are shown in Table 3. The statistical significance of the repeated measurement variance is shown in Table 4.

3.4.1. Outcome Measures

We evaluated the effectiveness of the chatbot program using measurements of vaccination information, motivation, self-efficacy, and behavioral intention as variables. Based on the IMB research model, we measured if there was a difference in the child’s vaccination information, motivation, self-efficacy (behavioral skills), and behavioral intention scores between the experimental and control groups.
In order to evaluation of the effectiveness of the chatbot and measure the outcomes, every 3 weeks, the measurements of vaccination information, motivation, self-efficacy, and behavioral intention were taken and scored.
The questionnaires were provided via the chatbot. First, to measure “Vaccination Information”, we verified the content validity of the 12 questions. Five experts were asked to provide their views to the measurement tools. These experts included a pediatrician, a researcher at the Vaccination Management Division of the KCDC, a professor at a nursing college, and two doctors at a nursing college. Twelve questions with a CVI index of 80% or more were selected, with a slight revision. In total, 12 yes or no questions regarding vaccination were formulated. Correct answers to each question scored one point. A higher correct answer rate indicated a higher level of child vaccination information.
Second, to measure “Vaccination Motivation”, six questions were revised from the pertussis vaccination motivation measurement tool developed by Kim (2008). The reliability of the original tool was Cronbach’s alpha = 0.82. The supplemented tool by experts was the same.
Third, to measure “Self-Efficacy”, five questions were modified from Lee and Yang (2017) with reference to the tools developed by Ko (2012). The original tool’s Cronbach’s alpha was 0.91 and that of the supplemented tool in this is 0.85.
Fourth, to measure “Behavioral Intention”, based on Yeon et al. (2010) and Han (2011), seven questions were used to revise and supplement the measurement for the intention of vaccination, with Cronbach’s alpha = 0.9.
Conveying vaccination information via a chatbot has not been previously applied in the vaccination field. Therefore, the chatbot can be a new method to facilitate vaccination information communication. Thus, it is expected that the groups using the chatbot will have better access to information than those who do not, and that obtaining information through the chatbot will be useful.

3.4.2. Vaccination Information

To verify the content validity of the first measurement tool, five experts were consulted. The content validity index (CVI) was calculated based on the CVI score [20]. Subsequently, we developed 12 yes or no questions regarding vaccination. Correct answers added one point to the score. Higher scores indicate a higher level of child vaccination information.

3.4.3. Vaccination Motivation

Six questions from the pertussis vaccination individual motivation measurement tool developed by Kim [21] were revised and included in the study. The original tool’s Cronbach’s alpha was 0.82, the same as this study.

3.4.4. Vaccination Self-Efficacy

Self-efficacy refers to parents’ ability to successfully carry out their children’s vaccination schedules. This variable was measured by revising and supplementing the self-efficacy tool developed to investigate the implementation of the rotavirus vaccination [12]. The Cronbach’s alpha was 0.91 in the original tool and 0.85 in this study.

3.4.5. Behavioral Intention to Vaccinate

Behavioral intention refers to taking preventive action, that is, a parent’s intention to voluntarily implement their child’s vaccination schedule. Based on the metrics used in previous studies [22,23], we developed seven questions to supplement existing measurements for the intention to vaccinate. The Cronbach’s alpha was 0.94.

3.4.6. Turing Test

Alan Turing’s [24] seminal 1950 research express the criteria for intelligence in the computing machinery and intelligence fields. They can be summarized as follows: from a separate location, an interviewer interacts with a human and a computer within a predetermined period. If the interviewer cannot distinguish between the human and the computer through separate conversations with each other, or if the interviewer thinks the computer is human, the computer is considered intelligent [25]. Since the Turing test exists only as a theory and there is no actual discrete measurement tool for AI, we developed four questions to assess the chatbot’s intelligence. Thus, in this study, after using the non-face-to-face chatbot for 12 weeks, the experimental participants were asked if the conversation felt like it was with a person or a machine.

3.5. Statistical Analysis

SPSS v.21 software was used (IBM Corp., Armonk, NY, USA) for statistical analysis. Participants’ general characteristics were analyzed using frequency, percentage, mean, and standard deviation. Pearson’s chi-square test (Pearson’s x2) was used for the homogeneity test for the experimental and control groups. The results for knowledge, motivation, self-efficacy, and behavioral intention were analyzed by repeated-measures ANOVA.

4. Results

4.1. Baseline Characteristics

This study had a quasi-experimental format, employing a control group that did not use the chatbot and an intervention or experimental group that did. A total of 63 subjects were included in the analysis after excluding subjects who dropped out (one in the control group and four in the intervention group). In Korean culture, a child’s primary caregiver is typically the mother; therefore, the proportion of female participants was larger. Testing for homogeneity of general characteristics (such as sex, age of the child in months, and age of parents) between the intervention and control groups showed no significant differences (Table 5).

4.2. Changes in Vaccination Knowledge, Motivation, Self-Efficacy, and Behavioral Intention

After 4 weeks, the average vaccination knowledge score was 6.13 (±2.63) in the experimental group and 3.34 (±2.55) in the control group, which was statistically significant (t = −4.27, p = 0.000). After 8 weeks, the experimental group showed a higher mean (8.29 ± 2.15) than the control group (4.44 ± 2.06), which was statistically significant (t = −7.26, p = 0.000). After 12 weeks, the mean of the experimental group (8.74 ± 1.70) was higher than that of the control group (3.88 ± 2.14), which was also statistically significant (t = −10.00, p = 0.000). The differences in motivation, self-efficacy, and behavioral intention between the experimental group and control group are shown in (Table 6). Changes in motivation, self-efficacy, and behavioral intention were statistically significant.

4.3. Turing Test Results

A typical question asked regarding the Turing test was: “Having used the chatbot for 12 weeks, did you feel like you were talking to a real person?” About 25.71% of respondents said that they felt like they were talking to a real person. On the other hand, 74.29% of respondents said they felt like they were talking to a machine. A series of questions were asked to determine which function of the chatbot could be improved to make the conversation more human-like. The experimental group considered the accuracy of the answers and the tone, humor, and emoticons used by the chatbot. Conversely, when the chatbot provided answers that were accurate but too detailed, users felt that the chatbot was machine-like. When the respondents were asked about what improvements could be made to the chatbot, the answers were as follows: use of natural tone, providing specific answers to questions, and giving many short answers rather than sharing too much information at once.

5. Discussion

According to this study the significant effect of increasing parents’ access to vaccination-related information by employing a newly designed child vaccination chatbot. A child vaccination chatbot can be used as a useful tool as it can increase parents’ motivation and intention to vaccinate their children. Parents in the experimental group showed higher scores for the four variables compared to the control group.
Our results suggest that engaging parents with an interactive tool in a familiar technological environment may improve vaccination rates. In the second quarter of 2019, Kakao Talk had 44.417 million users [17]. One of the main characteristics of Kakao Talk is that it can be used habitually and helps users interact conveniently with services such as chatbots. Moreover, it can promote social networking and emotional exchange among members [26]. There is an active research stream regarding health promotion based on Kakao Talk, and the beneficial effects of using this platform have been suggested in a previous study [25].
Through this study, parents who missed their vaccination scheduled were interviewed to determine the functional needs that could be applied to upgrade the chatbot. Through the Kakao Plus Friend Service, notification messages were sent automatically to parents when their child’s vaccination date was imminent, and users were provided weekly information related to vaccinations regularly, including articles, newsletters, and webtoons. The service also included information about the location of the nearest vaccination center and local healthcare institutions. The most important factor for increasing child vaccinations seemed to be providing accurate information promptly.
Data were collected through prior research on child vaccination at the KCDC. Parents who missed a child’s vaccination date indicated a high demand for continuously provided information on vaccinations. Based on the interviews, the chatbot’s algorithm was revised and updated to meet users’ needs even as the study was ongoing. Further adjustments should be made to ensure the best use of the platform.
According to this research, providing personalized real-time information based on the changes of child’s age was essential. Furthermore, it is important to notify parents promptly of their child’s vaccination schedule. By reflecting on the current environment and providing credible, personalized information in real time through a familiar platform, parents feel reassured and capable of safely implementing their child’s vaccination schedule.
A typical question raised by the parents are about precautions to take before, during, and after vaccination. As they have not many choices, parents tend to look for answers through the internet. However, it is difficult to obtain suitable specific information based on a child’s age. Nadarzynski et al. [27] stated that to maximize the effect of information provided through AI-based interventions, the opinions of users and healthcare professionals must be considered. Accordingly, the present research used standardized data provided by the KCDC to craft the chatbot’s responses. As demonstrated by previous studies, providing reliable information based on an easy-to-access mobile chatbot can have a positive effect on vaccination motivation.
A key motivator for maintaining participation in most activities is the provision of social rewards. Previous research has shown that engagement can be boosted by expressing empathy when people post comments or photos [28]. When the participants posted a photo of their child receiving a vaccination, a message of encouragement and coupons were sent to motivate them to continue. Another method for motivating people is praising [29]. Thus, rewards, showing interest, responding, and leaving affirming comments positively affect motivation to vaccinate children.
Jones [30] showed that providing mobile-based information was effective at enhancing self-efficacy. Similarly, in our study, the experimental group showed higher self-efficacy scores, indicating that they were confident that they could successfully implement the vaccination schedule recommended by the KCDC.
The information–motivation–behavioral skills model developed by Fisher et al. [31] posits that when information related to health behavior is provided properly, it motivates people to behave in a healthy manner [32]. Thus, as self-efficacy improves, so do health behaviors. This study indicates that as parents increased their motivation and self-efficacy through using a chatbot, their vaccination behavioral intention also increased.
In this study, a feedback exercise based on the Turing test was conducted at the end of week 12. When asked if the chatbot interactions seemed human, 74.29% of respondents said no. Although in 2014, there is one case regarding Eugene Goostman chatbot passing the Turing test, no other chatbots or computer programs have showed a significant difference since [33]. However, there are ways to improve chatbot interactions to make them feel more natural. According to previous studies, to compensate for the disadvantages of text-oriented web programs, it is better to limit the length of the text provided by the app and use more images and video content to increase users’ engagement [34]. These factors must be considered to improve the app.

6. Conclusions

In conclusion, a development of a child vaccination chatbot was done to help parents receive information about vaccinations via a mobile social network platform. The child vaccination chatbot provided helpful and timely information for parents raising children who needed vaccinations. The study indicates that using the child vaccination chatbot increased vaccination motivation, self-efficacy, and vaccination rates by providing parents with the necessary information to follow their child’s vaccination schedule. Despite the strengths of our study, several limitations must be considered.
The method of information provision proposed here has several implications. First, follow-up research to monitor the effects of the child vaccination chatbot is needed in a long-term perspective. Second, additional research is required to determine how it can complement current practices and how economically effective the chatbot can be. Third, future studies should determine how healthcare efficiency might be improved using chatbots.
A chatbot can reduce the time and cost required to address frequently asked questions related to vaccination by providing human-like, real-time conversation based on reliable and accurate information. Although there are concerns over the potential of AI to replace humans, being able to offload repetitive tasks to a machine tool can both enhance user experience and allow healthcare personnel to concentrate on creative, productive, and caring tasks.
The vaccination schedule differs according to a child’s age. An asset of our newly developed child vaccination chatbot, providing a real-time consultation messenger service, is that it was programed based on the age of child in question. Moreover, regularly reminding parents of their child’s vaccination schedule and regularly sending encouraging messages and coupon rewards to boost vaccine motivation showed a positive impact on social motivation to increase the confidence of parents. As pediatric clinics and community healthcare centers bustle with parents, the child vaccination chatbot can help save time and resources by responding to common questions. With the chatbot taking on the Q&A process, medical staff can focus more on providing care and treatment. This can eventually contribute to providing better care for children and parents. Moreover, this study can be applied to community medical practice in social distance situations such as the COVID-19 pandemic by efficiently providing in-depth, customized information to parents.

Author Contributions

Conceptualization, Y.-J.H. and M.P.; methodology, Y.-J.H.; software, Y.-J.H.; validation, Y.-J.H.; formal analysis, Y.-J.H. and M.P.; investigation, Y.-J.H.; resources, Y.-J.H.; data curation, Y.-J.H.; writing—original draft preparation, Y.-J.H.; writing—review and editing, J.-H.L. and J.K.; visualization, Y.-J.H. and M.P.; supervision, J.-H.L. and J.K.; project administration, Y.-J.H.; funding acquisition, Y.-J.H. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the National Research Foundation of Korea (NRF), grant number 2020R1F1A1077151. National Research Foundation of Korea (NRF) is a organization specialized in research management in Korea Science and Engineering Foundation, Korea Research Foundation and Korea Foundation for International Cooperation of Science and Technology. The NRF plans, evaluates and manages all projects related to academic and R&D activities of universities, research institutes and industries covering all areas of humanities, social sciences and engineering.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and was approved by the Institutional Review Board of Seoul National University (IRB no. 2002/001-008) on 25 May 2020.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Acknowledgments

This research was supported by the National Research Foundation of Korea (NRF) grant by the Korea government (MSIT)(No. NRF-2019R1F1A1049523).

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Korea Center for Disease Control and Prevention. Available online: http://www.kdca.go.kr/index.es?sid=a3 (accessed on 29 August 2020).
  2. Park, M.-B.; Kim, C.-B.; Joo, H.-S. Factors influencing influenza vaccination coverage. J. Korean Con. Ass. 2013, 13, 300–311. [Google Scholar] [CrossRef]
  3. Santoli, J.M.; Lindley, M.C.; DeSilva, M.B.; Kharbanda, E.O.; Daley, M.F.; Galloway, L.; Gee, J.; Glover, M.; Herring, B.; Kang, Y.; et al. Effects of the COVID-19 pandemic on routine pediatric vaccine ordering and administration—United States, 2020. MMWR Morb. Mortal Wkly. Rep. 2020, 69, 591–593. [Google Scholar] [CrossRef]
  4. Korea Center for Disease Control. Vaccination Information Search Site. Available online: https://nip.cdc.go.kr/irgd/index.html (accessed on 29 August 2020).
  5. Nam, H.J.; Lee, S.G.; Jeon, S.Y.; Om, J.E.; Park, K.S. Investigation of children with no vaccinations recorded on the National Immunization Registry Information System. J. Korean Soc. Matern. Child Health 2017, 21, 176–181. [Google Scholar] [CrossRef]
  6. Heiss, S.N.; Carmack, H.J.; Chadwick, A.E. Effects of interpersonal communication, knowledge, and attitudes on pertussis vaccination in Vermont. J. Healthc. Commun. 2015, 8, 207–219. [Google Scholar] [CrossRef]
  7. Humiston, S.G.; Rosenthal, S.L. Challenges to vaccinating adolescents: Vaccine implementation issues. Pediatr. Infect. Dis. J. 2005, 24 (Suppl. 6), S134–S140. [Google Scholar] [CrossRef] [PubMed]
  8. Forster, A.S.; Rockliffe, L.; Chorley, A.J.; Marlow, L.A.V.; Bedford, H.; Smith, S.G.; Waller, J. A qualitative systematic review of factors influencing parents’ vaccination decision-making in the United Kingdom. SSM Popul. Health 2016, 2, 603–612. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  9. Sandora, T.J.; Gidengil, C.A.; Lee, G.M. Pertussis vaccination for healthcare workers. Clin. Microbiol. Rev. 2008, 21, 426–434. [Google Scholar] [CrossRef] [Green Version]
  10. Snyder, L.B.; Hamilton, M.A.; Mitchell, E.W.; Kiwanuka-Tondo, J.; Fleming-Melici, F.; Proctor, D. A meta-analysis of the effect of mediated health communication campaigns on behavior change in the United States. J. Health Commun. 2010, 9 (Suppl. 1), 71–96. [Google Scholar] [CrossRef] [PubMed]
  11. Kim, K.-J.; Hwang, T.-Y.; Lee, K.-S. Knowledge, health belief, and vaccination behavior on Hepatitis A among university students. J. Agric. Med. Community Health 2016, 41, 119–128. [Google Scholar] [CrossRef] [Green Version]
  12. Lee, S.H.; Yang, S.J. Factors associated with rotavirus vaccination behavior among mothers of infants: Using the health belief model. J. Korean Acad. Public Health Nurs. 2017, 31, 34–46. [Google Scholar] [CrossRef] [Green Version]
  13. Wong-Villacres, M.; Evans, H.; Schechter, D.; DiSalvo, B.; Kumar, N. Consejero automatico: Chatbots for supporting Latino parents’ educational engagement. In Proceedings of the Tenth International Conference on Information and Communication Technologies and Development, Ahmedabad, India, 4–7 January 2019; Association of Computing Machinery: New York, NY, USA, 2019; Volume 53, pp. 1–5. [Google Scholar] [CrossRef]
  14. DeCarlo, M.; Crenshaw, C.; Fominaya, A.; Parsons, A.; Secret, M.; Stewart, M. Open Textbooks and Student Learning in Social Work Research Methods: Results from a Mixed-Methods and Student-Engaged Pilot Project. Available online: osf.io/preprints/socarxiv/vjf5p (accessed on 21 December 2019).
  15. Patrick, K.; Calfas, K.J.; Norman, G.J.; Rosenberg, D.; Zabinski, M.F.; Sallis, J.F.; Rock, C.L.; Dillon, L.W. Outcomes of a 12-month web-based intervention for overweight and obese men. Ann. Behav. Med. 2011, 42, 391–401. [Google Scholar] [CrossRef]
  16. Faul, F.; Erdfelder, E.; Lang, A.G.; Buchner, A. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 2007, 39, 175–191. [Google Scholar] [CrossRef]
  17. Rothwell, J.C.; Julious, S.A.; Cooper, C.L. A study of target effect sizes in randomized controlled trials published in the Health Technology Assessment journal. Trials 2018, 19, 1–13. [Google Scholar] [CrossRef] [Green Version]
  18. Maeil Business News Korea. Available online: https://pulsenews.co.kr/view.php?year=2020&no=218180 (accessed on 2 March 2020).
  19. Reyes, R.; Garza, D.; Garrido, L.; De La Cueva, V.; Ramirez, J. Methodology for the implementation of virtual assistants for education using Google Dialogflow. In Mexican International Conference on Artificial Intelligence 2019: Advances in Soft Computing; Martínez-Villaseñor, L., Batyrshin, I., Marín-Hernández, A., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 440–451. [Google Scholar] [CrossRef]
  20. Lynn, M.R. Determination and quantification of content validity. Nurs. Res. 1986, 35, 382–385. [Google Scholar] [CrossRef] [PubMed]
  21. Kim, S.M. A Study on Pertussis Immunity State and Influencing Factors for Adult Pertussis Vaccination in Healthcare Workers. Master’s Thesis, Inha University, Inchon, Korea, 2008. [Google Scholar]
  22. Choi, W.-H.; Seo, Y.-M.; Kim, B.R. Factors influencing dementia preventive behavior intention in the elderly. J. East-West Nurs. Res. 2019, 25, 138–146. [Google Scholar] [CrossRef]
  23. Han, K.-H.; Jo, S. Does culture matter? A cross-national investigation of women’s responses to cancer prevention campaigns. Health Care Women Int. 2012, 33, 75–94. [Google Scholar] [CrossRef] [PubMed]
  24. Turing, A.M. Computing machinery and intelligence. Mind 1950, 59, 433–434. [Google Scholar] [CrossRef]
  25. Piao, M.; Ryu, H.; Lee, H.; Kim, J. Use of the Healthy Lifestyle Coaching Chatbot app to promote stair-climbing habits among office workers: Exploratory randomized controlled trial. JMIR mHealth uHealth 2020, 8, e15085. [Google Scholar] [CrossRef]
  26. Kim, J.-M. Analyzing the user’s using attitude of KakaoTalk Plus Friend–coupon attitude as a mediator. J. Dig. Conv. 2018, 16, 327–336. [Google Scholar] [CrossRef]
  27. Nadarzynski, T.; Miles, O.; Cowie, A.; Ridge, D. Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: A mixed-methods study. Dig. Health 2019, 5, 1–12. [Google Scholar] [CrossRef]
  28. Eyal, N. Hooked: How to Build Habit-Forming Products; Penguin Canada: Toronto, ON, Canada, 2014. [Google Scholar]
  29. Verplanken, B. Beyond frequency: Habit as mental construct. Br. J. Soc. Psychol. 2010, 45, 639–656. [Google Scholar] [CrossRef] [PubMed]
  30. Jones, J.M.; Lewis, F.M.; Griffith, K.; Cheng, T.; Secord, S.; Walton, T.; Bernstein, L.J.; Maheu, C.; Catton, P. Helping Her Heal-Group: A pilot study to evaluate a group delivered educational intervention for male spouses of women with breast cancer. Psycho-Oncol. 2013, 22, 2102–2109. [Google Scholar] [CrossRef]
  31. Fisher, J.D.; Fisher, W.A.; Shuper, P.A. The information-motivation-behavioral skills model of HIV preventive behavior. In Emerging Theories in Health Promotion Practice and Research, 2nd ed.; DiClemente, R.J., Crosby, R.A., Kegler, M.C., Eds.; Jossey-Bass Inc./Wiley: San Francisco, CA, USA, 2009; pp. 21–64. [Google Scholar]
  32. Chang, S.J.; Choi, S.; Kim, S.-A.; Song, M. Intervention strategies based on information-motivation-behavioral skills model for health behavior change: A systematic review. Asian Nurs. Res. 2014, 8, 172–181. [Google Scholar] [CrossRef] [Green Version]
  33. Warwick, K.; Shah, H. Can machines think? A report on Turing test experiments at the Royal Society. J. Exp. Theor. Artif. Intell. 2016, 28, 989–1007. [Google Scholar] [CrossRef] [Green Version]
  34. Kwon, E.J. Development and Evaluation of Information Support Program for Spouses of Women with Breast Cancer Using Mobile Phone. Ph.D. Thesis, Seoul National University, Seoul, Korea, 2015. [Google Scholar]
Applsci 11 12142 g001 550
Figure 1. The main screen of the child vaccination chatbot.
Figure 1. The main screen of the child vaccination chatbot.
Applsci 11 12142 g001
Applsci 11 12142 g002 550
Figure 2. Immunization inquiries about COVID-19.
Figure 2. Immunization inquiries about COVID-19.
Applsci 11 12142 g002
Applsci 11 12142 g003 550
Figure 3. Searching for a public health center.
Figure 3. Searching for a public health center.
Applsci 11 12142 g003
Applsci 11 12142 g004 550
Figure 4. Child vaccination questions and answers.
Figure 4. Child vaccination questions and answers.
Applsci 11 12142 g004
Applsci 11 12142 g005 550
Figure 5. The dialogue architecture of the child vaccination chatbot.
Figure 5. The dialogue architecture of the child vaccination chatbot.
Applsci 11 12142 g005
Table
Table 1. The research design of the study.
Table 1. The research design of the study.
GroupInterventionPost-Test
(After 4 Weeks)		Post-Test
(After 8 Weeks)		Post-Test
(After 12 Weeks)
ExperimentalX 1E1 2E2 2E3 2
Control C1 3C2 3C3 3
1 X: child vaccination chatbot baseline. 2 E1, E2, E3: Measure child vaccination information, motivation, self-efficacy, and behavioral intention for the experimental group. 3 C1, C2, C3: Measure child vaccination information, motivation, self-efficacy, and behavioral intention for the control group.
Table
Table 2. Child vaccination chatbot’s service scenario.
Table 2. Child vaccination chatbot’s service scenario.
Scenarios
1. Parent or guardian can join the study through the Kakao Plus Friend platform and start the child vaccination chatbot.
2. Enter the child’s name and age.
3. Real time response to provide the vaccination information corresponding according to the age of the child.
4. Sending a notification message one week in advance the vaccination schedule.
5. Provide vaccination related information (e.g., measles).
6. Sending weekly based message related to immunization information articles, newsletters, webtoons, images, etc.,
7. Parents can check the child’s vaccination history.
8. Provide a message to encourage and send a drink coupon when a vaccination certificate is uploaded to motivate the parent child vaccination.
9. Inform nearby locations where to reach vaccination (e.g., community healthcare centers, medical institutions).
Table
Table 3. Results of repeated measurement analysis of usability test of ANOVA.
Table 3. Results of repeated measurement analysis of usability test of ANOVA.
VariableGroupPhase 1 Phase 2Phase 3
MeanSDMeanSDMeanSD
Usability testControl4.010.634.080.714.340.69
Table
Table 4. Statistical significance of repeated measurement variance analysis.
Table 4. Statistical significance of repeated measurement variance analysis.
Variable SSdfMSFp
Usability testWithin
group		Lag score1442.81911442.8193304.6850.000
Error11.788270.437
Table
Table 5. Homogeneity test of general characteristics at baseline.
Table 5. Homogeneity test of general characteristics at baseline.
CharacteristicsCategoriesExperimental Group
n (%)		Control Group
n (%)		x2 or tp
Gender of the
Chatbot User		Male5 (16.1)12 (37.5)3.650.056
Female26 (83.9)20 (62.5)
Age of the child (month)16.10 ± 13.1717.50 ± 11.430.4520.653
Age of the parent34.19 ± 4.3536.75 ± 9.151.4090.164
Table
Table 6. Outcome variables in the experimental and control group (n = 63).
Table 6. Outcome variables in the experimental and control group (n = 63).
VariableGroupPost-Test
(After 4 Weeks)		Post-Test
(After 8 Weeks)		Post-Test
(After 12 Weeks)
Mean ± SDt, p-ValueMean ± SDt, p-ValueMean ± SDt, p-Value
KnowledgeExp6.13 ± 2.63−4.278.29 ± 2.15−7.268.74 ± 1.70−10.00
Con3.34 ± 2.55<0.0014.44 ± 2.06<0.0013.88 ± 2.14<0.001
MotivationExp2.83 ± 0.83−3.283.21 ± 0.57−7.193.63 ± 0.38−9.81
Con2.17 ± 0.77<0.052.28 ± 0.45<0.0012.45 ± 0.56<0.001
Self-EfficacyExp3.22 ± 0.68−2.263.59 ± 0.34−7.743.76 ± 0.34−8.47
Con2.66 ± 1.23<0.052.66 ± 1.26<0.0012.43 ± 0.82<0.001
Behavioral IntentionExp4.22 ± 0.98−5.234.64 ± 0.48−9.484.71 ± 0.41−9.65
Con2.69 ± 1.32<0.0012.64 ± 1.09<0.0012.76 ± 1.07<0.001
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Hong, Y.-J.; Piao, M.; Kim, J.; Lee, J.-H. Development and Evaluation of a Child Vaccination Chatbot Real-Time Consultation Messenger Service during the COVID-19 Pandemic. Appl. Sci. 2021, 11, 12142. https://doi.org/10.3390/app112412142

AMA Style

Hong Y-J, Piao M, Kim J, Lee J-H. Development and Evaluation of a Child Vaccination Chatbot Real-Time Consultation Messenger Service during the COVID-19 Pandemic. Applied Sciences. 2021; 11(24):12142. https://doi.org/10.3390/app112412142

Chicago/Turabian Style

Hong, Yeong-Joo, Meihua Piao, Jeongeun Kim, and Jae-Ho Lee. 2021. "Development and Evaluation of a Child Vaccination Chatbot Real-Time Consultation Messenger Service during the COVID-19 Pandemic" Applied Sciences 11, no. 24: 12142. https://doi.org/10.3390/app112412142

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop