Development and Evaluation of a Child Vaccination Chatbot Real-Time Consultation Messenger Service during the COVID-19 Pandemic

Round 1

Reviewer 1 Report

1) The research can be improved by explaining the type of sampling method used in this study in further details.

2) The authors should improve on the literature review to show the relationship between changes in vaccination information, motivation, self-efficacy, and behavioral intention to vaccinate towards the efficacy of the 'Child Vaccination Chatbot'. 

Author Response

 "Please see the attachment." 

Response to Reviewer 1 Comments

Thank you very much for your thorough and valuable comments, as they greatly helped us to improve our manuscript. We have incorporated your comments into our revised manuscript.

Point 1: The research can be improved by explaining the type of sampling method used in this study in further details.

According to the reviewer’s comment 1 have added the related description.

  Response 1: The convenience sampling method was used in this study. The participants were randomly recruited online community via “Mom Café” (http://cafe.naver.com/ttokks) by announcing recruitment on the bulletin board for the parents who were in the inclusion criteria

Point 2: The authors should improve on the literature review to show the relationship between changes in vaccination information, According to the reviewer’s comment 1 have added the related description.

Response 2: Previous research emphasizes the importance of providing adequate information regarding child vaccinations. According to Kauffman et al. (2018) ‘s research providing parents about child vaccination information can improve parental awareness of vaccinations which can influence to raise vaccination rates. Moreover, it showed that providing educational intervention for parents or child immunization-related information can give a positive influence in vaccination performance. As shown in the previous research. one of the important factor in vaccination implementation is the provision of vaccination information to parents.

According to the previous research from Nadarzynski's et al. (2019), people tend to use a chatbot usually for minor health problems. Public Health- organizations and pediatric clinics provide repeated trivial vaccination related. The Q&A method is similar to the FAQ type. However, it is different in the speed of providing answers because it usually provides information in real time with appropriate answers (Wang, Kraut, & Levine, 2015).

Author Response File: Author Response.docx

Reviewer 2 Report

Dear Hong et al.

I had the privilege to review your manuscript entitled: "Development and evaluation of the 'Child Vaccination Chatbot' Real-Time Consultation Messenger Service during the COVID-19 pandemic".  I found the manuscript to be high towards addressing the growing vaccine hesitancy observer globally.

I have one main concern and a few minor recommendations with the work presented.

Main concern

Although your results indicate that the chatbot successfully increased parents' knowledge regarding vaccination, I feel that the results lack information on the actual use of the chatbot by said parents.  It would be interesting to the reader to see some of the questions raised by parents to the chatbot and whether any common themes arise.  Also, do you have access to other data regarding the use of the chatbot (i.e. duration used, the average number of questions per user, how often users returned to the chatbot, etc.)?  Including these data in the study will guide future researchers to improve their research.

Minor recommendations

line 57:  here you state that "overcoming" issues needs a new way of management, however your application is an adjunct to current practices.  I would reconsider the use of the word "overcome".  A suggestion would be "Improvements to existing vaccination management practices are needed".

line 63:  if few studies exist, it indicates that there does exist previous studies using chatbots.  I would include one or two examples and the results in its paragraph

line 154: Your welcome = you're welcome (to be changed in the chatbot, not necessarily for the publication).

I wish you the best for this publication and many citations.

Kind regards 

Author Response

 "Please see the attachment."

Response to Reviewer 2 Comments

Thank you very much for your thorough and valuable comments, as they greatly helped us to improve our manuscript. We have incorporated your comments into our revised manuscript.

Point 1: Although your results indicate that the chatbot successfully increased parents' knowledge regarding vaccination, I feel that the results lack information on the actual use of the chatbot by said parents.  It would be interesting to the reader to see some of the questions raised by parents to the chatbot and whether any common themes arise.

According to the reviewer’s comment 1 have added the related description.

 Response 1: Converatin below show the questions raised by parents to the chatbot. The conversation was made in real-time.

Real-time Converation 1.

__________________________________________________________________________

Parent: Hello :) Today, my two-year-old baby was vaccinated with the 3rd round of Japanese encephalitis and meningococcus (Menbio).

Parent:  Hello :)

Today, my two-year-old baby was vaccinated with the 3rd round of Japanese encephalitis and meningococcus (Menbio).

Chatbot: Yes, under stood

Parent:  However, the doctor told me that the meningococcal vaccination, which I know is done only once, requires another vaccination two months later.

Chatbot:: So what is your main question?

Parent: How many vaccinations do I need in total? If it is a one-time vaccination, I wonder if I don’t need an additional second vaccination.

Chatbot: There are two types of meningococcal vaccines available in Korea.

Menveo: From 2 months to 55 years of age, 1 dose. If vaccination is started at 2-6 months, 3 doses at 2-month intervals 4th vaccination is 6 months at intervals. Inoculation after 1 year of age 2 doses if the vaccination is started between 7 and 23 months of age

Menactra: 9-23 months 2 doses at 3-month intervals. 24 months - 55 years of age 1 dose

The frequency of vaccination varies depending on when your child is vaccinated and the type of medication.

* For details such as the inoculation period and frequency of inoculation, you need to consult with the doctor.

Parent: I am worried that the vaccination I will receive today requires two times of vaccination to generate antibodies.

Chatbot: Efficacy effect on serotype A for each vaccine The age and frequency of vaccination may vary, so consult with a specialist before vaccination.

Parent: Thank you

___________________________________________________________________________

Real-time Converation 2.

Parent: After child vaccination my child has a mild fever. What shall I do?

Chatbot: Check the child’s temperature regularly.

Parent:: The temperature is 37.3 degrees

Chatbot: Apply a lukewarm temperature wet towel and gently massage the childs body avoiding the vaccination area.  

Parent:: When should I see the pediatrician?

Chatbot: If the temperature goes above 38 degrees and doesn’t drop consider visiting the hospital.

___________________________________________________________________________

Point 2: Also, do you have access to other data regarding the use of the chatbot (i.e. duration used, the average number of questions per user, how often users returned to the chatbot, etc.)?  Including these data in the study will guide future researchers to improve their research.

According to the reviewer’s comment 1 have added the related description.

Response 2: The duration of using the Chatbot is 12 weeks, the average number of question per user is twice a day and usually the questions were concentrated as the vaccination schedule was imminent or after the vaccination. The frequency of visiting the Chatbot has a pattern corresponding to the child vaccination schedule.

Starting from February 1st, 2020, the experimental parent group used the Chatbot for 12weeks. The function of the Chatbot include notifications of the child vaccination schedule one week prior to the vaccination. Also regularly providing articles related to vaccinations, webtoons and news letters. In order to encourage child vaccination encouraging text messages; coupons and information was sent via Chatbot. Moreover, providing directions to the nearby vaccination centers and pediatric clinics. Automatic answers were provided related to vaccinations questions in real time. The control group, were provided a vaccination information brochure just once during the first week of the study and also were allowed to freely search the internet. During the 12weeks trial every 3weeks both groups usability test was performed. To analyze the chatbot users' perception of its performance in each phase, a repeated measures ANOVA was performed. The results showed that the mean value gradually increased from the first (M= 4.01, SD=.63) to the second (M=4.08, SD=.71) and the third (M=4.34, SD=.69). This was statistically significant (F=3304.685, p= .000). The post-test results indicate that it is statistically significant between the first and third phase, showing that the users' awareness of the usefulness of the chatbot increased as progressed. The results of the usability test’s are shown in Table 10. The Statistical significance of the repeated measurement variance are shown in Table 11.

Table 9. Results of repeated measurement analysis of usability test of ANOVA

Table 10. Statistical significance of repeated measurement variance analysis

Point 3: Minor recommendations

line 57:  here you state that "overcoming" issues needs a new way of management, however your application is an adjunct to current practices.  I would reconsider the use of the word "overcome".  A suggestion would be "Improvements to existing vaccination management practices are needed".

Response 3

 I have revised the sentence above according to the reviewer’s comment

line 63:  if few studies exist, it indicates that there does exist previous studies using chatbots. I would include one or two examples and the results in its paragraph

I have revised the sentence above according to the reviewer’s comment

According to the previous research from Nadarzynski's et al. (2019), people tend to use a chatbot usually for minor health problems. Public Health- organizations and pediatric clinics provide repeated trivial vaccination related. The Q&A method is similar to the FAQ type. However, it is different in the speed of providing answers because it usually provides information in real time with appropriate answers (Wang, Kraut, & Levine, 2015). by communicating with the parents. Moreover, according to the previous research

line 154: Your welcome = you're welcome (to be changed in the chatbot, not necessarily for the publication).

I have revised the sentence above according to the reviewer’s comment

Author Response File: Author Response.docx

Reviewer 3 Report

This article proposed the development and evaluation of the “Child Vaccination Chat-2 bot” for COVID-19 vaccination in order to enhance the motivation, self-efficacy, etc. First of all, the article needs to edit by native speaker. The article proposed the chatbot to provide vaccination information. However, what is the novelty of this research? What is the research significant of this study? As such, I would wonder the research contribution of this article.

Why the chatbot is used but not the other approaches? How the chatbot is delivered? What social network? There are several figures presenting the chatbot scenarios, these are written in Korean. These figures can be simplified and translated to English.

In the method, it is also unclear what the articles want to measure to quantify the outcomes. The most important is there is lack of the research theory and model for such measurement (authors can refer to some articles on measuring with the research model, e.g. COVID-19 Crisis: Exploring Community of Inquiry in Online Learning for Sub-Degree Students). It is also unclear how the outcomes are measured. what are the questions? What these measures mean and what it should be measured?

Why figure 6 is included? What does this imply?

Author Response

 "Please see the attachment."

Response to Reviewer 3 Comments

Thank you very much for your thorough and valuable comments, as they greatly helped us to improve our manuscript. We have incorporated your comments into our revised manuscript.

Point 1: This article proposed the development and evaluation of the “Child Vaccination Chat-2 bot” for COVID-19 vaccination in order to enhance the motivation, self-efficacy, etc. First of all, the article needs to edit by native speaker.

According to the reviewer’s comment this article went through an English editing sevice provided by MBTI

We are attaching the English editing certificate.

Point 2: The article proposed the chatbot to provide vaccination information. However, what is the novelty of this research?

According to the child age the due vaccination schedule differs. The newly developed child vaccination chatbot based on a real-time consultation messenger service was programed based on the child's age. Which can provide customized information in real-time. Moreover, regularly reminding the vaccination schedule and regularly sending encouraging messages and coupon rewards to boost vaccine motivation, showed positive impact on social motivation to increase confidence of parents

What is the research significant of this study? As such, I would wonder the research contribution of this article.

As pediatric clinics and community healthcare center bustle with parent’s, Child Vaccination Chatbot can contribute to reduce responding to repeated questions. By reducing the repeated Q&A process the medical staffs can save time and focus on more sophisticated medical practice. This can eventually contribute to provide better care for children and parents. Moreover, this study can be applied to community medical practice in social distance situations such as COVID-19 pandemic by providing efficient provision of in-depth customized information to parents.

Point 3: Why the chatbot is used but not the other approaches? How the chatbot is delivered? What social network?

The reason using Chatbot based on SNS platform, Kakao Plus Friend, is because the most widely used domestic SNS platform is Kakao. "Kakao talk" is a widely used messenger service like "Whats App", "WeChat" etc. Comparing methods of information provision by conventional documentation, websites and chatbot Information update speed can be done in real-time and provides information related to the context while conventional documentation and websites do not. Chatbot is also can provide two-way interaction providing customized information. There are also no time-space constraints as the Chatbot is mobile based and can be ubiquitous Most of all unlike traditional applications, does not require users to install software separately Therfore, users do not have to figure out how to us a chatbot but can directly obtain information by communicating daily.

Point 3-1: There are several figures presenting the chatbot scenarios, these are written in Korean. These figures can be simplified and translated to English.

 We have translated and changed all the figures in English. Thank you for your suggestion.

Point 4: In the method, it is also unclear what the articles want to measure to quantify the outcomes.

Based on the IMB research model this research measures if there is a difference in the child’s vaccination information, motivation, self-efficacy(behavioral skills), behavioral intention scores between the experimental and control groups.

Point 5: The most important is there is lack of the research theory and model for such measurement (authors can refer to some articles on measuring with the research model, e.g. COVID-19 Crisis: Exploring Community of Inquiry in Online Learning for Sub-Degree Students).

This study is based upon the theory that information, motivation, and behavioral skills relates to one’s health behavior and can be a crucial factor in the performance of health activities. Through this study the first step was to develop a child vaccination guidance chatbot. Second, was to evaluate its effectiveness, and ultimately induce changes in users’ behaviors.

Based on the information-motivation-behavioral skills(IMB) theory, this study focused on changes in providing vaccination knowledge, vaccination behavior motivation, self-efficacy and vaccination behavior skills of parents. Information-motivation-behavioral skills(IMB) model developed by Fisher et al. (2009), shows providing appropriate health behavior information results in increased motivation and improved self-efficacy can ultimately lead to changes in health behaviors. The IMB theory has conceptualized action-related elements to maintain self-care behaviors (Fisher, J. D., & Fisher, W. A., 1992; Fisher, J. D., Fisher, W. A., & Shuper, 2009).

Point 6: It is also unclear how the outcomes are measured.

In order to evaluation of the effectiveness of the chatbot and measure the outcomes, every 3 weeks the measurements of vaccination information, motivation, and self-efficacy behavioral intention variables were measured in tests and scored. The questionnaires were provided via chatbot.

First, to measure the “Vaccination Information” verifying the content validity of the 12 questions was performed. Five experts were requested to provide their views to the measurement tools. These included a pediatrician, a researcher at the Vaccination Management Division of the KCDC, a professor at a nursing college, and two doctors at a nursing college. 12 questions with the CVI index of 80% or more were selected, with a slight revision.

Totally, twelve questions consist of ‘yes’ or ‘no’ questions regarding vaccination. Correct answers to each question score one point. A higher correct answers’ rate stands for the higher level of child vaccination information.

Second, to measure the “Vaccination Motivation” six questions were revised from the pertussis vaccination motivation measurement tool developed by Kim (2008). The reliability of the original tool was Cronbach's alpha = .82. In addition, supplemented tool by experts was the same.

Third, to measure the “Self efficacy” five questions modified by Lee and Yang (2017) with the reference to the tools developed by Ko (2012). The original tool’s Cronbach's alpha = .91 and the supplemented tool In this study, is Cronbach's alpha = .85.

Third, to measure the “Behavioral intention” Yeon et al. (2010) and Han (2011), seven questions were used to revise and supplement the measurement for the intention of vaccination. Cronbach's alpha = .9.

Point 7: what are the questions?

The purposes of this study is to develop the real-time consultation messenger service chatbot for vaccination questions and answers. Applying the developed chatbot to the child raising parents, and evaluate the effect on the score changes for vaccination information, motivation, self-efficacy, and vaccination behavioral intention

Point 8: What these measures mean and what it should be measured?

Conveying vaccination information via chatbot has not been previously applied in the vaccination field. Therefore, chatbot can be a new method to vaccination information communication. Thus, it is expected that the groups using the chatbot will have better access to information than those that do not, and that obtaining information through the chatbot will be useful.

Point 9: Why figure 6 is included? What does this imply?

It shows the statistical result of Turing test against the Child vaccination chatbot users. Turing test shows concept of how the patient feel about using a chatbot. It is a common method of measuring the outcomes of a Artificial Intelligence. However, the figure itself doesn’t match the whole flow.

Therefore, we deleted figure6.  

Author Response File: Author Response.docx

Reviewer 4 Report

Hong et al propose an interesting study on the use of a Chatbot to encourage the vaccination of children. I consider the manuscript well done and I propose the following recommendations:

• line 101 please cite in in the style requested by the journal
• Conclusion is too long. Please be brief and relate to the purpose and objectives of your study.