Evaluation of the Internal and External Validity of Social media and Mobile Technology-driven HPV Vaccination Interven- tions: Systematic Review using Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) Framework

Social media HPV vaccination interventions show promise for increasing HPV vaccination rates. An important consideration for the implementation of effective interventions into realworld practice in the translation potential, or external validity, of the intervention. To this end, we conducted a systematic literature review to describe the current body of evidence regarding the external validity of social media HPV vaccination-related interventions. Constructs related to external validity were based on the RE-AIM (reach, effectiveness, adoption, implementation, maintenance) framework. Seventeen articles published between 2006 and 2020 met inclusion criteria. Three researchers independently coded each article using a validated RE-AIM (reach, effectiveness/efficacy, adoption, implementation, maintenance) framework. Discrepant codes were discussed with a fourth reviewer to gain consensus. Of these 17 studies, three were pilot efficacy studies, 10 were RCTs to evaluate effectiveness, one was a population-based study, and three did not explicitly state which type of study was conducted. Reflecting this distribution of study types, across all studies the mean level of reporting RE-AIM dimensions varied with reach recording 90.8%, effectiveness (72.1%), adoption (40.3%), implementation (45.6%), and maintenance (26.5%). This review suggests that while the current HPV vaccination social media-driven interventions provide sufficient information on internal validity (reach and effectiveness), few have aimed to gather data on external validity needed to translate the interventions into real world implementation. Our data suggest that implementation research is needed to move HPV vaccination-related interventions into practice. Included in this review are recommendations for enhancing the design and reporting of these HPV vaccination social media-related interventions.


Introduction
The human papillomavirus (HPV) vaccine protects against HPV-associated cancers, including most cervical cancer, as well as vulvar, vaginal, anal, penile, and oropharyngeal cancer. Cervical cancer is the fourth most common cancer among women worldwide, with approximately 570,000 new cervical cancer cases reported in 2018, representing 6.6% of female cancers [1]. The incidence of oral and anal cancers is increasing [2,3]. The HPV vaccine is recommended for adolescents [11][12] years, with catch-up vaccination through age 26 and FDA approval for adults up to age 45 years [4].
As part of the Global Strategy for the Elimination of Cervical Cancer as a Public Health Problem, the World Health Organization's (WHO) goal is for 90% vaccination of girls age 15 by 2030. The Healthy People 2030 goal is to increase the proportion of adolescents who receive recommended doses of the HPV vaccine with a target goal of 80% [5]. Although integrated programs and efforts to increase the HPV vaccination have occurred in many countries over the last 14 years, HPV vaccine rates remain low [6]. For instance, in 2019, only 54.2% of adolescents in the US [7] and 15% of adolescents globally [8] were current on HPV vaccinations.
Multiple factors including lack of opportunity for vaccination, parental attitudes or perceptions towards vaccination, lack of recommendations from healthcare providers, concerns about the vaccine's effect on sexual behavior, religious objection, low perceived risk of HPV infection, social influences, irregular preventive care, and vaccine cost have contributed to the low vaccination rates [9][10][11]. In efforts to address these barriers, researchers have included mobile technology-related media including text-message, e-mail, phone calls, and private Facebook messages, in their interventions to increase vaccination awareness, uptakes, and dose completion [12,13]. These interventions have been efficacious in increasing HPV vaccination uptake and completion [12][13][14][15].
However, the prospect of translating these efficacious interventions into regular clinical practice is unknown due in part to the lack of reported external validity [16]. External validity is the ability to generalize an evidence-based study to different measures, persons, settings, and times [17]. Several translational researchers have argued that reporting detailed components and processes of evidence-based studies would increase studies' generalizability (external validity) and the ability to translate those interventions into practice [18][19][20].
To address the research-practice issue, Glasgow and colleagues developed the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework [21] with a set of metrics critical for evaluating the generalizability of an evidence-based intervention into routine practice. They proposed that the translatability of an intervention is best evaluated through the five dimensions (RE-AIM). The reach dimension is designed to assess the proportion of potentially eligible individuals who participate in the intervention study. Efficacy/effectiveness is the function of the intended positive impact of the intervention. Adoption reflects the potential settings and inter-vention agents that participate in a study. Implementation refers to the quantity and quality of delivery of the intervention's various components. Finally, the maintenance dimension is the longerterm efficacy/effectiveness of an intervention on an individual. (See Table 1 for the definitions and indicators for each RE-AIM dimension). This framework can be used to organize and evaluate threats to the transferability of research to practice. Further, by evaluating a study through the fivedimension lens of the framework, researchers are able to assess internal and external validity equally [16,21].
The RE-AIM framework has been used recently in several systematic reviews to evaluate the internal and external validities of health intervention studies such as weight management intervention [18], physical activity intervention [22,23], worksite health behavior interventions [24], community settings [20], school-based health promotion [25,26], childhood obesity prevention [27][28][29], children dietary interventions with parents [30], injury prevention strategies [31], faithbased intervention [32], mobile phone-based intervention for diabetes self-management [33] and HIV prevention intervention [34]. While encouraging, there is little reporting on its potential use for translating HPV vaccination social media-driven intervention methods into regular practice settings, specifically at the population level. The two most recent comprehensive literature reviews on HPV vaccination mobile technology-related interventions evaluated the effectiveness of interventions in increasing the HPV vaccine [35], and the effectiveness of communication technology interventions on HPV vaccine [36]. However, the scope of these two previous literature reviews was narrow focusing primarily on internal validity with limited information on the external validity of the HPV vaccination intervention studies [18,19].
Our study aimed to use the RE-AIM framework to evaluate HPV vaccination intervention studies that included mobile technology to increase HPV vaccination uptake. To our knowledge, this is the first review using the RE-AIM framework to evaluate the HPV vaccination mobile technology-related interventions. Unlike the two previous reviews on HPV vaccine interventions, our current systematic review was structured to determine the translation potential or external validity of published HPV vaccination intervention studies by determining the extent to which those studies reported information across all five of the RE-AIM framework dimensions. We provided recommendations for future research based on these findings.

Search Strategy and Selection Criteria
We conducted an extensive literature search to identify research articles related to HPV vaccination technology-based interventions. We searched nine databases (PubMed, EMBASE, Medline, ERIC, CINAHL, Academic Search Complete, Web of Science, PsycINFO, Cochrane Library) using the following terms: human papillomavirus OR human papillomavirus* OR HPVAND social media OR social medium OR Web 2.0 OR twitter messaging OR Instagram OR Facebook OR WhatsApp OR Tito OR text message OR mobile technology AND intervention OR RCT. An article was included in the review if it met the following inclusion criteria: published in English, between 2006 and 2020, and in peer-reviewed journals; outcome variables included HPV vaccination completion and/or vaccination uptake; the intervention's mode of delivery included social media (WhatsApp, Facebook), and text messages. Articles were excluded if they were cross-sectional studies and included assessment of only participants' knowledge, attitude, and intention (see Figure 1). Excluded because outcome measure did not include vaccination uptake 3

RE-AIM Criteria
A modified version of the RE-AIM 30-item data extraction tool (https://www.re-aim.org) was used to code eligible articles on the degree to which internal and external validity indicators were reported. The RE-AIM dimensions and corresponding indicators are listed in Table 1.

Reach dimension:
Seven indicators were used to evaluate the reach dimension of the study. They included the description of the target population, description of the participants' HPV vaccination behavior, recruitment strategies, inclusion and exclusion criteria, description of the sample size determination, and participation rate.

Effectiveness dimension:
Eight metrics for effectiveness included the efficacy of the intervention in changing vaccination behavior, measurement of primary and/or secondary outcome (i.e. vaccination completion, or uptake), a short-term assessment, intent-to-treat assessment, description of imputation procedure, the measure of robustness across subgroups, and short-term attrition assessment.

Adoption dimension:
Seven indicators used for adoption were a description of intervention location, intervention delivery staff, the method used to identify delivery staff, inclusion and exclusion criteria for the staff, and rate of staff participation.

Implementation dimension:
Four metrics used for the implementation dimension were intention frequency, duration, the extent to which the intervention was implemented as planned, completion rates, and measurement of cost of implementing the intervention.

Maintenance dimension:
The four metrics for maintenance included follow-up (3 and 6 months) assessments, attrition rates, continuation, and institutionalization of the program.

Coding and Analysis
Articles that met the inclusion criteria for this review were independently coded by three graduate research assistants (BP, NT, and CM) and supervised by the principal investigator of the research team (MA). Each reviewer coded a ''yes'' indicating the presence or ''no,'' indicating the absence of the RE-AIM indicators outlined above. Following the individual coding, the PI and the three research assistants met to discuss articles and coding results, resolve uncertainty, and gain consensus in coding. The articles that had; "yes" for any indicator were scored as 1 and "no" was  Table 1). Based on the 33-item RE-AIM indicators, articles that scored between 0 -11 indicated fewer reporting of RE-AIM indicators, between 12 -22 indicated moderate reporting, and between 23-33 indicated high reporting of RE-AIM indicators.

Adoption Dimension
For the adoption dimension, the most reported indicators were the description of intervention location (88.2%) and the staff who delivered intervention (64.7%). Organizational spread and measures of the cost of adoption were two adoption indicators that received the lowest reporting score of 5.9% (see Table 1). Only Gerend et al's [37] article reported six out of the eight adoption dimension indicators. The remaining articles reported a few of the adoption dimension indicators with scores ranging from 1 to 5 (out of 8 indicators). (See Figure 2).

Implementation
Per our inclusion criteria for this review, all 17 articles included in the review used some form of technology including social media (Facebook); mobile phone (text messages), and emails to deliver the intervention. The most addressed implementation dimension indicators were intervention frequency and duration (88.2%) and participation/completion rates (82.4%) (See Table 1). The extent to which the protocol was delivered as intended was the least reported indicator (11.8%) and none of the studies reported the total cost of implementing the intervention. However, eight reported incentives given to each participant, and one reported that vaccines were given to participants at no cost. (See Figure 2).

Maintenance
The most common maintenance dimension indicator addressed in the reviewed articles was the 3-month and 6-month follow-up assessment (70.6%). The long-term attrition rate was addressed in 35.3% of articles. None of the articles addressed whether the programs were institutionalized or were still in place. (See Table 2).

Summary
This systematic review utilized the RE-AIM framework to evaluate the impact of HPV vaccine intervention studies that incorporated mobile technology to increase vaccine uptake and recommended dose completion. Eleven articles' outcome of measure included vaccination uptake [37-39, 41, 44, 47, 49-53]; and six articles' outcome measure was vaccination completion [15,40,42,43,45,48]. The vaccination uptake rates in the review ranged from 6.6% to 89% and the completion rates range from 17% to 88%, indicating successful implementation of many of the interventions.  [25, 27-30, 34, 54, 55]. Several articles targeted college students [37,38,43,44,48,53] which is not surprising given that the majority of those within the college-age groups are social media consumers. The reach dimension indicator rarely discussed is the target population denominator. While reporting of this indicator tends to be challenging [55], by not reporting the target population denominator, there is no context given to help determine the sample sizes. In the efficacy studies the concern is recruiting enough participants to provide the necessary power to detect effect size; therefore, understanding the target population that was exposed to recruitment materials can provide an estimate of the likely reach the program will achieve [18]. short-term assessments, were regularly reported which is consistent with the literature [25, 27-30, 34, 54, 55]. Across all 17 articles, less than half the authors described their chosen method of analysis for missing variables and/or attrition whether to use intent-to-treat or per-protocol analysis (analysis by treatment administered) approach. The intention-to-treat principle states that all randomized participants are included in the statistical analysis and analyzed according to the group they were originally assigned, regardless of what treatment (if any) they received [56]. Whenever treatment groups are not analyzed according to the group to which they were originally assigned, the risk of bias increases [57]. Similar to the conclusion drawn by Hollis and Campbell [58] the intention to treat approach is often insufficiently described and inadequately applied. This lack of reporting can diminish the accurate (unbiased) conclusions regarding the effectiveness of an intervention [57].
Authors should explicitly describe the handling of any deviations from randomized allocation and discuss missing responses and their potential effect on the studies' outcomes [58]. However, the use of randomization in some of the studies [15,38,40,42,43,45,52] may attenuate the possible confounding bias. Although most of the articles reported short-term pre-post assessments, only four articles [38,43,45,53] calculated short-term attrition rates. Attrition prevents a full intention to treat analysis and it can occur when participants have missing data and/or loss to follow-up. We argue that researchers need to be more explicit about the loss to follow-up, especially if rates are high [59].

Adoption Dimension
The average score of the adoption dimension was 38% with observed scores for the adoption indicators ranging from 6% to 88%. The description of the location of the intervention score (88%) for our reviewed articles is higher than the previous reviews scores of 48% to 60% [18,54], but lower when compared with the scores in Allen et al's [16]  maintained. The full support of adopting agents directly influences implementation fidelity and program sustainability [60].

Implementation
In the implementation dimension, we were interested in intervention studies that incorporated social media technology to deliver the intervention. Although the authors did not evaluate whether using a specific social media type was effective in delivering the intervention, based on our review we found that interventions that utilized Facebook [41,50] and text messages [15,45,49] reported significant improvements in HPV vaccination. The most frequent technology used was text messaging [15,37,39,43,[45][46][47][48][49]. Additionally, only eight articles reported the underlying theoretical framework used in intervention development [38,39,41,42,[49][50][51][52]. Theories provide a systematic view of phenomena by specifying the relationship between program inputs (resources), program activities (how the program is implemented), and their outputs or outcomes [61][62][63].  The application of the RE-AIM framework to evaluate the effectiveness of HPV vaccine social mobile interventions is limited, yet our review demonstrates the utility of RE-AIM to evaluate HPV interventions and highlights the potential transferability of selected HPV intervention programs to a broader audience.
3. To increase the potential to translate HPV vaccine research findings to practice, researchers should place a greater emphasis on obtaining and reporting external validity information, such as adoption, implementation, and maintenance dimensions. Providing external validity information enhances other researchers' and practitioners' ability to judge the generalizability of effects and the comparative utility of interventions [20,64]. All stakeholders, including researchers, reviewers, editorial board, and funders, should emphasize the need for external validity information [20]

Conclusions
In conclusion, our findings show that social media-related HPV vaccination intervention studies demonstrated some effect on vaccination uptake (at least one dose of vaccination rate of 48% of the study population); reached larger study participants; and demonstrated that college students and college-aged groups are the targets of most social media intervention studies. While most articles in our review met the Consolidated Standards Of Reporting Trials (CONSORT) metrics for reporting, specifically for internal validity reporting; the adoption, implementation, and maintenance dimensions for the RE-AIM framework were underreported. To ensure that these successful community-based interventions can be translated into practices, stakeholders should not only embrace the reporting of all the RE-AIM dimensions but should encourage researchers to adhere to external validity reporting standards similar to CONSORT internal validity reporting Supplementary Materials: The following are available online at www.mdpi.com/xxx/s1, Figure S1: Table S1: title, Video S1: title.