Trends in Vaccine Refusal and Acceptance Using Electronic Health Records from a Large Pediatric Hospital Network, 2013–2020: Strategies for Change

Understanding trends in vaccine refusal is critical to monitor as small declines in vaccination coverage can lead to outbreaks of vaccine-preventable diseases. Using electronic heath record (EHR) data from the Children’s Hospital of Philadelphia’s 31 outpatient primary care sites, we created a cohort of 403,448 children less than age 20 years who received at least one visit from 1 January 2013 through 31 December 2020. The sample represented 1,449,061 annualized patient and 181,131 annualized preventive vaccination visits per year. We characterized trends in vaccine refusal and acceptance using a repeated cross-sectional observational analysis of electronic health records (EHR) data using a single annual merged observation measure for patients seen multiple times for preventive healthcare within a calendar year. Refusals were identified for 212,900 annualized patient-visit year observations, which represented 14.6% of annualized patient-visit year observations and 25.1% of annualized vaccine patient-year observations. The odds of having a refusal marker were significantly increased in patients seen in suburban practices (aOR [CI]: 2.35 [2.30–2.40, p < 0.001]), in patients with increased age 11–17 years (aOR [CI]: 3.85 [3.79–3.91], p < 0.001), and those eligible for the VFC program (aOR [CI]: 1.10 [1.08–1.11]. Parental refusal (61.0%) and provider decisions (32.0%) were the most common documented in progress notes for not administering vaccines, whereas contraindications (2.5%) and supply issues (1.8%) were the least common. When offered, vaccine acceptance increased for human papillomavirus, hepatitis B, measles-mumps-rubella-containing and varicella-containing vaccines and decreased for hepatitis A and meningococcal vaccines. Repeated offering of vaccines was central to increasing acceptance, in part due to increased opportunities to address specific concerns.


Background
Vaccine hesitancy-the delay in acceptance or refusal of vaccination despite availability of vaccine supply and services-is an urgent public health issue as evidenced in part by recent, large measles outbreaks in the United States and current challenges to achieving the COVID-19 vaccine coverage necessary to slow the spread of COVID-19 [1][2][3][4]. Although childhood vaccination rates are generally high in the United States [5], some parents are generally hesitant to vaccinate and others are doubtful about specific Advisory Committee on Immunization Practices (ACIP) routinely-recommended vaccines. For example, approximately 26% of parents report hesitancy related to influenza vaccine and about 23% for human papillomavirus vaccine (HPV) vaccine. Concerns are often centered around vaccine side effects, disease severity, and vaccine effectiveness [6,7]. Similar concerns have been expressed about COVID-19 vaccines [8][9][10].
Vaccine hesitancy is complex, varying over time, context, and vaccine [11]. Though some parents may opt to vaccinate their children in spite of minor concerns, others choose to delay all vaccines or only accept certain vaccines [12,13]. Even parents with minor concerns often have questions and may not know how to get their questions answered or who to trust [14]. As such, the nature of the provider-patient family relationship (good, neutral, non-existent) is critical [15]. A trusting provider-patient relationship lends itself to an environment supportive of vaccine conversations and positions the provider to influence parental positions on vaccination [12,13].
Because individual vaccine decisions affect the cumulative health of a community, adequate vaccine coverage and timely vaccine acceptance are critical to understand. Experience has demonstrated that even small declines in vaccination coverage can result in pockets of susceptible individuals in communities, leading to substantial public health and economic consequences [16]. As such, public health officials and large hospital networks, like the Children's Hospital of Philadelphia (CHOP), have sought to not only monitor local vaccination environments, but to also target specific vaccines with low coverage rates (e.g., HPV vaccine) [17]. Focused efforts to identify children due for recommended vaccines and implementation of strategies to catch-up on missed vaccines have been demonstrated to increase coverage levels [18,19]. To understand trends of vaccine refusal over time and to identify whether trends varied by vaccine, we used CHOP's electronic health record (EHR) to evaluate patient-and vaccine-level trends from 2013 to 2020.

Methods
Data Source and Study Subjects: The CHOP network is comprised of 31 primary care sites that serve over one million patients annually. The network includes six urban sites, of which three are "academic" where resident and fellow training occurs and 25 suburban primary care sites (22 in Pennsylvania and three in New Jersey), where all providers have completed training [20]. In addition to having providers in training, academic sites have a higher proportion of patients eligible for the Vaccines for Children Program (VFC), the federal safety net that ensures eligible children who are uninsured, American Indian or Alaska native, or Medicaid-eligible have public insurance coverage for vaccination services [21].
The study was completed using EHR data (Epic Systems, Inc., Verona, WI, USA) from all CHOP network sites. Data represented children through to 19 years of age who received care between 1 January 2013, and 31 December 2020. The cohort included 403,488 unique patients seen at least once during the study period and represented 1,449,061 annualized patient visits.
Variables and Outcomes: We extracted patient-level demographic variables (gender, race, ethnicity), visit-level data [year of service, age at time of visit (0-11 months, 1-10 years, 11-17 years, 18-19 years), insurance/VFC eligibility status, clinic location, diagnosis codes], and vaccination-level data (documentation of vaccine administration, contraindications, refusal, and reasons for refusal). Annualized patient-year observation cohorts were created for each calendar year (2013-2020) by calculating the number of patients multiplied by the number of visits per year. Demographics were obtained from information listed in the first preventive care visit of each year. For visit-level and vaccination-level data, we aggregated all EHR system generated prompts for ACIP-routinely recommended vaccines [22] to determine when vaccinations were due and to identify vaccine acceptance and vaccine refusal markers over each year of observation for each patient. Vaccine acceptance was defined as accepted (i.e., administered vaccination) over the number of times offered.
Vaccine refusal was identified using four EHR markers: ( be documented in a section of the EHR called the "problem list". This section helps providers track active and resolved chronic health conditions that may impact current or future patient care decisions [23]. This marker was counted as a patient-level marker and is the only marker that disables clinical decision support to the provider. (4) Marker 4. Vaccine refusal letter. Some patient records include a signed vaccine refusal letter which discusses the benefits of vaccination. Parents sign that they have reviewed the letter and wish to decline vaccinations for their child.
Vaccine refusal is defined as one or more of these markers in a patient record and is not specific to a specific antigen. Patient refusal status could change over time and can be independent of vaccination intent.
Analysis: We performed a repeated cross-sectional observational analysis of utilization patterns of EHR refusal markers by year and over time. Our primary unit of analysis was person-years, with data from patients who were seen for multiple visits within a calendar year merged into a single observation, the "annualized patient visit". Vaccine refusal and acceptance was calculated across annual cohorts of patients based on demographic variables (practice type, gender, race, ethnicity, age, and payor type) and vaccine (i.e., antigen). Confidence intervals around proportions were calculated using the prop.test() function in R.
We analyzed the relationship between individual study variables of gender, race, ethnicity, VFC-eligibility (as a marker for insurance status), practice group location, and age at visit with the main outcome of documentation of refusal markers. Significance of each variable was assessed using chi-square, and all significant variables were evaluated using multivariable logistic regression analysis using the glm() function in R.
For adolescents only, we calculated vaccination coverage by the Healthcare Effectiveness Data and Information Set ® (HEDIS) Immunizations for Adolescents (IMA) quality measure [one dose of meningococcal vaccine, one tetanus, diphtheria, acellular pertussis (Tdap) or tetanus (TD) vaccine and the complete human papillomavirus vaccine (HPV) series by an adolescents' 13th birthday (2 or 3 doses depending on year of analysis). HPV 2-dose recommendations for those younger than 15 years of age were published in 2016 [24][25][26]. All analyses were performed using RStudio version 1.0.136 (RStudio, PBC, Boston, MA, USA) [27]. This study was considered exempt by the CHOP Institutional Review Board.

Results
A total of 403,488 unique patients were seen at a preventive care visit at least once during the study period (Table 1). Most patients (67.6%) received preventive care at suburban sites (n = 272,895), followed by 21.6% at academic (n = 86,952) and 10.8% at urban non-academic sites (n = 43,461) ( Table 1).
Markers of refusal were present at 212,900 annualized patient visits from all 1,449,061 annualized patient visit years of observation. This accounted for 14.6% annualized visits in total and 25.1% of 847,890 annualized patient visits when a vaccine was due ( Table 2).    Visits with a refusal marker increased from 13.1% of all annualized visits in 2013 to a peak of 17.3% in 2019, corresponding to an increase from 21.4% to 28.5% of annualized visits when a vaccine was due (Table 2). Suburban practices in general had the highest refusal rates, reaching 20.0% of all annualized visits in 2019. Refusal rates were highest among children aged 11-17 years and among children who were not eligible for the VFC program (i.e., privately insured). Refusals declined in 2020 across all sites and most demographic'variables.
In the multivariable model, the odds of having a documented refusal marker were significantly increased in patients seen in suburban practices ( Table 3).
About half of patients with refusal markers (40.3-50.1%) received at least one vaccination (Table 4) compared with a majority of patients without refusal markers (88.2-91.0%) among annualized visits where at least one vaccine was due (data not shown). Across all study years, clinical documentation (refusal marker 1) was most utilized (83.0%) compared with other refusal markers measured (Table 4). A total of 13% of all annualized patient visits included multiple markers. The most common combination of markers (30.7%) was clinical documentation (marker 1) and visit diagnosis (marker 2). For 2.9% (n = 11,899) of those patients with a "problem list" refusal indicator (marker 3), the entry was ultimately removed. The average duration of the entry was 2.37 years (or 832 days). The median age at "problem list" entry removal was 4.5 years (inter quartile range: 2.0-11.8 years).
Vaccine coverage rates among adolescents as measured by the HEDIS criteria increased from 41.8% in 2013 to 61.8% in 2020 with the lowest vaccination coverage in suburban sites and the highest in urban academic sites (Table 6). This increase was driven largely by improvements in vaccination rates in the suburban sites, which increased from 36.6% to 57.0% across the study period.
Of the 289,622 annualized visits with "some" or "no" vaccinations given associated with marker 1 (mean of 36,184 annual visits), parental or patient refusal was most often the reason for not vaccinating (60.1%), followed by other clinician or practice related reasons (31.9%), incomplete patient history (4.3%) contraindications to vaccination (2.5%), and vaccine supply shortage (1.8%).  Vaccine acceptance is defined as vaccine administration over all offerings in the given year for the given antigen. a Antigens are aggregated across all vaccine product types containing the given antigen (i.e., monovalent and combination products).

Discussion
The aim of this study was to use EHR records to examine trends in vaccine refusal and acceptance in a large outpatient pediatric network over a period of eight years (2013-2020). The study cohort included over 400,000 unique patients and almost 1.5 million annualized patient visits. The overall refusal rate was >14% of annualized patient-visit-year observations and 25.1% of annualized vaccine patient-year observations. Refusal rates increased from 13.1% of annualized visits in 2013 to a peak of 17.3% in 2019. Refusal rates were higher in suburban than in urban sites, highest in children aged 11-17 years and those privately insured. Trends in vaccine acceptance and refusal varied by antigen and enhance the existing literature [1][2][3][4]28,29]. Vaccination coverage in adolescents increased during the study period, a promising finding given the concerted effort to increase HPV vaccination nationwide [30]. Parental or patient refusal was the most common reason for not vaccinating (60.1%).
These results empirically demonstrate continued parental concerns for at least one ACIP-routinely recommended vaccine and though not documented in this study, concerns differ among antigens [28,29]. Providers, as a trusted source of medical information and vaccine advice, should continue to offer vaccines that have been refused, recognizing that over time parents may change their mind [28,29]. Sometimes parents and families need more time and multiple touchpoints with trusted sources to inform their decision-making. Reasons that influence decision-making and subsequent behavior could be many (e.g., provider presentation and offering of vaccinations, social changes, and new information).
To raise vaccination coverage for HPV, CHOP engaged in a quality improvement effort focused on engaging providers and prioritizing HPV vaccination through clinical-focused decision support, EHR-based alerts, transparent data reporting, on-line education sessions, and standing orders, as well as shifting initiation of vaccination earlier to nine years of age [17]. Other strategies anchored in family-focused decision support (e.g., automated reminder phone calls, education support), vaccine champions, and automation and standardization of QI efforts have also proven effective in raising coverage not only in pediatric but also adult efforts to increase vaccination coverage of recommended vaccines like HPV closer to national goals [30][31][32][33][34]. These strategies help support provider recommendations and engagements with families, which are central to increasing vaccination coverage rates.
However, there is also ambivalence in the patient population, as acceptance of some vaccines-hepatitis A, pneumococcal and meningococcal-have decreased. Other vaccines most frequently denied by parents for children include the hepatitis B, rotavirus, MMR, varicella, pneumococcal, and polio vaccines, with uptake of influenza and HPV vaccines remaining suboptimal as well [35,36]. Much of the existing literature have cited common concerns for hesitancy: lack of perceived need, vaccine safety, distrust in healthcare providers and the government, perceived lack of involvement in the decision-making process, immune system overload, religious objections, and lack of adequate time and resource to support vaccine conversations [28,29,35].
Prevalence of vaccine refusal markers also increased more rapidly in our academic urban sites that serve a higher proportion of VFC-eligible children, which suggest vaccine refusal patterns in urban settings may now be mirroring those more typically noted in suburban settings [37]. Our findings echo other studies on vaccine hesitancy and reinforce the notion that confidence in vaccines is nuanced and informed by many factors, which leads to confidence in some vaccines but not others [1-4,6-8, 28,29].
This study was unique in evaluating the clinician documentation of vaccine refusal during routine care. Documentation was collected routinely and repeatedly in our primary care network throughout our study period and makes our analyses unique compared to prior studies of vaccine hesitancy. However, these population-level data were not analyzed at the individual patient level, limiting the ability to identify risk factors and predictors for refusal or acceptance. One interesting finding, and its potential implications, that is worth noting is the removal of "problem list" entries which occurred around four years of age (Table 4), suggestive of the importance of preschool or kindergarten-based vaccine requirements. Although this study cannot prove causality, it provides hypothesisgenerating evidence to further evaluate potential differences among variables such as setting and insurance status.
In light of the potential for weakening of all vaccine mandates as a result of pushback against COVID-19-based mandates, exploring parental decision-making and views on vaccines can help inform policies and practices around vaccination [38,39]. Toward how to better inform the provider-patient conversation, facilitating detailed documentation of specific concerns about hesitancy in the EHR may be worth exploring as the use of clinical documentation versus other refusal markers such as the refusal letter is relatively easy.
Our study faced some notable limitations. First, the definition of refusal used in this study was based on specific elements of the EHR that may not capture nuanced reasons for vaccine refusal. Furthermore, refusal in our data did not specify whether refusal was to specific antigens or all antigens. Our markers relied on clinical documentation, and it is likely that some clinicians were inconsistent in documenting refusal, which may lead to an under reporting of rates.
Second, the data were not analyzed by patient, but rather aggregated for analysis. In addition, the same cohort of individuals was not followed over time. However, these estimates are still useful for understanding changes to vaccine refusal over time.
An additional limitation of this study is the impact of the COVID-19 pandemic (2020-2022) on perceptions of risk and attitudes about vaccines and vaccinations and the possible selection bias in that children coming to office visits since the COVID-19 pandemic (Spring 2020) may not be the same children who came in the prior year (i.e., parents more favorably disposed to health care or a group less concerned about infectious disease and thus, less fearful about coming to the office).
Finally, our study was conducted in a single network of primary care practices, which may limit the generalizability of our results. However, there are diverse populations and practice cultures represented across the 31 practices in our network, and our analysis of trends in vaccine refusal may generalize to other health systems with similar demographics.

Conclusions
Although vaccine refusal is complex and based on a variety of factors, repeated offering of vaccines at multiple visits may increase vaccine acceptance and, therefore, vaccination coverage. Health systems should ensure that multiple opportunities exist for providers and patients to have vaccine conversations to both build upon and leverage the trust common to this relationship.  Informed Consent Statement: Informed consent not required as per IRB for this study of retrospective data in the electronic record.
Data Availability Statement: Deidentified aggregate data can be made available upon request of the authors.