Factors Associated with COVID-19 Vaccine Hesitancy after Implementation of a Mass Vaccination Campaign

An online cross-sectional study on COVID-19 vaccination adhesion was conducted in Portugal nine months after vaccination rollout (September–November 2021). Logistic regression was used to identify factors associated with hesitancy to take the COVID-19 vaccine in the community-based survey, “COVID-19 Barometer: Social Opinion”. Hesitancy was 11%; however, of those, 60.5% stated that they intended to take the vaccine. Hesitancy was associated with factors such as lower monthly household income; no intention of taking the flu vaccine this year; perceived reasonable health status; having two or more diseases; low confidence in the health service response; worse perception of the adequacy of anti-COVID-19 government measures; low or no perceived risk of getting COVID-19; feeling agitated, anxious or sad some days; and lack of trust in the safety and efficacy of the vaccines. Confidence in vaccines, namely against COVID-19, is paramount for public health and should be monitored during vaccination rollout. Clear communication of the risks and benefits of vaccination needs improvement to increase adherence and public confidence.


Introduction
Until 22 November 2021, more than 262 million COVID-19 cases and 5.2 million COVID-19 deaths had been confirmed worldwide [1]. The availability of vaccines has brought a new breath of hope for a return to "normal life" [2]. However, with this accelerated vaccine development, the issue of vaccine hesitancy has resurfaced [3][4][5][6], leveraging both scientific and public debate on this topic and its impact on the success of immunisation programs [3][4][5][6][7][8][9][10]. The European Union vaccination targeted a 70% vaccination rate of the adult population by the summer of 2021, reaching 71.6% on 12 September 2021 [2]. However, pockets of low vaccine uptake, which can compromise the vaccination coverage, still exist in this region [2]. Hence, the risk of increasing numbers of severe cases, hospitalisations and deaths associated with this disease still represent a major public health concern [2].
Vaccine hesitancy was identified by the World Health Organization (WHO) as one of the top 10 threats to global health in 2019 [11], and it can be defined as the delay in acceptance, reluctance, or refusal of vaccination despite the availability of vaccination services [11,12]. Vaccine hesitancy is the outcome of a complex decision-making process [2,12], Additionally, the world has been facing the emergence of new virus mutations with potential higher transmissibility and virulence [41] and new threats to immunity, including waning and new variants [42]. By the end of 2021, many countries have experienced new waves of the COVID-19 pandemic, and on 4 November the World Health Organization declared, once again, Europe as the epicentre of the pandemic [43]. Thus, understanding the reasons behind vaccine hesitancy [2], specifically throughout the vaccination campaign rollout, is of paramount importance to tailor policies and media campaigns to guarantee a high coverage for these new vaccines [3]. Hence, this study aims to assess and identify factors associated with COVID-19 vaccine hesitancy in Portugal, nine months after the rollout of the country's vaccination program, in a context of high vaccination coverage.

Study Design
We used data from the community-based survey "COVID-19 Barometer: Social Opinion", launched by the National School of Public Health at the NOVA University of Lisbon (ENSP/NOVA), to identify and monitor the perception of the population regarding the pandemic and its impact on their health, wellbeing and daily-life [44]. The ongoing questionnaire was implemented at the beginning of the COVID-19 pandemic in 2020, with more than 220,000 answers by 30 November 2021. The questionnaire is flexible and allows for the inclusion of new questions and withdrawal of old ones according to the pandemic context. We analysed data between 18 September and 26 November 2021, approximately nine months after the country's vaccination program began. This time frame was selected because, according to the country's vaccination strategy and availability, every citizen older than 16 years old willing to be vaccinated would have access to it and could be vaccinated before our study started. By 20 September, 84.1% of the eligible population was fully vaccinated, and that number increased to 88.4% by 29 November [40]. Participants who were not living in Portugal were excluded from the analysis. Participants could fill in the questionnaire once or every two weeks, which would allow the analysis of trends and variations in responses [44]. We only used one questionnaire per participant for this study and considered the last time the participant answered the questionnaire between 18 September and 26 November 2021.

Outcome
The outcome question was "What is your intention to get the COVID-19 vaccine when called?". In this question, participants could answer whether they wanted to take the vaccine, were undecided, did not want to take it, or were already vaccinated. For the analysis, the dependent variable was categorised into vaccinated and hesitant. Participants who answered "I am already vaccinated" corresponded to vaccinated individuals. In contrast, the remaining participants, i.e., participants who answered "I will take it", "I have not decided yet" and "No" were considered hesitant as all these answers represented some form of delay, reluctance or vaccination refusal [12].

Independent Variables
The independent variables collected by the survey "COVID-19 Barometer: Social Opinion" and included in this analysis were those considered potentially associated with vaccine hesitancy. These variables were grouped following the Working Group on Vaccine Hesitancy framework as contextual, individual and group, and vaccine-specific influences [12], with a fourth category added regarding disease-specific variables. This classification was used in a prior cross-sectional analysis in Portugal using data from before the vaccines were widely available for the population [38]. Table 1 presents the variables considered for this analysis and their categories.

Statistical Analysis
The study outcome (dependent variable) was binary, vaccinated or hesitant, and the independent variables were grouped based on the vaccine hesitancy determinant matrix in Table 1. This self-reported vaccination rate corresponded to the primo-vaccination rate, as at this time, the booster campaign had not yet been extended to the general population. Variables were initially described using absolute and relative frequencies.
Logistic regression models were fitted. We estimated a crude odds-ratio (OR) for all variables with the corresponding 95% confidence interval (95% CI). Our interest was to assess and identify factors associated with COVID-19 vaccine hesitancy, an explanatory aim and not predictive, and we adjusted separate logistic regression, adjusted for age, gender, education, and month of the questionnaire, for each variable of interest.
We conducted a sensitivity analysis without participants who answered "I will take the COVID-19 vaccine". We assumed that those participants might be less hesitant than those who answered that they would delay or refuse to take the vaccine.
All statistical analyses were performed using R 4.0.2 [45].

Results
A total of 3232 individuals were included in the analysis, of which 2875 (89%) were vaccinated and the remaining 375 (11%) were hesitant. The distribution of hesitant consisted mainly of participants who answered "I will take the vaccine" (60.5%), followed by participants who said they would not take the vaccine (27.5%) and undecided participants (12%). The characteristics of the sample are presented in Tables 2-5.  Most of the individuals that answered the questionnaire were women (75%), aged between 25 and 64 years (69%) and had a university degree (76%). Around 55% of those hesitant about the COVID-19 vaccine perceive their health status as very good/good. The majority of vaccinated participants (90%) perceived COVID-19 vaccines as safe and effective, while the proportion of hesitant participants who perceived the COVID-19 vaccines as safe and effective was around 64%.
The results of the regression models are presented according to the groups on the vaccine hesitancy determinant matrix. The OR and aOR and their respective 95% CI are in the Supplementary Materials (Table S1). The sensitivity analysis can also be found in the Supplementary Materials (Table S2).

Determinants of Vaccine Hesitancy: Contextual Influences
Participants aged 65 to 79 years had higher odds of hesitancy than participants aged 50 to 64 years (aOR: 1.58, 95% CI: 1.20, 2.07). Higher odds of vaccine hesitancy were Vaccines 2022, 10, 281 7 of 14 also found for individuals with no education/basic education and secondary education, when compared to those with a university degree (no education/basic education aOR: 1.60, 95% CI: 1.04, 2.41; secondary aOR: 1.46, 95% CI: 1.12, 1.87). Participants with higher monthly household incomes had lower odds of hesitancy than participants with a monthly household income smaller than EUR 650. Participants who answered the questionnaire in September had lower hesitancy odds than those who answered in November (aOR: 0.67, 95% CI: 0.47, 0.93) (Table S1, Figure 1). compared to those with a university degree (no education/basic education aOR: 1.60, 95% CI: 1.04, 2.41; secondary aOR: 1.46, 95% CI: 1.12, 1.87). Participants with higher monthly household incomes had lower odds of hesitancy than participants with a monthly household income smaller than EUR 650. Participants who answered the questionnaire in September had lower hesitancy odds than those who answered in November (aOR: 0.67, 95% CI: 0.47, 0.93) (Table S1, Figure 1).
In the sensitivity analysis, i.e., excluding individuals who answered that they would take the vaccine, education was no longer associated with vaccine hesitancy. In turn, gender and loss of income became statistically significant, with women presenting lower odds of hesitancy (aOR: 0.57, 95% CI: 0.40, 0.83), and participants who lost income during the pandemic presenting higher odds of hesitancy (aOR: 1.46, 95% CI 1.01, 2.09). Interestingly, the results for age changed in the sensitivity analysis, with participants aged less than 50 years presenting higher odds of being hesitant than participants between 50 and 64 years old (16-24 years aOR: 2.88, 95% CI 1.05, 6.73; 25-49 aOR: 1.99, 95% CI: 1.36, 2.96). The association with monthly household income was still present (Table S2, Figure 1).

Figure 1.
Forest plot of vaccine hesitancy for contextual influences. Adjusted odds ratio (adjusted for gender, age, education, and period of questionnaire) and the respective 95% confidence intervals are denoted by black dots and black lines, respectively. Forest plot confidence intervals were cut off at 3. (A) Results of the main analysis-vaccinated vs. hesitant (refuse, undecided and would take the vaccine); (B) results for the sensitivity analysis, removing individuals who would take the vaccine.

Determinants of Vaccine Hesitancy: Individual Influences
Individuals who did not intend to take the flu vaccine this year had higher hesitancy odds than those who took or intended to take the flu vaccine (aOR: 2.65, 95% CI: 2.00, 3.53). Those who perceived their health status as reasonable had lower odds of hesitancy In the sensitivity analysis, i.e., excluding individuals who answered that they would take the vaccine, education was no longer associated with vaccine hesitancy. In turn, gender and loss of income became statistically significant, with women presenting lower odds of hesitancy (aOR: 0.57, 95% CI: 0.40, 0.83), and participants who lost income during the pandemic presenting higher odds of hesitancy (aOR: 1.46, 95% CI 1.01, 2.09). Interestingly, the results for age changed in the sensitivity analysis, with participants aged less than 50 years presenting higher odds of being hesitant than participants between 50 and 64 years old (16-24 years aOR: 2.88, 95% CI 1.05, 6.73; 25-49 aOR: 1.99, 95% CI: 1.36, 2.96). The association with monthly household income was still present (Table S2, Figure 1).

Determinants of Vaccine Hesitancy: Individual Influences
Individuals who did not intend to take the flu vaccine this year had higher hesitancy odds than those who took or intended to take the flu vaccine (aOR: 2.65, 95% CI: 2.00, 3.53). Those who perceived their health status as reasonable had lower odds of hesitancy than those who perceived it as very good/good (aOR: 0.75, 95% CI: 0.59, 0.94). Participants who had two or more diseases also had lower odds of hesitancy than participants without diseases (aOR: 0.59, 95% CI: 0.42, 0.81). Participants who reported feeling agitated, sad, or anxious some days due to the physical distance measures had lower odds of hesitancy than participants who reported never having these feelings (aOR: 0.65, 95% CI: 0.51, 0.84) (Table S1, Figure 2). Results were similar in the sensitivity analysis (Table S2, Figure 2). than those who perceived it as very good/good (aOR: 0.75, 95% CI: 0.59, 0.94). Participants who had two or more diseases also had lower odds of hesitancy than participants without diseases (aOR: 0.59, 95% CI: 0.42, 0.81). Participants who reported feeling agitated, sad, or anxious some days due to the physical distance measures had lower odds of hesitancy than participants who reported never having these feelings (aOR: 0.65, 95% CI: 0.51, 0.84) (Table S1, Figure 2). Results were similar in the sensitivity analysis (Table S2, Figure 2).

Figure 2.
Forest plot of vaccine hesitancy for individual influences. Adjusted odds-ratio (adjusted for gender, age, education, and period of questionnaire) and the respective 95% confidence intervals are denoted by black dots and black lines, respectively. Forest plot confidence intervals and estimates were cut off at 3. (A) Results of the main analysis-vaccinated vs. hesitant (refuse, undecided and would take the vaccine); (B) results for the sensitivity analysis, removing individuals who would take the vaccine.
Results were similar in the sensitivity analysis. Increased odds of hesitancy were found for participants who had low confidence in the health services' response to COVID-19 and non-COVID-19 and who perceived the measures implemented by the government as inadequate. Participants who perceived high risk of getting COVID-19 and developing severe disease had lower odds of hesitancy (Table S2, Figure 3).

Determinants of Vaccine Hesitancy: COVID-19-Vaccine-Related Influences
Participants who perceived the COVID-19 vaccines as unsafe had higher hesitancy odds than those who perceived them as safe (aOR: 15.82, 95% CI 11.67, 21.54). Similarly, participants who perceived the COVID-19 vaccines as ineffective had higher hesitancy odds than those who perceived them as effective (aOR: 10.32, 95% CI 7.81, 13.64) (Table  S1). In the sensitivity analysis, results for the safety and efficacy of the COVID-19 vaccines were similar, but the association with time was no longer significant (Table S2).

Discussion
The present study investigated the factors associated with vaccine hesitancy in 3232 participants from the "COVID-19 Barometer: Social Opinion", nine months after the vaccination rollout in Portugal, following completion of the mass vaccination campaign among people aged 16 years and older. Most participants were already vaccinated (89%) at the time of the study. This self-reported vaccination rate corresponded to the primo- Results were similar in the sensitivity analysis. Increased odds of hesitancy were found for participants who had low confidence in the health services' response to COVID-19 Vaccines 2022, 10, 281 9 of 14 and non-COVID-19 and who perceived the measures implemented by the government as inadequate. Participants who perceived high risk of getting COVID-19 and developing severe disease had lower odds of hesitancy (Table S2, Figure 3).

Determinants of Vaccine Hesitancy: COVID-19-Vaccine-Related Influences
Participants who perceived the COVID-19 vaccines as unsafe had higher hesitancy odds than those who perceived them as safe (aOR: 15.82, 95% CI 11.67, 21.54). Similarly, participants who perceived the COVID-19 vaccines as ineffective had higher hesitancy odds than those who perceived them as effective (aOR: 10.32, 95% CI 7.81, 13.64) (Table S1). In the sensitivity analysis, results for the safety and efficacy of the COVID-19 vaccines were similar, but the association with time was no longer significant (Table S2).

Discussion
The present study investigated the factors associated with vaccine hesitancy in 3232 participants from the "COVID-19 Barometer: Social Opinion", nine months after the vaccination rollout in Portugal, following completion of the mass vaccination campaign among people aged 16 years and older. Most participants were already vaccinated (89%) at the time of the study. This self-reported vaccination rate corresponded to the primo-vaccination rate, as, at this time, the booster campaign had not yet been extended to the general population. We found that the following factors were associated with hesitancy in the main and sensitivity analysis: (i) contextual factors: lower monthly household income; (ii) individual factors: no intention of taking the flu vaccine this year, perceiving their health status as reasonable and having two or more diseases; (iii) COVID-19 influences: low confidence in the health service response to COVID-19 and non-COVID-19, worse perception of the adequacy of measures implemented by the government, perceiving a low or non-existent risk of getting COVID-19, and feeling agitated, anxious or sad some days due to the physical distance measures; and (iv) COVID-19-vaccine-specific factors: lack of trust in the safety and efficacy of the vaccines.
We found that 11% of the participants were still hesitant in a highly adhering setting. To the best of our knowledge, this is the first estimate following completion of the mass vaccination in individuals aged 16 and above. By 14 November 2021, official reports reported a 100% vaccination rate for individuals older than 64, 99% for individuals between 50 and 64, 94% for individuals between 25 and 49, 91% for individuals between 18 and 24 and 87% for individuals between 12 and 17 years old [46]. Our results and further data on un-or incompletely vaccinated hospitalised individuals [47] indicate that existing vaccination coverage is likely overestimated. Yet, 60.5% of those hesitant corresponded to participants who stated that they would take the vaccine, thus representing a further opportunity for public health authorities and professionals to invite individuals to get vaccinated. It is crucial to explore further why some individuals are keen to take the vaccine but have not done it yet at the time of the study. The safety and efficacy perception in the vaccines among the vaccinated group were 96% and 94%, respectively. Relatively high values were also found among the hesitant group (65% and 63%, respectively), which may indicate that, despite perceiving vaccines as safe and effective, other factors in the COVID-19 disease-specific category may make people hesitant (e.g., less fear for health or worry about COVID-19, perception of lower risk of infection, belief that it is not a severe disease). These factors have been explored in the literature [48], and further quantitative and qualitative studies can shed some light on possible reasons for this phenomenon and potentially reduce hesitancy by tackling some causes behind no vaccination for these individuals. Additional research could also further explore whether this delay is related to fears associated with vaccination or individual health status, or due to practical reasons, such as difficult access, inability to leave work, among others.
On the other hand, hesitancy has decreased since the beginning of the vaccination programme. At the beginning of the year, we found a hesitancy rate of 65% using data from the same survey [38]. This difference indicates the complexity of vaccine hesitancy and how hesitancy changes over time as more information is made available. Our previous study found that questionnaires answered before the release of information regarding the safety and efficacy of COVID-19 vaccines were strongly associated with higher odds of hesitancy [38]. These results obtained using the same questionnaire at two very different time points highlight the need to assess vaccination intent and uptake [3] once a vaccination program is rolled out and over time. Hesitancy rates and the characteristics of hesitant subjects might vary, and monitoring of hesitancy rates was previously recommended by the ECDC [2]. This is particularly relevant as booster doses are currently being administered [49] with a lower uptake than previously observed for the primo vaccination.
Negative perceptions towards the efficacy [5,8,13,14,17,19,21,25,31,34,37] and safety [5,8,10,13,14,[16][17][18][19]21,25,27,[29][30][31]33,37,40] of COVID-19 vaccines were highly linked to vaccine hesitancy. Given the complexity of this topic, aligned with the growing misinformation present in several media regarding the development process of vaccines, concerns about the safety and efficacy of the vaccines are understandable [2,15]. Additionally, the population's level of trust in vaccines and the availability of understandable and reliable information on these uncertain times can be impacting factors in the adherence to this vaccination [50]. There is a need to implement strategies to address the population perception and misconceptions regarding the efficacy and safety of COVID-19 vaccines, as these are strongly associated with hesitancy. Health status and having diseases were other important factors in vaccine hesitancy. This study found that participants who perceived their health status as reasonable and had two or more diseases had lower odds of hesitancy than participants with a good health perception and no diseases [8,18,34,36,40,46]. Additionally, we found that participants who felt negative emotions some days had lower odds of hesitancy.
We also found higher odds of hesitancy for participants with low confidence in the health services response to COVID-19 and non-COVID-19 and who perceived the measures implemented by the government as inadequate, matching prior publications on this topic [15,46]. Higher odds of hesitancy were also found for participants who perceived their risk of getting COVID-19 as low or non-existent, which was in agreement with the literature [5,8,10,[14][15][16][17][18]22,25,27,30,31,33,37,46]. Risk perception and trust in the authorities and health institutions can vary according to a country's pandemic context during a specific period [51]. For instances, the low infection rate during the period of analysis (with a 7-day rolling average ranging from 94.11 to 266.88 per million inhabitants on 18 September and 26 November, respectively [40]) could be detrimental to the vaccination drive in this particular time period.
Furthermore, no intention of taking the flu vaccine in the current year was associated with hesitancy to the COVID-19 vaccination, matching prior data on this topic [5,8,18,27,28,46] and suggesting that global attitudes towards these vaccines may cluster together to influence decision making.
Lower odds of hesitancy were found for participants with higher monthly household incomes [2,[4][5][6]8,10,14,16,17,20,34]. We also found lower odds of hesitancy for individuals with a university degree, which is in agreement with other studies [4,6,[8][9][10]13,14,17,18,21,22,24,27,28,34,46]. However, analysing only participants that refused or were undecided, this association is no longer significant. Loss of income during the pandemic was described as a hesitancy driver in the sensitivity analysis. This association aligned with the outcomes of the prior analysis of this survey conducted at the beginning of the year [46]. It is important to further explore these results in different socioeconomic groups with different educations, as vaccine hesitancy determinants do not have associations in only one direction [12]. For instance, our previous results showed that higher household income was associated with lower odds of hesitancy for individuals without a university degree. In contrast, higher household income was associated with higher odds of hesitancy for individuals with a university degree [38].
We found divergent results in the main and sensitivity analyses regarding age. Considering all participants who reported not being vaccinated, higher odds of hesitancy were found for individuals aged between 65 and 79 years old than individuals between 50 and 64 years old. However, removing from the analyses participants who reported intention to vaccinate, we found that participants younger than 49 had higher odds of hesitancy than individuals between 50 and 64 years old. The literature is in agreement with higher hesitancy for younger age groups [2][3][4]6,[8][9][10]13,14,[16][17][18]20,22,24,30,33,36,37,40,46]. We categorised age using the same age groups used by the official Portuguese reports for our analysis, which indicate a similar trend [46]. Other results lost or gained statistical significance in the sensitivity analysis. Women had lower odds of hesitancy only in the sensitivity analysis, although in the main analysis, the upper limit of the confidence interval was 1.02. In contrast, our previous study found increased odds of delaying vaccination for women [38], suggesting once more the changes associated with hesitancy over time. However, several studies reported lower odds of hesitancy for women [21], although this result was not always replicated [3,6,10,13,18,20,24,26,28]. When considering the month of response, the lower hesitancy for participants answering the questionnaire in September (versus November responders) lost statistical significance during the sensitivity analysis. Nevertheless, this result points to a temporal variation in vaccination hesitancy, as previously reported by other authors [13][14][15]18], highlighting the need for continuous monitoring.
One of the strengths of the present study is the collection of data through a nationwide survey, analysed using a comprehensive framework on vaccine hesitancy that allowed us to identify contextual, individual, and vaccine-specific factors associated with vaccine uptake and hesitancy. This study was based on an online, voluntary survey and presented some limitations. The sample is not representative of the Portuguese population, as there was a clear preponderance of women and participants with higher education levels. There is a possibility of sampling bias (due to the online nature of the survey), selection bias (participants might be more conscious of the seriousness of COVID-19 than non-participants), and response, non-response and social desirability bias. Despite these limitations, our study indicates that official data on coverage are likely overestimated.
In conclusion, Portuguese adults showed good adherence to the COVID-19 mass vaccination efforts, with hesitancy remaining in a sub-group mostly due to delayed vaccination. Negative vaccine-related perceptions, low risk perceptions and general sentiments of distrust were identified as important factors associated with hesitancy. These factors can inform the development of tailored interventions to tackle COVID-19 vaccine hesitancy.