Development and Validation of a Questionnaire to Assess Knowledge and Attitudes toward COVID-19 Preventive Measures in Romania

The World Health Organization warns about the threat of the COVID-19 sixth wave. Our aim was to propose the first validated Romanian questionnaire to assess people’s level of education and attitudes towards general measures to protect against COVID-19 infection. Our study was conducted on a sample of 194 people. The first version of the questionnaire consisted of 40 items. Items that did not meet psychometric criteria were removed. Latent components/factors were identified through exploratory factorial analysis (EFA). The Cronbach’s alpha coefficient was used to assess internal fidelity. The EFA identified three factors. Factor 1 was named “Compliance with protective measures”, factor 2 was “Attitudes toward vaccination” and factor 3 was “Attitudes regarding potential COVID-19 therapies”. The final version of the questionnaire consists of 16 items. The test’s final score predicted the presence of vaccination with an accuracy of 0.773. The questionnaire score, the diagnosis of diabetes, the advice provided by healthcare workers and the medical profession proved to be significant predictors of vaccination. The implementation of our questionnaire within national programs could identify populational areas that need specific interventions to reach vaccination targets and prevent a full-blown sixth wave of the COVID-19 pandemic in Romania.


Introduction
The global COVID pandemic has affected more than half a billion people and has so far led to around 6,332,618 deaths, driving economic, political and social changes. The protective measures against SARS-CoV-2 virus infection are currently both general and recommended in all epidemics with respiratory-transmitted pathogens (social distancing, wearing protective masks), but also specific such as testing and vaccination. The role of these measures is to reduce the number of severe forms of disease and death but also to prevent the overcapacity of hospitalization and intensive care at the national level [1].
In a time of measures' relaxation (the removal of the mandatory wearing of the mask in public places, the changes to the rules for testing and isolating new cases) after a period of decline in the incidence of COVID-19 cases throughout March, April and May 2022, our paper provides a warning signal based on latest World Health Organization (WHO) briefing (6 July 2022) that urges countries to readopt tried and tested public health measures and plug immunity gaps as infections new sub-lineages have risen 30% in the past two weeks (as of 6 July 2022) and increased in four out of six WHO regions [2]. This warning comes at a time when many countries have still not reached WHO's vaccination targets (Bulgaria 29.9%, Slovakia 50.8%, Croatia 55.6%, Slovenia 57.1%, Poland 59.6%) [3] or gained access to antivirals. This is also the case for Romania, with a cumulative uptake of full vaccination in the total population of 42.3% as of 14 July 2022 [3]. Moreover, even for those vaccinated, vaccine efficiency declined with the appearance of new mutations in the structure of spike protein, thus amplifying the existent hesitancy in adherence to preventive measures [4]. Breakthrough infections in vaccinated populations with the recently emerged SARS-CoV-2 Omicron variant in the fifth wave of the pandemic exposed some of these limitations of COVID-19 vaccines and highlighted the need for other medical treatments such as drug therapy [5]. However, antiviral drugs prove to reduce the risk for severe outcomes and death in non-hospitalized patients or when given in the first 5 days after COVID diagnosis [6][7][8], but fewer data are on efficiency in hospitalized patients with severe disease [8]. These findings reinforce the need not to underestimate general preventive measures and the role of massive testing.
Currently, there is no validated Romanian questionnaire to assess people's attitudes toward anti-COVID vaccination and general preventive measures. Such a tool is vital for improving people's attitudes and compliance and reaching vaccination targets.
In this context, we considered the opportune warning from WHO, and we aligned ourselves to this call by proposing the first validated Romanian questionnaire to assess people's level of education and attitudes towards general measures to protect against COVID-19 infection. Another objective of our study was to characterize the level of adherence and the main motivations behind acceptance or refusal to follow the imposed preventive measures. The ultimate goal was to use the proposed questionnaire within national programs to identify populational areas with low adherence and establish specific educational programs to increase the willingness to vaccinate against COVID-19 in an effort to prevent a full-blown sixth wave.

Questionnaire Development
We designed a survey tool, in the Romanian language, based on the guidelines provided by the World Health Organization Regional Office for Europe (WHO) [18] for the European member states that wish to conduct behavioral insights studies related to COVID-19.
The items of the questionnaire were proposed and reviewed by three physicians. The variables measured by this questionnaire addressed protective measures against COVID-19 infection: social distancing, masking, testing, vaccination, green certificate and following medical advice. As defined by the European Commission, the green certificate is "a proof that a person has been vaccinated against COVID-19, has received a negative test result or has recovered from COVID-19" [19].
The Likert scale was used to assess the answer to each question: 0-"Strongly Disagree", 1-"Disagree", 2-"Neither Agree nor Disagree", 3-"Agree", 4-"Strongly Agree". The final score of the test represents the sum of all item scores. The lower the total score, the more people would tend to have a positive attitude toward vaccination and governmental measures.

Questionnaire Validation
The tested hypothesis was the following: patients with a better questionnaire score on the knowledge of preventive measures will have more frequent correct proangiogenic behavior.
After its development, the validation process followed several steps.

Face Validity
Two experts evaluated the proposed items for their clarity, adequacy and significance in relation to appropriate measuring of knowledge and attitudes toward SARS-CoV-2 infection. Each item was evaluated by experts in relation to three standards: - Relevance to the purpose of the questionnaire; -Good matching with the recommendations provided to patients by national education programs; - The relevance of the measures and behaviors we seek to assess.
Each of these three aspects, corresponding to an item, was assessed on a Likert scale from 1 to 5. Items considered by both experts to be unimportant were deleted.
The questionnaire was then tested for ease of use or ambiguity in a pilot test with ten nursing students. Items with an ambiguous definition of terms detected by the pilot test were removed or rephrased.

Study Design and Participants
An observational study was then conducted on a convenience sample of 194 people. The criteria for inclusion was the completion of a Google form-based questionnaire distributed to the general population through social media (Facebook) between 7 March 2022 and 9 June 2022. All incomplete questionnaires were excluded. Respondents were not paid for their participation. The dissemination of the questionnaire drawn up on Google sheets was conducted through Facebook by the authors.
Approval from the Research Ethics Committee of the "Gr. T. Popa" University of Medicine and Pharmacy Iaşi (Nr. 155/23.02.2022) was obtained before contacting participants and collecting data. All participants provided signed informed consent. The informed consent was included as an introduction to the questionnaire. All data were anonymized to maintain participant confidentiality.

Data Collection
Demographic data (age, gender, educational level, profession, medical history, vaccination history) and questionnaire answers revealing knowledge, attitudes and behaviors related to the COVID-19 outbreak were recorded for each responder.

Psychometric Evaluation
We used psychometric criteria on the data obtained in the study to evaluate the following characteristics: (a) Items to which at least 5% of people responded in the same way were retained [20,21]; (b) The discrimination index (DI) is a measure of how well an item can differentiate between good candidates and less able ones. For a specific item, DI, expressed as a percentage, was determined by calculating the difference between the number of people in the upper quartile and, respectively, in the lower quartile of the test's total score, considering only responders who answered correctly to that item. Items with a coefficient of less than 20% do not have a sufficient capacity to discriminate between subjects and were therefore removed [21]; (c) The ability of each element to discriminate between people with different levels of knowledge was measured by correlating the score for each element with the overall test score (TSC-total score correlation). All items with significant correlations at a p < 0.001 were retained [21].

Identifying Underlying Components
The next step was to identify latent components/factors through exploratory factorial analysis. Factorial analysis validates the construction of the questionnaire and reduces the number of variables to a number of components called factors that encompass a subset of the variables sharing a certain communality expressed through multicollinearity.
The Pearson correlation coefficient was determined for conducting the discriminative analysis. In order to evaluate construct validity, we first assessed the homogeneity of variance with the Levene test. Afterward, we determined whether there were significant differences in the mean scores using the ANOVA test. The construct validity was tested using exploratory analysis. The Kaiser-Meyer-Olkin measure of sampling adequacy (KMO) was used to signal in advance whether the sample size is large enough to extract factors reliably. The Bartlett's Sfericity Test was used to assess the presence of multicollinearity, and the p-value of Bartlett's test of sphericity was considered significant if it was lower than 0.05. The factor loading value of individual items is set to be greater than 0.25. We considered variables with a correlation coefficient greater than 0.4 as variables that contribute to the formation of a factor.
Principal component analysis (PCA) was used to reduce the dimensionality of the dataset in case of the presence of interrelated variables while retaining as much as possible of the variation.
Several indices were used to assess a good model fit for the construct, including the ratio of chi-square to degree of freedom (χ 2 /df) < 5.0, root mean square error of approximation (RMSEA) ≤ 0.08, comparative fit index (CFI) > 0.9, Tucker Lewis Index (TLI) > 0.9 and p > 0.05 for the chi-square test [22].

Internal Fidelity
The Cronbach's alpha coefficient was used to assess internal fidelity. The Guttman Split-Half coefficient showed the probability of test-retest reproducibility.

Statistical Analysis
The database containing the questionnaire responses was managed with Microsoft Office Excel 2007. Statistical processing was performed using the JASP 0.16.1 [15] and JAMOVI [14] software.
In order to test the normality of the distribution of the questionnaire scores, the Kolmogorov-Smirnov test was applied.
Sample size calculation was undertaken using the single proportion formula: n = Z α/2 2 ·P·(1 − p)/d 2 at 95% confidence interval, where Z α/2 = 1.96, P = prevalence of 50% and d = 5% of marginal error [23]. We also defined a reference group to estimate the sample size. Due to the lack of other available data, we used a population over 15 years old, with residency in the North East Region of Romania, reported on 1 July 2019 (n = 2,620,857) to define the size of the reference group [17]. Based on these calculations, the minimum sample size required was 370 (95% CI) or 165 (80% CI).
In order to emphasize the conformity and utility of our survey in screening for people's attitudes toward vaccination, we performed both simple and forwarded binomial logistic regression to predict the vaccination status using the test's score or other variables resulting from the factor analysis.
For ANOVA analyses, in the presence of two independent groups with α = 0.05 and a power of 95%, the sample size needed is estimated to be 210 participants. In the case of 190 participants, the power decreases to 93%. For logistical regression with 194 participants, OR = 1.3 and critical z 0.99, the power was estimated with the use of the G*Power 3.1.9.7 software at a value of 0.68.

Results
Our sample included 194 people, of which 137 (70.6%) were women, with a mean age of 41.5 years old and a standard deviation (SD) of 12.4. Age varied between 18 and 89 years old. Of all respondents, 58.2% inhabited urban areas. Regarding the geographical area, the participants came from the north-eastern part of Romania, and almost half of them were from Iasi county: Iasi 110, Bacau 14, Botosani 24, Galati 13, Neamt 12, Suceava 14, Vaslui 7. A significant proportion of the study sample declared to work in healthcare (27.53%) ( Table 1). The first version of the questionnaire reviewed by experts consisted of 40 items, to which the participants were instructed to respond according to their conviction ( Table 2).  The score for the items M2, M5, M6, M7, M8, T2, T4, V1, V2, V4, V5, V6, V7, V8, S1, S3, S4,  S5, S6, S7, S8, CV2, CV3 and CV4 was reversed.  The mean score of the questionnaire was 42.8 (95% IC: 41-44.6; SD = 12.6). Significant differences in the score were observed between subgroups regarding the area of residence, the presence of diabetes, the awareness of the potential severity of SARS-CoV2 disease and the profession (Table 3). Vaccinated people had significantly lower scores.  According to the psychometric criteria, items SD1, M2, M3, M4, M6, V1, V2, V3, V4, V6, V10, S3, S4, S6 and S7 did not meet the recommendation that at least 5% of people should respond in the same way and were eliminated. Items M2, M3, M4, M6, V2, V3, V6, S3, S4, S6 and S7 were withdrawn due to the DI that was lower than 20%. Item V3 was removed due to the lack of ability to discriminate between people with different levels of knowledge.
The exploratory factorial analysis for the identification of latent components identified three factors that together explained 52.7% of the total variation and were, therefore, chosen as the components of our questionnaire (Table 4, Figure 1). T The value of the Kaiser-Meyer-Olkin coefficient (0.791) and the result of Bartlett's test (χ 2 = 1571.865, p < 0.001) indicated the adequacy of the sample size chosen for our analysis. The names of the factors are given according to the variables that they encompass. Factor 1 was named "Compliance with protective measures", factor 2 was "Attitudes toward vaccination" and factor 3 was "Attitudes regarding potential COVID-19 therapies". For factor 1, there are high correlations between items exploring social distancing, testing and the protective role of mask-wearing. For factor 2, the items selected were addressing potential harmful side effects or inefficiency of the vaccines and attitudes toward restrictions associated with the green certificate. For factor 3, the questions investigated behaviors toward potential therapies for SARS-CoV-2 infection, uncertainty toward their scientific validation and distrust in the scientific plausibility of protective measures.   Principal component analysis (PCA) simplified the complexity of high-dimensional data while retaining trends and patterns. After performing PCA and excluding the variables with loadings lower than 0.5, the final model contains 16 items (Tables 5 and 6) and confirms the factorial structure of the proposed questionnaire with the three dimensions identified by the exploratory factorial analysis.   Principal component analysis (PCA) simplified the complexity of high-dimensional data while retaining trends and patterns. After performing PCA and excluding the variables with loadings lower than 0.5, the final model contains 16 items (Tables 5 and 6) and confirms the factorial structure of the proposed questionnaire with the three dimensions identified by the exploratory factorial analysis. The value of the alpha Cronbach coefficient shows a good internal fidelity of the questionnaire (0.853). The Guttman Split-Half coefficient shows the probability of testretest reproducibility. By applying the Split-Half method to show the probability of testretest reproducibility, a value of the Guttman Split-Half coefficient of 0.859 was obtained, which indicates that the fidelity of the scale is acceptable (Table 7). As another argument for the survey's ability to screen people's attitudes toward vaccination, we identified that the test's final score predicts the presence of vaccination for a cut-off value of 50% with an accuracy of 0.773 and an AUC of 0.793 ( Figure 2). The value of the alpha Cronbach coefficient shows a good internal fidelity of the questionnaire (0.853). The Guttman Split-Half coefficient shows the probability of testretest reproducibility. By applying the Split-Half method to show the probability of testretest reproducibility, a value of the Guttman Split-Half coefficient of 0.859 was obtained, which indicates that the fidelity of the scale is acceptable (Table 7). As another argument for the survey's ability to screen people's attitudes toward vaccination, we identified that the test's final score predicts the presence of vaccination for a cut-off value of 50% with an accuracy of 0.773 and an AUC of 0.793 ( Figure 2).  Table 8 illustrates the predictive power of various variables used in the factorial analysis to estimate the presence of vaccination. In total, the questionnaire score, the diagnosis of diabetes, the advice provided by healthcare workers and the medical profession proved to be significant predictors of vaccination.  Table 8 illustrates the predictive power of various variables used in the factorial analysis to estimate the presence of vaccination. In total, the questionnaire score, the diagnosis of diabetes, the advice provided by healthcare workers and the medical profession proved to be significant predictors of vaccination.

Discussion
Our paper represents the first validation study of a Romanian questionnaire aimed at screening attitudes toward vaccination and preventive measures against SARS-CoV-2 infection. Our article contributes to the literature with a scientific method to validate a questionnaire in a non-English language. As such, our method can serve as a model to be used by researchers in other countries to develop and validate surveys in their language. Moreover, our article characterizes the level of adherence and the main motivations behind acceptance or refusal to follow the COVID-19 preventive measures of a Romanian population.
Even though we are now in a period of relaxation of the public health measures due to an apparent withdrawal of the epidemic in the first half of the year, recent publications warn about several countries entering the sixth wave of the epidemic [24][25][26][27]. According to the WHO's latest briefing (6 July 2022), preventive measures and vaccination programs should again be implemented [2]. Several reviews dealing with the worldwide attitudes and hesitancy towards COVID-19 vaccination were published [28][29][30]. In countries with high levels of vaccination, knowledge and level of education proved to have positive associations with attitude and adherence to precautionary measures [23]. On the contrary, in Eastern Europe, a lower overall proportion of vaccine acceptance has been reported [30]. Increasing the vaccination rate is crucial in combating the COVID-19 pandemic, but it requires the prior identification of the underlying causes/specific determinants of hesitancy towards vaccines specific to this region.
Similar to other middle-income countries, Romania faces several barriers and difficulties in re-establishing control measures against the COVID-19 epidemic. The means of controlling the SARS-CoV-2 virus infection spreading were accompanied by economic costs such as unemployment, corporate bankruptcies and a disproportionate impact on lessskilled and less-educated workers. Social costs included increased domestic violence and damaged educational systems [31]. Furthermore, the media attention and uncertainty about how to protect against infection generate contradictory attitudes and behaviors among the population, starting from accepting protective measures and vaccination to denying their effects and even the disease. Along with the conspiracy theories, the imposition of compulsory vaccination has increased social tensions in many countries, with vaccination being associated with the threat to individual human freedoms, such as the right to free choice of health or the right to work. This perception of the threat to freedom has provided an opportunity for various political factions to condemn vaccination, to question the effectiveness of current vaccines amid the tension generated by general protection measures seen as restrictive in order to gain electoral capital. These perspectives emphasize that while COVID-19 vaccines are becoming increasingly available, they are still met with reluctance, and thus, safety measures (e.g., face masks, personal hygiene and social distancing) are still of key importance in protecting personal and public health against COVID-19 [32].
In order to meet the WHO's recommendations and overcome the particular Romanian patterns of public health measures adherence and acceptance, we developed and validated a questionnaire that addresses knowledge, attitudes and practices focusing on preventive measures against COVID-19. It is divided into six categories related to social distancing, mask-wearing, testing, vaccination, use of supplements and the benefit of a green certificate.
The analysis of the completed questionnaire database offers a global picture of the adherence motivations to general preventive measures against COVID-19 in Romania. By knowing the actual reasons behind not reaching the country's vaccination target, specific measures can be taken to increase people's confidence and compliance. For instance, cardiovascular patients within our 194-sized sample did not have higher vaccination rates, although cardiovascular diseases are known risk factors for developing severe forms of COVID-19 [33][34][35][36]. However, the presence of diabetes was significantly associated with a better score in our questionnaire and with a higher proportion of vaccinated people. Thus, specific measures directed toward cardiovascular patients to raise awareness through educational programs could be effective in levering up vaccination rates. Other main determinants of vaccine acceptance and preventive measures compliance, as identified by our study, were related to vaccines' safety and efficiency, the people's trust in the government and medical system, and health literacy. These results show us that there is room for progress in COVID-19 health literacy through intensified education. Gaining people's trust in the government and healthcare system through communication and education should also be a priority. Our results also suggest a need for targeted community awareness interventions for the most vulnerable populations, those with no school education, the elderly and people living in rural areas.
Moreover, we conducted a multivariate analysis that identified several predictors of vaccination adherence: questionnaire's final score, medical profession, medical advice versus mass media consultation, the presence of diabetes mellitus and the fear of the disease.
In addition to proposing a Romanian survey regarding COVID-19 attitudes, we also proceeded to the scientific validation of the questionnaire in order to obtain pertinent information in a reliable and valid way [21] and ensure that the questionnaire "measures what is intended to be measured" [37]. The application of improper measurement tools that are not validated can lead to inaccurate and misleading findings, resulting in a poor plan for interventions and, therefore, too unreliable efficacy.
Validity is assessed through two categories of tests, which evaluate theoretical construct and empirical construct. The theoretical construct is tested through face validity. The purpose of face validity is to ascertain that the items of the questionnaire fully represent the domain that is intended to be judged [37]. We used literature reviews, critical incidents, direct observations and expert judgment approaches to construct a questionnaire of 40 items with acceptable face validity. The literature review did not identify, to date, any similar validated questionnaire in the Romanian language [38]. Face validity is an important aspect related to the empirical construct of the questionnaire [20,21]. The next validation step was represented by the psychometric evaluation, which prompted the elimination of 15 questionnaire items due to not meeting psychometric criteria. The remaining 25 items were included in the exploratory factor analysis. The factor analysis revealed a three factors structure with 20 items. The resulting model explained 52.7% of the variance in our study. Furthermore, PCA was applied, resulting in a final model of 16 items.
In order to strengthen the evidence that the questionnaire "measures what is intended to be measured", we proved that the test is able to discriminate between vaccinated and unvaccinated people.
Our results are similar to other resembling COVID-19 questionnaire validations in other languages. A Korean study was based on similar sample size (229) and obtained an alpha Cronbach coefficient of 0.75 for validating a survey that assesses COVID-19 knowledge, attitudes and practices among nursing students [39]. Likewise, the validation of an Indian questionnaire to assess knowledge, attitude, practices and concerns regarding COVID-19 vaccination among the general population was based on 201 participants and resulted in an alpha Cronbach coefficient of 0.86. With the exception of a french questionnaire aimed to assess COVID-19 knowledge and behaviors that was validated on a big cohort of 8045 participants [9], our study fits the current international scientific landscape for the population sample as well as validity results.

Limitations
Firstly, the number of survey respondents was small. This low interest may be explained by the fact that the incidence of COVID-19 cases declined during the spring of 2022, preventive measures were relaxed, and the mass media and people's attention was focused on the war in Ukraine, with which Romania shares a significant long border. However, this number is still appropriate for factor analysis and testing of the questionnaire. Secondly, our sample might not be representative of the general population in Romania as the proportion of vaccinated people was 74,8%, much higher than that reported by the authorities of 42.3% [3] (as of 14 July 2022). Moreover, a significant proportion were healthcare workers, thus explaining the high rate of vaccination. The percentage of women (70.6%) in our sample is also higher compared to the general population. In our future validation studies, we aim to overcome these limitations by including a larger population with similar variables' distributions as the general population.

Conclusions
In the context of the re-emergence of new variants of SARS-CoV-2 spreading globally, with steady recommendations from the WHO to re-establish some general preventive measures and reinforce vaccination programs, this study aimed to achieve the following: (1) propose the first validated Romanian questionnaire aimed at screening attitudes toward COVID-19 measures; (2) identify the main motivations to accept or reject anti-COVID-19 measures in a Romanian population sample; and (3) pave the way to the integration of our questionnaire as part of the national programs to take the pulse of people's attitudes, raise awareness and identify populational areas that need specific interventions, in order to reach vaccination targets and prevent a full-blown sixth wave of the COVID-19 pandemic in Romania. Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.