Development and Validation of a Questionnaire to Measure Knowledge of and Attitude toward COVID-19 among Nursing Students in Greece

Background: During the COVID-19 pandemic, nursing students have had a key role in supporting the healthcare sector. They can join healthcare professionals in clinical practice or provide information to increase citizens’ levels of knowledge and their compliance with the restriction measures. The study aimed to develop and validate a tool to measure knowledge of and attitudes toward COVID-19 among nursing students in Greece. Methods: A questionnaire was developed through theoretical research and expert consultation. A cross-sectional study was conducted among 348 undergraduate nursing students of the Department of Nursing, Hellenic Mediterranean University, recruited by convenient sampling. Validity and reliability were analyzed. Results: The Kaiser–Meyer–Olkin measure was 0.84, indicating that the sample size was adequate for factor analysis. In addition, the p-value for Bartlett’s test of sphericity was <0.001, denoting that the correlation matrix was suitable for factor analysis. The construct validity of the questionnaire was determined through exploratory factor analysis (EFA), which revealed that 16 items lead to four factors: knowledge, attitude toward restriction measures, compliance with them, and volunteering. One of the key findings of this study was that participants preferred to receive information from valid sources rather than social media during the crucial period of the “infodemic”. Conclusions: The questionnaire was shown to have satisfying psychometric properties and, therefore, can be used as a tool in future research in the area of nursing students’ knowledge, attitudes, compliance, and volunteering during the COVID-19 pandemic.


Introduction
The COVID-19 pandemic is a social phenomenon; the first cases were identified in China in December 2019 and effected intense health, social, and demographic changes. The World Health Organization (WHO), on 30 December 2019, received a media statement by the Wuhan Municipal Health Commission regarding cases of 'viral pneumonia' in Wuhan, People's Republic of China, while on 30 January 2020, WHO announced a public warning regarding a health emergency of To our knowledge, no fully validated tool exploring nursing students' knowledge of and attitudes toward the COVID-19 public health crisis exists. Due to a shortage of appropriate research tools for answering this research question, we aimed to develop and validate a new instrument for the purpose of this study.

Development of Survey Questionnaire
The study questionnaire, which comprised 24 items, was constructed using reference materials, guidance, and information on COVID-19 developed by WHO, the CDC, and National Health Services (NHS). The survey covered the domains of student demographics, general awareness, information sources, knowledge, and attitude toward COVID-19, as well as level of adherence to the restriction measures. The tool was constructed with input by diverse public health professionals and professors. The first version of the questionnaire was validated by face and content validation methods by five selected experts (two nurses, one physician, and two faculty members). This was done in order to assess its readability and validity before pilot testing among ten randomly selected nursing students to confirm the clarity and acceptability [19]. Finally, all the participants' comments were incorporated and led to a new modified version of the tool with a better understanding and a more suitable order of questions. The final survey link was delivered to students via the educational platform of the institute.

Validation Process-Pilot Study
The questionnaire was evaluated for face validity by an interdisciplinary team of ten senior researchers (experts) in the fields of education and community nursing. Special attention was given to item construction, in order to avoid ambiguity or incomprehensibility [20]. The experts involved in the process critically scrutinized the scales on completeness and the items in terms of possible misunderstandings or ambiguities.
A pilot study with 30 students was conducted in order to estimate the face validity of the questionnaire. Students were asked to participate in the pilot study by completing an online Google Form developed for the purposes of the present study. Special attention was given to item construction to avoid ambiguity or incomprehensibility [20]. The participants carefully checked the scales on completeness and the items in terms of possible misunderstandings or ambiguities. All the items were clear and comprehensive, and finally, only a few syntax corrections were made by the researchers. All returned questionnaires had all items answered and were used for the statistical analysis.

Full Implementation Study
Following the validation process, a cross-sectional study was conducted in Greece during the period of total lockdown due to the pandemic (April-May 2020). Data were collected from April 27 to May 5 by using an online survey, from a convenience sample of undergraduate students who were attending distance classes organized by the university (Hellenic Mediterranean University). Participants were informed about all aspects of the study and voluntarily confirmed their willingness to participate. No personal data were recorded and all questionnaires were completed anonymously. Study approval was obtained by the Hellenic Mediterranean University ethical committee (ethical number 16/27.04.2020).

Statistical Analysis
The construct validity of the questionnaire was estimated with exploratory factor analysis, identifying the underlying factors. The varimax rotation method was used to identify correlations between items and construct the factors. Accordingly, the level for acceptable factor loading was set at >0.4 and for acceptable eigenvalues, set at >1. The Kaiser-Meyer-Olkin test was also used to measure the adequacy of the sample size for factor analysis, with values >0.7 considered as acceptable. Bartlett's test of sphericity was applied to estimate the covariance between the items and values <0.05 indicated that the correlation matrix was suitable for factor analysis. Internal consistency for the factors was measured with the use of the raw coefficient alpha and values >0.7 were considered as acceptable.
For each factor that emerged from factor analysis, a total score was calculated by adding the answers in the factor's items and dividing by the number of items. Thus, a total score from 1 to 5 was created, with higher values indicating greater agreement.
Continuous variables are presented as mean (standard deviation), while categorical variables are presented as numbers (percentages). The Kolmogorov-Smirnov test (p > 0.05) was used to test the normality assumption for the continuous variables. Bivariate analyses between demographic characteristics and total factor scores included a Student's t-test, Spearman's correlation coefficient, and Pearson's correlation coefficient. The Student's t-test was used to compare a continuous variable with a dichotomous one, while Spearman's correlation coefficient was used to correlate a continuous variable with an ordinal one. Furthermore, the correlation between two continuous variables that followed normal distribution was assessed with Pearson's correlation coefficient. Then, multivariable linear regression was performed with total factor scores as the dependent variables. Accordingly, the backward stepwise linear regression was applied and the coefficients' beta, 95% confidence intervals, and p-values were calculated. All tests of statistical significance were two-tailed, and p-values < 0.05 were considered as statistically significant. Statistical analysis was performed with the IBM SPSS 21.0 (IBM Corp. Released 2012. IBM SPSS Statistics for Windows, Version 21.0 Armonk, NY, USA).

Demographic Characteristics
From the 451 students initially approached, 348 of them completed the questionnaire (a response rate of 77.16%) and their demographic characteristics are presented in Table 1. The mean age was 23.6 years, while the majority of students were female (84.8%), single (91.7%), and living with others during pandemic (92%) in Crete (65.5%). 55.2% lived with high-risk groups, 29.9% worked before the pandemic, and 10.9% have been working during the pandemic.

Factor Analysis
The Kaiser-Meyer-Olkin measure was 0.84, indicating that the sample size was adequate for factor analysis. Additionally, the p-value for Bartlett's test of sphericity was <0.001, denoting that the correlation matrix was suitable for factor analysis. The results of the exploratory factor analysis are presented in Table 2. There were four factors, including the 16 items out of the 24 questionnaire items. This four-factor model explained 50% of the questionnaire's variance. According to common sense and the meaning of items, we characterized the factors as the following: (a) COVID-19 knowledge, (b) Attitudes toward restriction measures, (c) Compliance with restriction measures, (d) Volunteering.

Reliability Analysis
The reliability analysis for the questionnaire is presented in Table 3. According to the raw coefficient alpha and the Spearman-Brown coefficient, the questionnaire developed very good reliability. In particular, all raw coefficients alpha values and Spearman-Brown coefficients were >0.70, except one. The raw coefficient alpha for the overall instrument was 0.80 and the Spearman-Brown coefficient was 0.77. The raw coefficient alpha values for the four factors that emerged from the factor analysis ranged from 0.71 to 0.78.

Descriptive Statistics
Descriptive statistics for the 24 questionnaire items and the four factors are presented in Table 4. Mean total scores for the four factors were above the mid-point value (= 3) of the scale, indicating high knowledge levels (mean = 4.22), positive attitudes toward restriction measures (mean = 4.14), high levels of compliance with restriction measures (mean = 4.12), and intentions of students to volunteer in clinical settings (n = 3.34). Regarding other aspects of the questionnaire, students preferred to be more informed by official organizations (e.g., WHO, Centers for Disease Control and Prevention, etc.) (mean = 3.87), public media (mean = 3.35), and electronic databases (e.g., PubMed) (mean = 3.21), compared with social media (e.g., Facebook, Instagram, etc.) (mean = 2.46). Moreover, students believed that quarantine measures could have a moderate effect on their health (mean = 2.93), while the COVID-19 pandemic could change people's lives in a significant way (mean = 3.90).

Bivariate and Multivariable Analysis
Bivariate analyses between demographic characteristics and the factors' scores that emerged from factor analysis are shown in Table 5. There were no significant associations between demographic characteristics and COVID-19 knowledge. Students that lived with others during the pandemic had more positive attitudes toward the restriction measures (p = 0.04), while females showed greater compliance with restriction measures (p = 0.02). In bivariate analyses, age, marital status, maternal educational level, and working before the pandemic were associated with volunteering. However, in the multivariable linear regression analysis, only increased age was associated with increased intention to volunteer (coefficient beta = 0.02, 95% confidence interval = 0.004 to 0.03, p = 0.011).

Discussion
The abrupt spread of the COVID-19 pandemic worldwide and the declaration of the disease as a Public Health Emergency of International Concern were bad omens for public health. Healthcare providers and health science students were always at high risk of infectious diseases and, conclusively, their knowledge of and their attitudes toward the new virus are of major importance for future interventions and health policy planning.
According to our knowledge, this is the first study to develop a tool with these dimensions investigating Greek nursing students' views during the COVID-19 confinement. In this study, which was conducted one month after the announcement of lockdown in Greece, the researchers attempted to investigate COVID-19-related knowledge, the attitude toward this new situation, the level of adherence towards the restriction measures, and volunteerism among nursing students. The developed instrument was comprehensively tested and showed satisfactory psychometrical properties, and it can be used as a valid research tool in future studies in this field. The instrument accounted for 50% of the total variance, indicating that the four factors model was statistically appropriate, including COVID-19 knowledge, attitudes toward restriction measures, compliance with restriction measures, and volunteering. Regarding the dimension of COVID-19 knowledge, we conclude that knowledge is an essential issue, as has been confirmed by several studies [21][22][23][24]. Increased awareness and promotion of positive attitudes among students are imperative to changing students' health practices and improving compliance with preventive measures. Positive attitudes and compliance with restriction measures could significantly improve the public's preventive health behaviors and the preparedness for COVID-19 [21][22][23]25,26]. Finally, a relatively new issue emerged from our factor analysis: the factor "volunteering". Health care students internationally volunteered to assist in hospitals due to the COVID-19 pandemic, providing crucial aid to hospital functioning and patients' care in healthcare systems [27,28].
One of the main results of this study was the students' high knowledge levels (mean = 4.22), as well as their preference to get informed by official organizations and official electronic databases compared with social media. Recent studies show the importance of the need for health science students to be well-informed regarding the symptoms of COVID-19 and prevention strategies [17]. Further, a study in Turkey regarding nursing students revealed that almost half of the respondents (48%) were well-informed about the coronavirus disease, despite the fact that knowledge was associated with an increase in their stress levels [29]. Furthermore, a study based on medical and health students in India found a lack of availability of credible knowledge, with the majority (65.17%) getting informed through social media, and a small percentage of them (11.47%) not informed regarding the pandemic [19]. Notably, there is evidence indicating that health science students have higher levels of knowledge compared with social science students [30], and also have an obligation to adopt true knowledge and disseminate valid evidence regarding the spread of the virus [17].
Literature highlights the need for students' attention to the value of the knowledge refinement process and evidence-based answers through critical appraisal of the information before applying or sharing it [31]. At the same time, authors in cooperation with public health organizations and WHO suggest frameworks, strategic partnerships, and coordinated actions for infodemic management, involving health professionals, students, researchers, and stakeholders [32][33][34]. As a further step, researchers claim that virus-related knowledge and health literacy are needed in order to achieve higher levels of compliance with restriction measures to control citizens' fear, resist the infodemic, and to promote citizens' trust in reliable information and recommendations [15].
Moreover, in this study, most of the participants hold a positive attitude toward restriction measures. Similarly, a national study in the United Kingdom conducted among final year medical students showed that the majority (93.6%) believed that the measures during the pandemic are necessary [35]. In contrast, in the US, 37.8% of the students presented unwillingness to comply with restriction measures, while in Cyprus, researchers showed that women and individuals over the age of 30 were more likely to implement the measures [36,37].
Additionally, the present study reveals the intentions of students to volunteer in clinical settings. This is in line with previous studies' findings in Spain, which found that a high percentage of students expressed the desire to assist nurses in providing patient care during the pandemic [38]. Internationally, nursing and medical students were willing to volunteer for certain tasks in order to assist the healthcare system and support the COVID-19 response effort [39]. Volunteerism in the period of the COVID-19 crisis, although can cause uncertainty and fear for students acting in the frontlines, is still considered a valuable lesson for their future interprofessional practice, and therefore, a high percentage of students report a willingness to get involved [28,35].
Moreover, students reported that the COVID-19 pandemic could change people's lives in a significant way, which is in line with previous findings. Similarly, a study conducted in Hong Kong showed that the vast majority of participants (97%) were worried about COVID-19 and its impact on their daily routines [40], while another study conducted in the United Kingdom reported medical students' disappointment due to the worldwide travel restriction [35].
In the present study, a portion of the participants alleged that the quarantine measures could have a moderate effect on their health (mean = 2.46). Rawls and Gibson stated that the pandemic may cause negative economic, psychological, and cultural consequences [41], as this situation affects students' psychological status and creates anxiety and fears for their future careers [38,42]. In addition, there is evidence linking social isolation with uncertainty, insecurity, and instability to students, as well as with emotional difficulties during student life [36]. Generally speaking, studies have shown that the COVID-19 pandemic and measures to control it have a great impact on individuals' quality of life and mental health, while in a study conducted in the general population of Cyprus, the psychological morbidity was associated with being a university student [42].
The present study has some strengths, as well as limitations. A possible limitation of our study is its cross-sectional study design, which does not allow us to assess possible changes in nursing students' attitudes, their levels of knowledge, or their adherence to restrictions over the different stages of the COVID-19 pandemic. On the other hand, the cross-sectional design allows researchers to use real-time data and clearly depicts the participants' knowledge and attitudes at that point in time, during the confinement. It is worth noting that to our knowledge, this is the first study to develop and validate a tool to assess nursing students' knowledge and attitudes toward COVID-19 in a university environment during the period of quarantine. Another limitation is that we did not perform a confirmatory factor analysis since we did not have a prior theory regarding the number and the structure of the factors. Thus, there is a need for further studies in this field to confirm and expand our findings.

Conclusions
The questionnaire developed proved to have satisfactory psychometric properties in terms of validity and reliability. This instrument can be used as a tool in future research among different student populations in the area of knowledge, attitudes, compliance, and volunteering during the COVID-19 pandemic.