Preferences and Scores of Different Types of Exams during COVID-19 Pandemic in Faculty of Veterinary Medicine in Spain: A Cross-Sectional Study of Paper and E-exams

Abstract

The World Health Organization (WHO) officially declared the novel coronavirus (COVID-19) as a pandemic on 11 March 2020, and educational institutions have had to modify most of their activities (face-to-face activities were suspended). This situation forced academic institutions to modify the evaluation format of students. The use of proctoring systems quickly became widespread, although some controversies arose. The two main discussions regarding these systems are the integrity of the assessment and the capacity of the students to adapt to this new assessment method, without changes in theirs scores. To elucidate two controversies, we have analyzed the preferences and the scores obtained from a trial of 660 scores from 332 students of the third grade of Veterinary Medicine. The experiment involved three modalities of exam: an online format from home using the Respondus Lockdown Browser system (Modality 1), online in person using the Respondus Lockdown Browser system with the supervision of a teacher (Modality 2), or paper format in person with the supervision of a teacher (Modality 3). The results obtained showed that the students preferred Modality 1 (online at home with Respondus Lockdown Browser system). No statistical differences between the scores obtained by students were found between the three modalities analyzed. The proctoring system is a good method to adjudicate exams in higher education institutions, and the scores of students are similar to those obtained through traditional evaluation and control systems.

1. Introduction

The World Health Organization (WHO) officially declared the novel coronavirus (COVID-19) as a pandemic on 11 March 2020 [1]. It has spread rapidly due to the high contagion capacity of the virus and the routes of transmission (mainly by aerosols when coughing and sneezing). In addition, around 30% of patients have a wide range of symptoms that can be life-threatening [2]. Due to the above, higher education institutions have had to modify most of their activities, as face-to-face activities were suspended. Adaptation to this situation has been a challenge around the world, and faculties have had to adjust their teaching and assessment systems [3]. While online teaching activities have been well received by both teachers and students, assessment activities and, more specifically, online exams with proctoring systems have given rise to significant controversies, mainly in science education [4,5,6,7]. Some prevailing concerns among students are related to privacy and various environmental and psychological factors [5]. In fact, Dragan et al. (2020) concluded that while remote supervision proved to be timely solution, the emotional needs of students who may be feeling stressed must be considered [8].

Two of the most significant challenges of real-time online examination monitoring systems are the integrity of the assessment and the ability of students to adapt to this new assessment method. To analyze the dishonesty of students in remote assessments and some ways to avoid them, some studies have been realized. A study carried out by Guangul et al. (2020) concluded that combining various assessments methods (for example, reporting with online submission) helps minimize academic dishonesty [4]. Li et al. (2021) developed an optimization-based anti-collusion approach to remote online testing, minimizing the gain from collusion [9]. Recently, Pettit et al. (2021) have published a review in which, by analyzing the studies carried out so far, they give a series of recommendations, in order to improve the authentication of candidates and prevent cheating [7].

Integrity of evaluation in online exams can be proctored in a variety of ways, including taking the exam in person or using a real-time supervisor system [5]. One of these real-time online supervisor systems is the Respondus Lockdown Browser system, which provides real-time online monitoring services using a microphone and a webcam. Students connect to the exam online through the Respondus program, which guides them through the process and monitors them during the exam. Before the start of the exam, students must show identification to verify their identity, as well as a 360° view of their workspace in order to ensure there are no unauthorized materials. Students are required to maintain a visual and audio connection throughout all the session. The second challenge of online assessment systems is the ability of students to adapt to these new systems, and that the scores obtained are not affected by these new methods.

The main aim of this work was to compare the preferences and the scores obtained for students in the third grade of Veterinary Medicine by exam online through the Respondus Lockdown Browser at home, with the Respondus Lockdown Browser with teacher supervision, and in person paper format with teacher supervision.

2. Materials and Methods

2.1. Population and Sample

This trial was performed in the third grade of Veterinary Medicine (ages between 21 to 23) under the same educational conditions (professors, times, among others). A total of 660 scores were obtained from two different exams performed on different days (separated by two weeks), each with scores ranging from 0 to 10 points. The first exam (332 students scores, 75 males and 257 females) consisted of a total of 36 multiple choice questions (with the same value), where the student had to choose the correct answer (only one correct) from a total of three, with each incorrect option subtracting 0.5 a point from a correct one. The second exam (328 students scores, 74 males and 254 females) consisted of a total of six questions (three mathematical problems and three relations of concepts or incomplete sentences that the student had to complete). The score of each question was variable and specified in the exam. During the entire period the exams (of both modalities) were monitored to detect possible copies among the students.

2.2. Study Design

The students could choose exam modality and vigilance system before the exam. The different options were as follows. Modality 1: Online format at home using the Respondus Lockdown Browser system [10]. Modality 2: Online format presential using the Respondus system and with professor supervision. Modality 3: Paper format presential with professor’s supervisor. To ensure that students performed exams under similar conditions they had the possibility of communication with supervisors. Students from Modality 1 could communicate with supervisors by clicking on a link embedded on the exam. This link redirect students to a videoconference with supervisors. On the other hand, students from Modalities 2 and 3 could communicate directly with supervisors. The Respondus system is a proctoring strategy for computer-based exams through which students must connect to the online exam. This system is based on a control of the environment in which the student takes the exam. In this way, the system prevents the student from accessing web pages other than those of the exam and records the session during the exam, in addition to giving a series of recommendations to the student before starting the exam. The program first indicates the terms of use (Figure 1). Thereafter, a chain of screens is presented where the student is clearly explained the steps to follow and the regulations for taking the exam (Figure 2).

2.3. Statistical Analysis

The statistical analysis was performed using R statistical software (R Core Team (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL http://www.R-project.org (accessed on 15 June 2021)) and Rcmdr package, freely available on CRAN. The 95% confidence intervals for prevalence estimates were calculated using the Wilson score interval method. Categorial factors were compared with Pearson’s Chi-squared test and Fisher exact tests. The Shapiro-Wilk test for normality and Levene’s test for homoscedasticity were used to detect significant difference among group variances. A general lineal model was carried out to analyze the differences between scores, and the variables analyzed were gender of student, modality of exam, and interactions between them. Results were expressed as mean and standard deviation (SD). The statistical significance was set a p-value < 0.01.

3. Results

Regarding preferences, no differences between gender was observed and the tendencies in preferences and scores obtained was similar (

Table 1. Preferences (%) in function of gender between exam modalities chosen by the students.

	Modality 1			Modality 2			Modality 3
	Male	Female	Total	Male	Female	Total	Male	Female	Total
First exam	73.61	72.27	72.56 b	8.33	8.20	8.23 a	18.06	19.53	19.21 a
Second exam	77.33	74.20	74.92 b	1.33	6.35	5.20 a	21.34	19.45	19.88 a
Total	75.51	73.23	73.74 b	4.76	7.28	6.72 a	19.73	19.49	19.54 a

Modality 1: Online format at home using the Respondus Lockdown Browser system. Modality 2: Online format presential using the Respondus system and with professor as supervisor. Modality 3: Paper format presential with professor as supervisor. Means within a row with different superscripts (a or b) were significantly different, p-value < 0.01.

). However, significant differences were observed between the different modalities offered. In the first exam, students preferred (p < 0.01) Modality 1 (72.56%; 53 males and 185 females) over the others where no significative difference were observed (Modality 2: 8.23%; 6 males and 21 females, and Modality 3: 9.91%; 13 males and 50 females). The same results were observed in the second exam, where students preferred (p < 0.01) Modality 1 (74.92%; 58 males and 187 females) over the others where no significative difference were observed (Modality 2: 5.20%; 1 male and 16 females. Modality 3: 19.88%; 16 males and 49 females).

Regarding the scores obtained, no copy attempts were detected in any modality. Although in general terms the students got better scores in the second exam (p < 0.01), no significant differences were found due to gender or modalities (

Table 2. Scores (mean ± standard deviation) in function of gender obtained in the different exams for each choose modalities.

	Modality 1			Modality 2			Modality 3
	Male	Female	Total	Male	Female	Total	Male	Female	Total
First exam	4.57 ± 0.21	4.87 ± 0.11	4.74 ± 0.12	5.60 ± 0.64	5.16 ± 0.34	5.78 ± 0.38	5.71 ± 0.43	5.89 ± 0.22	5.80 ± 0.26
Second exam	7.70 ± 0.23	7.97 ± 0.13	7.83 ± 0.12	8.00 ± 1.74	8.87 ± 0.43	8.43 ± 0.86	7.59 ± 0.43	8.32 ± 0.25	7.91 ± 0.24
Total	6.13 ± 0.16	6.42 ± 0.09	6.28 ± 0.09	6.80 ± 0.90	7.01 ± 0.28	6.91 ± 0.47	6.65 ± 0.31	7.06 ± 0.17	6.85 ± 0.18

Modality 1: Online format at home using the Respondus Lockdown Browser system. Modality 2: Online format presential using the Respondus system and with professor as supervisor. Modality 3: Paper format presential with professor as supervisor. Means withing a row with different letters were significantly different, p-value < 0.01.

).

4. Discussion

The COVID-19 pandemic has forced both teachers and students to modify their academic activities, among which is the evaluation of the knowledge acquired by the student. Some studies indicate that the acceptance of the new methodologies has been correct in different countries [11,12,13]. Results obtained in this study showed that students preferred the online format exam at home using the Respondus Lockdown Browser system (Modality 1), regardless of the type of exam and the gender of the student. This preference for online systems is in disagreement with the observed data in previous reports where students preferred paper exams [14,15,16]. In these studies, the main reasons for students objections (to the online format) were diverse, including the possibility of cheating or even obtaining lower scores due to a low concentration [14,15]. The first reason is easily eliminated by using a proctoring system during the electronic remote examination (as the Respondus Lockdown Browser system was used in our study). Furthermore, other approaches could be useful to minimize academic dishonesty (as to prepare different questions for each student and online submission) [4]. The second reason does not have supporting scientific data, in spite of the fact that electronic remote testing can be difficult to overcome [5], and test anxiety increases in some students with the electronic remote exams [8,16] which could be due to the use of unknown technologies [8]. Unlike previous studies, the students in our study had previously used the Respondus Lockdown Browser system. Moreover, it is interesting to highlight that all the students had the possibility of communication with supervisors independently of the modality. This strategy allowed all students to perform the exam under similar conditions. Otherwise, students in the classroom could have access to supervisors whereas students at home could not. Therefore, we hypothesized that training and the possibility of communication with supervisors could explain an increased preference for this modality.

The scores in our study were similar to the results of other authors [8,15,16,17]. Advances in online exam software and remote monitoring have prevented cheating, as well as issues with cybersecurity and student authentication [3,7]. Furthermore, [18] observed in a study with veterinary students in Germany that they perceived the electronic exam as fairer and less stressful than other types of exams, such as oral ones, and that the preparation time for an oral exam was longer than for an electronic exam.

These results suggest that evaluation through remote electronic examinations with a proctoring system such as Respondus Lockdown Browser could be a good evaluation system for university students and that, once they get used to this system, it could replace traditional evaluation systems. This study was carried out using two different evaluations within the same subject of the Veterinary degree. The kinds of questions and average marks (for each evaluation) were considerably different. Therefore, the obtained results (between groups) were similar in both evaluations. This fact allows a degree of certainty about the different evaluation systems of said subject. For example, Arja et al. (2021) analyzed the online evaluation in medical faculty [18]. This study showed similar results, but the authors conclude by recommending a combination of online assessments and face-to-face modalities, mainly in clinical subjects or degrees. Therefore, the design of the study and its results should be taken with caution when generalizing them for other subjects or degrees. As another limitation, it is necessary to consider that there are other types of evaluations not taken into account in this study. For this reason, it would be interesting to carry out a larger study in the future that includes other subjects from other degrees. Further studies regarding the preferences and scores obtained of students in the Veterinary degree and others are necessary before these evaluation procedures can be used on a regular system. Studies covering a larger number of students, with other evaluation and control systems and other university degrees would be interesting.

5. Conclusions

Unlike results found by other studies, regardless of the exam modality, students preferred the electronic remote examination with a Respondus Lockdown Browser proctoring system. No differences in scores were obtained by students in the different modalities and surveillance system (professor or electronic programs). According to our results, electronical exams are a good alternative to presential paper exams.