Comparing Face-to-Face, Emergency Remote Teaching and Smart Classroom: A Qualitative Exploratory Research Based on Students’ Experience during the COVID-19 Pandemic

Abstract

The COVID-19 pandemic significantly disrupted traditional face-to-face teaching worldwide and forced education institutions to adopt new, online teaching formats to enable students to continue with their studies. This research focuses on students’ perceptions of three teaching different modalities: face-to-face (F2F), Emergency Remote Teaching (ERT) and Smart Classroom (SC), which were implemented in response to the restrictions enforced to combat the spread of COVID-19. A qualitative study based on two user experience techniques, Pocket Bipolar Laddering and Emotional Appraisal, was carried out on a group of second-year ICT engineering university students at La Salle Campus Barcelona. The former technique consists in identifying a maximum of three positive and three negative salient items, while the latter is intended to rate pairs of opposite feelings. In the SC format, saving time on travel to university was considered an advantage of online learning, while disadvantages included less effective instructor–student interaction, distractions when off-campus and teamwork issues. These shortcomings can be addressed by specific online teaching training to develop a more active form of learning and foment student participation. However, both ERT and SC modalities were considered effective solutions to cope with the social and mobility restrictions imposed during the pandemic.

1. Introduction

The outbreak of COVID-19 in December 2019 quickly became a worldwide pandemic in 2020 [1] due to its high transmission and mortality rates. COVID-19 originates from the coronavirus SARS-CoV-2, which causes viral respiratory disease [2]. Hence, in order to cope with the multifaced issues derived from the virus, different measures were adopted by governments and institutions in order to minimize the spread of the disease [3,4,5]. As public health systems became inundated with patients, human interactions were increasingly restricted [4,6,7,8]. In 2020, the direct effects of COVID-19 and all the adopted measures that were taken to try to stop the pandemic had a significant impact on worldwide economies [9,10,11].

In the educational context, institutions implemented different solutions to cope with the effects of the pandemic and to enable students to achieve their learning outcomes. This study presents the salient opinions and feelings of ICT engineering undergraduates who experienced three class modalities at La Salle URL over three consecutive academic semesters: face-to-face (F2F), Emergency Remote Teaching (ERT), and Smart Classroom (SC).

The general aim of this research was to assess three different class modalities according to the undergraduates’ perceptions of the 2019–2020 academic year at La Salle URL. It is expected that the results will shed light on F2F, ERT and SC formats and provide insight from a students’ perspective to enable course coordinators to make further teaching adjustments to improve learning experiences and outcomes.

The article contributes to the literature by analyzing the user experience of a group of ICT engineering students who attended three class modalities due to the outbreak of the COVID-19 pandemic. Given that all the participants selected for this research were students with strong ICT skills, eventual technological barriers should not have interfered with their perception of these three formats.

2. State of the Art

In the educational sector, different measures were taken, including mobility restrictions, to limit or even forbid physical access to educational premises [12,13]. To cope with the impossibility of face-to-face learning, remote learning was considered as a solution to face the pandemic constraints [14,15,16]. In addition, the measures taken to curb the pandemic began to have negative effects on students’ health in terms of anxiety, depression and stress [17,18,19,20,21]; interpersonal interactions [22]; and a decrease in physical activity while increasing sedentary behaviors [23,24].

To deal with the COVID-19 pandemic, different initiatives were deployed by academic institutions with the objective of maintaining learning frameworks as much as possible [25,26,27]. In this line, educational institutions adopted different solutions, such as webcasts, open educational resources, flipped instruction using video clips, group learning or online courses [25], learning management systems, instant message platforms, e-mails, webinars or online videos [26].

In this context, most educational institutions decided to apply Emergency Remote Teaching (ERT) [28,29,30], which involved a temporal shift from face-to-face (F2F) class sessions to remote teaching. This latter modality raises the following points: (1) online education allows competition among universities within a worldwide context [31], as it is considered as a kind of distance education option [32,33,34] that was already offered by various institutions in pre-pandemic times; (2) online learning has some pros, in terms of flexibility in time or location and the fact that it may be self-paced by the student, among others [35,36]; (3) online learning presents some challenges in terms of assessment [37,38,39], engagement [40,41,42] or student participation [43,44]; (4) online learning can be implemented both synchronously or asynchronously, each with its own particular implications [45,46,47]; (5) online teaching implies certain challenges for the academic staff [48,49]; (6) online teaching has its own idiosyncrasies, requiring adaptions in the practice of teaching [50,51,52,53]. On the other hand, online teaching is not conceptually the same as ERT, since the latter is not an initially planned learning strategy [54]. Hence, when shifting to ERT, not all educational institutions started off in the same position which resulted in quite different ERT approaches.

2.1. The Smart Classroom Concept

An SC format is characterized by the introduction of interconnected electronic devices in classrooms, such as cameras, screens, smart-boards or sensors [55,56,57,58]. Therefore, it can be said that the SC is an instrumental tool that enables smart education [58,59], which facilitates personalized learning and the possibility of learning anytime and anywhere [58]. A number of studies have been carried out to evaluate the SC format from different perspectives by means of diverse surveys, as shown in different papers, such as [57,60,61]. In the following subsections, the technological tools deployed at La Salle Campus in response to the outbreak of the pandemic are briefly explained from a users’ perspective, not from a technical viewpoint.

The SC format is a technological implementation that leverages teaching and learning solutions. Measuring outcomes once a specific class format has been implemented is considered relevant as it enables course designers to make fine adjustments when required. The concept of performing research focused on students’ opinions is also considered appropriate in order to create the whole picture. In that line, some researchers have focused on evaluating students’ feedback during the COVID-19 pandemic about a specific topic, such as one measuring the impact of COVID-19 in their lives by means of a questionnaire [62], one about the learning experience of nursing students during the first month of confinement by means of a qualitative study [63], and one about students’ perceptions about the forced transition to remote learning [64].

2.2. Technologies Deployed at La Salle URL to Cope with COVID-19

The ICT engineering programs taught at La Salle URL were initially designed in a F2F format. Once attendance restrictions were imposed by government institutions, some changes were implemented to enable teaching activities to continue. An explanation of these changes is given next, distinguishing two different stages: firstly, an ERT approach, and secondly, a more sophisticated technological solution based on an SC model. Implementing an SC solution in the campus facilities was a strategic decision adopted by the institution at a management level. The initial goal of the SC technological solution was to cope with campus closures and the mobility restrictions derived from the COVID-19 pandemic in order to continue teaching students in a way that was as similar as possible to the F2F format in classrooms and laboratories [65].

2.2.1. ERT: 2019–2020 Academic Year, Second Semester

In February 2020, the second semester of the 2020–2021 academic year started, with mobility limitations that restricted student access to La Salle URL Campus. As La Salle URL was already teaching some online programs, ERT could be implemented, taking advantage of the existing resources with the acquisition of additional new equipment. The required elements to implement the ERT are listed as follows:

• Computer, Internet connection, webcam, microphone and speakers. Instructors and students had to have their own equipment. In some cases, La Salle provided the required equipment on request. Furthermore, additional devices, such as drawing tablets, were supplied when required for teaching purposes.
• The corporative Learning Management System incorporated a module named “Online Teaching and Learning Platform” to allow the online interconnections of users.

2.2.2. Smart Classroom: 2020–2021 Academic Year

In September 2020, most classrooms at La Salle URL were equipped with technological devices to make them SCs. This deployment enabled offering class sessions with students inside the classroom (on-campus) or connected remotely to the classroom in a synchronous way (off-campus). The aim of this SC format [65] was to maintain most of the advantages that F2F classes offered [66], despite the COVID-19 pandemic. The complete system installed in most classrooms and laboratories [67] is described as follows:

• Personal Computer (PC): It can be connected to the SMART Board and to the Internet.
• Wi-Fi in all the campus facilities, available in all the classrooms. It enables all authorized users to connect to the Internet by means of their own devices.
• Sound system: The classroom is equipped with a sound system consisting of microphones and speakers to allow interaction between those in the physical classroom and students at home who need a basic audio system.
• Image system: It consists of an equipment based on two cameras and two TVs. One of the cameras automatically follows the movements of the instructor as s/he moves around the classroom. However, at the instructor’s request, the camera can directly focus on the board. The other camera shows a general view of the classroom. The two TV can be configured to show different views, such as the mosaic of all students or just the image of a single student attending classes in an off-campus format. In addition, off-campus students need a device with a camera and a screen to allow proper interaction.
• SMART Board: It is a computer with a huge touch screen that functions as a board. Instructors use this computer to initiate a virtual session where all the authorized participants can connect in order to follow the class session. The screen may display contents from other computers, or just act as a board. The instructor can allow students to write remotely on the board.
• Software and licenses to allow the creation and connection to virtual sessions to undertake class activities online.
• Remote access to different devices (such as specialized computers) located in laboratories at the campus facilities.

3. Methods

The specific research objectives were: (1) to gain insight into the potential for the SC to create new expectations for students once restrictions on COVID-19 are cleared, which could mean rethinking the old F2F format; (2) to obtain greater knowledge about students’ perceptions when experiencing the ERT and the SC formats; (3) to detect the possible negative effects of the ERT and SC formats on students from an emotional point of view when compared with the F2F format. This research is an exploratory study of the perceptions of second-year undergraduates who have attended classes in three different technological scenarios in the context of undergraduate engineering studies from a user experience approach [68,69]. A specific technique, Bipolar Laddering (BLA) can be used to collect qualitative data about the user experience of respondents, minimizing the bias of their perceptions as it is based on open-ended questions. Furthermore, to complete information about students’ experience, an Emotional Appraisal technique can provide additional data once their experience is completed in a specific class format.

3.1. Participants and Procedure

The surveyed students were enrolled in the course Value Chain and Financial Economics (VC&FE), a compulsory subject that forms part of all the ICT Engineering degrees at La Salle-URL. This research study’s open-ended questions were formulated to find out the general perception and the emotions of the students regarding the different class formats deployed in each of the three different scenarios, regardless of the subject itself.

The research took place once the 2020–2021 first semester classes had finished. The surveyed students were chosen because they had started their engineering classes through the F2F modality in September 2019. Later, during their second semester of 2019–2020 academic year, the outbreak of the pandemic affected most aspects of people’s lives, and therefore, these students shifted to the ERT format. Finally, during the first semester of the 2020–2021 academic year, they were taught through an enhanced F2F modality, developed to allow attending classes remotely and live, via an SC format. Hence, these second-year students were asked to assess their user experience once they had had class sessions in three different modalities: F2F, ERT and SC.

In the 2020–2021 academic year, the total number of enrolled students in the subject VC&FE was 164, and in terms of gender, there were 128 males (78.05%) and 36 females (21.95%). The survey was carried out synchronously and voluntarily on the last day of class with those students who were present in the classroom and remotely connected at home. It was not necessary to perform a User Profile Test to verify possible differences in the universe to select a suitable sample, since all VC&FE students can be considered homogenous in terms of this research: all students had experienced the three class formats one after the other; all undergraduates were studying an ICT engineering degree; all students had experienced on-campus classes for one semester and all had experienced the SC format when they were physically in the classroom at the beginning of the first semester of the 2020–2021 academic year, and all had attended off-campus classes when more restrictive measures were enforced by the authorities. A total of 43 students started the survey but only 39 completed it correctly. In any case, as it is a qualitative study, the number of participants is much higher than the minimum established in the literature [70]. In terms of gender, valid questionnaires were answered by 30 males (76.92%) and 9 females (23.08%). The mean age of respondents was 19.7 years old (SD = 1.22). Most of these 39 students started their undergraduate studies in 2019–2020 (82.05%), while the rest (17.95%) had started their studies the previous year.

Data were collected by means of a voluntary and anonymous open-ended questionnaire structured through the pocket BLA technique as shown in Figure A1 that students uploaded in the Learning Management System of the university. This option was implemented since this is the platform that students usually use to upload their homework. Students were asked to introduce their answers at the end of the last session of VC&FE in January 2021, and it took them a maximum of fifty minutes to complete the task. Students did not include any personal information in the form, and data were treated once a label to each of the respondents had been assigned (U1, U2, U3, …), and the files uploaded by the students were deleted.

3.2. Methodology Applied to Answer the Research Objectives

User experience can be defined as analyzing products or services from the user’s perspective once a product of a service has been used [71]. Additionally, other aspects such as emotions can be considered when analyzing user experience [72]. The techniques chosen in the study were based on the opinions of the students reflecting their previous experience in class and the feelings they have had. A qualitative approach to listen to the students’ opinions was considered a good option to draw initial conclusions, in line with other research works [73]. The students were surveyed using two techniques: Bipolar Laddering and Emotional Appraisal.

3.2.1. Bipolar Laddering

To assess students’ experience, the Bipolar Laddering technique was employed. This instrument is based on a Socratic interview/survey, where the respondents provide their own ideas [74]. Several steps must be taken to perform the BLA [74,75]: (1) users explain all the strong and weak points once the user has completed the experience; (2) users assess (from 0, minimum; to 10, maximum) each one of the identified strong and weak points; (3) users must give their own opinions on how to improve and resolve, according to their thoughts, all the identified points. Then, points are classified and clustered according to the following conditions: (1) positive common elements or positive points cited by at least two users; (2) positive particular elements or positive points cited only by one user; (3) negative common elements or negative points cited by at least two users; (4) negative particular elements or negative items mentioned by one user. The results are shown in different tables displaying the mean associated with each element and the percentage of users who have cited each one of the items.

The Bipolar Laddering technique can be implemented by means of a BLA interview or by using a pocket BLA. This latter option consists of a survey that substitutes the interview carried out when using the BLA interview by an open-ended questionnaire. Both are designed to identify the most salient positive and negative items once the user has experienced a product or a service. The first option is driven by means of an interview, and each interview can take a considerable amount of time, since the interviewer can ask respondents for clarifications. The second option consists of just asking the user to write the most relevant items in a document on completion of the experience. This method provides fewer data per person since respondents are asked to write just a limited number of different elements, but more data when all the answers are aggregated. The latter option is preferred when the goal is to obtain data from many people investing a reasonable amount of time, as was the case of this research. A standard BLA interview activity based on an experience can take around thirty minutes. In the case of this research, it was not just a single experience but three different learning experiences that were to be evaluated, so the pocket BLA technique was selected. Hence, the pocket BLA consists of identifying a maximum of three positive and three negative items, following all the other instructions explained in the previous paragraph. This system has been applied previously in other educational projects [76,77,78].

The open-ended questions posed by means of the pocket BLA technique are shown in Appendix A. The process was explained to the students, and they were then asked to identify three positive and three negative elements of each modality based on their perceptions: F2F, ERT and SC. Students were asked to “Identify, according your perceptions, three positive and three negative elements of each one of three class modalities that you have experienced during the last three semesters”. Figure 1 presents the components to be completed once the student had identified each element, in other words, a written description of the positive and negative elements, from the three different class modalities. The combined data obtained from each element provide a better understanding of students’ perceptions about the item and the feedback from users.

3.2.2. Emotional Appraisal

Positive emotions seem to have a positive effect on human beings [79,80]. In fact, previous research has focused on emotions and their impact on students’ interest, motivation to learn and on their academic achievement [81,82]. Therefore, efforts to analyze the emotions experienced by students in a class format may help to complete the picture about their perceptions.

To capture user’s emotions, Emotional Appraisals was used. This questionnaire, in which the user has to rate each pair of opposite feelings shown in Figure 2 by selecting a percentage that spans between both antonymous pairs of feelings was designed by Schmidt-Atzert [83]. The evaluation of each pair of feelings depends on how close (or far) it is located from each one of the pairs. Assessing all the pairs of feelings on the list enables us to draw a complete map of emotions once the user has experienced a product, service or experience. Hence, it results in the election of a more positive or negative positioning of the pairs of feelings being assessed [77,83,84]. Having surveyed the three different class modalities, an Emotional Appraisal Questionnaire was employed to capture user’s feelings about the three class modalities that students had experienced. Within the same form, with the boxes to be completed about the pocket BLA included below, eleven pairs of opposite feelings were given to be assessed by the students as seen in Figure A1, included in Appendix A. Students received the instruction, “Assess the pair of emotions that you have had once experienced each class modality”. The antonymous moods to be weighted for each class modality are listed in Figure 2.

4. Results and Findings

Results from the pocket BLA questionnaire on each of the semesters are shown. All the mentioned positive and negative items are listed, including scores and mention indexes. The students’ comments that ranked above 20% in terms of mention index are also included. Furthermore, a comparison of students’ emotions per semester is presented from a students’ emotional appraisal of each analyzed period.

4.1. Pocket BLA: 2019–2020 First Semester

As previously highlighted, all surveyed students attended ICT engineering courses in the 2019–2020 academic year. During the first semester, classes were taught in an F2F way. Therefore, instructors and students were physically in the classrooms and the laboratories located in the campus facilities. The comments of the students were reworded to be able to capture their ideas in a coherent way, and later, their opinions were grouped.

Table 1. Positive (P) and negative (N) elements both common (C) and particular (P) of the F2F modality (2019–2020 first semester). In bold, the items with a Mention Index higher than 20%.

Item	Description	Average Score	VARP	Mention Index
19s1PCE1	Instructor–student interaction.	8.85	1.21	15/39
19s1PCE2	More concentration. Fewer distractions.	8.40	1.44	15/39
19s1PCE3	Teamwork. Promotion of relationship and cooperation.	8.20	1.56	10/39
19s1PCE4	Classes are better understood.	8.00	0.33	7/39
19s1PCE5	Accesibility (instructor).	9.33	0.89	4/39
19s1PCE6	Resolution of doubts.	8.75	0.69	4/39
19s1PCE7	More entertaining classes.	8.67	0.89	3/39
19s1PCE8	Greater attention. Involvement of instructors and students.	8.50	0.25	2/39
19s1PCE9	More confidence when communicating with the instructor.	10.00	0.00	2/39
19s1PPE1	Good student/classroom ratio.	8.00	-	1/39
19s1NCE1	Waste of time due to travel.	2.67	1.55	13/39
19s1NCE2	Recorded classes are not available.	3.00	0.40	5/39
19s1NCE3	Possibility of contagion. COVID-19 pandemic.	2.00	2.80	5/39
19s1NCE4	Need to get up very early in the morning.	3.00	0.00	3/39
19s1NCE5	All contacts and activities were in person.	2.67	3.55	3/39
19s1NCE6	Classes: low interaction.	3.50	0.25	2/39
19s1NCE7	Too many people in class.	1.50	2.25	2/39
19s1NCE8	Difficulty to see board content in class if too far.	0.50	0.00	2/39
19s1NPE1	Teamwork: sometimes may be difficult.	4.00	-	1/39
19s1NPE2	Long time spent on university campus.	5.00	-	1/39
19s1NPE3	Students not receiving content of the boards.	5.00	-	1/39
19s1NPE4	Distraction from companions, if they are friends.	5.00	-	1/39
19s1NPE5	Shame when speaking in front of classmates.	2.00	-	1/39
19s1NPE6	Not very flexible. Activites related to time and space.	0.00	-	1/39
19s1NPE7	Few exercises solved.	2.00	-	1/39
19s1NPE8	It can be difficult being concentrated in classrooms.	2.00	-	1/39

shows the positive and negative items for F2F mentioned by the students. For each common element, the mean score is shown with its variance and the mention index. The mention index is presented as the number of students citing each item, compared to the total number of participants. For each particular element, the score of the item is depicted. Each one of the items was labeled according to a code of maximum length of nine positions: the first two ones, referring to the year; the third and the fourth ones, referring to the semester; the fifth one, referring to positive (P) or negative (N); the sixth one, referring to common (C) or particular (P); the seventh one, referring to element (E); and the remaining ones, which enumerate each one of the elements within the same category.

From the data obtained, the next step was to polarize the elements based on two criteria: (1) positive or negative elements; (2) number of citations of each element, being referred to as “common elements” if cited for more than one student or “particular elements” if the element is just cited by one student. In this type of analysis, the positive and negative common elements are the most representative because they are the most cited. Depending on the reference rate and its average obtained value, the most relevant elements were identified.

Students’ open answers as to how to improve the item 19s1PCE1 (Instructor–Student interaction) were: “keep it as it is”, a widely repeated answer; “include more options to contact with the instructor”; or simply no comment. According to analysis of the results of the item 19s1PCE2 (More concentration, Fewer distractions), comments were in the line of: “deploying much more dynamic practices in class”; “giving more dynamic explanations, including more examples”, “more silence in class; students sometimes talk in class”, “less use of personal computers during the class sessions” or “nothing to do; it depends on the student”. Finally, observations about how to improve 19sPCE3 (Teamwork, Promotion of relationship and cooperation) were as follows: “teamworking is easier to perform F2F than contacting online”; “doing more teamwork activities, besides including different students in different teams”; “increasing student-student interactions”; “teamworking in real cases” or “doing some more teamwork activities out of the scheduled class timetable at the university”.

Student’s comments about how to improve item 19s1NCE1 (waste of time due to travel) were “nothing to do”, “allowing the option of doing some classes online” or just including “no comment”.

4.2. Pocket BLA. 2019–2020 Second Semester

During the second semester, surveyed students attended classes on the ERT modality.

Table 2. Positive (P) and negative (N) elements, both common (C) and particular (P), of the ERT modality (2019–2020 second semester). In bold are the items with a Mention Index higher than 20%.

Item	Description	Average Score	VARP	Mention Index
19s2PCE1	No time is wasted on campus trips from home.	7.82	2.14	12/39
19s2PCE2	Emergency Remote Teaching solution.	7.75	0.94	9/39
19s2PCE3	Comfort of being at home.	7.67	2.88	4/19
19s2PCE4	Resolving doubts remotely.	7.00	3.25	4/39
19s2PCE5	Possibility to review the classes.	9.00	1.00	3/39
19s2PCE6	Accessibility. No time or physical location restrictions.	8.00	0.00	3/39
19s2PPE1	No risk arising from COVID-19.	7.00	-	1/39
19s2PPE2	Viewing class contents.	10.00	-	1/39
19s2PPE3	Exams done remotely.	10.00	-	1/39
19s2NCE1	Very easy to be less attentive or distracted.	2.45	1.88	11/39
19s2NCE2	Teaching tools with shortcomings arising from tech.	2.25	5.93	10/39
19s2NCE3	Technical problems. Internet connection…	0.80	0.16	6/39
19s2NCE4	Less interaction and relationship with instructors.	3.80	0.96	6/39
19s2NCE5	More difficulty following the class.	2.00	1.50	5/39
19s2NCE6	Classes: lower quality compared with F2F classes.	2.67	2.88	3/39
19s2NCE7	Teamwork: quite difficult.	4.50	0.25	2/39
19s2NCE8	Less interaction between students.	1.00	1.00	2/39
19s2NCE9	Less dynamic activities.	3.00	1.00	2/39
19s2NCE10	Stressful exams. Little time to make the resolution.	0.00	0.00	2/39
19s2NCE11	Technical difficulties due to fluid communication.	4.50	0.25	2/39
19s2NPE1	Difficulty resolving doubts.	1.00	-	1/39
19s2NPE2	Most work done individually.	5.00	-	1/39
19s2NPE3	Recorded classes are not available.	2.00	-	1/39
19s2NPE4	Difficulties using the technological tools.	5.00	-	1/39

shows all the items about ERT commented by the students, including the mention index and the average score of each one of the items.

Comments about how to improve the item 19s2PCE1 (No time is wasted on campus travel from home) were: “keep it as it is”, a widely repeated answer; or just including no comment. Observations about item 19s2PCE2 (ERT solution) were: “including methodologies with the goal of increasing the quality of classes”; “trying to minimize problems associated with ERT”.

Comments about how to improve the item 19s2NCE1 (Very easy to be less attentive or distracted) were: “doing F2F classes”; “increasing students’ attention by means of solving more exercises”; “it is difficult to solve”; “including more breaks”; “majorly depends on the environment of the student at home”; “more dynamic classes” or just including no comment. Observations about item 19s2NCE2 (Teaching tools with shortcomings arising from tech.) were: “more dynamic classes”; “Smart Classrooms”; “using tablets”; “improving online classes” or just including no comment.

4.3. Pocket BLA. 2020–2021 First Semester

Table 3. Positive (P) and negative (N) elements both common (C) and particular (P) of the SC modality (2020–2021 first semester). In bold, the items with a Mention Index higher than 20%.

Item	Description	Average Score	VARP	Mention Index
20s1PCE1	Versatility: Attending classes on’ or off-campus.	8.94	2.05	18/39
20s1PCE2	Recording class sessions. It allows reviewing contents.	9.18	1.05	12/39
20s1PCE3	Very good teaching method.	8.14	1.55	8/39
20s1PCE4	Blackboard/PC. Very good teaching possibilities.	9.40	0.64	7/39
20s1PCE5	Agile technological system. Better than virtual classes.	8.14	1.55	7/39
20s1PCE6	No time is wasted on campus trips.	8.33	0.88	4/39
20s1PCE7	Classroom cameras: allow a great interaction.	8.75	0.68	4/39
20s1PCE8	It allows good interactivity b/w instructors and students.	8.33	0.22	3/39
20s1NCE1	Students less attentive if they attend classes virtually.	2.71	1.63	9/39
20s1NCE2	In the classroom, students learn more and interact better.	2.25	1.68	8/39
20s1NCE3	Sometimes writings on the board do not look good.	2.00	4.66	5/39
20s1NCE4	Teamwork: quite difficult.	2.67	2.88	3/39
20s1NCE5	Having to take F2F exams.	1.00	0.66	3/39
20s1NCE6	Class session recordings are deleted too soon.	4.00	1.00	3/39
20s1NCE7	Difficulties in interactions between students (class/remote).	3.00	0.00	3/39
20s1NCE8	Students appear on a screen for a long time.	2.00	0.00	2/39
20s1NCE9	Students do not experience the campus environment.	3.00	0.00	2/39
20s1NCE10	Difficulties in understanding contents/subjects.	3.00	0.00	2/39
20s1NPE1	Sending too many communications via e-mail.	4.00	-	1/39
20s1NPE2	If noise in the classroom, remote learning is difficult.	4.00	-	1/39
20s1NPE3	Improves ERT, but Virtual F2F option is worse than F2F.	5.00	-	1/39
20s1NPE4	Virtual F2F conditioned by technology of each student.	2.00	-	1/39
20s1NPE5	Exams: Little time to make the resolution.	0.00	-	1/39
20s1NPE6	Less interaction between students.	2.00	-	1/39
20s1NPE7	Complications connecting specific link of the class group.	4.00	-	1/39
20s1NPE8	Virtual F2F conditions the way of learning of students.	4.00	-	1/39

shows all the items about SC commented by the students. It includes the mention index and the average score of each one of the items.

Comments on how to improve the item 20s1PCE1 (Versatility: Attending classes ‘on-campus’ or ‘off-campus’) were: “students can choose between on-campus and off-campus”, a widely repeated answer; “adapting off classroom and on classroom activities in a better way”; “nothing to improve” or just including no comment. Observations about the item 20s1PCE2 (Recording of class sessions, It allows reviewing contents) were: “maintaining recorded classes available for more time” or “nothing to improve”. Finally, notes about 20s1PCE3 (Very good teaching method) were: “increasing drawings and writings” and the majority of students did not include any additional comments.

All the positive elements were identified by at least two students. Hence, no single positive particular element was obtained from the questionnaire. Comments about how to improve the item 20s1NCE1 (Students are less attentive if they attend classes virtually) were: “attending F2F on-campus classes”; “increase interactions with off-campus students to enhance their engagement”; “difficult to solve; it is an issue that depends of the student”; “increasing interaction” or just including no comment. Observations about the item 20s1NCE2 (At classroom students learn more knowledge and interact better with the instructor) were: “quite difficult to cope with this issue”; “implement a system to solve students’ doubts privately”; “doing more questions to be answered (mandatory)”; “attending classes on-campus instead of off-campus” or “increasing even more interaction options”.

The findings of the qualitative research obtained by means of the Pocket BLA technique in terms of reliability and validity match the criteria presented in previous studies [85,86,87,88]. Reliability was achieved by means of replicating the same methodologies and techniques deployed in other studies to collect students’ experiences in an educational context, e.g., [78,89]. The study is considered valid since data collection was carried out in a consistent way with other research works [78,89], the obtained results are consistent with a study that had analyzed different class formats [90], and a great number of respondents (thirty-nine students) form the base elements to analyze our research. Regarding credibility, it should be noted that the analyzed data were provided directly from students’ opinions and perceptions experienced in the three class formats one after another for the same period of time, being recognizable as true data by the surveyed students. When analyzing transferability, it can be said that the context in which data were extracted was clearly explained, and the sample of surveyed students was homogeneous with the defined universe, besides collecting data from a great number of respondents. Dependability was achieved since the collection process was clearly explained, while data were analyzed separately by three authors of this paper, previously to synthetizing the results. The transcription of all students’ opinions and the comparison of findings with some partial previous results obtained in other studies [90] contribute to the confirmability of the study.

4.4. Students’ Emotional Appraisal in the Three Analyzed Semesters

The results of the Emotional Appraisal were subject to a statistical treatment that was carried out using the MATLAB ^® software, adding [91] to the software package to calculate the Cronbach’s alpha test. To check the consistency of the collected dataset, the Cronbach’s alpha test [92] was performed. The aggregated dataset had a value of α = 0.88, which indicated high consistency since the α was greater than 0.70 [93]. In addition, three additional Cronbach’s alpha tests were performed with the data subsets obtained by eliminating each subset of class modality, obtaining the following results: without F2F, α = 0.91; without ERT, α = 0.82; and finally, without SC, α = 0.87. Thus, all the different subsets of data showed consistency.

Table 4. Data from the Emotional Appraisal: F2F, ERT and SC.

Emotional Pair	Pairs of Emotions	F2F Mean (SD)	ERT Mean (SD)	SC Mean (SC)
EP1	Confidence/Suspicion	0.82 (0.16)	0.51 (0.26)	0.73 (0.27)
EP2	High Quality/Low Quality	0.83 (0.14)	0.49 (0.28)	0.75 (0.23)
EP3	Useful/Useless	0.78 (0.23)	0.59 (0.25)	0.76 (0.26)
EP4	Interesting/Boring	0.69 (0.19)	0.47 (0.25)	0.60 (0.25)
EP5	Known/Unknown	0.79 (0.27)	0.47 (0.32)	0.60 (0.29)
EP6	Comfortable/Uncomfortable	0.68 (0.23)	0.58 (0.32)	0.71 (0.24)
EP7	Attractive/Not Attractive	0.67 (0.20)	0.48 (0.30)	0.70 (0.29)
EP8	Innovative/Conventional	0.37 (0.29)	0.67 (0.23)	0.81 (0.25)
EP9	Simple/Complex	0.67 (0.26)	0.56 (0.24)	0.56 (0.22)
EP10	Nearby/Distant	0.80 (0.24)	0.38 (0.29)	0.56 (0.23)
EP11	Funny/Not Funny	0.59 (0.23)	0.40 (0.24)	0.49 (0.28)

presents the results of students’ emotional appraisal in terms of means and standard deviation (SD) of the different class modalities. Respondents assessed eleven pairs of feelings (see Figure 2), according to the procedure mentioned in Section 3.

An ANOVA test was done, which requires a normal distribution of the data. To verify this assumption, a Kolmogorov–Smirnov test was run on each subset of data. Only five of the thirty-three subsets did not show a normal distribution (EP3, F2F; EP3, ERT; EP3, SC; EP8, SC; EP10, ERT), which were not written in bold in Table 4.

Figure 3 shows the different boxplots of each one of the class modalities grouped by each single pair of emotions, displaying also the mean values in green dots.

Results of the ANOVA test to analyze reliability are presented in

Table 5. Degrees of freedom and value of the F test for each of the eleven ANOVA tests.

Emotional Pair	Pairs of Emotions	F Test
EP1	Confidence/Suspicion	F(2,91) = 12.84, p < 0.0001
EP2	High Quality/Low Quality	F(2,90) = 17.8, p < 0.0001
EP3	Useful/Useless	F(2,88) = 4.99, p = 0.0089
EP4	Interesting/Boring	F(2,91) = 6.26, p = 0.0028
EP5	Known/Unknown	F(2,85) = 7.68, p = 0.0009
EP6	Comfortable/Uncomfortable	F(2,91) = 2.22, p = 0.1143
EP7	Attractive/Not Attractive	F(2,91) = 6.24, p = 0.0029
EP8	Innovative/Conventional	F(2,88) = 21.06, p < 0.0001
EP9	Simple/Complex	F(2,87) = 1.89, p = 0.1570
EP10	Nearby/Distant	F(2,87) = 18.85, p < 0.0001
EP11	Funny/Not Funny	F(2,88) = 3.94, p = 0.0230

, showing the degrees of freedom and the values of the F test. All emotion pairs except EP6 and EP9 show significant differences between some of the means of the three class modalities. As can be seen in Table 5, not all the degrees of freedom related to the participants are the same (they range between 85 and 91), since some of them did not score the complete list of the emotional pair for each modality.

On the other hand,

Table 6. Results from the ANOVA, showing the p-value of each pairwise comparison of modalities.

	EP1	EP2	EP3	EP4	EP5	EP6	EP7	EP8	EP9	EP10	EP11
F2F-ERT	0.000	0.000	0.020	0.002	0.001	0.311	0.021	0.000	0.217	0.000	0.017
F2F-SC	0.291	0.382	0.972	0.322	0.053	0.895	0.931	0.000	0.192	0.002	0.287
ERT-SC	0.001	0.000	0.022	0.074	0.203	0.109	0.004	0.067	0.999	0.011	0.323

presents the p-values obtained (alpha 0.05) from all the pairwise tests once a multivariate analysis was performed to show the pairwise comparison between the different class modalities. Pairs showing a statistically significant difference of means were written in bold in Table 6.

Despite the statistical treatment of the results of the Emotional Appraisal, it should be noted that these results were obtained in the context of an exploratory qualitative analysis. Therefore, this analysis was appropriate in the context of this research, which was designed to collect opinions and feelings of ICT engineering undergraduates once they had experienced three class modalities.

5. Discussion

The results shown in Section 4 were the basis for comparing students’ perceptions of the F2F, ERT and SC teaching and learning modalities. The appraisals of these three experiences were carried out jointly when students had finished the third period of classes. This assessment was based on the user experience as explained in Section 3.

5.1. Students’ Experience

An analysis of the results from the pocket BLA assessment enables us to find out the strengths and weaknesses of each type of class, according to students’ perceptions, as follows:

• Instructor–student interaction was highly valued by students (mean, 8.85; mention index, 38.46%) when attending F2F classes. When experiencing SC classes, some comments are in the same line (mean, 8.33; mention index, 7.69%), whereas others believe that students that are in the classroom interact more than students that are attending off-campus classes (mean, 2.25; mention index, 20.51%). Issues about instructor–student interaction when using the SC had been previously identified in different works, as shown in [57]. Interaction is assessed in ERT classes with a mean score of 3.8 (mention index, 15.38%). Quite surprisingly, students perceive this issue as less problematic in ERT than in SC perhaps because they are living SC as their reality, and ERT is perceived as something that happened and that is not going to happen again since now SC is available. These opinions support the idea that being physically in the classroom allows better interaction with the instructor. Hence, instructors should put their efforts into trying to minimize this deficit when attending off-campus students in the SC format. Specific training for instructors could help to cope with this issue, increasing instructors’ skills to keep interacting with people in the classroom while increasing interaction with off-campus students.
• According to students’ perceptions, they reach higher levels of concentration and are less distracted (mean, 8.40; mention index, 38.46%) in F2F classes. In contrast, students perceive that they are less attentive or more distracted when experiencing both ERT (mean, 2.45; mention index, 28.21%) and SC classes (mean, 2.71; mention index, 23.08%). This item is in the same line as that of other authors, as posited in [57,64]. Therefore, students who remain off-campus should be given the tools to increase their engagement. In addition, instructors could introduce new activities with the objective of specifically engaging off-campus students and making them participate in class.
• Regarding teamwork, most students perceived that F2F classes are the best option (mean 8.20; mention index 25.64%). Just one student considered that it was a difficult task when experiencing the F2F modality (score, 4.00). However, the perception of difficulty in performing this activity was higher when doing ERT classes (mean, 4.5; mention index, 5.13%) or SC classes (mean, 2.67; mention index, 7.69%). This result reinforces the idea that teamwork and Project-Based Learning are key skills for engineering students, since both play a key role once they enter the labor market and are both competencies that should be developed and trained in class [94,95,96].
• When analyzing F2F, students consider that the amount of time needed to commute to the university (mean, 2.67; mention index, 33.33%) as negative. In contrast, commuting is considered a positive item when dealing with ERT classes (mean, 7.82; mention index, 30.77%) or SC classes (mean, 8.33; mention index, 10.26%), since undergraduates can minimize their physical presence at the university. This aspect would lead us to rethink the F2F model and perhaps evolve it towards a blended model, combining on-campus days with off-campus days depending on the kind of teaching activity. On the other hand, minimizing commuting because of the COVID-19 pandemic lockdown may have an impact on student’s sedentary behavior and decrease physical activity. These latter outcomes have harmful effects on health, as shown in different studies [23,24].
• Teaching tools have shortcomings that arise when using ERT from the specific technology adopted (mean, 2.25; mention index, 25.64%).
• ERT is perceived as a very good solution to keep on doing classes while social and mobility restrictions remain in place (mean, 7.75; mention index, 23.08%).
• SC classes are perceived as a versatile solution since they enable students to attend classes in person on-campus or off-campus at the students’ convenience (mean, 8.94; mention index, 46.15%). In fact, this is one of the main advantages of online classes [35,36].
• Recording class sessions in the SC modality is considered a useful option (mean, 9.18; mention index, 30.77%), since it enables students to review classes ubiquitously. In contrast, formerly, when experiencing F2F, this option of recording did not exist, and once students had experienced the option of reviewing recorded classes, they perceived not having this new technological possibility as a negative (mean, 3.0; mention index, 12.82%).

5.2. Students’ Emotional Appraisal

A comparison of students’ emotional responses to the three different ways of learning by means of diverse technologies is shown in Figure 4. Each point of the plot represents the mean of the evaluation obtained for each emotion pair in each class modality.

As can be seen in Table 5, there were statistically significant differences between the modalities for all pairs of emotions except for the pairs Comfortable/Uncomfortable and Simple/Complex. On the other hand, Table 6 highlights all those pairwise combinations that present a significant difference in their means in bold.

Students’ preferences can be assessed through a comparison of the means of each emotional pair associated with each class modality. F2F classes appear as the winning option when trading off the different assessed feelings. F2F classes are perceived as the option that offers a higher degree of positive feelings with respect to the other options in terms of being nearby to students, confidence, interesting, being perceived as known and high quality. However, the SC modality is quite close to the F2F modality in most comparisons, such as usefulness, confidence and quality. In fact, the SC format is superior to the other options in terms of the perception of innovation, comfort and attraction. When evaluating their perceptions about the complexity/simplicity of the three formats, the students rated all of them with a similar score, in line with the initial assumption that ICT students were not conditioned by potential technical issues. Finally, based on analysis of ERT classes compared with the other alternatives, it has the lowest perceptions in terms of positive feelings, except in innovation, where it is quite close to SC classes. The perceived low quality of classes, the low interest that this methodology generates on the student while learning and the distance among people involved in the class sessions are three of the major drawbacks of ERT when compared with the other two options. Hence, F2F classes appear to be the option that generates the most positive assessment of students’ emotions. However, SC classes are not far behind F2F classes in terms of positive feelings.

An analysis of Table 6 reaffirms some of the previous statements, given that comparisons are statistically significant. First, shifting from F2F to ERT had a negative impact on the majority of the students, as evidenced by the results shown in the row labeled F2F-ERT. Second, shifting from ERT to SC improved several aspects, such as confidence, high quality, usefulness, attractiveness and nearby. Third, when comparing F2F to SC, the greatest differences appeared in the perception of innovation (in favor of SC) and closeness (in favor of F2F), as reinforced by the figures displayed in the row labeled F2F-SC.

Although students considered ERT and SC classes to be worse than F2F classes in terms of feelings, students’ comments in the pocket BLA survey emphasized that ERT and SC classes have been a good choice to cope with all mobility restrictions, allowing them to continue learning despite the COVID-19 pandemic.

5.3. Some Additional Observations

The analysis of the results appear to favor the implementation of a blended learning model, mixing on-campus days with off-campus days. This approach could preserve certain advantages of attending classes physically on-campus while reducing some of the issues detected by surveyed students. This solution could be developed to allow off-campus sessions in the case of future mobility restrictions, or when students face a situation in which they cannot attend F2F classes, which could result in a more sustainable model. It should be noted that a change in these characteristics is not a simple or immediate task since it requires the redesign of subjects and degrees. Furthermore, if the adoption of this model is considered, this change should be consistent with the overall strategy of the institution.

Even though the SC model has great potential, students detected certain limitations when they experienced this system. For example, they indicated that the instructor often ended up interacting more with the students in the classroom than those at home, or that it was difficult to follow what was being done in the classroom. Thus, simply emitting class activities is not enough. Activities should be carefully redesigned to facilitate the simultaneous participation and interaction of both on-campus and off-campus students. Henceforth, instructors should be properly trained to take advantage of all the available educational teaching options, besides developing some communication skills.

5.4. Limitations and Directions for Future Studies

The results of this study reflect students’ opinions once classes were concluded, although final exams were still not done. However, the aim of this research is centered only on the comparison of teaching and learning when experiencing F2F, ERT and SC classes, as formulated in the survey. The first limitation of this study is the insufficient knowledge base to evaluate the effect of the deployment of different technological education formats to face the situation derived from the COVID-19 pandemic. Another limitation of this research is that the only targeted students of the survey were second-year students. Third- and fourth-year engineering students were not surveyed because they had experienced F2F classes for a longer period of time. An initial design constraint could become a limitation of this research since surveyed students were skilled enough in ICT technology, so technical difficulties in the use of the different teaching and/or learning platform did not arise. Appraising user experience and emotion for students of other academic fields, not necessarily equally skilled in ICT due to their studies and background, could offer different results, both in terms of user experience and emotion feedback. To cope with this latter restriction research including university students of diverse academic fields could be carried out.

Having identified the relevant topics about the different formats in this exploratory research, a future research line might include a quantitative survey of all the students to assess all the different items to obtain information statistically significant information. Another research line is to identify how the different learning methodologies and assessment activities can be adapted to take advantage of the SC class format when there are students on- and off-campus.

6. Conclusions

The COVID-19 pandemic has affected and changed multiple spheres of our lives, including the domain of education. To cope with mobility restrictions, universities have implemented different solutions to continue teaching their students. This research shows the students’ response to three different class modalities: F2F, ERT and SC classes. Methodologically, user experience and emotions have been selected to measure second-year ICT engineering student’s perceptions about classes. According to this research, students perceive F2F classes as better than the other two options in most facets, except in the amount of time that students spend to arrive at the university. From analyzing the SC format, three elements can be highlighted in terms of issues: instructor–student interaction, greater distractions when off-campus and teamwork difficulties. Therefore, more effective training in the development of the appropriate teaching and learning skills within the SC context could help address these shortcomings. These training activities should be aimed mainly at increasing active participation, either through deliverables or specific activities to promote student interaction. Despite the aforementioned issues, the students’ assessment of SC classes is quite close to F2F classes, which coincides with other studies [90]. In fact, with the current design, the SC model is quite a good solution despite some drawbacks when compared to F2F. ERT and SC modalities were considered an effective solution to the pandemic in order to cope with restrictions of F2F classes. To conclude, SC is emerging as a solution with huge potential provided educational institutions are able to make the necessary changes at different levels to implement it. Similarly, it is gaining ground as a sustainable solution to cope with the current challenges derived from present educational uncertainties caused by the outbreak of COVID-19.