Next Article in Journal
Envisioning Climate Change Adaptation Futures Using Storytelling Workshops
Next Article in Special Issue
Analysis of the Dynamical Capabilities into the Public Research Institutes to Their Strategic Decision-Making
Previous Article in Journal
Collaborative Distributed Planning with Asymmetric Information. A Technological Driver for Sustainable Development
Previous Article in Special Issue
Mixed Analysis of the Flipped Classroom in the Concrete and Steel Structures Subject in the Context of COVID-19 Crisis Outbreak. A Pilot Study
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Comparing Face-to-Face, Emergency Remote Teaching and Smart Classroom: A Qualitative Exploratory Research Based on Students’ Experience during the COVID-19 Pandemic

Department of Engineering, Universitat Ramon Llull (URL), 08022 Barcelona, Spain
Department of Management and Technology, Universitat Ramon Llull (URL), 08022 Barcelona, Spain
Department of Architecture, Universitat Ramon Llull (URL), 08022 Barcelona, Spain
Author to whom correspondence should be addressed.
Sustainability 2021, 13(12), 6625;
Received: 9 March 2021 / Revised: 4 June 2021 / Accepted: 7 June 2021 / Published: 10 June 2021
(This article belongs to the Special Issue Information Systems, E-learning and Knowledge Management)


The COVID-19 pandemic significantly disrupted traditional face-to-face teaching worldwide and forced education institutions to adopt new, online teaching formats to enable students to continue with their studies. This research focuses on students’ perceptions of three teaching different modalities: face-to-face (F2F), Emergency Remote Teaching (ERT) and Smart Classroom (SC), which were implemented in response to the restrictions enforced to combat the spread of COVID-19. A qualitative study based on two user experience techniques, Pocket Bipolar Laddering and Emotional Appraisal, was carried out on a group of second-year ICT engineering university students at La Salle Campus Barcelona. The former technique consists in identifying a maximum of three positive and three negative salient items, while the latter is intended to rate pairs of opposite feelings. In the SC format, saving time on travel to university was considered an advantage of online learning, while disadvantages included less effective instructor–student interaction, distractions when off-campus and teamwork issues. These shortcomings can be addressed by specific online teaching training to develop a more active form of learning and foment student participation. However, both ERT and SC modalities were considered effective solutions to cope with the social and mobility restrictions imposed during the pandemic.

1. Introduction

The outbreak of COVID-19 in December 2019 quickly became a worldwide pandemic in 2020 [1] due to its high transmission and mortality rates. COVID-19 originates from the coronavirus SARS-CoV-2, which causes viral respiratory disease [2]. Hence, in order to cope with the multifaced issues derived from the virus, different measures were adopted by governments and institutions in order to minimize the spread of the disease [3,4,5]. As public health systems became inundated with patients, human interactions were increasingly restricted [4,6,7,8]. In 2020, the direct effects of COVID-19 and all the adopted measures that were taken to try to stop the pandemic had a significant impact on worldwide economies [9,10,11].
In the educational context, institutions implemented different solutions to cope with the effects of the pandemic and to enable students to achieve their learning outcomes. This study presents the salient opinions and feelings of ICT engineering undergraduates who experienced three class modalities at La Salle URL over three consecutive academic semesters: face-to-face (F2F), Emergency Remote Teaching (ERT), and Smart Classroom (SC).
The general aim of this research was to assess three different class modalities according to the undergraduates’ perceptions of the 2019–2020 academic year at La Salle URL. It is expected that the results will shed light on F2F, ERT and SC formats and provide insight from a students’ perspective to enable course coordinators to make further teaching adjustments to improve learning experiences and outcomes.
The article contributes to the literature by analyzing the user experience of a group of ICT engineering students who attended three class modalities due to the outbreak of the COVID-19 pandemic. Given that all the participants selected for this research were students with strong ICT skills, eventual technological barriers should not have interfered with their perception of these three formats.

2. State of the Art

In the educational sector, different measures were taken, including mobility restrictions, to limit or even forbid physical access to educational premises [12,13]. To cope with the impossibility of face-to-face learning, remote learning was considered as a solution to face the pandemic constraints [14,15,16]. In addition, the measures taken to curb the pandemic began to have negative effects on students’ health in terms of anxiety, depression and stress [17,18,19,20,21]; interpersonal interactions [22]; and a decrease in physical activity while increasing sedentary behaviors [23,24].
To deal with the COVID-19 pandemic, different initiatives were deployed by academic institutions with the objective of maintaining learning frameworks as much as possible [25,26,27]. In this line, educational institutions adopted different solutions, such as webcasts, open educational resources, flipped instruction using video clips, group learning or online courses [25], learning management systems, instant message platforms, e-mails, webinars or online videos [26].
In this context, most educational institutions decided to apply Emergency Remote Teaching (ERT) [28,29,30], which involved a temporal shift from face-to-face (F2F) class sessions to remote teaching. This latter modality raises the following points: (1) online education allows competition among universities within a worldwide context [31], as it is considered as a kind of distance education option [32,33,34] that was already offered by various institutions in pre-pandemic times; (2) online learning has some pros, in terms of flexibility in time or location and the fact that it may be self-paced by the student, among others [35,36]; (3) online learning presents some challenges in terms of assessment [37,38,39], engagement [40,41,42] or student participation [43,44]; (4) online learning can be implemented both synchronously or asynchronously, each with its own particular implications [45,46,47]; (5) online teaching implies certain challenges for the academic staff [48,49]; (6) online teaching has its own idiosyncrasies, requiring adaptions in the practice of teaching [50,51,52,53]. On the other hand, online teaching is not conceptually the same as ERT, since the latter is not an initially planned learning strategy [54]. Hence, when shifting to ERT, not all educational institutions started off in the same position which resulted in quite different ERT approaches.

2.1. The Smart Classroom Concept

An SC format is characterized by the introduction of interconnected electronic devices in classrooms, such as cameras, screens, smart-boards or sensors [55,56,57,58]. Therefore, it can be said that the SC is an instrumental tool that enables smart education [58,59], which facilitates personalized learning and the possibility of learning anytime and anywhere [58]. A number of studies have been carried out to evaluate the SC format from different perspectives by means of diverse surveys, as shown in different papers, such as [57,60,61]. In the following subsections, the technological tools deployed at La Salle Campus in response to the outbreak of the pandemic are briefly explained from a users’ perspective, not from a technical viewpoint.
The SC format is a technological implementation that leverages teaching and learning solutions. Measuring outcomes once a specific class format has been implemented is considered relevant as it enables course designers to make fine adjustments when required. The concept of performing research focused on students’ opinions is also considered appropriate in order to create the whole picture. In that line, some researchers have focused on evaluating students’ feedback during the COVID-19 pandemic about a specific topic, such as one measuring the impact of COVID-19 in their lives by means of a questionnaire [62], one about the learning experience of nursing students during the first month of confinement by means of a qualitative study [63], and one about students’ perceptions about the forced transition to remote learning [64].

2.2. Technologies Deployed at La Salle URL to Cope with COVID-19

The ICT engineering programs taught at La Salle URL were initially designed in a F2F format. Once attendance restrictions were imposed by government institutions, some changes were implemented to enable teaching activities to continue. An explanation of these changes is given next, distinguishing two different stages: firstly, an ERT approach, and secondly, a more sophisticated technological solution based on an SC model. Implementing an SC solution in the campus facilities was a strategic decision adopted by the institution at a management level. The initial goal of the SC technological solution was to cope with campus closures and the mobility restrictions derived from the COVID-19 pandemic in order to continue teaching students in a way that was as similar as possible to the F2F format in classrooms and laboratories [65].

2.2.1. ERT: 2019–2020 Academic Year, Second Semester

In February 2020, the second semester of the 2020–2021 academic year started, with mobility limitations that restricted student access to La Salle URL Campus. As La Salle URL was already teaching some online programs, ERT could be implemented, taking advantage of the existing resources with the acquisition of additional new equipment. The required elements to implement the ERT are listed as follows:
  • Computer, Internet connection, webcam, microphone and speakers. Instructors and students had to have their own equipment. In some cases, La Salle provided the required equipment on request. Furthermore, additional devices, such as drawing tablets, were supplied when required for teaching purposes.
  • The corporative Learning Management System incorporated a module named “Online Teaching and Learning Platform” to allow the online interconnections of users.

2.2.2. Smart Classroom: 2020–2021 Academic Year

In September 2020, most classrooms at La Salle URL were equipped with technological devices to make them SCs. This deployment enabled offering class sessions with students inside the classroom (on-campus) or connected remotely to the classroom in a synchronous way (off-campus). The aim of this SC format [65] was to maintain most of the advantages that F2F classes offered [66], despite the COVID-19 pandemic. The complete system installed in most classrooms and laboratories [67] is described as follows:
  • Personal Computer (PC): It can be connected to the SMART Board and to the Internet.
  • Wi-Fi in all the campus facilities, available in all the classrooms. It enables all authorized users to connect to the Internet by means of their own devices.
  • Sound system: The classroom is equipped with a sound system consisting of microphones and speakers to allow interaction between those in the physical classroom and students at home who need a basic audio system.
  • Image system: It consists of an equipment based on two cameras and two TVs. One of the cameras automatically follows the movements of the instructor as s/he moves around the classroom. However, at the instructor’s request, the camera can directly focus on the board. The other camera shows a general view of the classroom. The two TV can be configured to show different views, such as the mosaic of all students or just the image of a single student attending classes in an off-campus format. In addition, off-campus students need a device with a camera and a screen to allow proper interaction.
  • SMART Board: It is a computer with a huge touch screen that functions as a board. Instructors use this computer to initiate a virtual session where all the authorized participants can connect in order to follow the class session. The screen may display contents from other computers, or just act as a board. The instructor can allow students to write remotely on the board.
  • Software and licenses to allow the creation and connection to virtual sessions to undertake class activities online.
  • Remote access to different devices (such as specialized computers) located in laboratories at the campus facilities.

3. Methods

The specific research objectives were: (1) to gain insight into the potential for the SC to create new expectations for students once restrictions on COVID-19 are cleared, which could mean rethinking the old F2F format; (2) to obtain greater knowledge about students’ perceptions when experiencing the ERT and the SC formats; (3) to detect the possible negative effects of the ERT and SC formats on students from an emotional point of view when compared with the F2F format. This research is an exploratory study of the perceptions of second-year undergraduates who have attended classes in three different technological scenarios in the context of undergraduate engineering studies from a user experience approach [68,69]. A specific technique, Bipolar Laddering (BLA) can be used to collect qualitative data about the user experience of respondents, minimizing the bias of their perceptions as it is based on open-ended questions. Furthermore, to complete information about students’ experience, an Emotional Appraisal technique can provide additional data once their experience is completed in a specific class format.

3.1. Participants and Procedure

The surveyed students were enrolled in the course Value Chain and Financial Economics (VC&FE), a compulsory subject that forms part of all the ICT Engineering degrees at La Salle-URL. This research study’s open-ended questions were formulated to find out the general perception and the emotions of the students regarding the different class formats deployed in each of the three different scenarios, regardless of the subject itself.
The research took place once the 2020–2021 first semester classes had finished. The surveyed students were chosen because they had started their engineering classes through the F2F modality in September 2019. Later, during their second semester of 2019–2020 academic year, the outbreak of the pandemic affected most aspects of people’s lives, and therefore, these students shifted to the ERT format. Finally, during the first semester of the 2020–2021 academic year, they were taught through an enhanced F2F modality, developed to allow attending classes remotely and live, via an SC format. Hence, these second-year students were asked to assess their user experience once they had had class sessions in three different modalities: F2F, ERT and SC.
In the 2020–2021 academic year, the total number of enrolled students in the subject VC&FE was 164, and in terms of gender, there were 128 males (78.05%) and 36 females (21.95%). The survey was carried out synchronously and voluntarily on the last day of class with those students who were present in the classroom and remotely connected at home. It was not necessary to perform a User Profile Test to verify possible differences in the universe to select a suitable sample, since all VC&FE students can be considered homogenous in terms of this research: all students had experienced the three class formats one after the other; all undergraduates were studying an ICT engineering degree; all students had experienced on-campus classes for one semester and all had experienced the SC format when they were physically in the classroom at the beginning of the first semester of the 2020–2021 academic year, and all had attended off-campus classes when more restrictive measures were enforced by the authorities. A total of 43 students started the survey but only 39 completed it correctly. In any case, as it is a qualitative study, the number of participants is much higher than the minimum established in the literature [70]. In terms of gender, valid questionnaires were answered by 30 males (76.92%) and 9 females (23.08%). The mean age of respondents was 19.7 years old (SD = 1.22). Most of these 39 students started their undergraduate studies in 2019–2020 (82.05%), while the rest (17.95%) had started their studies the previous year.
Data were collected by means of a voluntary and anonymous open-ended questionnaire structured through the pocket BLA technique as shown in Figure A1 that students uploaded in the Learning Management System of the university. This option was implemented since this is the platform that students usually use to upload their homework. Students were asked to introduce their answers at the end of the last session of VC&FE in January 2021, and it took them a maximum of fifty minutes to complete the task. Students did not include any personal information in the form, and data were treated once a label to each of the respondents had been assigned (U1, U2, U3, …), and the files uploaded by the students were deleted.

3.2. Methodology Applied to Answer the Research Objectives

User experience can be defined as analyzing products or services from the user’s perspective once a product of a service has been used [71]. Additionally, other aspects such as emotions can be considered when analyzing user experience [72]. The techniques chosen in the study were based on the opinions of the students reflecting their previous experience in class and the feelings they have had. A qualitative approach to listen to the students’ opinions was considered a good option to draw initial conclusions, in line with other research works [73]. The students were surveyed using two techniques: Bipolar Laddering and Emotional Appraisal.

3.2.1. Bipolar Laddering

To assess students’ experience, the Bipolar Laddering technique was employed. This instrument is based on a Socratic interview/survey, where the respondents provide their own ideas [74]. Several steps must be taken to perform the BLA [74,75]: (1) users explain all the strong and weak points once the user has completed the experience; (2) users assess (from 0, minimum; to 10, maximum) each one of the identified strong and weak points; (3) users must give their own opinions on how to improve and resolve, according to their thoughts, all the identified points. Then, points are classified and clustered according to the following conditions: (1) positive common elements or positive points cited by at least two users; (2) positive particular elements or positive points cited only by one user; (3) negative common elements or negative points cited by at least two users; (4) negative particular elements or negative items mentioned by one user. The results are shown in different tables displaying the mean associated with each element and the percentage of users who have cited each one of the items.
The Bipolar Laddering technique can be implemented by means of a BLA interview or by using a pocket BLA. This latter option consists of a survey that substitutes the interview carried out when using the BLA interview by an open-ended questionnaire. Both are designed to identify the most salient positive and negative items once the user has experienced a product or a service. The first option is driven by means of an interview, and each interview can take a considerable amount of time, since the interviewer can ask respondents for clarifications. The second option consists of just asking the user to write the most relevant items in a document on completion of the experience. This method provides fewer data per person since respondents are asked to write just a limited number of different elements, but more data when all the answers are aggregated. The latter option is preferred when the goal is to obtain data from many people investing a reasonable amount of time, as was the case of this research. A standard BLA interview activity based on an experience can take around thirty minutes. In the case of this research, it was not just a single experience but three different learning experiences that were to be evaluated, so the pocket BLA technique was selected. Hence, the pocket BLA consists of identifying a maximum of three positive and three negative items, following all the other instructions explained in the previous paragraph. This system has been applied previously in other educational projects [76,77,78].
The open-ended questions posed by means of the pocket BLA technique are shown in Appendix A. The process was explained to the students, and they were then asked to identify three positive and three negative elements of each modality based on their perceptions: F2F, ERT and SC. Students were asked to “Identify, according your perceptions, three positive and three negative elements of each one of three class modalities that you have experienced during the last three semesters”. Figure 1 presents the components to be completed once the student had identified each element, in other words, a written description of the positive and negative elements, from the three different class modalities. The combined data obtained from each element provide a better understanding of students’ perceptions about the item and the feedback from users.

3.2.2. Emotional Appraisal

Positive emotions seem to have a positive effect on human beings [79,80]. In fact, previous research has focused on emotions and their impact on students’ interest, motivation to learn and on their academic achievement [81,82]. Therefore, efforts to analyze the emotions experienced by students in a class format may help to complete the picture about their perceptions.
To capture user’s emotions, Emotional Appraisals was used. This questionnaire, in which the user has to rate each pair of opposite feelings shown in Figure 2 by selecting a percentage that spans between both antonymous pairs of feelings was designed by Schmidt-Atzert [83]. The evaluation of each pair of feelings depends on how close (or far) it is located from each one of the pairs. Assessing all the pairs of feelings on the list enables us to draw a complete map of emotions once the user has experienced a product, service or experience. Hence, it results in the election of a more positive or negative positioning of the pairs of feelings being assessed [77,83,84]. Having surveyed the three different class modalities, an Emotional Appraisal Questionnaire was employed to capture user’s feelings about the three class modalities that students had experienced. Within the same form, with the boxes to be completed about the pocket BLA included below, eleven pairs of opposite feelings were given to be assessed by the students as seen in Figure A1, included in Appendix A. Students received the instruction, “Assess the pair of emotions that you have had once experienced each class modality”. The antonymous moods to be weighted for each class modality are listed in Figure 2.

4. Results and Findings

Results from the pocket BLA questionnaire on each of the semesters are shown. All the mentioned positive and negative items are listed, including scores and mention indexes. The students’ comments that ranked above 20% in terms of mention index are also included. Furthermore, a comparison of students’ emotions per semester is presented from a students’ emotional appraisal of each analyzed period.

4.1. Pocket BLA: 2019–2020 First Semester

As previously highlighted, all surveyed students attended ICT engineering courses in the 2019–2020 academic year. During the first semester, classes were taught in an F2F way. Therefore, instructors and students were physically in the classrooms and the laboratories located in the campus facilities. The comments of the students were reworded to be able to capture their ideas in a coherent way, and later, their opinions were grouped.
Table 1 shows the positive and negative items for F2F mentioned by the students. For each common element, the mean score is shown with its variance and the mention index. The mention index is presented as the number of students citing each item, compared to the total number of participants. For each particular element, the score of the item is depicted. Each one of the items was labeled according to a code of maximum length of nine positions: the first two ones, referring to the year; the third and the fourth ones, referring to the semester; the fifth one, referring to positive (P) or negative (N); the sixth one, referring to common (C) or particular (P); the seventh one, referring to element (E); and the remaining ones, which enumerate each one of the elements within the same category.
From the data obtained, the next step was to polarize the elements based on two criteria: (1) positive or negative elements; (2) number of citations of each element, being referred to as “common elements” if cited for more than one student or “particular elements” if the element is just cited by one student. In this type of analysis, the positive and negative common elements are the most representative because they are the most cited. Depending on the reference rate and its average obtained value, the most relevant elements were identified.
Students’ open answers as to how to improve the item 19s1PCE1 (Instructor–Student interaction) were: “keep it as it is”, a widely repeated answer; “include more options to contact with the instructor”; or simply no comment. According to analysis of the results of the item 19s1PCE2 (More concentration, Fewer distractions), comments were in the line of: “deploying much more dynamic practices in class”; “giving more dynamic explanations, including more examples”, “more silence in class; students sometimes talk in class”, “less use of personal computers during the class sessions” or “nothing to do; it depends on the student”. Finally, observations about how to improve 19sPCE3 (Teamwork, Promotion of relationship and cooperation) were as follows: “teamworking is easier to perform F2F than contacting online”; “doing more teamwork activities, besides including different students in different teams”; “increasing student-student interactions”; “teamworking in real cases” or “doing some more teamwork activities out of the scheduled class timetable at the university”.
Student’s comments about how to improve item 19s1NCE1 (waste of time due to travel) were “nothing to do”, “allowing the option of doing some classes online” or just including “no comment”.

4.2. Pocket BLA. 2019–2020 Second Semester

During the second semester, surveyed students attended classes on the ERT modality. Table 2 shows all the items about ERT commented by the students, including the mention index and the average score of each one of the items.
Comments about how to improve the item 19s2PCE1 (No time is wasted on campus travel from home) were: “keep it as it is”, a widely repeated answer; or just including no comment. Observations about item 19s2PCE2 (ERT solution) were: “including methodologies with the goal of increasing the quality of classes”; “trying to minimize problems associated with ERT”.
Comments about how to improve the item 19s2NCE1 (Very easy to be less attentive or distracted) were: “doing F2F classes”; “increasing students’ attention by means of solving more exercises”; “it is difficult to solve”; “including more breaks”; “majorly depends on the environment of the student at home”; “more dynamic classes” or just including no comment. Observations about item 19s2NCE2 (Teaching tools with shortcomings arising from tech.) were: “more dynamic classes”; “Smart Classrooms”; “using tablets”; “improving online classes” or just including no comment.

4.3. Pocket BLA. 2020–2021 First Semester

Table 3 shows all the items about SC commented by the students. It includes the mention index and the average score of each one of the items.
Comments on how to improve the item 20s1PCE1 (Versatility: Attending classes ‘on-campus’ or ‘off-campus’) were: “students can choose between on-campus and off-campus”, a widely repeated answer; “adapting off classroom and on classroom activities in a better way”; “nothing to improve” or just including no comment. Observations about the item 20s1PCE2 (Recording of class sessions, It allows reviewing contents) were: “maintaining recorded classes available for more time” or “nothing to improve”. Finally, notes about 20s1PCE3 (Very good teaching method) were: “increasing drawings and writings” and the majority of students did not include any additional comments.
All the positive elements were identified by at least two students. Hence, no single positive particular element was obtained from the questionnaire. Comments about how to improve the item 20s1NCE1 (Students are less attentive if they attend classes virtually) were: “attending F2F on-campus classes”; “increase interactions with off-campus students to enhance their engagement”; “difficult to solve; it is an issue that depends of the student”; “increasing interaction” or just including no comment. Observations about the item 20s1NCE2 (At classroom students learn more knowledge and interact better with the instructor) were: “quite difficult to cope with this issue”; “implement a system to solve students’ doubts privately”; “doing more questions to be answered (mandatory)”; “attending classes on-campus instead of off-campus” or “increasing even more interaction options”.
The findings of the qualitative research obtained by means of the Pocket BLA technique in terms of reliability and validity match the criteria presented in previous studies [85,86,87,88]. Reliability was achieved by means of replicating the same methodologies and techniques deployed in other studies to collect students’ experiences in an educational context, e.g., [78,89]. The study is considered valid since data collection was carried out in a consistent way with other research works [78,89], the obtained results are consistent with a study that had analyzed different class formats [90], and a great number of respondents (thirty-nine students) form the base elements to analyze our research. Regarding credibility, it should be noted that the analyzed data were provided directly from students’ opinions and perceptions experienced in the three class formats one after another for the same period of time, being recognizable as true data by the surveyed students. When analyzing transferability, it can be said that the context in which data were extracted was clearly explained, and the sample of surveyed students was homogeneous with the defined universe, besides collecting data from a great number of respondents. Dependability was achieved since the collection process was clearly explained, while data were analyzed separately by three authors of this paper, previously to synthetizing the results. The transcription of all students’ opinions and the comparison of findings with some partial previous results obtained in other studies [90] contribute to the confirmability of the study.

4.4. Students’ Emotional Appraisal in the Three Analyzed Semesters

The results of the Emotional Appraisal were subject to a statistical treatment that was carried out using the MATLAB ® software, adding [91] to the software package to calculate the Cronbach’s alpha test. To check the consistency of the collected dataset, the Cronbach’s alpha test [92] was performed. The aggregated dataset had a value of α = 0.88, which indicated high consistency since the α was greater than 0.70 [93]. In addition, three additional Cronbach’s alpha tests were performed with the data subsets obtained by eliminating each subset of class modality, obtaining the following results: without F2F, α = 0.91; without ERT, α = 0.82; and finally, without SC, α = 0.87. Thus, all the different subsets of data showed consistency.
Table 4 presents the results of students’ emotional appraisal in terms of means and standard deviation (SD) of the different class modalities. Respondents assessed eleven pairs of feelings (see Figure 2), according to the procedure mentioned in Section 3.
An ANOVA test was done, which requires a normal distribution of the data. To verify this assumption, a Kolmogorov–Smirnov test was run on each subset of data. Only five of the thirty-three subsets did not show a normal distribution (EP3, F2F; EP3, ERT; EP3, SC; EP8, SC; EP10, ERT), which were not written in bold in Table 4.
Figure 3 shows the different boxplots of each one of the class modalities grouped by each single pair of emotions, displaying also the mean values in green dots.
Results of the ANOVA test to analyze reliability are presented in Table 5, showing the degrees of freedom and the values of the F test. All emotion pairs except EP6 and EP9 show significant differences between some of the means of the three class modalities. As can be seen in Table 5, not all the degrees of freedom related to the participants are the same (they range between 85 and 91), since some of them did not score the complete list of the emotional pair for each modality.
On the other hand, Table 6 presents the p-values obtained (alpha 0.05) from all the pairwise tests once a multivariate analysis was performed to show the pairwise comparison between the different class modalities. Pairs showing a statistically significant difference of means were written in bold in Table 6.
Despite the statistical treatment of the results of the Emotional Appraisal, it should be noted that these results were obtained in the context of an exploratory qualitative analysis. Therefore, this analysis was appropriate in the context of this research, which was designed to collect opinions and feelings of ICT engineering undergraduates once they had experienced three class modalities.

5. Discussion

The results shown in Section 4 were the basis for comparing students’ perceptions of the F2F, ERT and SC teaching and learning modalities. The appraisals of these three experiences were carried out jointly when students had finished the third period of classes. This assessment was based on the user experience as explained in Section 3.

5.1. Students’ Experience

An analysis of the results from the pocket BLA assessment enables us to find out the strengths and weaknesses of each type of class, according to students’ perceptions, as follows:
  • Instructor–student interaction was highly valued by students (mean, 8.85; mention index, 38.46%) when attending F2F classes. When experiencing SC classes, some comments are in the same line (mean, 8.33; mention index, 7.69%), whereas others believe that students that are in the classroom interact more than students that are attending off-campus classes (mean, 2.25; mention index, 20.51%). Issues about instructor–student interaction when using the SC had been previously identified in different works, as shown in [57]. Interaction is assessed in ERT classes with a mean score of 3.8 (mention index, 15.38%). Quite surprisingly, students perceive this issue as less problematic in ERT than in SC perhaps because they are living SC as their reality, and ERT is perceived as something that happened and that is not going to happen again since now SC is available. These opinions support the idea that being physically in the classroom allows better interaction with the instructor. Hence, instructors should put their efforts into trying to minimize this deficit when attending off-campus students in the SC format. Specific training for instructors could help to cope with this issue, increasing instructors’ skills to keep interacting with people in the classroom while increasing interaction with off-campus students.
  • According to students’ perceptions, they reach higher levels of concentration and are less distracted (mean, 8.40; mention index, 38.46%) in F2F classes. In contrast, students perceive that they are less attentive or more distracted when experiencing both ERT (mean, 2.45; mention index, 28.21%) and SC classes (mean, 2.71; mention index, 23.08%). This item is in the same line as that of other authors, as posited in [57,64]. Therefore, students who remain off-campus should be given the tools to increase their engagement. In addition, instructors could introduce new activities with the objective of specifically engaging off-campus students and making them participate in class.
  • Regarding teamwork, most students perceived that F2F classes are the best option (mean 8.20; mention index 25.64%). Just one student considered that it was a difficult task when experiencing the F2F modality (score, 4.00). However, the perception of difficulty in performing this activity was higher when doing ERT classes (mean, 4.5; mention index, 5.13%) or SC classes (mean, 2.67; mention index, 7.69%). This result reinforces the idea that teamwork and Project-Based Learning are key skills for engineering students, since both play a key role once they enter the labor market and are both competencies that should be developed and trained in class [94,95,96].
  • When analyzing F2F, students consider that the amount of time needed to commute to the university (mean, 2.67; mention index, 33.33%) as negative. In contrast, commuting is considered a positive item when dealing with ERT classes (mean, 7.82; mention index, 30.77%) or SC classes (mean, 8.33; mention index, 10.26%), since undergraduates can minimize their physical presence at the university. This aspect would lead us to rethink the F2F model and perhaps evolve it towards a blended model, combining on-campus days with off-campus days depending on the kind of teaching activity. On the other hand, minimizing commuting because of the COVID-19 pandemic lockdown may have an impact on student’s sedentary behavior and decrease physical activity. These latter outcomes have harmful effects on health, as shown in different studies [23,24].
  • Teaching tools have shortcomings that arise when using ERT from the specific technology adopted (mean, 2.25; mention index, 25.64%).
  • ERT is perceived as a very good solution to keep on doing classes while social and mobility restrictions remain in place (mean, 7.75; mention index, 23.08%).
  • SC classes are perceived as a versatile solution since they enable students to attend classes in person on-campus or off-campus at the students’ convenience (mean, 8.94; mention index, 46.15%). In fact, this is one of the main advantages of online classes [35,36].
  • Recording class sessions in the SC modality is considered a useful option (mean, 9.18; mention index, 30.77%), since it enables students to review classes ubiquitously. In contrast, formerly, when experiencing F2F, this option of recording did not exist, and once students had experienced the option of reviewing recorded classes, they perceived not having this new technological possibility as a negative (mean, 3.0; mention index, 12.82%).

5.2. Students’ Emotional Appraisal

A comparison of students’ emotional responses to the three different ways of learning by means of diverse technologies is shown in Figure 4. Each point of the plot represents the mean of the evaluation obtained for each emotion pair in each class modality.
As can be seen in Table 5, there were statistically significant differences between the modalities for all pairs of emotions except for the pairs Comfortable/Uncomfortable and Simple/Complex. On the other hand, Table 6 highlights all those pairwise combinations that present a significant difference in their means in bold.
Students’ preferences can be assessed through a comparison of the means of each emotional pair associated with each class modality. F2F classes appear as the winning option when trading off the different assessed feelings. F2F classes are perceived as the option that offers a higher degree of positive feelings with respect to the other options in terms of being nearby to students, confidence, interesting, being perceived as known and high quality. However, the SC modality is quite close to the F2F modality in most comparisons, such as usefulness, confidence and quality. In fact, the SC format is superior to the other options in terms of the perception of innovation, comfort and attraction. When evaluating their perceptions about the complexity/simplicity of the three formats, the students rated all of them with a similar score, in line with the initial assumption that ICT students were not conditioned by potential technical issues. Finally, based on analysis of ERT classes compared with the other alternatives, it has the lowest perceptions in terms of positive feelings, except in innovation, where it is quite close to SC classes. The perceived low quality of classes, the low interest that this methodology generates on the student while learning and the distance among people involved in the class sessions are three of the major drawbacks of ERT when compared with the other two options. Hence, F2F classes appear to be the option that generates the most positive assessment of students’ emotions. However, SC classes are not far behind F2F classes in terms of positive feelings.
An analysis of Table 6 reaffirms some of the previous statements, given that comparisons are statistically significant. First, shifting from F2F to ERT had a negative impact on the majority of the students, as evidenced by the results shown in the row labeled F2F-ERT. Second, shifting from ERT to SC improved several aspects, such as confidence, high quality, usefulness, attractiveness and nearby. Third, when comparing F2F to SC, the greatest differences appeared in the perception of innovation (in favor of SC) and closeness (in favor of F2F), as reinforced by the figures displayed in the row labeled F2F-SC.
Although students considered ERT and SC classes to be worse than F2F classes in terms of feelings, students’ comments in the pocket BLA survey emphasized that ERT and SC classes have been a good choice to cope with all mobility restrictions, allowing them to continue learning despite the COVID-19 pandemic.

5.3. Some Additional Observations

The analysis of the results appear to favor the implementation of a blended learning model, mixing on-campus days with off-campus days. This approach could preserve certain advantages of attending classes physically on-campus while reducing some of the issues detected by surveyed students. This solution could be developed to allow off-campus sessions in the case of future mobility restrictions, or when students face a situation in which they cannot attend F2F classes, which could result in a more sustainable model. It should be noted that a change in these characteristics is not a simple or immediate task since it requires the redesign of subjects and degrees. Furthermore, if the adoption of this model is considered, this change should be consistent with the overall strategy of the institution.
Even though the SC model has great potential, students detected certain limitations when they experienced this system. For example, they indicated that the instructor often ended up interacting more with the students in the classroom than those at home, or that it was difficult to follow what was being done in the classroom. Thus, simply emitting class activities is not enough. Activities should be carefully redesigned to facilitate the simultaneous participation and interaction of both on-campus and off-campus students. Henceforth, instructors should be properly trained to take advantage of all the available educational teaching options, besides developing some communication skills.

5.4. Limitations and Directions for Future Studies

The results of this study reflect students’ opinions once classes were concluded, although final exams were still not done. However, the aim of this research is centered only on the comparison of teaching and learning when experiencing F2F, ERT and SC classes, as formulated in the survey. The first limitation of this study is the insufficient knowledge base to evaluate the effect of the deployment of different technological education formats to face the situation derived from the COVID-19 pandemic. Another limitation of this research is that the only targeted students of the survey were second-year students. Third- and fourth-year engineering students were not surveyed because they had experienced F2F classes for a longer period of time. An initial design constraint could become a limitation of this research since surveyed students were skilled enough in ICT technology, so technical difficulties in the use of the different teaching and/or learning platform did not arise. Appraising user experience and emotion for students of other academic fields, not necessarily equally skilled in ICT due to their studies and background, could offer different results, both in terms of user experience and emotion feedback. To cope with this latter restriction research including university students of diverse academic fields could be carried out.
Having identified the relevant topics about the different formats in this exploratory research, a future research line might include a quantitative survey of all the students to assess all the different items to obtain information statistically significant information. Another research line is to identify how the different learning methodologies and assessment activities can be adapted to take advantage of the SC class format when there are students on- and off-campus.

6. Conclusions

The COVID-19 pandemic has affected and changed multiple spheres of our lives, including the domain of education. To cope with mobility restrictions, universities have implemented different solutions to continue teaching their students. This research shows the students’ response to three different class modalities: F2F, ERT and SC classes. Methodologically, user experience and emotions have been selected to measure second-year ICT engineering student’s perceptions about classes. According to this research, students perceive F2F classes as better than the other two options in most facets, except in the amount of time that students spend to arrive at the university. From analyzing the SC format, three elements can be highlighted in terms of issues: instructor–student interaction, greater distractions when off-campus and teamwork difficulties. Therefore, more effective training in the development of the appropriate teaching and learning skills within the SC context could help address these shortcomings. These training activities should be aimed mainly at increasing active participation, either through deliverables or specific activities to promote student interaction. Despite the aforementioned issues, the students’ assessment of SC classes is quite close to F2F classes, which coincides with other studies [90]. In fact, with the current design, the SC model is quite a good solution despite some drawbacks when compared to F2F. ERT and SC modalities were considered an effective solution to the pandemic in order to cope with restrictions of F2F classes. To conclude, SC is emerging as a solution with huge potential provided educational institutions are able to make the necessary changes at different levels to implement it. Similarly, it is gaining ground as a sustainable solution to cope with the current challenges derived from present educational uncertainties caused by the outbreak of COVID-19.

Author Contributions

Conceptualization, J.P. and I.I.; methodology, J.P., I.I. and E.V.; validation, I.I., E.V., D.R. and D.F.; formal analysis, J.P., I.I., D.R. and E.V.; investigation, J.P., I.I., E.V. and D.F.; resources, J.P., I.I., E.V. and D.F.; data curation, D.F.; writing—original draft preparation, J.P. and I.I.; writing—review and editing, J.P., I.I., E.V., D.R. and D.F.; visualization, J.P., I.I., E.V., D.R. and D.F.; supervision, I.I., D.R. and D.F.; project administration, I.I., and D.F.; funding acquisition, I.I. and D.F. All authors have read and agreed to the published version of the manuscript.


This research has been funded by the ‘Secretaria d’Universitats i Recerca’ of the Department of Business and Knowledge of ‘Generalitat de Catalunya’ (Grant identification “2017 SGR 934”).

Institutional Review Board Statement

The research presented, as well as the design, collection and management of its data, has been POSITIVE evaluated and APPROVED, by the Ethics Committee of the Ramon Llull University with the file number: CER URL_2020_2021_009.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Data available on request from the authors. Data are not public for privacy reasons.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

This appendix includes the Pocket BLA questionnaire and the Emotional Appraisal that students completed. What is presented is a screenshot of the Excel file with which respondents answered the questionnaire.
Figure A1. Pocket BLA and Emotional Appraisal Questionnaire.
Figure A1. Pocket BLA and Emotional Appraisal Questionnaire.
Sustainability 13 06625 g0a1


  1. Cucinotta, D.; Vanelli, M. WHO declares COVID-19 a pandemic. Acta Biomed. Atenei Parm. 2020, 91, 157. [Google Scholar] [CrossRef]
  2. Bordi, L.; Nicastri, E.; Scorzolini, L.; Di Caro, A.; Capobianchi, M.R.; Castilletti, C.; Lalle, E. Differential diagnosis of illness in patients under investigation for the novel coronavirus (SARS-CoV-2), Italy, February 2020. Eurosurveillance 2020, 25, 2000170. [Google Scholar] [CrossRef] [PubMed][Green Version]
  3. European Centre for Disease Prevention and Control. Considerations Relating to Social Distancing Measures in Response to the COVID-19 Epidemic; European Centre for Disease Prevention and Control: Stockholm, Sweden, 2020. [Google Scholar]
  4. World Health Organization. Overview of Public Health and Social Measures in the Context of COVID-19; World Health Organization: Geneva, Switzerland, 2020. [Google Scholar]
  5. Goniewicz, K.; Khorram-Manesh, A.; Hertelendy, A.J.; Goniewicz, M.; Naylor, K.; Burkle, F.M. Current response and management decisions of the European Union to the COVID-19 outbreak: A review. Sustainability 2020, 12, 3838. [Google Scholar] [CrossRef]
  6. Prem, K.; Liu, Y.; Russell, T.W.; Kucharski, A.J.; Eggo, R.M.; Davies, N.; Flasche, S.; Clifford, S.; Pearson, C.A.B.; Munday, J.D.; et al. The effect of control strategies to reduce social mixing on outcomes of the COVID-19 epidemic in Wuhan, China: A modelling study. Lancet Public Health 2020, 5, 261–270. [Google Scholar] [CrossRef][Green Version]
  7. World Health Organization. Considerations in Adjusting Public Health and Social Measures in the Context of COVID-19; World Health Organization: Geneva, Switzerland, 2020. [Google Scholar]
  8. Lahiri, A.; Jha, S.S.; Bhattacharya, S.; Ray, S.; Chakraborty, A. Effectiveness of preventive measures against COVID-19: A systematic review of In Silico modeling studies in indian context. Indian J. Public Health 2020, 64, 156. [Google Scholar] [CrossRef]
  9. Organisation for Economic Co-operation and Development. Coronavirus: The World Economy at Risk; Organisation for Economic Co-operation and Development, 2020. Available online: (accessed on 10 April 2021).
  10. Baldwin, R.; Weder, B. Economics in the Time of COVID-19; VOX: New York, NY, USA, 2020. [Google Scholar]
  11. Fernandes, N. Economic Effects of Coronavirus Outbreak (COVID-19) on the World Economy. SSRN Electron. J. 2020. [Google Scholar] [CrossRef]
  12. World Health Organization. Considerations for School-Related Public Health Measures in the Context of Annex to Considerations in Adjusting Public Health and Social Measures in the Context of COVID-19; World Health Organization: Geneva, Switzerland, 2020. [Google Scholar]
  13. CEPAL-UNESCO La educación en tiempos de la pandemia de COVID-19. Geopoliticas 2020. Available online: (accessed on 10 April 2021).
  14. United Nations. Education during COVID 19 and beyond. United Nations 2020. Available online: (accessed on 10 April 2021).
  15. UNESCO. COVID-19 Educational Disruption and Response Impact. UNESCO 2020. Available online: (accessed on 10 April 2021).
  16. Hodges, C.; Moore, S.; Lockee, B.; Trust, T. The Difference Between Emergency Remote Teaching and Online Learning. Educ. Rev. 2020, 27, 1–12. [Google Scholar]
  17. Fawaz, M.; Samaha, A. E-learning: Depression, anxiety, and stress symptomatology among Lebanese university students during COVID-19 quarantine. Nurs. Forum. 2021, 56, 52–57. [Google Scholar] [CrossRef]
  18. Cheung, D.K.; Tam, D.K.Y.; Tsang, M.H.; Zhang, D.L.W.; Lit, D.S.W. Depression, anxiety and stress in different subgroups of first-year university students from 4-year cohort data. J. Affect. Disord. 2020, 274, 305–314. [Google Scholar] [CrossRef] [PubMed]
  19. Wang, X.; Hegde, S.; Son, C.; Keller, B.; Smith, A.; Sasangohar, F. Investigating mental health of US college students during the COVID-19 pandemic: Cross-sectional survey study. J. Med. Internet Res. 2020, 22, e22817. [Google Scholar] [CrossRef]
  20. Husky, M.M.; Kovess-Masfety, V.; Swendsen, J.D. Stress and anxiety among university students in France during Covid-19 mandatory confinement. Compr. Psychiatry 2020, 102, 152191. [Google Scholar] [CrossRef]
  21. Odriozola-González, P.; Planchuelo-Gómez, Á.; Irurtia, M.J.; de Luis-García, R. Psychological effects of the COVID-19 outbreak and lockdown among students and workers of a Spanish university. Psychiatry Res. 2020, 290, 113108. [Google Scholar] [CrossRef] [PubMed]
  22. Philippe, R.; Schiavio, A.; Biasutti, M. Adaptation and destabilization of interpersonal relationships in sport and music during the Covid-19 lockdown. Heliyon 2020, 6, e05212. [Google Scholar] [CrossRef] [PubMed]
  23. Karalis, T.; Raikou, N. Teaching at the Times of COVID-19: Inferences and Implications for Higher Education Pedagogy. Int. J. Acad. Res. Bus. Soc. Sci. 2020, 10, 479–493. [Google Scholar] [CrossRef]
  24. Stockwell, S.; Trott, M.; Tully, M.; Shin, J.; Barnett, Y.; Butler, L.; McDermott, D.; Schuch, F.; Smith, L. Changes in physical activity and sedentary behaviours from before to during the COVID-19 pandemic lockdown: A systematic review. BMJ Open Sport Exerc. Med. 2021, 7, e000960. [Google Scholar] [CrossRef]
  25. Huang, R.H.; Liu, D.J.; Tlili, A.; Yang, J.F.; Wang, H.H. Handbook on Facilitating Flexible Learning During Educational Disruption: The Chinese Experience in Maintaining Undisrupted Learning in COVID-19 Outbreak; Smart Learn. Inst. Beijing Norm. Univ. UNESCO: London, UK, 2020. [Google Scholar]
  26. Assunção Flores, M.; Gago, M. Teacher education in times of COVID-19 pandemic in Portugal: national, institutional and pedagogical responses. J. Educ. Teach. 2020, 46, 507–516. [Google Scholar] [CrossRef]
  27. Mishra, L.; Gupta, T.; Shree, A. Online teaching-learning in higher education during lockdown period of COVID-19 pandemic. Int. J. Educ. Res. Open 2020, 1, 100012. [Google Scholar] [CrossRef]
  28. Petillion, R.J.; McNeil, W.S. Student experiences of emergency remote teaching: Impacts of instructor practice on student learning, engagement, and well-being. J. Chem. Educ. 2020, 97, 2486–2493. [Google Scholar] [CrossRef]
  29. Moorhouse, B.L. Adaptations to a face-to-face initial teacher education course ‘forced’ online due to the COVID-19 pandemic. J. Educ. Teach. 2020, 46, 609–611. [Google Scholar] [CrossRef][Green Version]
  30. Whittle, C.; Tiwari, S.; Yan, S.; Williams, J. Emergency remote teaching environment: a conceptual framework for responsive online teaching in crises. Inf. Learn. Sci. 2020, 121, 311–319. [Google Scholar] [CrossRef]
  31. Dykman, C.; Davis, C.K. Part One—The Shift Toward Online Education. J. Inf. Syst. 2008, 19, 11–16. [Google Scholar]
  32. Anderson, T.; Dron, J. Three Generations of Distance Education Pedagogy. Int. Rev. Res. Open Distrib. Learn. 2011, 12, 80. [Google Scholar] [CrossRef][Green Version]
  33. Taylor, J. Fifth Generation Distance Education. Instr. Sci. Technol. 2001, 4, 1–14. [Google Scholar]
  34. Bozkurt, A. From Distance Education to Open and Distance Learning: A Holistic Evaluation of History, Definitions, and Theories. In Handbook of Research on Learning in the Age of Transhumanism; IGI Global: Hershey, PA, USA, 2019; pp. 252–273. [Google Scholar]
  35. Zhang, D.; Zhao, J.L.; Zhou, L.; Nunamaker, J.F. Can e-learning replace classroom learning? Commun. ACM 2004, 47, 75–79. [Google Scholar] [CrossRef]
  36. Arkorful, V.; Abaidoo, N. The role of e-learning, advantages and disadvantages of its adoption in higher education. Int. J. Instr. Technol. Distance Learn. 2015, 12, 29–42. [Google Scholar]
  37. Gaytan, J.; McEwen, B.C. Effective online instructional and assessment strategies. Am. J. Distance Educ. 2007, 21, 117–132. [Google Scholar] [CrossRef]
  38. Bender, T. Discussion-Based Online Teaching to Enhance Student Learning: Theory, Practice and Assessment; Stylus Publishing: Sterling, Virginia, 2012. [Google Scholar]
  39. Gikandi, J.W.; Morrow, D.; Davis, N.E. Online formative assessment in higher education: A review of the literature. Comput. Educ. 2011, 57, 2333–2351. [Google Scholar] [CrossRef]
  40. Dixson, M.D. Creating effective student engagement in online courses: What do students find engaging? J. Scholarsh. Teach. Learn. 2010, 10, 1–13. [Google Scholar]
  41. Bolliger, D.U.; Martin, F. Instructor and student perceptions of online student engagement strategies. Distance Educ. 2018, 39, 568–583. [Google Scholar] [CrossRef]
  42. Dumford, A.D.; Miller, A.L. Online learning in higher education: exploring advantages and disadvantages for engagement. J. Comput. High. Educ. 2018, 30, 452–465. [Google Scholar] [CrossRef]
  43. Hrastinski, S. A theory of online learning as online participation. Comput. Educ. 2009, 52, 78–82. [Google Scholar] [CrossRef]
  44. Vonderwell, S.; Zachariah, S. Factors that influence participation in online learning. J. Res. Technol. Educ. 2005, 38, 213–230. [Google Scholar] [CrossRef]
  45. Murphy, E.; Rodríguez-Manzanares, M.A.; Barbour, M. Asynchronous and synchronous online teaching: Perspectives of Canadian high school distance education teachers. Br. J. Educ. Technol. 2011, 42, 583–591. [Google Scholar] [CrossRef]
  46. Yamagata-Lynch, L.C. Blending online asynchronous and synchronous learning. Int. Rev. Res. Open Distance Learn. 2014, 15, 189–212. [Google Scholar] [CrossRef][Green Version]
  47. Hrastinski, S. Asynchronous & Synchronous E-Learning. Educ. Q. 2008, 31, 51–55. [Google Scholar]
  48. König, J.; Jäger-Biela, D.J.; Glutsch, N. Adapting to online teaching during COVID-19 school closure: teacher education and teacher competence effects among early career teachers in Germany. Eur. J. Teach. Educ. 2020, 43, 608–622. [Google Scholar] [CrossRef]
  49. Carrillo, C.; Flores, M.A. COVID-19 and teacher education: a literature review of online teaching and learning practices. Eur. J. Teach. Educ. 2020, 43, 466–487. [Google Scholar] [CrossRef]
  50. Stern, J. Introduction to Online Teaching and Learning. Int. J. Sci. Educ. 2018, 3, 1–10. [Google Scholar] [CrossRef]
  51. Gewin, V. Five tips for moving teaching online as COVID-19 takes hold. Nature 2020, 580, 295–297. [Google Scholar] [CrossRef][Green Version]
  52. Ko, S.; Rossen, S. Teaching Online: A Practical Guide; Taylor & Francis Group: Bowral, Australia, 2017. [Google Scholar]
  53. Bailey, C.J.; Card, K.A. Effective pedagogical practices for online teaching: Perception of experienced instructors. Internet High. Educ. 2009, 12, 152–155. [Google Scholar] [CrossRef]
  54. Jandrić, P.; Hayes, D.; Truelove, I.; Levinson, P.; Mayo, P.; Ryberg, T.; Monzó, L.D.; Allen, Q.; Stewart, P.A.; Carr, P.R.; et al. Teaching in the Age of Covid-19. Postdigital Sci. Educ. 2020, 2, 1069–1230. [Google Scholar] [CrossRef]
  55. Kwet, M.; Prinsloo, P. The ‘smart’ classroom: a new frontier in the age of the smart university. Teach. High. Educ. 2020, 25, 510–526. [Google Scholar] [CrossRef]
  56. Hwang, G.-J. Definition, framework and research issues of smart learning environments—A context-aware ubiquitous learning perspective. Smart Learn. Environ. 2014, 1, 1–14. [Google Scholar] [CrossRef]
  57. Saini, M.K.; Goel, N. How smart are smart classrooms? A review of smart classroom technologies. ACM Comput. Surv. 2019, 52, 1–28. [Google Scholar] [CrossRef][Green Version]
  58. Zhu, Z.T.; Yu, M.H.; Riezebos, P. A research framework of smart education. Smart Learn. Environ. 2016, 3, 1–17. [Google Scholar] [CrossRef][Green Version]
  59. Uskov, V.L.; Howlett, R.; Jain, L.; Vlacic, L. Smart Education and e-Learning 2017; Springer: Berlin, Germany, 2018. [Google Scholar]
  60. Yang, J.; Pan, H.; Zhou, W.; Huang, R. Evaluation of smart classroom from the perspective of infusing technology into pedagogy. Smart Learn. Environ. 2018, 5, 1–11. [Google Scholar] [CrossRef]
  61. MacLeod, J.; Yang, H.H.; Zhu, S.; Li, Y. Understanding students’ preferences toward the smart classroom learning environment: Development and validation of an instrument. Comput. Educ. 2018, 122, 80–91. [Google Scholar] [CrossRef]
  62. Branquinho, C.; Kelly, C.; Arevalo, L.C.; Santos, A.; Gaspar de Matos, M. “Hey, we also have something to say”: A qualitative study of Portuguese adolescents’ and young people’s experiences under COVID-19. J. Community Psychol. 2020, 48, 2740–2752. [Google Scholar] [CrossRef]
  63. Ramos-Morcillo, A.J.; Leal-Costa, C.; Moral-García, J.E.; Ruzafa-Martínez, M. Experiences of nursing students during the abrupt change from face-to-face to e-learning education during the first month of confinement due to COVID-19 in Spain. Int. J. Environ. Res. Public Health 2020, 17, 5519. [Google Scholar] [CrossRef] [PubMed]
  64. Parker, S.W.; Hansen, M.A.; Bernadowski, C. COVID-19 Campus Closures in the United States: American Student Perceptions of Forced Transition to Remote Learning. Soc. Sci. 2021, 10, 62. [Google Scholar] [CrossRef]
  65. ZOOM ‘Ways of Learning are Changing’: La Salle Campus Barcelona Gives Students Flexibility to Learn From Anywhere Using Zoom. 2021. Available online: (accessed on 10 May 2021).
  66. La Salle Universitat Ramon Llull (URL). La Salle URL Smart Learning. Available online: (accessed on 12 December 2020).
  67. La Salle Universitat Ramon Llull (URL). La Salle URL Smart Classroom. Available online: (accessed on 12 December 2020).
  68. Norman, D. El Diseño Emocional; Paidós: Barcelona, Spain, 2005. [Google Scholar]
  69. Norman, D. The Design of Everyday Things: Revised and Expanded Edition; Basic Books: New York, NY, USA, 2013. [Google Scholar]
  70. Nielsen, J. Why You Only Need to Test with 5 Users. Nielsens, Jakob. 2000. Available online: (accessed on 10 February 2021).
  71. ISO 9241-210:2019. Ergonomics of Human-System Interaction—Part 210: Human-Centered Design for Interactive Systems; International Organization for Standardization: Geneva, Switzerland, 2019; Available online: (accessed on 8 March 2021).
  72. Zarour, M.I. User Experience Aspects and Dimensions: Systematic Literature Review. Int. J. Knowl. Eng. 2017, 3, 52–59. [Google Scholar] [CrossRef]
  73. Biasutti, M. The student experience of a collaborative e-learning university module. Comput. Educ. 2011, 57, 1865–1875. [Google Scholar] [CrossRef]
  74. Pifarré, M.; Tomico, O. Bipolar laddering (BLA): A participatory subjective exploration method on user experience. In Proceedings of the 2007 Conference on Designing for User eXperiences, DUX’07, Chicago, IL, USA, 5–7 November 2007; pp. 2–13. [Google Scholar]
  75. Pifarré, M.; Sorribas, X.; Villegas, E. BLA (Bipolar Laddering) Applied to YouTube. Performing Postmodern Psychology Paradigms in User Experience Field. In Advanced Technologies; IntechOpen: London, UK, 2009. [Google Scholar] [CrossRef][Green Version]
  76. Fonseca, D.; Valls, F.; Redondo, E.; Villagrasa, S. Informal interactions in 3D education: Citizenship participation and assessment of virtual urban proposals. Comput. Human Behav. 2016, 55, 504–518. [Google Scholar] [CrossRef][Green Version]
  77. Labrador, E.; Villegas, E. Unir Gamificación y Experiencia de Usuario para mejorar la experiencia docente. RIED. Rev. Iberoam. Educ. A Distancia 2016, 19, 125–142. [Google Scholar] [CrossRef][Green Version]
  78. Labrador, E.; Villegas, E. Sistema Fun Experience Design (FED) aplicado en el aula. ReVisión 2014, 7. Available online: (accessed on 6 March 2021).
  79. Fredrickson, B.L. The broaden-and-build theory of positive emotions. Philos. Trans. R. Soc. B Biol. Sci. 2004, 359, 1367–1377. [Google Scholar] [CrossRef]
  80. Tugade, M.M.; Fredrickson, B.L. Resilient Individuals Use Positive Emotions to Bounce Back From Negative Emotional Experiences. J. Pers. Soc. Psychol. 2004, 86, 320. [Google Scholar] [CrossRef][Green Version]
  81. Pekrun, R. The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educ. Psychol. Rev. 2006, 18, 315–341. [Google Scholar] [CrossRef]
  82. Pekrun, R.; Frenzel, A.C.; Goetz, T.; Perry, R.P. The Control-Value Theory of Achievement Emotions. An Integrative Approach to Emotions in Education. In Emotion in Education; Elsevier BV: Amsterdam, The Nederlands, 2007; pp. 13–36. [Google Scholar]
  83. Schmidt-Atzert, L. Psicología de las Emociones; Herder: Barcelona, Spain, 1985. [Google Scholar]
  84. Villegas, E.; Labrador, E.; Fonseca, D.; Fernández-Guinea, S. Methodology I’M IN applied to workshop: successful educational practice for consultants in user experience with gamification fields. Univ. Access Inf. Soc. 2019, 18, 507–521. [Google Scholar] [CrossRef]
  85. Morse, J.M.; Barrett, M.; Mayan, M.; Olson, K.; Spiers, J. Verification Strategies for Establishing Reliability and Validity in Qualitative Research. Int. J. Qual. Methods 2002, 1, 13–22. [Google Scholar] [CrossRef]
  86. Leung, L. Validity, reliability, and generalizability in qualitative research. J. Fam. Med. Prim. Care 2015, 4, 324. [Google Scholar] [CrossRef]
  87. Cypress, B.S. Rigor or reliability and validity in qualitative research: Perspectives, strategies, reconceptualization, and recommendations. Dimens. Crit. Care Nurs. 2017, 36, 253–263. [Google Scholar] [CrossRef]
  88. Bengtsson, M. How to plan and perform a qualitative study using content analysis. NursingPlus Open 2016, 2, 8–14. [Google Scholar] [CrossRef][Green Version]
  89. Fonseca, D.; Navarro, I.; de Renteria, I.; Moreira, F.; Ferrer, Á.; de Reina, O. Assessment of Wearable Virtual Reality Technology for Visiting World Heritage Buildings: An Educational Approach. J. Educ. Comput. Res. 2018, 56, 940–973. [Google Scholar] [CrossRef]
  90. He, L.; Yang, N.; Xu, L.; Ping, F.; Li, W.; Sun, Q.; Li, Y.; Zhu, H.; Zhang, H. Synchronous distance education vs traditional education for health science students: A systematic review and meta-analysis. Med. Educ. 2021, 55, 293–308. [Google Scholar] [CrossRef]
  91. Schlegel, A. Cronbach’s Alpha. Available online: (accessed on 10 May 2021).
  92. Cronbach, L.J. Coefficient Alpha and the Internal Structure of Tests. Psychometrika 1951, 16, 297–334. [Google Scholar] [CrossRef][Green Version]
  93. Taber, K.S. The Use of Cronbach’s Alpha When Developing and Reporting Research Instruments in Science Education. Res. Sci. Educ. 2018, 48, 1273–1296. [Google Scholar] [CrossRef]
  94. Petchamé, J.; Iriondo, I.; Riu, D.; Masi, T.; Almazano, A.; Fonseca, D. Project Based Learning or the Rethinking of an Engineering Subject: Measuring Motivation. In Proceedings of the TEEM’20: 8th International Conference on Technological Ecosystems for Enhancing Multiculturality, Salamanca, Spain, 21–23 October 2020; pp. 267–272. [Google Scholar]
  95. Mills, H.; Treagust, D. Engineering Education. Is problem-based or project-based learning the answer? Australas. J. Eng. Educ. 2003, 3, 2–16. [Google Scholar]
  96. Kolmos, A.; De Graaff, E. Problem-Based and Project-Based Learning in Engineering Education: Merging models. In Cambridge Handbook of Engineering Education Research; Cambridge University Press: Cambridge, UK, 2014; pp. 141–161. [Google Scholar]
Figure 1. Pocket BLA: Template to be filled out by the participants for each class modality. This figure shows the particular case of the F2F class format.
Figure 1. Pocket BLA: Template to be filled out by the participants for each class modality. This figure shows the particular case of the F2F class format.
Sustainability 13 06625 g001
Figure 2. Template to be filled out by the participants for each class modality. It includes all the pairs of opposite feelings of the Emotional Appraisal Questionnaire.
Figure 2. Template to be filled out by the participants for each class modality. It includes all the pairs of opposite feelings of the Emotional Appraisal Questionnaire.
Sustainability 13 06625 g002
Figure 3. Boxplots: Comparing Emotional Pairs one-to-one in F2F, ERT and SC Class Modalities. The means of each group are marked with green dots. The different subfigures show: (a) results from pairs 1 to 4; (b) results from pairs 5 to 8; (c) results from pairs 9 to 11.
Figure 3. Boxplots: Comparing Emotional Pairs one-to-one in F2F, ERT and SC Class Modalities. The means of each group are marked with green dots. The different subfigures show: (a) results from pairs 1 to 4; (b) results from pairs 5 to 8; (c) results from pairs 9 to 11.
Sustainability 13 06625 g003aSustainability 13 06625 g003b
Figure 4. Students’ Emotional Appraisal, comparing F2F, ERT and SC.
Figure 4. Students’ Emotional Appraisal, comparing F2F, ERT and SC.
Sustainability 13 06625 g004
Table 1. Positive (P) and negative (N) elements both common (C) and particular (P) of the F2F modality (2019–2020 first semester). In bold, the items with a Mention Index higher than 20%.
Table 1. Positive (P) and negative (N) elements both common (C) and particular (P) of the F2F modality (2019–2020 first semester). In bold, the items with a Mention Index higher than 20%.
ItemDescriptionAverage ScoreVARPMention Index
19s1PCE1Instructor–student interaction.8.851.2115/39
19s1PCE2More concentration. Fewer distractions.8.401.4415/39
19s1PCE3Teamwork. Promotion of relationship and cooperation.8.201.5610/39
19s1PCE4Classes are better understood.8.000.337/39
19s1PCE5Accesibility (instructor).9.330.894/39
19s1PCE6Resolution of doubts.8.750.694/39
19s1PCE7More entertaining classes.8.670.893/39
19s1PCE8Greater attention. Involvement of instructors and students.8.500.252/39
19s1PCE9More confidence when communicating with the instructor.10.000.002/39
19s1PPE1Good student/classroom ratio.8.00-1/39
19s1NCE1Waste of time due to travel.2.671.5513/39
19s1NCE2Recorded classes are not available.3.000.405/39
19s1NCE3Possibility of contagion. COVID-19 pandemic.2.002.805/39
19s1NCE4Need to get up very early in the morning.3.000.003/39
19s1NCE5All contacts and activities were in person.2.673.553/39
19s1NCE6Classes: low interaction.3.500.252/39
19s1NCE7Too many people in class.1.502.252/39
19s1NCE8Difficulty to see board content in class if too far.0.500.002/39
19s1NPE1Teamwork: sometimes may be difficult.4.00-1/39
19s1NPE2Long time spent on university campus.5.00-1/39
19s1NPE3Students not receiving content of the boards.5.00-1/39
19s1NPE4Distraction from companions, if they are friends.5.00-1/39
19s1NPE5Shame when speaking in front of classmates.2.00-1/39
19s1NPE6Not very flexible. Activites related to time and space.0.00-1/39
19s1NPE7Few exercises solved.2.00-1/39
19s1NPE8It can be difficult being concentrated in classrooms.2.00-1/39
Table 2. Positive (P) and negative (N) elements, both common (C) and particular (P), of the ERT modality (2019–2020 second semester). In bold are the items with a Mention Index higher than 20%.
Table 2. Positive (P) and negative (N) elements, both common (C) and particular (P), of the ERT modality (2019–2020 second semester). In bold are the items with a Mention Index higher than 20%.
ItemDescriptionAverage ScoreVARPMention Index
19s2PCE1No time is wasted on campus trips from home.7.822.1412/39
19s2PCE2Emergency Remote Teaching solution.7.750.949/39
19s2PCE3Comfort of being at home.7.672.884/19
19s2PCE4Resolving doubts remotely.7.003.254/39
19s2PCE5Possibility to review the classes.9.001.003/39
19s2PCE6Accessibility. No time or physical location restrictions.8.000.003/39
19s2PPE1No risk arising from COVID-19.7.00-1/39
19s2PPE2Viewing class contents.10.00-1/39
19s2PPE3Exams done remotely.10.00-1/39
19s2NCE1Very easy to be less attentive or distracted.2.451.8811/39
19s2NCE2Teaching tools with shortcomings arising from tech.2.255.9310/39
19s2NCE3Technical problems. Internet connection…0.800.166/39
19s2NCE4Less interaction and relationship with instructors.3.800.966/39
19s2NCE5More difficulty following the class.2.001.505/39
19s2NCE6Classes: lower quality compared with F2F classes.2.672.883/39
19s2NCE7Teamwork: quite difficult.4.500.252/39
19s2NCE8Less interaction between students.1.001.002/39
19s2NCE9Less dynamic activities.3.001.002/39
19s2NCE10Stressful exams. Little time to make the resolution.0.000.002/39
19s2NCE11Technical difficulties due to fluid communication.4.500.252/39
19s2NPE1Difficulty resolving doubts.1.00-1/39
19s2NPE2Most work done individually.5.00-1/39
19s2NPE3Recorded classes are not available.2.00-1/39
19s2NPE4Difficulties using the technological tools.5.00-1/39
Table 3. Positive (P) and negative (N) elements both common (C) and particular (P) of the SC modality (2020–2021 first semester). In bold, the items with a Mention Index higher than 20%.
Table 3. Positive (P) and negative (N) elements both common (C) and particular (P) of the SC modality (2020–2021 first semester). In bold, the items with a Mention Index higher than 20%.
ItemDescriptionAverage ScoreVARPMention Index
20s1PCE1Versatility: Attending classes on’ or off-campus.8.942.0518/39
20s1PCE2Recording class sessions. It allows reviewing contents.9.181.0512/39
20s1PCE3Very good teaching method.8.141.558/39
20s1PCE4Blackboard/PC. Very good teaching possibilities.9.400.647/39
20s1PCE5Agile technological system. Better than virtual classes.8.141.557/39
20s1PCE6No time is wasted on campus trips.8.330.884/39
20s1PCE7Classroom cameras: allow a great interaction.8.750.684/39
20s1PCE8It allows good interactivity b/w instructors and students.8.330.223/39
20s1NCE1Students less attentive if they attend classes virtually.2.711.639/39
20s1NCE2In the classroom, students learn more and interact better.2.251.688/39
20s1NCE3Sometimes writings on the board do not look good.2.004.665/39
20s1NCE4Teamwork: quite difficult.2.672.883/39
20s1NCE5Having to take F2F exams.1.000.663/39
20s1NCE6Class session recordings are deleted too soon.4.001.003/39
20s1NCE7Difficulties in interactions between students (class/remote).3.000.003/39
20s1NCE8Students appear on a screen for a long time.2.000.002/39
20s1NCE9Students do not experience the campus environment.3.000.002/39
20s1NCE10Difficulties in understanding contents/subjects.3.000.002/39
20s1NPE1Sending too many communications via e-mail.4.00-1/39
20s1NPE2If noise in the classroom, remote learning is difficult.4.00-1/39
20s1NPE3Improves ERT, but Virtual F2F option is worse than F2F.5.00-1/39
20s1NPE4Virtual F2F conditioned by technology of each student.2.00-1/39
20s1NPE5Exams: Little time to make the resolution.0.00-1/39
20s1NPE6Less interaction between students.2.00-1/39
20s1NPE7Complications connecting specific link of the class group.4.00-1/39
20s1NPE8Virtual F2F conditions the way of learning of students.4.00-1/39
Table 4. Data from the Emotional Appraisal: F2F, ERT and SC.
Table 4. Data from the Emotional Appraisal: F2F, ERT and SC.
Emotional PairPairs of EmotionsF2F Mean (SD)ERT Mean (SD)SC Mean (SC)
EP1Confidence/Suspicion0.82 (0.16)0.51 (0.26)0.73 (0.27)
EP2High Quality/Low Quality0.83 (0.14)0.49 (0.28)0.75 (0.23)
EP3Useful/Useless0.78 (0.23)0.59 (0.25)0.76 (0.26)
EP4Interesting/Boring0.69 (0.19)0.47 (0.25)0.60 (0.25)
EP5Known/Unknown0.79 (0.27)0.47 (0.32)0.60 (0.29)
EP6Comfortable/Uncomfortable0.68 (0.23)0.58 (0.32)0.71 (0.24)
EP7Attractive/Not Attractive0.67 (0.20)0.48 (0.30)0.70 (0.29)
EP8Innovative/Conventional0.37 (0.29)0.67 (0.23)0.81 (0.25)
EP9Simple/Complex0.67 (0.26)0.56 (0.24)0.56 (0.22)
EP10Nearby/Distant0.80 (0.24)0.38 (0.29)0.56 (0.23)
EP11Funny/Not Funny0.59 (0.23)0.40 (0.24)0.49 (0.28)
Table 5. Degrees of freedom and value of the F test for each of the eleven ANOVA tests.
Table 5. Degrees of freedom and value of the F test for each of the eleven ANOVA tests.
Emotional PairPairs of EmotionsF Test
EP1Confidence/SuspicionF(2,91) = 12.84, p < 0.0001
EP2High Quality/Low QualityF(2,90) = 17.8, p < 0.0001
EP3Useful/UselessF(2,88) = 4.99, p = 0.0089
EP4Interesting/BoringF(2,91) = 6.26, p = 0.0028
EP5Known/UnknownF(2,85) = 7.68, p = 0.0009
EP6Comfortable/UncomfortableF(2,91) = 2.22, p = 0.1143
EP7Attractive/Not AttractiveF(2,91) = 6.24, p = 0.0029
EP8Innovative/ConventionalF(2,88) = 21.06, p < 0.0001
EP9Simple/ComplexF(2,87) = 1.89, p = 0.1570
EP10Nearby/DistantF(2,87) = 18.85, p < 0.0001
EP11Funny/Not FunnyF(2,88) = 3.94, p = 0.0230
Table 6. Results from the ANOVA, showing the p-value of each pairwise comparison of modalities.
Table 6. Results from the ANOVA, showing the p-value of each pairwise comparison of modalities.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Petchamé, J.; Iriondo, I.; Villegas, E.; Riu, D.; Fonseca, D. Comparing Face-to-Face, Emergency Remote Teaching and Smart Classroom: A Qualitative Exploratory Research Based on Students’ Experience during the COVID-19 Pandemic. Sustainability 2021, 13, 6625.

AMA Style

Petchamé J, Iriondo I, Villegas E, Riu D, Fonseca D. Comparing Face-to-Face, Emergency Remote Teaching and Smart Classroom: A Qualitative Exploratory Research Based on Students’ Experience during the COVID-19 Pandemic. Sustainability. 2021; 13(12):6625.

Chicago/Turabian Style

Petchamé, Josep, Ignasi Iriondo, Eva Villegas, David Riu, and David Fonseca. 2021. "Comparing Face-to-Face, Emergency Remote Teaching and Smart Classroom: A Qualitative Exploratory Research Based on Students’ Experience during the COVID-19 Pandemic" Sustainability 13, no. 12: 6625.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop