Development of a Framework to Assess Challenges to Virtual Education in an Emergency Remote Teaching Environment: A Developing Country Student Perspective—The Case of Peru

Abstract

The COVID-19 pandemic forced most countries’ higher-education systems to shift to distance learning, which has been called either “Corona Teaching” or, more formally, “Emergency Remote Teaching (ERT).” Students were suddenly faced with a new class format delivery and the many challenges of virtual education. The present study aims to identify and measure the challenges in three stages: (1) a qualitative method approach was used to gather the opinions of 50 students that were then analyzed and coded to identify their perceived major challenges; (2) a survey was completed by 165 students to prioritize the relative importance of the previously identified challenges using the AHP as the weighting approach; (3) an assessment framework was developed, using statistical techniques to measure the extent of the challenges for specific stakeholders based on survey responses. The main challenges students face are inadequate physical facilities at home, difficulties with the learning platforms, and financial concerns. These results are applicable beyond the present research context. For the first time, an ERT assessment framework of the challenges was developed using composite indicators derived from students’ opinions and perspectives. This ERT framework allows for the exploration of a community of students’ vulnerability to the challenges within the context of an emergency remote environment.

1. Introduction

At the beginning of the COVID-19 pandemic (March 2020), many higher education institutions closed their doors for a few weeks to redesign their courses and train their faculty and staff in online instruction. Not all institutions had this opportunity or were savvy enough to thoroughly prepare and quickly switch to virtual instruction. Virtual schooling is possible asynchronously or synchronously. In the asynchronous format, there is no simultaneous student and faculty interaction during class delivery (although there might be some contact in the form of Q&As at the individual or group level). In the synchronous format, the instructor and students participate in a virtual classroom at a specified time. When the pandemic forced institutions to only provide distance education, synchronous virtual education was the format of choice, simply because it most closely resembled in-class instruction. Even so, once classes were restarted in the virtual delivery mode, there was conflicting evidence about how effective this virtual instruction was and the challenges students faced in this new learning format in the context of the pandemic. This was the motivation for the present research about the challenges online university students faced during the pandemic using a student sample from Peru.

Virtual education, also called virtual instruction, virtual learning, technology-mediated learning (TML), or other similar terms, can be defined for the purpose of this study as education mediated by technology; that is, “an environment in which the learner’s interactions with learning materials (readings, assignments, exercises, etc.), peers, and/or instructors are mediated through advanced information technologies” [1] (p. 2). The term “information technology” broadly refers to computing, communication, and data management technologies and their convergence, including the whole ICT spectrum.

1.1. Virtual Instruction

Virtual instruction has been studied for a few decades now. A solid body of research has explored the determinants for successful learning outcomes and students’ satisfaction. For example, Eom and Ashill [2] found that instructor–student dialogue, student–student dialogue, instructor activities, and course design positively affected online user satisfaction and learning outcomes at universities. This finding is important because the first three determinants are fundamentally different in an online format than they are in a classroom setting. Indeed, it is well known that student satisfaction is tightly coupled with the perception of the instructor’s engagement in course interaction, which is more natural in physical classroom settings [3]. There is a debate about whether there is fundamentally less faculty–student interaction online than there is in the physical classroom. Proponents of online education suggest that this is not the case and that the interaction is simply different in web-based learning; therefore, this is a practical and viable option to meet all of the learners’ educational needs [4,5].

Furthermore, online education advocates suggest that online instruction presents a more customized format in which instructors interact with each student [6]. Empirical studies show that skill-based learning outcomes can be equally achieved in online classes as in classroom settings [7]. A marketing perspective has been used to research student satisfaction with online learning. Parahoo, Santally [8] developed a predictive model for this purpose that showed that (in order of importance) university reputation, physical facilities, faculty empathy, and student interaction were needed for student satisfaction with online learning. Additionally, more than 20 years ago, some researchers underlined the importance of integrating distance learning objectives and functions into essential student services and student functions at all organizational units of higher education institutions [9,10]. However, all these studies have been conducted in a common education situation, i.e., students who had chosen to register for online classes and instructors who had agreed (and probably been trained) to teach in that format.

1.2. Emergency Remote Teaching during Covid-19

During the pandemic, students and teachers were forced into a virtual instruction format, which has been called “corona teaching” in the literature [11].The term might be understood as “teaching efforts to use the scarce technological resources available to teach their courses as if they were still in a classroom situation.” In other words, this consists in “virtually transforming face-to-face classes, but without changing the curriculum or the methodology” [11]. This is also consistent with the concept of emergency remote teaching (ERT), which, in contrast with long-planned online experiences, is a temporary shift of instructional delivery to an alternate delivery mode due to crisis circumstances [12,13]. Some studies also refer to this new global experimentation with remote teaching as emergency online education [14]. Indeed, the pandemic forced a quick adaptation to an educational modality never before experienced without the corresponding and necessary training [15,16,17,18,19,20]. This situation caused stress and frustration for teachers, students, and families worldwide. In addition to this stress, there was the ever-present fear of catching the coronavirus infection, death of family members, neighbors, and friends, as well as social isolation, increased screen exposure, and other stressors that caused a range of negative mental health consequences [21,22,23,24,25]. In some cases, this psychosocial process caused trauma and the development of the so-called “coronaphobia”[26]. García-Morales, Garrido-Moreno [27] briefly reviewed the literature to explore the barriers and challenges that universities faced during the COVID-19 disruption and argued that higher education has been transformed as a result. Students reported that a major challenge they experienced was technical problems [28], while other authors highlighted different ways in which virtual education could increase the digital divide [29]. Indeed, inequality observed before the pandemic became even more visible, challenging, and problematic due to the digital divide during the turbulent time of the spread of coronavirus in countries such as Peru [24,30]. In addition, several psychological barriers were reported, such as difficulty in maintaining attention, boredom, and inability to self-organize. Teachers also faced challenges adapting to the new emergency remote teaching environment [31]. Fülop, Breaz [32] analyzed teachers’ acceptance of new technologies and the impact on their well-being and university sustainability in an emerging country (Romania).They found several factors that caused teachers to be discontent when adapting to new technologies during the COVID-19 pandemic. Studies exploring the issues and challenges associated with distance learning in this unusual emergency context are still incipient, especially in developing countries such as Peru where technical facilities and Internet access are limited (Appendix A). This research aims to fill this gap in the literature, which is of great importance because students’ perceived challenges with distance learning, combined with the emergency remote teaching situation, negatively impacts both the expected performance improvements as well as the expected ease of use, both of which will lead to a decrease in acceptance of virtual instruction as a whole. Assessing the state of readiness for virtual instruction is considered to be the best practice to prepare for this type of instruction; however, this assessment needs to take into account the emergency remote teaching circumstances by focusing less on traditional outcomes (e.g., course satisfaction) and more on the context (e.g., home infrastructure), input (e.g., financial resources), and process(e.g., connectivity and computer availability) elements from a student’s perspective [12]. Furthermore, this assessment framework may help train faculty in emergency remote teaching [33].

Therefore, based on the previous literature discussion and, more specifically, following Hodges, Moore [12]’s recommendation of exploring broad questions while focusing the assessment on context, input, and process, the following research questions were proposed:

• RQ1: What are the most common challenges students face in virtual instruction within the context of a pandemic?
• RQ2: What is the importance of these virtual instruction challenges from the students’ perspective?
• RQ3: Is it possible to assess the extent of challenges faced by online students?

The first research question is exploratory, which suggests a qualitative method approach, while the second and third questions constitute an assessment of the findings of the first question and was established using a quantitative approach. In summary, this study’s use of mixed-method research fits the rationale for applying this approach [34] and concludes with the development of an assessment framework to evaluate the challenges of the emergency remote teaching environment (ERT) within the context of the pandemic in Peru, but which are also important to other ERT environments.

2. Materials and Methods

This study has three phases: first, identification of the challenges of virtual education in the context of the pandemic (open question survey); second, prioritization of those challenges (AHP pairwise comparison survey), and third, development of a virtual challenge assessment framework. The methodology and materials used for the three phases of this research are summarized in Figure 1.

2.1. Methodology for Phase I: Identification of the Challenges of Distance Learning

Given that a study of the challenges of virtual instruction within the context of a health crisis is rare or even unprecedented, it was decided that a qualitative study would be used in this phase. A traditional closed survey would have been highly inaccurate and incomplete in this study stage due to The novelty of the research subject matter and the many unknowns, especially in a developing country such as Peru. The lack of remote work experience among professors, students, and other family members on such a vast scale during the state of emergency and developing pandemic was another complication. Qualitative research is recommended for unprecedented and poorly understood situations, so a qualitative approach was used for the first stage of this study. The lockdown prevented organizing personal meetings to conduct surveys with respondents; therefore, the Peruvian Institute of Public Opinion (IOP) offered to collect survey data online1. The open qualitative questions (4, 5, 6, and 7 in the survey) were as follows:

• RQ4. What problems do YOU face learning using virtual instruction?
• RQ5. What problems does YOUR INSTITUTION face using virtual instruction?
• RQ6. What problems does YOUR FAMILY face due to virtual instruction?
• RQ7. What INFRASTRUCTURE problems do you face at home using virtual instruction?

Participants were allowed to list up to ten problems for each dimension/question. The IOP e-mailed potential student participants and posted the survey on college websites to collect responses from students associated with their host higher-education institution from June–July 2020. Since the survey was posted on websites, it was impossible to determine a response rate. The responses were meticulously, individually, and independently coded by two of the authors until theoretical saturation was reached. In qualitative analysis, the sample size is considered adequate when theoretical saturation is reached [35]. Theoretical saturation occurs when no new properties of the data categories emerge in further data collection [35]. In this study, once the authors had analyzed 50 cases, no new codes were created; theoretical saturation had been reached. Theoretical saturation means that codes (based on distinctive responses) and categories (based on a grouping of related responses) became repetitive, and there was no need to create new answer codes or group them into new categories. The demographics of the participants in the qualitative research phase are given in

Table 1. Qualitative participant demographics (N = 50).

Demographics	Characteristics	No	%
Age	Less than 20 years old	15	30%
	21–24 years old	25	50%
	25–29 years old	4	8%
	More/egual 30 years old	6	12%
Gender	Female	32	64%
	Male	18	36%
Discipline	Administration	5	10%
	Economy	3	6%
	Engineering	1	2%
	Others	41	82%

.

Once the responses to the open questions were collected, they were coded and assembled into thematic groups following standard practices for this type of qualitative study [36]. Seven major challenge thematic groups: quality of virtual instruction (C1), connectivity & equipment (C2), personal issues (C3), home infrastructure & study environment (C4), learning platform & access to resources (C5), financials (C6) and university administration & costs (C7) were identified and will be discussed in Section 3 of this study.

2.2. Methodology for Phase II: Prioritization of Challenges

Once specific challenge groups had been identified, it was decided that a survey research approach was most suitable to prioritize these challenges. The participants were asked to pairwise compare the seven major challenge groups (C1 to C7) to calculate their relative priorities (weights) using the Analytic Hierarchy Process (AHP) methodology [37]. The responses to the indicators’ questions and the calculated priorities (weights) in this stage were then used to develop an AHP assessment framework (see Appendix B for a brief discussion of the AHP methodology).

Five hundred seventy postgraduate MBA students from one of the top private Peruvian universities were invited to participate in this survey. Of this total, 165 students responded, yielding a survey response rate of 29%. The results were analyzed using the Analytic Hierarchy Process (AHP) to derive their relative importance (priorities). During this prioritization phase, efforts to comply with accepted best practices in AHP studies were heeded [38]. The participants’ demographics are provided in

Table 2. Prioritization survey demographics (N = 165).

Demographics	Characteristics	No	%
Age	Less than 20 years old	1	0.63%
	25–29 years old	25	15%
	30–34 years old	45	27.5%
Gender	More than 34 years old	94	57%
	Male	104	64%
	Female	54	36%

.

2.3. Methodology for Phase III: Development of an Assessment Framework

The most frequent quotes within each previously identified category (C1 to C7) in Phase I (

Table 3. Open question theme codes and their count.

Theme Code	Themes	Count	% Count
100	Internet connectivity and lack of equipment by students Internet connectivity and lack of equipment by Hi-Ed institutions	99	20%
150		11	2%
201	Inadequate physical facilities to study (e.g., study in bed) Inadequate environment to study (e.g., constant interruptions)	55	11%
203		29	6%
300	Learning Platform (e.g., too cumbersome)	14	3%
400	Quality of Teaching (e.g., teachers untrained for virtual education) Exams (e.g., not enough time)	96	19%
500		6	1%
600	Access to Resources (e.g., library books)	22	4%
700	Lack of class interaction with students and teachers	26	5%
800	Personal Problems (e.g., physical exhaustion, lack of focus)	47	9%
850	Financial problems (student)	6	1%
900	Personal Organization Problems Mental Health	23	5%
1000		28	6%
1100	University Admin University Costs	4	1%
1200		9	2%
1300	Financial problems (family)	20	4%
Grand Total		495	100%

) became the variables’ indicators (survey questions). They were edited to be in the form of a question (Q10 to Q20), each with sub-questions, and administered as part of the survey in the previous phase. Standard procedures for the construction of this type of assessment framework were followed according to established practices [39,40].

An indicator is “a quantitative or qualitative measure derived from a series of observed facts that can reveal a relative position in a given area and, when measured over time, can point out the direction of change.” [39] (p. 7). This direction of change may help identify performance trends and highlight potential issues needing improvement. In this study, each indicator is constituted by the response to one of the questions posed in the survey (Appendix A: Q10–Q20). These indicators can be converted into thematic indicators by grouping them along with common themes or categories (Challenges C1 to C7 in this study). A composite indicator is the compilation of the thematic indicators into a synthetic index and is presented as a measure of a challenge dimension.

There is a rich tradition of using AHP-based assessment frameworks taking advantage of the rating model approach. These frameworks have been used to evaluate proposals, such as vendor assessment in public bids [41] and similar evaluation tasks [42]. More recently, they have also been employed, in combination with advanced statistical techniques, as a psychometric assessment tool for organizational behavioral dimensions [43]. Following these practices, the questionnaire used in the previous stage also included questions developed from the most frequent quotes from students in the qualitative phase of the study. Based on this, an AHP assessment model for the top virtual instruction challenges in the context of the pandemic was proposed, as shown in Figure 2. Each challenge constitutes one of the dimensions to be considered when assessing the challenges to virtual instruction. Below each challenge dimension, a list of survey questions (also called indicators) was developed to assess the specific dimension.

3. Results

Next, the results of the data analysis for each of the three phases will be provided.

3.1. Phase I Results: Identification of the Challenges of Distance Learning

In this phase, the responses of the 50 participants to the proposed four open questions were coded into thematic challenges. For coding purposes, one of the authors read the answers to each question, created codes for the responses to each question, and grouped the codes into common themes. For example, the coded responses “201—Not appropriate environment to study” and “202—Interruptions…” were grouped under the theme “200—Inadequate Environment to Study”. The coding process followed standard recommendations for qualitative studies [36]. Next, another author who had not participated in the first analysis read the answers to each question, tried to code the answers according to the codes and themes created by the first analyst, and added new codes/categories as needed. There was a 5% discrepancy rate between the two authors’ coding, mainly placing a coded answer within a specific theme rather than within the themes themselves. The discrepancies were discussed, and an agreement was reached when needed. The most important result of this exercise was that the codes and, more importantly, the theme grouping of the codes or themes made sense.

To summarize the data analysis process, the codes (a total of 153) correspond to I.D.s given to specific different answers (e.g., answer codes 601–607 in

Table 4. Partial coding example.

600	Access to Resources
601	Lack of access to libraries
602	Presentations are needed in addition to videos
603	Insufficient study material
604	lack of access to laboratories
605	Lack of access to class resources due to lack of programs
606	Lack of access to needed services (e.g., printing)
607	Lack of access to the specific course material (e.g., design)

) provided by participants, and the themes (a total of 16 in Table 3) were created as a grouping of codes with a standard topic answer (e.g., the set of answer codes 601–607 constitutes the theme “600—Access to Resources” as shown in Table 3). When a participant provided an answer semantically identical to a previous response, a new code was not created, but the frequency count for the specific answer code was increased.

The final grouping of codes into similar themes and their frequency count is shown in Table 4.

A further examination of the themes in Table 3 allowed for those with close or similar ideas to be grouped into higher-level categories. For example, the themes coded 100 and 150 refer to challenges related to Internet connectivity and lack of proper equipment; therefore, they can be considered a single higher-level challenge or category or can also be shown as a cluster of the original themes. Each grouping of themes (from Table 4) into higher-level categories is summarized in

Table 5. Challenges for virtual education.

ID	Theme Codes	Challenge Categories	Count	% Total
C1	400/500/700	Quality of Virtual Instruction	128	26%
C2	100/150	Connectivity &Equipment	110	22%
C3	800/900/1000	Personal Issues	98	20%
C4	200	Home Infrastructure & Study Environment	84	17%
C5	300/600	Learning Platform & Access to Resources	36	7%
C6	850/1300	Financial Problems	26	5%
C7	1100/1200	University Admin &Costs	13	3%
		Grand Total	495	1

.

The categories (or groups of challenges) presented in Table 5 were listed in their count frequency in the coded participants’ responses. A detailed discussion of these grouped themes (C1 to C7), as shown in Table 5, is as follows:

• C1—The perceived quality of virtual instruction

This challenge constituted 26% of the answers and is the largest grouped category; this challenge was the most common concern among the students. This included concerns related to the quality of teaching (19%, code 400), exams (1%, 500), and the lack of class interaction (5%, code 700). This challenge can be defined as the student’s perception that the quality of virtual instruction is lesser than its physical counterpart due to the lack of class interactions (either among peers or with faculty) and suitable course assessment. Related quotes2:

“Faculty are poorly trained to teach in a virtual environment”

[C7P7_1]

“Instruction quality is lower”

[C31P6_1]

“Class time is less than in physical sessions”

[C13P6_3]

“The most important thing in my career is practice, which is difficult to do…”

[C13P6_1] (an architecture student)

“Too many [exam] questions for such a short time”

[C41P6_2]

“Class participation is not the same in a virtual environment”

[C40P6_1]

• C2—Connectivity and proper equipment

This challenge constituted 22% of the answers and is the second-largest cluster of issues. Within this category, 90% of the responses were related to the students’ technology limitations (code 100), and the rest were about the institution’s limitations (code 150). This challenge is defined as lacking suitable Internet connectivity or equipment to participate in virtual education classes. Related quotes:

“There are not enough computers [for everybody] at home”

[C8P8_1]

“My brothers have to study online, and there are not enough computers for all of us”

[C24P8_2]

“Connection problems in peak hours”

[C48P6_3]

“Not all of us have the opportunity to have Internet access”

[C6P7_1]

“More than 50% of students do not have Internet access”

[C50P7_2]

“I think that internet access to both faculty and students is something…that could be improved”

[C1P7_1]

“[University] servers cannot cope with the traffic”

[C8P7_1]

• C3—Home infrastructure and study environment

The third-largest cluster (17%) can be defined as the student’s lack of proper infrastructure (e.g., a desk) and study environment at home (e.g., lack of privacy, noisy family). Related quotes:

“Too much noise and little space to study at home”

[C36P8_2]

“I don’t have either a suitable chair or table, so I study in bed”

[C26P9_1]

“I don’t have a suitable place to listen to my classes. I ask [my family] to be quiet”

[C16P8_1]

“And there are three of us sitting at the same table”

[C19P8_3]

• C4—Personal issues

This cluster included issues such as personal issues (e.g., physical exhaustion, code 800), organizational issues (e.g., time management, code 900), and mental health (code 1000). It can be defined as the set of challenges related to physical and cognitive issues at a personal level. Nine percent of students surveyed a reported difficulty concentrating and physical exhaustion, five percent expressed problems organizing their work and their family life, and six percent directly reported coping with mental issues such as depression. This group constitutes 20% of the total responses (the third largest category). Related quotes:

“I do not have spaces for either recreation or sharing with peers”

[C8P6_1]

“Get tired of spending so much time in front of a screen”

[C39P6_1]

“I pay little attention to the class”

[C26P8_2]

“I need to care for the little ones while my parents are at work”

[C34P8_2]

“Lockdown gets young people depressed”

[C11P6_5]

“[I have a] family, friends who are either sick or have died”

[C18P6_1]

“They [university] don’t care about our mental health”

[C20P7_4]

• C5—Distance learning resources

This challenge cluster can be defined as the ability of the student to use and access the distance learning platform (code 300) and learning resources at large in the institution (code 600). There may be difficulties accessing the learning platform due to a lack of information and its extreme complexity. Additionally, students complained about the lack of access to physical and academic material (e.g., books, labs). About 3% considered the platform too cumbersome to be appropriately used, and 4% complained about the lack of access to library books. Related quotes:

“Lack of expertise to use the platform by both teachers and students”

[C50P6_6]

“I don’t have access to needed services (e.g., printer…)”

[C30P6_7]

“Not having access to library books”

[C18P6_1]

• C6—Finances

This challenge can be defined as the student’s financial concerns that impact education. These concerns may be directly related to the student’s (code 850) or the family’s financial situation (code 1300) during the pandemic. This cluster constitutes the fourth most commonly reported challenge, 10% of the total. Related quotes:

“I struggle to pay monthly university fees”

[C31P6_2]

“[My family’s] economic problems [are] the main problem”

[C50P8_1]

“Salary reduction for the only family provider”

[C30P8_1]

• C7—University administration and costs

This challenge cluster can be defined as those non-technical concerns related to the university (e.g., lack of leadership during the crisis, code 1200) and other miscellaneous issues (code 1100). These responses constituted 2.6% of the total answers. Related quotes:

“University tuition has not decreased even though we are not using their facilities anymore”

[C02P7_01, C41P7_01, C45P7_01]

“They [universities] have no concern for students economic situation. They ignore our requests to decrease tuition”

[C21P7_02]

“University authorities and faculty lack leadership”

[C51P7_01]

In summary, the four most prevalent challenges reported for distance learning are as follows:

• Quality of instruction/learning;
• Poor Internet connectivity and lack of proper equipment;
• Personal and psychological issues;
• Lack of appropriate home infrastructure.

The following three challenges were next in prevalence, although they were reported with less frequency:

• Learning platform and access to resources;
• Financial issues related to students and families;
• General concerns related to the university and others.

3.2. Phase II Results: Prioritization of Virtual Instruction Challenges

The first stage of this study identified the challenges faced by distance learning students. The frequency of quotes about these challenges allowed the researchers to identify the students’ everyday challenges but not their level of importance. Once the difficulties were identified and categorized, the second stage of the research aimed to prioritize the challenges based on the students’ perceptions.

In the first stage of this study, seven challenges (or clusters of challenges) were identified as follows: (C1) Learning and Instruction Quality, (C2) Connectivity and Equipment, (C3) Personal Issues, (C4) Home Infrastructure and Study Environment, (C5) Distance Learning Resources, (C6) Finances, and (C7) University Administration and Costs. The Analytic Hierarchy Process (AHP) method was applied to determine the relative importance of the presented challenges (C1–C7), and pairwise comparison questions were created. The first step in the AHP method requires formulating the prioritization as a hierarchical model, as shown in Figure 3 for this specific task.

The second step required asking the participant(s) a series of pairwise comparison questions related to the relative importance of the challenges. These data are used to derive the relative weights or importance of the challenges.

The comparison questions were formulated as follows: “with respect to the challenge importance3, which is more important for you?”; “C1—Learning & Instruction Quality” or “C2—Information Technology”? Once the more important criterion was determined, the question was, “To what extent…?” The participant had to select a relative intensity using Saaty’s fundamental scale from 1 to 9, which ranges from “Equally important” to “Extremely more important” (see Appendix B for an example of survey questions).

Following standard practice in AHP survey research to address consistency in group judgments, a minimum spanning tree approach was used, i.e., asking only comparison questions above the main diagonal [44]. This way, only six comparisons (C1:C2, C2:C3, C3:C4, C4:C5, C5:C6, and C6:C7) were needed, and a consistency ratio of 0 was assured. The other judgments in the pairwise comparison matrix were calculated (e.g., C1/C3 can be calculated as the result of multiplying the judgments (C1/C2) * (C2/C3)). This is common in similar studies because anonymous surveys do not allow for negotiating judgments with the individual participants to address consistency issues [45]. The pairwise comparison (PWC) matrices of all the survey respondents were combined using a geometric mean to create an aggregated PWC matrix. Finally, the overall priorities were calculated by simply normalizing the pairwise comparison matrix (see Appendix B for a more detailed explanation). The priority results are shown in Figure 4.

These results show that students consider the following challenges of virtual education as the most important (70.8% of the overall importance):

• C4—Lack of proper home infrastructure and study environment (W4 = 0.2088);
• C5—Learning platform and access to resources (W5 = 0.2900);
• C6—Financial issues related to students and families (W6 = 0.2087).

These three challenges combined constitute 70.8% of the overall importance. On the other hand, the most commonly occurring issues, as reported by the students (Table 3), were C1 (quality of virtual instruction), C2 (connectivity and proper equipment), and C3 (personal issues). However, high counts of student reports of a challenge suggest prevalence, but not necessarily the level of importance, as this prioritization phase of the study unveils. In other words, an AHP analysis allows us to determine the priority or degree of importance(based on the stakeholders themselves) and which challenges must be addressed first due to their pressing importance.

3.3. Phase III Results: Development of a Virtual Instruction Challenge Assessment Framework

The development of a virtual instruction challenge assessment framework requires composite indicators and synthesis indices of individual indicators that have been widely used in policy analysis and public communication [39]. They have been beneficial for benchmarking country performances, and their number has been steadily increasing since the milestone survey review by Bandura [46].

3.3.1. Identification and Development of Relevant Indicators/Variables

The availability of usable data that could become relevant indicators constitutes one of the most challenging parts of the process. The advantage of the current study is identifying the variables directly from the stakeholders (survey respondents) during the first phase (qualitative approach) and quantifying them in phase II of the research. Standard statistical techniques such as listwise deletion for missing data were used to process the collected data and ensure their overall quality before use.

The approach to developing indicators based on stakeholders’ input uses the most frequently quoted statements as questions to assess each of the constructs (challenge clusters: C1 to C7) proposed in the study. This helps develop a tentative survey to determine this study’s challenge clusters (C1 to C7). The original set consists of 60 questions (indicators) unequally distributed among the seven categories, as shown in Figure 2.

3.3.2. Standardization of Variables to Allow Comparisons

There is a need to standardize or normalize the variables before aggregating them. A problem occurs when the different indicators correspond to other variables expressed in their units (e.g., income, population, age). However, standardization is not a problem for survey data collected through a standard scale. Furthermore, when using AHP rating models, the scales of the variables (C1 to C7) are further standardized to allow for the aggregation of the variables [47].

3.3.3. The Weighting of Assessment Variables

The aggregated variables (C1 to C7 in the study) must first be weighted. While it is not unusual to consider all the variables equally weighted for simplicity, these weights are vital because stakeholders do not consider the different dimensions to be equally important and because they seriously impact the overall assessment score. Additionally, it is reasonable to expect that not all the variables will have an equal impact or be considered equally important by the different stakeholders. Fortunately, the C1–C7 variables had already been weighted using an AHP approach through surveying students in the previous phase of this study (Figure 4).

One important consideration is the aggregation of the indicators used for each challenge category. While indicators have been collected for each variable, it is necessary to determine whether the indicators converge toward a single variable and can therefore be aggregated. Greco, Ishizaka [48] reviewed the issues surrounding composite indicators’ weighting, aggregation, and robustness. In particular, they identified many participatory methods for this purpose, such as the Analytic Hierarchy Process (AHP), the Budget Allocation Process, and Conjoint Analysis. They also identified data-driven weighting methods such as Correlation Analysis, Multiple Linear Regression Analysis, Data Envelopment Analysis (DEA), and Principal Component and Factor Analysis. Furthermore, they discussed the strengths and weaknesses of using these methods. They concluded that using one or the other depended on the nature and use of the assessment framework.

For this study, the decision was made to use principal component and exploratory factor analysis (EFA) to analyze the convergence of the individual indicators into their thematic variables or categories (C1 to C7), given that this is a common and well-known approach for this purpose. This approach allows for those indicators whose contribution to the variable scale (loading factors) may be too low to be considered, or whose contribution to the internal reliability of the variable (measured as scale reliability through Cronbach’s Alpha coefficient) may be detrimental to be discarded. Additionally, by determining an upper threshold for an indicator to be present as well as a lower threshold for an indicator to be discarded during the EFA (exploratory factor analysis) process, there is no need to take into account the now minor loading differences of the surviving indicators (survey items) of each construct. They can be equally aggregated to obtain each of their corresponding challenge variables. Loading factors of 0.4 are widely used as a lower threshold [49]. Another commonly used recommendation is that for a sample size of 100, the loading should be greater than 0.512; for 200, it should be greater than 0.364 [50]. The sample size for this study phase was between 100 and 200; therefore, the lower threshold of 0.4 was considered reasonable. Still, most factors’ loading was significantly above 0.512, usually in the 0.7–0.9 range (EFA original statistical results are shown in Appendix C). The summary of the scale reliability and measurement variables is given in

Table 6. Scale reliability and descriptive statistics.

							1	2	3	4	5	6	7	8	9	10	11	12	13	#
		Variable	Items	Alpha	Mean	SD	C1.1	C1.2	C1.3	C2	C3.1	C3.2	C3.3	C3.4	C4	C5.1	C5.2	C6	C7.1	C7.2
1	C1.1.	QUALITY	4	0.841	1.787	0.916	1
2	C1.2.	INTERACTION	9	0.914	2.780	1.106	0.622 **	1
3	C1.3.	ASSESSMENT	5	0.817	2.528	0.983	0.571 **	0.714 **	1
4	C2.	EQUIPMENT	5	0.791	1.630	0.784	0.248 **	0.174	0.205 *	1
5	C3.1	PERSONAL	6	0.834	3.154	1.019	0.528 **	0.691 **	0.619 **	0.239 *	1
6	C3.2	MENTAL CONCERNS	3	0.730	3.061	1.061	0.277 **	0.404 **	0.387 **	0.267 **	0.339 **	1
7	C3.3	MENTAL ENVIRONMT	3	0.725	2.769	1.059	0.460 **	0.507 **	0.635 **	0.068	0.550 **	0.344 *	1
8	C3.4	MENTAL SADNESS	2	0.720	3.535	1.136	0.077	0.053	0.119	0.134	0.084	0.421 **	0.091	1
9	C4	HOME INFRASTR.	4	0.845	2.577	1.201	0.555 **	0.482 **	0.440 **	0.358 **	0.604 **	0.172	0.410 **	0.057	1
10	C5.1	LEARNING PLATFORM	4	0.722	1.702	0.731	0.662 **	0.425 **	0.373 **	0.470 **	0.380 **	0.346 **	0.276 **	0.129	0.471 **	1
11	C5.2	ACCESS TO RESOURCES	4	0.815	2.715	1.128	0.557 **	0.590 **	0.547 **	0.261 **	0.505 **	0.328 **	0.387 **	0.072	0.466 **	0.492 **	1
12	C6	FINANCIALS	4	0.817	2.915	1.019	0.229 *	0.379 **	0.385 **	0.366 **	0.306 **	0.401 **	0.260 **	0.223 *	0.297 **	0.214 *	0.302 **	1
13	C7.1	UNIVERSITY ADMIN	2	0.717	2.398	1.067	0.536 **	0.498 **	0.603 **	0.13	0.388 **	0.337 **	0.472 **	0.147	0.317 **	0.369 **	0.499 **	0.365 **	1
14	C7.2.	UNIVERSITY COSTS	2	0.819	4.040	1.091	0.181	0.289 **	0.225 *	0.062	0.311 **	0.311 **	0.215 *	0.213 *	0.14	0.266 **	0.318 **	0.308 **	0.269 **	1

Listwise N = 117; ** Significant at 0.1 level (two-tailed); * Significant at 0.5 level (two-tailed).

.

On the other hand, the Analytic Hierarchy Process (AHP) was used to weight the significant categories (C1 to C7), which allows for the prioritization of the different challenge variables identified by stakeholders4 through pairwise comparisons. It can easily integrate each stakeholder’s weighting judgment into the prioritization process, which makes the AHP a highly convenient weighting process. Greco, Ishizaka [48] recognize this technique’s importance and popularity and argue that the only drawback is that the number of pairwise comparisons may be too high for the participants. By only asking for the minimum set of comparisons needed to calculate all the remaining ones, the total number of pairwise comparisons was six. The weighting of the different dimensions or challenge variables was already available from the prioritization of challenges in the previous phase of this research (Figure 4). As a result of the analysis in this section, the final AHP assessment framework can be defined as shown in Figure 5. The respective survey questions are provided in Appendix E.

3.3.4. Sensitivity Analysis

Given that the assessment index results may depend heavily on selecting, weighing, standardizing, and aggregating the variables, sensitivity tests are recommended when using assessment frameworks based on composite indicators [39,40]. The present study grouped the indicators into specific variables (challenge dimensions) according to their EFA construct loadings. The values of the variables were already standardized since the data were collected through a standard scale (1—least challenged to 5—most challenged). When using an assessment framework, it is good practice to explore how robust the results would be if the variable weights differed. For this purpose, the assessment process and related sensitivity analysis are illustrated in a case study in Appendix D.

4. Discussion

While other studies have found similar results in semi-structured surveys [28], this current study, due to the different phases of identification, prioritization, and statistical analysis of each challenge item as an assessment indicator, allows for a more refined discussion of the challenges of virtual instruction, not just within the context of the COVID-19 pandemic but also within the broader context of emergency remote teaching.

Table 7. Virtual Teaching Challenge Themes during the Pandemic (Phase I and II).

Challenge ID	Description	Challenge Rank * Phase I	Importance Priority Phase II	Priority Rank ** Phase II
C1	Perceived quality of instruction/learning	1	0.0679	6
C2	Poor Internet connectivity and lack of proper equipment	2	0.0882	4
C3	Personal and psychological issues	3	0.0668	7
C4	Lack of appropriate home infrastructure	4	0.2088	2
C5	Learning platform and access to resources	5	0.2900	1
C6	Financial issues related to students and families	6	0.2087	3
C7	General concerns related to the university and others	7	0.0696	5

Note—* ranked by frequency of quotes,** Ranked by priority.

summarizes the results obtained in the present study based on the results of phase I and II of the present research and references in the literature.

The perception of the quality of virtual instruction/learning (C1) as being lesser than that of face-to-face learning was the most common claim by students in this study (Table 7) and has been recognized as an essential challenge for students (as well as for many teachers) by many researchers despite the extensive literature proving otherwise [12,51]. However, it was possible to identify two different types of student statements in the present study. Those related to objective claims such as “Class time is less than in physical sessions” and “Too many exam questions for such a short time” and those rather generic or subjective in nature, such as “instruction quality is lower” in the virtual mode. It is worth noting that in all these cases, virtual instruction is measured using the face-to-face methodology as the reference for good practices rather than accepting that they are different teaching delivery modes. Indeed, class time in virtual instruction may be less than in physical sessions because many practitioners argue that it is harder for students to stay focused for a long time in virtual sessions [52]. In other words, the length of a class is not a measure of its quality.

Similarly, the only way to assess whether a number of questions are suitable for a specific period depends on the level of difficulty both in content and form (e.g., short essay) of the questions. In summary, the perception of lower quality virtual instruction seems to have a significant component of mental model prejudice that considers face-to-face teaching as the ideal mode and compares the different processes of virtual instruction against them. Furthermore, this prejudicial perception of lower quality instruction should not be surprising among students who were forced to switch teaching delivery formats almost overnight due to the pandemic without any strategic assessment. The argument here is also supported by the fact that during the prioritization phase of the study, the students gave less importance to this category of Quality of Instruction (relative weight of 6.8%) than to the categories of Learning Platform and Access to Resources (29.0%), Home infrastructure and Learning Environment (20.9%), and Financial Issues (20.9%) as shown in Table 7. Another possible explanation for the results is that the prioritization phase took place a few weeks later than the identification phase. Either the perceived quality of instruction had improved in those few weeks, or the students had not accepted this lower quality mode of instruction (the alternative was no instruction at all) but were now more concerned about the practical issues just mentioned.

Technical issues have also been recognized as important, and by some researchers [28], as the most crucial challenge faced by students during the pandemic emergency remote teaching. The current study was able to dig deeper into this issue and identified two types of technical issues: Connectivity and Equipment (C2) and Learning Platform and Access (C5).The first technical challenge (C2) refers to the availability of proper Internet connectivity and suitable equipment to participate in virtual instruction. In contrast, (C5) refers to the ability to understand and access the course learning platforms and the university’s online resources. Availability of Internet connection and proper equipment (C2) is a key component of the digital divide which was expected to be aggravated by the pandemic [53]. During phase I of the current study (qualitative),the student participants frequently mentioned this challenge (ranked #2 in Table 7) with quotes such as “There are not enough computers at home,” “Connection problems in peak hours,”, etc., while the access to the learning platforms and resources (“Lack of expertise to use the platform by both teachers and students, ”I don’t have access to needed services [e.g., printers]”, “…library books”) was considered to be less of an issue (ranked #5). However, during phase II of this study, which took place two months after the first one, the survey of 165 students showed that Connectivity and Equipment was not (or no longer) the most important technical challenge for students (8.8% of importance), but rather the issues with the Learning Platform and Access to resources (29.0%) were more important as shown in Table 7. Why the difference from phase I to phase II? The ideal explanation would be that there were improvements to the digital divide and those who had problems obtaining Internet connectivity and equipment had solved their issues; however, a more likely explanation is that the students with these problems got accustomed to their limitations and due to this, the issues with the learning platforms as well as the access to academic resources became the next challenging technical online issue5 Additionally, keeping in mind that a large number of participating students (about half the students) had never had a single online course (as derived from the demographics section of the survey),their experience dealing with learning platforms was minimal. In other words, the issues with learning platforms may have been caused in many cases by their lack of familiarity or insufficient training rather than by technical issues. Access to resources, sometimes as essential as printers and library books, imposed a more critical challenge, particularly for students who required laboratory classes. This concern about laboratory access has been expressed by students pursuing science and engineering disciplines in other countries such as China [54].

The qualitative stage of this study (phase I) also allowed for the identification of Personal and Psychological Issues (C3), such as the impact of a lack social interaction with peers and friends. Student quotes such as “I don’t have spaces for recreation or sharing with peers,” “…get tired in front of a screen”, and ”Lockdown gets young people depressed” were ubiquitous and led to ranking these challenges as #3 in the qualitative phase of the study (Table 7). In all fairness, many of these challenges are mainly related to the emergency remote teaching environment created by the pandemic and associated lockdown more than by the nature of virtual teaching itself. Virtual education does not stop people from having a face-to-face social life with friends and family, but the pandemic lockdown did. Furthermore, the severity of the pandemic implied that students (and everyone) felt as ifCOVID-19 infection was a sword of Damocles hanging over everyone’s head (“[I have a] family, friends who are either sick or have died”). While the qualitative phase showed these personal issues to be significant (ranked #3 in Table 7), a few weeks later, they had become far less important (6.7%) than most of the other challenges, as shown in Figure 4. Still, the lesson is that these personal and psychological issues are critical and should be dealt with early in an emergency remote teaching situation.

One virtual challenge, particularly related to the context of developing countries, was given by identifying Home Infrastructure and Study Environment (C3) as a significant challenge for students. Some quotes, such as “Too much noise and little space to study at home”, “I don’t have a suitable chair, nor table, so I study in bed”, and “There are three of us sitting at the same table” showed the reality of many students struggling with the online classes during the pandemic in a developing country. Interestingly, this challenge was frequently mentioned during phase I of the present study (ranked #3 in Table 7). At the same time, it retained its importance during phase II, namely the prioritization of the challenges (20.9% of the relative importance in Figure 4). In hindsight, this makes sense because changes to a home infrastructure are not easily achieved or are not always possible. This challenge is the one that is most related to socio-economic considerations. Adding a room or study space to a house is not easy. Home Infrastructure was identified as an important challenge during phase I of the study (ranked #4 in Table 7), and it was prioritized as one of the top three concerns (20.9% of the importance in Figure 4) during phase II. Therefore, this is undoubtedly an essential consideration for students in emergency remote teaching environments in developing countries. This Home Infrastructure and Study Environment challenge has also been found in many other countries in the developing world such as countries in the African continent, exposing the problems of virtual education caused by socio-economic differences [55].

Another challenge of importance for students was the financial concerns related to not only paying university tuition6 but also the viability of their overall family subsistence. Quotes such as “I struggle to pay monthly university fees”, [My family] economic problems [is the] main problem”, and “Salary reduction for the only family provider” illustrate the great importance of Financial Issues related to students and families (C6). This challenge was prioritized as the third most crucial issue during phase II of the present study (20.9% in Figure 4) and the students’ sixth most frequently mentioned concern (Table 7). This challenge is related to problems derived from socio-economic status that were aggravated during the pandemic and were also identified by other researchers [29,55].

There were also challenges related to General Concerns with the University (C7) costs and attitudes. Students perceived that university tuition costs should have been decreased given that the physical facilities were not being used. In general, they complained that the universities were not as concerned as they should have been about the students’ economic situation (“…they ignore our requests…”, “University and faculty authorities lack leadership”). The study’s first phase showed this as the least ranked challenge (7th ranked, Table 7), similar to the prioritization during the second phase, which also showed it to be the least essential concern (about 7% of relative importance in Table 7).

Finally, a significant and novel contribution of the present study is that by using the derived information on the challenges faced by students through direct data collection during the two phases of the study (identification and prioritization), it was possible to develop an assessment framework (during their phase).This may allow the evaluation of the exposure of student communities in emergency remote teaching environments and the progress of interventions (by using the assessment framework before and after the intervention) to address the identified challenges. To our knowledge, no student perspective assessment frameworks have previously been developed. So far, the knowledge acquired during the COVID-19 pandemic is also applicable to other emergency remote teaching environments worldwide, particularly in developing countries.

5. Conclusions

During the pandemic, governments worldwide closed all educational institutions to control the spread of the SARS-CoV-2 virus. This forced educational institutions to adapt quickly and move forward by providing unplanned online learning, also called emergency remote teaching (ERT), directly impacting and challenging the students, teachers, and universities. At the beginning of the pandemic, when this study was started, there was no other research about ERT challenges within the context of a pandemic. This study aimed to identify the challenges students face in an ERT environment and in a developing country such as Peru, and to create an assessment framework to evaluate those challenges and to be used by communities of at-risk students in future ERT situations. Being able to systematically assess the extent of the challenges that students in a given community face contributes to the ability to pinpoint the specific issues that must be addressed in each case, which constitutes the significant contribution of this research. Furthermore, many of the findings and implications of this study are also applicable to other countries, even more developed countries.

The main contributions of this research are as follows:

• The qualitative research performed using the survey method allowed for the identification of the most frequently reported challenges and difficulties faced by students studying online in ERT environments:
  • Perceived quality of virtual instruction (25%);
  • Poor Internet connectivity and lack of proper equipment (22%);
  • Lack of appropriate home infrastructure to study (17%);
  • Personal and psychological issues (9%).
• The application of the Analytic Hierarchy Process helped prioritize the perceived importance of the most critical student challenges related to virtual education from the students’ perspective (three challenges constitute 70.8% of the overall significance):
  • Distance Learning Resources: learning platform and access to resources (29%);
  • Home Infrastructure: lack of proper home infrastructure (20.88%);
  • Finances: financial issues related to students and families (20.87%).
• The use of the AHP and composite indictors have allowed for the development of an assessment framework to evaluate the nature and extent of the challenges faced by specific students or communities of students in ERT environments, making it possible to develop interventions to address those challenges.

Identifying possible approaches to address salient challenges was not part of the study. Moreover, several concerns expressed by the students could be addressed through well-known behavioral intervention strategies related to changes in perception or managerial initiatives. However, we would consider it a disservice to the practitioner community not to discuss possible actions to address the identified challenges based on observed practices and extant literature.

A series of possible considerations and actions that educational institutions and governments can use regarding the immediate and long-term adaptation to the changing environment of tertiary education may result in positive outcomes during this turbulent time. Concerning the challenges found in the presented research, the following actions/considerations can be applied:

• Provide careful instructional design and planning, using a systematic model for design and development [56].
• Consider online learning design options including the following nine dimensions: modality, pacing, student–instructor ratio, pedagogy, instructor role online, student role online, online communication synchrony, the role of online assessments, and source of feedback [51].
• Provide structured and planned educational material (content, methodologies, and shared goals) and more adequate e-learning platforms by using suitable interactive digital learning resources (video, animations, quizzes, and games) to maintain students’ attention [53].
• Survey students about their capacity to engage in remote learning, including areas such as equipment, family responsibilities, home environment, etc. This information is needed to understand how realistic it is for students to adapt to instructors’ plans for delivery and to work with instructors to adjust them according to student capacity to participate in distance learning [57,58].
• Ensure the reliability of the selected technological delivery systems, the provision of and access to learner support systems, support for faculty professional development for online teaching pedagogies and tools, policy and governance issues related to distance program development, and quality assurance [12].
• Identify weaknesses in infrastructure including power, broadband, and equipment that needs to be strengthened when possible or workarounds when it is not possible (e.g., providing access to hotspots, affordable devices such as tablets, computers) [53,58].
• Implement a blended approach to reinforce a feeling of community belonging. Increasing interaction (student–content, student–student, and student–teacher) increases the learning outcomes when meaningfully integrated. According to experts, students need face-to-face interactions, so face-to-face lessons should complement online classes [53,59].
• Provide the opportunity for teachers to develop blended teaching competencies to prepare them to teach in different formats, settings, and situations and support their ongoing learning and growth related to teaching with technology [33].
• Incorporate a human-centered design approach into teacher education programs based on three premises: (a) building empathy, (b) engaging in pedagogical problem solving, and (c) establishing an online community of inquiry [60,61].
• Develop more inclusive tools, platforms, and devices to make digital learning resources accessible to people with disabilities [53].
• Consider dedicated (financial, logistical, and pedagogical) support programs for at-risk students [58].

Despite the presented challenges, the COVID-19 pandemic has fostered innovative solutions through digital transformation in various economic sectors, enriched campus-based programs with online activities, and developed learning and development opportunities for the university community. Students highlight gains on a personal level, such as greater self-discipline, better time management, responsibility, resilience, autonomy, and flexibility [62,63]. Teachers developed various new digital competencies and invested significant effort into building their students’ digital capabilities [64]. This effort has led to positive student attitudes and willingness to incorporate more online aspects into post-pandemic face-to-face learning [65]. For example, Whittle et al. [13] state that in an emergency remote teaching (ERT) environment, the educator must revisit and reevaluate their learning design frequently, both during and following the emergency situation, to determine the efficacy of the current approach and identify necessary adjustments as soon as possible. As variables such as technology access or standardized learning goals change, teachers must evaluate their current approach to determine what elements remain viable in the changing learning environment. As indicated by García-Morales et al. [27], the university system must strive to overcome this situation in order to be competitive and provide high-quality education in a scenario of digital transformation, disruptive technological innovations, and accelerated change. In emergencies and more planned contexts, the potential need for remote teaching must become part of a teacher’s skill set [66].

Finally, our newest contribution, the development of an assessment framework using composite indicators to evaluate the vulnerability of student groups concerning the identified challenges, may be helpful to plan interventions for emergency remote teaching environments beyond the specific context of this study. The authors intend to use these results as a reference for educators, universities, governments, and politicians to be more prepared for future crises and disasters.