E-Learning Critical Success Factors during the COVID-19 Pandemic: A Comprehensive Analysis of E-Learning Managerial Perspectives

: During the COVID-19 pandemic, educational institutions were shut down all over the world, which impacted over 60% of students and caused a massive disruption of the education system. The goal of this paper was to identify the critical success factors for E-learning during COVID-19 using the multi-criteria Analytic Hierarchy Process (AHP) and Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) techniques to enhance the educational process. Data were generated by interviewing 69 E-learning managers in educational institutions during COVID-19 based on deﬁned evaluation criteria and E-learning approaches through several channels. We found that technology management, support from management, increased student awareness to use E-learning systems, and demanding a high level of information technology from instructors, students, and universities were the most inﬂuential factors for E-learning during COVID-19. Among the ﬁve learning systems, blended learning was the most suitable learning system to practice. These results demonstrated that, regardless of how extraordinary the technology is in an educational institution, the readiness of E-learning execution played a large role in boosting the educational process during the COVID-19 pandemic.


Introduction
Prior to the COVID-19 pandemic, E-learning was growing approximately 15.4% yearly in educational institutions around the world without uncertainties or pressure on those institutions or on students [1]. However, as this research was conducted during COVID-19, the situation has changed dramatically. Educational institutions began providing most of their services online, including lecturers and different assessments via several platforms for over 60% of students around the world due to global restriction measures to minimize the spread of COVID-19 [2]. Referring to the data released by the World Health Organization (WHO), COVID-19 has been reported in over 216 countries, and there are areas with millions of confirmed cases [3]. Many countries have taken precautionary measures, including lockdowns of schools and universities, and switching to full E-learning mode during the spread of the Coronavirus, to avoid future expected waves [4]. This action was in response to social distance rules, which were strongly recommended by the WHO to prevent the spread of COVID-19 [5]. This lockdown began in the middle of the spring semester, which was unplanned for both instructors and students.
However, many studies have previously examined the critical success factors (CSFs) in the education sector from both the instructor's and students' perspectives for future improvement in the In 2011, the government of Saudi Arabia established the Saudi Electronic University (SEU), with tens of thousands of students enrolled in its different programs, including undergraduate and graduate studies. Since the occurrence of these events, King Abdulaziz University implemented many technological tools in order to enhance their practice in the E-learning system, such as the Learning Management System (LMS), which supports fresh and junior students by providing access to over 16,000 e-books as well as other academic resources online [12]. All these efforts for E-learning impacted the education transformation from on-campus to distance learning during the COVID-19 pandemic, which happened suddenly without enough time to plan.

TOPSIS and E-learning
In 1981, Hwang and Yoon developed the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) method [13]. This method is commonly known for the Multiple Criteria Decision-Making practice to determine the best alternative among a set of alternatives [14]. This method essentially determines the distance of both the positive and the negative alternatives of the ideal solution [15]. The TOPSIS technique is widely used in many industries, including education; specifically, E-learning is becoming more popular and is growing over time.
For instance, researchers in [16] evaluated the criteria of five different approaches of E-learning, where each approach's performance was rated and computed using the TOPSIS method. They concluded that the Flipped Classroom, a student-centered approach with online material provided to student's prior to classes, was suggested as the most convenient E-learning approach, whereas 'strategic readiness for E-learning implementation' was ranked as essential criteria. Additionally, in [17], the authors evaluated and selected the learning objectives of e-content and educational material in web-based learning systems. They combined both the Analytic Hierarchy Process (AHP) and TOPSIS methods to ease the selection process. They suggested that the learning outcomes metadata is an effective procedure to produce e-content.
Eight criteria were evaluated using the Preference Ranking Organization Method for Enrichment Evaluation (PROMETHEE) and the TOPSIS method to examine the E-learning readiness as well as to weight the criteria based on Simple Additive Weighting (SAW) and AHP to determine the weaknesses and improve the implementation of E-learning [18]. Indeed, TOPSIS has been commonly used to evaluate different criteria and factors in E-learning as well as different educational system approaches. Therefore, conducting this technique will help to find the most CSFs for E-learning during COVID-19.

Analytic Hierarchy Process (AHP) and E-learning
In 1980, Saaty [19] developed the Analytic Hierarchy Process (AHP). This is an effective decision-making process where quantitative and qualitative features are considered [16]. When the decision relies on several criteria, the AHP method is one of the most effective techniques to use [16]. Although the method has been criticized over its process, almost all fields, including resource allocation, management, and education, have used this technique for making important and responsible strategic decisions [20]. For example, Dweiri [20] researchers investigated and prioritized the critical factors of violations of academic integrity in Saudi Arabia within educational institutions. They stated that twelve essential factors of E-learning were prioritized using an analytic hierarchy process. They concluded that the most critical factor was the inappropriate guidelines provided to students, whereas a shortage of feedback was the least critical factor amongst all factors.
In [21], the authors evaluated the CSFs of E-learning from both the instructor and student perspectives at Sebelas Maret University. They stated that they used AHP and fuzzy techniques to determine the ranking of the CSFs from both perspectives. They concluded that the five CSFs from the instructor's point of view were fiscal policy, regulatory policy, course quality, relevant content, and technical support. Whereas course quality, relevant content, completeness of the content, attitudes toward students, and flexibility in taking courses were the five critical success factors from the students' perspective. However, the focus in this paper was on E-learning during the COVID-19 pandemic, which is an abnormal situation. Thus, we implemented the AHP method in order to calculate the weight of the different criteria.

Critical Success Factors and Type of E-Learning System
The critical success factors are referred to as "characteristics, conditions, or variables that, when properly sustained, maintained, or managed, can have a significant impact on the success of a firm competing in a particular industry" [22]. By finding the CSFs, stakeholders can boost these factors for better outcomes. The types of E-learning systems are identified in Table 1, to determine the most appropriate type of system associated with AHP and the TOPSIS technique during COVID-19. The factors that were evaluated in this paper are identified and defined in Table 2. Figure 1 summarizes the critical success factor hierarchy problem discussed in this paper based on the multiple-criteria decision analysis problem representation. Those criteria and alternatives were identified from previous studies on E-leaning and were categorized to represent the majority of the criteria that were evaluated in other studies under different conditions.

Factors Prior Research Definition
Student Characteristics [25,[29][30][31] This factor focuses on the student's environment while learning. It includes the student's pace of learning, commitment, attitude, motivation, knowledge of computer systems, and demographics.
Instructor Characteristics [8,25,[29][30][31] This factor focuses on the instructor's environments while teaching. It includes the instructor's attitude, flexibility, knowledge of learning technology, teaching style, and efficacy in student motivation.
Learning Environment [8,25,29,31] This factor focuses on the learning environment and facilities that are provided for both students and instructors. It includes a learning management system, technical infrastructure, interactive learning, and access and navigation.
Instructional Design [8,25,29,30] This factor focuses on the instructional system to meet the objectives of the institution. It includes the content quality, objective clarity, learning strategies, and learning psychology. Support [8,25,[29][30][31] This factor focuses on supporting both the instructors and students to enhance their experience. It includes communication tools, help disk availability, and training.
Information Technology [21,25,26,29,30,32] This factor focuses on the information technology system to deliver learning materials and objectives. It includes ease of use, reliability, efficiency, privacy, and information.

Factors Prior Research Definition
Technology Knowledge [8,25,[29][30][31] This factor focuses on the knowledge of using technology for both instructors and students. It includes the use of computers, the use of software, and communication interaction.
Level of Collaboration [8,21,30,31] This factor focuses on the collaboration level between faculty members. It includes the lack of social interaction, project team supervision, and managerial support.
Knowledge Management [8,21,30,31] This factor focuses on the management knowledge within the educational institution for faculty members and administration. It includes the management team, managing delivery and maintenance, time management, thinking strategies, and implementation expertise.
Educ. Sci. 2020, 10, x FOR PEER REVIEW 4 of 18 were fiscal policy, regulatory policy, course quality, relevant content, and technical support. Whereas course quality, relevant content, completeness of the content, attitudes toward students, and flexibility in taking courses were the five critical success factors from the students' perspective. However, the focus in this paper was on E-learning during the COVID-19 pandemic, which is an abnormal situation. Thus, we implemented the AHP method in order to calculate the weight of the different criteria.

Critical Success Factors and Type of E-Learning System
The critical success factors are referred to as "characteristics, conditions, or variables that, when properly sustained, maintained, or managed, can have a significant impact on the success of a firm competing in a particular industry" [22]. By finding the CSFs, stakeholders can boost these factors for better outcomes. The types of E-learning systems are identified in Table 1, to determine the most appropriate type of system associated with AHP and the TOPSIS technique during COVID-19. The factors that were evaluated in this paper are identified and defined in Table 2. Figure 1 summarizes the critical success factor hierarchy problem discussed in this paper based on the multiple-criteria decision analysis problem representation. Those criteria and alternatives were identified from previous studies on E-leaning and were categorized to represent the majority of the criteria that were evaluated in other studies under different conditions.

Materials and Methods
The methodology in this research consists of three parts; a survey, the AHP method, and the TOPSIS method. Further details will be provided in the next subsections.

Data Collection
Firstly, the research team visited 69 educational institutions and planned several meetings with the distance learning managers to create an appropriate picture of the problem. Table 3 summarizes the E-learning management staff demographic data in terms of age, gender, nationality, academic degree, and job title. This shows that most participants (90%) had a PhD, and that, with respect to job title, the biggest group were Associate Professors (45%). Furthermore, the majority were Saudi nationals (71%). The managers evaluated the criteria associated with each E-learning system. During the first meeting, the discussion of the decision criteria were as follows: Instructor Characteristics, Student Characteristics, Information Technology, Support, Technology Knowledge, Course, Instructional Design, E-Learning Environment, Level of Collaboration, and Knowledge Management. The decision alternatives were as follows: Blended Learning, Flipped Classroom, Information and Communication Technology (ICT) Supported, Face-to-Face Learning, Synchronous Learning, and Asynchronous Learning. The managers of E-learning rated each criterion associated with each e-learning system toward the CSFs of E-learning in the COVID-19 pandemic, as shown in Table 4.

The Analytic Hierarchy Process (AHP)
To apply the AHP method, the following steps were performed based on [19,33] Step 1: Weight each criterion and decision alternatives. The ratings are given in Table 5. For instant, if the E-learning managers determined that, in the E-learning process and based on the criteria and the rankings shown in Table 5, the Instructor Characteristics were very strongly preferable to the Student Characteristics (in other words, the E-learning Instructor Characteristics had a greater influence on the learning process compared with the Students Characteristics), they will give rank 7 to the Instructor Characteristics in the pair-wise comparison matrix when comparing the Instructor Characteristics versus Students Characteristics. On the other hand, the intermediate ratings (2, 4, 6, and 8) were also used. The reciprocal ratings (1/9, 1/8, etc.) were used in cases where a second alternative was chosen over the first, assigning a rating in the case of comparing an alternate with itself, as shown in Table 6. For example, Student Characteristics pair-wise compared to Instructor Characteristics at 1/7, which indicates that the Student Characteristics were seven times more preferable to Instructor Characteristics from the managerial perspective.
Step 2: Develop the pair-wise comparison matrix and rate the relative importance among every pair of decision alternatives, as shown in Table 6. The alternatives are listed in the matrix horizontally (first alternative) and vertically (second alternative) as associated with the numerical ratings.
Step 3: Set up the normalized matrix by dividing all numbers in the pair-wise comparison matrix column by the sum of its column, as shown in Table 7.
Step 4: Determine the average priority vector for all the rows in the normalized matrix. Then use these averages to create the priority vector of all alternative preferences associated with the criterion where the sum of this vector is 1, as shown in Table 8.
Step 5: Calculate the consistency ratio and use it to measure the subjective input. A ratio of less than 0.1 was considered good. For the ratios greater than 0.1, the subjective input was considered for re-evaluation.
Step 6: Develop the priority matrix to obtain the priority vectors for each criterion using the results of Step 4.
Step 7: Develop the criterion pair-wise development matrix following the one used to create the alternative pair-wise comparison matrices utilizing the ratings of Step 2, normalize the matrix following Step 3, and develop the priority vector criterion in Step 4.
Step 8: Develop the overall priority vector through multiplying the criteria priority vector in Step 7 with the priority matrix in Step 6.
Microsoft Excel was used to perform all these steps. Table 5. Analytic Hierarchy Process (AHP) ratings.

Linguistic Rating Numerical Rating
Extremely preferred 9 Very strongly preferred 7 Strongly preferred 5 Moderately preferred 3 Equally preferred 1

The Technique for Order Preference by Similarity to Ideal Solution (TOPSIS)
To apply the TOPSIS method, the following steps were performed, based on [32,34] Step 1: Form the matrix expressed as follows: A m X m1 X m2 · · · · · · X mn where: Ai = ith alternative project and Xij = the numerical outcome of the ith alternative project with respect to the jth criterion. Table 9 shows the structure of the TOPSIS matrix.
Step 2: Normalize the decision matrix (D) by applying the following formula: (2) Table 10 shows the TOPSIS normalize matrix.
Step 3: Construct the weighted normalized decision matrix that is shown in Table 11 by multiplying the normalized decision matrix by its relative weights. The following formula is used to calculate the weighted normalized value vij: Step 4: Define both the positive and negative ideal solutions: A − = min v ij jieJ , max v ij jieJ ∀ J = 1, 2, 3, . . . , n where: J is associated with the benefit criteria and J is associated with the cost criteria. Table 12 shows the TOPSIS positive and negative ideal solutions.
Step 5: Calculate the separation of all alternatives from the positive ideal as follows: Step 6: Calculate the relative distance of Ai with respect to A*, the ideal solution, which is defined as: where performance alternatives become better with larger values of Ci*.
Step 7: Rank the order of preference. All steps were performed using Microsoft Excel.

Results and Discussion
The data were collected by interviewing 69 E-learning managers. The response rate for both the first and second rounds of interviews was 100%. Every factor's weight was calculated using AHP software and Microsoft Excel 2013. The inconsistency ratio was calculated to achieve consistent weights among all participants. Table 8 shows that Knowledge Management (0.347996), Support (0.169233), Student Characteristics (0.111681), and Information Technology (0.106554) were the most critical success factors that influenced the E-learning process during the COVID-19 pandemic.
The primary purpose of the TOPSIS technique is to discover the best alternative that should have the shortest distance, which is the Euclidean distance, from the ideal solution. The data for the E-learning system alternatives based on their criteria are provided in Table 9, which shows the calculation of the square root of the squared summation for the given criteria for each E-learning system. The value in every cell was divided by the root summation of the square value, which gives the normalized decision matrix, as shown in Table 10. Table 11 shows the weighted normalized decision matrix, along with the ideal best and the ideal worst values. Table 12 shows the Euclidean distance from the value of both the ideal best and worst along with the performance score for every alternative using the TOPSIS method.
The research team used Microsoft Excel 2013 software to analyze and employ both AHP and TOPSIS techniques. The demonstration of the steps and analysis was provided in Section 3. As the result of the analytic hierarchy process for the E-learning critical success factors during the COVID-19 pandemic, Table 12 shows that Blended Learning appeared to be the best decision alternative for educational institutions to consider when selecting an E-learning system during the COVID-19 pandemic, with a total weight of 0.811. This was followed by Asynchronous Learning, which was considered to be the second-best alternative, with a total weight of 0.564, and then by the ICT Supported Face-to-Face Learning with a total weight of 0.563. This was then followed by the Flipped Classroom with a total weight of 0.483 and, finally, Synchronous Learning with a total weight of 0.066.
The findings revealed that the course type and contents did not have such a great impact on learning outcomes as previous studies showed, because it is a managerial perspective and all the courses are quoted with the same weight/importance [35]. A focus on providing the users (students and instructors) with more training in knowledge management would be wise [36]. Educational institutional support is very important in the success of E-learning. This is compatible with [37], which stated that providing computer and training support to students positively impacted the students' use of the learning system. Our study found that student characteristics played a large role in educational systems during COVID-19.
The findings also indicated that students must understand their role during the social distancing measures, build their own attitude and commitment, and find ways to self-motivate in order to gain successful learning outcomes. The findings for student characteristics are consistent with a study from [26,38].
The primary objective of this paper was to identify and prioritize the critical success factors of E-learning system adoption during the COVID-19 pandemic. We employed AHP and TOPSIS, and this study may assist educational institutions in gaining a better understanding of the critical success factors for E-learning adoption during the COVID-19 pandemic. Focusing on crises, such as the COVID-19 pandemic, the current study is relevant as no previous research has discussed issues where the entire educational system worldwide was affected with many interruptions. Many educational institutions shifted from in-class education toward E-learning. E-learning system adoption is not an easy process, nor can one system fit all different types of disciplines and institutions around the globe. This research tackles the different possible systems and their critical attainment factors.

Limitations
This research tackled the different possible systems and their critical attainment factors. There are many studies regarding general education and E-learning; however, there was a lack of literature that included the effect of pandemics that would reflect the same situation that the world is facing during COVID-19. One of the drawbacks of the multi-criteria decision analysis tools is that they are very sensitive to the perspective the study is focused on. Therefore, the same applied tools would provide different results and findings based on whether the issue was tackled from the student perspective or the instructor perspective. Finally, this study took place in Saudi Arabia, and might not apply to other countries due to differences in perspective and regulations.

Conclusions
The educational process worldwide has been interrupted due to the COVID-19 pandemic. E-learning is becoming much more necessary, and is very important in education. Educational institutions during COVID-19 face the unique challenges of smoothly maintaining the process of learning while ensuring that it is still beneficial. Therefore, these institutions must understand what drives instructors and learners toward the E-learning system. The main focus of this study was to classify and prioritize E-learning systems during the COVID-19 pandemic as well as to recognize practical implications.
This study prioritized different systems of E-learning using multi-criteria approaches. We discovered that the most significant factors influencing E-learning success during the COVID-19 pandemic were related to technology knowledge management, support from management, increased student awareness of utilizing E-learning systems, and demanding a high level of information technology from the instructors, students, and universities. This finding should be seriously considered as no matter how great the technology is, readiness for E-learning implementation still plays the leading role in improving the educational process. Blended Learning was the most preferred E-learning system out of the five methods discussed in this study. The results of this study provide useful information to the E-learning managers of universities in their process of implementing modern technologies in education.