Next Article in Journal
Prospective Analysis of Levels and Correlates of Physical Activity during COVID-19 Pandemic and Imposed Rules of Social Distancing; Gender Specific Study among Adolescents from Southern Croatia
Next Article in Special Issue
A Linked Open Data-Oriented Sustainable System for Transparency and Open Access to Government Data: A Case Study of the Public’s Response to Women’s Driving in Saudi Arabia
Previous Article in Journal
The Bearing Stiffness Effect on In-Wheel Motors
Open AccessArticle

A Hierarchical Model to Evaluate the Quality of Web-Based E-Learning Systems

1
Department of Computer Sciences, Bahria University Lahore Campus, Punjab 54600, Pakistan
2
Department of Software Engineering, University of Gujrat, Punjab 50700, Pakistan
3
College of Computer and Information Sciences (CCIS), King Saud University, Riyadh 11653, Saudi Arabia
4
Department of Computers and Systems Engineering, Faculty of Engineering at Helwan, Helwan University, Cairo 11795, Egypt
5
Center of Excellence in Information Assurance (CoEIA), King Saud University, Riyadh 12372, Saudi Arabia
6
Department of Software Engineering, National University of Modern Languages, Islamabad 44000, Pakistan
*
Author to whom correspondence should be addressed.
Sustainability 2020, 12(10), 4071; https://doi.org/10.3390/su12104071
Received: 18 April 2020 / Revised: 13 May 2020 / Accepted: 13 May 2020 / Published: 15 May 2020
(This article belongs to the Special Issue Sustainable Information Systems)

Abstract

The rapid growth of Information and Communication Technologies (ICT)—specifically, the Internet—has given emergence to e-learning. Resultantly, web-based e-learning systems are being increasingly developed to enhance the learning process. However, the utilization of such systems is low, mainly owing to poor quality content and overall design problems. To improve usage, it is imperative to identify the factors with the most significant impact on the quality of these systems so that the e-learning industry keeps these factors in consideration while developing e-learning systems. This study focused on the identification and prioritization of factors related to the design quality of e-learning systems through a hierarchical quality model. Thus, firstly, an extensive literature review was conducted to identify the factors that most affect the quality of web-based e-learning systems. Secondly, among the identified factors, only those with the most significant effect were considered. To identify the most important quality criteria, a survey was conducted. An instrument was deployed among 157 subjects, including e-learning designers, developers, students, teachers, and educational administrators. Finally, a second instrument was distributed among 51 participants to make a pairwise comparison among the criteria and rank them according to their relative importance. The identified and prioritized factors were classified into four main categories. Among these four factors, content was identified as the most important factor, whereas design was found to be the least important factor.
Keywords: analytical hierarchical process (AHP); content; e-learning; quality; usability analytical hierarchical process (AHP); content; e-learning; quality; usability

1. Introduction

As the recent coronavirus outbreak prompted universities to start shifting classes online either for a few weeks or for the remainder of the spring semester of 2020, e-learning and remote education have popped up as the magical alternative for in-person classes in the time of the COVID-19 pandemic. However, e-learning was not originally launched as a result of the outbreak of coronavirus; in fact, it has been used as a solution for numerous problems in education systems for a while, especially in developing countries. The most important among these problems is the shortage of qualified and experienced faculty, which constricts the learning opportunities in not only rural areas, but also urban [1]. E-learning systems are growing tremendously as an alternative means to offer educational services at self-pace, anywhere and anytime. Ozkan and Koseler define e-learning as an electronic mechanism used to deliver learning material to learners [2]. According to the IEEE Technology Standard Committee, e-learning systems are learning systems that use web browsers to impart an online learning experience. Another definition is that e-learning is a combination of a computer, browser, and internet to provide online education and training [3,4].
There are several e-learning means to deliver education, such as CDs, online chats, forums, video conferencing, and webcasting. At present, e-learning websites are considered a common source of delivering subject-related material. The learners interact with these websites in order to perform learning tasks efficiently, effectively, and satisfactorily. However, quality continues to be a serious issue in these websites, since most of them are insufficiently effective. Therefore, learners find it difficult to perform their tasks using most of these websites [4]. To impart the satisfactory learning experience, the websites should be effective, efficient, and usable so that extraneous cognitive load can be avoided. Thus, the quality of e-learning websites is an important aspect to meet the program learning objectives (PLO), and students heavily rely on high-quality e-learning websites to attain the course learning outcomes (CLO).
Consequently, educational institutes around the globe are switching towards an electronic mode of education in order to attract more and more international students. Besides developed countries, the developing countries are also ambitious to adopt the concept of e-learning; Singapore and Malaysia have completely revamped their educational system to integrate information and communication technologies (ICT) in their learning environments [5,6]. For the maximum utilization of e-learning websites to attain the PLO and CLO, quality is a mandatory aspect. Previous research has shown that the quality of e-learning resources influences e-learning students’ satisfaction, retention, and loyalty [7]. Developing countries are striving for the implementation of e-learning resources with their limited budget to contend with educational problems, such as a lack of qualified teachers, issues of access, and poor learning outcomes [8]. Web-based e-learning systems have successfully been implemented in many places; however, many of them have failed to realize the objectives, motives, and expectations behind their development. There could be many reasons for such failure, but the most obvious is the poor quality of e-learning resources [9]. Hence, the availability of guidelines for the design and development of high-quality web-based e-learning systems is crucial.
It is widely recognized that the quality of web-based e-learning systems is of utmost importance to achieve the learning objectives and to resolve chronic educational problems [10]. According to IEEE, the quality “is the degree to which a system, component, or process meets customer expectations” [11]. The design of e-learning websites should enable efficient and effective human–computer interactions, as well as reduce the user’s workload by taking advantage of the computer’s capabilities. For the best possible learning outcomes, designers need to consider some important user interface parameters while developing e-learning websites. The exposure of such design parameters requires evidence-based information rather than mere opinions, and this can be achieved through the sound mechanism of evaluation for e-learning websites.
There are two major dimensions of e-learning systems with respect to evaluation: the first is pedagogical and the second is related to the design and development aspects. Numerous frameworks and models exist that could be used to assure the quality of web-based e-learning. However, these models lack the guidance of the e-learning industry, which is, among other factors, the most important factor affecting the quality of e-learning websites. Secondly, such models are mainly feasible to evaluate the pedagogical aspects of e-learning systems rather than the design and development aspects [12]. In this study, we argue that an evaluation model is required to discover the factors related to the quality of the design and development of e-learning systems and to determine the relative importance of such factors in order to achieve a high-quality e-learning product. The outcome of this work would help designers, developers, and software project managers in making decisions regarding the parameters they should prioritize while producing a high-quality e-learning product. How the quality of web-based e-learning systems could be improved is an open research question that should be addressed forthwith. This study raises two important questions: (i) which factors influence the quality of e-learning through websites? (ii) How will these factors be ranked?
This study aimed particularly at (i) the identification of the key factors (i.e., the most important) affecting the quality of e-learning websites; (ii) the prioritization of the identified factors by defining their relative importance; (iii) the proposed hierarchical quality model for the evaluation of e-learning websites. The methodology used to attain these goals was as follows: firstly, an extensive literature review was conducted to explore and identify the factors that mostly affect the quality of web-based e-learning systems. Secondly, among the identified factors, only those with the most significant impact were considered in this study. To identify the most important quality criteria, an electronic and manual survey was conducted, and an instrument was deployed among 157 subjects, including e-learning designers, developers, students, teachers, and educational administrators. Finally, another instrument was distributed among 51 participants to make a pairwise comparison among the criteria and rank them according to their relative importance. The identified and prioritized key quality factors were classified into four categories, including content, design, usability, and organization. Among these four factors, content was identified as the most important factor, whereas design was found to be the least important factor.
The rest of this paper is organized as follows: Section 2 presents related work and Section 3 describes research methodology. In Section 4, we present and discuss the findings of this study and in Section 5 we propose the quality evaluation model based on these findings. Finally, in Section 6, we give our conclusions and future work.

2. Related Work

Researchers have introduced many different evaluation models and frameworks [13,14]. However, these models lack the evaluation of websites with respect to software design and development perspectives. The e-learning quality models and frameworks introduced by prior research are as follows:
Jabr and Al-omari proposed a web service-based e-learning framework that increased the efficiency and effectiveness of collaborative learning in terms of reusability, interoperability, accessibility, and modularization [15]. The model does not evaluate the e-learning websites either from pedagogical or developmental perspectives. Amin and Salih proposed a model premised on critical success factors (CSFs) from a university student’s perspective. They surveyed e-learning CSFs and classified them into four dimensions. The model aimed to achieve high quality in web-based e-learning systems [16]. It considered only the single fraction of stakeholders to propose the dimension for the attainment of high quality in e-learning solutions. In our opinion, the model partially fulfils the quality needs of e-learning websites, as the findings are only based on the views of students.
Wang Shee and Wang proposed a multi-criteria evaluation method for web-based e-learning systems, considering only a single aspect of e-learning, which was learner’s satisfaction. The main dimensions of the model were learner interface, learning community, system, and personalization [17]. There are also other important aspects of e-learning websites, such as effectiveness and efficiency, which are not catered to by the proposed model. Chua and Dyson introduced the ISO 9126 quality model as a tool to evaluate e-learning systems in order to improve the quality [18]. Djouab and Bari introduced an extension of the ISO 9126 software quality model. However, the extended model was neither validated nor were the guidelines related to the usage given [19].
Researchers have also proposed quality evaluation models specifically for websites. The website quality evaluation model (Web-QEM) was designed to evaluate the quality of web applications, which helps to meet the quality requirements, as well as discover the missing and poorly designed features. The Web-QEM model is based on the ISO 9126-model, so its quality characteristics include usability, reliability, efficiency, and functionality. This model evaluates the quality of a website, taking many different steps. It is an effective model to measure the quality of websites. However, Web-QEM does not address all the factors related to the software dimension of modern academic websites, such as learning experience [20].
Vida and Jons proposed the Web Quality Model (WQM) to evaluate web applications with respect to three different dimensions, including web features, ISO/IEC 9126-1-based quality characteristics, and lifecycle process. The model lacked in presenting a step by step evaluation criteria and sub-characteristics of the factors. The SERVQUAL model was proposed to assess the satisfaction and quality of websites. The model intended to measure the gap between the expectation of the customers and their usage experience. The model addressed the five quality attributes called RATER: i.e., reliability, assurance, tangibles, empathy, and responsiveness. However, it did not consider the sub-attributes of the RATER quality constructs [21]. Tsigereda proposed a model to evaluate the quality of academic websites. It evaluated websites from the perspective of students using four important quality factors, including content, reliability, efficiency, and functionality arranged in a hierarchal structure along with sub-quality factors. The downside of this model was that it did not take all important stakeholders into account, such as faculty members, educational administrators, and designers/developers. Moreover, the hierarchy model was not assigned weights or ranks to the quality factors [22].
Although a variety of e-learning quality models and frameworks have been proposed by researchers, certain limitations have been observed in these e-learning evaluation models. Firstly, some of these models have not considered the stakeholders of paramount importance with regard to e-learning websites. For example, the student’s perspective is an important aspect of the quality of e-learning websites. The e-learning websites should not be something that are merely imparted to a passive student. Instead, high quality e-learning websites should be developed through a process of co-production between the students and the learning system. The teacher’s perspective is also critical to ensure the quality of e-learning websites. It is the teacher who can offer suggestions regarding the teaching/learning method. Similarly, the educational experts can provide an appropriate learning strategy for the students being served through the educational website. Furthermore, the players from the e-learning industry, including designers and developers, can also make suggestions to evaluate the web-based e-learning systems [23]. Secondly, the majority of the quality models focused on pedagogical characteristics, including the learner, instructor, institution, social, and management, etc.. The software characteristics, such as design, usability, and digital contents have been ignored. Thirdly, some frameworks have addressed the limited number of software quality factors, including usability, portability, and reliability. The quality of e-learning websites from a software perspective cannot be gauged using such a limited number of quality factors. Finally, website evaluation is a multi-criteria decision making (MCDM) problem, as there are a number of different factors involved. MCDM methods are extensively applied in other areas, such as e-commerce [24,25], but, as per the author’s knowledge, they are rarely explored for the evaluation of web-based e-learning systems.
Keeping the shortcomings of previous research in mind, this study has considered the software perspective and key stakeholder of e-learning websites to propose a hierarchical quality evaluation model for e-learning websites. The domain of e-learning websites is very broad and there are numerous perspectives for the quality of e-learning websites, such as pedagogical, personal, institutional, software, and technical. The focus of this study is only on the software perspective.

3. Methodology

The methodology used in this research consists of three phases: in the first phase, a literature review was conducted to explore and identify the relevant factors that impact the quality of e-learning websites. As a result of this extensive literature review, the relevant quality factors were identified. In the second phase, a survey was conducted to determine the importance of various identified quality factors, and only those with a significant impact (key quality factors) were further considered in our study. Finally, in the third phase, the relative importance (weight) of each key quality factor was estimated using the most commonly used pairwise comparison method—that is, the Analytical Hierarchical Process (AHP). The output of this phase is a ranked list of quality factors for web-based e-learning systems, which was the basis of our proposed quality evaluation model. The graphical representation of the research methodology is depicted in Figure 1.
The details of the three phases of research methodology are discussed below.

3.1. Phase 1: Identify Relevant Quality Factors

To identify the factors that affect the quality of web-based e-learning systems, an extensive systematic literature review (SLR) [26] was conducted using different sources, including Google Scholar, Science Direct and IEEE Xplore. As a result, a broad range of quality factors (or dimensions) were identified; furthermore, the identified dimensions were classified into main dimensions and sub-dimensions. The main dimensions were content, design, organization, and usability, which are discussed in detail in Section 4.

3.2. Phase 2: Determine the Importance of Identified Quality Factors

As a result of phase 1, various relevant quality factors were identified. However, these factors did not have the same effect on e-learning websites: i.e., they were not of the same importance in terms of their impact on the quality of websites. Therefore, in the second phase, we attempted to determine the importance of the relevant quality factors, and only the factors whose importance exceeded a predetermined threshold (key factors) were further considered in our study. To determine the key quality factors, this study adopted the survey method, as the survey is an effective method to gather data from a wide range of the population. To collect data, a Likert scale questionnaire was developed, which consisted of a set of questions to evaluate the importance of the relevant quality factors identified in phase 1. The Likert scale questionnaire had five points, with the choices ranging from ‘very important’ to ‘not important’. The participants were asked to express their opinion and feelings by selecting an appropriate choice with respect to each element contributing to a certain quality dimension. To qualitatively analyze the collected data, IBM-SPSS (v.20) was used. The numeric values from 1 to 5 were assigned to each choice of the Likert scale, where 5 was assigned to ‘very important’ and 1 to ‘not important’.
The survey targeted four categories of respondents. The first category included web developers, particularly those who were involved in the development of learning applications. The second category of respondents were educational administrators, specifically those who had experience with e-learning websites or, at a minimum, were aware of the role of e-learning applications in delivering education. The third category was the academicians, which included course instructors and educators. The fourth category consisted of students. The last two categories involved those who had been using e-learning websites and were acquainted with a technology-infused learning environment. To choose the appropriate sample, a non-probability technique that is also known as snowball sampling was employed. The technique begins with the identification of a subject possessing desired characteristics. Afterward, the social network of identified subjects is utilized to select further participants [27]. This technique is also called referral sampling or chain sampling because the correctly identified sample helps researchers to further find more like-minded participants. In this sampling technique, the size of a sample increases gradually, similar to falling snow [28]. The students were conveniently available as compared to other subjects, which was why their number was high in the sample. However, a representation of all categories of subjects was ensured in the sample.
The sample selected contained 157 subjects to collect survey responses. The demographic profile of respondents was as follows: the respondents were comprised of both males and females. The male respondents were 124 (80.5%) and the females were 30 (19.5%). In terms of qualification, 22 (14.3%) respondents were undergraduates, 66 (42.9%) were graduates, 61 (39.6%) were postgraduates, and five (3.2%) had a doctorate degree. Amongst them, 100 (65%) were students, 20 (13%) were related to academia, 20 others (13%) were web developers, and 14 (9.1%) were related to educational administration, as shown in below Table 1.

3.3. Phase 3: Prioritize Key Quality Factors Using AHP Method

The core purpose of this phase was to determine the relative weights and ranking of the key quality factors identified in phase 2. The pairwise comparison method was adopted to estimate the weights of quality dimensions. A pairwise comparison generally is any process of comparing entities in pairs to judge which of each entity is preferred or has a greater amount of quantitative property, or whether or not the two entities are identical. It stems from the Analytic Hierarchy Process (AHP) [29], a famous multi-criteria decision-making framework which is used in the scientific study of preferences, voting systems, and requirements engineering. We applied the pairwise comparison approach [30] to assign relative weights for the key quality factors as follows:
Step 1:
The pairwise comparison matrix was completed. Two factors were evaluated at a time in terms of their relative importance. Index values from 1 to 9 were used. If factor Fi was exactly as important as factor Fj, this pair received an index of 1. If Fi was extremely more important than Fj, the index was 9. All gradations were possible in between, as shown in Table 2. For a “less important” relationship, the fractions 1/1 to 1/9 were available: if Fi was extremely less important than Fj, the rating was 1/9.
The index values were estimated based on experts’ consensus; for this purpose, a questionnaire was prepared and distributed among the participants who had at least three years of experience with using e-learning websites in universities. The participants were restricted with the amount of experience because only a certain amount of years of experience could enable them to make a decision about the relative significance of factors with respect to e-learning quality. Experts were asked to evaluate the importance of each key quality factor with respect to other quality factors and record their estimates, as shown in Table 2. If the estimates of different experts were different, a consensus method was used to reduce divergence. The final approved estimates were entered row by row into a cross-matrix C (n x n). First, the diagonal of C was filled by values of 1, as per Equation (1). Second, the right upper half of C was filled until each factor had been compared to every other one. If Fi to Fj was rated with the relative importance of m (i.e., C i j = m ) , Fj to Fi had to be rated with 1/m (i.e., C j i = 1 / m ) . Lastly, the lower left half of C was filled with the corresponding fractions, as per Equation (2). (Note that i and j are positive integers ≤ n, C i j is the element of C located in row i and column j).
C i j = 1 ,   i = j
C i j = 1 C j i   ,   i j
Step 2:
the normalized comparison matrix was calculated. A normalized comparison matrix C was created by dividing each element in matrix C by the sum of the elements in its column. This is shown in Equation (3).
C i j = C i j / i = 1 n C i j
Step 3:
the relative weights of the factors were calculated. To obtain the weight w i of each individual factor Fi, the mean of each row in C was calculated, as shown in Equation (4).
w i = 1 n j = 1 n C i j
These weights were already normalized; their sum was 1, as illustrated in Equations (5) and (6).
0 w i 1
i = 1 n w i = 1
Step 4:
the consistency of the results of the pairwise comparisons were checked. According to Saaty, if the consistency ratio was less than 10%, then it was acceptable; otherwise, the pairwise comparisons should be revised [31]. The ratio of consistency (CR) is given in Equation (7).
C R = C I R I
where CI is consistency index and is given in Equation (8).
C I = λ m a x n n 1
λmax is the maximum eigenvalue and n is the rank of the pairwise comparison matrix [28].
RI specifies the random index of consistency, which had a different value based on the number of criteria, as shown in Table 3.
A variety of software is available in the market, such as Expert Choice, HIPRE3, Criterium, and AUTOMAN, which can be used to incorporate AHP. However, the Expert Choice software is considered as the standard AHP software [11], which is why it was used in this research. In this study, AHP was implemented using the following steps.
(a)
The research objective was defined. The objective of this study was to evaluate and rank the key quality factors of e-learning websites.
(b)
The objective and the evaluation criteria were organized in a hierarchical structure. The first level (level 1) of hierarchy describes the objective, the second level (level 2) describes the main quality criteria, and the third level (level 3) gives the sub-criteria related to each main criterion. This is shown in Figure 2.
(c)
The relative weights for criteria and sub-criteria were computed following the steps from 1 to 4, as shown above. A pairwise comparison required n (n—1)/2 comparisons [31], where ‘n’ represented the number of criteria or sub-criteria.

4. Findings and Discussion

The findings of the three phases discussed above are discussed below.

4.1. Findings of Phase 1

The SLR of phase 1 resulted in the extraction of a wide range of pertinent quality dimensions, which could play an important role in assessing the quality of e-learning websites. These dimensions were extensively studied and classified into four main quality dimensions, including content, design, organization, and usability. Table 4 shows the main quality dimensions and the sub-dimensions of each main dimension.

4.2. Findings of Phase 2

In phase 2, a survey was conducted to evaluate the importance of each quality sub-dimension on a 5-point Likert scale. The survey questionnaire was shared with 157 participants (see Table 1) who were personally approached (via telephone conversation and face-to-face meetings when possible). The objectives of the study and the value of their responses in shaping future web-based e-learning systems were illustrated. In addition to the Google form of the survey, a separate, easy to read and understand tutorial was shared with them. The tutorial provided a clear and concise description of each sub-dimension so that the participants apprehended it well before responding. All the participants responded, and only three of the 157 surveys were found to be incomplete and vague; thus, they were excluded from the data analysis. A total of 22 sub-dimensions were evaluated, and the data obtained through the questionnaire were analyzed using the statistical software package (SPSS). The results are shown in Table 5.
As the target of MCDM methods, adopted one of them in this study (i.e. AHP), is to compare limited factors with each other. While working in MCDM, detailed literature review produces a large list of criteria and sub criteria which needs to be reduced because it further has to be presented to respondents. So, there is a need of cutoff point to eliminate some less important factors after conducting survey in the second phase and before applying the pairwise comparison of AHP in the last phase. The less important factors are eliminated using some threshold value/cutoff that is selected on the basis of experts and researchers choice. So, after reviewing all the responses and consulting with experts in the area, mean value 4 was considered most appropriate as threshold value, as if we set less than 4, very few factors were supposed to eliminated and if set it greater than 4 then most of the important factors were expected to be eliminated.
The results in Table 5, shows that ‘Timely’ quality sub-dimension has been perceived very important by most of the respondents 58.4% and 37% respectively, only one participant considered it unimportant with respect to content. The mean value of ‘Timely’ is also highest (4.51). Among the sub-dimensions of content, ‘Multilingual’ has been found the least important one with lowest mean importance value 3.56. Similarly, with respect to ‘Design’ dimension, the ‘Attractive’ sub-dimension has been professed very important by most of the respondents, 48.7% and 36.4% correspondingly. Only few participants stated that it is less important (3.9%) and unimportant (4.5%). The mean importance value of ‘Attractive’ sub-dimension is the highest (4.21). Among sub-dimensions of ‘Design’, ‘Appropriateness’ has been found the least important with lowest mean 3.64. With regard to ‘Organization’ dimension, the ‘Index’ quality sub-dimension was observed very important by 47.4% and important by 36.4% participants, only one participant (0.6%) reflected that it is unimportant in terms. The mean value of ‘Index’ has been also found the highest (4.23). Among sub-dimensions of ‘Organization’, ‘Consistency’ was the least important quality sub-dimension with lowest mean 3.58. In terms of ‘Usability’, ‘User friendliness’ has been identified very important sub-dimension by majority of the respondents (69%) with highest mean importance value of 4.59. ‘Availability’ has been recognized as the least important quality sub-dimension with lowest mean importance value of 3.96. Only those sub-dimensions whose mean importance value equals or exceeds 4 are identified as key factors (shown in green) and further considered in phase 3. Thus, sub-dimensions with mean importance value less than 4 (shown in red) are excluded.

4.3. Findings of Phase 3

In phase 3, AHP was applied to estimate the relative weights and to rank the key quality dimension (aka criteria) and sub-dimensions (aka sub-criteria). The AHP was implemented through Expert Choice software. The steps of implementation and the corresponding findings were discussed below.
(a)
Defining the goal: the first step in AHP was to define the goal or the objective of the study. The goal of this study was to propose a hierarchical quality model that evaluated the quality of e-learning websites.
(b)
Organizing the objective and the evaluation criteria in the hierarchical structure. Based on the findings of phase 1 and phase 2, shown in Table 4 and Table 5, the key quality factors comprised criteria and sub-criteria. The main quality criteria of e-learning websites are content, design, organization, and usability; these criteria comprised a total of 22 sub-criteria. Out of the 22 sub-criteria, only 15 were considered in phase 3, while seven with mean values less than 4 were excluded. The criteria excluded were multilingual, reliability of content, appropriateness, color, consistency, domain, and availability, as shown in Figure 3. The remaining 15 sub-criteria, along with their main criteria, were inserted in Expert Choice software to construct the hierarchical quality model, as shown in Figure 4.
(c)
Compute the relative weights of the key criteria and sub-criteria. In this step, pairwise comparisons have been performed as described in above steps from 1 to 4. Pairwise comparison process is the core of AHP which is a widely used technique in Multi-Criteria Decision Making (MCDM). The most critical step here is step 1 where experts have to make consensus on the index values entered to the pairwise comparison matrices. Experts were asked to fill in a pairwise comparison matrix that compares the importance of the four main criteria with respect to each other according to the scale shown in Table 2. Similarly, experts have to complete pairwise comparison matrices that compare the importance of sub-criteria of each main criterion to each other. For this purpose, a questionnaire has been prepared and distributed among participants who have at least three years of experience on using e-learning websites. The questionnaire has been distributed among 51 participants including students, academicians, web developers and educational administrators. All participants have responded to the questionnaire and their estimate entered to Expert Choice software as shown in below figures. The below figures show different screenshots of Expert Choice software which represent the pairwise comparison indices assigned by the experts when comparing different criteria against each other. Figure 5 present the relative importance of Content with respect to Usability as shown by the weights assigned by participants to these factors. Figure 6 showed the Content with respect to Organization along with weights given by participants to both factors. Figure 7 illustrates Content with respect to Design and Figure 8 showed the relative importance of Organization as compared to Design. In the same way, the relative importance of the Usability vs. Design and Organization vs. Design has been assessed through Expert choice. The consistency test [35] has also been performed in Expert Choice software. The survey responses which have consistency more than 10% were excluded. As a result, 30 responses were excluded out of 51. The remaining 21 responses were used to compute the final weights.
In order to complete all steps of the pairwise comparisons, the Expert Choice software was deployed to estimate the relative weights (priority) of the criteria and sub-criteria. These weights were the final relative importance value of each quality evaluation criteria and sub-criteria. The weights were categorized into “global weights”, which were the weights of the four main criteria, and “local weights”, which were the relative weights of sub-criteria. The global weights are shown on the right-hand side of the screenshot of Figure 5 and listed in Table 6. On the other hand, local weights are shown in Table 7, Table 8, Table 9 and Table 10. The local and global weights were computed in the Expert Choice software in accordance with Equations (3)–(5). The global weights included the weighted importance value for content, design, organization, and usability, and the local weights were relative importance values for the 15 sub-criteria.
As shown in Table 6, the weight of content was 0.407, which indicated that it was the most important criterion. The weight of the usability criteria was found to be 0.294, which represented the second most important criterion to evaluate the quality of e-learning websites. The overall weights and ranking of the main criteria are shown below.
The local weights of the sub-criteria of the four main criteria (i.e., content, design, usability, and organization) along with their ranks are shown in Table 7, Table 8, Table 9 and Table 10. Table 7 shows that timely was the most important sub-criteria in the content category, with the highest weight of 0.302, while relevant was at the lowest position and its weight was 0.188. Table 8 shows that user friendly was the most important criteria in usability, with the highest weight of 0.401, while reliability was the least important sub-criterion, with the lowest local weight of 0.206. As shown in Table 9, index was the most important sub-criterion of organization with the highest weight of 0.313, while navigation was observed to be the least important sub-criterion with the local weight of 0.149. Finally, Table 10 shows that attractive was the most important sub-criterion in design, with the highest weight of 0.389, while multimedia elements were found to be the least important sub-criterion, with the lowest local weight of 0.142. The above results are aggregated in Table 11.

5. Proposed Model for Quality Evaluation

The objective of this research study is to develop a hierarchical quality model that could be used to evaluate the quality of e-learning websites. Thus, the main outcome this research is a hierarchal model to evaluate the quality of e-learning websites, this model is shown in Figure 9.
‘Content’ is found as most significant quality factor, the basic objective of e-learning systems is to facilitate and help learners by imparting them with apposite learning content and services to meet their learning needs [60]. The content factor is related to the learning material and the features presented by e-learning websites to learners. Therefore, this factor plays an important role in the success of web based e-learning systems as an effective content is the main concern of learners when visiting an educational system. The related sub-factors are also important as all of them emphasize on the effectiveness of content so that quality learning experience could be delivered. Firstly, ‘Timely’ sub-factor has been noticed to be the most important factor among others, it indicates that the content of the website should be updated on regular basis to reflect the latest development and progress made in the field or area of study [31]. Secondly, the content should always be accurate and authentic so that e-learning system would be perceived as trustworthy learning source. Thirdly, the learning material should be presented using variety of resources including video, images, graphics, audio, animation and simulation to add value and to accommodate the learning needs of learners with diverse learning preferences [31]. Lastly, the variety of learning material should be relevant to the topic being discussed and to the level of students to whom the contents are being delivered. The finding of this research is similar to previous research in which emphasis is also given on ‘Content’ and declared it as one of the most important factors to evaluate the quality of web based e-learning systems [61].
The second most important quality factor in the proposed hierarchical model was usability. Usability is the ability of a system to help users in achieving their goals effectively, efficiently, and satisfactorily within the context of use [60,62]. Thus, web-based e-learning systems should support the learning process and allow learners to efficiently use the system. It should also be appropriate for the intended learning tasks [61]. The related sub-factors were important, as they emphasized the usability of the system. The user friendly sub-factor indicated that the interface should be easy to use, allow learners to achieve learning objectives when they use e-learning systems for first time, and help them to perform learning tasks quickly in the future [62]. The e-learning websites should consist of “interactive features” that allow learners to fully manipulate the course and have control over the learning objects [61]. Reliability indicated that the web-based e-learning system should be accessed by a user anywhere and anytime using a diverse range of devices and platforms. Previous research supports the finding of this study. It is underlined that every software system has some functionality that is provided to its users, such as teaching/learning functionality in e-learning systems. However, the value of system functionality is noticeable only when the system can be effectively and efficiently utilized by its users. The real effectiveness of a software is realized when there is an appropriate balance between the functionality and usability of the system. Usability is, therefore, a significant factor which complements the system functionality and enables users to engage with the system to perform all actions and services offered by the system. In the absence of such factors, users remain unable to take full advantage of the system’s services and will not enjoy the system while using it [62]. To achieve usability, human-centered methodology may be adopted in the development process [63].
The third important quality factor is organization, which is related to the overall structure of the website. It describes how the important aspects of a web-based system are organized. The related sub-factors emphasized the need for an appropriate structure and the organization of educational websites so that it is accessible to all kinds of learners. Index is the main page of the website; it should present all learning categories and the classification of learning materials so that learners can easily access the intended learning path [48,52]. The logo should be available on e-learning websites to indicate that the learning content is being delivered from authentic and credible sources, which is a deep concern for learners of this information age [53,54,64,65]. The “links” or hyperlinks on websites must be active, as the dead links irritates the learners [54]. On long pages, a list of contents with links should be available at the top of the page to take learners to the corresponding content farther down the web page [66]. The links should be designed with glosses as short phrases to guide the learners to what is actually located behind the links. Lastly, navigation is a method used to find information on a web-based e-learning system. A navigation page assists learners in locating and connecting to desired pages. The navigation scheme should be designed in a way that allows learners to locate and access learning material effectively and efficiently [66].
Finally, the quality factor design is concerned with the design and visual aspects of the website. It emphasizes the requirement that the important items of the e-learning website should be placed consistently and aligned appropriately. The design of the website must ensure that the web pages show a reasonable amount of white space, as too much white space can demand substantial scrolling, whereas too little may offer a presentation that looks too busy. The sub-factors of design include attractiveness, text, browser compatibility, and multimedia elements. The overall design of the website should be attractive to retain the learner’s attention for a longer period of time so that they spend most of their time with the learning application [32,35,66]. The text should be presented with a font size, style, foreground, and background color that can support the learners in reading and comprehending the learning materials rather than hindering them [53,54,65]. Different learners have different browser features and different defaults: for example, learners with visual impairments have a tendency to select larger fonts, whereas some other learners may turn off backgrounds, use fewer color options, or override the font. Therefore, the e-learning web-based system should be compatible with common browsers and with browser settings most commonly used by the learners [46,47,66]. Moreover, the design of web-based e-learning system should have the support of multimedia elements, as these elements enforce the user to concentrate on the learning items [66]. In short, for effective solutions, there is a need to rethink the customer’s needs and innovate.

6. Conclusions and Future Work

The quality of a product or a service directly affects the satisfaction of its user. E-learning websites are continually being designed even by developing countries to reinforce their education systems. The basic motive behind such rapid development of web-based e-learning resources is to achieve academic excellence. However, these websites suffer deficiency and ineffectiveness in their usage, owing to their poor quality. An evaluation of an e-learning website in terms of quality is a challenging task. Nevertheless, the quality level of e-learning websites can be investigated with the help of any suitable framework or model. This study has extensively reviewed the literature to identify the key quality factors that would be used to evaluate the quality of e-learning websites. The AHP technique has been used to prioritize the quality factors for the evaluation of e-learning websites. Finally, an effective evaluation model has been proposed. However, there are certain limitations of this research that should be addressed in the future. To achieve a quality evaluation model, this study only considers stakeholders from the urban areas of developing countries. The stakeholders belong to a mainstream community of students and teachers; however, there are many others with different incapacities and disabilities who are not included in this study. The students also differ from each other in terms of their learning abilities, capacities, orientation, and emotions, which are also not considered. The future research would be planned with two independent studies that consider a large sample from rural environments, as well as urban environments of developing countries to reveal if there are any divergent results. Additionally, the individual differences of learners and their special needs should be considered so that a universal quality evaluation model can be proposed. E-learning systems are mainly web-based systems that are open to computer threats, so security is an important concern in these systems and will help in leveraging overall cybersecurity [67]. The e-learning is penetrating into education at a very fast pace. Information should be actively secured to prevent the loss of privacy, availability, and integrity [68]. It is, therefore, vital to identify and understand the threats to the systems in order to develop secure e-learning systems. Web based e-learning systems should explore with respect to cloud computing technologies which have changed the way to develop and access applications. Moreover, instead of AHP, there are other multi-criteria decision analysis techniques that exist, such as Fuzzy AHP and TOPSIS, which should be explored in future research.

Author Contributions

Conceptualization, A.H.M. and A.S.; Formal analysis, A.S., A.A., A.E.Y. and A.-B.S.A.-T.; Funding acquisition, A.E.Y. A.-B.S.A-T and K.S.; Investigation, A.A. and A.-B.S.A.-T.; Methodology, A.H.M. A.E.Y. and A.A.; Project administration, K.S. and B.S.; Resources, B.S.; Software, A.H.M. and A.A.; Supervision, A.H.M. and A.S.; Validation, A.H.M. and A.E.Y.; Writing—original draft, A.S.; Writing—review & editing, A.S. and A.E.Y. All authors have read and agreed to the published version of the manuscript.

Funding

This research has no external funding.

Acknowledgments

The authors extend their appreciation to the Deanship of Scientific Research at King Saud University for funding this work through research group no (RG-1439-022).

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Jindal, A. Ranking Criteria classification for e-learning website selection and evaluation. BMIET J. Sci. Technol. Manag. 2018, 2, 20–26. [Google Scholar]
  2. Ozkan, S.; Koseler, R. Multi-dimensional students’ evaluation of e-learning systems in the higher education context: An empirical investigation. Comput. Educ. 2009, 53, 1285–1296. [Google Scholar] [CrossRef]
  3. Shee, D.Y.; Wang, Y.-S. Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications. Comput. Educ. 2008, 50, 894–905. [Google Scholar] [CrossRef]
  4. Lee, J.-K.; Lee, W.-K. The relationship of e-Learner’s self-regulatory efficacy and perception of e-Learning environmental quality. Comput. Hum. Behav. 2008, 24, 32–47. [Google Scholar] [CrossRef]
  5. Koh, T.S.; Lee, S.C. Digital skills and education: Singapore’s ICT master planning for the school sector. In Toward a Better Future: Education and Training for Economic Development in Singapore Since 1965; Lee, S.K., Goh, C.B., Fredriksen, B., Tan, J.P., Eds.; The World Bank: Washington, DC, USA, 2008; pp. 167–190. [Google Scholar]
  6. Chan, F.M. ICT in Malaysian schools: Policy and strategies. In Proceedings of the A Workshop on the Promotion of ICT in Education to Narrow the Digital Divide, Tokyo, Japan, 15–22 October 2000. [Google Scholar]
  7. Pham, L.; Limbu, Y.B.; Bui, T.K.; Nguyen, H.T.; Pham, H.T. Does e-learning service quality influence e-learning student satisfaction and loyalty? Evidence from Vietnam. Int. J. Educ. Technol. High. Educ. 2019, 16, 7. [Google Scholar] [CrossRef]
  8. Chung, J. Integrating ICTs into Education: Lessons Learned. A Collective Case Study of Six Asian Countries: Indonesia, Malaysia, Philippines, Singapore, South Korea, Thailand. Asia-Pacific Collab. Educ. J. 2005, 1, 77–81. [Google Scholar]
  9. Farid, S.; Ahmad, R.; Alam, M. A Hierarchical Model for E-learning Implementation Challenges using AHP. Malays. J. Comput. Sci. 2015, 28, 166–188. [Google Scholar] [CrossRef]
  10. Deepwell, F. Embedding quality in e-learning implementation through evaluation. J. Educ. Technol. Soc. 2007, 10, 34–43. [Google Scholar]
  11. Fitzpatrick, R. Software quality: Definitions and strategic issues. In Staffordshire University, School of Computing Report; Dublin Institute of Technology: Dublin, Ireland, 1996. [Google Scholar]
  12. Mardle, S.; Pascoe, S.; Herrero, I. Management Objective Importance in Fisheries: An Evaluation Using the Analytic Hierarchy Process (AHP). Environ. Manag. 2004, 33, 1–11. [Google Scholar] [CrossRef]
  13. Masoumi, D.; Lindstrom, B. Quality in e-learning: A framework for promoting and assuring quality in virtual institutions. J. Comput. Assist. Learn. 2011, 28, 27–41. [Google Scholar] [CrossRef]
  14. Govindasamy, T. Successful implementation of e-learning: Pedagogical considerations. Internet Higher Educ. 2001, 4, 287–299. [Google Scholar] [CrossRef]
  15. Jabr, M.; Al-Omari, H. e-Learning Management System Using Service Oriented Architecture Department of Computer Science, Amman Arab University for Graduate Studies, Amman, Jordan King Abdulaziz University for Science and Technology, Riyadh, Saudi Arabia. J. Comput. Sci. 2010, 6, 285–295. [Google Scholar] [CrossRef]
  16. Amin, F.M.H.; Salih, N.K. New model to achieve software quality assurance in e-learning application. IJCSI 2017, 14, 65–70. [Google Scholar]
  17. Farid, S.; Ahmad, R.; Niaz, I.; Itmazi, J.; Asghar, K. Identifying perceived challenges of e-learning implementation. In Proceedings of the First International Conference on Modern Communication & Computing Technologies (MCCT’14), Researchgate.Net, Nawabshah, Pakistan, 26–28 February 2014; pp. 1–11. [Google Scholar]
  18. Chua, B.B.; Dyson, L.E. Applying the ISO 9126 Model to the Evaluation of an e-Learning System; Faculty of Information Technology University of Technology: Sydney, Australia, 2004; pp. 184–190. [Google Scholar]
  19. Djouab, R.; Bari, M. An ISO 9126 Based Quality Model for the e-Learning Systems. Int. J. Inf. Educ. Technol. 2016, 6, 370–375. [Google Scholar] [CrossRef]
  20. Olsina, L.; Rossi, G. Application Quality with WebQEM. IEEE Multimed. 2012, 9, 20–29. [Google Scholar] [CrossRef]
  21. Uppal, M.A.; Ali, S.; Gulliver, S. Factors determining e-learning service quality. Br. J. Educ. Technol. 2018, 49, 412–426. [Google Scholar] [CrossRef]
  22. Tsigereda, W.M. A framework for evaluating Academic Website’s quality. In Proceedings of the Third International Conference on Computational Intelligence and Informatics; Springer: Berlin, Germany, 2010. [Google Scholar]
  23. A Guide to Quality in Online Learning. Available online: https://www.chea.org/userfiles/uploads/A%20Guide%20to%20Quality%20in%20Online%20Learning.pdf (accessed on 5 April 2020).
  24. Sohaib, O.; Naderpour, M.; Hussain, W. SaaS E-Commerce Platforms Web Accessibility Evaluation. In Proceedings of the 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), IEEE, Rio de Janeiro, Brazil, 8–13 July 2018; pp. 1–7. [Google Scholar]
  25. Sohaib, O.; Naderpour, M.; Hussain, W.; Martínez, L. Cloud computing model selection for e-commerce enterprises using a new 2-tuple fuzzy linguistic decision-making method. Comput. Ind. Eng. 2019, 132, 47–58. [Google Scholar] [CrossRef]
  26. Hamrioui, S.; de la Torre Díez, I.; Garcia-Zapirain, B.; Saleem, K.; Rodrigues, J.J. A systematic review of security mechanisms for big data in health and new alternatives for hospitals. Wirel. Commun. Mob. Comput. 2017, 2017, 6. [Google Scholar] [CrossRef]
  27. Devi, K.; Sharma, A.K. Framework for Evaluation of Academic Website. Int. J. Comput. Tech. 2016, 3, 234–239. [Google Scholar]
  28. Fitzpatrick, R. Additional Quality Factors for the World Wide Web Keywords. In Proceedings of the Second World Congress for Software Quality, Oulu, Finland, 20–22 June 2000. [Google Scholar]
  29. Saaty, R.W. The analytic hierarchy process—What it is and how it is used. Math. Model. Pergamon J. 1987, 9, 161–176. [Google Scholar] [CrossRef]
  30. Youssef, A.E.; Mostafa, A.M. Critical Decision-Making on Cloud Computing Adoption in Organizations Based on Augmented Force Field Analysis. IEEE Access 2019, 7, 167229–167239. [Google Scholar] [CrossRef]
  31. Saaty, T.L. How to Make a Decision: The Analytic Hierarchy Process. Bonifay(M.) Carre (M.-B.) Rigoir dir.—Fouill. à Marseille. Les mobiliers (Ier-VIIe s.).Paris Errance (Etudes Massaliètes 5) 1998, 1994, 200–205. [Google Scholar]
  32. Kanagaraj, D.J.; Sudhahar, C. Website quality: Imperatives for effective industrial marketing through websites’ usage intensity augmentation. Int. J. Electron. Cust. Relatsh. Manag. 2015, 9, 203. [Google Scholar] [CrossRef]
  33. Reitsamer, B.; Kaschig, A.; Heinz, T.; Stokburger-Sauer, N. The Effects of Website Quality Perception on Users’ Responses—A Multidisciplinary Approach. In Proceedings of the Americas Conference on Information Systems, Savannah, GA, USA, 2014; pp. 1–12. [Google Scholar]
  34. Shafiee, M.M.; Rahimzadeh, S.; Haghighizade, R. The effect of implementing SEO techniques and websites design methods on e-tourism development: A study of travel agencies e-tourism websites. In Proceedings of the 2016 10th International Conference on e-Commerce in Developing Countries: With Focus on e-Tourism (ECDC); Institute of Electrical and Electronics Engineers (IEEE), Isfahan, Iran, 15–16 April 2016; pp. 1–8. [Google Scholar]
  35. Lane, E.F.; Verdini, W.A. A Consistency Test for AHP Decision Makers. Decis. Sci. 1989, 20, 575–590. [Google Scholar] [CrossRef]
  36. Tse, A.C.B.; Chan, C.-F. The Relationship between Interactive Functions and Website Ranking. J. Advert. Res. 2004, 44, 369–374. [Google Scholar] [CrossRef]
  37. Lim, K.H.; Benbasat, I. The Influence of Multimedia on Improving the Comprehension of Organizational Information. J. Manag. Inf. Syst. 2002, 19, 99–127. [Google Scholar] [CrossRef]
  38. Yang, Z.; Cai, S.; Zhou, Z.; Zhou, N. Development and validation of an instrument to measure user perceived service quality of information presenting Web portals. Inf. Manag. 2005, 42, 575–589. [Google Scholar] [CrossRef]
  39. Hillier, M. The role of cultural context in multilingual website usability. Electron. Commer. Res. Appl. 2003, 2, 2–14. [Google Scholar] [CrossRef]
  40. Exploring Website Evaluation Criteria using the Repertory Grid Technique: A Web Designers’ Perspective Complete Research Submission Total Word Count: 5,268. Technology 2003, 1–22.
  41. Vidgen, R.; Barnes, S. An integrative approach to the assessment of ecommerce quality. J. Electron. Commer. Res. 2002, 3, 114–127. [Google Scholar]
  42. Abanumy, A.; Al-badi, A.; Mayhew, P. e-Government Website Accessibility: In-Depth Evaluation of Saudi Arabia and Oman. Evaluation 2005, 3, 99–106. [Google Scholar]
  43. Lin, O.; Joyce, D. Critical Success Factors for Online Auction Web Sites. Online 2004, 14, 341–344. [Google Scholar]
  44. Basu, A. Context-driven assessment of commercial Web sites. In Proceedings of the 36th Annual Hawaii International Conference on System Sciences, Big Island, HI, USA, 6–9 January 2003. [Google Scholar]
  45. Lautenbach, M.A.E.; ter Schegget, I.E.; Schoute, A.E.; Witteman, C.L.M. Evaluating the usability of web pages: A case study. Artificial Intelligence Preprint Series 1999, 11, 1–14. [Google Scholar]
  46. Li, X.; Zeng, H. Modeling web application for cross-browser compatibility testing. In Proceedings of the 15th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), Las Vegas, NV, USA, 30 June–2 July 2014; pp. 1–5. [Google Scholar]
  47. Xu, B.; Xu, L.; Nie, C.; Chu, W.; Chang, C. Applying combinatorial method to test browser compatibility. In Proceedings of the 15th International Symposium on Multimedia Software Engineering, Institute of Electrical and Electronics Engineers (IEEE), Taichung, Taiwan, 10–12 December 2003; pp. 156–162. [Google Scholar]
  48. Wenham, D.; Zaphiris, P. User interface evaluation methods for internet banking web sites: A review, evaluation and case study. In Proceedings of the HCII’03 Human–Computer Interaction International Conference, Crete, Greece, 22–27 June 2003; pp. 721–725. [Google Scholar]
  49. Weideman, M.; Ngindana, M. Website navigation architectures and their effect on website visibility -a literature survey. S. Afr. Inst. Comput. Sci. Inf. Technol. 2004, 59, 292–296. [Google Scholar]
  50. Djonov, E. Website hierarchy and the interaction between content organization, webpage and navigation design: A systemic functional hypermedia discourse analysis perspective. Inf. Des. J. 2007, 15, 144–162. [Google Scholar] [CrossRef]
  51. Diniz, E.; Porto, R.M.; Adachi, T. Internet Banking in Brazil: Evaluation of Functionality, Reliability and Usability. Evaluation 2005, 8, 41–50. [Google Scholar]
  52. Hadullo, K.; Oboko, R.; Omwenga, E. Factors affecting asynchronous e-learning quality in developing countries university settings. Int. J. Educ. Develop. Using ICT 2018, 14. [Google Scholar]
  53. George, J.; Iowa State University; Mirsadikov, A.; Mennecke, B. Website Credibility Assessment: An Empirical - Investigation of Prominence-interpretation Theory. AIS Trans. Human-Computer Interact. 2016, 8, 40–57. [Google Scholar] [CrossRef]
  54. Lin, F.-J.; Huarng, K.; Chen, Y.-M.; Lin, S.-M. Quality evaluation of Web services. In Proceedings of the IEEE International Conference on E-Commerce Technology for Dynamic E-Business, Beijing, China, 13–15 September 2005; pp. 226–233. [Google Scholar]
  55. Hasan, L.; Abuelrub, E. Assessing the quality of web sites. Appl. Comput. Inform. 2011, 9, 11–29. [Google Scholar] [CrossRef]
  56. Rababah, O.; Masoud, F. Key Factors for Developing a Successful E-commerce Website. Commun. IBIMA 2010, 9. [Google Scholar] [CrossRef]
  57. Chen, Y.-H.; Hsu, I.-C.; Lin, C.-C. Website attributes that increase consumer purchase intention: A conjoint analysis. J. Bus. Res. 2010, 63, 1007–1014. [Google Scholar] [CrossRef]
  58. Tsai, W.T.; Zhang, D.; Chen, Y.; Huang, H.; Paul, R.; Liao, N. A software reliability model for web services. In Proceedings of the 8th IASTED International Conference on Software Engineering and Applications, Cambridge, MA, USA, November 2004; pp. 144–149. [Google Scholar]
  59. Zheng, Z.; Lyu, M.R. Personalized Reliability Prediction of Web Services. ACM Trans. Softw. Eng. Methodol. 2013, 22, 1–25. [Google Scholar] [CrossRef]
  60. Qureshi, K.; Irfan, M. Usability Evaluation of e-Learning Applications, a Case Study of It’s Learning from a Student’s Perspective. Master’s Thesis, Blekinge Institute of Technology, Karlskrona, Sweden, 2009. [Google Scholar]
  61. Kazaine, I. Evaluating the quality of e-learning material. In Proceedings of the 11th International Scientific and Practical Conference. Volume II; Rēzeknes Tehnoloģiju akadēmija: Rēzekne, Latvia, 2017; Volume 74, p. 77. [Google Scholar]
  62. Onacan, M.; Alper, E. Usability evaluation of learning management system in a higher education institution: A scale development study. J. Glob. Strateg. Manag. 2016, 10. [Google Scholar] [CrossRef]
  63. Sohaib, O.; Solanki, H.; Dhaliwa, N.; Hussain, W.; Asif, M. Integrating design thinking into extreme programming. J. Ambient. Intell. Humaniz. Comput. 2018, 10, 2485–2492. [Google Scholar] [CrossRef]
  64. Baratis, E.; Petrakis, E.; Milios, E. Automatic Website Summarization by Image Content: A Case Study with Logo and Trademark Images. IEEE Trans. Knowl. Data Eng. 2008, 20, 1195–1204. [Google Scholar] [CrossRef]
  65. Lowry, P.B.; Wilson, D.W.; Haig, W.L. A Picture is Worth a Thousand Words: Source Credibility Theory Applied to Logo and Website Design for Heightened Credibility and Consumer Trust. Int. J. Human-Computer Interact. 2013, 30, 63–93. [Google Scholar] [CrossRef]
  66. Shneiderman, B.; Michael, L. Research-Based Web Design & Usability Guidelines; Washington DC, Department of Health and Human Services: Washington, DC, USA, 2006.
  67. Al-Muhtadi, J.; Shahzad, B.; Saleem, K.; Jameel, W.; Orgun, M.A. Cybersecurity and privacy issues for socially integrated mobile healthcare applications operating in a multi-cloud environment. Health Inform. J. 2019, 25, 315–329. [Google Scholar] [CrossRef]
  68. Sahi, M.A.; Abbas, H.; Saleem, K.; Yang, X.; Derhab, A.; Orgun, M.A.; Iqbal, W.; Rashid, I.; Yaseen, A. Privacy preservation in e-healthcare environments: State of the art and future directions. IEEE Access 2017, 6, 464–478. [Google Scholar] [CrossRef]
Figure 1. Research methodology.
Figure 1. Research methodology.
Sustainability 12 04071 g001
Figure 2. Hierarchical structure in AHP.
Figure 2. Hierarchical structure in AHP.
Sustainability 12 04071 g002
Figure 3. Graphical representation of mean values of quality sub-criteria.
Figure 3. Graphical representation of mean values of quality sub-criteria.
Sustainability 12 04071 g003
Figure 4. The hierarchical quality model for e-learning website.
Figure 4. The hierarchical quality model for e-learning website.
Sustainability 12 04071 g004
Figure 5. Examples for pairwise comparisons, as accomplished by Expert Choice.
Figure 5. Examples for pairwise comparisons, as accomplished by Expert Choice.
Sustainability 12 04071 g005
Figure 6. Example for pairwise comparison (content & organization) as accomplished by Expert Choice.
Figure 6. Example for pairwise comparison (content & organization) as accomplished by Expert Choice.
Sustainability 12 04071 g006
Figure 7. Example for pairwise comparison (content & design) as accomplished by Expert Choice.
Figure 7. Example for pairwise comparison (content & design) as accomplished by Expert Choice.
Sustainability 12 04071 g007
Figure 8. Example for pairwise comparison (usability & organization) as accomplished by Expert Choice.
Figure 8. Example for pairwise comparison (usability & organization) as accomplished by Expert Choice.
Sustainability 12 04071 g008
Figure 9. Quality Evaluation Model.
Figure 9. Quality Evaluation Model.
Sustainability 12 04071 g009
Table 1. Demographic profile of participants.
Table 1. Demographic profile of participants.
DemographicFrequencyPercent
Male12480.5
Female3019.5
QualificationFrequencyPercent
Undergraduate2214.3
Graduate6642.9
Postgraduate6139.6
PhD53.2
ProfessionFrequencyPercent
Students10065.0
Academicians2013.0
Web Developers2013.0
Educational Administrators149.0
Table 2. Index values in pairwise comparison matrix Analytic Hierarchy Process (AHP) scale.
Table 2. Index values in pairwise comparison matrix Analytic Hierarchy Process (AHP) scale.
DefinitionIndexDefinitionIndex
Equally important1Equally important1/1
Moderately more important3Moderately less important1/3
Much more important5Much less important1/5
Far more important7Far less important1/7
Extremely more important9Extremely less important1/9
Table 3. Relationship between RI and n.
Table 3. Relationship between RI and n.
N12345678910
RI000.580.91.121.241.321.411.451.49
Table 4. Extracted quality dimensions.
Table 4. Extracted quality dimensions.
Quality Main DimensionsQuality Sub-DimensionsLiterature Reference
ContentTimely[32,33]
Relevant[34,35]
Multilanguage[28]
Variety of Presentation[36,37]
Accuracy[38]
Reliability of Content[39,40]
DesignAttractive[32,35]
Appropriateness[41]
Color[42,43]
Multimedia elements[37]
Text[44,45]
Browser Compatibility[46,47]
OrganizationIndex[48]
Navigation[49,50]
Consistency[51]
Links[52]
Logo[53,54]
Domain[55]
UsabilityUser friendly[56,57]
Reliability[58,59]
Availability[38]
Interactive features[41,43]
Table 5. Analysis of data collected through the survey.
Table 5. Analysis of data collected through the survey.
CriteriaSub-Criteria Contribution LevelMeanSt. Deviation
Very ImportantImportantNeutralLess ImportantNot Important
ContentTimely15490574214.510.679
%58.437.02.61.30.6
Relevant154627416204.270.698
%40.348.110.41.30
Multilingual1543945402463.561.143
%25.329.226.015.63.9
Variety of Presentation1545462251214.010.943
%35.140.316.27.80.6
Accuracy154565928834.020.967
%36.438.318.25.21.9
Reliability of Content1545658171763.921.126
%36.437.711.011.03.9
DesignAttractive154755610674.211.040
%48.736.46.53.94.5
Appropriateness1543660263113.641.071
%23.439.016.920.10.6
Color1544068281083.791.064
%26.044.218.26.55.2
Multimedia Elements1546449251064.011.094
%41.631.816.26.53.9
Text1545273171114.060.891
%33.847.411.07.10.6
Browser Compatibility1546851221124.120.990
%44.233.114.37.11.3
OrganizationIndex1547356141014.230.913
%47.436.49.16.50.6
Navigation1546063191024.100.941
%39.040.912.36.51.3
Consistency1543846432253.581.107
%24.729.927.914.33.2
Links1545958181904.021.000
%38.337.711.712.30
Logo154526922924.04.914
%33.844.814.35.81.3
Domain1543862341643.741.028
%24.740.322.110.42.6
UsabilityUser Friendly154106379024.590.720
%68.824.05.801.3
Reliability154547519514.140.804
%35.148.712.33.20.6
Availability1545455321123.960.983
%35.135.720.87.11.3
Interactive Features154566323844.030.980
%36.440.914.95.22.6
Table 6. Global weights and global rank of main criteria.
Table 6. Global weights and global rank of main criteria.
CriteriaWeights (Global)Rank (Global)
Content0.4071
Usability0.2942
Organization0.1883
Design0.1114
Table 7. Local weights and rank of sub-criteria of content.
Table 7. Local weights and rank of sub-criteria of content.
Sub-CriteriaWeights (Local)Rank (Local)
Timely0.3021
Accuracy0.2762
Variety of Presentation0.2343
Relevant0.1884
Table 8. Local weights and rank of sub-criteria of usability.
Table 8. Local weights and rank of sub-criteria of usability.
Sub-CriteriaWeights (Local)Rank (Local)
User Friendly0.4011
Interactive Features0.3942
Reliability0.2063
Table 9. Local weights and rank of sub-criteria of organization.
Table 9. Local weights and rank of sub-criteria of organization.
Sub-CriteriaWeights (Local)Rank (Local)
Index0.3131
Logo0.3012
Links0.2373
Navigation0.1494
Table 10. Local weights and rank of sub-criteria of design.
Table 10. Local weights and rank of sub-criteria of design.
Sub-CriteriaWeights (Local)Rank (Local)
Attractive0.3891
Text0.2782
Browser Compatibility0.1913
Multimedia Elements0.1424
Table 11. Overall weights and rankings of criteria and sub-criteria.
Table 11. Overall weights and rankings of criteria and sub-criteria.
FactorsGlobal Weight with Contribution %Global Ranking of FactorsSub-FactorsLocal Weight with Contribution %Local Ranking of Sub Factors
Content0.407 (40.7%)1Timely0.302 (30.2%)1
Accuracy0.276 (27.6%)2
Variety of Presentation0.234 (23.4%)3
Relevant0.188 (18.8%)4
Usability0.294 (29.4%)2User Friendly0.401 (40.1%)1
Interactive Features0.394 (39.4%)2
Reliability0.206 (20.6%)3
Organization0.188 (18.8%)3Index0.313 (31.3%)1
Logo0.301 (30.1%)2
Links0.237 (23.7%)3
Navigation0.149 (14.9%)4
Design0.111 (11.1%)4Attractive0.389 (38.9%)1
Text0.278 (27.8%)2
Browser Compatibility0.191 (19.1%)3
Multimedia Elements0.142 (14.2%)4
Back to TopTop