A Hierarchical Model to Evaluate the Quality of Web-Based E-Learning Systems

Abstract

The rapid growth of Information and Communication Technologies (ICT)—specifically, the Internet—has given emergence to e-learning. Resultantly, web-based e-learning systems are being increasingly developed to enhance the learning process. However, the utilization of such systems is low, mainly owing to poor quality content and overall design problems. To improve usage, it is imperative to identify the factors with the most significant impact on the quality of these systems so that the e-learning industry keeps these factors in consideration while developing e-learning systems. This study focused on the identification and prioritization of factors related to the design quality of e-learning systems through a hierarchical quality model. Thus, firstly, an extensive literature review was conducted to identify the factors that most affect the quality of web-based e-learning systems. Secondly, among the identified factors, only those with the most significant effect were considered. To identify the most important quality criteria, a survey was conducted. An instrument was deployed among 157 subjects, including e-learning designers, developers, students, teachers, and educational administrators. Finally, a second instrument was distributed among 51 participants to make a pairwise comparison among the criteria and rank them according to their relative importance. The identified and prioritized factors were classified into four main categories. Among these four factors, content was identified as the most important factor, whereas design was found to be the least important factor.

1. Introduction

As the recent coronavirus outbreak prompted universities to start shifting classes online either for a few weeks or for the remainder of the spring semester of 2020, e-learning and remote education have popped up as the magical alternative for in-person classes in the time of the COVID-19 pandemic. However, e-learning was not originally launched as a result of the outbreak of coronavirus; in fact, it has been used as a solution for numerous problems in education systems for a while, especially in developing countries. The most important among these problems is the shortage of qualified and experienced faculty, which constricts the learning opportunities in not only rural areas, but also urban [1]. E-learning systems are growing tremendously as an alternative means to offer educational services at self-pace, anywhere and anytime. Ozkan and Koseler define e-learning as an electronic mechanism used to deliver learning material to learners [2]. According to the IEEE Technology Standard Committee, e-learning systems are learning systems that use web browsers to impart an online learning experience. Another definition is that e-learning is a combination of a computer, browser, and internet to provide online education and training [3,4].

There are several e-learning means to deliver education, such as CDs, online chats, forums, video conferencing, and webcasting. At present, e-learning websites are considered a common source of delivering subject-related material. The learners interact with these websites in order to perform learning tasks efficiently, effectively, and satisfactorily. However, quality continues to be a serious issue in these websites, since most of them are insufficiently effective. Therefore, learners find it difficult to perform their tasks using most of these websites [4]. To impart the satisfactory learning experience, the websites should be effective, efficient, and usable so that extraneous cognitive load can be avoided. Thus, the quality of e-learning websites is an important aspect to meet the program learning objectives (PLO), and students heavily rely on high-quality e-learning websites to attain the course learning outcomes (CLO).

Consequently, educational institutes around the globe are switching towards an electronic mode of education in order to attract more and more international students. Besides developed countries, the developing countries are also ambitious to adopt the concept of e-learning; Singapore and Malaysia have completely revamped their educational system to integrate information and communication technologies (ICT) in their learning environments [5,6]. For the maximum utilization of e-learning websites to attain the PLO and CLO, quality is a mandatory aspect. Previous research has shown that the quality of e-learning resources influences e-learning students’ satisfaction, retention, and loyalty [7]. Developing countries are striving for the implementation of e-learning resources with their limited budget to contend with educational problems, such as a lack of qualified teachers, issues of access, and poor learning outcomes [8]. Web-based e-learning systems have successfully been implemented in many places; however, many of them have failed to realize the objectives, motives, and expectations behind their development. There could be many reasons for such failure, but the most obvious is the poor quality of e-learning resources [9]. Hence, the availability of guidelines for the design and development of high-quality web-based e-learning systems is crucial.

It is widely recognized that the quality of web-based e-learning systems is of utmost importance to achieve the learning objectives and to resolve chronic educational problems [10]. According to IEEE, the quality “is the degree to which a system, component, or process meets customer expectations” [11]. The design of e-learning websites should enable efficient and effective human–computer interactions, as well as reduce the user’s workload by taking advantage of the computer’s capabilities. For the best possible learning outcomes, designers need to consider some important user interface parameters while developing e-learning websites. The exposure of such design parameters requires evidence-based information rather than mere opinions, and this can be achieved through the sound mechanism of evaluation for e-learning websites.

There are two major dimensions of e-learning systems with respect to evaluation: the first is pedagogical and the second is related to the design and development aspects. Numerous frameworks and models exist that could be used to assure the quality of web-based e-learning. However, these models lack the guidance of the e-learning industry, which is, among other factors, the most important factor affecting the quality of e-learning websites. Secondly, such models are mainly feasible to evaluate the pedagogical aspects of e-learning systems rather than the design and development aspects [12]. In this study, we argue that an evaluation model is required to discover the factors related to the quality of the design and development of e-learning systems and to determine the relative importance of such factors in order to achieve a high-quality e-learning product. The outcome of this work would help designers, developers, and software project managers in making decisions regarding the parameters they should prioritize while producing a high-quality e-learning product. How the quality of web-based e-learning systems could be improved is an open research question that should be addressed forthwith. This study raises two important questions: (i) which factors influence the quality of e-learning through websites? (ii) How will these factors be ranked?

This study aimed particularly at (i) the identification of the key factors (i.e., the most important) affecting the quality of e-learning websites; (ii) the prioritization of the identified factors by defining their relative importance; (iii) the proposed hierarchical quality model for the evaluation of e-learning websites. The methodology used to attain these goals was as follows: firstly, an extensive literature review was conducted to explore and identify the factors that mostly affect the quality of web-based e-learning systems. Secondly, among the identified factors, only those with the most significant impact were considered in this study. To identify the most important quality criteria, an electronic and manual survey was conducted, and an instrument was deployed among 157 subjects, including e-learning designers, developers, students, teachers, and educational administrators. Finally, another instrument was distributed among 51 participants to make a pairwise comparison among the criteria and rank them according to their relative importance. The identified and prioritized key quality factors were classified into four categories, including content, design, usability, and organization. Among these four factors, content was identified as the most important factor, whereas design was found to be the least important factor.

The rest of this paper is organized as follows: Section 2 presents related work and Section 3 describes research methodology. In Section 4, we present and discuss the findings of this study and in Section 5 we propose the quality evaluation model based on these findings. Finally, in Section 6, we give our conclusions and future work.

2. Related Work

Researchers have introduced many different evaluation models and frameworks [13,14]. However, these models lack the evaluation of websites with respect to software design and development perspectives. The e-learning quality models and frameworks introduced by prior research are as follows:

Jabr and Al-omari proposed a web service-based e-learning framework that increased the efficiency and effectiveness of collaborative learning in terms of reusability, interoperability, accessibility, and modularization [15]. The model does not evaluate the e-learning websites either from pedagogical or developmental perspectives. Amin and Salih proposed a model premised on critical success factors (CSFs) from a university student’s perspective. They surveyed e-learning CSFs and classified them into four dimensions. The model aimed to achieve high quality in web-based e-learning systems [16]. It considered only the single fraction of stakeholders to propose the dimension for the attainment of high quality in e-learning solutions. In our opinion, the model partially fulfils the quality needs of e-learning websites, as the findings are only based on the views of students.

Wang Shee and Wang proposed a multi-criteria evaluation method for web-based e-learning systems, considering only a single aspect of e-learning, which was learner’s satisfaction. The main dimensions of the model were learner interface, learning community, system, and personalization [17]. There are also other important aspects of e-learning websites, such as effectiveness and efficiency, which are not catered to by the proposed model. Chua and Dyson introduced the ISO 9126 quality model as a tool to evaluate e-learning systems in order to improve the quality [18]. Djouab and Bari introduced an extension of the ISO 9126 software quality model. However, the extended model was neither validated nor were the guidelines related to the usage given [19].

Researchers have also proposed quality evaluation models specifically for websites. The website quality evaluation model (Web-QEM) was designed to evaluate the quality of web applications, which helps to meet the quality requirements, as well as discover the missing and poorly designed features. The Web-QEM model is based on the ISO 9126-model, so its quality characteristics include usability, reliability, efficiency, and functionality. This model evaluates the quality of a website, taking many different steps. It is an effective model to measure the quality of websites. However, Web-QEM does not address all the factors related to the software dimension of modern academic websites, such as learning experience [20].

Vida and Jons proposed the Web Quality Model (WQM) to evaluate web applications with respect to three different dimensions, including web features, ISO/IEC 9126-1-based quality characteristics, and lifecycle process. The model lacked in presenting a step by step evaluation criteria and sub-characteristics of the factors. The SERVQUAL model was proposed to assess the satisfaction and quality of websites. The model intended to measure the gap between the expectation of the customers and their usage experience. The model addressed the five quality attributes called RATER: i.e., reliability, assurance, tangibles, empathy, and responsiveness. However, it did not consider the sub-attributes of the RATER quality constructs [21]. Tsigereda proposed a model to evaluate the quality of academic websites. It evaluated websites from the perspective of students using four important quality factors, including content, reliability, efficiency, and functionality arranged in a hierarchal structure along with sub-quality factors. The downside of this model was that it did not take all important stakeholders into account, such as faculty members, educational administrators, and designers/developers. Moreover, the hierarchy model was not assigned weights or ranks to the quality factors [22].

Although a variety of e-learning quality models and frameworks have been proposed by researchers, certain limitations have been observed in these e-learning evaluation models. Firstly, some of these models have not considered the stakeholders of paramount importance with regard to e-learning websites. For example, the student’s perspective is an important aspect of the quality of e-learning websites. The e-learning websites should not be something that are merely imparted to a passive student. Instead, high quality e-learning websites should be developed through a process of co-production between the students and the learning system. The teacher’s perspective is also critical to ensure the quality of e-learning websites. It is the teacher who can offer suggestions regarding the teaching/learning method. Similarly, the educational experts can provide an appropriate learning strategy for the students being served through the educational website. Furthermore, the players from the e-learning industry, including designers and developers, can also make suggestions to evaluate the web-based e-learning systems [23]. Secondly, the majority of the quality models focused on pedagogical characteristics, including the learner, instructor, institution, social, and management, etc.. The software characteristics, such as design, usability, and digital contents have been ignored. Thirdly, some frameworks have addressed the limited number of software quality factors, including usability, portability, and reliability. The quality of e-learning websites from a software perspective cannot be gauged using such a limited number of quality factors. Finally, website evaluation is a multi-criteria decision making (MCDM) problem, as there are a number of different factors involved. MCDM methods are extensively applied in other areas, such as e-commerce [24,25], but, as per the author’s knowledge, they are rarely explored for the evaluation of web-based e-learning systems.

Keeping the shortcomings of previous research in mind, this study has considered the software perspective and key stakeholder of e-learning websites to propose a hierarchical quality evaluation model for e-learning websites. The domain of e-learning websites is very broad and there are numerous perspectives for the quality of e-learning websites, such as pedagogical, personal, institutional, software, and technical. The focus of this study is only on the software perspective.

3. Methodology

The methodology used in this research consists of three phases: in the first phase, a literature review was conducted to explore and identify the relevant factors that impact the quality of e-learning websites. As a result of this extensive literature review, the relevant quality factors were identified. In the second phase, a survey was conducted to determine the importance of various identified quality factors, and only those with a significant impact (key quality factors) were further considered in our study. Finally, in the third phase, the relative importance (weight) of each key quality factor was estimated using the most commonly used pairwise comparison method—that is, the Analytical Hierarchical Process (AHP). The output of this phase is a ranked list of quality factors for web-based e-learning systems, which was the basis of our proposed quality evaluation model. The graphical representation of the research methodology is depicted in Figure 1.

The details of the three phases of research methodology are discussed below.

3.1. Phase 1: Identify Relevant Quality Factors

To identify the factors that affect the quality of web-based e-learning systems, an extensive systematic literature review (SLR) [26] was conducted using different sources, including Google Scholar, Science Direct and IEEE Xplore. As a result, a broad range of quality factors (or dimensions) were identified; furthermore, the identified dimensions were classified into main dimensions and sub-dimensions. The main dimensions were content, design, organization, and usability, which are discussed in detail in Section 4.

3.2. Phase 2: Determine the Importance of Identified Quality Factors

As a result of phase 1, various relevant quality factors were identified. However, these factors did not have the same effect on e-learning websites: i.e., they were not of the same importance in terms of their impact on the quality of websites. Therefore, in the second phase, we attempted to determine the importance of the relevant quality factors, and only the factors whose importance exceeded a predetermined threshold (key factors) were further considered in our study. To determine the key quality factors, this study adopted the survey method, as the survey is an effective method to gather data from a wide range of the population. To collect data, a Likert scale questionnaire was developed, which consisted of a set of questions to evaluate the importance of the relevant quality factors identified in phase 1. The Likert scale questionnaire had five points, with the choices ranging from ‘very important’ to ‘not important’. The participants were asked to express their opinion and feelings by selecting an appropriate choice with respect to each element contributing to a certain quality dimension. To qualitatively analyze the collected data, IBM-SPSS (v.20) was used. The numeric values from 1 to 5 were assigned to each choice of the Likert scale, where 5 was assigned to ‘very important’ and 1 to ‘not important’.

The survey targeted four categories of respondents. The first category included web developers, particularly those who were involved in the development of learning applications. The second category of respondents were educational administrators, specifically those who had experience with e-learning websites or, at a minimum, were aware of the role of e-learning applications in delivering education. The third category was the academicians, which included course instructors and educators. The fourth category consisted of students. The last two categories involved those who had been using e-learning websites and were acquainted with a technology-infused learning environment. To choose the appropriate sample, a non-probability technique that is also known as snowball sampling was employed. The technique begins with the identification of a subject possessing desired characteristics. Afterward, the social network of identified subjects is utilized to select further participants [27]. This technique is also called referral sampling or chain sampling because the correctly identified sample helps researchers to further find more like-minded participants. In this sampling technique, the size of a sample increases gradually, similar to falling snow [28]. The students were conveniently available as compared to other subjects, which was why their number was high in the sample. However, a representation of all categories of subjects was ensured in the sample.

The sample selected contained 157 subjects to collect survey responses. The demographic profile of respondents was as follows: the respondents were comprised of both males and females. The male respondents were 124 (80.5%) and the females were 30 (19.5%). In terms of qualification, 22 (14.3%) respondents were undergraduates, 66 (42.9%) were graduates, 61 (39.6%) were postgraduates, and five (3.2%) had a doctorate degree. Amongst them, 100 (65%) were students, 20 (13%) were related to academia, 20 others (13%) were web developers, and 14 (9.1%) were related to educational administration, as shown in below

Table 1. Demographic profile of participants.

Demographic	Frequency	Percent
Male	124	80.5
Female	30	19.5
Qualification	Frequency	Percent
Undergraduate	22	14.3
Graduate	66	42.9
Postgraduate	61	39.6
PhD	5	3.2
Profession	Frequency	Percent
Students	100	65.0
Academicians	20	13.0
Web Developers	20	13.0
Educational Administrators	14	9.0

.

3.3. Phase 3: Prioritize Key Quality Factors Using AHP Method

The core purpose of this phase was to determine the relative weights and ranking of the key quality factors identified in phase 2. The pairwise comparison method was adopted to estimate the weights of quality dimensions. A pairwise comparison generally is any process of comparing entities in pairs to judge which of each entity is preferred or has a greater amount of quantitative property, or whether or not the two entities are identical. It stems from the Analytic Hierarchy Process (AHP) [29], a famous multi-criteria decision-making framework which is used in the scientific study of preferences, voting systems, and requirements engineering. We applied the pairwise comparison approach [30] to assign relative weights for the key quality factors as follows:

Step 1:

The pairwise comparison matrix was completed. Two factors were evaluated at a time in terms of their relative importance. Index values from 1 to 9 were used. If factor F_i was exactly as important as factor F_j, this pair received an index of 1. If F_i was extremely more important than F_j, the index was 9. All gradations were possible in between, as shown in

Table 2. Index values in pairwise comparison matrix Analytic Hierarchy Process (AHP) scale.

Definition	Index	Definition	Index
Equally important	1	Equally important	1/1
Moderately more important	3	Moderately less important	1/3
Much more important	5	Much less important	1/5
Far more important	7	Far less important	1/7
Extremely more important	9	Extremely less important	1/9

. For a “less important” relationship, the fractions 1/1 to 1/9 were available: if F_i was extremely less important than F_j, the rating was 1/9.

The index values were estimated based on experts’ consensus; for this purpose, a questionnaire was prepared and distributed among the participants who had at least three years of experience with using e-learning websites in universities. The participants were restricted with the amount of experience because only a certain amount of years of experience could enable them to make a decision about the relative significance of factors with respect to e-learning quality. Experts were asked to evaluate the importance of each key quality factor with respect to other quality factors and record their estimates, as shown in Table 2. If the estimates of different experts were different, a consensus method was used to reduce divergence. The final approved estimates were entered row by row into a cross-matrix C (n x n). First, the diagonal of C was filled by values of 1, as per Equation (1). Second, the right upper half of C was filled until each factor had been compared to every other one. If F_i to F_j was rated with the relative importance of m (i.e., $C_{ij}=m)$, F_j to F_i had to be rated with 1/m (i.e., $C_{ji}=1/m)$. Lastly, the lower left half of C was filled with the corresponding fractions, as per Equation (2). (Note that i and j are positive integers ≤ n, $C_{ij}$ is the element of C located in row i and column j).

$$C_{ij}=1, i=j$$

(1)

$$C_{ij}=\frac{1}{C_{ji} }, i\ne j$$

(2)

Step 2:

the normalized comparison matrix was calculated. A normalized comparison matrix $C^{\prime }$ was created by dividing each element in matrix C by the sum of the elements in its column. This is shown in Equation (3).

$$C_{ij}^{\prime }=C_{ij}/{\displaystyle \sum }_{i=1}^n{C}_{ij}$$

(3)

Step 3:

the relative weights of the factors were calculated. To obtain the weight $w_i$ of each individual factor F_i, the mean of each row in $C^{\prime }$ was calculated, as shown in Equation (4).

$$w_i=\frac{1}{n}{\displaystyle \sum }_{j=1}^n{C}_{ij}^{\prime }$$

(4)

These weights were already normalized; their sum was 1, as illustrated in Equations (5) and (6).

$$0\le w_i\le 1$$

(5)

$${\displaystyle \sum }_{i=1}^n{w}_i=1$$

(6)

Step 4:

the consistency of the results of the pairwise comparisons were checked. According to Saaty, if the consistency ratio was less than 10%, then it was acceptable; otherwise, the pairwise comparisons should be revised [31]. The ratio of consistency (CR) is given in Equation (7).

$$CR=\frac{CI}{RI}$$

(7)

where CI is consistency index and is given in Equation (8).

$$CI=\frac{{\lambda }_{max}-n}{n-1}$$

(8)

λ_max is the maximum eigenvalue and n is the rank of the pairwise comparison matrix [28].

RI specifies the random index of consistency, which had a different value based on the number of criteria, as shown in

Table 3. Relationship between RI and n.

N	1	2	3	4	5	6	7	8	9	10
RI	0	0	0.58	0.9	1.12	1.24	1.32	1.41	1.45	1.49

.

A variety of software is available in the market, such as Expert Choice, HIPRE3, Criterium, and AUTOMAN, which can be used to incorporate AHP. However, the Expert Choice software is considered as the standard AHP software [11], which is why it was used in this research. In this study, AHP was implemented using the following steps.

(a)

The research objective was defined. The objective of this study was to evaluate and rank the key quality factors of e-learning websites.

(b)

The objective and the evaluation criteria were organized in a hierarchical structure. The first level (level 1) of hierarchy describes the objective, the second level (level 2) describes the main quality criteria, and the third level (level 3) gives the sub-criteria related to each main criterion. This is shown in Figure 2.

(c)

The relative weights for criteria and sub-criteria were computed following the steps from 1 to 4, as shown above. A pairwise comparison required n (n—1)/2 comparisons [31], where ‘n’ represented the number of criteria or sub-criteria.

4. Findings and Discussion

The findings of the three phases discussed above are discussed below.

4.1. Findings of Phase 1

The SLR of phase 1 resulted in the extraction of a wide range of pertinent quality dimensions, which could play an important role in assessing the quality of e-learning websites. These dimensions were extensively studied and classified into four main quality dimensions, including content, design, organization, and usability.

Table 4. Extracted quality dimensions.

Quality Main Dimensions	Quality Sub-Dimensions	Literature Reference
Content	Timely	[32,33]
	Relevant	[34,35]
	Multilanguage	[28]
	Variety of Presentation	[36,37]
	Accuracy	[38]
	Reliability of Content	[39,40]
Design	Attractive	[32,35]
	Appropriateness	[41]
	Color	[42,43]
	Multimedia elements	[37]
	Text	[44,45]
	Browser Compatibility	[46,47]
Organization	Index	[48]
	Navigation	[49,50]
	Consistency	[51]
	Links	[52]
	Logo	[53,54]
	Domain	[55]
Usability	User friendly	[56,57]
	Reliability	[58,59]
	Availability	[38]
	Interactive features	[41,43]

shows the main quality dimensions and the sub-dimensions of each main dimension.

4.2. Findings of Phase 2

In phase 2, a survey was conducted to evaluate the importance of each quality sub-dimension on a 5-point Likert scale. The survey questionnaire was shared with 157 participants (see Table 1) who were personally approached (via telephone conversation and face-to-face meetings when possible). The objectives of the study and the value of their responses in shaping future web-based e-learning systems were illustrated. In addition to the Google form of the survey, a separate, easy to read and understand tutorial was shared with them. The tutorial provided a clear and concise description of each sub-dimension so that the participants apprehended it well before responding. All the participants responded, and only three of the 157 surveys were found to be incomplete and vague; thus, they were excluded from the data analysis. A total of 22 sub-dimensions were evaluated, and the data obtained through the questionnaire were analyzed using the statistical software package (SPSS). The results are shown in

Table 5. Analysis of data collected through the survey.

Criteria	Sub-Criteria		Contribution Level					Mean	St. Deviation
			Very Important	Important	Neutral	Less Important	Not Important
Content	Timely	154	90	57	4	2	1	4.51	0.679
		%	58.4	37.0	2.6	1.3	0.6
	Relevant	154	62	74	16	2	0	4.27	0.698
		%	40.3	48.1	10.4	1.3	0
	Multilingual	154	39	45	40	24	6	3.56	1.143
		%	25.3	29.2	26.0	15.6	3.9
	Variety of Presentation	154	54	62	25	12	1	4.01	0.943
		%	35.1	40.3	16.2	7.8	0.6
	Accuracy	154	56	59	28	8	3	4.02	0.967
		%	36.4	38.3	18.2	5.2	1.9
	Reliability of Content	154	56	58	17	17	6	3.92	1.126
		%	36.4	37.7	11.0	11.0	3.9
Design	Attractive	154	75	56	10	6	7	4.21	1.040
		%	48.7	36.4	6.5	3.9	4.5
	Appropriateness	154	36	60	26	31	1	3.64	1.071
		%	23.4	39.0	16.9	20.1	0.6
	Color	154	40	68	28	10	8	3.79	1.064
		%	26.0	44.2	18.2	6.5	5.2
	Multimedia Elements	154	64	49	25	10	6	4.01	1.094
		%	41.6	31.8	16.2	6.5	3.9
	Text	154	52	73	17	11	1	4.06	0.891
		%	33.8	47.4	11.0	7.1	0.6
	Browser Compatibility	154	68	51	22	11	2	4.12	0.990
		%	44.2	33.1	14.3	7.1	1.3
Organization	Index	154	73	56	14	10	1	4.23	0.913
		%	47.4	36.4	9.1	6.5	0.6
	Navigation	154	60	63	19	10	2	4.10	0.941
		%	39.0	40.9	12.3	6.5	1.3
	Consistency	154	38	46	43	22	5	3.58	1.107
		%	24.7	29.9	27.9	14.3	3.2
	Links	154	59	58	18	19	0	4.02	1.000
		%	38.3	37.7	11.7	12.3	0
	Logo	154	52	69	22	9	2	4.04	.914
		%	33.8	44.8	14.3	5.8	1.3
	Domain	154	38	62	34	16	4	3.74	1.028
		%	24.7	40.3	22.1	10.4	2.6
Usability	User Friendly	154	106	37	9	0	2	4.59	0.720
		%	68.8	24.0	5.8	0	1.3
	Reliability	154	54	75	19	5	1	4.14	0.804
		%	35.1	48.7	12.3	3.2	0.6
	Availability	154	54	55	32	11	2	3.96	0.983
		%	35.1	35.7	20.8	7.1	1.3
	Interactive Features	154	56	63	23	8	4	4.03	0.980
		%	36.4	40.9	14.9	5.2	2.6

.

As the target of MCDM methods, adopted one of them in this study (i.e. AHP), is to compare limited factors with each other. While working in MCDM, detailed literature review produces a large list of criteria and sub criteria which needs to be reduced because it further has to be presented to respondents. So, there is a need of cutoff point to eliminate some less important factors after conducting survey in the second phase and before applying the pairwise comparison of AHP in the last phase. The less important factors are eliminated using some threshold value/cutoff that is selected on the basis of experts and researchers choice. So, after reviewing all the responses and consulting with experts in the area, mean value 4 was considered most appropriate as threshold value, as if we set less than 4, very few factors were supposed to eliminated and if set it greater than 4 then most of the important factors were expected to be eliminated.

The results in Table 5, shows that ‘Timely’ quality sub-dimension has been perceived very important by most of the respondents 58.4% and 37% respectively, only one participant considered it unimportant with respect to content. The mean value of ‘Timely’ is also highest (4.51). Among the sub-dimensions of content, ‘Multilingual’ has been found the least important one with lowest mean importance value 3.56. Similarly, with respect to ‘Design’ dimension, the ‘Attractive’ sub-dimension has been professed very important by most of the respondents, 48.7% and 36.4% correspondingly. Only few participants stated that it is less important (3.9%) and unimportant (4.5%). The mean importance value of ‘Attractive’ sub-dimension is the highest (4.21). Among sub-dimensions of ‘Design’, ‘Appropriateness’ has been found the least important with lowest mean 3.64. With regard to ‘Organization’ dimension, the ‘Index’ quality sub-dimension was observed very important by 47.4% and important by 36.4% participants, only one participant (0.6%) reflected that it is unimportant in terms. The mean value of ‘Index’ has been also found the highest (4.23). Among sub-dimensions of ‘Organization’, ‘Consistency’ was the least important quality sub-dimension with lowest mean 3.58. In terms of ‘Usability’, ‘User friendliness’ has been identified very important sub-dimension by majority of the respondents (69%) with highest mean importance value of 4.59. ‘Availability’ has been recognized as the least important quality sub-dimension with lowest mean importance value of 3.96. Only those sub-dimensions whose mean importance value equals or exceeds 4 are identified as key factors (shown in green) and further considered in phase 3. Thus, sub-dimensions with mean importance value less than 4 (shown in red) are excluded.

4.3. Findings of Phase 3

In phase 3, AHP was applied to estimate the relative weights and to rank the key quality dimension (aka criteria) and sub-dimensions (aka sub-criteria). The AHP was implemented through Expert Choice software. The steps of implementation and the corresponding findings were discussed below.

(a)

Defining the goal: the first step in AHP was to define the goal or the objective of the study. The goal of this study was to propose a hierarchical quality model that evaluated the quality of e-learning websites.

(b)

Organizing the objective and the evaluation criteria in the hierarchical structure. Based on the findings of phase 1 and phase 2, shown in Table 4 and Table 5, the key quality factors comprised criteria and sub-criteria. The main quality criteria of e-learning websites are content, design, organization, and usability; these criteria comprised a total of 22 sub-criteria. Out of the 22 sub-criteria, only 15 were considered in phase 3, while seven with mean values less than 4 were excluded. The criteria excluded were multilingual, reliability of content, appropriateness, color, consistency, domain, and availability, as shown in Figure 3. The remaining 15 sub-criteria, along with their main criteria, were inserted in Expert Choice software to construct the hierarchical quality model, as shown in Figure 4.

(c)

Compute the relative weights of the key criteria and sub-criteria. In this step, pairwise comparisons have been performed as described in above steps from 1 to 4. Pairwise comparison process is the core of AHP which is a widely used technique in Multi-Criteria Decision Making (MCDM). The most critical step here is step 1 where experts have to make consensus on the index values entered to the pairwise comparison matrices. Experts were asked to fill in a pairwise comparison matrix that compares the importance of the four main criteria with respect to each other according to the scale shown in Table 2. Similarly, experts have to complete pairwise comparison matrices that compare the importance of sub-criteria of each main criterion to each other. For this purpose, a questionnaire has been prepared and distributed among participants who have at least three years of experience on using e-learning websites. The questionnaire has been distributed among 51 participants including students, academicians, web developers and educational administrators. All participants have responded to the questionnaire and their estimate entered to Expert Choice software as shown in below figures. The below figures show different screenshots of Expert Choice software which represent the pairwise comparison indices assigned by the experts when comparing different criteria against each other. Figure 5 present the relative importance of Content with respect to Usability as shown by the weights assigned by participants to these factors. Figure 6 showed the Content with respect to Organization along with weights given by participants to both factors. Figure 7 illustrates Content with respect to Design and Figure 8 showed the relative importance of Organization as compared to Design. In the same way, the relative importance of the Usability vs. Design and Organization vs. Design has been assessed through Expert choice. The consistency test [35] has also been performed in Expert Choice software. The survey responses which have consistency more than 10% were excluded. As a result, 30 responses were excluded out of 51. The remaining 21 responses were used to compute the final weights.

In order to complete all steps of the pairwise comparisons, the Expert Choice software was deployed to estimate the relative weights (priority) of the criteria and sub-criteria. These weights were the final relative importance value of each quality evaluation criteria and sub-criteria. The weights were categorized into “global weights”, which were the weights of the four main criteria, and “local weights”, which were the relative weights of sub-criteria. The global weights are shown on the right-hand side of the screenshot of Figure 5 and listed in

Table 6. Global weights and global rank of main criteria.

Criteria	Weights (Global)	Rank (Global)
Content	0.407	1
Usability	0.294	2
Organization	0.188	3
Design	0.111	4

. On the other hand, local weights are shown in

Table 7. Local weights and rank of sub-criteria of content.

Sub-Criteria	Weights (Local)	Rank (Local)
Timely	0.302	1
Accuracy	0.276	2
Variety of Presentation	0.234	3
Relevant	0.188	4

,

Table 8. Local weights and rank of sub-criteria of usability.

Sub-Criteria	Weights (Local)	Rank (Local)
User Friendly	0.401	1
Interactive Features	0.394	2
Reliability	0.206	3

,

Table 9. Local weights and rank of sub-criteria of organization.

Sub-Criteria	Weights (Local)	Rank (Local)
Index	0.313	1
Logo	0.301	2
Links	0.237	3
Navigation	0.149	4

and

Table 10. Local weights and rank of sub-criteria of design.

Sub-Criteria	Weights (Local)	Rank (Local)
Attractive	0.389	1
Text	0.278	2
Browser Compatibility	0.191	3
Multimedia Elements	0.142	4

. The local and global weights were computed in the Expert Choice software in accordance with Equations (3)–(5). The global weights included the weighted importance value for content, design, organization, and usability, and the local weights were relative importance values for the 15 sub-criteria.

As shown in Table 6, the weight of content was 0.407, which indicated that it was the most important criterion. The weight of the usability criteria was found to be 0.294, which represented the second most important criterion to evaluate the quality of e-learning websites. The overall weights and ranking of the main criteria are shown below.

The local weights of the sub-criteria of the four main criteria (i.e., content, design, usability, and organization) along with their ranks are shown in Table 7, Table 8, Table 9 and Table 10. Table 7 shows that timely was the most important sub-criteria in the content category, with the highest weight of 0.302, while relevant was at the lowest position and its weight was 0.188. Table 8 shows that user friendly was the most important criteria in usability, with the highest weight of 0.401, while reliability was the least important sub-criterion, with the lowest local weight of 0.206. As shown in Table 9, index was the most important sub-criterion of organization with the highest weight of 0.313, while navigation was observed to be the least important sub-criterion with the local weight of 0.149. Finally, Table 10 shows that attractive was the most important sub-criterion in design, with the highest weight of 0.389, while multimedia elements were found to be the least important sub-criterion, with the lowest local weight of 0.142. The above results are aggregated in

Table 11. Overall weights and rankings of criteria and sub-criteria.

Factors	Global Weight with Contribution %	Global Ranking of Factors	Sub-Factors	Local Weight with Contribution %	Local Ranking of Sub Factors
Content	0.407 (40.7%)	1	Timely	0.302 (30.2%)	1
			Accuracy	0.276 (27.6%)	2
			Variety of Presentation	0.234 (23.4%)	3
			Relevant	0.188 (18.8%)	4
Usability	0.294 (29.4%)	2	User Friendly	0.401 (40.1%)	1
			Interactive Features	0.394 (39.4%)	2
			Reliability	0.206 (20.6%)	3
Organization	0.188 (18.8%)	3	Index	0.313 (31.3%)	1
			Logo	0.301 (30.1%)	2
			Links	0.237 (23.7%)	3
			Navigation	0.149 (14.9%)	4
Design	0.111 (11.1%)	4	Attractive	0.389 (38.9%)	1
			Text	0.278 (27.8%)	2
			Browser Compatibility	0.191 (19.1%)	3
			Multimedia Elements	0.142 (14.2%)	4

.

5. Proposed Model for Quality Evaluation

The objective of this research study is to develop a hierarchical quality model that could be used to evaluate the quality of e-learning websites. Thus, the main outcome this research is a hierarchal model to evaluate the quality of e-learning websites, this model is shown in Figure 9.

‘Content’ is found as most significant quality factor, the basic objective of e-learning systems is to facilitate and help learners by imparting them with apposite learning content and services to meet their learning needs [60]. The content factor is related to the learning material and the features presented by e-learning websites to learners. Therefore, this factor plays an important role in the success of web based e-learning systems as an effective content is the main concern of learners when visiting an educational system. The related sub-factors are also important as all of them emphasize on the effectiveness of content so that quality learning experience could be delivered. Firstly, ‘Timely’ sub-factor has been noticed to be the most important factor among others, it indicates that the content of the website should be updated on regular basis to reflect the latest development and progress made in the field or area of study [31]. Secondly, the content should always be accurate and authentic so that e-learning system would be perceived as trustworthy learning source. Thirdly, the learning material should be presented using variety of resources including video, images, graphics, audio, animation and simulation to add value and to accommodate the learning needs of learners with diverse learning preferences [31]. Lastly, the variety of learning material should be relevant to the topic being discussed and to the level of students to whom the contents are being delivered. The finding of this research is similar to previous research in which emphasis is also given on ‘Content’ and declared it as one of the most important factors to evaluate the quality of web based e-learning systems [61].

The second most important quality factor in the proposed hierarchical model was usability. Usability is the ability of a system to help users in achieving their goals effectively, efficiently, and satisfactorily within the context of use [60,62]. Thus, web-based e-learning systems should support the learning process and allow learners to efficiently use the system. It should also be appropriate for the intended learning tasks [61]. The related sub-factors were important, as they emphasized the usability of the system. The user friendly sub-factor indicated that the interface should be easy to use, allow learners to achieve learning objectives when they use e-learning systems for first time, and help them to perform learning tasks quickly in the future [62]. The e-learning websites should consist of “interactive features” that allow learners to fully manipulate the course and have control over the learning objects [61]. Reliability indicated that the web-based e-learning system should be accessed by a user anywhere and anytime using a diverse range of devices and platforms. Previous research supports the finding of this study. It is underlined that every software system has some functionality that is provided to its users, such as teaching/learning functionality in e-learning systems. However, the value of system functionality is noticeable only when the system can be effectively and efficiently utilized by its users. The real effectiveness of a software is realized when there is an appropriate balance between the functionality and usability of the system. Usability is, therefore, a significant factor which complements the system functionality and enables users to engage with the system to perform all actions and services offered by the system. In the absence of such factors, users remain unable to take full advantage of the system’s services and will not enjoy the system while using it [62]. To achieve usability, human-centered methodology may be adopted in the development process [63].

The third important quality factor is organization, which is related to the overall structure of the website. It describes how the important aspects of a web-based system are organized. The related sub-factors emphasized the need for an appropriate structure and the organization of educational websites so that it is accessible to all kinds of learners. Index is the main page of the website; it should present all learning categories and the classification of learning materials so that learners can easily access the intended learning path [48,52]. The logo should be available on e-learning websites to indicate that the learning content is being delivered from authentic and credible sources, which is a deep concern for learners of this information age [53,54,64,65]. The “links” or hyperlinks on websites must be active, as the dead links irritates the learners [54]. On long pages, a list of contents with links should be available at the top of the page to take learners to the corresponding content farther down the web page [66]. The links should be designed with glosses as short phrases to guide the learners to what is actually located behind the links. Lastly, navigation is a method used to find information on a web-based e-learning system. A navigation page assists learners in locating and connecting to desired pages. The navigation scheme should be designed in a way that allows learners to locate and access learning material effectively and efficiently [66].

Finally, the quality factor design is concerned with the design and visual aspects of the website. It emphasizes the requirement that the important items of the e-learning website should be placed consistently and aligned appropriately. The design of the website must ensure that the web pages show a reasonable amount of white space, as too much white space can demand substantial scrolling, whereas too little may offer a presentation that looks too busy. The sub-factors of design include attractiveness, text, browser compatibility, and multimedia elements. The overall design of the website should be attractive to retain the learner’s attention for a longer period of time so that they spend most of their time with the learning application [32,35,66]. The text should be presented with a font size, style, foreground, and background color that can support the learners in reading and comprehending the learning materials rather than hindering them [53,54,65]. Different learners have different browser features and different defaults: for example, learners with visual impairments have a tendency to select larger fonts, whereas some other learners may turn off backgrounds, use fewer color options, or override the font. Therefore, the e-learning web-based system should be compatible with common browsers and with browser settings most commonly used by the learners [46,47,66]. Moreover, the design of web-based e-learning system should have the support of multimedia elements, as these elements enforce the user to concentrate on the learning items [66]. In short, for effective solutions, there is a need to rethink the customer’s needs and innovate.

6. Conclusions and Future Work

The quality of a product or a service directly affects the satisfaction of its user. E-learning websites are continually being designed even by developing countries to reinforce their education systems. The basic motive behind such rapid development of web-based e-learning resources is to achieve academic excellence. However, these websites suffer deficiency and ineffectiveness in their usage, owing to their poor quality. An evaluation of an e-learning website in terms of quality is a challenging task. Nevertheless, the quality level of e-learning websites can be investigated with the help of any suitable framework or model. This study has extensively reviewed the literature to identify the key quality factors that would be used to evaluate the quality of e-learning websites. The AHP technique has been used to prioritize the quality factors for the evaluation of e-learning websites. Finally, an effective evaluation model has been proposed. However, there are certain limitations of this research that should be addressed in the future. To achieve a quality evaluation model, this study only considers stakeholders from the urban areas of developing countries. The stakeholders belong to a mainstream community of students and teachers; however, there are many others with different incapacities and disabilities who are not included in this study. The students also differ from each other in terms of their learning abilities, capacities, orientation, and emotions, which are also not considered. The future research would be planned with two independent studies that consider a large sample from rural environments, as well as urban environments of developing countries to reveal if there are any divergent results. Additionally, the individual differences of learners and their special needs should be considered so that a universal quality evaluation model can be proposed. E-learning systems are mainly web-based systems that are open to computer threats, so security is an important concern in these systems and will help in leveraging overall cybersecurity [67]. The e-learning is penetrating into education at a very fast pace. Information should be actively secured to prevent the loss of privacy, availability, and integrity [68]. It is, therefore, vital to identify and understand the threats to the systems in order to develop secure e-learning systems. Web based e-learning systems should explore with respect to cloud computing technologies which have changed the way to develop and access applications. Moreover, instead of AHP, there are other multi-criteria decision analysis techniques that exist, such as Fuzzy AHP and TOPSIS, which should be explored in future research.