Evaluation of Open and Distance Education Websites: A Hybrid Multi-Criteria Decision-Making Approach

Abstract

Higher education institutions and organizations have new opportunities thanks to digital technologies. Universities worldwide seek to provide the most outstanding available student services, particularly those that promote student achievements in their program objectives. The World Wide Web, in particular, has advanced Internet-based information technology, dramatically impacting all types of education delivery. Therefore, it has rapidly expanded the Open and Distance Education (ODE) system. This study aims to evaluate the performance of the higher institutions’ ODE websites within 5 main criteria (Navigation, Accessibility, Design, Content Readability, and Announcements) and 20 sub-criteria. The case study has taken place in Türkiye, and the institutions available for the study were Anadolu University ODE; Ankara University ODE; Ataturk University ODE; Istanbul University ODE. This paper utilized two Multi-criteria Decision Making (MCDM) techniques: Fuzzy Analytical Hierarchy Process (Fuzzy AHP) and Fuzzy Weighted Aggregated Sum Product Assessment (Fuzzy WASPAS). The criteria were determined with the help of the literature, which was searched, then categorized and weighted with Fuzzy AHP. The evaluation step in the process was conducted with Fuzzy WASPAS to select the best-performed alternative ODE websites. According to research findings, Design is the essential criterion, followed by Accessibility, Content Readability, Announcement, and Navigation criteria. Our research identified and recommended the development areas for further research and proposed theoretical and practical implications as well as managerial decisions to be considered for the ODE website improvements.

1. Introduction

New opportunities have emerged in many sectors, including education and training, due to the expeditious development of computer and communication technology. Digital technologies create new avenues for educational organizations, likewise higher educational institutions. They offer services in different levels of programs, such as undergraduate, undergraduate completions, and associate degrees, along with proper delivery methods on campus-based, online, open-distance, and hybrid modes. Due to the advancement of Internet-based information technology, particularly the World Wide Web, the emerging Open and Distance Education (ODE) system is expanding rapidly which has significant effects on all forms of education delivery. The phrase “open and distance learning” refers to the reality that all or most of the instruction is given by a person who is not the learner, and the mission’s goal is including greater integration and communication, whether in terms of accessibility, curriculum, or other structural components. In order to make the entire educational system adaptable, learners and teachers cannot coexist in the same space where teaching and learning meet and must be physically separated from one another. In comparison to conventional online courses, online learning places a greater emphasis on the need for learners to be able to control (self-regulate) their own learning [1]. Self-regulated learning (SRL) is one of the critical elements that has been shown to positively influence students’ success in conventional online education settings [2]. When defining open and distance education or online education/learning, there is some confusion in the terms referred to. Open distance education or Open University are entire degree programs, and all activities run online. In contrast, distance education could be part of any degree program and involve an online course or supportive medium along with an educational program’s face-to-face (on campus) mode.

Distance education schools offer their programs in two main categories, “synchronous” and “asynchronous” learning programs [3]. The first online model is more interactive with instructors and classmates. Students use the institution’s website to log on at the scheduled time and access all related multimedia sources. The second model is organized by self-study mode; students are responsible for completing the degrees.

Website performance is one of the critical problems for the open education system. Infrastructure, such as internet access at all of the university’s dedicated learning facilities, is necessary for students to make the best use of the institution’s digital e-services [4]. The Internet provides excellent opportunities for higher education institutions with diverse program content, and website design with various features is vital for effective delivery and the value of trust in the platforms [5]. Therefore, the design enhances online educational resources [6]. Similarly, Oppong et al. (2013) highlighted the importance of resource sharing, content availability, and accessibility across the used website [7]. In Garg (2017)’s study, the website criteria was widely reviewed and highlighted, such as human elements with technical skills and aspects, infrastructure, organizational support, website functionalities (usability and accessibility), content availability and quality, system support and maintenance, design and appearance, ease of use, navigation, communication, interactivity, and multimedia design [8]. High-quality web-based information should be defined to improve online programs, interactive websites, and construction [9,10]. Arif et al. (2018) explored the factors influencing students’ use of web-based services and applied the “Unified Theory of Acceptance and Use of Technology (UTAUT)” [11]. The results revealed that students’ behavior intentions to utilize web-based services are significantly predicted by “performance expectations”, “effort expectations”, and “social influence”.

Open institutions should be well-placed to exhibit visible leadership in these fields. Analyzing the content of the Open University website reveals how prevalent this is [12]. Moreover, online open education services increasingly rely on web-based technologies [11]. These technologies are considered for teaching and supportive university services such as the library, facilities, learning centers, labs, and clubs. Since the accessibility of online materials is a vital component of open education, libraries have considered designing their infrastructure to enhance program outcomes [13].

Furthermore, Carvalho and Gasque (2018) proposed the development of skills and abilities of teachers and librarians whose informational literacy also plays a critical role in compliance with the infrastructure [14]. Similarly, Kimmons (2014) contributed that K–12 teachers’ levels of open education literacy should be enhanced to support programs’ efficiency by providing open resources accessibility and utilization [15]. In a 2009 study by McKinney et al., undergraduate psychology students took part in one of two circumstances to test the efficacy of audio lessons in higher education while students were unable to attend regular class sessions [16]. According to the results, students who listened to a podcast and took notes performed much better than those who only attended lectures.

Kabak et al. (2017) studied the evaluation of distance learning websites using the multi-criteria decision techniques ANP and TOPSIS to select the best website [17]. A similar study for the European open education system compared the two leading platforms [18] in terms of texts and visuals, which play a critical role in creating and deploying the data to reach proper operation performance levels. Online learning and open education are also emerging in medical education [19]. They studied the different collaborative software tools for virtual 3D anatomical models in a lab-based fully online course. E-learning websites are evaluated by the quality of content, platform structure, virtual environment, features, communication mediums, accessibility, adaptability, and interoperability perspectives [20,21,22]. In addition, the e-services that universities provide to their learners create better environments and boost the outcomes of the activities. These efforts should cover not only the course context but also the supplementary services which support the university’s online platforms from a broader perspective [23].

Previous studies also support the significance of social media in developing these. Moreover, Pang (2022) proposed the web-based flipped learning approach using social media as an emerging technology for learners’ engagement and critical thinking [24]. With social media usage and website platform support, students are more cooperative and collaborative in their field of study. Content-rich platforms also support distance education with various types of tasks to complete.

This study aims to evaluate the performance of the higher institutions’ ODE websites within 5 main criteria (navigation, accessibility, design, content readability, and announcements) and 20 sub-criteria in Türkiye. The paper also focuses on revealing the best alternative ODE in terms of service offerings and qualities according to the evaluated criteria [25]. Based on the scope and objectives of the study, the research questions are formed as follows:

• RQ1. What are the performance criteria for the evaluation of websites?
• RQ2. What are the weights of performance criteria, and how should the alternatives be prioritized?
• RQ3. How can the best-performing websites be selected using fuzzy methods?

The paper is structured as follows: an introduction which consists of a literature review, materials and methods (a hybrid Multi-Criteria Decision-Making using Fuzzy AHP and Fuzzy WASPAS methods), a case study, and discussions and conclusions. The criteria were determined with the help of the literature which was searched, then categorized and weighted with Fuzzy AHP. The evaluation step in the process was conducted with Fuzzy WASPAS to select the best-performing alternative ODE websites. Finally, the Section 4 includes limitations of the study and potential future research directions.

2. Materials and Methods

The study provides an evaluation approach that prioritizes the relative weights of various distance education websites. The proposed approach consists of four sequential steps. In the first step, 20 different sub-criteria under 5 main criteria are derived from the existing literature [26,27,28,29]. Identified criteria are weighted by stakeholders of distance education websites using the Fuzzy Analytical Hierarchy Process (Fuzzy AHP) outlined in the second step. Further, the third step applies the Fuzzy Weighted Aggregated Sum Product Assessment (Fuzzy WASPAS) method to rank the Turkish Open and Distance Education websites. Finally, the sensitivity analysis is conducted to ascertain the effect on the consequences of a change in a model.

The study’s primary purpose is to select the best educational institution among four serving Turkish Higher Education Institutions. The fuzzy approach is adopted in decision-making since the selection criteria are not defined as numerical values. The methodology used for the study is illustrated in Figure 1 below.

2.1. The Fuzzy AHP Method

The Fuzzy AHP method has been introduced by using fuzzy logic [30,31] and the classical AHP method [32,33,34,35] together. In the Fuzzy AHP method, pairwise comparisons are made by expressing the opinions of experts in the field with fuzzy numbers. As in the classical AHP, exact values are not used.

The Fuzzy AHP method is frequently used to eliminate uncertainty in decision processes. In the literature, there are many Fuzzy AHP methods, some of the well-known methods are Van Laarhoven and Pedrycz’s (1983), Buckley’s (1984), Enea and Piazza’s (2004), and Chang’s (1992; 1996) extended analysis methods [36,37,38,39,40]. Chang’s method is the most widely used in the field. This study explains this method in detail since the solution is carried out with Chang’s method. Chang’s extended analysis method has become the most preferred method in the literature because it is very similar to the classical AHP solution steps and is easy to process. In Chang’s method [39], verbal comparisons between criteria are represented by fuzzy triangular numbers (

Table 1. Fuzzy Numbers Corresponding to Linguistic Expressions.

Verbal Importance	Triangular Fuzzy Numbers	Inverse Triangular Fuzzy Numbers
Equal importance	(1,1,1)	(1/1,1/1,1/1)
A little more important	(1,3,5)	(1/5,1/3,1/1)
Strongly important	(3,5,7)	(1/7,1/5,1/3)
Very strongly important	(5,7,9)	(1/9,1/7,1/5)
Totally important	(7,9,9)	(1/9,1/9,1/7)

).

Step 1:

The fuzzy synthetic order value for the criteria is calculated by Equation (1) below.

$$S_i = {\displaystyle \sum }_{j=1}^m{M}_{gi}^j*{\left[{\displaystyle \sum }_{i=1}^n{\displaystyle \sum }_{j=1}^m{M}_{gi}^j\right]}^{-1}$$

(1)

To obtain the ${\displaystyle \sum }_{j=1}^m{M}_{gi}^j$ value in the equation, the following addition is performed.

$${\displaystyle \sum }_{j=1}^m{M}_{gi}^j = ({\displaystyle \sum }_{j=1}^m{l}_j,{\displaystyle \sum }_{j=1}^m{m}_j,{\displaystyle \sum }_{j=1}^m{u}_j)$$

(2)

The expression $\left[{\displaystyle \sum }_{i=1}^n{\displaystyle \sum }_{j=1}^m{M}_{gi}^j\right]$ in Equation (1) is calculated according to Equation (3).

$${\left[{\displaystyle \sum }_{i=1}^n{\displaystyle \sum }_{j=1}^m{M}_{gi}^j\right]}^{-1} = \left(\frac{1}{{{\displaystyle \sum }}_i^n{u}_i},\frac{1}{{{\displaystyle \sum }}_i^n{m}_i},\frac{1}{{{\displaystyle \sum }}_i^n{l}_i}\right)$$

(3)

Step 2:

The probability of the expression M₂ = (l₂, m₂, u₂) ≥ M₁ = (l₁, m₁, u₁) in obtaining the synthesis values is defined as follows:

$$V\left(M_2\ge M_1\right) = \begin{array}{cc}1& m_2\ge m_1\\ 0& l_1\ge u_2\\ \frac{l_1-u_2}{(m_2-u_2)-\left(m_1-l_1\right)}& other\end{array}$$

(4)

In order to compare M₁ and M₂, both the values of V(M₂ ≥ M₁) and V(M₁ ≥ M₂) are required.

Step 3:

All rows are compared among themselves. The values obtained as a result of these comparisons are taken as the minimum. Likewise, the second row is compared with all other rows, and the minimum is taken. This process is continued for all rows. The weight vector is obtained by combining the minimum values found for each row.

$$W^{\prime } = (d^{\prime }\left(A_1\right), d^{\prime }\left(A_2\right),\dots ,d^{\prime }{\left(A_n\right)}^T$$

(5)

Step 4:

W is calculated by normalizing the weight vector obtained in step 3.

$$W = (d^{\prime }\left(A_1\right), d^{\prime }\left(A_2\right),\dots ,d^{\prime }{\left(A_n\right)}^T$$

(6)

W ceases to be a fuzzy number.

$$d\left(A_i\right) = \frac{d^{\prime }\left(A_i\right)}{{{\displaystyle \sum }}_{i=1}^n{d}^{\prime }\left(A_i\right)}$$

(7)

2.2. The Fuzzy WASPAS Method

WASPAS is a multi-criteria decision-making approach that combines the results of two different models, the “Weighted Sum Model (WSM)” and the “Weighted Product Model (WPM)”. The Fuzzy WASPAS Method was proposed by Zavadskas et al. (2012) as one of the multi-criteria decision-making methods [41,42]. Later, this method was developed by Zavadskas et al. (2014) using heuristic fuzzy numbers with interval values [43]. Ghorabaee et al. (2016) extended the WASPAS method by using Interval Type-2 fuzzy numbers [44]. The steps of the Fuzzy WASPAS method [45] were defined as follows:

Step 1:

Construction of the fuzzy decision matrix: The fuzzy performance value ($\tilde{x}ij$) and weight (${\tilde{w}}_j$) of alternative i according to criterion j form the inputs of the decision matrix. There are m alternatives and n criteria.

$$\tilde{X} = \left[\begin{array}{ccc}\widetilde{\begin{array}{c}{X}_{11}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{X}_{1j}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{X}_{1n}\\ ⋮\end{array}}\\ \widetilde{\begin{array}{c}{X}_{i1}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{X}_{ij}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{X}_{in}\\ ⋮\end{array}}\\ \widetilde{\begin{array}{c}{X}_{m1}\\ \end{array}}\cdots & \widetilde{\begin{array}{c}{X}_{mj}\\ \end{array}}\cdots & \widetilde{\begin{array}{c}{X}_{mn}\\ \end{array}}\end{array}\right]i = \overline{1, m}, j=\overline{1, n}$$

(8)

Step 2:

Generation of the normalized decision matrix: Normalization operations are performed for all initial values of $\tilde{x}ij$.

$$\widetilde{{\overline{x}}_{ij}} = \{\begin{array}{c}{\tilde{x}}_{ij}=\frac{\tilde{x}ij}{ma{x}_i\tilde{x}ij} if benefical criterion \\ {\tilde{x}}_{ij}= \frac{mi{n}_i\tilde{x}ij}{\tilde{x}ij} if non-benefical criterion\end{array} i= \overline{1, m}, j= \overline{1, n}$$

(9)

Step 3:

Calculation of the weighted normalized fuzzy decision matrix (${\widehat{X}}_q$) for the Weighted Sum Model (WSM):

$$\widetilde{ {\widehat{X}}_q} = \left[\begin{array}{ccc}\widetilde{\begin{array}{c}{\widehat{X}}_{11}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{\widehat{X}}_{1j}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{\widehat{X}}_{1n}\\ ⋮\end{array}}\\ \widetilde{\begin{array}{c}{\widehat{X}}_{i1}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{\widehat{X}}_{ij}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{\widehat{X}}_{in}\\ ⋮\end{array}}\\ \widetilde{\begin{array}{c}{\widehat{X}}_{m1}\\ \end{array}}\cdots & \widetilde{\begin{array}{c}{\widehat{X}}_{mj}\\ \end{array}}\cdots & \widetilde{\begin{array}{c}{\widehat{X}}_{mn}\\ \end{array}}\end{array}\right]$$

(10)

$${\tilde{\widehat{x}}}_{ij}={\tilde{\overline{x}}}_{ij}{\tilde{w}}_j; i=\overline{1, m}, j=\overline{1, n}$$

Calculation of the weighted normalized fuzzy decision matrix (${\widehat{X}}_p$) for the weighted product model (WPM):

$$\widetilde{ {\widehat{X}}_p} = \left[\begin{array}{ccc}\widetilde{\begin{array}{c}{\stackrel{=}{X}}_{11}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{\stackrel{=}{X}}_{1j}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{\stackrel{=}{X}}_{1n}\\ ⋮\end{array}}\\ \widetilde{\begin{array}{c}{\stackrel{=}{X}}_{i1}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{\stackrel{=}{X}}_{ij}\\ ⋮\end{array}}\begin{array}{c}\cdots \\ \ddots \end{array}& \widetilde{\begin{array}{c}{\stackrel{=}{X}}_{in}\\ ⋮\end{array}}\\ \widetilde{\begin{array}{c}{\stackrel{=}{X}}_{m1}\\ \end{array}}\cdots & \widetilde{\begin{array}{c}{\stackrel{=}{X}}_{mj}\\ \end{array}}\cdots & \widetilde{\begin{array}{c}{\stackrel{=}{X}}_{mn}\\ \end{array}}\end{array}\right]$$

(11)

$${\widetilde{\stackrel{=}{x}}}_{ij} = {\tilde{\overline{x}}}_{ij}{\tilde{w}}_j; i=\overline{1, m}, j=\overline{1, n}$$

Turskis et al. (2015) proposed the following method for integrating the criteria weights obtained separately from multiple decision-makers [45]. The fuzzy group weights for n criteria are obtained as follows: $\tilde{W} = \left[{\tilde{w}}_1, {\tilde{w}}_n\right] = \left[{\tilde{w}}_j\right]$ (${\tilde{w}}_j$ is a fuzzy triangular number):

$${\tilde{x}}_j = \left(x_{j\alpha }, x_{j\beta }, x_{j\gamma }\right)$$

The minimum possible value of j criteria weight is: $w_{j\alpha } = \begin{array}{c}min y_{jk}\\ k\end{array}$, $y_{jk}$ is j criteria weight determined by k expert, $j = \overline{1, n}, k = \overline{1,p}$;

The most probable value of the j-criterion weight: $w_{j\beta } = {\left({\displaystyle \prod }_{k=1}^p{y}_{jk}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$p$}\right.}, j = \overline{1, n}$;

The maximum possible value of the j criterion weight: $w_{j\gamma } = \begin{array}{c}max y_{jk}\\ k\end{array}$.

$$j = \overline{1, n}, k = \overline{1,p}$$

Step 4:

Calculation of the values of the optimality function:

(a)

It is calculated according to the weighted sum model for each alternative as follows.

$${\tilde{Q}}_i = {\displaystyle \sum }_{j=1}^n{\tilde{\widehat{x}}}_{ij};i = \overline{1, m}$$

(12)

(b)

It is calculated according to the Weighted Product Model for each alternative as follows.

$${\tilde{P}}_i = {\displaystyle \prod }_{j=1}^n{\widetilde{\stackrel{=}{x}}}_{ij};i = \overline{1, m}$$

(13)

The values of ${\tilde{Q}}_i$ and ${\tilde{P}}_i$, which are the result of fuzzy performance measurement for each alternative, are fuzzy numbers. Defuzzification of fuzzy numbers is done with the following equations.

$$Q_i = \frac{1}{3}\left(Q_{i\alpha },Q_{i\beta }, Q_{i\gamma }\right)$$

$$P_i = \frac{1}{3}\left(P_{i\alpha },P_{i\beta }, P_{i\gamma }\right)$$

Step 5:

The integrated utility function value for each alternative of the Fuzzy WASPAS method can be determined as follows.

$$K_i = \lambda {\displaystyle \sum }_{j=1}^m{Q}_i+\left(1-\lambda \right){\displaystyle \sum }_{j=1}^m{P}_i, \lambda =0,\dots , 1, 0\le K_i\le 1$$

When determining the λ value, it is assumed that the Weighted Sum Model for all alternatives should equal the Weighted Product Model scores.

$$\lambda = \frac{{{\displaystyle \sum }}_{i=1}^m{P}_i}{{{\displaystyle \sum }}_{i=1}^m{Q}_{i }+{{\displaystyle \sum }}_{i=1}^m{P}_i}$$

Step 6:

Finally, the alternatives are ranked according to their Ki values. The alternative with the most significant Ki value is the most suitable. For the alternatives, the order of conformity can be established by considering the Ki values.

3. Case Study Analyses and Results

In this section, the case study was conducted with the help of proven methodologies for evaluating Turkish ODE websites. The section is structured into four sub-sections. First, the background information regarding the four main Turkish ODE institutions was provided in detail. Second, the evaluation criteria for the main and subcategories were determined. Third, the evaluation criteria for Turkish ODE websites (Anadolu University ODE; Ankara University ODE; Ataturk University ODE; Istanbul University ODE) were weighted with the Fuzzy AHP method. Finally, the ODE website performance was evaluated with the Fuzzy WASPAS method. Further, the study results were shared with the decision-makers of each university.

3.1. Background: Open and Distance Education Institutions in Türkiye

The higher education system in Türkiye comprises Public-Government and Private-Foundation. A total of 205 universities (129 public and 76 private) serve higher education programs [46]. The education forms are on-campus, open-distance, and distance education schools. Four state universities have been authorized in Türkiye. According to the rules and principles determined by the Council of Higher Education, these institutions offer educational services to various segments of society. These four institutions offer the entire program as Open-Distance Education; however, Distance Learning programs also have growing appeal, having grown by 30% of the total number of universities in the country. Recent decades have shown that interest in open and distance education has increased. The following structured information on ODE institutions is given in alphabetical order.

Anadolu University Open Education System, with 3.5 million active students [46], one of the world’s mega universities, is the first open higher education system in Eskisehir, Türkiye, established in 1982. The system offers 19 undergraduate and 41 associate degree programs for personnel of national education [47]. The university provides educational programs in Türkiye and 29 countries with 50 examination centers. As a leading institution in Open-Distance education, its mission and strategies are as follows,

“Open Education System has the privileged of achieving equal opportunity in education in all circumstances, not only the citizens with physical, hearing and visually impaired but also the citizens who are trying to complete their sentences in prisons” [47].

Ankara University, as an Open and Distance Educational School, was established in 2020 in the capital city of Ankara, Türkiye [48]. The school offers two undergraduate and 13 associate degree programs to 1.7 thousand students [46].

Ataturk University Open and Distance Educational School was established in 2012 in Erzurum, Türkiye [49]. The school has offerred 13 undergraduate, 4 undergraduate completion, and 30 associate degree programs to 487 thousand students [46].

Istanbul University is the oldest university in Türkiye, established in 1453. The institution has 46 programs (19 undergraduate, 5 undergraduate completions, and 22 associate degrees) with a total of 452 thousand students [46]. The Open and Distance Educational Schools started the programs of Distance Learning Units in 2009 and Open Education in 2011 [50]. The institution provides regular and collaborative programs with other schools at Istanbul University.

3.2. Determining the Evaluation Criteria

The study was conducted in several stages. First, the criteria were considered for evaluating websites from the support of the literature and research strategies. As shown in

Table 2. Criteria and Sub-criteria for Evaluation.

Criteria Title	Sub Criteria with Codes
Navigation	Easy access content menu (N1); Search through the website (N2); Speed-upload (N3)
Accessibility	Home page accessibility/return (A1); Accessibility from multiple devices/disability (A2); Social Media links (A3); Log-in subscribe with password (A4)
Design	Layout (proper font size, color, contrast, whitespace) (D1); Interactive media (picture, animation, video, audio, etc.) (D2; User friendliness (D3) Language preferences (D4); Browser adaptability (D5)
Content readability	Error (free) prevention (C1); Efficiency and flexibility (C2); Logical information hierarchy (C3)
Announcements	Informative Feedback (AN1); Help-Support (Live chat) (AN2); Contact address and Information (AN3); Copyright and updates (AN4); FAQ (frequently asked questions) (AN5)

, the criteria were finalized and divided into subcategories based on expert opinions. These criteria categories are coded for each main and sub-criteria as follows; navigation (three sub-criteria; N1 to N3), accessibility (four sub-criteria; A1 to A4), design (five sub-criteria; D1 to D5), content readability (three sub-criteria; C1 to C3), and announcements (five sub-criteria; AN1 to AN5).

The opinions of four experts were used both in categorizing the criteria and in the pairwise comparison with the Fuzzy AHP method, as well as in evaluating the alternative ODEs with the Fuzzy WASPAS method. Two of these experts are information technology department professionals, a website designer and a website administrator. Others are potential users of the ODEs that are the subject of the analysis, a faculty member with eight years of professional experience in the engineering department, and a postgraduate student. The five-point linguistic expression scale in Table 1 is used for the pairwise comparisons. The data collected were summarized based on the findings of the observations, and the article used the average judgments of the four experts.

3.3. The Weighting of Evaluation Criteria with the Fuzzy AHP Method

In the study’s second phase, the criteria and sub-criteria used to analyze the decision problem were weighted using the Fuzzy AHP method. For this purpose, the main criteria were first discussed, and the experts made a pairwise comparison using the fuzzy triangular numbers in Table 1. The verbal evaluations were converted into the fuzzy triangular numbers given in

Table 3. Fuzzy Pairwise Comparison Matrix for Main Criteria.

	Fuzzy Pairwise Comparison Matrix for Main Criteria
	N	A	D	C	AN
N	(1,1,1)	(1/7,1/5,1/3)	(1/5,1/3,1/1)	(1/9,1/7,1/5)	(1/5,1/3,1/1)
A	(3,5,7)	(1,1,1)	(1/7,1/5,1/3)	(1,3,5)	(3,5,7)
D	(1,3,5)	(3,5,7)	(1,1,1)	(5,7,9)	(1,3,5)
C	(5,7,9)	(1/5,1/3,1/1)	(1/9,1/7,1/5)	(1,1,1)	(1,3,5)
AN	(1,3,5)	(1/7,1/5,1/3)	(1/5,1/3,1/1)	(1/5,1/3,1/1)	(1,1,1)

and digitized. Calculations were made by considering the equations given in the Fuzzy AHP application steps detailed in the previous section, and the fuzzy triangular weight values (W) given in

Table 4. Calculated Fuzzy Weights for Main Criteria.

Fuzzy Weights for Sub-Criteria
N1			N2			N3			…	AN3			AN4			AN5
l	m	u	l	m	u	l	m	u	…	l	m	u	l	m	u	l	m	u
0.038	0.053	0.058	0.027	0.022	0.019	0.019	0.009	0.006	…	0.033	0.010	0.012	0.007	0.005	0.005	0.017	0.027	0.027

were obtained.

Then, the relevant sub-criteria for each main criterion were compared, and their effects on the solution were revealed.

Table 5. Sub-criteria—Navigation Pairwise Comparison Matrix.

	Navigation (N)
	N1	N2	N3
N1	(1,1,1)	(1,3,5)	(3,5,7)
N2	(1/5,1/3,1/1)	(1,1,1)	(1,3,5)
N3	(1/7,1/5,1/3)	(1/5,1/3,1/1)	(1,1,1)

,

Table 6. Sub-criteria—Accessibility Pairwise Comparison Matrix.

	Accessibility (A)
	A1	A2	A3	A4
A1	(1,1,1)	(1,3,5)	(1/7,1/5,1/3)	(1/7,1/5,1/3)
A2	(1/5,1/3,1/1)	(1,1,1)	(1/7,1/5,1/3)	(1/7,1/5,1/3)
A3	(3,5,7)	(3,5,7)	(1,1,1)	(1,3,5)
A4	(3,5,7)	(3,5,7)	(1/5,1/3,1/1)	(1,1,1)

,

Table 7. Sub-criteria—Design Pairwise Comparison Matrix.

	Design (D)
	D1	D2	D3	D4	D5
D1	(1,1,1)	(1,3,5)	(1,3,5)	(1,3,5)	(1,3,5)
D2	(1/5,1/3,1/1)	(1,1,1)	(1,3,5)	(1,3,5)	(1,3,5)
D3	(1/5,1/3,1/1)	(1/5,1/3,1/1)	(1,1,1)	(1,3,5)	(1,3,5)
D4	(1/5,1/3,1/1)	(1/5,1/3,1/1)	(1/5,1/3,1/1)	(1,1,1)	(1,1,1)
D5	(1/5,1/3,1/1)	(1/5,1/3,1/1)	(1/5,1/3,1/1)	(1,1,1)	(1,1,1)

,

Table 8. Sub-criteria—Content Readability Pairwise Comparison Matrix.

	Content Readability (C)
	C1	C2	C3
C1	(1,1,1)	(1,3,5)	(1/5,1/3,1/1)
C2	(1/5,1/3,1/1)	(1,1,1)	(1/5,1/3,1/1)
C3	(1,3,5)	(1,3,5)	(1,1,1)

and

Table 9. Sub-criteria—Announcements Pairwise Comparison Matrix.

	Announcements (AN)
	AN1	AN2	AN3	AN4	AN5
AN1	(1,1,1)	(1,3,5)	(1,3,5)	(3,5,7)	(1,3,5)
AN2	(1/5,1/3,1/1)	(1,1,1)	(1,3,5)	(1,3,5)	(1/5,1/3,1/1)
AN3	(1/5,1/3,1/1)	(1/5,1/3,1/1)	(1,1,1)	(1,3,5)	(1/5,1/3,1/1)
AN4	(1/7,1/5,1/3)	(1/5,1/3,1/1)	(1/5,1/3,1/1)	(1,1,1)	(1/5,1/3,1/1)
AN5	(1/5,1/3,1/1)	(1,3,5)	(1,3,5)	(1,3,5)	(1,1,1)

shows the pairwise comparison matrices of the sub-criteria determined for all the main criteria.

The final weighted values in

Table 10. Weightings of Main Criteria and Sub-criteria.

Main Criteria	Weights of Main Criteria	Sub-Criteria	Weights of Sub-Criteria
Navigation (N)	0.08	N1	0.59
		N2	0.27
		N3	0.13
Accessibility (A)	0.18	A1	0.11
		A2	0.07
		A3	0.49
		A4	0.32
Design (D)	0.41	D1	0.36
		D2	0.25
		D3	0.18
		D4	0.11
		D5	0.11
Content Readability (C)	0.12	C1	0.30
		C2	0.17
		C3	0.53
Announcements (AN)	0.11	AN1	0.37
		AN2	0.17
		AN3	0.18
		AN4	0.05
		AN5	0.22

were obtained from analyzing the expert opinions given in Table 5, Table 6, Table 7, Table 8 and Table 9 using the Fuzzy AHP method. Thus, the fuzzy and defuzzified weight values of all main and sub-criteria that are effective in the evaluation process of ODEs were obtained. The weight values given in Table 10 were defuzzified to understand the effect of each criterion on the solution results. In the next step, fuzzy triangular weight values were used for analysis with the Fuzzy WASPAS method.

As seen in Table 10, with a weight value of 0.41, the criterion that has the most critical effect on the solution among the five main criteria is Design, while Accessibility ranks second with a weight value of 0.18, and Content Readability is the third-most important criterion with a weight value of 0.12.

3.4. Evaluation of Website Performance with Fuzzy WASPAS Method

In this research step, Fuzzy WASPAS started forming fuzzy assessments of the ODEs (ODE1, ODE2, ODE3, and ODE4), taking into account the criteria by applying for fuzzy triangular numbers. In this part, the alternative ODEs described in Section 3.1 are coded randomly to ensure confidentiality and keep their names hidden. ODE details are kept confidential in order to eliminate legal issues. However, the results can be shared with the institutions with a confidentiality agreement during the consultancy sessions. Therefore, our study has considered these sensitivities.

Fuzzy linguistic expressions are first converted to fuzzy triangular numbers (

Table 11. Linguistic Expressions used in the Evaluation of Alternatives.

	N1	N2	N3	A1	A2	A3	A4	D1	D2	D3	D4	D5	C1	C2	C3	AN1	AN2	AN3	AN4	AN5
ODE1	T	E	T	T	T	T	T	T	T	T	T	T	T	T	T	T	T	T	T	T
ODE2	VS	T	S	VS	T	T	T	VS	T	VS	E	T	T	T	VS	T	T	T	T	T
ODE3	S	T	T	E	VS	T	T	S	S	VS	E	T	VS	VS	S	T	VS	T	T	T
ODE4	VS	S	VS	T	E	E	T	S	VS	S	E	T	VS	S	T	T	T	T	S	T

Equal importance (E), Strongly important (S), Very Strongly important (VS), Totally important (T).

). Thus, the initial fuzzy decision matrix is obtained. The normalized decision matrix shown in

Table 12. The Normalized Fuzzy Decision Matrix.

	N1			N2			N3			…	AN3			AN4			AN5
ODE1	0.78	1.00	1.00	0.11	0.11	0.11	0.78	1.00	1.00	…	0.78	1.00	1.00	0.78	1.00	1.00	0.78	1.00	1.00
ODE2	0.56	0.78	1.00	0.78	1.00	1.00	0.33	0.56	0.78	…	0.78	1.00	1.00	0.78	1.00	1.00	0.78	1.00	1.00
ODE3	0.33	0.56	0.78	0.78	1.00	1.00	0.78	1.00	1.00	…	0.78	1.00	1.00	0.78	1.00	1.00	0.78	1.00	1.00
ODE4	0.56	0.78	1.00	0.33	0.56	0.78	0.56	0.78	1.00	…	0.78	1.00	1.00	0.33	0.56	0.78	0.78	1.00	1.00

was achieved by applying Equation (9).

Equation (10) was applied to obtain the weighted-normalized Fuzzy Decision Matrix for WSM (

Table 13. The Weighted Normalized Matrix for WSM.

Weight	0.038	0.053	0.058	0.027	0.022	0.019	0.019	0.009	0.006	…	0.033	0.010	0.012	0.007	0.005	0.005	0.017	0.027	0.027
	N1			N2			N3			…	AN3			AN4			AN5
ODE1	0.029	0.053	0.058	0.003	0.002	0.002	0.015	0.009	0.006	…	0.026	0.010	0.012	0.005	0.005	0.005	0.013	0.027	0.027
ODE2	0.021	0.041	0.058	0.021	0.022	0.019	0.006	0.005	0.005	…	0.026	0.010	0.012	0.005	0.005	0.005	0.013	0.027	0.027
ODE3	0.013	0.029	0.045	0.021	0.022	0.019	0.015	0.009	0.006	…	0.026	0.010	0.012	0.005	0.005	0.005	0.013	0.027	0.027
ODE4	0.021	0.029	0.038	0.013	0.021	0.029	0.021	0.029	0.038	…	0.029	0.038	0.038	0.013	0.021	0.029	0.029	0.038	0.038

), and Equation (11) was applied to obtain the weighted-normalized Fuzzy Decision Matrix for WPM (

Table 14. The Weighted Normalized Matrix for WPM.

Weight	0.038	0.053	0.058	0.027	0.022	0.019	0.019	0.009	0.006	…	0.033	0.010	0.012	0.007	0.005	0.005	0.017	0.027	0.027
	N1			N2			N3			…	AN3			AN4			AN5
ODE1	0.991	1	1	0.943	0.954	0.959	0.995	1	1	…	0.992	1	1	0.998	1	1	0.996	1	1
ODE2	0.978	0.987	1	0.993	1	1	0.980	0.995	0.999	…	0.992	1	1	0.998	1	1	0.996	1	1
ODE3	0.959	0.970	0.986	0.993	1	1	0.995	1	1	…	0.992	1	1	0.998	1	1	0.996	1	1
ODE4	0.978	0.991	1	0.959	0.978	0.991	0.978	0.991	1	…	0.991	1	1	0.959	0.978	0.991	0.991	1	1

).

Equations (12) and (13) were applied to determine the values of the optimality function for WSM and WPM. Following these calculations, the obtained weight values and ranking of the alternative ODEs are given below (

Table 15. Calculations of the Optimality and Integrated Utility Function Values of the Fuzzy WASPAS Approach.

		l	m	u	Qi
	ODE1	0.687	0.887	0.890	0.822
	ODE2	0.589	0.787	0.868	0.748
	ODE3	0.473	0.648	0.768	0.630
	ODE4	0.393	0.535	0.611	0.513
	Sum				2.712				Ki		Alternative	Weight	Rank
									0.854		ODE1	0.300	1
						λ	0.52		0.770		ODE2	0.271	2
									0.641		ODE3	0.225	3
		l	m	u	Pi				0.579		ODE4	0.204	4
	ODE1	0.756	0.954	0.959	0.889			Sum	2.843
	ODE2	0.633	0.836	0.911	0.794
	ODE3	0.494	0.675	0.792	0.654
	ODE4	0.539	0.673	0.744	0.652
	Sum				2.988

).

4. Discussions and Conclusions

This study aimed to evaluate the websites of four leading universities in the field of open education in Türkiye with multiple criteria using the Fuzzy AHP and Fuzzy WASPAS methods. For this purpose, firstly, the literature in the relevant field was examined, and the evaluation criteria considered within the scope of the study were put forward. The criteria obtained were categorized with the help of expert opinions. The analyses and evaluations carried out within the scope of the study were handled in two stages. The Fuzzy AHP method was used in the first stage to weight the evaluation criteria. Since the criteria used in the evaluation do not have clear numerical values that can be expressed in terms of all decision-makers, fuzzy numbers were used.

After the analyses were performed with Fuzzy AHP, weight values were calculated for each main and sub-criteria. When the weights calculated for the five main criteria are examined, it can be seen that the criterion with the highest importance of 0.41 is “Design.” With a weight value of 0.18, “Accessibility” is the criterion with the second most important effect. The weight value of “Content Readability,” which ranks third in terms of its effect on the solution, was calculated as 0.12. The first three criteria are followed by “Announcements,” with a weight value of 0.11, and “Navigation,” with a weight value of 0.08, respectively. When the “Design” criterion, which has the highest priority value, is examined in terms of sub-criteria, it draws attention as the most prominent sub-criteria in terms of Layout (D1), Interactive Media (D2), and User Friendliness (D3) design, respectively. In light of the results obtained, it can be seen that, during the evaluation of the four alternative ODEs, the alternative that is better in terms of “Design” stands out more than the others and takes first place.

After the weights of the evaluation criteria were determined, alternative ODEs were evaluated with the Fuzzy WASPAS method. At this stage, fuzzy criterion weights were used during the analyses. The WASPAS Method was preferred because it aims to achieve a more accurate result by combining the results of two different models. As a result of the analysis performed with the Fuzzy WASPAS Method, the ranking of the alternatives was carried out.

Table 16. Weighting and Ranking of Alternatives with Fuzzy WASPAS.

	Weights and Ranking for Alternatives
Alternatives	ODE1	ODE2	ODE3	ODE4
Weights	0.300	0.271	0.225	0.204
Ranking	1	2	3	4

shows the alternative weights and ranking based on the analysis results obtained with the Fuzzy WASPAS Methods.

When the analysis results are examined, it can be seen that the first alternative is quite prominent compared to the others. While ODE1 was in the first place with a weight value of 0.300, it can be seen that ODE2 took the second place with a weight value of 0.271. When the fuzzy verbal evaluations made by the decision makers for ODE1 and ODE2 are examined, it can be seen that ODE1 has the best values in terms of criteria with high weight. ODE3 follows the first two alternatives with a 0.225 weight value, followed by ODE4 with 0.204. After applying the Fuzzy WASPAS method, sensitivity analysis was performed for different λ values, and it was examined whether the alternative ordering changed.

As seen from

Table 17. Alternative ODE’s Weights According to Different λ Values.

	W
	λ = 0.25	λ = 0.5	λ = 0.52	λ = 0.75	Ratings
ODE1	0.307	0.301	0.300	0.295	1
ODE2	0.275	0.271	0.271	0.267	2
ODE3	0.228	0.226	0.225	0.224	3
ODE4	0.217	0.205	0.204	0.193	4

, λ = 0.25, λ = 0.50, and λ = 0.75 changed the calculated w values; however, there was no change in the ranking results.

System requirements of the platforms ensure what needs to be accomplished for the proposed system to work effectively from the user perspectives and expert opinions. The study provided how to evaluate the ODEs with the proper criteria set and the entire study process. It is essential to investigate further how such factors would lead to the development and implementation of higher education institutions’ service offerings.

The evaluation criteria are split into five main categories to assess the performance and highlight the potential service quality level. Comparing the four main ODEs also supports the benchmarking among these platforms. According to the results, Design is the most valued and necessary element for website infrastructure, which guarantees the service provision, followed by Accessibility and Content Readability. Another benefit of having this kind of evaluation is to provide a road map for implementation. Therefore, institutions should concentrate on adapting the improvement areas in their websites to provide various and technologically-supported new features.

4.1. Theoretical and Research Implications

From a research and theoretical perspective, the findings of this study broaden the field of knowledge regarding Multi-Criteria Decision-Making (MCDM). This study also reveals that Open and Distance Education platforms in service usage, design features, accessibility, navigation, and content management are evaluated based on the research aims and strategies. The human–computer interaction for online education is vital. Our results also provide invaluable information on the design requirements preferred by online learners. Our paper also contributed to the MCDM literature by using Fuzzy AHP and Fuzzy WASPAS methods and provides an evaluation of the ODE website performance. This paper focuses on a single country case study to evaluate and conduct the analyses.

4.2. Managerial and Practical Implications

From a practical and managerial perspective, the results of this research paper provide information to users about ODE websites’ performance and usage of the system. Decision-makers benefit from the comparative results. This research also supports understanding the ODE website infrastructure with system requirements and administrative concerns to manage the programs. Design, Accessibility, and Content Readability are the essential criteria highlighting the concentration point for developing and improving the ODE services. Strategy makers should also focus on enhancing ODE website system characteristics, features, and content richness regarding service quality and operations. It details the particular design element that must be visible on an ODE website. Website designers could benefit from this information. Furthermore, with the help of technological advancement and new service strategies, we suggest tailoring ODE webpages to meet personality features to improve functioning.

4.3. Research Limitations

Despite the implications to the research field of the MCDM evaluation study, several limitations can be found in this study. First, evaluation criteria have been revealed and detailed using existing publications despite the scarcity of publications in the study area. Second, the conducted evaluation heavily depends on the service availability of the ODE institutions where the evaluation takes place in a single country comparison. Third, the usage of the MCDM method is also considered a limitation; therefore, the comparative analysis with other methods is not applied. Fourth, while determining the criteria weights, it is assumed that the main and sub-criteria were expressed in a hierarchical structure. The possible feedback and interaction among the sub-criteria are not taken into account. Finally, the experts who carried out the evaluation are not actual users of the ODE system; therefore, they evaluated user perspectives with limited accessibility.

4.4. Future Research Directions

The findings of this research highlight the need for further studies to evaluate ODE systems. Furthermore, potential future research streams can be proposed as follows:

The possible feedback and interaction between the sub-criteria can be evaluated using a method that considers the feedback, such as the analytical network process. The subsequent study can be carried out with students, academics, and system employees directly involved in the process to obtain a more detailed evaluation. Moreover, working with decision-makers, who are the system’s primary users, and administrators of the system may generate a different set of criteria for future studies. In addition, more detailed research and evaluations should be made based on each degree program that institutions offer to perform better in the ODE platform. Moreover, comparative case studies in a sector or country can extend the study with various application areas in similar domains or in general business, management, and engineering fields.

In conclusion, universities worldwide seek to provide the most outstanding available student services, particularly those that promote student achievements in their program objectives. The World Wide Web, in particular, has advanced internet-based information technology, dramatically impacting all types of education delivery. Therefore, it has rapidly expanded the Open and Distance Education (ODE) system. This case study took place in Türkiye, and the institutions available for the study are Anadolu University ODE; Ankara University ODE; Ataturk University ODE; Istanbul University ODE. This study focused on evaluating the performance of the websites of four leading universities in the field of ODE in Türkiye with multiple criteria using Fuzzy AHP and Fuzzy WASPAS approaches. For this purpose, firstly, the literature in the relevant field was examined, and the evaluation criteria considered within the scope of the study were put forward. ODE websites, using 5 main criteria (Navigation, Accessibility, Design, Content Readability, and Announcements) and 20 sub-criteria, were examined. The criteria obtained were categorized with the help of expert opinions. The analyses and evaluations carried out within the scope of the study were handled in two stages. The Fuzzy AHP method was used in the first stage to weight the evaluation criteria. Since the criteria used in the evaluation do not have clear numerical values that can be expressed in terms of all decision-makers, fuzzy numbers were used. According to the research findings, Design is the essential criterion, followed by the Accessibility, Content Readability, Announcement, and Navigation criteria. Our research identified and recommended the development areas for further research and proposed theoretical and practical implications and managerial decisions to be considered for ODE website improvements.