A Novel Integrated PIPRECIA–Interval-Valued Triangular Fuzzy ARAS Model: E-Learning Course Selection

: The development of information and communication technologies has revolutionized and changed the way we do business in various areas. The ﬁeld of education did not remain immune to the mentioned changes; there was a gradual integration of the educational process and the mentioned technologies. As a result, platforms for distance learning, as well as the organization of e-learning courses of various types, have been developed. The rapid development of e-learning courses has led to the problem of e-learning course selection and evaluation. The problem of the e-learning course selection can be successfully solved by using multiple-criteria decision-making (MCDM) methods. Therefore, the aim of the paper is to propose an integrated approach based on the MCDM methods and symmetry principles for e-learning course selection. The pivot pairwise relative criteria importance assessment (PIPRECIA) method is used for determining the weights of criteria, and the interval-valued triangular fuzzy additive ratio assessment (ARAS) method is used for the ranking of alternatives i.e., e-learning courses. The suitability of the proposed integrated model is demonstrated through a numerical case study.


Introduction
The role of information and communication technologies (ICT) has become very important in all aspects of life and business. These technologies have a very significant role in information elaboration and its transformation into knowledge, which is the main condition for someone to become an efficient part of the information society [1]. The information has become accessible and transferable from anywhere in the world. Also, education and its availability have an important role in the modern world and therefore, ICT has become an integral part of people's lives in all aspects [2]. The rapid and dynamic development of information and communication technologies has led to significant changes in people's personal, social, and work lives. Currently, people live in an information society that relies on the use of information and communication technologies, which is also a society of knowledge, intangible capital and learning, in which progress is based on knowledge and creativity [3].
Based on the previously mentioned fuzzy sets theory, somewhat later Bellman and Zadeh [25] introduced the fuzzy MCDM-based methodology, which was later widely accepted in the scientific community as well as being used to solve many decision-making problems. With the aim to solve a variety of complex MCDM problems, some extensions to the fuzzy set theory have been proposed, such as intuitionistic fuzzy sets [26], interval-valued fuzzy sets [27], bipolar fuzzy sets [28], and so on.
The additive ratio assessment (ARAS) method was developed by Zavadskas and Turskis [36]. The ARAS method has been applied to solve various decision-making problems. So, for example, Zavadskas and Turskis [36] have applied the ARAS method to evaluate the microclimate in office rooms, Zavadskas et al. [37] have applied the ARAS method to select the most appropriate foundation installment alternative, Karabašević et al. [38] have applied the ARAS method for personnel selection. Besides that, the ARAS method has also been applied for the ranking of companies according to the CSR indicators [39], selection of the software testing method [40], mineral prospectivity mapping [41], reduction of greenhouse gas emission [42], and so on.
In order to extend the applicability of the ARAS method and to enable the use of grey and fuzzy numbers, Turskis and Zavadskas [43] proposed a proper fuzzy extension (ARAS-F) and a grey extension (ARAS-G) [44]. It is also important to note that on the basis of the ARAS method, some other approaches are proposed. In this context, it is worth mentioning the ARCAS approach proposed by Stanujkic et al. [45] that is based on ordinary SWARA and ARAS methods and is adapted for negotiations.
Based on all the above, the main aim of this paper is to propose an integrated approach based on the MCDM methods for the selection of the e-learning course. The proposed approach is based on the use of the interval-valued triangular fuzzy (IVTFN) ARAS method for the ranking of alternatives and the PIPRECIA method for the determining weights of the criteria.
Therefore, the rest of this paper is organized as follows. Section 2 demonstrates the proposed methodology, based on the PIPRECIA and extended ARAS method. In order to highlight the proposed MCDM methodology, in Section 3 a case study of e-courses evaluation is considered. The discussions and conclusions are given at the end of the manuscript.

Methods
The proposed methodology for the evaluation of the e-courses is based on the PIPRECIA method and interval-valued triangular fuzzy ARAS method. The PIPRECIA method is used for defining weights of the criteria whereas the interval-valued triangular fuzzy ARAS method is used for the ranking of alternatives.

PIPRECIA Method
The pivot pairwise relative criteria importance assessment (PIPRECIA) method for determining the weights of criteria was proposed by Stanujkic et al. [46], as one of the extensions of the step-wise weight assessment ratio analysis (SWARA) method. Although it is a relatively new method, so far, the PIPRECIA method is applied for solving problems of personnel selection [47], assessment of ICT [48,49], supplier selection [50], mining method selection [51], and so forth.
The computational procedure of the PIPRECIA method can be demonstrated through the following steps [46]: Step 1. Determination of the set of the evaluation criteria. Step 2. Starting from the second criterion, it is necessary to determine the relative importance s j of the criterion j in relation to the previous (j − 1) criterion: (1) Step 3. Calculation of the coefficient k j as follows: Step 4. Calculation of the recalculated weight q j as follows: Step 5. Calculation of the relative weights of the evaluation criteria as follows: where w j represents the relative weight of the criterion j.

An Extension of the ARAS Method Based on the Use of Interval-Valued Fuzzy Numbers
Based on Stanujkic et al. [52], the computational procedure for selecting the most acceptable alternative by applying the IVTFN-ARAS method that includes only beneficial criteria could be demonstrated through the following steps: Step 1. Determination of the optimal performance rating for each criterion.
where x 0 j represents the interval-valued fuzzy optimal performance rating of criterion j.
Step 2. Calculation of the normalized decision matrix.
where r ij represents the normalized interval-valued fuzzy performance rating of alternative i in relation to the criterion j, c + j = m i=0 c ij . Step 3. Calculation of the weighted interval-valued normalized fuzzy decision matrix.
v ij = w j · r ij (12) where v ij represents the weighted normalized interval-valued fuzzy performance rating of alternative i in relation to the criterion j.
Step 4. Calculation of the overall interval-valued fuzzy performance ratings.
where S i represent overall interval-valued fuzzy performance rating of alternative i.
Step 5. Calculation of the degree of utility, for each alternative. As a result of performing the previous steps, the obtained overall performance ratings are IVFNs. Therefore, overall performance ratings have to be defuzzified before the calculation of the overall degree of utility. In this way, the same equation as in the ordinary ARAS method is used to determine the overall degree of utility.
Step 6. Ranking of alternative selections by the most efficient. This step is the same as in the original ARAS method.

A Numerical Case Study
This section describes the process of the selection of an e-learning course based on the opinions of twenty-four respondents by using the PIPRECIA method and the IVTFN-ARAS method. Four e-learning courses in the field of programming are evaluated, designated as: A 1 -Cubes (www.cubes.edu.rs); A 2 -ITAcademy (www.it-akademija.com); A 3 -Link-eLearning (www.link-elearning.com), and A 4 -Ok School (www.ok-school.com). The criteria used for the evaluation of the e-learning courses are obtained based on the analysis of the relevant literature [53][54][55], and their weights are determined by using the PIPRECIA method. Evaluation criteria and their corresponding weights that represent the attitudes of the group are shown in Table 1. Table 1. The group evaluation criteria and their weights.
Presentation method 0.14 C 3 Teaching method 0.13 C 4 e-learning environment 0.14 C 5 Learning materials 0.15 Quality of multimedia content 0.14 C 7 Group work and interactivity 0.15 At the beginning of the evaluation, all twenty-four respondents evaluated alternatives using the five-point Likert scale. The Likert scale was chosen because it is easy to use and its usage is understandable for respondents. Ratings obtained from three randomly selected respondents are shown in Tables 2-4 and Figures 1-3. Table 2. The ratings obtained from the fist of twenty-four respondents.
Criteria  Figure 1. The ratings obtained from the fist of twenty-four respondents. Table 3. The ratings obtained from the second of twenty-four respondents.  Table 3. The ratings obtained from the second of twenty-four respondents. Criteria The ratings obtained from the fist of twenty-four respondents. Table 3. The ratings obtained from the second of twenty-four respondents.   Subsequently, in the next step, a group decision matrix was formed. The elements of this matrix, shown in Table 5, are IVTFNs formed by the transformation of crisp ratings into IVTFNs, as is explained in Stanujkic [52].
Based on the data from Table 5, the optimal performance ratings are determined by using Equation (5). The obtained optimal performance ratings are shown in Table 6.
In the next two steps, normalized and weighted normalized decision-making matrices are calculated using Equations (11) and (12). The normalized and weighted normalized decision-making matrices are shown in Tables 7 and 8.  Subsequently, in the next step, a group decision matrix was formed. The elements of this matrix, shown in Table 5, are IVTFNs formed by the transformation of crisp ratings into IVTFNs, as is explained in Stanujkic [52].
Based on the data from Table 5, the optimal performance ratings are determined by using Equation (5). The obtained optimal performance ratings are shown in Table 6.
In the next two steps, normalized and weighted normalized decision-making matrices are calculated using Equations (11) and (12). The normalized and weighted normalized decision-making matrices are shown in Tables 7 and 8. Table 5. IVTF group decision-making matrix.  Table 6. The optimal IVTF performance ratings.  Table 7. The normalized IVTF performance rating.  Finally, the overall interval-valued triangular fuzzy performance ratings, obtained by using Equation (13), are shown in Table 9. In order to determine the quality of the e-courses, these values must be defuzzified using some of the well-known procedures [52].
Results obtained using the simplest of all of the considered defuzzification procedures are shown in Table 10. The relative quality, i.e., the degree of utility, of analyzed e-courses as well as their ranking orders, are also shown in Table 10. Table 10. The degree of utility and ranking order of analyzed e-courses. As can be seen from Table 10, the best alternative, i.e., e-course, is the alternative designated as A 2 . By varying the coefficient λ, greater importance can be given to l and u in relation to l' and u', and vice versa. The results obtained using Equation (16) for some characteristic values of the coefficient λ are shown in Table 11. Table 11. The degree of utility and ranking order of analyzed e-course for some characteristic values of λ. In this case, the alternative denoted as A 2 remains the best alternative i.e., an e-learning course in all cases. This indicates the stability of the chosen e-learning course.

BNP Q i Rank
However, in many cases variation of the coefficient lambda may have an impact on the ranking order of the considered alternatives, and this approach may be useful to analyze different scenarios such as pessimistic, realistic, and optimistic.

Discussion and Conclusions
The main aim of the paper is to introduce a methodology suitable for e-learning course evaluation and selection. The emergence of e-courses as the modern way of learning has provoked the need for finding the methods ideal for their assessment. In this paper, the methodology based on the PIPRECIA and the interval-valued triangular fuzzy ARAS methods is proposed. The applicability of the proposed methodology is presented through a numerical case study. When defining criteria, special attention was dedicated to the issues of organization and teaching. So, in this case, the attention was not directed on the technical and informational performance of an e-learning course, but towards the quality of the offered content and how the teaching process was implemented.
Based on the data obtained from the respondents, the PIPRECIA method was applied, and the weights of the criteria for each of the twenty-four respondents were obtained, as is shown in Table 1.
The obtained results show that the weights of the criteria are approximate, which impose the fact that the given features are nearly equally important to all respondents. This is entirely understandable because the e-learning course should satisfy all the requirements and, in that way, offer the quality of "service" to the users.
The reason for applying the PIPRECIA method for the determination of the weights lies in its simplicity and suitability for use in cases where a large number of decision-makers are involved in the evaluation process. The advantage of the PIPRECIA method over the well-known and widely used AHP method is reflected in a more straightforward computational procedure that does not diminish the reliability and relevance of the results obtained. Also, when interviewing respondents who are not familiar with the MCDM methods, the process of evaluating weights by using the PIPRECIA method is far more understandable to respondents, than is the case with the AHP method. If the PIPRECIA method is compared with the SWARA method (on which the PIPRECIA method was developed), it can be concluded that the PIPRECIA method has certain advantages over it. Namely, the SWARA method requires that the evaluation criteria should be sorted according to their intended significance, which complicates its application in group decision cases. Many complex decision-making problems require the participation of a group of respondents. In such cases, the individual attitudes of the respondents have to be transformed into group attitudes, with an as small as possible loss of information. In order to take into account the uncertainty and imprecision of the data on which decision-making is very often based, the application of the interval-valued triangular fuzzy ARAS method is proposed. The approach in which individual ratings are transformed into interval-valued triangular fuzzy numbers can be very useful in this regard. The interval-valued triangular fuzzy ARAS method may use such information to rank alternatives and/or analyze different scenarios. Thus, by applying this method, decision-makers have been given the opportunity to express their optimistic, pessimistic, and realistic attitudes.
In this paper, the numerical case study of the e-learning course selection was examined. The reason for that relies on the increasing importance of this kind of learning. In order to create the high quality e-learning course, it is necessary to determine the pros and cons of the considered course and its position relative to the competition. In that way, the creators will know what aspects of the course should be improved and what are of satisfactory quality. The application of the proposed integrated approach has proven to be quite justified and appropriate in this case. The reason is that if the e-learning courses were evaluated based solely on the use of crisp numbers, the obtained results would not include uncertainty. This would result in a decision that would not be completely realistic and, ultimately, unreliable. The obtained results confirmed this point of view. To get the most reliable results and to make the best possible decisions, it is necessary to respect the risk and uncertainty to the maximum extent possible. So, based on the conducted numerical case study, the e-learning course designated as A 2 is the best in terms of evaluated criteria.
As the examination of the literature has shown, the authors used different approaches for e-course evaluations. Chao and Chen [56] examined which factors are crucial for the quality of the e-learning courses. They applied the consistent fuzzy preference relations (CFPR) with AHP methodology. They evaluated four groups of factors that are elaborated in a particular number of criteria. The final results showed that the most influential criteria are: the e-learning material, friendly user interface, using the web discussion zone, and distant learning without time and space. The main point of this paper is the quality of the content of the e-learning courses. The assessment of the evaluation criteria showed that e-learning material has the greatest influence together with the group work and interactivity, which is in line with the results obtained from the mentioned authors.
Garg and Jain [57] applied the combination of the methods for defining the best e-learning website. They divide the evaluation criteria into two groups called quality factors and e-learning specific factors. The second group of factors is pointed to the quality of e-learning content and their results showed that the most important criterion is the ease of learning community, which could be considered as a counterpart to the group work and interactivity presented here.
Besides the mentioned works, others present the utilization of different MCDM techniques for e-learning course evaluation and selection. For the resolving of the problem of evaluation of e-course quality, the authors have proposed the application of the proximity indexed value (PIV) model [58], fuzzy ANP [59], DANP and VIKOR [60], and so on. The conducted numerical case study presented in this paper, as well as the comparison with the results of the given authors, confirmed that the proposed approach is also very beneficial for e-learning course evaluation and selection because the obtained results clearly outlined the crucial characteristics and the position of the specific e-learning course comparing to the others. Therefore, the proposed integrated PIPRECIA-IVTFN-ARAS model has proven to be useful and feasible, especially in circumstances where it is essential to make the most relevant and realistic decision possible. The proposed integrated model can be extended to other areas of the business as well.
As a direction for future research, a significantly larger sample could be used in order to obtain results from a macro point of view. Besides that, the learning outcomes of the learners that would study the ranked courses can be further investigated as well.