An Enhanced ELECTRE II Method for Multi-Attribute Ontology Ranking with Z-Numbers and Probabilistic Linguistic Term Set

Abstract

The high number of ontologies available on the web to date makes it increasingly difficult to select appropriate ontologies for reuse. Many studies have attempted to provide support for ontology selection and ranking; however, the existing studies provide support for ontology ranking from an objective perspective as opposed to a subjective perspective. They do not take into account the qualitative aspects of ontologies. Furthermore, the existing methods have a limited focus on group environments. In this paper, a multi-criteria decision-making approach is presented for ontology ranking with the development of an enhanced model combining the ELECTRE II model with the Z-Probabilistic Linguistic Term Set (ZPLTS). The ZPLTS-ELECTRE II model enables decision-makers to model ontology ranking problems using both numerical and linguistic data. Furthermore, the newly proposed model provides support for ontology ranking in group settings, with an emphasis on modeling the differing levels of credibility of decision-makers using the ZPLTS, which allows decision-makers to not only specify their opinion but also specify their level of credibility. The model was applied to rank a set of mental health ontologies obtained from the BioPortal repository. The results showed that the method was able to rank the ontologies successfully. The results were further compared with the traditional ELECTRE II and the PLTS ELECTRE II methods, displaying superior modeling capabilities. This paper demonstrated the effectiveness of the newly proposed ZPLTS-ELECTRE II model for ontology ranking in a real-world context, but the method is not constrained to the ontology ranking domain; rather, it may be applied to other real-world decision problems as well.

1. Introduction

The concept of ontology is becoming increasingly valuable in the field of artificial intelligence (AI) and big data [1]. An ontology is defined as an explicit specification of a shared conceptualization [2], and it is one of the core components facilitating knowledge representation and reasoning in AI. Ontologies describe different domains of discourse, and they play an important role in representing and expressing complex knowledge in a form that facilitates reasoning, computation, and dissemination. Ontologies provide an efficient solution to manage and advance the pressing issues of knowledge and information overload. Ontologies also provide an efficient and effective model for the comprehension of the myriad semantics associated with various knowledge and information, among different systems, agents, and technologies.

The field of ontology and ontological engineering has been studied extensively in recent years, and as such, scientists and subject matter experts have developed a wide range of ontologies belonging to various domains. Ontologies are also witnessing a surge in real-world applications; some of their applications are found in areas such as agriculture [3], finance [4], education [5], and engineering [6]. There are currently a wide range of ontologies freely available online for download, hosted in repositories such as the BioPortal [7] and the AgroPortal [8] ontology repositories. However, ontologies are inherently complex structures, which makes them highly arduous to architect and develop. A wide range of expertise is required for the research and development of an ontology. Accordingly, this process can be very costly and time consuming. Whilst some projects may require a new ontology to be developed de novo, for most projects, there is already a variety of existing ontologies available to choose from. Therefore, it is more efficient to reuse existing ontologies, possibly with some modifications, as opposed to developing new ontologies de novo. Unfortunately, the large number of ontologies available online also has a downside, giving rise to a new problem altogether—the problem of ontology selection.

The ontology selection problem arises due to the complexity associated with ontologies, which makes it challenging for users to comprehend and analyze existing ontologies. This problem is worsened when given the choice of multiple ontologies to choose from. It is therefore evident that there is an urgent need for the development of techniques and methods for evaluating and selecting relevant ontologies for the purpose of reuse.

There have been attempts by authors [9,10,11,12,13] to rank ontologies to aid their selection, but only in recent years has there been attempts to rank ontologies from a decision-making perspective [14,15,16,17]. The issue of ontology selection is essentially a multi-criteria decision-making (MCDM) problem, as there are multiple ontologies to choose from, whilst considering multiple characteristics and attributes. Authors have applied MCDM techniques for the task of ontology ranking and selection, such as ELECTRE, Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) [17], Weighted Linear Combination Ranking Technique (WLCRT) [16], and so on. However, these previous research studies present some shortcomings, which are elaborated on as follows.

1.

The existing MCDM solutions for ranking ontologies [14,15,16,17] to aid their selection focus on ranking ontologies from a singular perspective. However, in real-world application, the ranking and selection of suitable ontologies for reuse would involve more than a single person. Rather, a group of experts and stakeholders would collectively evaluate and decide on the suitable ontologies to reuse. At present, very little emphasis has been placed on ontology ranking and selection in a group setting in the existing literature.

2.

In real-world scenarios, ontologies would be ranked and selected not just according to their underlying characteristics, but a lot of importance would be given to how well the ontologies align with the needs and requirements of the organization and its stakeholders. However, the current MCDM solutions for ranking ontologies to aid their selection [14,15,16,17] focus on ranking ontologies based on their underlying characteristics and attributes. Little work has been completed that performs ontology ranking with MCDM methods from the perspectives of the actual users and knowledge engineers.

3.

The current MCDM solutions for ranking ontologies [14,15,16,17] to aid their selection rely on quantitative metrics represented with numerical values to evaluate and rank the ontologies; this renders the task of decision-makers and subject experts very difficult, as they are required to handle a large number of numerical values to express their evaluation of alternatives in the process of ranking ontologies. It would be valuable to allow decision-makers and subject experts to use natural language and linguistic terminologies to express their views in the ranking process of ontologies. At present, very little work has been completed pertaining to the usage of linguistic modeling in MCDM solutions for ranking and selection of ontologies.

To address the above-mentioned shortcomings of existing MCDM solutions for ontology ranking, this study proposes an enhanced ELECTRE II model based on the concept of the Z-Probabilistic Linguistic Term Set (ZPLTS), which is a combination of Z-numbers [18] and the Probabilistic Linguistic Term Set (PLTS) [19]. The ELECTRE II method is part of a family of outranking MCDM methods that were developed to rank alternatives from best to worst. It is one of the prominent models that is widely used in decision-making problems [20,21,22,23,24,25]. The ELECTRE methods were developed over five decades ago, and accordingly, they are a mature and well-studied MCDM tool, but nonetheless, the research and development on the methods is still very active up to today [25]. The ELECTRE family is also especially unique being that it has the ability to adapt to various problematic situations: that is, for selection problems, the ELECTRE I may be applied, for ranking problems, the ELECTRE II, III, and IV may be applied, and for classification problems, the ELECTRE Tri may be applied. The concept of PLTS allows decision-makers to provide their evaluations for alternatives with the use of linguistic terms as opposed to numerical values, making it easier to evaluate alternatives. Z-numbers allow a decision-maker to provide an indication of the credibility of their evaluation. The combination of these with the ELECTRE II method enables a powerful model for the task of ontology ranking for selection from a group perspective, using not only quantitative values but also linguistic evaluations from decision-makers. The contributions of the study are threefold: (1) the ability of modeling decision making in ELECTRE with both numerical and linguistic terms, (2) the use of ZPLTS to permit group decision making in ELECTRE and to enable the measurement of the credibility of the different decision-makers’ evaluations, (3) a comparative analysis of the proposed enhanced ELECTRE method with existing fuzzy ELECTRE methods on the task of ontology ranking in the mental health domain.

The rest of the paper is structured as follows. Section 2 reviews and discusses related literature. ELECTRE II is presented, and the various concepts relevant to the specification of the proposed ZPLTS-ELECTRE II method are defined in Section 3. In Section 4, the detailed specification of the proposed ZPLTS-ELECTRE II method is presented. The experimental results of the application of the ZPLTS-ELECTRE II method to rank a set of ontologies are described in Section 5, and the paper is concluded in Section 6.

2. Literature Review

As indicated earlier, there have been many studies focusing on ontology ranking [9,10,11,12,13] to aid ontology selection. In a study by Alani and Brewster [9], the authors investigated the application of ontology graph analysis measures such as the Centrality and Class Match Measures to rank ontologies, yielding the AKTiveRank system. However, the study gave rise to further questions, and therefore, the authors undertook further study [10] to modify AKTiveRank with the use of structural metrics of ontologies. Another study performed by Yu et al. [11] gave rise to the ARRO system for ontology ranking. ARRO had a substantial amount of similarities with the AKTiveRank system [10], as both systems perform the ranking of ontologies based on their relevance to the users’ query terms. In a study by Alipanah et al. [12], an algorithm was developed to rank ontologies based on a measurement from information theory, namely, Entropy-Based Distribution. In [13], the authors proposed an algorithm, named Onto-DSB, for ontology ranking based on the internal structure of ontologies. The algorithm was applied to an ontology set from Swoogle, and it outperformed the Swoogle and AKTiveRank techniques. The aforementioned studies do not use decision making in the ranking of ontologies.

In recent years, there has been an interest in the application of MCDM methods for the task of ontology ranking. One of the first studies was by Esposito et al. [14], where the authors implemented the ELECTRE I and ELECTRE III methods to rank ontologies. A similar study [15] applied the ELECTRE I to rank a set of biomedical ontologies. Fonou-Dombeu and Viriri [16] proposed a framework known as C-Rank, which uses a MCDM method called the Weighted Linear Combination Ranking technique (WLCRT) to rank ontologies. Another study by Fonou-Dombeu [17] applied three MCDM methods, namely, the Weighted Sum Model (WSM), the Weighted Product Model (WPM), and the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS), to rank ontologies. The studies [15,16,17] ranked ontologies based on their complexity metrics.

The aforementioned MCDM solutions for ranking ontologies to aid their selection perform the ranking based on the underlying structure and characteristics of the ontologies. However, in real-world applications, it would be important to rank ontologies for selection according to how well they align with the needs and requirements of the users and stakeholders for a project. In this regard, there have been some studies with an emphasis on ranking ontologies according to their applicability to the users’ needs. One prominent study was performed by Lozano-Tello and Gomez-Perez [26], where the authors proposed the OntoMetric model that allows users to evaluate the suitability of ontologies in light of the requirements of their projects. The study made use of 160 characteristics to compare ontologies, which yielded a score that was used to rank the ontologies. However, the system had the drawback of it being time consuming to perform such a large number of evaluations, and it also has the potential to be extremely biased depending on the decision-maker [27]. To overcome this issue, a recent study by Ma et al. [27] proposed an Ontology Usability Scale. The authors extracted 10 items from the 160 characteristics in [26] that were related to the usability of ontologies, which they then used to form the Ontology Usability Scale. The scale allows a user to provide their evaluation for the 10 items in light of each ontology with the use of a Likert scale. The qualitative attributes used in this study were adopted from the study in [27].

Despite the use of MCDM modeling in ontology ranking [14,15,16,17], only quantitative metrics have been used as evaluation criteria. There have been no attempts to combine MCDM modeling with usability metrics for ontology ranking and selection. Furthermore, there is little or no work that has been completed in facilitating ontology ranking and selection from the combined perspective of both the quantitative characteristics of ontologies as well as their level of usability for specific projects. The field of MCDM has also seen many improvements and enhancements in recent years, with the combination of other fields such as fuzzy set theory and linguistic terms. The aim of this study is to enhance the ELECTRE II MCDM method with the Z-Probabilistic Linguistic Term Set to address the above-mentioned problems.

The ELECTRE II method is part of the ELECTRE family of MCDM methods, along with ELECTRE I, III, IV, and Tri. ELECTRE I was originally developed by Bernard Roy [28] in 1968, and thereafter, the subsequent versions were developed as improvements of ELECTRE I. ELECTRE II [29] was developed to deal with ranking problems. The ELECTRE II model is based on the concepts of concordance and discordance, and it introduced the notion of strong and weak outranking graphs. Since its inception, ELECTRE II has been applied to a variety of domains, such as education [23], finance and investment [20], engineering [21], construction and built environment [22], and agriculture [24]. More applications of ELECTRE II can be found in [25]. These applications of ELECTRE II were based on the traditional ELECTRE II model; however, there have been many enhancements of the method by combining it with fuzzy structures for decision making in fuzzy environments. This enables the ELECTRE II model to deal with uncertain and imprecise data, thereby providing a solution to the problem of obtaining accurate information from decision-makers.

Authors have proposed many variations of ELECTRE II with different types of fuzzy sets. Govindan et al. [30] developed an ELECTRE II model based on fuzzy set theory to rank alternatives having uncertain information. The authors applied the method to the task of ranking third party logistics providers. In another study by Devadoss and Rekha [31], the intuitionistic ELECTRE II model was proposed and applied to rank inequalities faced by women in society. The intuitionistic fuzzy set was enhanced to overcome its restrictions, leading to the Pythagorean fuzzy set proposed by Yager [32]. Akram et al. [33] developed an ELECTRE II model to perform group decision making under the Pythagorean fuzzy environment, and the authors applied the method to a supplier selection scenario. Chen and Xu [34] combined the hesitant fuzzy set with ELECTRE II to yield the HF-ELECTRE II method. A study [35] integrated the concept of bipolar fuzzy sets with ELECTRE II to formulate the BF-ELECTRE II method. The authors then applied the model to a problem of selecting an optimal business location and choosing the best supplier. Another study by Tian et al. [36] developed an ELECTRE II method to perform under a single-valued neutrosophic environment in a group decision-making setting. Furthermore, due to the fact that in real-world decision problems, it is more appropriate for decision-makers to use words to express their opinions, as opposed to numerical values, authors have attempted to add linguistic modeling capabilities to ELECTRE II. Zadeh [37] proposed the concept of linguistic variables, allowing decision-makers to express their opinions in a more natural manner. Since decision-makers are often hesitant when expressing their evaluations, the hesitant fuzzy linguistic term set (HFLTS) was proposed by Rodriguez et al. [38]. This allowed decision-makers to provide more than one linguistic term for each evaluation. The concept of HFLTS was then applied to ELECTRE II by Liao et al. [39]. However, the main disadvantage of HFLTS is that it assigns an equal importance weighting to all linguistic terms, which is often not desired in real-world decision-making cases. To overcome this, Pang et al. [19] developed the concept of a probabilistic linguistic term set (PLTS) whereby a decision-maker is able to specify different probability values for each of their selected linguistic terms. PLTS was well received in the field of decision making, with the development of many decision-making models combining existing MCDM techniques with the concept of PLTS, such as PL-MULTIMOORA [40], PL-PROMETHEE [41], PL-VIKOR [42], and PL-CODAS [43]. The ELECTRE II method has also been enhanced to perform under the PLTS environment. Pan et al. [44] developed the PL-ELECTRE II model to solve the problem of evaluating therapeutic scheduling for patients suffering from brain-metastasized non-small cell lung cancer. In another study by Shen et al. [45], the authors developed the PLTS-ELECTRE II model for solving a venture capital evaluation problem. The PLTS-ELECTRE II model was able to model both quantitative criteria and qualitative linguistic criteria. An extensive study on the PLTS and its many applications to MCDM can be found in [46].

Even though the PLTS has been fairly successful in modeling real-world decision problems, they suffer from one weakness. Oftentimes, different decision-makers provide their evaluations for a decision problem from different perspectives. They have different skill sets and experiences. The PLTS does not factor this, and therefore, the PLTS was recently combined with the concept of Z-numbers by Chai et al. [18], leading to the Z-Probabilistic Linguistic Term Set (ZPLTS). The concept of Z-numbers was proposed by Zadeh [47] as a way of assigning a credibility value to an evaluation value, and it has been applied to a range of MCDM models in recent years, resulting in Z-PROMETHEE II [48], Z-TOPSIS [49], and Z-MABAC [50], to name a few. Z-numbers are able to represent information and its credibility, and hence, it provides a solution to problematic results caused by the lack of information credibility in decision-making problems. ZPLTS combines the PLTS with Z-numbers in order to allow decision-makers to express their evaluations along with their credibility, both in the form of linguistic values with associated probabilities. The ZPLTS allows for richer modeling and decision-making capabilities for real-world decision problems. To the best of our knowledge, despite its capabilities, the ZPLTS has not been integrated with the ELECTRE II model. This study proposed the ZPLTS-ELECTRE II, which is an enhanced ELECTRE II model that combines the traditional ELECTRE II method with a Z-Probabilistic Linguistic Term Set. The new ZPLTS-ELECTRE II method is then applied to rank a set of ontologies of the mental health domain. The next section presents ELECTRE II and defines the various concepts that are relevant in the specification of the proposed ZPLTS-ELECTRE II method.

3. Preliminaries

In this section, a short explanation of ELECTRE II is given, which is followed by the definitions of the concepts of Linguistic Term Set, Hesitant Fuzzy Linguistic Term Set, Probabilistic Linguistic Term Set, Z-number and Z-Probabilistic Linguistic Term Set. To enhance the readability of the study, a brief description of the mathematical symbols and notations used in the paper is provided in

Table 1. Descriptions of mathematical symbols and notations.

Notation	Description	Notation	Description
p	probability value	$L(p)$	probabilistic linguistic term set
$\#L(p)$	number of terms in $L(p)$	${\widehat{Z}}^{\#}$	z probabilistic linguistic term
$\widehat{Z}$	z probabilistic linguistic value	$\varphi$	number of terms
S	linguistic term set	$S^{\prime }$	linguistic term set
$\varsigma$	max term in S	$\zeta$	max term in $S^{\prime }$
$S(·)$	score	$D(·)$	deviation degree
$d(·)$	distance	$\mathrm{\Lambda }$	quantitative matrix
$\mathrm{\Gamma }$	decision-makers evaluation matrix	m	number of alternatives
n	number of criteria	t	number of quantitative criteria
b	decision-maker	E	decision matrix
$\omega$	criterion weight	$\alpha$	concordance threshold
$\beta$	discordance threshold	$S^F$	strong outranking
$S^f$	weak outranking	$\psi (·)$	assigned rank
$N^F$	non-dominant strong alternatives	$N^f$	non-dominant weak alternatives
$G^F$	strong outranking graph	$G^f$	weak outranking graph

.

3.1. ELECTRE II

The ELECTRE II method was developed to solve ranking problems, yielding a ranking of alternatives from best to worst. The method requires a set of alternatives and a corresponding set of criteria, a set of importance weights for each criterion, and five thresholds. The ELECTRE II method progresses by building the concordance and discordance relations between all alternative pairs, which are based on the number of criteria that agree that a particular alternative is at least as good as another and the number of criteria that disagree. These concordance and discordance relations are evaluated against the thresholds to assign alternative pairs into a strong outranking relation or a weak outranking relation. Finally, a forward and reverse ranking order procedure is applied to the strong and weak outranking relations to generate two sets of rankings. The two rankings are then combined to form the final ranking. Further details on the ELECTRE II method can be found in [29,51].

3.2. Linguistic Term Set and Hesitant Fuzzy Linguistic Term Set

Definition 1.

Let S be a linguistic term set where $S=\{S_t|t=-\varsigma ,\dots ,-1,0,1,\dots ,\varsigma \}$, with odd cardinality. The midterm is symbolic of indifference, and the other linguistic terms are placed symmetrically centered around the midterm.

During the decision-making process, it is often difficult for decision-makers to express their opinions using crisp values. Therefore, the concept of Linguistic Term Set (LTS) is useful to provide decision-makers the opportunity to better express themselves. An LTS is a set of linguistic terms that a decision-maker may pick from to evaluate a particular criterion. One issue that arises is that decision-makers may feel hesitant to give their opinions on a criterion. The Hesitant Fuzzy Linguistic Term Set (HFLTS) [38] was proposed as a solution to this problem, whereby a decision-maker is able to select more than one linguistic value from the LTS to evaluate a given criterion.

Definition 2.

Let $S=\{S_t|t=-\varsigma ,\dots ,-1,0,1,\dots ,\varsigma \}$ be an LTS. An HFLTS is defined as an ordered finite subset of S [38].

Example 1.

Let $S=\{s_{-2}=\mathit{very}\mathit{low},s_{-1}=\mathit{low},s_0=\mathit{medium},s_1=\mathit{high},s_2=\mathit{very}\mathit{high}\}$ be an LTS with five terms. An HFLTS is $h_s(x_1)=\{s_1,s_2\}$, which signifies high and very high. Another HFLTS signifying ’at least medium’ is $h_s(x_2)=\{s_0,s_1,s_2\}$, which is a subset of linguistic terms from S comprising ’medium’, ’high’, and ’very high’.

3.3. Probabilistic Linguistic Term Set

One problem with HFLTSs is that it assumes that all linguistic values within the set are equally important and are given equal probability. This is not always the case, as sometimes, a decision-maker may wish to assign varying probability distributions across an HFLTS. To accomplish this, the concept of a Probabilistic Linguistic Term Set (PLTS) was proposed [19].

Definition 3.

Let $S=\{S_t|t=-\varsigma ,\dots ,-1,0,1,\dots ,\varsigma \}$ be an LTS. A PLTS is defined as:

$$L(p)=\{L^m(p^m)|L^m\in S,p^m\le 0,m=1,2,\dots ,\#L(p),\sum _{m=1}^{\#L(p)}{p}^m\le 1\}$$

(1)

where $L^m$ is a linguistic term, $p^m$ is its associated probability, and the number of linguistic terms in $L(p)$ is denoted by $\#L(p)$.

3.4. Z-Number

A Z-number [47] enhances the process of selecting linguistic terms by adding a credibility variable. This enables different decision-makers to not only evaluate a criterion but also to specify their credibility levels.

Definition 4.

A Z-number, Z, is defined as an ordered pair of fuzzy numbers, $Z=(A,B)$. The value of A is a description or a restriction value, and the value of B represents the credibility measure for the values A.

3.5. Z-Probabilistic Linguistic Term Set

The concept of PLTS is very powerful, but one downside is that it ignores the credibility of decision-makers. This can lead to inaccurate results in the decision-making process. To overcome this, the Z-Probabilistic Linguistic Term Set (ZPLTS) was proposed [18].

Definition 5.

Let X be a non-empty set. A ZPLTS, ${\widehat{Z}}^{\#}$, is defined as ${\widehat{Z}}^{\#}=\{\langle x,\widehat{Z}\rangle |x\in X\}$, with $\widehat{Z}$ being a Z Probabilistic Value [18].

Definition 6.

Let $L_A(p)$ and $L_B(p)$ be two PLTSs. A Z Probabilistic Linguistic Value (ZPLV) is defined as [18]:

$$\widehat{Z}=(\widehat{A},\widehat{B})=(L_{\widehat{A}}(p),L_{\widehat{B}}(p))$$

(2)

where $L_{\widehat{A}}(p)=\{L_{\widehat{A}}^m(p_{\widehat{A}}^m)|L_{\widehat{A}}^m\in S,p_{\widehat{A}}^m\le 0,m=1,2,\dots ,\#L_{\widehat{A}}(p_{\widehat{A}}),{\sum }_{m=1}^{\#L_{\widehat{A}}(p_{\widehat{A}})}{p}_{\widehat{A}}^m\le 1\}$,

$L_{\widehat{B}}(p)=\{L_{\widehat{B}}^n(p_{\widehat{B}}^n)|L_{\widehat{B}}^n\in S^{\prime },p_{\widehat{B}}^n\le 0,n=1,2,\dots ,\#L_{\widehat{B}}(p_{\widehat{B}}),{\sum }_{n=1}^{\#L_{\widehat{B}}(p_{\widehat{B}})}{p}_{\widehat{B}}^n\le 1\}$, and

$S=\{-\varsigma ,\dots ,-1,0,1,\dots ,\varsigma \}$ and $S^{\prime }=\{-\zeta ,\dots ,-1,0,1,\dots ,\zeta \}$ are LTSs.

In order to normalize a ZPLV, both the value and the credibility sets, that is, $L_{\widehat{A}}(p)$ and $L_{\widehat{B}}(p)$, must be normalized. A ZPLV is normalized in two stages. Firstly, the probability distribution of the linguistic terms is normalized, and thereafter, the number of linguistic terms is normalized. The first stage is required whenever the sums of the probabilities are less than 1; then, the probabilities must be normalized to equate to 1.

Definition 7.

Let $\widehat{Z}$ be a ZPLV with $\widehat{Z}=(L_{\widehat{A}}(p),L_{\widehat{B}}(p))$, and ${\sum }_{m=1}^{\#L_{\widehat{A}}(p_{\widehat{A}})}{p}_{\widehat{A}}^m\le 1$ and${\sum }_{n=1}^{\#L_{\widehat{B}}(p_{\widehat{B}})}{p}_{\widehat{B}}^n\le 1$. The associated ZPLV after normalizing the probability distributions can be defined as [18]:

$$\widehat{Z}=({\dot{L}}_{\widehat{A}}(p),{\dot{L}}_{\widehat{B}}(p))=(\{L_{\widehat{A}}^m({\dot{p}}_{\widehat{A}}^m)\},\{L_{\widehat{B}}^n({\dot{p}}_{\widehat{B}}^n)\})$$

(3)

where ${\dot{p}}_{\widehat{A}}^m=\frac{p_{\widehat{A}}^m}{{\sum }_{m=1}^{\#L_{\widehat{A}}(p_{\widehat{A}})}{p}_{\widehat{A}}^m}$ and ${\dot{p}}_{\widehat{B}}^n=\frac{p_{\widehat{B}}^n}{{\sum }_{n=1}^{\#L_{\widehat{B}}(p_{\widehat{B}})}{p}_{\widehat{B}}^n}$,

for $m=1,2,\dots ,\#L_{\widehat{A}}(p_{\widehat{A}})$ and $n=1,2,\dots ,\#L_{\widehat{B}}(p_{\widehat{B}})$.

After the first normalization stage, that is, normalizing the probabilities to equate to 1, the second stage is to normalize the number of linguistic terms. This is required in order to calculate the distance between two ZPLVs.

Definition 8.

Let ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$ be two ZPLVs with ${\widehat{Z}}_1=(L_{\widehat{A}1}(p),L_{\widehat{B}1}(p))$ and

${\widehat{Z}}_2=(L_{\widehat{A}2}(p),L_{\widehat{B}2}(p))$. The number of linguistic terms in a PLTS is given by $\#L(p)$. In the case that $\#L_{\widehat{A}1}(p_{\widehat{A}1})>\#L_{\widehat{A}2}(p_{\widehat{A}2})$, then the number of terms need to be normalized as follows [18]:

1.

$L_{\widehat{A}2}$ is increased by adding ϕ terms to $L_{\widehat{A}2}$, where $\varphi =\#L_{\widehat{A}1}(p_{\widehat{A}1})-\#L_{\widehat{A}2}(p_{\widehat{A}2})$. The ϕ linguistic terms to be added may be any linguistic terms in $L_{\widehat{A}2}(p)$.

2.

The probabilities of the ϕ linguistic terms that were added must be set to 0.

The same process applies when $\#L_{\widehat{B}1}(p_{\widehat{B}1})>\#L_{\widehat{B}2}(p_{\widehat{B}2})$.

Example 2.

Let ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$ be two ZPLVs with ${\widehat{Z}}_1=(\{s_0(0.2),s_1(0.8)\},\{s_1^{\prime }(0.4),s_2^{\prime }(0.2),s_3^{\prime }(0.2)\})$ and ${\widehat{Z}}_2=(\{s_{-2}(0.2),s_{-1}(0.4),s_0(0.2)\},\{s_{-2}^{\prime }(0.2),s_0^{\prime }(0.2),s_2^{\prime }(0.4)\})$. In order to normalize the ZPLVs, firstly, all probabilities must add up to 1. This results in

${\widehat{Z}}_1=(\{s_0(0.2),s_1(0.8)\},\{s_1^{\prime }(0.5),s_2^{\prime }(0.25),s_3^{\prime }(0.25)\})$ and

${\widehat{Z}}_2=(\{s_{-2}(0.25),s_{-1}(0.5),s_0(0.25)\},\{s_{-2}^{\prime }(0.25),s_0^{\prime }(0.25),s_2^{\prime }(0.5)\})$. The next step is to ensure that the cardinality of the evaluation and credibility values for each ZPLV is equal, resulting in

${\widehat{Z}}_1=(\{s_0(0.2),s_1(0.8),s_1(0)\},\{s_1^{\prime }(0.4),s_2^{\prime }(0.2),s_3^{\prime }(0.2)\})$ and

${\widehat{Z}}_2=(\{s_{-2}(0.2),s_{-1}(0.4),s_0(0.2)\},\{s_{-2}^{\prime }(0.2),s_0^{\prime }(0.2),s_2^{\prime }(0.4)\})$.

In order to make use of ZPLVs, they need to be comparable with each other. Chai et al. [18] proposed the score and deviation methods to compare two ZPLVs.

Definition 9.

Let $\widehat{Z}=(\widehat{A},\widehat{B})=(L_{\widehat{A}}(p),L_{\widehat{B}}(p))$ be a ZPLV, where

$(\{L_{\widehat{A}}^m(p_{\widehat{A}}^m)|m=1,2,\dots ,\#L_{\widehat{A}}(p_{\widehat{A}})\},\{L_{\widehat{B}}^n(p_{\widehat{B}}^n)|n=1,2,\dots ,\#L_{\widehat{B}}(p_{\widehat{B}})\})$. The subscripts of $L_{\widehat{A}}^m$ and $L_{\widehat{B}}^n$ are denoted as $v_{\widehat{A}}^m$ and $v_{\widehat{B}}^n$, respectively. The score of $\widehat{Z}$ is defined as [18]:

$$S(\widehat{Z})=(\overline{\alpha }+\varsigma )({\overline{\alpha }}^{\prime }+\zeta )$$

(4)

where $\overline{\alpha }=\frac{{\sum }_{m=1}^{\#L_{\widehat{A}}(p_{\widehat{A}})}{v}_{\widehat{A}}^m{p}_{\widehat{A}}^m}{{\sum }_{m=1}^{\#L_{\widehat{A}}(p_{\widehat{A}})}{p}_{\widehat{A}}^m}$ and ${\overline{\alpha }}^{\prime }=\frac{{\sum }_{n=1}^{\#L_{\widehat{B}}(p_{\widehat{B}})}{v}_{\widehat{B}}^n{p}_{\widehat{B}}^n}{{\sum }_{n=1}^{\#L_{\widehat{B}}(p_{\widehat{B}})}{p}_{\widehat{B}}^n}$

Larger scores correlate with larger ZPLVs, allowing for their comparison. However, in the case that two ZPLVs have equal scores, then their deviation degrees may be calculated to differentiate between the ZPLVs further.

Definition 10.

Let $\widehat{Z}=(\widehat{A},\widehat{B})=(L_{\widehat{A}}(p),L_{\widehat{B}}(p))$, where

$(\{L_{\widehat{A}}^m(p_{\widehat{A}}^m)|m=1,2,\dots ,\#L_{\widehat{A}}(p_{\widehat{A}})\},\{L_{\widehat{B}}^n(p_{\widehat{B}}^n)|n=1,2,\dots ,\#L_{\widehat{B}}(p_{\widehat{B}})\})$, their subscripts are given as $v_{\widehat{A}}^m$ and $v_{\widehat{B}}^n$, and the score of $\widehat{Z}$ is given as $S(\widehat{Z})=(\overline{\alpha }+\varsigma )({\overline{\alpha }}^{\prime }+\zeta )$. The deviation degree of $\widehat{Z}$ is defined as [18]:

$$D(\widehat{Z})=\sqrt{\frac{{\sum }_{m=1}^{\#L_{\widehat{A}}(p_{\widehat{A}})}{\sum }_{n=1}^{\#L_{\widehat{B}}(p_{\widehat{B}})}{((p_{\widehat{A}}^m{v}_{\widehat{A}}^m+\varsigma )(p_{\widehat{B}}^n{v}_{\widehat{B}}^n+\zeta )-(\overline{\alpha }+\varsigma )({\overline{\alpha }}^{\prime }+\zeta ))}^2}{\#L_{\widehat{A}}(p_{\widehat{A}})\#L_{\widehat{B}}(p_{\widehat{B}})}}$$

(5)

After determining the score and deviation degree, ZPLVs can be compared as follows.

Definition 11.

Let ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$ be two ZPLVs. The comparison between ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$ is defined as [18]:

1. 

If $S({\widehat{Z}}_1)>S({\widehat{Z}}_2)$ then ${\widehat{Z}}_1>{\widehat{Z}}_2$.

2. 

If $S({\widehat{Z}}_1)<S({\widehat{Z}}_2)$ then ${\widehat{Z}}_1<{\widehat{Z}}_2$.

3. 

If $S({\widehat{Z}}_1)=S({\widehat{Z}}_2)$ then the deviation degree is required to compare ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$ as follows:

(a) 

If $D({\widehat{Z}}_1)>D({\widehat{Z}}_2)$ then ${\widehat{Z}}_1<{\widehat{Z}}_2$.

(b) 

If $D({\widehat{Z}}_1)<D({\widehat{Z}}_2)$ then ${\widehat{Z}}_1>{\widehat{Z}}_2$.

(c) 

If $D({\widehat{Z}}_1)=D({\widehat{Z}}_2)$ then ${\widehat{Z}}_1\approx {\widehat{Z}}_2$.

(d) 

If $L_{\widehat{A}1}(p)=L_{\widehat{A}2}(p)$ and $L_{\widehat{B}1}(p)=L_{\widehat{B}2}(p)$ then ${\widehat{Z}}_1={\widehat{Z}}_2$.

Example 3.

Let ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$ be two ZPLVs, with ${\widehat{Z}}_1=\{\{s_0(0.2),s_1(0.8)\},\{s_1^{\prime }(0.55),s_2^{\prime }(0.45)\}\}$ and ${\widehat{Z}}_2=\{\{s_{-1}(0.75),s_0(0.25)\},\{s_1^{\prime }(0.1),s_2^{\prime }(0.9)\}\}$. There are two terms in $L_{\widehat{A}1}(p_{\widehat{A}1})$, two terms in $L_{\widehat{B}1}(p_{\widehat{B}1})$, two terms in $L_{\widehat{A}2}(p_{\widehat{A}2})$, and two terms in $L_{\widehat{B}2}(p_{\widehat{B}2})$. The ς and ζ values are both 3. The scores of ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$, denoted as $S({\widehat{Z}}_1)$ and $S({\widehat{Z}}_2)$, are calculated as follows.

$S({\widehat{Z}}_1)=(\overline{\alpha }+3)({\overline{\alpha }}^{\prime }+3)$, and $\overline{\alpha }=\frac{0\times 0.2+1\times 0.8}{0.2+0.8}=0.8$, and ${\overline{\alpha }}^{\prime }=\frac{1\times 0.55+2\times 0.45}{0.55+0.45}=1.45$

$\therefore S({\widehat{Z}}_1)=(0.8+3)(1.45+3)=16.91$

Similarly, the score of ${\widehat{Z}}_2$ can be calculated as follows.

$S({\widehat{Z}}_2)=(\overline{\alpha }+3)({\overline{\alpha }}^{\prime }+3)$, and $\overline{\alpha }=\frac{-1\times 0.75+0\times 0.25}{0.75+0.25}=-0.75$, and ${\overline{\alpha }}^{\prime }=\frac{1\times 0.1+2\times 0.9}{0.1+0.9}=1.9$

$\therefore S({\widehat{Z}}_2)=(-0.75+3)(1.9+3)=11.025$

$\because S({\widehat{Z}}_1)>S({\widehat{Z}}_2)\Rightarrow {\widehat{Z}}_1>{\widehat{Z}}_2$.

Example 4.

Let ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$ be two ZPLVs, with ${\widehat{Z}}_1=\{\{s_0(0.8),s_1(0.2)\},\{s_0^{\prime }(0.3),s_1^{\prime }(0.7)\}\}$ and ${\widehat{Z}}_2=\{\{s_0(0.9),s_2(0.1)\},\{s_0^{\prime }(0.65),s_2^{\prime }(0.35)\}\}$. There are two terms in $L_{\widehat{A}1}(p_{\widehat{A}1})$, two terms in $L_{\widehat{B}1}(p_{\widehat{B}1})$, two terms in $L_{\widehat{A}2}(p_{\widehat{A}2})$, and two terms in $L_{\widehat{B}2}(p_{\widehat{B}2})$. The ς and ζ values are both 3. The scores of ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$, denoted as $S({\widehat{Z}}_1)$ and $S({\widehat{Z}}_2)$, are calculated as follows.

$S({\widehat{Z}}_1)=(0.2+3)(0.7+3)=11.84$, and $S({\widehat{Z}}_2)=(0.2+3)(0.7+3)=11.84$.

Since $S({\widehat{Z}}_1)=S({\widehat{Z}}_2)$, the deviation degrees of ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$ must be calculated to compare the ZPLVs further. This is performed as follows.

$D({\widehat{Z}}_1)=\sqrt{\frac{{(3\times 3-11.84)}^2+{(3\times 3.7-11.84)}^2+{(3.2\times 3-11.84)}^2+{(3.2\times 3.7-11.84)}^2}{2\times 2}}\approx 1.85$

Similarly, the deviation degree for ${\widehat{Z}}_2$ can be calculated as follows.

$D({\widehat{Z}}_2)=\sqrt{\frac{{(3\times 3-11.84)}^2+{(3\times 3.7-11.84)}^2+{(3.2\times 3-11.84)}^2+{(3.2\times 3.7-11.84)}^2}{2\times 2}}\approx 1.85$

Since $S({\widehat{Z}}_1)=S({\widehat{Z}}_2)$ and $D({\widehat{Z}}_1)=D({\widehat{Z}}_2)$, it implies that ${\widehat{Z}}_1\approx {\widehat{Z}}_2$.

To calculate the distance between two ZPLVs, a distance measure based on the Euclidean distance is used [18].

Definition 12.

Let two ZPLVS be defined as ${\widehat{Z}}_1=({\widehat{A}}_1,{\widehat{B}}_1)=(L_{{\widehat{A}}_1}(p),L_{\widehat{B}1}(p))$ =

$(\{L_{\widehat{A}1}^m(p_{\widehat{A}1}^m)|m=1,2,\dots ,\#L_{\widehat{A}1}(p_{\widehat{A}1})\},\{L_{\widehat{B}1}^n(p_{\widehat{B}1}^n)|n=1,2,\dots ,\#L_{\widehat{B}1}(p_{\widehat{B}1})\})$,

and ${\widehat{Z}}_2=({\widehat{A}}_2,{\widehat{B}}_2)=(L_{{\widehat{A}}_2}(p),L_{{\widehat{B}}_2}(p))$ =

$(\{L_{\widehat{A}2}^h(p_{\widehat{A}2}^h)|h=1,2,\dots ,\#L_{\widehat{A}2}(p_{\widehat{A}2})\},\{L_{\widehat{B}2}^k(p_{\widehat{B}2}^k)|k=1,2,\dots ,\#L_{\widehat{B}2}(p_{\widehat{B}2})\})$. If $\#L_{\widehat{A}1}(p_{\widehat{A}1})=\#L_{\widehat{A}2}(p_{\widehat{A}2})$ and $\#L_{\widehat{B}1}(p_{\widehat{B}1})=\#L_{\widehat{B}2}(p_{\widehat{B}2})$, then the distance between the two ZPLVs, ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$, is defined as:

$$d({\widehat{Z}}_1,{\widehat{Z}}_2)=\sqrt{\frac{{\sum }_{m=1}^{\#L_{\widehat{A}1}(p_{\widehat{A}1})}{\sum }_{n=1}^{\#L_{\widehat{B}1}(p_{\widehat{B}1})}{((p_{\widehat{A}1}^m{v}_{\widehat{A}1}^m+\varsigma )(p_{\widehat{B}1}^n{v}_{\widehat{B}1}^n+\zeta )-(p_{\widehat{A}2}^m{v}_{\widehat{A}2}^m+\varsigma )(p_{\widehat{B}2}^n{v}_{\widehat{B}2}^n+\zeta ))}^2}{\#L_{\widehat{A}1}(p_{\widehat{A}1})\#L_{\widehat{B}1}(p_{\widehat{B}1})}}$$

(6)

Example 5.

Let ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$ be two ZPLVs, with ${\widehat{Z}}_1=\{\{s_0(0.2),s_1(0.8)\},\{s_1^{\prime }(0.55),s_2^{\prime }(0.45)\}\}$ and ${\widehat{Z}}_2=\{\{s_{-1}(0.75),s_0(0.25)\},\{s_1^{\prime }(0.1),s_2^{\prime }(0.9)\}\}$. The distance between ${\widehat{Z}}_1$ and ${\widehat{Z}}_2$, denoted as $d({\widehat{Z}}_1,{\widehat{Z}}_2)$, is calculated as follows. There are two terms in $L_{\widehat{A}1}(p_{\widehat{A}1})$, two terms in $L_{\widehat{B}1}(p_{\widehat{B}1})$, two terms in $L_{\widehat{A}2}(p_{\widehat{A}2})$, and two terms in $L_{\widehat{B}2}(p_{\widehat{B}2})$. The ς and ζ values are both 3.

$d({\widehat{Z}}_1,{\widehat{Z}}_2)=\sqrt{\frac{{(3\times 3.55-2.25\times 3.1)}^2+{(3\times 3.9-2.25\times 4.8)}^2+{(3.8\times 3.55-3\times 3.1)}^2+{(3.8\times 3.9-3\times 4.8)}^2}{2\times 2}}$

$d({\widehat{Z}}_1,{\widehat{Z}}_2)\approx 2.83$.

The concepts defined above are applied to specify the proposed ZPLTS-ELECTRE II method in the next section.

4. ZPLTS-ELECTRE II

The ZPLTS-ELECTRE II method involves the creation of the quantitative, decision-makers evaluation and decision matrices, the definition of weights, thresholds and comparative sets, the determination of the concordance and discordance relations, and the construction of weak and strong outranking graphs. These building blocks of the ZPLTS-ELECTRE II method are defined and explained in the next subsections.

4.1. Quantitative Matrix

The quantitative matrix, denoted as $\mathrm{\Lambda }$, represents the performances of all m alternatives in light of t quantitative criteria. The quantitative matrix $\mathrm{\Lambda }$ is defined in Equation (7). Each element of $\mathrm{\Lambda }$, $a_{ij}$, represents the performance of the $i^{th}$ alternative in light of the $j^{th}$ criterion.

$$\mathrm{\Lambda }={\left[a_{i,j}\right]}_{m\times t}=\left[\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \dots & a_{1,t}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m,1}& a_{m,2}& \dots & a_{m,t}\end{array}\right]$$

(7)

4.2. Decision-Makers Evaluation Matrix

If the number of decision-makers who are going to evaluate the alternatives is denoted by r, then there will be r decision matrices. Each matrix is denoted by ${\mathrm{\Gamma }}^b$, where $b=1,2,\dots ,r$. There are m alternatives that each decision-maker needs to evaluate; hence, there are m rows in each matrix, and there are $(n-t)$ criteria that each decision-maker must judge for each alternative, resulting in $(n-t)$ columns in each decision matrix. ${\mathrm{\Gamma }}^b$ is defined in Equation (8). Each element ${\widehat{Z}}_{ij}$ is a ZPLV that represents the linguistic evaluation for the $i^{th}$ alternative regarding the $j^{th}$ criterion along with the linguistic term for the decision-makers’ credibility regarding that evaluation.

$${\mathrm{\Gamma }}^b={\left[{\widehat{Z}}_{i,j}\right]}_{m\times (n-t)}=\left[\begin{array}{cccc}{\widehat{Z}}_{1,1}& {\widehat{Z}}_{1,2}& \dots & {\widehat{Z}}_{1,n-t}\\ ⋮& ⋮& \ddots & ⋮\\ {\widehat{Z}}_{m,1}& {\widehat{Z}}_{m,2}& \dots & {\widehat{Z}}_{m,n-t}\end{array}\right]$$

(8)

4.3. Decision Matrix

The ${\mathrm{\Gamma }}^b$ matrices are then combined to form one matrix, $\mathrm{\Gamma }$. This is completed by combining the probability values for all of the like terms. Thereafter, the matrix $\mathrm{\Gamma }$ is normalized in two stages. Firstly, the probability distribution is normalized, and secondly, the number of linguistic variables is normalized. To normalize the probability distribution, whenever the sums of the probabilities are less than 1, the remaining probability value needs to be further allocated. That is, if $\widehat{Z}=(L_{\widehat{A}}(p),L_{\widehat{B}}(p))$ is a ZPLV with ${\sum }_{m=1}^{\#L_{\widehat{A}}(p_{\widehat{A}})}{p}_{\widehat{A}}^m<1$ and ${\sum }_{n=1}^{\#L_{\widehat{B}}(p_{\widehat{B}})}{p}_{\widehat{B}}^n<1$, then the remaining probabilities must be allocated so that the probability sums are equal to 1. This is completed by applying Equation (3).

After normalizing the probability distributions, the next step is to normalize the number of linguistic variables by making the number of terms in $L_A(p)$ and $L_B(p)$ equal, that is, $\#L_A(p)=\#L_B(p)$. This is demonstrated in Definition 8.

After combining and normalizing the ZPLV decision matrices to form one matrix, $\mathrm{\Gamma }$, the decision matrix, E, is built by concatenating $\mathrm{\Lambda }$ with $\mathrm{\Gamma }$, as shown in Equation (9).

$$E=\left[\mathrm{\Lambda }\mathrm{\Gamma }\right]=\left[\begin{array}{cccccc}{d}_{1,1}& \dots & d_{1,t}& {\widehat{Z}}_{1,1}& \dots & {\widehat{Z}}_{1,n-t}\\ ⋮& \ddots & ⋮& ⋮& \ddots & ⋮\\ d_{m,1}& \dots & d_{m,t}& {\widehat{Z}}_{m,1}& \dots & {\widehat{Z}}_{m,n-t}\end{array}\right]$$

(9)

4.4. Weights and Thresholds

A set of weights, $\omega =\{{\omega }_1,{\omega }_2,\dots ,{\omega }_n\}$, must be defined, where ${\omega }_j$ represents the importance weighting for the $j^{th}$ criterion. Three concordance thresholds, ${\alpha }_1$, ${\alpha }_2$, and ${\alpha }_3$, must also be defined such that $0.5\le {\alpha }_1\le {\alpha }_2\le {\alpha }_3\le 1$. An upper and a lower discordance threshold must be defined for each criterion as well. ${\beta }_j^{-}$ represents the lower discordance threshold, and ${\beta }_j^{+}$ represents the upper discordance threshold for criterion j, where ${\beta }_j^{+}\ge {\beta }_j^{-}\ge 0$.

4.5. Comparative Sets

Three sets are determined to compare each alternative with every other alternative. ${\mathcal{B}}^{-}(x,y)$ represents those criteria for which alternative x performs worst than alternative y. ${\mathcal{B}}^0(x,y)$ represents those criteria for which alternative x performs equally well as alternative y. ${\mathcal{B}}^{+}(x,y)$ represents those criteria for which alternative x performs better than alternative y.

In the case of quantitative criteria, ${\mathcal{B}}^{-}(x,y)$ is defined in Equation (10) with $1\le j\le t$, and for ZPLV criteria in Equation (11), where $t+1\le j\le n$.

$${\mathcal{B}}^{-}(x,y)=\{j|x_j<y_j\}$$

(10)

$${\mathcal{B}}^{-}(x,y)=\{j|(S({\widehat{Z}}_{x,j})<S({\widehat{Z}}_{y,j}))\mathrm{or}(S({\widehat{Z}}_{x,j})=S({\widehat{Z}}_{y,j})\mathrm{and}D({\widehat{Z}}_{x,j})>D({\widehat{Z}}_{y,j}))\}$$

(11)

In the case of quantitative criteria, ${\mathcal{B}}^0(x,y)$ is defined in Equation (12), where $1\le j\le t$, and for ZPLV criteria in Equation (13), where $t+1\le j\le n$.

$${\mathcal{B}}^0(x,y)=\{j|x_j=y_j\}$$

(12)

$${\mathcal{B}}^0(x,y)=\{j|S({\widehat{Z}}_{x,j})=S({\widehat{Z}}_{y,j})\mathrm{and}D({\widehat{Z}}_{x,j})=D({\widehat{Z}}_{y,j})\}$$

(13)

${\mathcal{B}}^{+}(x,y)$ is defined in Equation (14) in the case of quantitative criteria, where $1\le j\le t$, and in Equation (15) for ZPLV criteria with $t+1\le j\le n$. $S({\widehat{Z}}_{i,j})$ represents the score given in Equation (4) and $D({\widehat{Z}}_{i,j})$ represents the deviation degree given in Equation (5).

$${\mathcal{B}}^{+}(x,y)=\{j|x_j>y_j\}$$

(14)

$${\mathcal{B}}^{+}(x,y)=\{j|(S({\widehat{Z}}_{x,j})>S({\widehat{Z}}_{y,j}))\mathrm{or}(S({\widehat{Z}}_{x,j})=S({\widehat{Z}}_{y,j})\mathrm{and}D({\widehat{Z}}_{x,j})<D({\widehat{Z}}_{y,j}))\}$$

(15)

4.6. Concordance Relations

The concordance values for every alternative, $(x,y)$, can be determined by applying Equation (16). This represents the weighting of criteria that concords with the statement that alternative x is at least as good as alternative y.

$$C(x,y)=\frac{{\sum }_{j\in {\mathcal{B}}^{+}(x,y)}^{}{\omega }_j+{\sum }_{j\in {\mathcal{B}}^0(x,y)}^{}{\omega }_j}{{\sum }_{j=1}^n{\omega }_j}$$

(16)

4.7. Discordance Relations

The discordance value measures how true the statement that alternative x is outranked by alternative y is. To determine the discordance values for each alternative pair, three discordance sets are formed: the high discordance set ${\mathcal{Q}}^{+}(x,y)$, the medium discordance set ${\mathcal{Q}}^0(x,y)$, and the low discordance set ${\mathcal{Q}}^{-}(x,y)$. The discordance sets are formulated as follows.

The high discordance set is formulated by applying Equation (17) for quantitative criteria, where $j=1,2,\dots ,t$, and in the case of qualitative criteria, Equation (18) is applied, where $j=n-t+1,n-t+2,\dots ,n$.

$${\mathcal{Q}}^{+}(x,y)=\{j|y_j-x_j\ge {\beta }_j^{+}\}$$

(17)

$${\mathcal{Q}}^{+}(x,y)=\{j|d(x_j,y_j)\ge {\beta }_j^{+},j\in {\mathcal{B}}^{+}(y,x)\}$$

(18)

The medium discordance set is formulated by applying Equation (19) for quantitative criteria, where $j=1,2,\dots ,t$, and in the case of qualitative criteria, Equation (20) is applied, where $j=n-t+1,n-t+2,\dots ,n$.

$${\mathcal{Q}}^0(x,y)=\{j|{\beta }_j^{-}<y_j-x_j<{\beta }_j^{+}\}$$

(19)

$${\mathcal{Q}}^0(x,y)=\{j|{\beta }_j^{-}<d(x_j,y_j)<{\beta }_j^{+},j\in {\mathcal{B}}^{+}(y,x)\}$$

(20)

The low discordance set is formulated by applying Equation (21) for quantitative criteria, where $j=1,2,\dots ,t$, and in the case of qualitative criteria, Equation (22) is applied. When the discordance threshold is larger than 0 (${\beta }^{-}>0$) or is equal to 0 (${\beta }^{-}=0$), then Equation (23) is applied, where $j=n-t+1,n-t+2,\dots ,n$.

$${\mathcal{Q}}^{-}(x,y)=\{j|y_j-x_j\le {\beta }_j^{-}\}$$

(21)

$${\mathcal{Q}}^{-}(x,y)=\{j|(d(x_j,y_j)\le {\beta }_j^{-},j\in {\mathcal{B}}^{+}(y,x))or(j\in {\mathcal{B}}^{+}(x,y))or(j\in {\mathcal{B}}^0(x,y))\}$$

(22)

$${\mathcal{Q}}^{-}(x,y)=\{j|(j\in {\mathcal{B}}^0(x,y))or(j\in {\mathcal{B}}^{+}(x,y))\}$$

(23)

4.8. Strong and Weak Outranking Graphs

Each alternative pair may have a strong outranking relation, $x{S}^{F}y$, or a weak outranking relation, $x{S}^{f}y$. According to these relations, a strong outranking graph, and a weak outranking graph can be constructed. The conditions for the strong outranking relations are given in Equations (24) and (25).

$$x{S}^{F}y\iff \left\{\begin{array}{c}\hfill C(x,y)\ge {\alpha }_3\hfill \\ \hfill j\in {\mathcal{Q}}^0(x,y)\mathrm{or}{\mathcal{Q}}^{-}(x,y),\forall j\hfill \\ \hfill \frac{{\sum }_{j\in {\mathcal{B}}^{+}(x,y)}^{}{\omega }_j}{{\sum }_{j\in {\mathcal{B}}^{-}(x,y)}^{}{\omega }_j}\ge 1\hfill \end{array}\right.$$

(24)

or

$$x{S}^{F}y\iff \left\{\begin{array}{c}\hfill C(x,y)\ge {\alpha }_2\hfill \\ \hfill j\in {\mathcal{Q}}^{-}(x,y),\forall j\hfill \\ \hfill \frac{{\sum }_{j\in {\mathcal{B}}^{+}(x,y)}^{}{\omega }_j}{{\sum }_{j\in {\mathcal{B}}^{-}(x,y)}^{}{\omega }_j}\ge 1\hfill \end{array}\right.$$

(25)

The conditions for the weak outranking relations are given in Equations (26) and (27).

$$x{S}^{f}y\iff \left\{\begin{array}{c}\hfill C(x,y)\ge {\alpha }_2\hfill \\ \hfill j\in {\mathcal{Q}}^0(x,y),\forall j\hfill \\ \hfill \frac{{\sum }_{j\in {\mathcal{B}}^{+}(x,y)}^{}{\omega }_j}{{\sum }_{j\in {\mathcal{B}}^{-}(x,y)}^{}{\omega }_j}\ge 1\hfill \end{array}\right.$$

(26)

or

$$x{S}^{f}y\iff \left\{\begin{array}{c}\hfill C(x,y)\ge {\alpha }_1\hfill \\ \hfill j\in {\mathcal{Q}}^{-}(x,y),\forall j\hfill \\ \hfill \frac{{\sum }_{j\in {\mathcal{B}}^{+}(x,y)}^{}{\omega }_j}{{\sum }_{j\in {\mathcal{B}}^{-}(x,y)}^{}{\omega }_j}\ge 1\hfill \end{array}\right.$$

(27)

4.9. Exploit Outranking Relations

To build the strong and weak outranking graphs, the following procedures are followed. Regarding the strong outranking graph, if alternative x strongly outranks alternative y, $(x{S}^{F}y)$, then a directed arc is drawn from x to y. Similarly, if alternative x weakly outranks alternative y, $(x{S}^{f}y)$, then a directed arc is drawn in the weak outranking graph. According to these two graphs, a ranking can be determined as follows.

The ranking proceeds in three stages. First, the forward order is performed, then, the backward order is performed, and finally, the results of the two orders are combined to produce a final ranking. The forward order is performed as follows:

1.

Let $N_1^F$ denote the set of non-dominant alternatives in the strong outranking graph $G_1^F$. $N_1^F$ is filled by all those alternatives that are not outranked by any other alternatives, that is, all those alternatives that have no arcs going into them. The same is completed for the weak outranking graph, $G_1^f$, with the set $N_1^f$.

2.

The intersection of $N_1^F$ and $N_1^f$, $N_1^F\cap N_1^f$, is determined to produce the set $N_1$. The alternatives in $N_1$ are those that are not outranked in both the strong and the weak outranking graphs. All those alternatives in $N_1$ are given the forward rank of 1, that is, ${\psi }_1(x)=1$ for all alternatives in $N_1$.

3.

The nodes of those alternatives contained in $N_1$ can now be removed from the strong outranking and the weak outranking graphs along with their associated edges. After removing the nodes and edges, the resulting graphs are $G_2^F$ and $G_2^f$.

4.

The steps 1 to 3 are repeated until all alternatives have been ranked, with each iteration producing a new set of graphs $G_v^F$ and $G_v^f$. Eventually, all alternatives should be assigned a forward rank.

Thereafter, the reverse order is performed as follows:

1.

All the arrows in the strong and weak outranking graphs, $G_1^F$ and $G_1^f$, are reversed to form the mirror image graphs.

2.

Each alternative is assigned a rank, ${\psi }_2(x)$, in the same way as in the forward order from steps 1 to 3.

3.

Due to the graph reversals, each rank is transformed by applying Equation (28).

$${\displaystyle {\psi }_3(A_i)=1+\underset{A_v\in A}{max}{\psi }_2(A_v)-{\psi }_2(A_i)}$$

(28)

The final ranking can be obtained by combining the forward and reverse order rankings. This is completed by taking the mid-point of both rankings, as shown in Equation (29).

$$\overline{\psi }(A_i)=\frac{{\psi }_1(A_i)+{\psi }_3(A_i)}{2}$$

(29)

The steps and processes of the ZPLTS-ELECTRE II method presented above are summarized in Algorithm 1. The application of the proposed ZPLTS-ELECTRE II method is presented next to demonstrate its feasibility.

Algorithm 1 ZPLTS-ELECTRE II.

1:  Build quantitative matrix $\mathrm{\Lambda }$ comprising m alternatives with t quantitative criteria 2:  Build ZPLTS decision matrices by obtaining evaluation from r decision-makers. Each decision-maker expresses their evaluation in a decision matrix ${\mathrm{\Gamma }}^b$ 3:  Combine quantitative and linguistic decision matrices to form decision matrix E, with dimensions $m\times n$, by applying Equation (9) 4:  Define set of criteria importance weights, ${\omega }_j$, along with three concordance thresholds, ${\alpha }_1$, ${\alpha }_2$, and ${\alpha }_3$, as well as two discordance thresholds, ${\beta }_j^{-}$ and ${\beta }_j^{+}$, for the j criteria 5:  Construct comparative sets, ${\mathcal{B}}^{-}(x,y)$, ${\mathcal{B}}^0(x,y)$, and ${\mathcal{B}}^{+}(x,y)$, which are representative of the criteria for which an alternative is beaten by another alternative, is equal to another alternative, and beats another alternative 6:  Determine concordance relations for every alternative pair, $C(x,y)$, by applying Equation (16) 7:  Determine the high, medium, and low discordance sets for every alternative pair, ${\mathcal{Q}}^{+}(x,y)$, ${\mathcal{Q}}^0(x,y)$, and ${\mathcal{Q}}^{-}(x,y)$, by applying Equations (17) and (23) 8:  Build strong and weak outranking graphs, $x{S}^{F}y$ and $x{S}^{f}y$, by applying Equations (24) and (25) for the strong outranking relations, and Equations (26) and (27) for the weak outranking relations 9:  Assign a rank to each alternative using the forward and reverse order processes, ${\psi }_1(A_i)$ and ${\psi }_3(A_i)$, and thereafter combine the ranks using Equation (29)

5. Application of ZPLTS-ELECTRE II Method in Ontology Ranking

This section presents (1) the design of the experiment used to apply the ZPLTS-ELECTRE II method, (2) the description of the results obtained, (3) the comparative analysis of the performance of the ZPLTS-ELECTRE II method against the traditional ELECTRE II and fuzzy ELECTRE II methods, and (4) the discussion of the overall performance of the proposed ZPLTS-ELECTRE II method. The design of the experiment involves the choice of the dataset as well as the quantitative and qualitative attributes of the ontologies used as alternatives. The experiment is then carried out to apply the proposed ZPLTS-ELECTRE II method to rank the ontologies. The performance of the ZPLTS-ELECTRE II method is firstly compared against that of the traditional ELECTRE II and the PLTS ELECTRE II methods on the same dataset; thereafter, the ZPLTS-ELECTRE II method is compared against previous studies that have used MCDM methods in ontology ranking as well as the existing fuzzy ELECTRE II methods implemented in related studies. Finally, the strengths and weaknesses of the proposed ZPLTS-ELECTRE II method are described.

5.1. Experimental Design

This section presents the dataset, the qualitative and quantitative attributes, and the computer and software environments used in the experiments.

5.1.1. Dataset

The dataset used in the study is made up of nine ontologies pertaining to the mental health and well-being domain. In fact, the quality and state of mental well-being have seen a drastic decline in recent years [52,53]. The World Health Organization (WHO) estimated that 6.6 billion people around the world suffered from at least one form of mental health disorder [54].

Therefore, the issue of mental health and well-being is a growing concern around the world. Moreover, the advent of COVID-19 has further worsened the matter [55]. It is clear that there is an urgent need to study and develop techniques for the automated monitoring and management of the state of mental well-being throughout the world. Ontologies may also play a role in the development of technologies and systems to facilitate and provide support structures for mental health issues. For this reason, this study focused on the selection and ranking of real-world ontologies that model knowledge pertaining to mental wellness. The ontologies were obtained from the BioPortal Ontology Repository [7] and include:

1.

The Mental State Assessment Ontology (ONL-MSA)—ONL-MSA is a module of the OntoNeuroLOG [56] ontology, which was developed in the NeuroLog (http://neurolog.i3s.unice.fr/ accessed on 21 August 2022) project for enhancing the field of neuroimaging. The ONL-MSA ontology models knowledge pertaining to the mental state assessments.

2.

The APA Neuro Cluster Ontology (APANEUROCLUSTER)—it models the APA neuropsychology and neurology and includes the Assessment Diagnosis, Neurosciences, Neurological Disorders, and Neuroanatomy categories.

3.

The Ontologia de Saúde Mental (OSM)—it is the Portuguese equivalent of Mental Health Ontology, it was developed to assist in managing the Psychosocial Care Network in the Brazilian context, enabling the creation of intelligent computational tools and the development of mental health indicators.

4.

The Mental Functioning Ontology (MF)—it is an ontology that represents aspects of mental functioning, such as cognition and intelligence.

5.

The Alzheimer’s Disease Ontology (ADO)—it represents knowledge regarding Alzheimer’s disease. The main categories of the ontology include Health, Human, Neurologic Disease, and Neurological Disorder.

6.

The Neuroscience Information Framework Cell Ontology (NIFCELL)—it is part of the Neuroscience Information Framework (NIF) project (http://neuinfo.org accessed on 21 August 2022). The NIFCELL ontology expresses knowledge regarding cells and cell types from the Neuroscience Information Framework Standard Ontology (https://bioportal.bioontology.org/ontologies/NIFSTD accessed on 21 August 2022).

7.

The Epilepsy Semiology Ontology (EPISEM)—it was designed to capture the semiology of epilepsy. It models the signs and symptoms of epilepsy and represents the ictal, post-ictal, inter-ictal, and aura signs.

8.

The Cognitive Paradigm Ontology (COGPO) (http://www.cogpo.org/ accessed on 21 August 2022)—COGPO was developed to describe the experimental conditions within experiments related to the cognition and behavior of humans. The ontology defines the conditions of experiments in a standardized format.

9.

The Cognitive Atlas Ontology (COGAT) (https://www.cognitiveatlas.org/ accessed on 21 August 2022)—COGAT models and characterizes the state of current thought in cognitive science through a set of mental concepts and tasks. The ontology represents users’ knowledge with expertise in psychology, cognitive science, and neuroscience.

5.1.2. Quantitative Attributes

The quantitative metrics used in this study were adopted from the OntoMetrics framework [57], which defines various metrics for evaluating ontologies and provides an online environment for their automatic calculation. Several graph metrics have been defined to evaluate ontologies [57,58]. Five of these metrics were adopted in this study to demonstrate the effectiveness of the proposed ZPLTS-ELECTRE II method. The graph metrics are calculated from the graph and taxonomy tree of the ontology, thereby providing insights pertaining to the characteristics and attributes of the ontology [58]. The five graph metrics adopted measure the design complexity of ontologies and include the Absolute Leaf Cardinality (ALC), Absolute Root Cardinality (ARC), Average Depth (AD), Average Breadth (AB), and Average Number of Paths (ANP). The metrics are elaborated on as follows.

1.

The Absolute Leaf Cardinality metric, abbreviated ALC, is an indicator of the number of leaf nodes within a graph [58,59]; it expresses the dispersion of leaf nodes within a graph and is calculated by Equation (30).

$$m=n_{LEA\subseteq g}$$

(30)

where m is the ALC metric, and n is the cardinality of the set $LEA$, which is a subset of the directed graph g.

2.

The Absolute Root Cardinality metric, abbreviated ARC, expresses the number of root nodes in a graph [59,60]. The ARC is calculated by Equation (31).

$$m=n_{ROO\subseteq g}$$

(31)

where m is the ARC metric, and n is the cardinality of the set $ROO$, which is a subset of the directed graph g.

3.

Depth is a graph property of an ontology relating to the number of paths within the graph, considering the taxonomy is-a arcs only. The Average Depth metric, abbreviated AD, is determined by dividing the sum of the cardinalities of every path within a graph, by the cardinality of the set of all paths in that graph [58]. The AD is an indicator of the degree to which an ontology has a vertical modeling with many taxonomy is-a relations, which represent richer information content in the ontology [60]. The AD metric is calculated by Equation (32).

$$m=\frac{1}{n_{P\subseteq g}}\sum _P^j{N}_{j\in P}$$

(32)

where m represents the AD value, N is the cardinality of the paths j, P represents the set of paths within the graph g, and n represents the cardinality of P.

4.

The Average Breadth metric, abbreviated AB, expresses the average cardinality value of a generation in a graph. It is calculated by dividing the sum of the cardinalities of all generations by the number of generations in the graph [58]. The metric is an indicator of the degree to which the ontology has a horizontal modeling of its hierarchies with less taxonomy is-a relations [60]. The AB metric is calculated by Equation (33).

$$m=\frac{1}{n_{L\subseteq g}}\sum _L^j{N}_{j\in L}$$

(33)

where m represents the AB value, N is the cardinality of a generation, j a particular generation, $n_{L\subseteq g}$ is the cardinality of L, and L is the set of all generations within a graph g.

5.

The Average Number of Paths, abbreviated ANP, expresses the relationship between a concept and the root concept within the taxonomy hierarchy of the ontology. An ontology that has a high ANP value contains a large number of taxonomy/inheritance relationships along with a large number of interconnections between the classes within the ontology. If the ANP value is 1, then it signifies that the inheritance hierarchy of the ontology is a tree [61]. The ANP is calculated by Equation (34).

$$m=\frac{{\sum }_{i=1}^n{p}_i}{\left|C\right|}$$

(34)

where m is the ANP value, $p_i$ is the number of paths for a given concept, and $\left|C\right|$ is the total number of concepts.

5.1.3. Qualitative Attributes

The five qualitative attributes used in this study were adopted from a study by Ma et al. [27] where the authors developed an Ontology Usability Scale (OUS) to evaluate ontologies. The OUS comprised 10 questions that a decision-maker can use to evaluate an ontology. Based on the OUS, five criteria were adopted in this study. These five criteria are elaborated on as follows.

1.

Clarity of Purpose (CoP)—This criterion addresses the question of how clear is the purpose of the ontology [27]. The CoP criterion pertains to the semantics and documentation of the ontology.

2.

Quality of Subclass Definition (QoSD)—This criterion addresses the question of how properly the subclasses in the ontology are defined and whether the class hierarchy needs better organization or not [27]. The QoSD criterion pertains to the syntax and structure of the content of the ontology.

3.

Description of Concepts and Relations in Natural Language (DoCRNL)—This criterion describes how well the concepts and the relations of an ontology are described using natural language [27]. The DoCRNL criterion is expressive of the semantics and documentation of the ontology.

4.

Understandability of Conceptualization (UoC)—This criterion expresses how easy it is to comprehend the ontology’s conceptualization [27]. The UoC criterion pertains to the semantics and documentation aspects of an ontology.

5.

Description of Concepts using Attributes (DoCA)—This criterion concerns how well the attributes in the ontology describe its concepts [27]. The DoCA criterion pertains to the information content of the ontology.

5.1.4. Computer and Software Environments

The experiments in this study were carried out using a 64-bit Microsoft^® Windows^® 10 device with 12 GB of RAM and a 1 TB HDD. The device had an Intel^® Core^TM i-5 processor with a speed of 2.30 GHz. The ZPLTS-ELECTRE II algorithm was implemented using the Java 8 programming language together with the JetBrains IntelliJ IDEA Community Edition 2019.1.3 development environment.

5.2. Experimental Results and Discussions

This section presents the results of the application of ZPLTS-ELECTRE II method in the ranking of the ontologies in the dataset.

5.2.1. Quantitative Matrix

Initially, the quantitative matrix, $\mathrm{\Lambda }$, of the ZPLTS-ELECTRE II algorithm was formed with the five complexity metrics defined in Section 5.1.2 for each of the nine ontologies in the dataset as in

Table 2. Quantitative matrix $\mathrm{\Lambda }$ of the 5 complexity metrics for the ontologies in the dataset.

${\mathit{O}}_{\mathit{i}}$	ALC	ARC	AD	AB	ANP
$O_1$	26	447	1.97	53.2	532
$O_2$	7	494	1.98	62.62	250.5
$O_3$	1	228	3.80	5.28	40
$O_4$	14	16	1.60	2.15	7
$O_5$	7	1323	6.82	7.10	156.91
$O_6$	2	320	5.21	6.8	46.75
$O_7$	11	1364	4.76	4.04	181.4
$O_8$	8	181	3.19	10.47	49.75
$O_9$	6	3508	2.37	27.56	454.87

. These metrics were calculated with the OntoMetrics platform [59]. Column $O_i$ represents the ontologies in the dataset, with $1\le i\le 9$.

5.2.2. Qualitative Matrices

The qualitative criteria were evaluated by four decision-makers, and accordingly, four qualitative matrices were formed. These are ${\mathrm{\Gamma }}^b$, where $b=\{1,2,3,4\}$, $m=9$ as there are $nine$ ontology alternatives, and $n-t=10-5=5$ as there are $five$ quantitative criteria. The decision-makers were allowed to select a linguistic term for each criterion for every ontology along with an accompanying linguistic term representing their credibility level for that criterion evaluation. The Linguistic Term Set S in

Table 3. Linguistic Term Set S for evaluating ontology criteria.

Term	${\mathit{s}}_{-2}$	${\mathit{s}}_{-1}$	${\mathit{s}}_0$	${\mathit{s}}_1$	${\mathit{s}}_2$
Evaluation	Very bad	Bad	Average	Good	Very good

was used for the evaluation of criteria, and the Linguistic Term Set $S^{\prime }$ in

Table 4. Linguistic Term Set $S^{\prime }$ for expressing credibility level for evaluation.

Term	${\mathit{s}}_{-2}^{\prime }$	${\mathit{s}}_{-1}^{\prime }$	${\mathit{s}}_0^{\prime }$	${\mathit{s}}_1^{\prime }$	${\mathit{s}}_2^{\prime }$
Evaluation	Not at all sure	Not sure	Moderately sure	Sure	Very sure

was used for expressing levels of credibility.

There are five linguistic terms in S. The worst term is ’Very bad’, and the best term is ’Very good’. The middle term is ’Average’. The term ’Bad’ lies between ’Average’ and ’Very bad’, and the term ’Good’ lies between ’Average’ and ’Very good’.

There are also five linguistic terms in $S^{\prime }$. The worst term is ’Not at all sure’, and the best term is ’Very sure’. The middle term is ’Moderately sure’. The term ’Not sure’ lies between ’Moderately sure’ and ’Not at all sure’, and the term ’Sure’ lies between ’Moderately sure’ and ’Very sure’. The matrices representing the decision-makers’ evaluations; i.e., ${\mathrm{\Gamma }}^1$, ${\mathrm{\Gamma }}^2$, ${\mathrm{\Gamma }}^3$, and ${\mathrm{\Gamma }}^4$ are presented in

Table 5. Qualitative matrix for criteria evaluation by decision-maker 1 (${\mathrm{\Gamma }}^1$).

${\mathit{O}}_{\mathit{i}}$	CoP	QoSD	DoCRNL	UoC	DoCA
$O_1$	$(s_1,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_1,s_0^{\prime })$	$(s_0,s_1^{\prime })$	$(s_1,s_0^{\prime })$
$O_2$	$(s_0,s_0^{\prime })$	$(s_{-1},s_1^{\prime })$	$(s_{-1},s_1^{\prime })$	$(s_0,s_0^{\prime })$	$(s_0,s_1^{\prime })$
$O_3$	$(s_0,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_0,s_0^{\prime })$	$(s_1,s_0^{\prime })$	$(s_0,s_0^{\prime })$
$O_4$	$(s_{-1},s_0^{\prime })$	$(s_0,s_1^{\prime })$	$(s_{-1},s_1^{\prime })$	$(s_0,s_0^{\prime })$	$(s_0,s_1^{\prime })$
$O_5$	$(s_2,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_0,s_0^{\prime })$	$(s_1,s_0^{\prime })$
$O_6$	$(s_0,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_{-1},s_1^{\prime })$	$(s_0,s_1^{\prime })$
$O_7$	$(s_0,s_0^{\prime })$	$(s_0,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_{-1},s_0^{\prime })$	$(s_0,s_0^{\prime })$
$O_8$	$(s_{-1},s_1^{\prime })$	$(s_{-1},s_0^{\prime })$	$(s_{-1},s_0^{\prime })$	$(s_0,s_0^{\prime })$	$(s_0,s_0^{\prime })$
$O_9$	($s_1$, $s_1^{\prime }$)	($s_1$, $s_1^{\prime }$)	($s_1$, $s_1^{\prime }$)	($s_2$, $s_1^{\prime }$)	($s_2$, $s_1^{\prime }$)

,

Table 6. Qualitative matrix for criteria evaluation by decision-maker 2 (${\mathrm{\Gamma }}^2$).

${\mathit{O}}_{\mathit{i}}$	CoP	QoSD	DoCRNL	UoC	DoCA
$O_1$	$(s_1,s_{-1}^{\prime })$	$(s_2,s_0^{\prime })$	$(s_2,s_{-1}^{\prime })$	$(s_1,s_{-1}^{\prime })$	$(s_1,s_0^{\prime })$
$O_2$	$(s_0,s_1^{\prime })$	$(s_1,s_0^{\prime })$	$(s_{-1},s_0^{\prime })$	$(s_{-1},s_0^{\prime })$	$(s_{-1},s_1^{\prime })$
$O_3$	$(s_1,s_{-1}^{\prime })$	$(s_0,s_0^{\prime })$	$(s_0,s_{-1}^{\prime })$	$(s_0,s_{-1}^{\prime })$	$(s_0,s_{-1}^{\prime })$
$O_4$	$(s_0,s_0^{\prime })$	$(s_1,s_0^{\prime })$	$(s_0,s_0^{\prime })$	$(s_{-1},s_0^{\prime })$	$(s_{-1},s_0^{\prime })$
$O_5$	$(s_1,s_{-1}^{\prime })$	$(s_2,s_{-1}^{\prime })$	$(s_2,s_{-1}^{\prime })$	$(s_1,s_0^{\prime })$	$(s_1,s_{-1}^{\prime })$
$O_6$	$(s_1,s_0^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$
$O_7$	$(s_0,s_0^{\prime })$	$(s_1,s_0^{\prime })$	$(s_0,s_1^{\prime })$	$(s_{-1},s_1^{\prime })$	$(s_0,s_1^{\prime })$
$O_8$	$(s_{-1},s_{-1}^{\prime })$	$(s_1,s_0^{\prime })$	$(s_0,s_0^{\prime })$	$(s_{-1},s_{-1}^{\prime })$	$(s_{-1},s_0^{\prime })$
$O_9$	$(s_2,s_1^{\prime })$	$(s_2,s_1^{\prime })$	$(s_2,s_1^{\prime })$	$(s_2,s_0^{\prime })$	$(s_2,s_1^{\prime })$

,

Table 7. Qualitative matrix for criteria evaluation by decision-maker 3 (${\mathrm{\Gamma }}^3$).

${\mathit{O}}_{\mathit{i}}$	CoP	QoSD	DoCRNL	UoC	DoCA
$O_1$	$(s_0,s_1^{\prime })$	$(s_2,s_1^{\prime })$	$(s_1,s_0^{\prime })$	$(s_2,s_1^{\prime })$	$(s_1,s_1^{\prime })$
$O_2$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_{-1},s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_0,s_0^{\prime })$
$O_3$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$
$O_4$	$(s_{-1},s_1^{\prime })$	$(s_{-1},s_1^{\prime })$	$(s_{-1},s_1^{\prime })$	$(s_1,s_0^{\prime })$	$(s_0,s_0^{\prime })$
$O_5$	$(s_0,s_1^{\prime })$	$(s_2,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_2,s_1^{\prime })$	$(s_1,s_1^{\prime })$
$O_6$	$(s_{-1},s_0^{\prime })$	$(s_1,s_0^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_0^{\prime })$	$(s_0,s_0^{\prime })$
$O_7$	$(s_1,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$
$O_8$	$(s_0,s_0^{\prime })$	$(s_{-1},s_1^{\prime })$	$(s_{-1},s_0^{\prime })$	$(s_1,s_0^{\prime })$	$(s_0,s_0^{\prime })$
$O_9$	$(s_2,s_1^{\prime })$	$(s_2,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_2,s_0^{\prime })$	$(s_2,s_0^{\prime })$

and

Table 8. Qualitative matrix for criteria evaluation by decision-maker 4 (${\mathrm{\Gamma }}^4$).

${\mathit{O}}_{\mathit{i}}$	CoP	QoSD	DoCRNL	UoC	DoCA
$O_1$	$(s_0,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_1,s_0^{\prime })$	$(s_1,s_1^{\prime })$	$(s_0,s_1^{\prime })$
$O_2$	$(s_1,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_{-1},s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_0,s_0^{\prime })$
$O_3$	$(s_0,s_0^{\prime })$	$(s_0,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_1,s_0^{\prime })$
$O_4$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_1,s_0^{\prime })$	$(s_0,s_0^{\prime })$
$O_5$	$(s_0,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_1,s_1^{\prime })$
$O_6$	$(s_{-1},s_0^{\prime })$	$(s_1,s_0^{\prime })$	$(s_1,s_0^{\prime })$	$(s_0,s_0^{\prime })$	$(s_0,s_0^{\prime })$
$O_7$	$(s_1,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_0,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_0,s_1^{\prime })$
$O_8$	$(s_{-1},s_0^{\prime })$	$(s_{-1},s_1^{\prime })$	$(s_{-1},s_0^{\prime })$	$(s_1,s_0^{\prime })$	$(s_0,s_0^{\prime })$
$O_9$	$(s_1,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_1,s_1^{\prime })$	$(s_1,s_0^{\prime })$	$(s_1,s_0^{\prime })$

.

5.2.3. Decision Matrix

After obtaining the quantitative matrix, $\mathrm{\Lambda }$ (Table 2), and the qualitative matrices, ${\mathrm{\Gamma }}^1$ to ${\mathrm{\Gamma }}^4$ (Table 5, Table 6, Table 7 and Table 8), the decision matrix, E, is built. Firstly, the four qualitative matrices are combined by adding the probabilities of like terms for each criterion to form a single matrix, $\mathrm{\Gamma }$, in

Table 9. The $\mathrm{\Gamma }$ matrix before normalization.

${\mathit{O}}_{\mathit{i}}$	CoP	QoSD	DoCRNL	UoC	DoCA
$O_1$	$\{s_0(0.5),s_1(0.5)\},\{s_{-1}^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_1(0.5),s_2(0.5)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_1(0.75),s_2(0.25)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.75)\}$	$\{s_0(0.25),s_1(0.25),s_2(0.5)\},\{s_{-1}^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_0(0.25),s_1(0.75)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5)\}$
$O_2$	$\{s_0(0.75),s_1(0.25)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_{-1}(1)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5)\}$	$\{s_{-1}(0.25),s_0(0.75)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5)\}$
$O_3$	$\{s_0(0.75),s_1(0.25)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.25),s_1^{\prime }(0.5)\}$	$\{s_0(0.75),s_1(0.25)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_0(0.75),s_1(0.25)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.25),s_1^{\prime }(0.5)\}$	$\{s_0(0.75),s_1(0.25)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.25),s_1^{\prime }(0.5)\}$	$\{s_0(0.75),s_1(0.25)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.5),s_1^{\prime }(0.25)\}$
$O_4$	$\{s_{-1}(0.5),s_0(0.5)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5)\}$	$\{s_{-1}(0.25),s_0(0.5),s_1(0.25)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_{-1}(0.5),s_0(0.5)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\},\{s_0^{\prime }(1)\}$	$\{s_{-1}(0.25),s_0(0.75)\},\{s_0^{\prime }(0.75),s_1^{\prime }(0.25)\}$
$O_5$	$\{s_0(0.5),s_1(0.25),s_2(0.25)\},\{s_{-1}^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_1(0.5),s_2(0.5)\},\{s_{-1}^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_1(0.75),s_2(0.25)\},\{s_{-1}^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_0(0.25),s_1(0.5),s_2(0.25)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5)\}$	$\{s_1(1)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.25),s_1^{\prime }(0.5)\}$
$O_6$	$\{s_{-1}(0.5),s_0(0.25),s_1(0.25)\},\{s_0^{\prime }(0.75),s_1^{\prime }(0.25)\}$	$\{s_0(0.25),s_1(0.75)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5)\}$	$\{s_0(0.75),s_1(0.25)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_{-1}(0.25),s_0(0.75)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5)\}$	$\{s_0(1)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5)\}$
$O_7$	$\{s_0(0.5),s_1(0.5)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5)\}$	$\{s_0(0.75),s_1(0.25)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_0(0.75),s_1(0.25)\},\{s_1^{\prime }(1)\}$	$\{s_{-1}(0.5),s_0(0.25),s_1(0.25)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75)\}$	$\{s_0(1)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75)\}$
$O_8$	$\{s_{-1}(0.75),s_0(0.25)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.5),s_1^{\prime }(0.25)\}$	$\{s_{-1}(0.75),s_1(0.25)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5)\}$	$\{s_{-1}(0.75),s_0(0.25)\},\{s_0^{\prime }(1)\}$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.75)\}$	$\{s_{-1}(0.25),s_0(0.75)\},\{s_0^{\prime }(1)\}$
$O_9$	$\{s_1(0.25),s_2(0.75)\},\{s_1^{\prime }(1)\}$	$\{s_1(0.25),s_2(0.75)\},\{s_1^{\prime }(1)\}$	$\{s_1(0.75),s_2(0.25)\},\{s_1^{\prime }(1)\}$	$\{s_1(0.25),s_2(0.75)\},\{s_0^{\prime }(0.75),s_1^{\prime }(0.25)\}$	$\{s_2(1)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5)\}$

. The matrix $\mathrm{\Gamma }$ is then normalized for each criterion, as shown in

Table 10. Normalized qualitative matrix for criterion CoP.

${\mathit{O}}_{\mathit{i}}$	Criterion CoP
$O_1$	$\{s_0(0.5),s_1(0.5),s_1(0)\},\{s_{-1}^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_2$	$\{s_0(0.75),s_1(0.25),s_1(0)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_3$	$\{s_0(0.75),s_1(0.25),s_1(0)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.25),s_1^{\prime }(0.5)\}$
$O_4$	$\{s_{-1}(0.5),s_0(0.5),s_0(0)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5),s_1^{\prime }(0)\}$
$O_5$	$\{s_0(0.5),s_1(0.25),s_2(0.25)\},\{s_{-1}^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_6$	$\{s_{-1}(0.5),s_0(0.25),s_1(0.25)\},\{s_0^{\prime }(0.75),s_1^{\prime }(0.25),s_1^{\prime }(0)\}$
$O_7$	$\{s_0(0.5),s_1(0.5),s_1(0)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5),s_1^{\prime }(0)\}$
$O_8$	$\{s_{-1}(0.75),s_0(0.25),s_0(0)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.5),s_1^{\prime }(0.25)\}$
$O_9$	$\{s_1(0.25),s_2(0.75),s_2(0)\},\{s_1^{\prime }(1),s_1^{\prime }(0),s_1^{\prime }(0)\}$

,

Table 11. Normalized qualitative matrix for criterion QoSD.

${\mathit{O}}_{\mathit{i}}$	Criterion QoSD
$O_1$	$\{s_1(0.5),s_2(0.5),s_2(0)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_2$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_3$	$\{s_0(0.75),s_1(0.25),s_1(0)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_4$	$\{s_{-1}(0.25),s_0(0.5),s_1(0.25)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_5$	$\{s_1(0.5),s_2(0.5),s_2(0)\},\{s_{-1}^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_6$	$\{s_0(0.25),s_1(0.75),s_1(0)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5),s_1^{\prime }(0)\}$
$O_7$	$\{s_0(0.75),s_1(0.25),s_1(0)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_8$	$\{s_{-1}(0.75),s_1(0.25),s_1(0)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5),s_1^{\prime }(0)\}$
$O_9$	$\{s_1(0.25),s_2(0.75),s_2(0)\},\{s_1^{\prime }(1),s_1^{\prime }(0),s_1^{\prime }(0)\}$

,

Table 12. Normalized qualitative matrix for criterion DoCRNL.

${\mathit{O}}_{\mathit{i}}$	Criterion DoCRNL
$O_1$	$\{s_1(0.75),s_2(0.25),s_2(0)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.75),s_0(0)\}$
$O_2$	$\{s_{-1}(1),s_{-1}(0),s_{-1}(0)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_3$	$\{s_0(0.75),s_1(0.25),s_1(0)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.25),s_1^{\prime }(0.5)\}$
$O_4$	$\{s_{-1}(0.5),s_0(0.5),s_0(0)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_5$	$\{s_1(0.75),s_2(0.25),s_2(0)\},\{s_{-1}^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_6$	$\{s_0(0.75),s_1(0.25),s_1(0)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_7$	$\{s_0(0.75),s_1(0.25),s_1(0)\},\{s_1^{\prime }(1),s_1^{\prime }(0),s_1^{\prime }(0)\}$
$O_8$	$\{s_{-1}(0.75),s_0(0.25),s_0(0)\},\{s_0^{\prime }(1),s_0^{\prime }(0),s_0^{\prime }(0)\}$
$O_9$	$\{s_1(0.75),s_2(0.25),s_2(0)\},\{s_1^{\prime }(1),s_1^{\prime }(0),s_1^{\prime }(0)\}$

,

Table 13. Normalized qualitative matrix for criterion UoC.

${\mathit{O}}_{\mathit{i}}$	Criterion UoC
$O_1$	$\{s_0(0.25),s_1(0.25),s_2(0.5)\},\{s_{-1}^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_2$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5),s_1^{\prime }(0)\}$
$O_3$	$\{s_0(0.75),s_1(0.25),s_1(0)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.25),s_1^{\prime }(0.5)\}$
$O_4$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\},\{s_0^{\prime }(1),s_0^{\prime }(0),s_0^{\prime }(0)\}$
$O_5$	$\{s_0(0.25),s_1(0.5),s_2(0.25)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5),s_1^{\prime }(0)\}$
$O_6$	$\{s_{-1}(0.25),s_0(0.75),s_0(0)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5),s_1^{\prime }(0)\}$
$O_7$	$\{s_{-1}(0.5),s_0(0.25),s_1(0.25)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_8$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.75),s_0^{\prime }(0)\}$
$O_9$	$\{s_1(0.25),s_2(0.75),s_2(0)\},\{s_0^{\prime }(0.75),s_1^{\prime }(0.25),s_1^{\prime }(0)\}$

and

Table 14. Normalized qualitative matrix for criterion DoCA.

${\mathit{O}}_{\mathit{i}}$	Criterion DoCA
$O_1$	$\{s_0(0.25),s_1(0.75),s_1(0)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5),s_1^{\prime }(0)\}$
$O_2$	$\{s_{-1}(0.25),s_0(0.75),s_0(0)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5),s_1^{\prime }(0)\}$
$O_3$	$\{s_0(0.75),s_1(0.25),s_1(0)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.5),s_1^{\prime }(0.25)\}$
$O_4$	$\{s_{-1}(0.25),s_0(0.75),s_0(0)\},\{s_0^{\prime }(0.75),s_1^{\prime }(0.25),s_1^{\prime }(0)\}$
$O_5$	$\{s_1(1),s_1(0),s_1(0)\},\{s_{-1}^{\prime }(0.25),s_0^{\prime }(0.25),s_1^{\prime }(0.5)\}$
$O_6$	$\{s_0(1),s_0(0),s_0(0)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5),s_1^{\prime }(0)\}$
$O_7$	$\{s_0(1),s_0(0),s_0(0)\},\{s_0^{\prime }(0.25),s_1^{\prime }(0.75),s_1^{\prime }(0)\}$
$O_8$	$\{s_{-1}(0.25),s_0(0.75),s_0(0)\},\{s_0^{\prime }(1),s_0^{\prime }(0),s_0^{\prime }(0)\}$
$O_9$	$\{s_2(1),s_2(0),s_2(0)\},\{s_0^{\prime }(0.5),s_1^{\prime }(0.5),s_1^{\prime }(0)\}$

. Thereafter, the matrix $\mathrm{\Lambda }$ is concatenated with $\mathrm{\Gamma }$ to form the decision matrix E with dimensions $9\times 10$, as there are $nine$ ontologies and 10 criteria.

5.2.4. Weights and Thresholds

The criteria importance weightings were obtained by applying the mean weighting method; that is, all criteria were given equal importance. The weights and the discordance thresholds are shown in

Table 15. Criteria importance weights and discordance thresholds.

Index j	Criterion	${\mathit{\omega }}_{\mathit{j}}$	${\mathit{\beta }}_{\mathit{j}}^{-}$	${\mathit{\beta }}_{\mathit{j}}^{+}$
1	ALC	0.1	3	7
2	ARC	0.1	250	700
3	AD	0.1	0.75	3
4	AB	0.1	4.5	15
5	ANP	0.1	35	150
6	CoP	0.1	0.3	1
7	QoSD	0.1	0.5	1.1
8	DoCRNL	0.1	0.7	1.2
9	UoC	0.1	0.3	0.9
10	DoCA	0.1	0.5	1.1

. The concordance thresholds were set as ${\alpha }_1=0.55$, ${\alpha }_2=0.70$ and ${\alpha }_3=0.85$.

5.2.5. Comparative Sets

The comparative sets, ${\mathcal{B}}^{-}(x,y)$, ${\mathcal{B}}^0(x,y)$, and ${\mathcal{B}}^{+}(x,y)$ for comparing two alternatives against each other were created for all alternatives. To determine the comparative relationships, the scores and deviation degree values are required. These were calculated by applying Equations (4) and (5). The scores and deviation degree values for criteria 6 to 10, that is, the qualitative criteria, were calculated and displayed in the heatmaps in Figure 1. As an illustration, when comparing alternative 1 to alternative 2, that is, the Mental State Assessment Ontology and the APA Neuro Cluster Ontology, the comparative sets are formed as follows.

$${\mathcal{B}}^{+}(1,2)=\{1,5,6,7,8,9,10\}$$

Based on the set ${\mathcal{B}}^{+}(x,y)$, it can be observed that alternative 1 outperforms alternative 2 for criteria 1, 5, 6, 7, 8, 9, and 10.

$${\mathcal{B}}^0(1,2)=\{\}$$

Since the set ${\mathcal{B}}^0(x,y)$ is empty, it can be observed that there exists no criteria for which the performance of alternative 1 is equivalent to the performance of alternative 2.

$${\mathcal{B}}^{-}(1,2)=\{2,3,4\}$$

Based on the set ${\mathcal{B}}^{-}(x,y)$, it can be observed that alternative 1 is outperformed by alternative 2 for criteria 2, 3, and 4.

Figure 1. Score and deviation degree values for all alternatives in light of criteria 6 to 10. (a) Score values. (b) Deviation degree values.

Figure 1. Score and deviation degree values for all alternatives in light of criteria 6 to 10. (a) Score values. (b) Deviation degree values.

5.2.6. Concordance Relations

The concordance values between every alternative pair, $C(x,y)$, were calculated by applying Equation (16). The resulting concordance values are shown in the concordance matrix in

Table 16. Concordance matrix.

	${\mathit{O}}_1$	${\mathit{O}}_2$	${\mathit{O}}_3$	${\mathit{O}}_4$	${\mathit{O}}_5$	${\mathit{O}}_6$	${\mathit{O}}_7$	${\mathit{O}}_8$	${\mathit{O}}_9$
${\mathit{O}}_{\mathbf{1}}$	0	0.70	0.90	1.00	0.60	0.80	0.70	0.90	0.30
${\mathit{O}}_{\mathbf{2}}$	0.30	0	0.60	0.80	0.30	0.60	0.30	0.80	0.20
${\mathit{O}}_{\mathbf{3}}$	0.10	0.40	0	0.90	0.00	0.20	0.40	0.70	0.10
${\mathit{O}}_{\mathbf{4}}$	0.00	0.20	0.10	0	0.10	0.20	0.10	0.60	0.10
${\mathit{O}}_{\mathbf{5}}$	0.40	0.80	1.00	0.90	0	1.00	0.70	0.80	0.20
${\mathit{O}}_{\mathbf{6}}$	0.20	0.40	0.80	0.80	0.00	0	0.40	0.70	0.10
${\mathit{O}}_{\mathbf{7}}$	0.30	0.70	0.70	0.90	0.30	0.60	0	0.90	0.20
${\mathit{O}}_{\mathbf{8}}$	0.10	0.20	0.30	0.40	0.20	0.30	0.10	0	0.20
${\mathit{O}}_{\mathbf{9}}$	0.70	0.80	0.90	0.90	0.80	0.90	0.80	0.80	0

, where $O_i$ represents the $i^{th}$ ontology alternative, $1\le i\le 9$.

5.2.7. Discordance Relations

In order to determine the discordance relations between all criteria pairs, the differences in their performance for all 10 criteria need to be calculated. When considering quantitative criteria, that is, criteria 1 to 5, the difference between the performance of alternative x and alternative y at criterion j is the difference between $x_j$ and $y_j$, that is, $x_j-y_j$. The calculated differences between all alternatives at each criterion are presented in the heatmaps in Figure 2.

When considering qualitative criteria, that is, criteria 6 to 10, the difference between the performance of alternative x and alternative y at criterion j is the distance between $x_j$ and $y_j$, as given by Equation (6). The calculated distances between all alternatives at each criterion is presented in the heatmaps in Figure 3.

After analyzing the differences and distances between the criteria for all alternatives, the discordance relations are determined. For comparing each alternative pair, x and y, the comparison between their criteria performances are partitioned into one of three discordance sets: high discordance ${\mathcal{Q}}^{+}(x,y)$, medium discordance ${\mathcal{Q}}^0(x,y)$, or low discordance ${\mathcal{Q}}^{-}(x,y)$. To illustrate this, when considering the comparison between alternative 1 and 3, the comparative sets are formed as follows.

$${\mathcal{Q}}^{+}(1,3)=\{\}$$

The set ${\mathcal{Q}}^{+}(1,3)$ is an empty set, which implies that there is no criteria for which the comparison of alternatives 1 and 3 yields high discordance.

$${\mathcal{Q}}^0(1,3)=\{3\}$$

Since the set ${\mathcal{Q}}^0(1,3)$ has one element, that is, criterion 3, this means that there is medium discordance when comparing alternative 1 and 3 against criterion 3.

$${\mathcal{Q}}^{-}(1,3)=\{1,2,4,5,6,7,8,9,10\}$$

By observing the set ${\mathcal{Q}}^{-}(1,3)$, it can be seen that comparing alternative 1 with 3 yields the set comprising nine out of 10 criteria that have low discordance levels.

5.2.8. Strong and Weak Outranking Graphs

After determining the concordance and discordance relations, the strong and weak outranking relations were determined by applying Equations (24) to (27). The outranking relations obtained are displayed in

Table 17. Strong and weak outranking relationships.

	${\mathit{O}}_1$	${\mathit{O}}_2$	${\mathit{O}}_3$	${\mathit{O}}_4$	${\mathit{O}}_5$	${\mathit{O}}_6$	${\mathit{O}}_7$	${\mathit{O}}_8$	${\mathit{O}}_9$
${\mathit{O}}_{\mathbf{1}}$			$S^F$	$S^F$				$S^F$
${\mathit{O}}_{\mathbf{2}}$			$S^f$					$S^F$
${\mathit{O}}_{\mathbf{3}}$
${\mathit{O}}_{\mathbf{4}}$								$S^f$
${\mathit{O}}_{\mathbf{5}}$			$S^F$			$S^F$	$S^F$	$S^F$
${\mathit{O}}_{\mathbf{6}}$								$S^F$
${\mathit{O}}_{\mathbf{7}}$				$S^F$		$S^f$		$S^F$
${\mathit{O}}_{\mathbf{8}}$
${\mathit{O}}_{\mathbf{9}}$			$S^F$			$S^F$	$S^F$	$S^F$

, where $S^F$ represents a strong outranking relation, and $S^f$ represents a weak outranking relation.

Based on Table 17, the strong and weak outranking graphs can be constructed. The constructed graphs are shown in Figure 4. Each node, $O_i$, represents the $i^{th}$ ontology alternative, where $1\le i\le 9$. A directed edge from one node to another signifies that the node from which the edge begins outranks the node to which the edge points.

5.2.9. Exploitation of Outranking Relations

After constructing the outranking graphs, the outranking relations are exploited by applying the forward and reverse procedures, as shown in Section 4.9. Firstly, the forward procedure is performed as follows. In the first iteration, the strong and weak outranking graphs, $G_1^F$ and $G_1^f$, are observed, and the sets $N_1^F$ and $N_1^f$ are formed. The elements in these sets represent the non-dominant alternatives in $G_1^F$ and $G_1^f$. $N_1^F$ is formed as:

$$N_1^F= \{1, 2, 5, 9\}$$

and $N_1^f$ is formed as:

$$N_1^f= \{1, 2, 4, 5, 7, 9\}$$

Thereafter, the intersection between the non-dominant sets is formed as $N_1$:

$$N_1= \{1, 2, 5, 9\}$$

After determining the intersection $N_1$, the alternatives in $N_1$ are assigned a rank, that is, ${\psi }_1(1)=1$, ${\psi }_1(2)=1$, ${\psi }_1(5)=1$, and ${\psi }_1(9)=1$. All alternatives assigned a rank are then removed from the graphs along with their associated edges. The new graphs are then constructed, as shown in Figure 5.

The non-dominant sets and their intersection are then formed for the second iteration as follows:

$$N_2^F=\{3, 6, 7\}$$

and $N_2^f$ is formed as:

$$N_2^f=\{3, 4, 7\}$$

Thereafter, the intersection between the non-dominant sets are formed as $N_2$:

$$N_2=\{3, 7\}$$

The alternatives in the intersection are assigned the rank position of 2, that is, ${\psi }_1(3)=2$ and ${\psi }_1(7)=2$. The third iteration then proceeds with the new graphs in Figure 6.

The non-dominant sets and their intersection are then formed for the third iteration as follows:

$$N_3^F=\{4, 6\}$$

and $N_3^f$ is formed as:

$$N_3^f=\{4, 6\}$$

Thereafter, the intersection between the non-dominant sets are formed as $N_3$:

$$N_3=\{4, 6\}$$

The alternatives in the intersection are assigned the rank position of 3, that is, ${\psi }_1(4)=3$ and ${\psi }_1(6)=3$. Since there are still nodes in the graph, the fourth iteration then proceeds with the new graphs in Figure 7.

Since both the graphs in Figure 7 have only one node which is not dominated by any other nodes, their intersection will be equal, and hence, the alternative $O_8$ is assigned the 4th rank position, that is, ${\psi }_1(8)=4$. All alternatives in the graphs have been ranked, and the forward procedure is completed.

After performing the forward procedure, the reverse procedure is performed. The first step in the reverse procedure is to construct the mirror image of the strong and weak outranking graphs by reversing the direction of the edges. Thereafter, the same steps follow, as performed in the forward procedure. The resulting ranking is then transformed by applying Equation (28). Finally, to generate the final ranking of the alternatives, the forward ranking and the transformed reverse ranking are combined by applying Equation (29). The ranking obtained from the reverse procedure, ${\psi }_2(O_i)$, the transformed ranking, ${\psi }_3(O_i)$, and the final ranking, $\overline{\psi }(O_i)$ are shown in

Table 18. Final ranking results of the ontology alternatives.

Alternative ${\mathit{O}}_{\mathit{i}}$	${\mathit{\psi }}_1({\mathit{O}}_{\mathit{i}})$	${\mathit{\psi }}_2({\mathit{O}}_{\mathit{i}})$	${\mathit{\psi }}_3({\mathit{O}}_{\mathit{i}})$	$\overline{\mathit{\psi }}({\mathit{O}}_{\mathit{i}})$
$O_1$	1	3	2	1.5
$O_2$	1	2	3	2
$O_3$	2	1	4	3
$O_4$	3	2	3	3
$O_5$	1	4	1	1
$O_6$	3	2	3	3
$O_7$	2	3	2	2
$O_8$	4	1	4	4
$O_9$	1	4	1	1

.

The final ranking for the nine ontologies is determined as:

$$O_5, O_9 \succ O_1 \succ O_2, O_7 \succ O_3, O_4, O_6 \succ O_8$$

Accordingly, ontologies 5 and 9 are the best, and the worst ontology is ontology 8.

5.3. Comparative Analysis of ZPLTS-ELECTRE II

The performance of the ZPLTS-ELECTRE II method is compared with that of the traditional ELECTRE II [29] and PLTS ELECTRE II [62] methods for the same task of evaluating and ranking mental health ontologies.

5.3.1. Comparison with Traditional ELECTRE II

Since the traditional ELECTRE II cannot model linguistic criteria, only the quantitative criteria were used, that is, ALC, ARC, AD, AB, and ANP. Firstly, the alternatives and criteria form a decision matrix, which is the same as the matrix in Table 2. The decision matrix (Table 2) was then normalized to form the matrix in

Table 19. Decision matrix for traditional ELECTRE II.

Oi	ALC	ARC	AD	AB	ANP
$O_1$	1.00	0.13	0.29	0.85	1.00
$O_2$	0.27	0.14	0.29	1.00	0.47
$O_3$	0.04	0.06	0.56	0.08	0.08
$O_4$	0.54	0.00	0.24	0.03	0.01
$O_5$	0.27	0.38	1.00	0.11	0.29
$O_6$	0.08	0.09	0.76	0.11	0.09
$O_7$	0.42	0.39	0.70	0.06	0.34
$O_8$	0.31	0.05	0.47	0.17	0.09
$O_9$	0.23	1.00	0.35	0.44	0.86

.

The criteria were assigned equal weights, as with the application of ZPLTS-ELECTRE II. Since there were five criteria, each criterion received a weight of $0.20$. The concordance thresholds were set as $c_1=0.85$, $c_2=0.70$, and $c_3=0.55$. However, since the traditional ELECTRE II does not allow separate discordance thresholds to be specified for each criterion as in ZPLTS-ELECTRE II, the discordance thresholds, $d_1$ and $d_2$, were set as $d_1=0.40$ and $d_2=0.25$.

The next step was to calculate the concordance values between all alternative pairs, as shown in

Table 20. Concordance matrix for traditional ELECTRE II.

	${\mathit{O}}_1$	${\mathit{O}}_2$	${\mathit{O}}_3$	${\mathit{O}}_4$	${\mathit{O}}_5$	${\mathit{O}}_6$	${\mathit{O}}_7$	${\mathit{O}}_8$	${\mathit{O}}_9$
${\mathit{O}}_{\mathbf{1}}$	0	0.6	0.8	1.0	0.6	0.8	0.6	0.8	0.6
${\mathit{O}}_{\mathbf{2}}$	0.6	0	0.8	0.8	0.6	0.8	0.4	0.6	0.4
${\mathit{O}}_{\mathbf{3}}$	0.2	0.2	0	0.8	0.0	0.0	0.2	0.4	0.2
${\mathit{O}}_{\mathbf{4}}$	0.0	0.2	0.2	0	0.2	0.2	0.2	0.2	0.2
${\mathit{O}}_{\mathbf{5}}$	0.4	0.6	1.0	0.8	0	1.0	0.4	0.6	0.4
${\mathit{O}}_{\mathbf{6}}$	0.2	0.2	1.0	0.8	0.2	0	0.4	0.6	0.2
${\mathit{O}}_{\mathbf{7}}$	0.4	0.6	0.8	0.8	0.6	0.6	0	0.8	0.4
${\mathit{O}}_{\mathbf{8}}$	0.2	0.4	0.6	0.8	0.4	0.6	0.2	0	0.4
${\mathit{O}}_{\mathbf{9}}$	0.4	0.6	0.8	0.8	0.6	0.8	0.6	0.6	0

.

Thereafter, the discordance values were determined for every alternative pair, as shown in

Table 21. Discordance matrix for traditional ELECTRE II.

	${\mathit{O}}_1$	${\mathit{O}}_2$	${\mathit{O}}_3$	${\mathit{O}}_4$	${\mathit{O}}_5$	${\mathit{O}}_6$	${\mathit{O}}_7$	${\mathit{O}}_8$	${\mathit{O}}_9$
${\mathit{O}}_{\mathbf{1}}$	0	0.15	0.27	0.00	0.71	0.47	0.41	0.18	0.87
${\mathit{O}}_{\mathbf{2}}$	0.73	0	0.27	0.27	0.71	0.47	0.41	0.18	0.86
${\mathit{O}}_{\mathbf{3}}$	0.96	0.92	0	0.50	0.43	0.19	0.38	0.27	0.94
${\mathit{O}}_{\mathbf{4}}$	0.99	0.97	0.32	0	0.76	0.52	0.45	0.22	1.00
${\mathit{O}}_{\mathbf{5}}$	0.74	0.89	0.00	0.27	0	0.00	0.14	0.06	0.62
${\mathit{O}}_{\mathbf{6}}$	0.92	0.89	0.00	0.46	0.29	0	0.33	0.22	0.91
${\mathit{O}}_{\mathbf{7}}$	0.79	0.94	0.02	0.12	0.30	0.06	0	0.11	0.61
${\mathit{O}}_{\mathbf{8}}$	0.91	0.83	0.09	0.23	0.53	0.29	0.34	0	0.95
${\mathit{O}}_{\mathbf{9}}$	0.77	0.56	0.21	0.31	0.65	0.41	0.35	0.12	0

Thereafter, the strong and weak outranking graphs were constructed by considering the concordance and discordance matrices. The strong and weak outranking relations are shown in

Table 22. Strong and weak outranking relations for traditional ELECTRE II.

	${\mathit{O}}_1$	${\mathit{O}}_2$	${\mathit{O}}_3$	${\mathit{O}}_4$	${\mathit{O}}_5$	${\mathit{O}}_6$	${\mathit{O}}_7$	${\mathit{O}}_8$	${\mathit{O}}_9$
${\mathit{O}}_{\mathbf{1}}$		$S^f$	$S^f$	$S^F$				$S^F$
${\mathit{O}}_{\mathbf{2}}$			$S^f$	$S^f$				$S^f$
${\mathit{O}}_{\mathbf{3}}$
${\mathit{O}}_{\mathbf{4}}$
${\mathit{O}}_{\mathbf{5}}$			$S^F$	$S^f$		$S^F$		$S^f$
${\mathit{O}}_{\mathbf{6}}$			$S^F$					$S^f$
${\mathit{O}}_{\mathbf{7}}$			$S^F$	$S^F$	$S^f$	$S^f$		$S^F$
${\mathit{O}}_{\mathbf{8}}$			$S^f$	$S^F$		$S^f$
${\mathit{O}}_{\mathbf{9}}$			$S^F$	$S^f$			$S^f$	$S^f$

.

This is followed by the ontologies being ranked using the forward and reverse ranking procedures, and finally, they were combined [29]. The final ranking was as follows:

$$O_9 \succ O_1 \succ O_7 \succ O_2 \succ O_5 \succ O_6, O_8 \succ O_3, O_4$$

It can be observed that there are some differences in the ranking by ZPLTS-ELECTRE II with that of the traditional ELECTRE II, which is due to the fact that the traditional ELECTRE II does not model the qualitative linguistic criteria and the decision-makers credibility levels, and it only allows the definition of two discordance thresholds. Therefore, the traditional ELECTRE II method offers limited modeling capabilities in decision-making problems compared to the proposed ZPLTS-ELECTRE II method.

5.3.2. Comparison with PLTS ELECTRE II

The performance of the ZPLTS-ELECTRE II method was also compared with the PLTS ELECTRE II method [62] for the task of evaluating and ranking mental health ontologies. In the case of PLTS ELECTRE II, only PLTS data can be modeled; therefore, the decision matrix only comprises the qualitative linguistic criteria as in

Table 23. Decision matrix for PLTS ELECTRE II.

${\mathit{O}}_{\mathit{i}}$	CoP	QoSD	DoCRNL	UoC	DoCA
$O_1$	$\{s_0(0.5),s_1(0.5),s_1(0)\}$	$\{s_1(0.5),s_2(0.5),s_2(0)\}$	$\{s_1(0.75),s_2(0.25),s_2(0)\}$	$\{s_0(0.25),s_1(0.25),$ s2(0.5)}	$\{s_0(0.25),s_1(0.75),s_1(0)\}$
$O_2$	$\{s_0(0.75),s_1(0.25),s_1(0)\}$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\}$	$\{s_{-1}(1),s_{-1}(0),s_{-1}(0)\}$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\}$	$\{s_{-1}(0.25),s_0(0.75),s_0(0)\}$
$O_3$	$\{s_0(0.75),s_1(0.25),s_1(0)\}$	$\{s_0(0.75),s_1(0.25),s_1(0)\}$	$\{s_0(0.75),s_1(0.25),s_1(0)\}$	$\{s_0(0.75),s_1(0.25),s_1(0)\}$	$\{s_0(0.75),s_1(0.25),s_1(0)\}$
$O_4$	$\{s_{-1}(0.5),s_0(0.5),s_0(0)\}$	$\{s_{-1}(0.25),s_0(0.5),s_1(0.25)\}$	$\{s_{-1}(0.5),s_0(0.5),s_0(0)\}$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\}$	$\{s_{-1}(0.25),s_0(0.75),s_0(0)\}$
$O_5$	$\{s_0(0.5),s_1(0.25),s_2(0.25)\}$	$\{s_1(0.5),s_2(0.5),s_2(0)\}$	$\{s_1(0.75),s_2(0.25),s_2(0)\}$	$\{s_0(0.25),s_1(0.5),s_2(0.25)\}$	$\{s_1(1),s_1(0),s_1(0)\}$
$O_6$	$\{s_{-1}(0.5),s_0(0.25),s_1(0.25)\}$	$\{s_0(0.25),s_1(0.75),s_1(0)\}$	$\{s_0(0.75),s_1(0.25),s_1(0)\}$	$\{s_{-1}(0.25),s_0(0.75),s_0(0)\}$	$\{s_0(1),s_0(0),s_0(0)\}$
$O_7$	$\{s_0(0.5),s_1(0.5),s_1(0)\}$	$\{s_0(0.75),s_1(0.25),s_1(0)\}$	$\{s_0(0.75),s_1(0.25),s_1(0)\}$	$\{s_{-1}(0.5),s_0(0.25),s_1(0.25)\}$	$\{s_0(1),s_0(0),s_0(0)\}$
$O_8$	$\{s_{-1}(0.75),s_0(0.25),s_0(0)\}$	$\{s_{-1}(0.75),s_1(0.25),s_1(0)\}$	$\{s_{-1}(0.75),s_0(0.25),s_0(0)\}$	$\{s_{-1}(0.25),s_0(0.25),s_1(0.5)\}$	$\{s_{-1}(0.25),s_0(0.75),s_0(0)\}$
$O_9$	$\{s_1(0.25),s_2(0.75),s_2(0)\}$	$\{s_1(0.25),s_2(0.75),s_2(0)\}$	$\{s_1(0.75),s_2(0.25),s_2(0)\}$	$\{s_1(0.25),s_2(0.75),s_2(0)\}$	$\{s_2(1),s_2(0),s_2(0)\}$

. Furthermore, the PLTS ELECTRE II method cannot model credibility values, and therefore, the credibility aspect was omitted.

The criteria importance weights were set as $0.2$ for all $five$ criteria. Since the PLTS ELECTRE II makes use of strong, medium, and weak concordances and discordances, as well as indifference sets, the corresponding weights are also required. The strong, medium, and weak concordance weights were set as ${\omega }_C=1$, ${\omega }_C^{\prime }=0.9$, and ${\omega }_C^{″}=0.8$, respectively. The strong, medium, and weak discordance weights were set as ${\omega }_D=1$, ${\omega }_D^{\prime }=0.9$, and ${\omega }_D^{″}=0.8$, respectively. The indifference weight was set as ${\omega }_J^{=}=0.7$.

The score values for all ontologies at each criterion are shown in

Table 24. Score values for PLTS ELECTRE II.

	CoP	QoSD	DoCRNL	UoC	DoCA
${\mathit{O}}_{\mathbf{1}}$	0.50	1.50	1.25	1.25	0.75
${\mathit{O}}_{\mathbf{2}}$	0.25	0.25	−1.00	0.25	-0.25
${\mathit{O}}_{\mathbf{3}}$	0.25	0.25	0.25	0.25	0.25
${\mathit{O}}_{\mathbf{4}}$	−0.50	0.00	−0.50	0.25	−0.25
${\mathit{O}}_{\mathbf{5}}$	0.75	1.50	1.25	1.00	1.00
${\mathit{O}}_{\mathbf{6}}$	−0.25	0.75	0.25	−0.25	0.00
${\mathit{O}}_{\mathbf{7}}$	0.50	0.25	0.25	−0.25	0.00
${\mathit{O}}_{\mathbf{8}}$	−0.75	-0.50	−0.75	0.25	−0.25
${\mathit{O}}_{\mathbf{9}}$	1.75	1.75	1.25	1.75	2.00

, and the deviation values for all ontologies at each criterion are shown in

Table 25. Deviation values for PLTS ELECTRE II.

	CoP	QoSD	DoCRNL	UoC	DoCA
${\mathit{O}}_{\mathbf{1}}$	0.35	0.35	0.27	0.49	0.27
${\mathit{O}}_{\mathbf{2}}$	0.27	0.49	0.00	0.49	0.27
${\mathit{O}}_{\mathbf{3}}$	0.27	0.27	0.27	0.27	0.27
${\mathit{O}}_{\mathbf{4}}$	0.35	0.35	0.35	0.49	0.27
${\mathit{O}}_{\mathbf{5}}$	0.49	0.35	0.27	0.35	0.00
${\mathit{O}}_{\mathbf{6}}$	0.49	0.27	0.27	0.27	0.00
${\mathit{O}}_{\mathbf{7}}$	0.35	0.27	0.27	0.49	0.00
${\mathit{O}}_{\mathbf{8}}$	0.27	0.53	0.27	0.49	0.27
${\mathit{O}}_{\mathbf{9}}$	0.27	0.27	0.27	0.27	0.00

.

Thereafter, the concordance matrix was formulated, as shown in

Table 26. Concordance matrix for PLTS ELECTRE II.

	${\mathit{O}}_1$	${\mathit{O}}_2$	${\mathit{O}}_3$	${\mathit{O}}_4$	${\mathit{O}}_5$	${\mathit{O}}_6$	${\mathit{O}}_7$	${\mathit{O}}_8$	${\mathit{O}}_9$
${\mathit{O}}_{\mathbf{1}}$	0	0.92	0.90	0.92	0.46	0.92	0.86	0.92	0.14
${\mathit{O}}_{\mathbf{2}}$	0.0	0	0.14	0.66	0.0	0.38	0.18	0.66	0.0
${\mathit{O}}_{\mathbf{3}}$	0.0	0.82	0	0.94	0.0	0.70	0.66	0.90	0.0
${\mathit{O}}_{\mathbf{4}}$	0.0	0.46	0.0	0	0.0	0.18	0.18	0.84	0.0
${\mathit{O}}_{\mathbf{5}}$	0.66	0.96	0.92	0.96	0	0.90	0.92	0.96	0.14
${\mathit{O}}_{\mathbf{6}}$	0.0	0.58	0.32	0.78	0.0	0	0.62	0.76	0.0
${\mathit{O}}_{\mathbf{7}}$	0.14	0.72	0.46	0.78	0.0	0.48	0	0.76	0.0
${\mathit{O}}_{\mathbf{8}}$	0.0	0.46	0.0	0.28	0.0	0.18	0.18	0	0.0
${\mathit{O}}_{\mathbf{9}}$	0.94	0.96	0.92	1.0	0.92	0.92	0.94	0.96	0

.

The next step was to formulate the discordance matrix. This is shown in

Table 27. Discordance matrix for PLTS ELECTRE II.

	${\mathit{O}}_1$	${\mathit{O}}_2$	${\mathit{O}}_3$	${\mathit{O}}_4$	${\mathit{O}}_5$	${\mathit{O}}_6$	${\mathit{O}}_7$	${\mathit{O}}_8$	${\mathit{O}}_9$
${\mathit{O}}_{\mathbf{1}}$	0	0.0	0.0	0.0	0.41	0.0	0.0	0.0	0.92
${\mathit{O}}_{\mathbf{2}}$	0.9	0	0.75	0.32	0.9	0.75	0.75	0.22	0.92
${\mathit{O}}_{\mathbf{3}}$	0.63	0.0	0	0.0	0.64	0.22	0.11	0.0	0.92
${\mathit{O}}_{\mathbf{4}}$	1.0	0.58	0.58	0	1.0	0.58	0.58	0.0	1.0
${\mathit{O}}_{\mathbf{5}}$	0.12	0.0	0.0	0.0	0	0.0	0.0	0.0	0.75
${\mathit{O}}_{\mathbf{6}}$	0.65	0.54	0.54	0.39	0.75	0	0.53	0.39	0.9
${\mathit{O}}_{\mathbf{7}}$	0.63	0.12	0.54	0.12	0.75	0.29	0	0.12	0.9
${\mathit{O}}_{\mathbf{8}}$	0.9	0.60	0.66	0.49	0.9	0.60	0.66	0	0.92
${\mathit{O}}_{\mathbf{9}}$	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0

.

The concordance thresholds, $c^{-},c^0,c^{\ast }$, and the discordance thresholds, $d^0,d^{\ast }$, were set to $c^{-}=0.55,c^0=0.70,c^{\ast }=0.85$ and $d^0=0.25,d^{\ast }=0.40$. Thereafter, the strong and weak outranking graphs were built, as shown in

Table 28. Strong and weak outranking relationships for PLTS ELECTRE II.

	${\mathit{O}}_1$	${\mathit{O}}_2$	${\mathit{O}}_3$	${\mathit{O}}_4$	${\mathit{O}}_5$	${\mathit{O}}_6$	${\mathit{O}}_7$	${\mathit{O}}_8$	${\mathit{O}}_9$
${\mathit{O}}_{\mathbf{1}}$		$S^F$	$S^F$	$S^F$		$S^F$	$S^F$	$S^F$
${\mathit{O}}_{\mathbf{2}}$		$S^f$						$S^f$
${\mathit{O}}_{\mathbf{3}}$		$S^F$		$S^F$		$S^F$	$S^f$	$S^F$
${\mathit{O}}_{\mathbf{4}}$								$S^F$
${\mathit{O}}_{\mathbf{5}}$	$S^f$	$S^F$	$S^F$	$S^F$		$S^F$	$S^F$	$S^F$
${\mathit{O}}_{\mathbf{6}}$				$S^f$				$S^f$
${\mathit{O}}_{\mathbf{7}}$		$S^F$		$S^F$				$S^F$
${\mathit{O}}_{\mathbf{8}}$
${\mathit{O}}_{\mathbf{9}}$	$S^F$	$S^F$	$S^F$	$S^F$	$S^F$	$S^F$	$S^F$	$S^F$

.

The strong and weak outranking relations were then exploited using the forward and backwards ranking procedures. The final ranking obtained was:

$$O_9 \succ O_5 \succ O_1 \succ O_3 \succ O_7 \succ O_6 \succ O_2 \succ O_4 \succ O_8$$

It can be observed that the ranking from the PLTS ELECTRE II was similar to the ranking from ZPLTS-ELECTRE II. In both rankings, ontology $O_9$ was ranked first, $O_3$ was ranked fourth, and the last ontology was $O_8$. However, the ZPLTS-ELECTRE II method produced more tied rankings than that of the PLTS ELECTRE II method. The difference in rankings are expected due to the following issues. (1) The PLTS ELECTRE II only modeled the five qualitative linguistic criteria and was not able to combine numerical and linguistic data. (2) The PLTS ELECTRE II method was not able to model the credibility aspects of the decision-makers, and (3) the PLTS ELECTRE II method did not allow for individual discordance thresholds to be specified for each criterion, while the ZPLTS-ELECTRE II method did. In light of the above, the proposed ZPLTS-ELECTRE II method offers better modeling capabilities in decision-making problems than the PLTS ELECTRE II method.

5.4. Discussion

In this section, the ZPLTS-ELECTRE II method is further compared with other MCDM methods that have been applied for ranking ontologies as well as with similar existing ELECTRE II methods that are based on fuzzy sets, such as LTS and PLTS. Thereafter, the strengths, weaknesses, and limitations of the proposed ZPLTS-ELECTRE II method are discussed.

5.4.1. Comparison with Existing MCDM Methods for Ontology Ranking

The ZPLTS-ELECTRE II model is compared with other studies that have applied MCDM methods to rank ontologies. The comparison can be seen in

Table 29. Comparison of ZPLTS-ELECTRE II with other MCDM methods for ontology ranking.

Method	Quantitative Criteria	Qualitative Criteria	No. of Criteria	No. of Ontologies
ELECTRE I [15]	Yes	No	8	70
ELECTRE I/III [14]	Yes	No	5	12
WLCRT [16]	Yes	No	8	70
TOPSIS/WSM/WPM [17]	Yes	No	8	70
ZPLTS-ELECTRE II [this study]	Yes	Yes	10	9

.

It can be observed from Table 29 that from all the studies that have applied MCDM methods to rank ontologies, this study has used the largest number of criteria and both the qualitative and quantitative criteria as well as the complexity and usability metrics to perform ontology ranking. Therefore, this study extends the existing literature with the development of the ZPLTS-ELECTRE II method that (1) uses both the quantitative and qualitative criteria, (2) both the complexity and usability metrics, and (3) the largest number of criteria in the task of ontology ranking.

Furthermore, it is shown in Table 29 that nine ontologies were used in this study compared to more ontologies in related studies. The ontology alternatives (9) were chosen to demonstrate the effectiveness of the proposed ZPLTS-ELECTRE II method. Future applications of the ZPLTS-ELECTRE II method may effectively use more alternatives.

5.4.2. Comparison with Existing Fuzzy ELECTRE II Methods

The ZPLTS-ELECTRE II method is compared with other recent developments of ELECTRE II in

Table 30. Comparison of ZPLTS-ELECTRE II with other ELECTRE II enhancements.

Method	Enhancement	Credibility	Structure
PL-ELECTRE II [44]	PLTS	No	Possibility Degree
PLTS-ELECTRE II [45]	PLTS	No	Score and Deviation
PLTS ELECTRE II [62]	PLTS	No	Score and Deviation
HF-ELECTRE II [34]	HFLTS	No	Score and Deviation
ZPLTS-ELECTRE II [this study]	ZPLTS	Yes	Score and Deviation

.

Table 30 compares the ZPLTS-ELECTRE II method with existing enhanced ELECTRE II methods. It is shown in Table 30 that some of these methods share some similar [45,62] and dissimilar [34,44] features with the ZPLTS-ELECTRE II method. In particular, Table 30 shows that none of the existing enhanced ELECTRE II methods enables the measurement of the credibility level of a decision-maker’s evaluation as the proposed ZPLTS-ELECTRE II method does. The differences between the ZPLTS-ELECTRE II method and the other enhancements of ELECTRE II are that (1) the ZPLTS-ELECTRE II method offers better modeling capabilities in decision-making problems through the ability of decision-makers to better express their credibility and confidence levels with the use of Z-numbers as well as the ability of decision-makers to specify an individual discordance level for each criterion, and (2) the ZPLTS-ELECTRE II method provides the capability to model both quantitative numerical criteria as well as qualitative linguistic criteria, unlike the existing enhancements of ELECTRE II methods.

5.4.3. Strengths of ZPLTS-ELECTRE II

The advantages and strengths of the proposed ZPLTS-ELECTRE II method that make it superior to existing ELECTRE II enhancement methods are as follows:

1.

The method can model both quantitative numerical and qualitative linguistic criteria, thereby providing decision-makers with more flexible and realistic expression of their preferences.

2.

Individual discordance thresholds can be specified for each criterion. This provides the decision-makers with more flexibility in expressing their preferences.

3.

Decision-makers are able to express their level of confidence or credibility when providing their evaluations, thereby improving the ability of the model to capture the cognitive nature of the decision-making process.

4.

The applications of the ZPLTS-ELECTRE II method are not constrained to ontology ranking, but rather, it can be applied to any decision-making problem in any domain.

5.4.4. Weaknesses and Limitations of ZPLTS-ELECTRE II

The ZPLTS-ELECTRE II method has some weaknesses and limitations as follows:

1.

The different decision-makers are assigned equal weighting, and the final decision matrix is composed by combining the individual decision-makers’ evaluation matrices with equal importance given to all decision-makers. This may not be the case in some decision-making problems.

2.

The ZPLTS-ELECTRE II method is dependent on the decision-makers for the specifications of the thresholds and parameters, such as the criteria importance weights, the concordance thresholds, and the discordance thresholds.

3.

The ability of the ZPLTS-ELECTRE II method to model both linguistic and numerical data has the implication of individual discordance thresholds for each criterion. This is an advantage as it expands the method’s capability of modeling decision-problems. However, when comparing the method to other ELECTRE II methods, ZPLTS-ELECTRE II has the disadvantage of having a larger number of discordance thresholds to be analyzed and defined as opposed to other methods that may only require a smaller number of discordance thresholds.

6. Conclusions and Future Work

In this paper, the ELECTRE II MCDM method was combined with the concept of ZPLTS to develop a more expressive and real-world modeling technique for decision modeling and decision making, namely, ZPLTS-ELECTRE II. The method was applied to rank nine ontologies comprising knowledge pertaining to mental health and well-being. Ten attributes were used, five of which were quantitative complexity metrics, and the other five were qualitative attributes obtained from four decision-makers’ perspectives. The proposed ZPLTS-ELECTRE II method successfully ranked all ontologies from best to worst. In real-world ontology selection problems, it would be more appropriate to model both quantitative and qualitative attributes. Therefore, the ZPLTS-ELECTRE II method is particularly useful in eliciting ontology ranking and selection as a group decision-making problem. The new ZPLTS-ELECTRE II method provides knowledge engineers and ontologists an opportunity to evaluate and select ontologies from a multi-dimensional perspective as opposed to a singular perspective. The performance of the ZPLTS-ELECTRE II method was compared against that of the traditional ELECTRE II and the PLTS ELECTRE II methods on the same dataset; thereafter, the ZPLTS-ELECTRE II method was compared against previous studies that have used MCDM methods in ontology ranking as well as the existing fuzzy ELECTRE II methods implemented in related studies and displayed superior modeling capabilities to the existing methods.

The ZPLTS-ELECTRE II method can be used for the modeling of real-world decision-making problems, and it is not confined to ontology ranking. Some examples of real-world scenarios that would greatly benefit from the application of ZPLTS-ELECTRE II include (1) medical diagnosis by multiple doctors by modeling their symptoms qualitatively and quantitatively, (2) identifying and selecting the best site locations for implementing construction projects according to various stakeholders such as engineers, architects, developers, surveyors, etc., and (3) evaluating and selecting the appropriate candidates for job positions based on multiple evaluators, such as managers, staff members, team leaders, etc.

In the future, more operations for the ZPLTS can be developed in order to further extend the modeling capabilities of the ZPLTS-ELECTRE II method. Another future direction of research would be to apply the ZPLTS-ELECTRE II model for ontology ranking based on different criteria as well as to perform the ranking of ontologies from other domains apart from the mental health domain. The current study focuses on the PLTS, but in many cases, the decision-maker may encounter further uncertainty when providing their evaluation. Therefore, future studies may extend the current ZPLTS-ELECTRE II method to perform under further uncertain environments such as the interval-valued and uncertain probabilistic linguistic term set. Lastly, it would be interesting to apply the ZPLTS-ELECTRE II method to other fields to aid in the task of ranking and selection.