An Ordinal Consistency Indicator for Pairwise Comparison Matrix

The pairwise comparison (PC) matrix is often used to manifest human judgments, and it has been successfully applied in the analytic hierarchy process (AHP). As a PC matrix is formed by making paired reciprocal comparisons, symmetry is a striking characteristic of a PC matrix. It is this simple but powerful means of resolving multicriteria decision-making problems that is the basis of AHP; however, in practical applications, human judgments may be inconsistent. Although Saaty’s rule for the consistency test is commonly accepted, there is evidence that those so-called “acceptable” PC matrices may not be ordinally consistent, which is a necessary condition for a PC matrix to be accepted. We propose an ordinal consistency indicator called SDR (standard deviation of ranks), derive the upper bound of the SDR, suggest a threshold for a decision-maker to assess whether the ordinal consistency of a PC matrix is acceptable, and reveal a surprising fact that the degree of ordinal inconsistency of a small PC matrix may be more serious than a large one. We made a comparative analysis with some other indicators. Experimental results showed that the ordinal inconsistency measured by the SDR is invariant under heterogeneous judgment measurements with a varied spectrum of scales, and that the SDR is superior to the two compared indicators. Note that the SDR not only works for a multiplicative PC matrix but can also be used for additive and fuzzy PC matrices.


Introduction
Since Thurstone's eminent article was published, the pairwise comparison matrix (PC matrix for short) has been a well-known method used to help decision-makers to manifest experts' subjective judgments [1]. The PC matrix has been successfully applied in methodologies of multicriteria decision-making, such as the analytic hierarchy process (AHP). AHP is a theory of measurement with ratio scales through pairwise comparisons and relies on the judgments of experts to derive a priority vector [2,3]. Decision-making is a mental activity that people often face. Sometimes, it is a complex and difficult task, especially when we are confronted with multiple, usually conflicting, criteria, and when we need to rank multiple alternatives in order to find the best one [4]. To simplify such a complex problem, we may decompose it into smaller and more easily tractable ones; in addition, due to the limit of our capability of handling multi-entities simultaneously, we may compare two entities at a time and express the degree of our preference of one entity over the other one.
The AHP perfectly provides us with a comprehensive framework that deals with problem decomposition in a systematic way, and by using a PC matrix, it enables us simultaneously to cope with the intuitive, the rational, and the irrational when we handle real world decisions and complexities [3]. A debate about the main criticisms of the AHP can be found [5]. Despite their long history, the PC and the AHP are still very attractive subjects for research. There are some survey and review papers about the PC and the AHP [6][7][8].
matrices called CPC matrix (the "corner" PC matrix) and FPC matrix (the "full" PC matrix) are used to illustrate the superiority of the SDR [33]. Finally, the conclusion and discussion are given in Section 5.

Literature Review
Ever since Saaty's [2,14] development of the analytic hierarchy process (AHP), many applications in practical decision-making have been reported [41][42][43][44][45][46][47][48]; however, the main challenge of using pairwise comparison is its lack of consistency, which in practice is very often the case; in other words, most PC matrices are inconsistent. There are several causes of inconsistency, including psychological reasons, type errors, and an insufficient model structure [49].
Detecting and rectifying the inconsistency of a PC matrix is an important and active research area. Consequently, there are various suggestions have been proposed [13,36,37,40,[50][51][52][53][54]. Several inconsistency indicators have been proposed to estimate the deviation of experts' judgments from a situation of complete consistency [12,16,26,27]. Cavallo and D'Apuzzo provided a consistency indicator that has a natural meaning, and it is easy to compute in the additive and multiplicative cases [12]. Brunelli et al. proved the proportionality between some consistency indicators [26]. Brunelli et al. investigated and analyzed 10 indicators from a numerical perspective. The results show that there is a strong or weak correlation between these indicators, and since each indicator is, in fact, a different definition, it has its own advantages and disadvantages. In addition, it also pointed out that there is no recognized standard that can be used to measure the appropriateness of different indicators. [26,27].
Saaty's consistency indicator CI and CR rule of 0.1 have been widely debated [18,28,40,45,[54][55][56]. For example, by using the numerical results of a simulation analysis, Xu et al. showed that it is impossible to find some proper critical values of CR for different matrix sizes, and argued that Saaty's consistency test could be unreasonable [56]. Moreover, there is evidence that those so-called "acceptable" PC matrices may not be ordinally consistent [37,39,40].
However, relative to cardinal consistency, there are seldom studies devoted to ordinal consistency [36,54,57]. Genest and Zhang, based on a Gower plot [58], proposed a graphical method to detect the elements in a PC matrix that cause major ordinal and cardinal inconsistencies [57]. By using a Gower plot and two optimization models, Li and Ma proposed an iterative method to assist a decision-maker in detecting/adjusting ordinal and cardinal inconsistencies [54]. By using route matrices and digraphs, Yang et al. developed an adjustment procedure to help decision-makers correct the inconsistency [36].
In the next section, we introduce an ordinal consistency indicator that is quite different from the foregoing suggestions in that it is not only easy to understand but also easy to use to measure the degree of ordinal inconsistency of a PC matrix. Moreover, the proposed indicator not only works for multiplicative PC matrix but also can be used for an additive and fuzzy PC matrix, as we use only ordinal information instead of cardinal information.

The Proposed Indicator SDR
For an n × n PC matrix A = a ij , we first construct a corresponding rank matrix R: Here, entry r ij is the rank of a ij in the i-th row of A. For example, a 4 × 4 PC matrix A as follows: We will have a corresponding rank matrix R as follows: Note that the proposed method is applicable to the additive and fuzzy PC matrices. Evidently, entry r ij belongs to {1, 2, . . . , n}. Then an ordinal inconsistency indicator SDR (standard deviation of ranks) can be defined [59] as: The variable σ j is the standard deviation of the j-th column of the corresponding rank matrix R n×n . For the 4 × 4 PC matrix A mentioned above, we will obtain SDR[A] = 0.25, as σ 1 = σ 2 = 0.5 and σ 3 = σ 4 = 0. It is obviously that for a PC matrix A of order n, the minimum of SDR is 0, that is, A is completely ordinally consistent. Without surprise, the greater the SDR[A], the greater the inconsistency of the PC matrix A. The worst situation of inconsistency will occur when all entries of a column with a different rank and all entries of a raw with a different rank, if without tie between two entities (When the result of a comparison is a tie, we can use a more detailed judgment, for example, the numbers from 1.1 to 1.9 [60]). For example, Here, we have σ j =1.29, for all j = 1, 2, 3, 4. Based on such an observation, it is easy to derive the following proposition.

.
Proof. It is well known that the standard deviation σ of n observations is Now, let x i = i, and use σ 1∼n to stand for the standard deviation of these n numbers, then we have Thus, it is easy to derive the following result: ).
Evidently, the upper bound of SDR[A] is a function of n, i.e., the size of a PC matrix A. This property is important, because one of the reasons why Saaty's consistency check is criticized is that it cannot provide proper critical values for matrices of different sizes [29]. Table 1 lists the upper bounds of SDR[A] for a PC matrix A of size n from 3 to 15.     Here, the Greek small letter δ stands for a threshold, where 0 ≤ δ ≤ 1, and the choice of an adequate threshold depends on the decision-maker's attitude. The more rigorous attitude is, the higher (i.e., a smaller value of δ) the threshold is. Alternatively, we can set the threshold, in a statistical manner, to be under a certain percentile (denoted by p) of the distribution of SDR[A], for example, p = 0.05 or 0.1. Similarly, the choice of an appropriate percentile also depends on the decision-maker's attitude. To provide such a threshold, we examined the distribution of SDR[A] by systematically generating possible PC matrices. That is, a total of (2x − 1) ∑ n−1 i=1 i matrices, if we use an x-scales measurement for a PC matrix of size n. Here, the term of x-scales measurement means the decisionmaker can choose 2x − 1 symmetric reciprocal values to present his/her assessment about the relative comparisons of two entities. (Saaty [2] suggested a widely used symmetric reciprocal 9-scales 1 9 , 1 8 , · · · , 1 2 , 1, 2, · · · , 8, 9 with a neutral value of 1). Therefore, if we use a 4-scale measurement for a 4 × 4 PC matrix, there are a total of (117,639 = 7 6 ) matrices. Experimental results show that there are only 1166 4 × 4 PC matrices, which is less than one percent (1166/117,639 = 0.00991) of all matrices, which are completely ordinally consistent, i.e., SDR[A] = 0; therefore, it is reasonable to set the threshold as in 5th or 10th percentile. Strictly speaking, a PC matrix A can be called ordinally consistent only if SDR[ ] = 0; otherwise, it should be called ordinally inconsistent. However, in practice, we may use a threshold to decide whether a PC matrix of order n is acceptable or not, according to the following inequality:

SDR[ ] ≤ δ ~
Here, the Greek small letter δ stands for a threshold, where 0 ≤ δ ≤ 1, and the choice of an adequate threshold depends on the decision-maker's attitude. The more rigorous attitude is, the higher (i.e., a smaller value of δ) the threshold is. Alternatively, we can set the threshold, in a statistical manner, to be under a certain percentile (denoted by ) of the distribution of SDR[ ], for example, = 0.05 or 0.1. Similarly, the choice of an appropriate percentile also depends on the decision-maker's attitude. To provide such a threshold, we examined the distribution of SDR[ ] by systematically generating possible PC matrices. That is, a total of (2 − 1) ∑ matrices, if we use an x-scales measurement for a PC matrix of size n. Here, the term of x-scales measurement means the decision-maker can choose 2x − 1 symmetric reciprocal values to present his/her assessment about the relative comparisons of two entities. (Saaty [2] suggested a widely used symmetric reciprocal 9-scales [ , , ⋯ , , 1, 2, ⋯ , 8,9] with a neutral value of 1). Therefore, if we use a 4-scale measurement for a 4 × 4 PC matrix, there are a total of (117,639 = 7 ) matrices. Experimental results show that there are only 1166 4 × 4 PC matrices, which is less than one percent (1166/117,639 = 0.00991) of all matrices, which are completely ordinally consistent, i.e., SDR[ ] = 0; therefore, it is reasonable to set the threshold as in 5th or 10th percentile. Table 2 summarizes the results of seven percentiles ( = 0.01~0.5) for 3 × 3 PC matrix using three different judgment scale measurements (S: Saaty's [2], I: Inverse linear [10], B: Balanced [11]) with varied spectrum of scales (from 3-scales to 9-scales). Note that we conducted the simulation by systematically generating possible PC matrices, and that a tie between two entities is allowed.  Table 2 summarizes the results of seven percentiles (p = 0.01~0.5) for 3 × 3 PC matrix using three different judgment scale measurements (S: Saaty's [2], I: Inverse linear [10], B: Balanced [11]) with varied spectrum of scales (from 3-scales to 9-scales). Note that we conducted the simulation by systematically generating possible PC matrices, and that a tie between two entities is allowed. The results, presented in Table 2, show that the ordinal inconsistency of a PC matrix A measured by the SDR[A] is invariant under different judgment scale measurements with a varied spectrum of scales. This is because we use only ordinal information instead of cardinal information.

Comparisons
In this study, we made three different comparisons between the proposed indicator SDR and other indicators. Firstly, we examined the linear correlation via the Pearson , between pairs of some consistency indicators suggested in the literature. The results are reported in Table 3. (Since we conducted simulations by systematically generating possible PC matrices, the results are based on a 5-scale instead of Saaty's 9-scale measurement). The notations used in Table 3 are followed a study of Brunelli et al. [26]; however, we conducted simulations by systematically generating possible PC matrices instead of generating random matrices or perturbed consistent matrices the way they adopted. In Table 3, via figures in boldface, we can find that there is a strong linear correlation among the indicators of CI, CI*, GCI, and I CD . This result is consistent with the report of Brunelli et al. [26]. Note that there is no indicator that has a strong linear correlation with the SDR; this evidence makes the SDR unique among other indicators.
Secondly, to demonstrate the sensitivity of SDR in detecting ordinal inconsistency and to examine Saaty's consistency test, we conducted a simulation as follows. For 4 × 4 PC matrix, there are only 955,505 matrices out of all possible matrices (24,137,569 = 17 6 ) with CI ≤ 0.1, i.e., only 3.96% of 4 × 4 PC matrix satisfy Saaty's rule; this result confirms a previous study [55]. Among these "acceptable" matrices, we further identified 1056 matrices that with 0.0999 ≤ CI ≤ 0.09999, and compared their values of CI and SDR. Figure 3 shows the results.
Symmetry 2021, 13, x FOR PEER REVIEW 8 of 12 Figure 3 indicates that these so-called "acceptable" PC matrices are ordinally inconsistent. This result coincides with several previous studies [37,40]; however, ordinally consistent is a necessary condition of a PC matrix to be considered as "acceptable". Thirdly, we compared CI, SDR, and Dissonance in their capability of measuring ordinal inconsistency of PC matrix. Dissonance is an ordinal consistency indicator suggested by Siraj et al.; they also proposed a congruence measure used for cardinally inconsistent judgments [40]. To make this comparison, we use two types of PC matrices that have been analyzed in literature [33]. Koczkodaj and Szwarc analyzed a kind of PC matrix called "corner PC matrix or CPC," that with all ones except for two corners as follows: Trivially, the only possibility of this matrix to be consistent is when = 1. By using a distance-based inconsistency reduction algorithm, Koczkodaj and Szwarc proclaimed  Figure 3 indicates that these so-called "acceptable" PC matrices are ordinally inconsistent. This result coincides with several previous studies [37,40]; however, ordinally consistent is a necessary condition of a PC matrix to be considered as "acceptable".
Thirdly, we compared CI, SDR, and Dissonance in their capability of measuring ordinal inconsistency of PC matrix. Dissonance is an ordinal consistency indicator suggested by Siraj et al.; they also proposed a congruence measure used for cardinally inconsistent judgments [40]. To make this comparison, we use two types of PC matrices that have been analyzed in literature [33]. Koczkodaj and Szwarc analyzed a kind of PC matrix called "corner PC matrix or CPC," that with all ones except for two corners as follows: Trivially, the only possibility of this matrix to be consistent is when x = 1. By using a distance-based inconsistency reduction algorithm, Koczkodaj and Szwarc proclaimed that x > 1.5 is suspiciously high and the PC matrix needs to be re-examined [29]. In fact, by using the SDR, we find that a matrix CPC(x, n) even with a small value of x less than 1.5 would be inconsistent. Figure 4 shows the values of CI, SDR, and Dissonance of the CPC(x, 5). Here, we set x from 0.5 to 1.5 with an increment of 0.05.  When ≠ 1, in contrast to the values of SDR and Dissonance (this refers to the overall dissonance (Ψ) [40]; note that since these values fall on the x-axis, they will be visible only in color.) that are constants of 0.614738 and 0, respectively, the values of CI increasing from 0 to 0.00496 or 0.014654 depends on the values of x is 1.5 or 0.5, respectively. Note that the curve of CI asymmetric to = 1. The results show that the indicator SDR is superior to the two compared indicators in measuring the ordinal inconsistency of a PC matrix.
Koczkodaj and Szwarc analyzed another type of PC matrix, called FPC (the "full" PC matrix or the PC matrix full of x). The matrix FPC(x, n), with > 1, is defined as follows: Similarly, the only possibility of this matrix to be consistent is when = 1; however, unlike CPC(x, n), it has all erroneous triads. Figure 5 shows the values of CI, SDR, and Dissonance of the FPC(x, 5). Here, we also set x from 0.5 to 1.5 with an increment of 0.05. When x = 1, in contrast to the values of SDR and Dissonance (this refers to the overall dissonance (Ψ) [40]; note that since these values fall on the x-axis, they will be visible only in color). that are constants of 0.614738 and 0, respectively, the values of CI increasing from 0 to 0.00496 or 0.014654 depends on the values of x is 1.5 or 0.5, respectively. Note that the curve of CI asymmetric to x = 1. The results show that the indicator SDR is superior to the two compared indicators in measuring the ordinal inconsistency of a PC matrix.
Koczkodaj and Szwarc analyzed another type of PC matrix, called FPC (the "full" PC matrix or the PC matrix full of x). The matrix FPC(x, n), with x > 1, is defined as follows: Similarly, the only possibility of this matrix to be consistent is when x = 1; however, unlike CPC(x, n), it has all erroneous triads. Figure 5 shows the values of CI, SDR, and Dissonance of the FPC(x, 5). Here, we also set x from 0.5 to 1.5 with an increment of 0.05.
Similarly, the only possibility of this matrix to be consistent is when = 1; howev unlike CPC(x, n), it has all erroneous triads. Figure 5 shows the values of CI, SDR, a Dissonance of the FPC(x, 5). Here, we also set x from 0.5 to 1.5 with an increment of 0. The values of CI increasing from 0 to 0.0165 or 0.0487 depends on the values of x is 1.5 or 0.5, respectively. For indicator Dissonance, the values are zero, i.e., fall on the x-axis, with respect to x from 1 to 1.5, and are 0.03333 with respect to x from 0.95 to 0.5. All

Conclusions
This study attempted to reinforce Saaty's consistency test, as there is evidence that those so-called "acceptable" PC matrices may not be ordinally consistent. We propose an indicator called SDR ,. reveals a surprising fact that the degree of ordinal inconsistency of a small PC matrix may be more serious than a large one. We made three different comparative analyses with some other indicators. The results show that the indicator SDR measures exactly what ordinal inconsistency means and that the SDR is superior to the two compared indicators. In practice, we suggest decision-makers not only to follow Saaty's rule of 0.1 but also to check the SDR.
Since we use only ordinal information instead of cardinal information of a PC matrix in computing the SDR, it is worth noting that the ordinal inconsistency of a PC matrix measured by using the indicator SDR is invariant under heterogeneous judgment measurements with a varied spectrum of scales, and that the proposed SDR not only works for multiplicative PC matrix but can also be used for additive and fuzzy PC matrices. Finally, since it is somewhat difficult to analyze and derive the upper bound of SDR[A]. for a PC matrix A that a tie between two entities is allowed, we leave it as a topic for future research. Moreover, how we can use this ordinal consistency indicator to improve the cardinal consistency of a PC matrix seems worth investigating.