An Ordinal Consistency Indicator for Pairwise Comparison Matrix

Abstract

The pairwise comparison (PC) matrix is often used to manifest human judgments, and it has been successfully applied in the analytic hierarchy process (AHP). As a PC matrix is formed by making paired reciprocal comparisons, symmetry is a striking characteristic of a PC matrix. It is this simple but powerful means of resolving multicriteria decision-making problems that is the basis of AHP; however, in practical applications, human judgments may be inconsistent. Although Saaty’s rule for the consistency test is commonly accepted, there is evidence that those so-called “acceptable” PC matrices may not be ordinally consistent, which is a necessary condition for a PC matrix to be accepted. We propose an ordinal consistency indicator called SDR (standard deviation of ranks), derive the upper bound of the SDR, suggest a threshold for a decision-maker to assess whether the ordinal consistency of a PC matrix is acceptable, and reveal a surprising fact that the degree of ordinal inconsistency of a small PC matrix may be more serious than a large one. We made a comparative analysis with some other indicators. Experimental results showed that the ordinal inconsistency measured by the SDR is invariant under heterogeneous judgment measurements with a varied spectrum of scales, and that the SDR is superior to the two compared indicators. Note that the SDR not only works for a multiplicative PC matrix but can also be used for additive and fuzzy PC matrices.

1. Introduction

Since Thurstone’s eminent article was published, the pairwise comparison matrix (PC matrix for short) has been a well-known method used to help decision-makers to manifest experts’ subjective judgments [1]. The PC matrix has been successfully applied in methodologies of multicriteria decision-making, such as the analytic hierarchy process (AHP). AHP is a theory of measurement with ratio scales through pairwise comparisons and relies on the judgments of experts to derive a priority vector [2,3]. Decision-making is a mental activity that people often face. Sometimes, it is a complex and difficult task, especially when we are confronted with multiple, usually conflicting, criteria, and when we need to rank multiple alternatives in order to find the best one [4]. To simplify such a complex problem, we may decompose it into smaller and more easily tractable ones; in addition, due to the limit of our capability of handling multi-entities simultaneously, we may compare two entities at a time and express the degree of our preference of one entity over the other one.

The AHP perfectly provides us with a comprehensive framework that deals with problem decomposition in a systematic way, and by using a PC matrix, it enables us simultaneously to cope with the intuitive, the rational, and the irrational when we handle real world decisions and complexities [3]. A debate about the main criticisms of the AHP can be found [5]. Despite their long history, the PC and the AHP are still very attractive subjects for research. There are some survey and review papers about the PC and the AHP [6,7,8].

A PC matrix is presented as a square matrix A = [a_ij], where a_ij > 0, for every i, j = 1, n, manifest expert’s assessment based on relative comparisons of an entity i over another entity j. Here, an entity could be a criterion or an alternative. Several scales have been proposed to present subjective judgment [2,9,10,11]. It is worth noting that there is a general unified framework for PC matrices where entry a_ij of the matrix can be a preference ratio, i.e., multiplicative case, or a preference difference, i.e., additive case, or a value in the range of [0, 1] that measures the distance from the indifference, i.e., fuzzy case [12]. By using a logarithmic mapping and an exponential mapping, the multiplicative case and additive case are interchangeable [13]. In this study, we focus on the multiplicative case, but the results can also be used for additive and fuzzy PC matrices.

Since entry a_ij is an estimate of the underlying ratio (α_i/α_j), without surprising that a_ii = 1 for all i. A PC matrix is called reciprocal if $a_{ij}=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$a_{ji}$}\right.,\mathrm{for}\mathrm{all}i,j=1,2,\dots ,n$, and is called consistent if $a_{ij}{a}_{jk}=a_{ik},\mathrm{for}\mathrm{all}i,j,k=1,2,\dots ,n.$ All PC matrices discussed here are reciprocal because this is an essential and rational condition. As the pairwise comparisons that we consider are formed by making paired reciprocal comparisons, symmetry is a striking characteristic of a PC matrix. It is this simple, but powerful means of resolving multicriteria decision-making problems that is the basis of the AHP [14]; however, they are not necessarily consistent because this is a desirable property that is what we want to achieve, not only from the academic perspective but also from the decision-maker’s perspective. Although sometimes we need to deal with an incomplete PC matrix [15], hereinafter, a PC matrix means a complete reciprocal PC matrix.

Two theoretical issues relating to the usage of the PC matrix are of special interest: the choice of a prioritization technique and inconsistency evaluating [16]. There are several different methods commonly used to derive the estimated priority vector $\mathsf{\alpha }=\left({\alpha }_1,{\alpha }_2,\cdots ,{\alpha }_n\right)$ of a PC matrix [17,18,19,20,21,22,23]. It is well known that if a PC matrix A is inconsistent, different prioritization methods may derive different estimates of priority vector $\mathsf{\alpha }$; however, if A is consistent, different prioritization methods should derive the same estimate [18]. Unfortunately, the judgments of decision-makers may be inconsistent, especially in the situation of many entities involved.

This study is devoted to the second issue: inconsistency evaluating. The issue of inconsistency has attracted increasing attention from inception to recent years and resulted in various indicators of measurement have been proposed [16,24,25,26,27,28,29,30,31,32,33]; however, the usage of the indicators is justified only by some heuristics, and it is still unclear what they really “measure” [16]. Although a considerable number of studies have been performed, the discussion on inconsistency indicators is far from being over, and the ground is still fertile for debates [27]. Recently, there are some studies have been devoted to proposing a set of properties to characterize the inconsistency indicators [29,34,35].

There are two types of inconsistency: cardinal and ordinal. While cardinal consistency is a sufficient condition of a PC matrix to be considered acceptable, ordinal consistency is a necessary condition [36]. In other words, cardinal consistency guarantees ordinal consistency, and cardinal inconsistency does not imply ordinal inconsistency; however, ordinal inconsistency guarantees cardinal inconsistency [37]. It seems that only one inconsistency indicator is insufficient for describing the inconsistency of a PC matrix; thus, the improvement of the notion of inconsistency should be necessary [38]. Moreover, it has been confirmed that a relatively high percentage of comparison matrices satisfying Saaty’s rule of consistency test is ordinally inconsistent [37,39,40]. (Rule of “consistency ratio should be less than 0.1.”) However, Saaty’s rule of consistency is a test that is still commonly accepted; therefore, we focus on the ordinal inconsistency.

The remainder of this paper is organized as follows. In Section 2, a brief literature review is introduced. In Section 3, we present the proposed ordinal inconsistency indicator called SDR (standard deviation of ranks), derive the upper bound of the SDR that is a function of the size of the PC matrix, and suggest a threshold for the decision-maker to decide whether the ordinal consistency of a PC matrix is acceptable. In Section 4, some inconsistency indicators are compared and results are presented. Further, two kinds of PC matrices called CPC matrix (the “corner” PC matrix) and FPC matrix (the “full” PC matrix) are used to illustrate the superiority of the SDR [33]. Finally, the conclusion and discussion are given in Section 5.

2. Literature Review

Ever since Saaty’s [2,14] development of the analytic hierarchy process (AHP), many applications in practical decision-making have been reported [41,42,43,44,45,46,47,48]; however, the main challenge of using pairwise comparison is its lack of consistency, which in practice is very often the case; in other words, most PC matrices are inconsistent. There are several causes of inconsistency, including psychological reasons, type errors, and an insufficient model structure [49].

Detecting and rectifying the inconsistency of a PC matrix is an important and active research area. Consequently, there are various suggestions have been proposed [13,36,37,40,50,51,52,53,54]. Several inconsistency indicators have been proposed to estimate the deviation of experts’ judgments from a situation of complete consistency [12,16,26,27]. Cavallo and D’Apuzzo provided a consistency indicator that has a natural meaning, and it is easy to compute in the additive and multiplicative cases [12]. Brunelli et al. proved the proportionality between some consistency indicators [26]. Brunelli et al. investigated and analyzed 10 indicators from a numerical perspective. The results show that there is a strong or weak correlation between these indicators, and since each indicator is, in fact, a different definition, it has its own advantages and disadvantages. In addition, it also pointed out that there is no recognized standard that can be used to measure the appropriateness of different indicators. [26,27].

Saaty’s consistency indicator CI and CR rule of 0.1 have been widely debated [18,28,40,45,54,55,56]. For example, by using the numerical results of a simulation analysis, Xu et al. showed that it is impossible to find some proper critical values of CR for different matrix sizes, and argued that Saaty’s consistency test could be unreasonable [56]. Moreover, there is evidence that those so-called “acceptable” PC matrices may not be ordinally consistent [37,39,40].

However, relative to cardinal consistency, there are seldom studies devoted to ordinal consistency [36,54,57]. Genest and Zhang, based on a Gower plot [58], proposed a graphical method to detect the elements in a PC matrix that cause major ordinal and cardinal inconsistencies [57]. By using a Gower plot and two optimization models, Li and Ma proposed an iterative method to assist a decision-maker in detecting/adjusting ordinal and cardinal inconsistencies [54]. By using route matrices and digraphs, Yang et al. developed an adjustment procedure to help decision-makers correct the inconsistency [36].

In the next section, we introduce an ordinal consistency indicator that is quite different from the foregoing suggestions in that it is not only easy to understand but also easy to use to measure the degree of ordinal inconsistency of a PC matrix. Moreover, the proposed indicator not only works for multiplicative PC matrix but also can be used for an additive and fuzzy PC matrix, as we use only ordinal information instead of cardinal information.

3. The Proposed Indicator SDR

For an $n\times n$ PC matrix $A=\left[a_{ij}\right]$, we first construct a corresponding rank matrix R:

$$R_{n\times n}=\left[r_{ij}\right],\forall i,j=1,2,\cdots ,n.$$

Here, entry $r_{ij}$ is the rank of $a_{ij}$ in the i-th row of A. For example, a $4\times 4$ PC matrix $A$ as follows:

$$A=\left[\begin{array}{cc}\begin{array}{cc}1& 2\\ \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.& 1\end{array}& \begin{array}{cc}4& 5\\ 3& 6\end{array}\\ \begin{array}{cc}\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$4$}\right.& \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$3$}\right.\\ \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$5$}\right.& \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$6$}\right.\end{array}& \begin{array}{cc}1& 3\\ \raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$3$}\right.& 1\end{array}\end{array}\right].$$

We will have a corresponding rank matrix R as follows:

$$R=\left[\begin{array}{cc}\begin{array}{cc}4& 3\\ 4& 3\end{array}& \begin{array}{cc}2& 1\\ 2& 1\end{array}\\ \begin{array}{cc}4& 3\\ 3& 4\end{array}& \begin{array}{cc}2& 1\\ 2& 1\end{array}\end{array}\right].$$

Note that the proposed method is applicable to the additive and fuzzy PC matrices. Evidently, entry $r_{ij}$ belongs to $\left\{1,2,\dots ,n\right\}.$ Then an ordinal inconsistency indicator SDR (standard deviation of ranks) can be defined [59] as:

$$\mathrm{SDR}\left[A\right]=\frac{{{\displaystyle \sum }}_1^n{\sigma }_j}{n}.$$

The variable ${\sigma }_j$ is the standard deviation of the j-th column of the corresponding rank matrix $R_{n\times n}$. For the $4\times 4$ PC matrix $A$ mentioned above, we will obtain $\mathrm{SDR}\left[A\right]=0.25$, as ${\sigma }_1={\sigma }_2=0.5$ and ${\sigma }_3={\sigma }_4=0$. It is obviously that for a PC matrix A of order n, the minimum of SDR is 0, that is, A is completely ordinally consistent. Without surprise, the greater the $\mathrm{SDR}\left[A\right],$ the greater the inconsistency of the PC matrix A. The worst situation of inconsistency will occur when all entries of a column with a different rank and all entries of a raw with a different rank, if without tie between two entities (When the result of a comparison is a tie, we can use a more detailed judgment, for example, the numbers from 1.1 to 1.9 [60]). For example,

$${\mathrm{R}}_{4\times 4}=\left[\begin{array}{cc}\begin{array}{cc}1& 2\\ 2& 1\end{array}& \begin{array}{cc}3& 4\\ 4& 3\end{array}\\ \begin{array}{cc}3& 4\\ 4& 3\end{array}& \begin{array}{cc}2& 1\\ 1& 2\end{array}\end{array}\right].$$

Here, we have ${\sigma }_j=$1.29, for all $j=$ 1, 2, 3, 4. Based on such an observation, it is easy to derive the following proposition.

Proposition 1.

For a PC matrix A of order n, the upper bound of$SDR\left[A\right]$is$\sqrt{\frac{n\left(n+1\right)}{12}}$.

Proof. 

It is well known that the standard deviation $\mathsf{\sigma }$ of n observations is

$$\mathsf{\sigma }=\sqrt{\frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{x}}_{\mathrm{i}}-\mathrm{x}¯\right)}^2}{\left(\mathrm{n}-1\right)}}.$$

Now, let ${\mathrm{x}}_{\mathrm{i}}=\mathrm{i}$, and use ${\mathsf{\sigma }}_{1~\mathrm{n}}$ to stand for the standard deviation of these n numbers, then we have

$${\mathsf{\sigma }}_{1~\mathrm{n}}=\sqrt{\frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{(\mathrm{i}-\frac{\mathrm{n}+1}{2})}^2}{\left(\mathrm{n}-1\right)}}.$$

Thus, it is easy to derive the following result: (Since ${\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{i}=\frac{\mathrm{n}\left(\mathrm{n}+1\right)}{2},$ and ${\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{i}}^2=\frac{\mathrm{n}\left(\mathrm{n}+1\right)\left(2\mathrm{n}+1\right)}{6}$).

$${\mathsf{\sigma }}_{1~\mathrm{n}}=\sqrt{\frac{\mathrm{n}\left(\mathrm{n}+1\right)}{12}}.$$

 □

Evidently, the upper bound of $\mathrm{SDR}\left[A\right]$ is a function of n, i.e., the size of a PC matrix A. This property is important, because one of the reasons why Saaty’s consistency check is criticized is that it cannot provide proper critical values for matrices of different sizes [29].

Table 1. The upper bounds of the $\mathrm{SDR}\left[A\right]$ for PC matrix A of size $n=3~15$.

$\mathit{n}$	3	4	5	6	7	8	9	10	11	12	13	14	15
${\sigma }_{1~\mathrm{n}}$	1	1.29	1.58	1.87	2.16	2.45	2.74	3.03	3.32	3.61	3.89	4.18	4.47

lists the upper bounds of $\mathrm{SDR}\left[A\right]$ for a PC matrix A of size $n$ from 3 to 15.

Figure 1 reveals that the upper bound of $\mathrm{SDR}\left[A\right]$ steadily increases as the size of PC matrix A increases.

We use $\mathrm{NSDR}\left[A\right]$, which stands for the normalized $\mathrm{SDR}\left[A\right]$ (i.e., $\mathrm{NSDR}\left[A\right]=\frac{\mathrm{SDR}\left[A\right]}{n})$; thus, we will obtain the following proposition.

Proposition 2.

For a PC matrix A with order$n\ge 3$, the upper bound of$NSDR\left[A\right]$lies in the range of$\left[\frac{1}{\sqrt{12}},\frac{1}{3}\right].$

Proof. 

By definition, the upper bound of $\mathrm{NSDR}\left[A\right]$ is $\sqrt{\frac{\left(\mathrm{n}+1\right)}{12\mathrm{n}}}.$ Obviously, when $\mathrm{n}=3,$ the upper bound of $\mathrm{NSDR}\left[A\right]=\frac{1}{3},$ and $\underset{\mathrm{n}\to \infty }{\lim }\sqrt{\frac{\left(\mathrm{n}+1\right)}{12\mathrm{n}}}=\frac{1}{\sqrt{12}}.$ □

Figure 2 reveals a surprising fact that the upper bound of $\mathrm{NSDR}\left[A\right]$ declines as the size of PC matrix A increases.

Strictly speaking, a PC matrix A can be called ordinally consistent only if $\mathrm{SDR}\left[A\right]=0$; otherwise, it should be called ordinally inconsistent. However, in practice, we may use a threshold to decide whether a PC matrix of order n is acceptable or not, according to the following inequality:

$$\mathrm{SDR}\left[A\right]\le \mathsf{\delta }{\sigma }_{1~\mathrm{n}}$$

Here, the Greek small letter $\mathsf{\delta }$ stands for a threshold, where $0\le \mathsf{\delta }\le 1$, and the choice of an adequate threshold depends on the decision-maker’s attitude. The more rigorous attitude is, the higher (i.e., a smaller value of $\mathsf{\delta }$) the threshold is. Alternatively, we can set the threshold, in a statistical manner, to be under a certain percentile (denoted by $p$) of the distribution of $\mathrm{SDR}\left[A\right]$, for example, $p=$ 0.05 or 0.1. Similarly, the choice of an appropriate percentile also depends on the decision-maker’s attitude. To provide such a threshold, we examined the distribution of $\mathrm{SDR}\left[A\right]$ by systematically generating possible PC matrices. That is, a total of ${\left(2x-1\right)}^{{\displaystyle \sum }_{i=1}^{n-1}i}$ matrices, if we use an x-scales measurement for a PC matrix of size n. Here, the term of x-scales measurement means the decision-maker can choose 2x − 1 symmetric reciprocal values to present his/her assessment about the relative comparisons of two entities. (Saaty [2] suggested a widely used symmetric reciprocal 9-scales $\left[\frac{1}{9},\frac{1}{8},\cdots ,\frac{1}{2},1,2,\cdots ,8,9\right]$ with a neutral value of 1). Therefore, if we use a 4-scale measurement for a $4\times 4$ PC matrix, there are a total of (117,639 $=7^6$) matrices. Experimental results show that there are only 1166 $4\times 4$ PC matrices, which is less than one percent (1166/117,639 $=$ 0.00991) of all matrices, which are completely ordinally consistent, i.e., $\mathrm{SDR}\left[A\right]=0$; therefore, it is reasonable to set the threshold as in 5th or 10th percentile.

Table 2. The thresholds of the $\mathrm{SDR}\left[A\right]$ for a $3\times 3$ PC matrix A.

	$\mathit{p}=$	0.01	0.05	0.1	0.2	0.3	0.4	0.5
3-scales 4-scales 5-scales 6-scales 7-scales 8-scales 9-scales	S	0.00	0.00	0.00	0.19	0.19	0.39	0.53
	I	0.00	0.00	0.00	0.19	0.19	0.39	0.53
	B	0.00	0.00	0.00	0.19	0.19	0.39	0.53

summarizes the results of seven percentiles ($p=$ 0.01~0.5) for $3\times 3$ PC matrix using three different judgment scale measurements (S: Saaty’s [2], I: Inverse linear [10], B: Balanced [11]) with varied spectrum of scales (from 3-scales to 9-scales). Note that we conducted the simulation by systematically generating possible PC matrices, and that a tie between two entities is allowed.

The results, presented in Table 2, show that the ordinal inconsistency of a PC matrix A measured by the $\mathrm{SDR}\left[A\right]$ is invariant under different judgment scale measurements with a varied spectrum of scales. This is because we use only ordinal information instead of cardinal information.

4. Comparisons

In this study, we made three different comparisons between the proposed indicator SDR and other indicators. Firstly, we examined the linear correlation via the Pearson correlation coefficient ($\mathrm{Correl}\left(\mathrm{X},\mathrm{Y}\right)=\frac{{{\displaystyle \sum }}_{i=1}^n\left(x_i-\overline{x}\right)\left(y_i-\overline{y}\right)}{\sqrt{{{\displaystyle \sum }}_{i=1}^n{\left(x_i-\overline{x}\right)}^2{{\displaystyle \sum }}_{i=1}^n{\left(y_i-\overline{y}\right)}^2}}$), between pairs of some consistency indicators suggested in the literature. The results are reported in

Table 3. Linear correlation between pairs of consistency indicators.

	CI	SDR	CI*	GCI	HCI	RE	GW	KI	ICD
CI	1	0.743	0.967	0.998	0.933	0.798	0.924	0.713	0.946
SDR	0.743	1	0.619	0.772	0.705	0.855	0.845	0.829	0.705
CI*	0.967	0.619	1	0.951	0.9	0.716	0.839	0.61	0.882
GCI	0.998	0.772	0.951	1	0.93	0.813	0.938	0.744	0.943
HCI	0.933	0.705	0.9	0.93	1	0.765	0.937	0.646	0.89
RE	0.798	0.855	0.716	0.813	0.765	1	0.842	0.696	0.734
GW	0.924	0.845	0.839	0.938	0.937	0.842	1	0.803	0.867
KI	0.713	0.829	0.61	0.744	0.646	0.696	0.803	1	0.619
ICD	0.946	0.705	0.882	0.943	0.89	0.734	0.867	0.619	1

. (Since we conducted simulations by systematically generating possible PC matrices, the results are based on a 5-scale instead of Saaty’s 9-scale measurement). The notations used in Table 3 are followed a study of Brunelli et al. [26]; however, we conducted simulations by systematically generating possible PC matrices instead of generating random matrices or perturbed consistent matrices the way they adopted.

In Table 3, via figures in boldface, we can find that there is a strong linear correlation among the indicators of CI, CI*, GCI, and I_CD. This result is consistent with the report of Brunelli et al. [26]. Note that there is no indicator that has a strong linear correlation with the SDR; this evidence makes the SDR unique among other indicators.

Secondly, to demonstrate the sensitivity of SDR in detecting ordinal inconsistency and to examine Saaty’s consistency test, we conducted a simulation as follows. For $4\times 4$ PC matrix, there are only 955,505 matrices out of all possible matrices (24,137,569 $=$ ${17}^6$) with $\mathrm{CI}\le 0.1$, i.e., only 3.96% of $4\times 4$ PC matrix satisfy Saaty’s rule; this result confirms a previous study [55]. Among these “acceptable” matrices, we further identified 1056 matrices that with $0.0999\le \mathrm{CI}\le 0.09999$, and compared their values of CI and SDR. Figure 3 shows the results.

Figure 3 indicates that these so-called “acceptable” PC matrices are ordinally inconsistent. This result coincides with several previous studies [37,40]; however, ordinally consistent is a necessary condition of a PC matrix to be considered as “acceptable”.

Thirdly, we compared CI, SDR, and Dissonance in their capability of measuring ordinal inconsistency of PC matrix. Dissonance is an ordinal consistency indicator suggested by Siraj et al.; they also proposed a congruence measure used for cardinally inconsistent judgments [40]. To make this comparison, we use two types of PC matrices that have been analyzed in literature [33]. Koczkodaj and Szwarc analyzed a kind of PC matrix called “corner PC matrix or CPC,” that with all ones except for two corners as follows:

$$\mathrm{CPC}\left(x,n\right)=\left[\begin{array}{cc}\begin{array}{cc}1& 1\\ 1& 1\end{array}& \begin{array}{cc}\dots & x\\ \cdots & 1\end{array}\\ \begin{array}{cc}⋮& ⋮\\ x^{-1}& 1\end{array}& \begin{array}{cc}⋮& ⋮\\ \cdots & 1\end{array}\end{array}\right].$$

Trivially, the only possibility of this matrix to be consistent is when $x=1$. By using a distance-based inconsistency reduction algorithm, Koczkodaj and Szwarc proclaimed that x > 1.5 is suspiciously high and the PC matrix needs to be re-examined [29]. In fact, by using the SDR, we find that a matrix CPC(x, n) even with a small value of x less than 1.5 would be inconsistent. Figure 4 shows the values of CI, SDR, and Dissonance of the CPC(x, 5). Here, we set x from 0.5 to 1.5 with an increment of 0.05.

When $x\ne 1,$ in contrast to the values of SDR and Dissonance (this refers to the overall dissonance ($\mathsf{\Psi }$) [40]; note that since these values fall on the x-axis, they will be visible only in color). that are constants of 0.614738 and 0, respectively, the values of CI increasing from 0 to 0.00496 or 0.014654 depends on the values of x is 1.5 or 0.5, respectively. Note that the curve of CI asymmetric to $x=1$. The results show that the indicator SDR is superior to the two compared indicators in measuring the ordinal inconsistency of a PC matrix.

Koczkodaj and Szwarc analyzed another type of PC matrix, called FPC (the “full” PC matrix or the PC matrix full of x). The matrix FPC(x, n), with $x>1$, is defined as follows:

$$\mathrm{FPC}\left(x,n\right)=\left[\begin{array}{cc}\begin{array}{cc}1& x\\ x^{-1}& 1\end{array}& \begin{array}{cc}x& \begin{array}{cc}\cdots & x\end{array}\\ x& \begin{array}{cc}\cdots & x\end{array}\end{array}\\ \begin{array}{cc}⋮& ⋮\\ \begin{array}{c}{x}^{-1}\\ x^{-1}\end{array}& \begin{array}{c}⋮\\ x^{-1}\end{array}\end{array}& \begin{array}{cc}\ddots & \begin{array}{cc}⋮& ⋮\end{array}\\ \begin{array}{c}{x}^{-1}\\ \cdots \end{array}& \begin{array}{cc}\begin{array}{c}1\\ x^{-1}\end{array}& \begin{array}{c}x\\ 1\end{array}\end{array}\end{array}\end{array}\right]$$

Similarly, the only possibility of this matrix to be consistent is when $x=1$; however, unlike CPC(x, n), it has all erroneous triads. Figure 5 shows the values of CI, SDR, and Dissonance of the FPC(x, 5). Here, we also set x from 0.5 to 1.5 with an increment of 0.05.

The values of CI increasing from 0 to 0.0165 or 0.0487 depends on the values of x is 1.5 or 0.5, respectively. For indicator Dissonance, the values are zero, i.e., fall on the x-axis, with respect to x from 1 to 1.5, and are 0.03333 with respect to x from 0.95 to 0.5. All $\mathrm{SDR}\left[A\right]$, on the other hand, are 0.614738 except when $x=1,$ where $\mathrm{SDR}\left[A\right]=0$ as expected. From the results presented above, we can conclude that the indicator CI is not suitable for measuring ordinal inconsistency of a PC matrix, and that the proposed indicator SDR functions appropriately what its name implies; that is, it measures exactly what ordinal inconsistency means.

5. Conclusions

This study attempted to reinforce Saaty’s consistency test, as there is evidence that those so-called “acceptable” PC matrices may not be ordinally consistent. We propose an indicator called $\mathrm{SDR}\left[A\right]$ (standard deviation of ranks) to measure the ordinal inconsistency of a PC matrix A. The upper bound of $\mathrm{SDR}\left[A\right]$, $\sqrt{\frac{n\left(n+1\right)}{12}},$ is a function of size n of the PC matrix A. This property is important, because it enables us to find proper values of ordinal inconsistency for different PC matrix sizes. Then, we suggest a threshold for the decision-maker to decide whether the ordinal inconsistency of a PC matrix is acceptable. Moreover, we derive the range of the upper bound of normalized $\mathrm{SDR}\left[A\right]$ (namely, $\mathrm{NSDR}\left[A\right]$),$\sqrt{\frac{n+1}{12n}}$, of PC matrix $A$ with size of n. The range, from $\frac{1}{3}$. to $\frac{1}{\sqrt{12}},$. reveals a surprising fact that the degree of ordinal inconsistency of a small PC matrix may be more serious than a large one. We made three different comparative analyses with some other indicators. The results show that the indicator SDR measures exactly what ordinal inconsistency means and that the SDR is superior to the two compared indicators. In practice, we suggest decision-makers not only to follow Saaty’s rule of 0.1 but also to check the SDR.

Since we use only ordinal information instead of cardinal information of a PC matrix in computing the SDR, it is worth noting that the ordinal inconsistency of a PC matrix measured by using the indicator SDR is invariant under heterogeneous judgment measurements with a varied spectrum of scales, and that the proposed SDR not only works for multiplicative PC matrix but can also be used for additive and fuzzy PC matrices. Finally, since it is somewhat difficult to analyze and derive the upper bound of $\mathrm{SDR}\left[A\right]$. for a PC matrix $A$ that a tie between two entities is allowed, we leave it as a topic for future research. Moreover, how we can use this ordinal consistency indicator to improve the cardinal consistency of a PC matrix seems worth investigating.