Numerical Analysis of Consensus Measures within Groups

Abstract

Measuring the consensus for a group of ordinal-type responses is of practical importance in decision making. Many consensus measures appear in the literature, but they sometimes provide inconsistent results. Therefore, it is crucial to compare these consensus measures, and analyze their relationships. In this study, we targeted five consensus measures: ${\mathsf{\Phi }}_e$ (from entropy), ${\mathsf{\Phi }}_1$ (from absolute deviation), ${\mathsf{\Phi }}_2$ (from variance), ${\mathsf{\Phi }}_3$ (from skewness), and ${\mathsf{\Phi }}_{mv}$ (from conditional probability). We generated 316,251 probability distributions, and analyzed the relationships among their consensus values. Our results showed that ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2,$ and ${\mathsf{\Phi }}_3$ tended to provide consistent results, and the ordering ${\mathsf{\Phi }}_1\le {\mathsf{\Phi }}_e\le {\mathsf{\Phi }}_2\le {\mathsf{\Phi }}_3$ held at a high probability. Although ${\mathsf{\Phi }}_{mv}$ had a positive correlation with ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2,$ and ${\mathsf{\Phi }}_3$, it had a much lower tolerance for even a small proportion of extreme opposite opinions than ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2,$ and ${\mathsf{\Phi }}_3$ did.

1. Introduction

A consensus measure quantifies the consensus in ratings of a target. It provides fundamental implications of the group’s decision. For example, it can reveal whether the opinions of the group’s members are converging during a successive voting process [1], or whether averaging the members’ ratings to the group level is appropriate [2]. Because of its practicality, the problem of measuring consensus has received much attention, both in academic and applied research [3].

Many consensus measures appear in the literature. Most of them are derived from the deviation of individual ratings from the mean [3,4], while some are based on the extension of entropy [1], or the application of conditional probability [5]. Because consensus measures intend to quantify consensus, one tends to assume that similar conclusions can be drawn using different consensus measures. Although this assumption usually holds, it is still possible that a set of ratings which receives the lowest consensus score using one consensus measure may get a very high consensus score using another consensus measure (see Table 9). It is reasonable that using different consensus measures might lead to different conclusions because they are built on different theoretical concepts. For example, let A₁ and A₂ denote two sets of ratings collected at time t₁ and t₂, t₁ < t₂. Using one consensus measure might conclude that the consensus of A₁ is smaller than that of A₂ (i.e., the group members’ opinions are converging), but using another consensus measure might yield the opposite conclusion. Therefore, it is crucial to compare these consensus measures in more detail so that one can adequately interpret the meanings of the consensus values.

The objective of this study was to analyze the relationships among different consensus measures so that one can adequately utilize these consensus measures going forward. We first reviewed five consensus measures, and their properties. Then, we took a numerical analysis approach to comparing these consensus measures. This approach proceeded by generating a large number of possible rating distributions, and calculating their consensus scores using each consensus measure. Then, these consensus scores were analyzed to reveal the relationships among these consensus measures. Finally, we discussed how to interpret these consensus scores, and how to select a suitable consensus measure.

2. Review of Consensus Measures

2.1. Basic Properties of a Consensus Measure

In this paper, we assumed that a rating was an integer in $X=\left\{1, 2, \dots ,n\right\}$. For Likert-type scale responses, n = 5 or 7 is often used. Then, the ratings of all group members can be described as a probability distribution $p\left(x\right)$ over X. Let $p_i$ denote the probability $p\left(x=i\right)$ of getting a rating i. Then,

$$p_i\ge 0, \mathrm{for} i = 1 \mathrm{to} n,$$

(1)

$${{\displaystyle \sum }}_{i=1}^n{p}_i=1,$$

(2)

$$\mathrm{mean} m\left(p\right)={{\displaystyle \sum }}_{i=1}^{n}i{p}_i,$$

(3)

$$\mathrm{variance} V\left(p\right)={{\displaystyle \sum }}_{i=1}^n{p}_i{\left(i-m\left(p\right)\right)}^2.$$

(4)

Notably, the rating data are ordinal, and thus, calculating the mean or variance of $p\left(x\right)$ is inappropriate. However, mean, variance, or a combination of both was used intensively in the literature to design consensus measures for ordinal attributes.

Let $\mathsf{\Phi }$ denote a consensus measure, and $\mathsf{\Phi }\left(p\right)$. denote the consensus score of $p\left(x\right)$, based on $\mathsf{\Phi }$. It is common to restrict the range of $\mathsf{\Phi }\left(p\right)$ between zero and one. This restriction also facilitates comparing different consensus measures. Thus, $0\le \mathsf{\Phi }\left(p\right)\le 1$, and $\mathsf{\Phi }\left(p\right)=1$ and $\mathsf{\Phi }\left(p\right)=0$ indicate the maximum and minimum consensus scores, respectively [5]. In this paper, we divided the consensus measures into three categories, as described in the three subsections below.

2.2. Deviation-Based Consensus Measures

Deviation-based consensus measures use the absolute deviation of individual ratings from their mean to measure the consensus. They mainly differ in the power of the absolute deviation. In the literature, power = 1 or 2 was used to measure consensus. In this study, we extended the power to 3.

The average deviation ($AD$) [6] is the average difference between each rating and the mean, as shown in Equation (5). It is a measure of variability, and its range is between 0 and $\frac{n-1}{2}$, as proven in Corollary 1. Based on $AD$, we can design a consensus measure ${\mathsf{\Phi }}_1\left(p\right)$ such that $0\le {\mathsf{\Phi }}_1\left(p\right)\le 1$ (see Definition 1).

$$AD\left(p\right)= {\displaystyle \sum }_{i=1}^n{p}_i\left|i-m\left(p\right)\right|.$$

(5)

Corollary 1.

Given a probability distribution$p\left(x\right)$over$X=\left\{1,2,...,n\right\}$,$0\le AD\left(p\right)\le \frac{n-1}{2}$holds.

Proof. 

See Appendix A. ☐

Definition 1.

Consensus measure${\mathsf{\Phi }}_1\left(p\right)=1-\frac{AD\left(p\right)}{\left(n-1\right)/2}$.

Similar to $AD$, variance ($V$) is also a measure of variability, and is defined as the average of the squared difference between each rating and the mean, as shown in Equation (4). Its range is between 0 and ${\left(\frac{n-1}{2}\right)}^2$, as proven in Corollary 2. Elzinga et al. [4] designed a consensus measure ${\mathsf{\Phi }}_2\left(p\right)$ based on $V$ (see Definition 2).

Corollary 2.

Given a probability distribution$p\left(x\right)$over$X=\left\{1,2,...,n\right\}$,$0\le V\left(p\right)\le {\left(\frac{n-1}{2}\right)}^2$holds.

Proof. 

See Appendix B. ☐

Definition 2.

Consensus measure${\mathsf{\Phi }}_2\left(p\right)=1-\frac{V\left(p\right)}{{\left(\left(n-1\right)/2\right)}^2}$[4].

Notably, $AD$ uses the absolute difference between each rating and the mean, while variance uses the squared difference between each rating and the mean. We can raise the power of the absolute difference to three, and design a new consensus measure ${\mathsf{\Phi }}_3\left(p\right)$ as follows: let $S$ denote the average of the cubed absolute difference between each rating and the mean, as shown in Equation (6). The range of $S$ is between 0 and ${\left(\frac{n-1}{2}\right)}^3$, as proven in Corollary 3. A consensus measure ${\mathsf{\Phi }}_3\left(p\right)$ based on $S$ is shown in Definition 3.

$$S\left(p\right)= {{\displaystyle \sum }}_{i=1}^n{p}_i{\left|i-m\right|}^3.$$

(6)

Corollary 3.

Given a probability distribution$p\left(x\right)$over$X=\left\{1, 2, \dots , n\right\}$,$0\le S\left(p\right)\le {\left(\frac{n-1}{2}\right)}^3$holds.

Proof. 

See Appendix C. ☐

Definition 3.

Consensus measure${\mathsf{\Phi }}_3\left(p\right)=1-\frac{S\left(p\right)}{{\left(\left(n-1\right)/2\right)}^3}$.

The maximum values of ${\mathsf{\Phi }}_1\left(p\right)$, ${\mathsf{\Phi }}_2\left(p\right)$, and ${\mathsf{\Phi }}_3\left(p\right)$ all occur when $p_k=1$ for some $k\in X$ and $p_{i\in X\backslash \left\{k\right\}}=0$. The minimum values of ${\mathsf{\Phi }}_1\left(p\right)$, ${\mathsf{\Phi }}_2\left(p\right)$, and ${\mathsf{\Phi }}_3\left(p\right)$ all occur when $p_1=p_n=0.5$, and $p_{i\in X\backslash \left\{1,n\right\}}=0$. Please see the proofs of Corollaries 1, 2, and 3 in Appendix A, Appendix B, and Appendix C, respectively, for details.

Essentially, in ${\mathsf{\Phi }}_1\left(p\right)$, ${\mathsf{\Phi }}_2\left(p\right)$, and ${\mathsf{\Phi }}_3\left(p\right)$, raising the power of the absolute deviation increases the impact of those ratings further from the mean. An example is given below.

Example 1.

Given a probability distribution$p\left(x\right)$over$X=\left\{1, 2, 3, 4, 5\right\}$where$p_{i\in X}=0.2$, a (less consensus) probability distribution$q\left(x\right)$with more probabilities further from the mean is generated from$p\left(x\right)$by shifting 0.05 probability at$x=4$to$x=5$, i.e.,$q_1=q_2=q_3=0.2$,$q_4=0.15$, and$q_5=0.25$.

Table 1. From $p\left(x\right)$ to $q\left(x\right)$, consensus score reduces the most in ${\mathsf{\Phi }}_3$, less in ${\mathsf{\Phi }}_2$, and least in ${\mathsf{\Phi }}_1$.

	$\mathit{A}\mathit{D}$	$\mathit{V}$	$\mathit{S}$	${\mathsf{\Phi }}_1$	${\mathsf{\Phi }}_2$	${\mathsf{\Phi }}_3$
$p\left(x\right)$	1.2	2	3.6	0.4	0.5	0.55
$q\left(x\right)$	1.26	2.1475	3.9369	0.37	0.463125	0.507888
$\mathsf{\Phi }\left(p\right)-\mathsf{\Phi }\left(q\right)$	-	-	-	0.03	0.03688	0.04211

shows$AD, V, S$,${\mathsf{\Phi }}_1$,${\mathsf{\Phi }}_2$, and${\mathsf{\Phi }}_3$of$p\left(x\right)$and$q\left(x\right)$. The last row of Table 1 indicates that from$p$to$q$, the consensus is reduced by 0.03 with${\mathsf{\Phi }}_1$, 0.03688 with${\mathsf{\Phi }}_2$, and 0.04211 with${\mathsf{\Phi }}_3$. That is, the impact of increasing the probability further from the mean is greatest in${\mathsf{\Phi }}_3$, less in${\mathsf{\Phi }}_2$, and least in${\mathsf{\Phi }}_1$.

2.3. Conditional-Probability-Based Consensus Measure

Corollary 2 shows that the range of variance $V$ is between 0 and ${\left(\frac{n-1}{2}\right)}^2$, and the consensus measure ${\mathsf{\Phi }}_2$ is constructed based on this range. However, the range of $V$ is a function of the mean $m$. Specifically, for a given value of $m$, the range of $V$ is between $\left(m-⎣m⎦\right)\left(⎣m⎦+1-m\right)$ and $\left(m-1\right)\left(n-m\right)$, where $⎣m⎦$ is the greatest integer $\le n$. The size of this range is small as the value of $m$ approaches either end of the interval $\left[1,n\right]$, and is large as the value of $m$ approaches the center of the interval $\left[1,n\right]$. Thus, Akiyama et al. [5] proposed a new consensus measure via the conditional probability $p(V|m)$. Because this consensus measure is calculated using both $m$ and $V$, we denoted it as ${\mathsf{\Phi }}_{mv}\left(p\right)$ in this paper. Figure 1 shows the steps to calculate ${\mathsf{\Phi }}_{mv}\left(p\right)$ for a probability distribution $p\left(x\right)$ over $X=\left\{1, 2, 3, 4, 5\right\}$.

Table 2. Some examples of the probability distribution $p\left(x\right)$ satisfying ${\mathsf{\Phi }}_{mv}\left(p\right)=1$ or 0.

${\mathit{p}}_1$	${\mathit{p}}_2$	${\mathit{p}}_3$	${\mathit{p}}_4$	${\mathit{p}}_5$	${\mathsf{\Phi }}_{\mathit{m}\mathit{v}}\left(\mathit{p}\right)$
1	0	0	0	0	1
0.75	0.25	0	0	0	1
0.50	0.50	0	0	0	1
0	0.96	0.40	0	0	1
0.50	0	0	0	0.50	0
0.90	0	0	0	0.10	0
0.96	0	0	0	0.04	0
0.98	0	0	0	0.02	0

shows some examples of the probability distribution $p\left(x\right)$ with ${\mathsf{\Phi }}_{mv}\left(p\right)=1$ or 0. Unlike ${\mathsf{\Phi }}_1$, ${\mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$, ${\mathsf{\Phi }}_{mv}\left(p\right)=1$ not only occurs when $p_k=1$ for some $k\in X$, and $p_{i\in X\backslash \left\{k\right\}}=0$, but also occurs in many other cases. The first four examples in Table 2 show that the maximum value of ${\mathsf{\Phi }}_{mv}\left(p\right)$ occurs when all probabilities are distributed on one side, and none on the other side of $x$. Similarly, ${\mathsf{\Phi }}_{mv}\left(p\right)=0$ not only happens when $p_1=p_n=0.5$, and $p_{i\in X\backslash \left\{1,n\right\}}=0$, but also occurs in many other cases. The last three examples in Table 2 show that a small proportion of extreme opposite opinions can drag ${\mathsf{\Phi }}_{mv}\left(p\right)$ to zero.

2.4. Entropy-Based Consensus Measure

In the literature, the Shannon entropy equation and its extensions were used to quantify the diversity of a probability distribution [7]. Given a probability distribution $p\left(x\right)$, the Shannon entropy of $p\left(x\right)$ is $-{\displaystyle {{\displaystyle \sum }}_{i=1}^n}{p}_i\ln \left(p_i\right)$ where $n$ is the number of possible values of $x$, and $p_i$ denotes the probability of $x=i$. Because diversity appears to be the opposite concept of consensus, and the range of the Shannon entropy is between 0 and $\ln \left(n\right)$, a consensus measure between 0 and 1 based on the Shannon entropy equation can be defined as follows [1,8]:

$$\mathsf{\Phi }=1+\frac{{{\displaystyle \sum }}_{i=1}^n{p}_i\ln \left(p_i\right)}{\ln \left(n\right)}.$$

(7)

Notably, the Shannon entropy equation treats the variable $x$ as a nominal variable, and not as an ordinal variable; thus, the Shannon entropy equation and Equation (7) are inappropriate for quantifying the consensus of ordinal data, such as Likert-type scale responses. To resolve this problem, Tastle and Wierman [1,8] extended the Shannon entropy equation to define a new consensus measure, denoted as ${\mathsf{\Phi }}_e$ in this paper, as follows:

$${\mathsf{\Phi }}_e=1+{{\displaystyle \sum }}_{i=1}^n{p}_i{\log }_2\left(1-\frac{\left|i-m\right|}{n-1}\right),$$

(8)

where $m$ is the mean of $p\left(x\right)$, as defined in Equation (3). Similar to ${\mathsf{\Phi }}_1\left(p\right)$, ${\mathsf{\Phi }}_2\left(p\right)$, and ${\mathsf{\Phi }}_3\left(p\right)$, the maximum value of ${\mathsf{\Phi }}_e\left(p\right)$ only occurs when $p_k=1$ for some $k\in X$, and $p_{i\in X\backslash \left\{k\right\}}=0$; the minimum value of ${\mathsf{\Phi }}_e\left(p\right)$ only occurs when $p_1=p_n=0.5$, and $p_{i\in X\backslash \left\{1,n\right\}}=0$.

3. Experimental Study

3.1. Experiment Setup

Given a probability distribution, the five consensus measures reviewed in Section 2 often yielded different consensus scores, and sometimes the differences among these scores were substantial, and led to opposite conclusions. This phenomenon makes it difficult to interpret the meaning of these scores. In this study, we performed a numerical experiment to analyze the relationships among these five consensus measures.

This experiment used the probability distribution $p\left(x\right)$ over $X=\left\{1, 2, 3, 4, 5\right\}$, which is common for Likert-type scale data. Specifically, we wrote a small computer program containing a five-level for loop to generate 316,251 probability distributions, where the i-th level of the for loop changed the value of $p_i$ from 0 to 1 with a step size of 0.2, and cases not satisfying ${\displaystyle {{\displaystyle \sum }}_{i=1}^5}{p}_i=1$ were skipped. Thus, these 316,251 probability distributions covered all of the possible probability distributions of $p\left(x\right)$ satisfying $p_i\in \left\{0, 0.2, 0.4,\dots , 0.98, 1\right\}$ for $i$ = 1 to 5, and ${{\displaystyle \sum }}_{i=1}^5{p}_i=1$. Then, the consensus scores of each generated probability distribution were calculated and compared to study the relationships among the five consensus measures.

Table 3. The distribution of the mean values of the 316,251 generated probability distributions.

Range of Mean	$1\le \mathit{m}\le 2$	$2<\mathit{m}\le 3$	$3<\mathit{m}\le 4$	$4<\mathit{m}\le 5$
Number of probability distributions	16,390	143,747	140,878	15,236
Probability	5.18%	45.45%	44.55%	4.82%

shows the distribution of the mean values of the 316,251 probability distributions. Most of the generated probability distributions had mean values between 2 and 4.

3.2. Correlation

Table 4. Kendall rank correlation coefficients between consensus measures using all 316,251 probability distributions.

	${\mathsf{\Phi }}_1$	${\mathsf{\Phi }}_{\mathit{e}}$	${\mathsf{\Phi }}_2$	${\mathsf{\Phi }}_3$	${\mathsf{\Phi }}_{\mathit{m}\mathit{v}}$
${\mathsf{\Phi }}_1$	1	0.990202	0.967755	0.887252	0.925708
${\mathsf{\Phi }}_e$	0.990202	1	0.99008	0.940635	0.964478
${\mathsf{\Phi }}_2$	0.967755	0.99008	1	0.969419	0.970876
${\mathsf{\Phi }}_3$	0.887252	0.940635	0.969419	1	0.974605
${\mathsf{\Phi }}_{mv}$	0.925708	0.964478	0.970876	0.974605	1

shows the Kendall rank correlation coefficients between any two consensus measures. As expected, the results reflected higher than 0.887 correlation between any two consensus measures. That is, if a probability distribution A is ranked higher than another probability distribution B based on one consensus measure, it is very likely that A is also ranked higher than B based on another consensus measure. Let $\mathsf{\tau }\left({\mathsf{\Phi }}_i,{ \mathsf{\Phi }}_j\right)$ denote the Kendall rank correlation coefficient between ${\mathsf{\Phi }}_i$ and ${\mathsf{\Phi }}_j$. According to Table 4, the lowest correlation occurred at $\mathsf{\tau }\left({\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_3\right)$, and the highest occurs at $\mathsf{\tau }\left({\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e\right)$. Specifically, $\mathsf{\tau }\left({\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e\right)>\mathsf{\tau }\left({\mathsf{\Phi }}_2,{ \mathsf{\Phi }}_e\right)>\mathsf{\tau }\left({\mathsf{\Phi }}_3,{ \mathsf{\Phi }}_{mv}\right)>\mathsf{\tau }\left({\mathsf{\Phi }}_2,{ \mathsf{\Phi }}_{mv}\right)>\mathsf{\tau }\left({\mathsf{\Phi }}_2,{ \mathsf{\Phi }}_3\right)>\mathsf{\tau }\left({\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_2\right)>\mathsf{\tau }\left({\mathsf{\Phi }}_e,{ \mathsf{\Phi }}_{mv}\right)>\mathsf{\tau }\left({\mathsf{\Phi }}_e,{ \mathsf{\Phi }}_3\right)>\mathsf{\tau }\left({\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_{mv}\right)>\mathsf{\tau }\left({\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_3\right)$.

According to Table 3, only 5.18% and 4.82% of the 316,251 generated probability distributions had their mean values in the intervals [1, 2] and (4, 5], respectively. To check whether high correlation still existed for probability distributions with small or large mean values, we calculated the Kendall rank correlation coefficients using both subsets of probability distributions, and the results are shown in

Table 5. Kendall rank correlation coefficients between consensus measures using the 16,390 probability distributions where $1\le m\le 2$.

	${\mathsf{\Phi }}_1$	${\mathsf{\Phi }}_{\mathit{e}}$	${\mathsf{\Phi }}_2$	${\mathsf{\Phi }}_3$	${\mathsf{\Phi }}_{\mathit{m}\mathit{v}}$
${\mathsf{\Phi }}_1$	1	0.967110	0.930117	0.774093	0.785614
${\mathsf{\Phi }}_e$	0.967110	1	0.985489	0.904147	0.891701
${\mathsf{\Phi }}_2$	0.930117	0.985489	1	0.942186	0.900688
${\mathsf{\Phi }}_3$	0.774093	0.904147	0.942186	1	0.940492
${\mathsf{\Phi }}_{mv}$	0.785614	0.891701	0.900688	0.940492	1

and

Table 6. Kendall rank correlation coefficients between consensus measures using the 15,236 probability distributions where $4<m\le 5$.

	${\mathsf{\Phi }}_1$	${\mathsf{\Phi }}_{\mathit{e}}$	${\mathsf{\Phi }}_2$	${\mathsf{\Phi }}_3$	${\mathsf{\Phi }}_{\mathit{m}\mathit{v}}$
${\mathsf{\Phi }}_1$	1	0.965686	0.930352	0.772132	0.776873
${\mathsf{\Phi }}_e$	0.965686	1	0.986604	0.905085	0.886878
${\mathsf{\Phi }}_2$	0.930352	0.986604	1	0.941809	0.897223
${\mathsf{\Phi }}_3$	0.772132	0.905085	0.941809	1	0.939574
${\mathsf{\Phi }}_{mv}$	0.776873	0.886878	0.897223	0.939574	1

. Every value in Table 5 and Table 6 was smaller than its corresponding value in Table 4. Particularly, $\mathsf{\tau }\left({\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_3\right)$ dropped from 0.887252 in Table 4 to 0.774093 in Table 5, and 0.772132 in Table 6; $\mathsf{\tau }\left({\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_{mv}\right)$ dropped from 0.925708 in Table 4 to 0.785614 in Table 5, and 0.776873 in Table 6.

3.3. Range of Difference

Although Table 4 shows that a positive correlation existed between any two consensus measures of the 316,251 generated probability distributions, some of the generated probability distributions did not follow this general trend. In this section, we calculated the range of differences between two consensus measures to show that this difference was usually small, but was sometimes very big.

Table 7. Mean differences between any two consensus measures.

	${\mathsf{\Phi }}_1$	${\mathsf{\Phi }}_{\mathit{e}}$	${\mathsf{\Phi }}_2$	${\mathsf{\Phi }}_3$	${\mathsf{\Phi }}_{\mathit{m}\mathit{v}}$
${\mathsf{\Phi }}_1$	0	0.0381011	0.1258246	0.16606489	0.15698299
${\mathsf{\Phi }}_e$	0.0381011	0	0.0895491	0.1281988	0.1429986
${\mathsf{\Phi }}_2$	0.1258246	0.0895491	0	0.0491733	0.14278058
${\mathsf{\Phi }}_3$	0.16606489	0.1281988	0.0491733	0	0.149693
${\mathsf{\Phi }}_{mv}$	0.15698299	0.1429986	0.14278058	0.149693	0

shows the mean differences between any two consensus measures of the 316,251 generated probability distributions. All of the mean differences were small (<0.167), where the largest mean difference occurred between ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_3$, and the smallest mean difference occurred between ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_e$. The results were consistent with Table 4, where the smallest and the largest correlation coefficients were $\mathrm{R}\left({\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_3\right)$ and $\mathrm{R}\left({\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e\right)$, respectively.

Table 8. Maximum differences between two consensus measures.

	${\mathsf{\Phi }}_1$	${\mathsf{\Phi }}_{\mathit{e}}$	${\mathsf{\Phi }}_2$	${\mathsf{\Phi }}_3$	${\mathsf{\Phi }}_{\mathit{m}\mathit{v}}$
${\mathsf{\Phi }}_1$	0	0.108996	0.25	0.375	0.9216
${\mathsf{\Phi }}_e$	0.108996	0	0.165037	0.290037	0.858559
${\mathsf{\Phi }}_2$	0.25	0.165037	0	0.249661	0.9216
${\mathsf{\Phi }}_3$	0.375	0.290037	0.249661	0	0.849347
${\mathsf{\Phi }}_{mv}$	0.9216	0.858559	0.9216	0.849347	0

shows the maximum difference between any two consensus measures of the 316,251 generated probability distributions. Some of the maximum differences were very large. For example, the maximum difference between ${\mathsf{\Phi }}_{mv}$ and other consensus measures was larger than 0.84. Notably, all of the correlation coefficients between ${\mathsf{\Phi }}_{mv}$ and the other consensus measures were greater than 0.92 (see Table 4), and the mean difference between ${\mathsf{\Phi }}_{mv}$ and the other consensus measures was less than 0.16 (see Table 7). Thus, it is reasonable to infer that, although for most probability distributions, the difference between ${\mathsf{\Phi }}_{mv}$ and the other consensus measures was not large, but for some probability distributions, this difference could be huge. Therefore, it is important to understand for which kinds of probability distributions does such a big difference between various consensus measures occur.

The first four examples in

Table 9. Some examples of the probability distribution $p\left(x\right)$, and their consensus scores.

Example Number	${\mathit{p}}_1$	${\mathit{p}}_2$	${\mathit{p}}_3$	${\mathit{p}}_4$	${\mathit{p}}_5$	${\mathsf{\Phi }}_1$	${\mathsf{\Phi }}_{\mathit{e}}$	${\mathsf{\Phi }}_2$	${\mathsf{\Phi }}_3$	${\mathsf{\Phi }}_{\mathit{m}\mathit{v}}$
1	0.98	0	0	0	0.02	0.9216	0.858559	0.9216	0.849347	0
2	0.90	0	0	0	0.10	0.64	0.531004	0.64	0.4096	0
3	0.86	0	0	0	0.14	0.5184	0.415761	0.5184	0.268739	0
4	0	0	0.50	0	0.50	0.5	0.584963	0.75	0.875	0.833333
5	0.02	0	0	0.16	0.82	0.8032	0.796982	0.8944	0.85691	0.833333
6	0.98	0	0.02	0	0	0.9608	0.966392	0.9804	0.981168	0.833333
7	0.98	0	0	0.02	0	0.9412	0.940313	0.9559	0.936443	0.166667
8	0.	0.96	0	0	0.04	0.8848	0.884354	0.9136	0.880353	0.99176

show some of the generated probability distributions where the maximum differences between two consensus measures occurred. Example 1 had a large proportion (98%) of probability at $x=1$, thus rendering high consensus scores using ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e,{\mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$. However, this large proportion of probability at $x=1$ also made values of $m$ close to 1, where $m$ was the mean of the probability distribution. As discussed in Section 2.3, the range of variance is small when $m$ approaches either end of the interval [0, 1]. Thus, for values of $m$ close to 1, the range of variance was small, making ${\mathsf{\Phi }}_{mv}$ very sensitive to even a small proportion of probability at the opposite end of $x$ (2% at $x=5$ in this example). As a result, Example 1 yielded ${\mathsf{\Phi }}_{mv}=0$. This example was also one of the probability distributions among the 316,251 generated probability distributions that had the maximum difference (in Table 8) between ${\mathsf{\Phi }}_{mv}$ and other consensus measures.

Examples 2 and 3 in Table 9 were similar to Example 1, where a large proportion of probability occurred at $x=1$, and a small proportion of probability occurred at $x=5$. The values of ${\mathsf{\Phi }}_{mv}$ remained 0 for Examples 2 and 3. However, the difference between $p_1$ and $p_5$ decreased from Example 1 through to Example 3, making ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e,{\mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$ smaller for Examples 2 and 3 than for Example 1. Notably, Example 2 was one of the probability distributions that had the maximum difference (in Table 8) between ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_e$; Example 3 was one of the probability distributions that had the maximum difference between ${\mathsf{\Phi }}_2$ and ${\mathsf{\Phi }}_3$.

Example 4 had $p_3=p_5=0.5$, and yielded the maximum difference (in Table 8) between ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_2$, between ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_3$, between ${\mathsf{\Phi }}_e$ and ${\mathsf{\Phi }}_2$, and between ${\mathsf{\Phi }}_e$ and ${\mathsf{\Phi }}_3$. Suppose that the first four examples in Table 9 describe the voting results at four different stages during a successive voting process. From Example 1 through to Example 4, the value of ${\mathsf{\Phi }}_1$ decreased, indicating the group’s consensus was diverging. However, using ${\mathsf{\Phi }}_{mv}$ concluded the opposite. For ${\mathsf{\Phi }}_e$, ${\mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$, the consensus first decreased (from Example 1 through to Example 3), and then increased (from Example 4 onward). However, the differences between the consensus values in Examples 1 and 4 were 0.273596 with ${\mathsf{\Phi }}_e$, 0.1716 with ${\mathsf{\Phi }}_2$, and −0.02565 with ${\mathsf{\Phi }}_3$. Thus, using different consensus measures could lead to different conclusions.

A small change in the probability distribution could result in a different impact on different consensus measures. Consider Examples 1, 7, and 6. They differed by moving a small proportion (2%) of probability from $x=5$, to $x=4$, and to $x=3$, respectively. Although they were similar probability distributions, the value of ${\mathsf{\Phi }}_{mv}$ was 0 in Example 1, and gradually increased to 0.166667 in Example 7, but quickly increased to 0.833333 in Example 6. However, the values of ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e,{\mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$ did not change much among these three examples. Notably, the proportion of probabilities further from the mean had a greater negative impact on ${\mathsf{\Phi }}_3$, than on ${\mathsf{\Phi }}_2$ and ${\mathsf{\Phi }}_1$. Thus, by moving 2% of probability from $x=5$ to $x=4$ (i.e., moving closer to the mean), the ordering of ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$ changed from ${\mathsf{\Phi }}_3<{\mathsf{\Phi }}_2={\mathsf{\Phi }}_1$ in Example 1 to ${\mathsf{\Phi }}_3<{\mathsf{\Phi }}_1<{\mathsf{\Phi }}_2$ in Example 7. Then, by moving 2% of probability from $x=4$ to $x=5$, the ordering of ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$ changed to ${\mathsf{\Phi }}_1<{\mathsf{\Phi }}_2<{\mathsf{\Phi }}_3$ in Example 6.

The ordering of the values of these consensus measures depended on the probability distribution. For Examples 4, 5, and 6, the value of ${\mathsf{\Phi }}_{mv}$ was the same, but ${\mathsf{\Phi }}_1<{\mathsf{\Phi }}_e<{\mathsf{\Phi }}_2<{\mathsf{\Phi }}_{mv}<{\mathsf{\Phi }}_3$ held in Example 4, ${\mathsf{\Phi }}_e<{\mathsf{\Phi }}_1<{\mathsf{\Phi }}_{mv}<{\mathsf{\Phi }}_3<{\mathsf{\Phi }}_2$ held in Example 5, and ${\mathsf{\Phi }}_{mv}<{\mathsf{\Phi }}_1<{\mathsf{\Phi }}_e<{\mathsf{\Phi }}_2<{\mathsf{\Phi }}_3$ held in Example 6. In Example 7, ${\mathsf{\Phi }}_{mv}$ was the smallest among all consensus measures; however, in Example 8, ${\mathsf{\Phi }}_{mv}$ was the greatest.

3.4. Ordering

From the examples in Table 9, it appeared that no fixed ordering existed among the consensus scores calculated using different consensus measures. Figure 2 shows the distributions of consensus scores of the 316,251 probability distributions generated in this experiment. The distributions of consensus scores based on ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2,$ and ${\mathsf{\Phi }}_3$ were similar, but were very different from the distribution of consensus scores based on ${\mathsf{\Phi }}_{mv}$. For the consensus values close to 1, the ordering of the probabilities among ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2,$ and ${\mathsf{\Phi }}_3$ was ${\mathsf{\Phi }}_1<{\mathsf{\Phi }}_e<{\mathsf{\Phi }}_2<{\mathsf{\Phi }}_3$, but for the consensus values close to 0, the ordering of the probabilities became ${\mathsf{\Phi }}_1\ge {\mathsf{\Phi }}_e\ge {\mathsf{\Phi }}_2\ge {\mathsf{\Phi }}_3$.

In

Table 10. The probability of scores based on one consensus measure to be equal to or less than scores based on another consensus measure for the 316,251 generated probability distributions.

$\le$	${\mathsf{\Phi }}_{\mathit{e}}$	${\mathsf{\Phi }}_2$	${\mathsf{\Phi }}_3$	${\mathsf{\Phi }}_{\mathit{m}\mathit{v}}$
${\mathsf{\Phi }}_1$	94.66%	100%	96.41%	58.12%
${\mathsf{\Phi }}_e$	-	100%	96.96%	52.04%
${\mathsf{\Phi }}_2$	-	-	84.35%	36.84%
${\mathsf{\Phi }}_3$	-	-	-	28.01%

, we compared the consensus scores of the 316,251 generated probability distributions, and calculated the probabilities of scores based on one consensus measure being less than or equal to scores based on another consensus measure. According to Table 10, ${\mathsf{\Phi }}_1\le {\mathsf{\Phi }}_2$ and ${\mathsf{\Phi }}_e\le {\mathsf{\Phi }}_2$ always held, while ${\mathsf{\Phi }}_2\le {\mathsf{\Phi }}_3$, ${\mathsf{\Phi }}_e\le {\mathsf{\Phi }}_3$, ${\mathsf{\Phi }}_1\le {\mathsf{\Phi }}_3$, and ${\mathsf{\Phi }}_1\le {\mathsf{\Phi }}_e$ also held at very high probabilities. Thus, ${\mathsf{\Phi }}_1\le {\mathsf{\Phi }}_e\le {\mathsf{\Phi }}_2\le {\mathsf{\Phi }}_3$ was the most probable ordering among the scores based on these four consensus measures. The orderings between ${\mathsf{\Phi }}_{mv}$, and ${\mathsf{\Phi }}_1$ or ${\mathsf{\Phi }}_e$ were not apparent, where ${\mathsf{\Phi }}_1\le {\mathsf{\Phi }}_{mv}$ and ${\mathsf{\Phi }}_e\le {\mathsf{\Phi }}_{mv}$ only held at 58.12% and 52.04% probabilities, respectively. Finally, ${\mathsf{\Phi }}_2>{\mathsf{\Phi }}_{mv}$ and ${\mathsf{\Phi }}_3>{\mathsf{\Phi }}_{mv}$ were likely to occur because ${\mathsf{\Phi }}_2\le {\mathsf{\Phi }}_{mv}$ and ${\mathsf{\Phi }}_3\le {\mathsf{\Phi }}_{mv}$ held at 36.84% and 28.01% probabilities, respectively.

3.5. Relationships

To visually inspect the relationships among different consensus measures, we plotted the consensus values of the 316,251 generated probability distributions in two-dimensional (2D) scatter charts.

Figure 3 shows the scatter charts of ${\mathsf{\Phi }}_1$ scores versus scores based on the other consensus measures, where the red dashed lines represent equality between two consensus scores. As expected, a positively correlated trend existed. No fixed ordering existed between ${\mathsf{\Phi }}_1$ and the other consensus measures except that ${\mathsf{\Phi }}_1\le {\mathsf{\Phi }}_2$ always held, as shown in Figure 3b. According to Figure 3a–c, as the value of ${\mathsf{\Phi }}_1$ approached 0 or 1, the ranges of ${\mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2$ and ${\mathsf{\Phi }}_3$ narrowed, indicating that the maximum differences between ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$ decreased. However, when the value of ${\mathsf{\Phi }}_1$ approached 0.5, the ranges of ${\mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$ increased, indicating that the maximum differences between ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$ also increased. Furthermore, the maximum difference between ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_e$ was smaller than both the maximum differences between ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_2$, and between ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_3$.

Figure 3d shows that, for ${\mathsf{\Phi }}_1<1$, as the value of ${\mathsf{\Phi }}_1$ increased, the range of ${\mathsf{\Phi }}_{mv}$ increased, and the maximum difference between ${\mathsf{\Phi }}_1$ and ${\mathsf{\Phi }}_{mv}$ became huge. For any probability distribution satisfying ${\mathsf{\Phi }}_1=1$, its ${\mathsf{\Phi }}_{mv}$ was also 1. However, for any probability distribution satisfying ${\mathsf{\Phi }}_{mv}=1$, its value of ${\mathsf{\Phi }}_1$ was not necessarily 1. In fact, there were only $n$ probability distributions satisfying ${\mathsf{\Phi }}_1=1$, that is, when $p_k=1$ for some $k\in X$, and $p_{i\in X\backslash \left\{k\right\}}=0$ (this statement also applies to ${\mathsf{\Phi }}_e, {\mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$). However, there were many probability distributions satisfying ${\mathsf{\Phi }}_{mv}=1$ (see Table 2 for examples).

Figure 4 shows the scatter charts of the consensus scores based on ${\mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2,{ \mathsf{\Phi }}_3$, and ${\mathsf{\Phi }}_{mv}$. No fixed ordering existed among these consensus measures except that ${\mathsf{\Phi }}_e\le {\mathsf{\Phi }}_2$ always held, as shown in Figure 4a. According to Figure 4a,b,d, for ${\mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$, as the value of one consensus measure approached either end of the interval [0, 1], the range of another consensus measure decreased. According to Figure 4a,b, the maximum difference between ${\mathsf{\Phi }}_e$ and ${\mathsf{\Phi }}_2$ was smaller than that between ${\mathsf{\Phi }}_e$ and ${\mathsf{\Phi }}_3$. According to Figure 3b and Figure 4a,d, the maximum difference between ${\mathsf{\Phi }}_2$ and ${\mathsf{\Phi }}_e$ was smaller than those between ${\mathsf{\Phi }}_2$ and ${\mathsf{\Phi }}_1$, and between ${\mathsf{\Phi }}_2$ and ${\mathsf{\Phi }}_3$. Figure 4c,e,f show a similar pattern to Figure 3d. As the value of ${\mathsf{\Phi }}_e$ (or ${\mathsf{\Phi }}_2$, ${\mathsf{\Phi }}_3$) increased (before reaching 1), the range of ${\mathsf{\Phi }}_{mv}$ increased, and the maximum difference between ${\mathsf{\Phi }}_e$ (or ${\mathsf{\Phi }}_2$ and ${\mathsf{\Phi }}_3$) and ${\mathsf{\Phi }}_{mv}$ became huge.

4. Discussions

Given a probability distribution, using different consensus measures often yields different consensus scores. If there exists a fixed ordering among these scores, then consistent results can be drawn using different consensus measures. Unfortunately, such an ordering depends on the given probability distribution. However, according to Table 10, the following orderings among the consensus scores held at high probabilities: ${\mathsf{\Phi }}_1\le {\mathsf{\Phi }}_e\le {\mathsf{\Phi }}_2\le {\mathsf{\Phi }}_3$, ${\mathsf{\Phi }}_2>{\mathsf{\Phi }}_{mv}$, and ${\mathsf{\Phi }}_3>{\mathsf{\Phi }}_{mv}$.

Because there exists no fixed ordering among consensus scores based on different consensus measures, it is crucial to know the relationships among the consensus measures. Figure 3 and Figure 4 revealed that, for ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2,$ and ${\mathsf{\Phi }}_3$, as the value of one consensus measure approached either end of the interval [0, 1], the ranges of the other consensus measures decreased. Thus, one can expect smaller differences among ${\mathsf{\Phi }}_e, {\mathsf{\Phi }}_1, {\mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$ for consensus scores close to 0 or 1, than for consensus scores close to 0.5.

According to Figure 3d and Figure 4c,e,f, the range of ${\mathsf{\Phi }}_{mv}$ increased rapidly as the value of ${\mathsf{\Phi }}_e, {\mathsf{\Phi }}_1, {\mathsf{\Phi }}_2$, or ${\mathsf{\Phi }}_3$ increased. Thus, ${\mathsf{\Phi }}_{mv}$ often gave results inconsistent with those from ${\mathsf{\Phi }}_e, {\mathsf{\Phi }}_1, {\mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$, especially when the value of ${\mathsf{\Phi }}_e, {\mathsf{\Phi }}_1, {\mathsf{\Phi }}_2$, or ${\mathsf{\Phi }}_3$ was large. Looking at these figures from another perspective, the ranges of ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2,$ and ${\mathsf{\Phi }}_3$ decreased as the value of ${\mathsf{\Phi }}_{mv}$ increased. Notably, ${\mathsf{\Phi }}_{mv}$ tended to give low scores to probability distributions where some probability was located at the opposite end of the mean. Thus, for values of ${\mathsf{\Phi }}_{mv}$ close to zero, one should also check the values of ${\mathsf{\Phi }}_1,{ \mathsf{\Phi }}_e,{ \mathsf{\Phi }}_2,$ and ${\mathsf{\Phi }}_3$ for possibly inconsistent results.

Choosing a consensus measure remains a task for the users. If one has a low tolerance for even a small proportion of extreme opposite opinions, then ${\mathsf{\Phi }}_{mv}$ is a good choice. Otherwise, the other consensus measures tend to provide consistent results. If one prefers to emphasize the opinions further from the mean, then ${\mathsf{\Phi }}_3$ is a good choice. Otherwise, either ${\mathsf{\Phi }}_1$ or ${\mathsf{\Phi }}_e$ can be used, both yielding similar results. Finally, ${\mathsf{\Phi }}_2$ provides a middle ground between ${\mathsf{\Phi }}_3$ and ${\mathsf{\Phi }}_1$.

5. Conclusions

An understanding of the characteristics of consensus measures helps users interpret results. For example, according to Figure 3b, ${\mathsf{\Phi }}_1$ tended to yield a smaller consensus score than ${\mathsf{\Phi }}_2$ for the same probability distribution; thus, a probability distribution $A$ with ${\mathsf{\Phi }}_1\left(A\right)=0.6$ might have more consensus than another probability distribution $B$ with ${\mathsf{\Phi }}_2\left(B\right)=0.7$, even though ${\mathsf{\Phi }}_1\left(A\right)<{\mathsf{\Phi }}_2\left(B\right)$.

In essence, two opposite forces shape the design of a consensus measure: the force of obeying the majority, and the force of respecting the minority. Consensus measure ${\mathsf{\Phi }}_e$ stressed on the former, and the opinion of the minority has a weak impact on the consensus scores. In contrast, ${\mathsf{\Phi }}_{mv}$ emphasizes the latter, and the opinion of the minority substantially influences the consensus scores, as shown in the first four examples in Table 9.

Deviation-based consensus measures (i.e., ${\mathsf{\Phi }}_1, {\mathsf{\Phi }}_2$, and ${\mathsf{\Phi }}_3$) allow users to adjust the strengths of these two forces. As described in Section 2.2, raising the power of the absolute deviation in the deviation-based consensus measures increases the impact of ratings further from the mean. Intuitively, unless the probabilities of all opinions are distributed evenly on opposite sides of the mean (e.g., $p_1$ = $p_n$ = 0.5), ratings further from the mean represent the opinions of the minority. Thus, going from ${\mathsf{\Phi }}_1$ through to ${\mathsf{\Phi }}_3$, the impact of the minority increases. Overall, fine-tuning the balance between the force of obeying the majority, and the force of respecting the minority in a consensus measure provides the consensus measure with more flexibility for various situations, and is a direction of research worth exploring.