Measures of Qualitative Variation in the Case of Maximum Entropy

Abstract

Asymptotic behavior of qualitative variation statistics, including entropy measures, can be modeled well by normal distributions. In this study, we test the normality of various qualitative variation measures in general. We find that almost all indices tend to normality as the sample size increases, and they are highly correlated. However, for all of these qualitative variation statistics, maximum uncertainty is a serious factor that prevents normality. Among these, we study the properties of two qualitative variation statistics; VarNC and StDev statistics in the case of maximum uncertainty, since these two statistics show lower sampling variability and utilize all sample information. We derive probability distribution functions of these statistics and prove that they are consistent. We also discuss the relationship between VarNC and the normalized form of Tsallis (α = 2) entropy in the case of maximum uncertainty.

1. Introduction

Whenever the scale of measurement is nominal or ordinal, the classical measures of dispersion, like standard deviation and variance, cannot be used. In such cases, the only way to measure dispersion is to use measures, which involve frequencies of random observations. Wilcox [1] made the first attempt to gather some of the qualitative variation indices together pointing out the utility of these measures for statistical handling of qualitative data. One of the rare attempts of deriving probability functions for qualitative measures can be seen in Swanson [2].

Qualitative measures are widely used in social and biological sciences. Appropriate for the area of application, qualitative variation indices are preferred compared to diversity indices, or vice versa. A diversity index is a quantitative measure that accounts for the number of categories in a dataset. Measures of diversity are distinguished from measures of variation such that the former refers to counting numbers of discrete types [3]. Diversity is the antonym of concentration, whilst a near synonym of variety. The term concentration is more common in some areas of ecology, and economics. Diversity is more likely in sociology and communication [4]. For example, in economics, concentration is a measure of competitiveness in a market. The more concentrated the market, the less competitive it will be. Heip et al. [5] introduced a list of diversity and evenness indices, most of which could also be seen as qualitative variation indices. The same parallelism between qualitative variation indices and diversity indices can be found in [6,7,8,9,10] as well.

Although there are some differences in explanations of these measures, some mathematical analogies between them are straightforward for some inferential purposes. For this reason it is not surprising to find a concentration measure, which was originally proposed for measuring diversity before it was reversed (a measure is “reversed” by taking its reciprocal, or subtracting it from its maximum value, etc.). For example, Simpson’s D statistic is a measure of diversity, and “the reversed form” of it gives the Gini concentration index, which is a special case of Tsallis entropy. The Herfindahl-Hirschmann concentration index is obtained by subtracting Simpson’s D from 1.

Statistical entropy is a measure of uncertainty of an experiment as well as being a measure of qualitative variation. Once the experiment has been carried out, uncertainty is not present [11]. Thus, it can also be evaluated as the measure of information; one can get through sampling, or ignorance, before experimentation [12]. Jaynes [13] proposes that the maximizing of Shannon’s entropy provides the most appropriate interpretation for the amount of uncertainty. The principle of maximum entropy also coincides with Laplace’s well–known principle of insufficient reasoning. Pardo [14] and Esteban and Morales [15] provide theoretical background for different entropy measures.

In our study, we first present the three most common entropy measures, namely Shannon, Rényi, and Tsallis entropies with their variances, and then we discuss their asymptotic behaviour. Moreover, we list various qualitative variation indices as axiomatized by Wilcox [1], including Shannon and Tsallis entropies, and VarNC and StDev statistics. By simulations we check the normality of these measures for various entropy assumptions. We observe that maximum entropy is a serious factor, which prevents normality. We formulate the probability density functions of VarNC and StDev statistics and the first two moments under the assumption of maximum uncertainty. We also show that VarNC is special case of normalized Tsallis (α = 2) entropy under the same assumptions. We discuss the relationship between qualitative variation indices and power divergence statistics since entropy measures the divergence of a distribution from maximum uncertainty.

2. Entropy and Qualitative Variation

2.1. Common Entropy Measures

There are various entropy measures formulated by various authors in literature. Most commonly used ones are Shannon, Rényi, and Tsallis entropies. We give basic properties of these three measures below.

2.1.1. Shannon Entropy

In his study on mathematical theory for communication, Shannon [16] developed a measure of uncertainty or entropy, which was later named as “Shannon entropy”. If the discrete random variable X takes on the values $x_1,x_2,\dots ,x_K$ with respective probabilities $p_1,p_2,\dots ,p_K$, Shannon entropy is defined as:

$$\mathrm{H}=-{\displaystyle \sum }_{i=1}^K{p}_i\log p_i.$$

(1)

In case of maximum uncertainty (i.e., the case in which all probabilities are equal), this becomes ${\mathrm{H}}_{\mathrm{s}}=\log K$. The upper limit of Shannon entropy depends on the number of categories K. The estimator of Shannon entropy, $\widehat{H}$, is calculated by using sample information as:

$$\widehat{H}=-{\displaystyle \sum }_{i=1}^K{\widehat{p}}_i\log {\widehat{p}}_i$$

(2)

where probabilities ${\widehat{p}}_i$ are estimated by maximum likelihood method. Although this estimator is biased, the amount of bias can be reduced by increasing the sample size [17]. Zhang Xing [18] gives the variance of Shannon’s entropy with sample size n as follows:

$$\mathrm{Var}\left(\widehat{H}\right)=\frac{1}{n}\left({\displaystyle \sum }_{i=1}^K{p}_i{\ln }^2{p}_i-{\mathrm{H}}^2\right)+\frac{K-1}{2n^2}+\mathrm{O}\left(n^{-3}\right).$$

(3)

2.1.2. Rényi Entropy

Rényi entropy [19] is defined as:

$${\mathrm{H}}_{\mathrm{R}}=\frac{\log {{\displaystyle \sum }}_{i=1}^K{p}_i^{\alpha }}{1-\alpha }\mathrm{for}\alpha >0\mathrm{and}\alpha \ne 1.$$

(4)

Shannon entropy is a special case of Rényi entropy for $\alpha \to 1$. The variance of Rényi entropy estimator can be approximated by:

$$\mathrm{Var}\left({\widehat{H}}_{\mathrm{R}}\right)=\frac{1}{n}\left[{\left(\frac{\alpha }{\alpha -1}\right)}^2{\left({\displaystyle \sum }_{i=1}^K{p}_i^{\alpha }\right)}^{-2}\left({\displaystyle \sum }_{i=1}^K{p}_i^{2\alpha -1}-{\left({\displaystyle \sum }_{i=1}^K{p}_i^{\alpha }\right)}^2\right)\right].$$

(5)

2.1.3. Tsallis Entropy

Another generalization of Shannon entropy is mainly due to Constantino Tsallis. Tsallis entropy is also known as q-entropy and is a monotonic function of Rényi entropy. It is given by (see [20]):

$${\mathrm{H}}_{\mathrm{T}}=\frac{1-{{\displaystyle \sum }}_{i=1}^K{p}_i^{\alpha }}{\alpha -1}=\frac{1-e^{\left(1-\alpha \right){\mathrm{H}}_R}}{\alpha -1},\mathrm{for}\alpha >0\mathrm{and}\alpha \ne 1.$$

(6)

For $\alpha =2$, Tsallis entropy is identical to Gini Concentration Index [21]. The variance of this entropy estimator is given by [14]:

$$\mathrm{Var}\left({\widehat{H}}_{\mathrm{T}}\right)=\frac{1}{n}\left[{\left(\frac{\alpha }{\alpha -1}\right)}^2\left({\displaystyle \sum }_{i=1}^K{p}_i^{2\alpha -1}-{\left({\displaystyle \sum }_{i=1}^K{p}_i^{\alpha }\right)}^2\right)\right].$$

(7)

2.1.4. Asymptotic Sampling Distributions of Entropy Measures

Agresti and Agresti [22] present some information about sampling properties of Gini concentration index. They also introduce some tests to compare the qualitative variation of two groups. Magurran [23] discusses some statistical tests for comparing the entropies of two samples. Agresti [24] provides the method of deriving sampling distributions for qualitative variation statistics. Pardo [14] emphasizes that entropy-based uncertainty statistics can also be derived from divergence statistics. He also discusses some inferential issues in detail. For the asymptotic behaviour of entropy measures, one may refer to Zhang Xing [18] and Evren and Ustaoğlu [25] under the condition of maximum uncertainty.

2.2. Qualitative Variation Statistics

In this section we give a list of qualitative variation indices axiomatized by Wilcox [1] and discuss sampling properties and relationship with power divergence statistic. Wilcox [1] notes that in textbook treatments of measures of variation, range, semi-interquartile range, average deviation and standard deviation are presented and discussed. However, the presentation and discussion of measures of variation suitable for a nominal scale is often completely absent. His paper represents a first attempt to gather and to generate alternative indices of qualitative variation at introductory level.

2.2.1. Axiomatizing Qualitative Variation

Wilcox points out that any measure of qualitative variation must satisfy the following:

• Variation is between zero and one;
• When all of the observations are identical, variation is zero;
• When all of the observations are different, the variation is one.

2.2.2. Selected Indices of Qualitative Variation

A list of selected qualitative variation statistics, some of which might have already been called differently by different authors in various economical, ecological, and statistical studies, are listed in Table 1.

Table 1. A list of selected qualitative variation indices ($f_i$ denotes the frequency of category i, n is the sample size and K is the number of categories).

Index	Defining Formula	Min	Max	Explanation
Variation ratio or Freeman’s index (VR)	$1-\frac{f_{mode}}{n}$	0	$\frac{\left(K-1\right)}{K}$	$f_{mode}$ is the frequency of the modal class.
Index of deviations from the mode (ModVR)	$\frac{nK-K{f}_{mode}}{n\left(K-1\right)}$	0	1	Normalized form of the variation ratio.
Index based on a range of frequencies (RanVR)	$\frac{f_{min}}{f_{mode}}$	0	1	$f_{min,}{f}_{mode}$ are minimum and maximum frequencies.
Average deviation (AVDEV)	$\frac{1-{{\displaystyle \sum }}_{i=1}^K\left|f_i-\frac{n}{K}\right|}{2\frac{n}{K}\left(K-1\right)}$	0	1	Analogous to mean deviation. K is the number of categories.
Variation index based on the variance of cell frequencies (VarNC)	$1-\frac{{{\displaystyle \sum }}_{i=1}^K{\left(f_i-\frac{n}{K}\right)}^2}{\frac{n^2\left(K-1\right)}{K}}$	0	1	Normalized form of Tsallis entropy when $\mathsf{\alpha }=2$. Analogous to variance.
Std deviation (StDev)	$1-\sqrt{\frac{{{\displaystyle \sum }}_{i=1}^K{\left(f_i-\frac{n}{K}\right)}^2}{\frac{n^2\left(K-1\right)}{K}}}$	0	1	Analogous to standard deviation.
Shannon entropy (H)	$-{\displaystyle \sum }_{i=1}^K{p}_{i}log{p}_i$	0	logK	The base of the logarithm is immaterial.
Normalized entropy (HRel)	$-\frac{1}{logK}{\displaystyle \sum }_{i=1}^K{p}_{i}log{p}_i$	0	1	Normalization is used to force the index between 0 and 1.
B index	$1-\sqrt{1-{\left[\sqrt[K]{{\displaystyle \prod }_{i=1}^K{f}_i\frac{K}{n}}\right]}^2}$	0	1	B index considers the geometric mean of cell probabilities.
M1 (Tsallis entropy for $\alpha =2)$	$1-{\displaystyle \sum }_{i=1}^K{p}_i^2$	0	$\frac{\left(K-1\right)}{K}$	It is also known as Gini Concentration Index.
Heip Index (HI)	$\frac{exp\left(\mathrm{H}\right)-1}{K-1}$	0	1	If Shannon entropy is based on natural logarithms.

2.2.3. Normalizing (Standardizing) an Index

In general, if an index I fails to satisfy any of the requirements in Section 2.2.1, the following transformation can be used for remedy:

$$I_T=\frac{I-I_{min}}{I_{max}-I_{min}}.$$

(8)

Note that this has the same form as the distribution function of a uniform distribution. Since any distribution function is limited between 0 and 1, this transformation is useful in improving the situation. The term “normalization” or “standardization” is not related to normal distribution. Rather, it is intentionally used to indicate that any “normalized” index takes values from the interval $\left[0,1\right]$. For example, $\mathrm{V}$$\mathrm{R}=1-\frac{f_{mode}}{n}$ and whenever all observations come from one category, (Variation ratio) $\mathrm{VR}$ is zero. On the other hand, when $f_i=\frac{n}{K}\mathrm{for}\mathrm{i}=1,2,\dots ,K,$ $\mathrm{VR}=\frac{K-1}{K}$. Dividing $\mathrm{VR}$ by $\frac{K-1}{K}$ normalizes $\mathrm{VR}$. In other words, normalizing $\mathrm{VR}$ this way produces the index (Index of deviations from the mode) ModVR.

2.2.4. Power-Divergence Statistic and Qualitative Variation

Loosely speaking, for discrete cases, a statistic of qualitative variation measures the divergence between the distribution under study, and the uniform discrete distribution. For the general exposition of statistical divergence measures, one may refer to Basseville [26] and Bhatia and Singh [27]. Cressie and Read [28] show that ordinary chi-square and log-likelihood ratio test statistics for goodness of fit can be taken as the special cases of power-divergence statistic. Chen et al. [29] and Harremoёs [30] explain the family of power-divergence statistics based on different parametrizations. Power-divergence statistic is an envelope for goodness of fit testing and is defined as:

$$\mathrm{PD}\left(\lambda \right)=\frac{2}{\lambda \left(\lambda +1\right)}{\displaystyle \sum }_{i=1}^K{f}_i\left[{\left(\frac{f_i}{e_i}\right)}^{\lambda }-1\right]$$

(9)

where $f_i$ is the observed frequency, $e_i$ is the expected frequency under the null hypothesis, and $\lambda$ is a constant. Under the assumption of maximum uncertainty $e_i=\frac{n}{K}$, it becomes $f_i=n{p}_i$ and:

$$\mathrm{PD}\left(\lambda \right)=\frac{2n}{\lambda \left(\lambda +1\right)}\left[K^{\lambda }{\displaystyle \sum }_{i=1}^K{p}_i^{\lambda +1}-1\right].$$

(10)

By substituting $\alpha =\lambda +1$ in Equation (6), Tsallis entropy can be formulated alternatively as:

$${\mathrm{H}}_{\mathrm{T}}\left(\lambda +1\right)=\frac{1-{{\displaystyle \sum }}_{i=1}^K{p}_i^{\lambda +1}}{\lambda },\mathrm{for}\lambda >1\mathrm{and}\lambda \ne 0.$$

(11)

The normalized Tsallis entropy (${\mathrm{H}}_{\mathrm{TN}})$ can also be found as:

$${\mathrm{H}}_{\mathrm{TN}}\left(\lambda +1\right)=\frac{1-{{\displaystyle \sum }}_{i=1}^K{p}_i^{\lambda +1}}{1-K^{-\lambda }},\mathrm{for}\lambda >1\mathrm{and}\lambda \ne 0.$$

(12)

From Equation (12), we obtain the power-divergence statistic as:

$$\mathrm{PD}\left(\lambda \right)=\frac{2n\left(K^{\lambda }-1\right)\left(1-{\mathrm{H}}_{\mathrm{TN}}\left(\lambda +1\right)\right)}{\lambda \left(\lambda +1\right)}.$$

(13)

This result is in agreement with intuition. In the case of maximum entropy, normalized Tsallis entropy will be equal to one and $\mathrm{PD}\left(\lambda \right)=0$ as expected.

3. Normality Tests for the Various Measures of Qualitative Variation

3.1. Tests for Normality and Scenarios Used for the Evaluation

In order to test the normality of the above given qualitative indices under various entropy values, distributions with four, six and eight categories are studied. These distributions are chosen to investigate the differences between the behaviour of these indices in cases of both maximum entropy and lower entropy. Samples of 1000, 2000, and 5000 units are taken with corresponding runs for all distributions. The distributions are shown in Table 2, labelled from 1 to 6. Note that odd-numbered distributions correspond to maximum entropy cases. We have observed that none of the indices distribute normally for maximum entropy distributions no matter how large the sample size is. Therefore, we present only the results for lower entropy distributions.

Table 2. Six parent discrete distributions used in the simulations.

Distribution	1	2	3	4	5	6
X	f(x)	f(x)	f(x)	f(x)	f(x)	f(x)
1	0.25	0.35	0.166	0.5	0.125	0.05
2	0.25	0.1	0.166	0.2	0.125	0.65
3	0.25	0.45	0.166	0.1	0.125	0.05
4	0.25	0.1	0.166	0.1	0.125	0.05
5	-	-	0.166	0.05	0.125	0.05
6	-	-	0.166	0.05	0.125	0.05
7	-	-	-	-	0.125	0.05
8	-	-	-	-	0.125	0.05

3.2. Test Results for General Parent Distributions

To test the asymptotic normality of the indices, Schapiro-Wilk W, Anderson-Darling, Martinez-Iglewicz, Kolmogorov-Smirnov, D’Agostino Skewness, D’Agostino Kurtosis, and D’Agostino Omnibus tests are used. Success rates of these tests are shown in percentages for the non-maximum uncertainty distributions 2, 4, and 6 in Table 3. For instance, a rate 71% means five of the above mentioned seven tests accepted normality (5/7 = 0.714; the numbers are rounded to the nearest integer).

Table 3. Normality results in percentages for 2nd, 4th, and 6th distributions.

Index	Distribution 2			Distribution 4			Distribution 6
	n = 1000	n = 2000	n = 5000	n = 1000	n = 2000	n = 5000	n = 1000	n = 2000	n = 5000
Variation ratio	71	86	100	100	100	100	86	100	100
ModVR	71	86	100	100	100	100	86	100	100
RanVR	100	71	100	100	71	100	100	86	43
Average Deviation	100	71	100	86	100	100	86	100	100
VarNC	57	100	100	14	100	100	100	100	100
StDev	100	100	100	86	100	100	100	100	100
Shannon entropy	71	100	100	14	100	100	100	100	100
HRel	71	100	100	14	100	100	100	100	100
B index	100	71	100	100	100	100	100	100	57
M1	57	100	100	14	100	100	100	100	100
Heip index	28	100	100	57	100	100	100	100	100

As a general tendency as the sample size increases, nine of the eleven indices tend to normality for all non-maximum entropy distributions. Nevertheless the normality of two indices, namely RanVR and the B index, is affected by dimensionality and sample size. Moreover, sampling variability of these two indices is found to be considerably higher as compared to the other nine indices. This phenomenon can be seen in the coefficient variation diagrams of indices in Figure 1 for the six distributions in Table 2 with three different sample sizes.

Figure 1. Sampling variability of qualitative variation statistics. The vertical axis represents the coefficient of variation: (a) Distribution 1; (b) Distribution 2; (c) Distribution 3; (d) Distribution 4; (e) Distribution 5; (f) Distribution 6.

3.3. Test Results for Cases of Maximum Entropy

When the entropy is at the maximum, the variability of VarNC, StDev, Shannon, Hrel, and M1 statistics is comparatively low, as seen in Figure 1. On the other hand when the level of uncertainty is lower, the variability of VarNC and StDev statistics is still one among the lower scores. In addition, because of the close relationship between VarNC and StDev statistics with the chi-square distribution in case of maximum entropy, sampling properties of these two statistics can be deduced exactly; we address this issue in Section 4.

4. Sampling Properties

4.1. VarNC Statistic

The VarNC statistic is the analogous form of variance for analysing nominal distributions. It also equals the normalized form of Tsallis entropy when $\alpha =2$. The VarNC statistic can be evaluated in analogy to the variance of discrete distributions. It is defined as:

$$\mathrm{VarNC}=1-\frac{{{\displaystyle \sum }}_{i=1}^K{\left(f_i-\frac{n}{K}\right)}^2}{\frac{n^2\left(K-1\right)}{K}}.$$

(14)

Under maximum entropy assumption, the quantity:

$$\mathrm{X}=\frac{{{\displaystyle \sum }}_{i=1}^K{\left(f_i-\frac{n}{K}\right)}^2}{\frac{n}{K}}$$

(15)

fits a chi-square distribution with K − 1 degrees of freedom [1]. Thus, the probability density of VarNC statistics can be written as:

$$f\left(x\right)=\frac{x^{\frac{K-1}{2}-1}{e}^{-\frac{x}{2}}}{2^{\frac{K-1}{2}}\mathsf{\Gamma }\left(\frac{K-1}{2}\right)},x>0.$$

(16)

If we let $Z=\mathrm{VarNC}=a+bX$ for $a=1$ and $b=\frac{-1}{n\left(K-1\right)}$, then with $f\left(z\right)=f\left(x\right)\left|\frac{dx}{dz}\right|$ we obtain:

$$f\left(z\right)=c*{\left(1-y\right)}^{\frac{K-3}{2}}{e}^{-\frac{n\left(K-1\right)\left(1-y\right)}{2}}\mathrm{for}0<y<1$$

(17)

where $c=\frac{1}{\mathsf{\Gamma }\left(\frac{K-1}{2}\right)}{\left(\frac{n\left(K-1\right)}{2}\right)}^{\frac{\left(K-1\right)}{2}}.$

It can be shown that VarNC equals the normalized version of Tsallis (α = 2) entropy with $f_i=n{p}_i$;

$${\displaystyle \sum }_{i=1}^K{\left(f_i-\frac{n}{K}\right)}^2={\displaystyle \sum }_{i=1}^K{\left(n{p}_i-\frac{n}{K}\right)}^2=n^2{\displaystyle \sum }_{i=1}^K{\left(p_i-\frac{1}{K}\right)}^2$$

(18)

$${\displaystyle \sum }_{i=1}^K{\left(p_i-\frac{1}{K}\right)}^2={\displaystyle \sum }_{i=1}^K{p}_i^2-\frac{1}{K}$$

(19)

$$\mathrm{VarNC}=1-\frac{n^2\left({{\displaystyle \sum }}_{i=1}^K{p}_i^2-\frac{1}{K}\right)}{\frac{n^2\left(K-1\right)}{K}}=\frac{K}{K-1}\left(1-{\displaystyle \sum }_{i=1}^K{p}_i^2\right).$$

(20)

VarNC is also inversely proportional to the variance of cell probabilities of a multinomial distribution. The variance of Tsallis (α = 2) entropy can be found by direct substitution of α = 2 in Equation (7):

$$\mathrm{Var}\left(\mathrm{VarNC}\right)=\frac{1}{n}{\left(\frac{K}{K-1}\right)}^2\left({\displaystyle \sum }_{i=1}^K{p}_i^3-{\left({\displaystyle \sum }_{i=1}^K{p}_i^2\right)}^2\right)$$

(21)

which is larger than the variance of Tsallis (α = 2) entropy.

By deriving moments, one can show the consistency of VarNC statistics:

$$\mathrm{E}\left(\mathrm{VarNC}\right)=1-\frac{1}{n\left(K-1\right)}\left(K-1\right)=\frac{n-1}{n}$$

(22)

$$\mathrm{Var}\left(\mathrm{VarNC})\right)=\frac{2\left(K-1\right)}{n^2{\left(K-1\right)}^2}=\frac{2}{n^2\left(K-1\right)}$$

(23)

Under the assumption of maximum entropy, VarNC is biased since $\mathrm{E}\left(\mathrm{varNC}\right)\ne 1$; however, it is consistent since $\underset{n\to \infty }{\lim }\mathrm{E}\left(\mathrm{VarNC}\right)=1$, and $\underset{n\to \infty }{\lim }\mathrm{Var}\left(\mathrm{VarNC}\right)=0$ (see [31]). Finally, it can be noted that for larger K values, $\mathrm{VarNC}$ can be approximated by a normal distribution with $\mu =\frac{\left(n-1\right)}{n}$ and ${\sigma }^2=\frac{2}{n^2\left(K-1\right)}$.

4.2. StDev Statistic

The StDev statistic was proposed by Wilcox [1] as the analogous formulation of ordinary standard deviation for qualitative distributions. It is defined as:

$$\mathrm{StDev}=1-\sqrt{\frac{{{\displaystyle \sum }}_{i=1}^K{\left(f_i-\frac{n}{K}\right)}^2}{\frac{n^2\left(K-1\right)}{K}}}.$$

(24)

The statistic Y = StDev is a function of $X=\frac{{{\displaystyle \sum }}_{i=1}^K{\left(f_i-\frac{n}{K}\right)}^2}{\frac{n}{K}}$ as it holds $Y=1-\sqrt{\frac{X}{n\left(K-1\right)}}.$ That means that Y is $a-\sqrt{bX}$ with $a=1$ and $b=\frac{1}{n\left(K-1\right)}$ where X has the same probability density function as in Equation (16). Then by the transformation of probability densities one obtains:

$$f\left(y\right)=c*{\left(1-y\right)}^{K-2}{e}^{-\frac{n\left(K-1\right){\left(1-y\right)}^2}{2}}\mathrm{for}0<y<1$$

(25)

where $c=\frac{2^{\frac{3-K}{2}}{\left(n\left(K-1\right)\right)}^{\left(K-1\right)/2}}{\mathsf{\Gamma }\left(\frac{K-1}{2}\right)}.$

By deriving moments, one can show the consistency of StDev statistics. By Equation (24), we write:

$$\mathrm{E}\left(Y\right)=1-\mathrm{E}\left[\sqrt{\frac{X}{n\left(K-1\right)}}\right].$$

(26)

If we let g(x) = $\sqrt{\frac{X}{n\left(K-1\right)}}$, by Taylor series expansion we have:

$$\mathrm{E}\left(g\left(X\right)\right)\cong \mathrm{E}\left(g\left(x_0\right)+g^{\prime }\left(x_0\right)\left(x-x_0\right)+g^{″}\left(x_0\right)\frac{{\left(x-x_0\right)}^2}{2!}\right).$$

(27)

For $x_0=\mathrm{E}\left(X\right)=K-1$, we get $g\left(x_0\right)=\frac{1}{\sqrt{n}}$, $g^{\prime }\left(x_0\right)=\frac{1}{2\sqrt{n}\left(K-1\right)}$, and $g^{″}\left(x_0\right)=-\frac{1}{4{\left(K-1\right)}^2\sqrt{n}}$.

Then we obtain:

$$\mathrm{E}\left(Y\right)=\mathrm{E}\left(\mathrm{StDev}\right)\cong 1-\frac{1}{\sqrt{n}}\left[1-\frac{1}{4\left(K-1\right)}\right].$$

(28)

Similarly, ignoring the quadratic and higher terms in the Taylor-series expansion yields:

$$\text{Var}\left(Y\right)=\text{Var}\left(\text{StDev}\right)=\text{Var}\left(g\left(X\right)\right)\cong {\left(g^{\prime }\left(x_0\right)\right)}^2\text{Var}\left(X\right)=\frac{1}{2n\left(K-1\right)}.$$

(29)

StDev is biased, but consistent since, as $n\to \infty$, it holds $\mathrm{E}\left(Y\right)\to 1$ and $\underset{n\to \infty }{\lim }\mathrm{Var}\left(Y\right)=0.$

4.3. Probability Distribution of VarNC and StDev under Maximum Entropy

The probability distributions of the statistics VarNC and StDev under the assumption of maximum entropy are shown in Figure 2, for two different n and K values.

Figure 2. The probability distributions of the statistics VarNC and StDev under the assumption of maximum uncertainty for various selected parameters n and K: (a) n = 10, K = 5; (b) n = 50, K = 5; (c) n = 10, K = 10; (d) n = 50, K = 10.

5. Discussion

All categorical distributions can be modelled by multinomial distribution. As the sample size increases indefinitely, multinomial distribution tends to a multivariate normal distribution. All qualitative variation statistics discussed are functions of cell counts of multinomial distribution. This fact implies the asymptotic normality of various qualitative variation measures which are simply the functions of cell counts or probabilities, themselves. In our simulation studies, we have observed this tendency for most of the investigated qualitative variation statistics whenever the uncertainty is not at maximum, except for RanVR and the B index for some dimensionalities and sample sizes. The RanVR statistic mainly uses two special numbers, the minimum and maximum frequencies. In other words, the RanVR statistic is not sufficient since it does not use all relevant sample information. For this reason, higher sampling variability is expected a priori. This situation is especially important as dimensionality (K) increases. On the other hand, the B Index is a function of the geometric mean of the probabilities. This way of multiplicative formulation of uncertainty causes higher sampling variability and may be a factor preventing normality.

None of the indices which we studied distribute normally in the case of maximum entropy, no matter how large the sample size is. This implies that maximum entropy is a factor preventing normality. Secondly, when there is little or no information about cell probabilities of multinomial distribution, the principle of insufficient reasoning justifies assuming maximum entropy distributions. In such cases, VarNC and StDev may be used in modelling the qualitative variation, since the probability distributions of these two statistics can be derived based on the relation between VarNC, StDev, and chi-square distribution. In this study we have derived the probability functions of these two statistics and shown that both statistics discussed are consistent. We have also shown that the variance of VarNC is less than that of StDev statistic and VarNC has some additional appealing properties because it is simply the normalized version of Tsallis ($\mathsf{\alpha }=2)$ when the uncertainty is at maximum.