On the Use of Variability Measures to Analyze Source Coding Data Based on the Shannon Entropy

Abstract

Source coding maps elements from an information source to a sequence of alphabetic symbols. Then, the source symbols can be recovered exactly from the binary units. In this paper, we derive an approach that includes information variation in the source coding. The approach is more realistic than its standard version. We employ the Shannon entropy for coding the sequences of a source. Our approach is also helpful for short sequences when the central limit theorem does not apply. We rely on a quantifier of the information variation as a source. This quantifier corresponds to the second central moment of a random variable that measures the information content of a source symbol; that is, considering the standard deviation. An interpretation of typical sequences is also provided through this approach. We show how to use a binary memoryless source as an example. In addition, Monte Carlo simulation studies are conducted to evaluate the performance of our approach. We apply this approach to two real datasets related to purity and wheat prices in Brazil.

1. Introduction

Entropy is a fundamental concept in science [1,2]. In information theory [3], entropy is the process of coding a discrete memoryless source (DMS) through the first Shannon theorem [4,5]. The law of large numbers was used to prove this theorem [6]. The concept of information theory and its relation to entropy is found in [7]. The reader is referred to [8,9,10] for applications of entropy in statistics to show the relevance of entropy. Two relevant works on statistical mechanics by maximizing entropy were presented in [11,12].

In the entropy context, source coding maps from an information source to a sequence of alphabetic symbols (often using bits). This mapping occurs in such a way that the source symbols may be recovered exactly from the binary units (source coding with no information loss) or recovered within some distortion (source coding with information loss). Such a mapping is the concept behind data compression. Source coding represents symbols of an information source with as few code digits per source symbol as possible [13]. A distributed conditional arithmetic encoding based on source symbol purging has been proposed. Some works devoted to coding finite-length source sequences can be found in [14] and the references therein.

Although various works focus on entropy, they are only associated with the mean (first statistical moment) and not with the variance (second central moment) of the information produced by the source symbols. Information variation (or variability) is an aspect that should be considered in all information analysis. Then, instead of using the entropy as a lower bound on the average letter length of the source coding sequences, one can adopt an interval based on information variation according to a confidence level stated by the code designer. Nevertheless, a point estimate is often considered rather than an interval estimate. To estimate the entropy of symbol sequences, an excellent reference is [15].

To the best of our knowledge, there are no published studies dealing with DMS in a general context, independent of the probabilistic model that generates the data. Only one unpublished work on the Shannon entropy of sequences exists, in which a plugin estimator was proposed in 2018 by the authors of this paper [16]. Then, in another unpublished work in 2021 [17], the estimation of the variance of the Shannon entropy was stated. This work utilized a plugin estimator for the variance and provided asymptotic properties of the estimator over the conditions of a probabilistic simplex model. The study presented in [17] is different from that proposed earlier in [16] and formalized in the present investigation because we do not fix a probabilistic model but look at sequence-based empirical properties and not at the model. In [17], the asymptotic properties of the plugin estimator are verified. In the present study, we use the variance of the plugin estimator and the entropy estimator to characterize the DMS. For example, we can determine with a high degree of confidence whether the source is degenerate or it generates low or high entropy, which, in practical terms, helps us to determine if the sequence is predictable or not. In an old study presented in 1974 [18], some moments of an estimate of the Shannon measure of information were introduced for the multinomial distribution. In that work, a resampling mechanism was employed to directly evaluate the plugin estimator’s variability and verify what entropy estimator would be most appropriate in an ergodic environment based on the multinomial distribution. Therefore, the proposal introduced in [18] is based on a multinomial probabilistic model and not a general framework such as we state in the present investigation.

A work presented in 2008 [19] proposed an estimation of the entropy of binary time series. In that work, an estimator of the variance was provided via bootstrapping and did not utilize a plugin method. In [19], the authors were not interested in the variation directly as a classification mechanism, but the variation was employed to assess the Shannon estimator’s behavior calculating the bias and variance via bootstrapping.

Therefore, the primary objective of our investigation is to develop a method that includes information variation in the source coding. We use a quantifier of information variation as a source. This quantifier is related to the second central moment of a random variable that measures the information content of a source symbol—that is, through the variance and its corresponding standard deviation. Our approach is also helpful for short sequences in which the central limit theorem does not have enough support to be applied. Based on our approach, we also provide an interpretation of typical sequences. Note that the relevance of describing information variation lies in improving predictability by considering the inherent variability of the source, which is often not considered. Therefore, our proposal gives an improvement in the source coding performance because although the literature provides several Shannon measures, it does not usually consider the variability of DMS. If this variability is high, then such Shannon measures have low predictability.

The rest of the paper is organized as follows. Section 2 introduces the proposed approach for information variation in sources of coding sequences. In Section 3, we state an evaluation of entropy for finite sequences. In Section 4, an estimation strategy for atypical sequences is developed. Section 5 presents an illustration that uses binary DMS, in which the output is zero with a probability or one with its complement. In Section 6, we evaluate the proposed approach with Monte Carlo simulation studies. In Section 7, empirical applications of our proposal are provided. Concluding remarks are established in Section 8.

2. Variation of Information

Consider a DMS represented by a random variable U that accepts symbols from a finite alphabet, that is, the sequence stated as

$$\mathbb{A}\equiv \left\{a_k\right\},k\in \{1,\dots ,K\},$$

(1)

with probability of occurrence at a time instant, l namely, denoted by $\left\{p_l\left(a_k\right)\right\}$, for $k\in \{1,\dots ,K\}$, where

$$p_l\left(a_k\right)=P_l(U=a_k),l\in \{1,\dots ,L\}.$$

(2)

The stationary distribution for the DMS model defined in (2) is independent over time, that is, $p_l\left(a_k\right)=p\left(a_k\right)=P(U=a_k)$, for all $l\in \{1,\dots ,L\}$. In addition, for each symbol $a_k$ produced, the associated amount of information is given by

$$\mathcal{I}\left(a_k\right)=-{log}_2\left(p\left(a_k\right)\right),a_k\in \mathbb{A},$$

(3)

with $\mathbb{A}$ being defined in (1). For simplicity, let us refer to $p\left(a_k\right)$ established in (3) as $p_k$.

A smooth functional of the probability distribution of the source (first moment) is recognized as the Shannon entropy of the DMS [20] and formulated as

$$H\left(U\right)=\mathrm{E}\left(\mathcal{I}\left(a_k\right)\right)=\mathrm{E}(-{log}_2\left(p\left(a_k\right)\right))=-\sum _{k=1}^K{p}_k{log}_2\left(p_k\right),$$

(4)

where $\mathrm{E}$ is the expected value operator and $\mathcal{I}\left(a_k\right)$ is the information amount of U defined in (3), with $\mathcal{I}\left(a_k\right)$ being a random variable.

We propose to use information variation as a compound of a functional based on the second moment of the probability distribution of the source established as

$$\begin{array}{ccc}\hfill F^2\left(U\right)& =& \mathrm{Var}\left(U\right)=\mathrm{E}({(U-\mathrm{E}\left(U\right))}^2)=\mathrm{E}\left(U^2\right)-{\left(\mathrm{E}\left(U\right)\right)}^2\hfill \\ & =& {\displaystyle \sum _{k=1}^K}{p}_k{\left({log}_2\left(p_k\right)\right)}^2-H^2\left(U\right),\hfill \end{array}$$

(5)

where $\mathrm{Var}$ is the variance operator. One can prove that $F^2\ge 0$ by the Jensen inequality. Therefore, the variation can be computed employing

$$F\left(U\right)=\sqrt{{\displaystyle \sum _{k=1}^K}(p_k-p_k^2){\left({log}_2\left(p_k\right)\right)}^2-\sum _{i\ne j}{p}_i{p}_j{log}_2\left(p_i\right){log}_2\left(p_j\right)},$$

(6)

with the expression stated in (6) being obtained by applying to (4) the multiplication rule for series, also called the Lagrange identity, which allows us to attain at

$$H^2\left(U\right)={\left(-\sum _{k=1}^K{p}_k{log}_2\left(p_k\right)\right)}^2=\sum _{k=1}^K{p}_k^2{\left({log}_2\left(p_k\right)\right)}^2+\sum _{i\ne j}{p}_i{p}_j{log}_2\left(p_i\right){log}_2\left(p_j\right).$$

(7)

Therefore, by using (5) and (7), after a simple algebra, we reach the formula given in (6).

Definition 1 (Degenerated sources).

We define two types of degenerated sources as follows:

(i) 

[Type I source] This is denoted by $U^{*}$ and associated with the sequence $\left\{p_k\right\}$, for $k\in \{1,\dots ,K\}$, where there exists $k^{*}$, such that $p_{k^{*}}=1$ and $p_k=0$, for all $k\ne k^{*}$.

(ii) 

[Type II source] This is denoted by $U^{**}$ and related to the sequence $\left\{p_k\right\}$, for $k\in \{1,\dots ,K\}$, where $p_k=1/J$, for all $k\in I\subset \{1,\dots ,K\}$ and $J=\left|\right|I\left|\right|$, with $\left|\right|·\left|\right|$ denoting the cardinality of a set.

Proposition 1.

The variation of information is null if and only if the source is degenerated; that is, $F\left(U\right)=0$, for any $U=U^{*}$ or $U=U^{**}.$

Proof. 

(⇐) $F\left(U^{*}\right)=0$ follows directly from $p_k=p_k^2$, for all k, and $p_i{p}_j=0$, for all $i\ne j$. Regarding the formula stated in (6), for type II sources, there are $J>1$ identical terms in the first summation and $J^2-J$ terms in the second summation. Therefore, we have that

$$\begin{array}{cc}\hfill {\displaystyle \sum _{k=1}^J}(p_k-p_k^2){\left({log}_2\left(p_k\right)\right)}^2& =J\left(\frac{1}{J}-\frac{1}{J^2}\right){\left(log\left(\frac{1}{J}\right)\right)}^2,\hfill \\ \hfill {\displaystyle \sum _{i\ne j}}{p}_i{p}_j{log}_2\left(p_i\right)log\left(p_j\right)& =(J^2-J){\left(log\left(\frac{1}{J}\right)\right)}^2\frac{1}{J^2}.\hfill \end{array}$$

(⇒) Without loss of generality, consider the case where $\left\{p_k\right\}$ is sorted in descending order; that is, $p_K\le p_{K-1}\le \dots \le p_1$. This means that $p_k>0$ because the terms $p_k=0$ do not count. Thus, the possible solutions are of the type $p_k=1/J$, for $k\in \{1,\dots ,J\}\le K$ and $p_k=0$, for all $K>J$. Note that two trivial solutions are the following:

• $p_1=1$ and $p_k=0$, for $k>1$, with $J=1$.
• $p_k=1/J$, for $k\in \{1,\dots ,J\}$, with $J=K$.

Therefore, this concludes the proof. □

It is worth noting that the sources denoted by $U^{*}$ and $U^{**}$ in Definition 1 are two particular types of degenerated distributed DMS. The first type has just one symbol with nonzero probability, so it always sends the same symbol with probability one. The second type is a uniformly distributed source, which is often a limiting case. Although $U^{*}$-sources are a subset of $U^{**}$-sources (by assuming $J=\left|\right|I\left|\right|=1$), we distinguish them as distinct types of sources.

3. Evaluation of Entropy for Finite Sequences

Let $\left\{x_l\right\}$ be a sequence of observed counts of alphabetical symbols in a DMS sample of size L. We define the sample relative frequency of the k-th symbol $a_k$ employing

$${\widehat{p}}_k=\frac{m_k}{L},k\in \{1,\dots ,K\},$$

(8)

where $m_k$ is the number of symbols $a_k$ in the sample.

The most common manner of estimating $H\left(U\right)$ defined in (4) is nonparametrically using the plugin estimator provided as

$$\widehat{H}\left(U\right)=-\sum _{k=1}^K{\widehat{p}}_k{log}_2\left({\widehat{p}}_k\right),$$

(9)

where ${\widehat{p}}_k$ is given in (8) and K is the length of the sequence. Note that the estimate stated in (9) has been a subject of much research for decades [21,22,23,24,25]. Based on (5), we propose to use a statistic as a plugin estimator of information variation formulated as

$${\widehat{F}}^2\left(U\right)=\sum _{k=1}^K{\widehat{p}}_k{\left({log}_2\left({\widehat{p}}_k\right)\right)}^2-{\left(\widehat{H}\left(U\right)\right)}^2.$$

(10)

We can prove that the estimators $\widehat{H}\left(U\right)$ and ${\widehat{F}}^2\left(U\right)$ defined in (9) and (10) are consistent and asymptotically unbiased for $H\left(U\right)$ and $F^2\left(U\right)$, respectively. These desirable statistical properties can be proved by using a Taylor expansion of the functions $\widehat{H}\left(U\right)$ and ${\widehat{F}}^2\left(U\right)$ about the point $(p_1,\dots ,p_K)$ and confining ourselves to derivatives of fourth order.

Consider a DMS sample of size L. Then, based on the central limit theorem, as $L\to \infty$, we have that $\widehat{H}\left(U\right)\stackrel{\mathcal{D}}{\to }Z\left(U\right)$, that is, $\widehat{H}\left(U\right)$ converges in distribution to $Z\left(U\right)$, where $Z\left(U\right)\sim \mathcal{N}(H\left(U\right),F^2\left(U\right)/L)$. Thus, we obtain that

$$\widehat{H}\left(U\right)\dot{\sim }\mathcal{N}\left(H\left(U\right),\frac{F^2\left(U\right)}{L}\right),$$

(11)

where ∼ and $\dot{\sim }$, as usual in statistics, mean “distributed exactly as” and “approximately (or asymptotically) distributed as”, respectively.

If the condition $L\to \infty$ does not hold to apply the central limit theorem, as also usual in statistics, we can consider the assumption established as

$$\widehat{H}\left(U\right)\sim \mathcal{N}\left(H\left(U\right),\frac{F^2\left(U\right)}{L}\right).$$

(12)

In addition, as ${\widehat{F}}^2\left(U\right)$ is an estimator of the information variation [22], we may state that

$$\frac{(L-1){\widehat{F}}^2\left(U\right)}{F^2\left(U\right)}\sim {\chi }^2(L-1),$$

(13)

where ${\chi }^2(L-1)$ is the chi-square distribution with $L-1$ degrees of freedom.

Consequently, from the result presented in (12), we get

$$Z\left(U\right)=\frac{\sqrt{L}(\widehat{H}\left(U\right)-H\left(U\right))}{F\left(U\right)}\sim \mathcal{N}(0,1).$$

(14)

Hence, based on the expressions given in (13) and (14), we obtain

$$T\left(U\right)=\frac{\sqrt{L}(\widehat{H}\left(U\right)-H\left(U\right))}{\widehat{F}\left(U\right)}\sim t(L-1),$$

(15)

where $t(L-1)$ is the Student-t distribution with $L-1$ degrees of freedom. Naturally, for short sequences, it is more appropriate to use the Student-t statistic defined in (15). Thus, a $100(1-\alpha )\%$ confidence bound for $H\left(U\right)$ is expressed as

$$H_{\mathrm{t}}=\widehat{H}\left(U\right)+t_{1-\alpha }(L-1)\frac{\widehat{F}\left(U\right)}{\sqrt{L}},$$

(16)

where $t_{1-\alpha }(L-1)$ is the upper $100(1-\alpha )$-th percentile of the Student-t distribution with $L-1$ degrees of freedom [26]. Note that, for large L, based on the asymptotic distribution reached in (11), or for small L, considering the assumption established in (14), in the case of a known variance $F^2\left(U\right)$, and the theory stated in [26], we can calculate a confidence bound for $H\left(U\right)$ by means of

$$H_{\mathrm{N}}=\widehat{H}\left(U\right)+z_{1-\alpha }\frac{F\left(U\right)}{\sqrt{L}},$$

(17)

where $z_{1-\alpha }={\Phi }^{-1}(1-\alpha )$ is the upper $100(1-\alpha )$-th percentile of the standard normal distribution, which is the inverse of the cumulative distribution function $\Phi$.

If one sets a confidence level at, say $95\%$, then, for sequence outcomes of the source (all with small length L), the sample entropy value should not exceed the bound $H_{\mathrm{t}}$ given in (16) in $95\%$ of cases; or the bound $H_{\mathrm{M}}$ given in (17) in $95\%$ of cases, if L is large. As a result, rather than the fundamental limit of $H\left(U\right)$, which can be reached asymptotically as the length of the source sequences increases indefinitely, we can expect to code the output with $H_{\mathrm{t}}$ bits per source symbol.

The efficiency of a source coding run for a specific given sequence is determined by using a conventional expression that is defined as

$$\eta =\frac{H}{\overline{L}},$$

(18)

where $\overline{L}$ is the average number of bits per source letter. Instead of adopting the conventional formula stated in (18), we propose to assess such an efficiency (especially in the cases of short sequences and high entropy variability DMS) at a confidence level of $100(1-\alpha )\%$, according to

$${\eta }_{\alpha }=\frac{\widehat{H}\left(U\right)+t_{1-\alpha }(L-1)\widehat{F}\left(U\right)/\sqrt{L}}{\overline{L}}.$$

4. Atypical Sequences

Let us now shift the focus to long sequences. In this scenario, a statistical interpretation for atypical sequences can be formulated.

All sequences generated from the source that result in a sample entropy are considered typical if any of them fall within a $100(1-\alpha )\%$ confidence interval indicated as

$$\left[\widehat{H}\left(U\right)-z_{1-\alpha /2}\frac{\widehat{F}\left(U\right)}{\sqrt{L}},\widehat{H}\left(U\right)+z_{1-\alpha /2}\frac{\widehat{F}\left(U\right)}{\sqrt{L}}\right]\equiv \widehat{H}\left(U\right)\pm z_{1-\alpha /2}\frac{\widehat{F}\left(U\right)}{\sqrt{L}}.$$

(19)

Definition 2 ($\epsilon$-atypical sequence).

Let $\epsilon >0$ be an arbitrary value. For any sequence (a source sample) with a length L of symbols, the sample entropy is called an ε-atypical sequence if it is outside the range

$$[H-\epsilon ,H+\epsilon ].$$

(20)

The following question arises. What is the confidence level $100(1-\alpha )\%$ at which the confidence interval given in (19) for the entropy coincides with the interval stated in Definition 2 through the expression given in (20)? To respond to this question, we must impose the relationship formulated as

$${\Phi }^{-1}(1-\alpha /2)=\frac{\epsilon \sqrt{L}}{F\left(U\right)}.$$

(21)

In the case of the expression given in (21), the confidence level is interpreted as the probability of an atypical sequence occurring; that is, $P\left(\mathrm{atypical}\mathrm{sequence}\right)=\alpha$. Therefore, we have that

$$\underset{L\to \infty }{lim}P\left(\mathrm{atypical}\mathrm{sequence}\right)=\underset{L\to \infty }{lim}2\left(1-\Phi \left(\frac{\epsilon \sqrt{L}}{F\left(U\right)}\right)\right)=0.$$

Typical sequences (in the sense that the sample entropy is $\epsilon$-closed to the source’s entropy) constitute virtually 100% of the sequences if the sequence length L is sufficiently large.

Another key concept in information theory is the asymptotic equipartition property. The extension $U^L$ of the source U considers L-grams as the new symbols. This has an alphabet $\left\{{\underline{u}}_i\right\}$, with probabilities $\left\{p\left({\underline{u}}_i\right)\right\}$, for $i\in \{1,\dots ,K^L\}$. Note that, although we have

$$P\left(\underset{L\to \infty }{lim}\widehat{H}\left(U\right)=H\left(U\right)\right)=1,$$

it is possible to show that $\epsilon$-typical sequences hold [27] with the interval stated as

$$2^{-L\left(H\right(U)+\epsilon )}\le p\left({\underline{u}}_i\right)\le 2^{-L\left(H\right(U)-\epsilon )},i\in \{1,\dots ,K^L\}.$$

(22)

Therefore, from the formula given in (22), we get the set of typical sequences formulated as

$$p\left({\underline{u}}_i\right)\approx 2^{-LH\left(U\right)},i\in \mathbb{T}\subset \{1,\dots ,K^L\}.$$

(23)

It is worth noting that the expression given in (23) is constant for all $i\in \mathbb{T}$, with

$$\Vert \mathbb{T}\Vert =2^{LH\left(U\right)}.$$

By the definition of degenerated sources stated in Definition 1, we see that large extensions of a DMS become type II asymptotically degenerated. The entropy becomes exactly $LH\left(U\right)$ with no information variation.

5. Illustrative Example

As an illustration, consider the following example. Assume a binary DMS with an output of zero with a probability of p or one with a probability of $q=1-p$. For this case, the quantifiers given in (4) and (5) simplify, respectively, to two expressions stated as

$$\begin{array}{cc}\hfill H_2\left(p\right)& =-p{log}_2\left(p\right)-q{log}_2\left(q\right),\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill F_2\left(p\right)& =\sqrt{pq{\left({log}_2\left(p\right)-{log}_2\left(q\right)\right)}^2}.\hfill \end{array}$$

(25)

The binary entropy $H_2$ has a well-known shape [20]. We sketch $H_2$ and $F_2$ defined in (24) and (25) over p in Figure 1.

Regarding p, there are five values worth mentioning. Consider the ordered set of values stated as

$$p\in \{0,p_1,0.5,p_2,1.0\},$$

(26)

where $p_1$ and $p_2$ defined in (26) are points at which the function $F_2$ reaches a maximum. We can see in Figure 1 that $0<p_1<0.5$ and $0.5<p_2<1$. From this figure, note that three limit cases have no variance (binary degenerated sources): (i) $p=0$, (ii) $p=1$ (in both cases, the source transmits just the same symbol and has null entropy), and (iii) $p=0.5$. Surprisingly, in this last case (iii), with equiprobable sequences and maximum entropy, there is no variance in the amount of information.

Observe that, whatever the length of the sequence of source symbols, the average information per source letter is equal to the entropy. However, the points of maximum variability in the information content seem less, which is well known in the literature.

We begin by looking into the behavior of $\mathrm{d}{F}_2\left(p\right)/\mathrm{d}p$ stated as

$$\frac{\mathrm{d}{F}_2\left(p\right)}{\mathrm{d}p}=\frac{({\log }_2\left(p\right)-{log}_2(1-p))\left(\frac{2}{log\left(2\right)}+(1-2p)({log}_2\left(p\right)-{log}_2(1-p))\right)}{2\sqrt{p(1-p){({log}_2\left(p\right)-{log}_2(1-p))}^2}}.$$

(27)

A plot of the derivative of $F_2$ defined in (27) is shown in Figure 2. Note that the three points in this figure have infinite derivative values (singular points), which correspond to the three limit cases mentioned earlier in (i)–(iii). From the expression stated in (27), we define a function formulated as

$$g\left(p\right)=\frac{2}{log\left(2\right)}+(1-2p)({log}_2\left(p\right)-{log}_2(1-p)).$$

(28)

Notice that the two critical points (stationary values) of $F_2\left(p\right)$ occur when $g\left(p\right)=0$. In Figure 3, we can observe the plot of the function g defined in (28).

The Newton-Raphson (NR) algorithm was applied to approximate the two roots of g defined in (28). With a tolerance of ${10}^{-5}$ and four iterations, the values $p_1=0.083222$ (with an initial value to start the algorithm equal to 0.1) and $p_2=0.916778$ (with an initial value equal to 0.9) were obtained. The value of the derivative is around 0.956137 at the two maximum points, that is, $F_2\left(p_1\right)=F_2\left(p_2\right)\approx 0.956137$. In addition, we calculate the intersection points of the two curves of Figure 1, where $H_2\left(p\right)=F_2\left(p\right)$. In the range between the two points, the standard deviation of the entropy is less than the entropy itself. We use this range to characterize the binary DMS with low entropy variability.

By using the coefficient of variation (CV) as described in [26,28], we consider the formula defined as

$$\mathrm{CV}\left(p\right)=\frac{F_2\left(p\right)}{H_2\left(p\right)}.$$

(29)

We plot the CV stated in (29) through Figure 4. From the expression formulated in (29), we reach the function given by

$$D\left(p\right)=H_2\left(p\right)-F_2\left(p\right).$$

(30)

Figure 5 shows the plot of the function D expressed in (30) over p. Let $p_3$ and $p_4$ be the roots of the function D, with $p_3<p_4$. Again, the NR algorithm is used, and after three iterations and a tolerance of ${10}^{-5}$, the values $p_3=0.219076$ (with an initial value to start the algorithm equal to 0.3) and $p_4=0.780924$ (with an initial value equal to 0.7) were obtained.

The following definition is derived from analyzing the behavior of the CV over p as expressed in (29).

Definition 3 (Low entropy variability DMS).

A binary DMS that takes the values zero with a probability p, and one with a probability $1-p$, associated with the function D defined in (30) whose roots are $p_3$ and $p_4$, is called a low-entropy variability DMS if and only if we have

$$p_3=0.219076\le p\le 0.780924=p_4.$$

6. Monte Carlo Simulation Studies

To illustrate the previous results, we consider a binary source U generated by a Bernoulli process $X_t$ (sequence of identically independent distributed random variables). Assume that we run a Bernoulli trial every second, for t seconds, with $p=P(X_t=1)$ and $P(X_t=0)=1-p$, and the value equal to one being the arrival of a bit, for example.

We use Monte Carlo simulation to assess the distributional behavior of $Z\left(U\right)$ and $T\left(U\right)$, as given in expressions stated in (14) and (15), respectively. The number of Monte Carlo replications equals $M=5000$. The time is $t\in \{20,50,100,250,500\}$ in seconds, and we set $p\in \{0.010,0.106,0.202,0.298,0.394,0.490\}$ only by symmetry.

The kernel density estimates of $Z\left(U\right)$ applied to $X_t$ are shown in Figure 6. By visual inspection, we observe that the kernel density estimate of the distribution of $Z\left(U\right)$ shows evidence of a good approximation to the standard normal distribution (symmetric around to mean equal to zero, tails are decaying to zero, and turning points around of $-1$ and 1).

We observe that the approximation to the standard normal distribution is poor for short periods ($t=20$), considering all values of p as expected (the law of large numbers requires observing the source for a sufficiently large period). When p is close to 0 or 0.5, the source tends to degenerate, and the approximation of the standard normal distribution of $Z\left(U\right)$ is more challenging to achieve.

Normality of the distribution given in (14) was also checked through the Kolmogorov-Smirnov (KS) test [29].

Table 1. Null rejection rates of the KS test to detect goodness of fit for the normal distribution approximation of $Z\left(U\right)$ with the indicated values of t and p.

	p
t	0.010	0.106	0.202	0.298	0.394	0.490
20	0.02	0.11	0.07	0.11	0.15	0.51
50	0.02	0.04	0.05	0.08	0.11	0.49
100	0.02	0.06	0.05	0.05	0.08	0.37
250	0.04	0.04	0.06	0.06	0.06	0.25
500	0.07	0.05	0.05	0.05	0.05	0.20

contains the proportion of times that the KS test rejects the null hypothesis of normality. We use a 5% significance level, and then about 5% of these tests should reject the null hypothesis. As expected, the results generally show strong acceptance of the null hypothesis. However, the type I error is inflated for short periods of observation of the source, or p tending to 0.5.

We also compute the coverage probability (CP) of a 95% confidence bound based on the results stated in (17). The CP is computed as

$$\mathrm{CP}\left(\widehat{H}\left(U\right)\right)=\frac{1}{M}\sum _{r=1}^M\mathcal{I}(H\left(U\right)\in [0,H_{\mathrm{N}}^{\left(r\right)}]),$$

(31)

where $\mathcal{I}$ is an indicator function such that $H\left(U\right)$ belongs to the r-th bound $[0,H_{\mathrm{N}}^{\left(r\right)}]$, with $H_{\mathrm{N}}^{\left(r\right)}$ being the r-th upper limit estimates of the 95% confidence bound defined in (17).

Table 2. A CP of a 95% confidence bound for the normal distribution approximation of $Z\left(U\right)$ with the indicated value of p and t.

	p
t	0.010	0.106	0.202	0.298	0.394	0.490
20	0.98	0.89	0.93	0.89	0.85	0.49
50	0.98	0.96	0.95	0.92	0.89	0.51
100	0.98	0.94	0.95	0.95	0.92	0.63
250	0.96	0.96	0.94	0.94	0.94	0.75
500	0.93	0.95	0.95	0.95	0.95	0.80

reports empirical values for the CP defined in (31). From the results in this table, we conclude that the CP approaches the 95% nominal level as the sample size increases, as expected. For example, for $t=20$ and $p=0.202$, the CP is 93%, and for $t=500$ and $p=0.106$, the CP is 95%.

Figure 7 depicts the kernel estimates of the distribution of $T\left(U\right)$ across t and p. The boxplot of $\widehat{F}\left(U\right)$ is displayed in each panel. All densities look roughly with the Student-t distribution (symmetric around zero and fast decay of tails to zero when t is large by the convergence of the Student-t distribution to the normal one). Note that, in each panel, as t increases, the fluctuations of the source stabilize around a value that depends only on p. Then the information of $\widehat{F}\left(U\right)$ provides a scale correction of the statistic given in (15), as expected by the law of large numbers. In short periods, $T\left(U\right)$ is more stable than $Z\left(U\right)$, as expected. The KS test with the Student-t distribution as a reference confirms these conjectures.

Table 3. Null rejection rates of the KS test to detect goodness of fit for the Student-t distribution with $(t-1)$ degrees of freedom with the indicated values of t and p.

	p
t	0.010	0.106	0.202	0.298	0.394	0.490
20	0.82	0.16	0.16	0.11	0.11	0.17
50	0.61	0.06	0.05	0.09	0.11	0.11
100	0.37	0.06	0.06	0.06	0.12	0.08
250	0.09	0.05	0.06	0.06	0.09	0.05
500	0.05	0.05	0.05	0.05	0.07	0.04

contains the proportion of times that the KS test rejects the null hypothesis of a Student-t distribution. We use a 5% significance level. The type I error is also inflated for short-time observation ($t\le 100$) of the source, and p close to 0 or 0.5. As expected, the results generally agree with the null hypothesis.

Table 4. A CP of a 95% confidence bound for the Student-t distribution with $(t-1)$ degrees of freedom of $T\left(U\right)$ for the indicated value of p and t.

	p
t	0.010	0.106	0.202	0.298	0.394	0.490
20	0.18	0.84	0.83	0.89	0.89	0.83
50	0.39	0.94	0.94	0.92	0.88	0.88
100	0.64	0.94	0.94	0.94	0.89	0.92
250	0.91	0.95	0.94	0.95	0.92	0.96
500	0.94	0.95	0.95	0.94	0.94	0.97

reports empirical values for the CP defined in (31) changing the limit $H_{\mathrm{N}}^{\left(r\right)}$ by $H_{\mathrm{t}}^{\left(r\right)}$ stated in (16). From the results in this table, we conclude that the CP approaches the 95% nominal level as the sample size increases, as expected. For example, for $t=50$ and $p=0.202$, the CP is 94%, and for $t=250$ and $p=0.394$, the CP is 92%.

7. Empirical Applications

The first application with real data is related to purity of information. To show the use of information variation, we consider the Iris dataset obtained from [30], which is widely used in pattern recognition. This set can be considered a DMS with three prior symbols ($K=3$), each with 50 observations and a total of $L=150$ instances. The symbols correspond to three species of iris plants: setosa (symbol 1), virginica (symbol 2), and versicolor (symbol 3). For each species, the variables are petal length (PL), petal width (PW), sepal length (SL), and sepal width (SW). The Shannon entropy given in (9) provides a measure of information purity. We take subsets of the Iris data containing only one species. We are interested in identifying how pure each species is, and we assume that the source includes information variation when collecting the data. Because we do not know the true probability distribution of the source, we need to employ the plugin estimators defined in (9) and (10). We intuitively know that we have 100% knowledge of the Iris species because we have only a single species selected. As a result, we must be at the lower bound of entropy without information variation, which means that for each species, we have $\widehat{H}\left(U\right)=0$ and $\widehat{F}\left(U\right)=0,$ respectively. Then, we have complete knowledge of the contents of each set. If we ask to draw one observation at random and predict the species based only on the location of each species, for example, of Iris virginica species, we know that a kind of Iris virginica will be sketched. Nonetheless, in the Iris dataset, we have three species of iris, each representing 1/3 of the dataset. Thus, we expect a high entropy if we include the complete data. If we imagine an urn filled with different-colored balls representing the species of iris, we have less knowledge about what color of the ball (what species) we would draw randomly from the urn. In this case, $\widehat{H}\left(U\right)=1.59$ and $\widehat{F}\left(U\right)=0.$

A second application with real data is associated with wheat prices in Brazil, whose daily time series is available at www.cepea.esalq.usp.br/br/indicador/milho.aspx, accessed on 1 January 2022. The data were observed from 2 February 2002 to 25 October 2019. First, we subtract the original series to get stationarity. The series is flat, oscillating about a mean above zero, and the autocorrelation trails off as one might expect for a stationary series (see Figure 8). Now, we construct a new series of symbols +1 and −1 for detecting changes in prices for the differentiated series, such as in a nonparametric run test [31]. The new series can be considered a binary source that reflects whether the series of differences assumes positive ($+1$: price increase) or negative ($-1$: price decrease) values from wheat prices in Brazil. We compute the Shannon and information variation measures, getting $\widehat{H}\left(U\right)=0.923$ and $\widehat{F}\left(U\right)=0.208$. As our source is binary, we note that the entropy estimate is close to one, indicating that increasing or decreasing prices are practically equiprobable.

Because the observation time is relatively long ($L=5744$), we can utilize the approximation given in (19). In this way, if we consider a confidence level of 95%, we obtain the interval (0.918; 0.922) for the Shannon entropy. The result indicates that, with 95% confidence, our series is typical and $p=P\left(\mathrm{price}\mathrm{increase}\right)\approx 0.5$. This result suggests that it is challenging to predict and detect changes in wheat prices in Brazil so that other methods should be studied. Note that this is simply an illustrative example to show how our new methodology can be used. We do not intend to solve a new problem with our methodology for now. Ideas about cryptography, cryptocompression, and cryptocurrencies, as well as other possible applications, are proposed in the conclusions of the present article related to future research.

8. Concluding Remarks

In this paper, we have proposed and derived a novel approach that includes information variation in the source coding. This approach is more realistic than its standard version. We have employed the Shannon entropy for coding sequences of a source. In addition, the new approach is helpful for short sequences in which the central limit theorem does not have enough support to be applied. We employed a quantifier of information variation for a source related to the variance of the information content of a source symbol. Introducing a new parameter for a discrete memoryless source, namely the information variation, seems to be as essential as the concept of entropy itself. For short sequences, it is not the source’s entropy that should be used to calculate the efficiency of source coders, but rather the sample entropy. In addition to proposing the calculation of a more realistic estimate for the entropy, now depending on the sequence length from the source, a naive and didactic interpretation of typical sequences was presented. We introduced illustrations by assuming a binary discrete memoryless source, in which the output was either zero with a certain probability or one with its complementary probability. In addition, Monte Carlo simulation studies were conducted to evaluate the performance of our approach, providing evidence of the adequate performance of our proposal. Moreover, we have applied our approach to two real datasets related to purity and wheat prices in Brazil.

Potential further applications of our proposal are in assessing the randomness of sequences. Some open questions from the present investigation remain. Is there always a maximum variation in the sample entropy for some particular (nonbinary) probability assignment? The answer needs further developments that we leave for future work. In addition, the usage of other types of estimation methods; statistical moments—robust statistical methods and entropy configurations and sequences, as well as different types of inequalities [32,33,34,35,36,37]—will be considered, and we hope to report their findings in future publications. Similarly, as we have assumed statistical independence in our proposal, the extension to dependent sequences is a good topic for future extension. Moreover, traditional autoregressive models and a copula-based Markov chain are now available to be further considered [38]. We suggest investigating the possibility of incorporating a nonparametric estimation of the Shanon diversity index into our proposal [39]. In this way, we can obtain a reasonable confidence level when bias is suspected in the sample.

Our work may be of particular interest to specialists in the field of cryptography in solving digital security problems [40]. Coding methods and their application in information security systems, cryptocompression systems, and data compression are relevant topics to be considered [10,41]. For example, the application of arithmetic coding methods in cryptographic information protection systems is considered in [42]. Volatile time series and information flow analyses based on cryptocurrency data have been studied in [43]. A new approach to predicting cryptocurrency is stated in [44]. These frameworks open new avenues to apply the results obtained in the present research.