Some Order Preserving Inequalities for Cross Entropy and Kullback–Leibler Divergence

Abstract

Cross entropy and Kullback–Leibler (K-L) divergence are fundamental quantities of information theory, and they are widely used in many fields. Since cross entropy is the negated logarithm of likelihood, minimizing cross entropy is equivalent to maximizing likelihood, and thus, cross entropy is applied for optimization in machine learning. K-L divergence also stands independently as a commonly used metric for measuring the difference between two distributions. In this paper, we introduce new inequalities regarding cross entropy and K-L divergence by using the fact that cross entropy is the negated logarithm of the weighted geometric mean. We first apply the well-known rearrangement inequality, followed by a recent theorem on weighted Kolmogorov means, and, finally, we introduce a new theorem that directly applies to inequalities between K-L divergences. To illustrate our results, we show numerical examples of distributions.

1. Introduction

Cross entropy and Kullback–Leibler (K-L) divergence are applied to and discussed in studies on machine learning [1], visualization (e.g., to interpret the pipeline of visualization using the cost–benefit model [2] and several other applications [3]), computer graphics [4], and many other fields. Cross entropy and K-L divergence are two fundamental quantities of information theory. Cross entropy gives the average code length needed to represent one distribution by another distribution, and the excess code needed over the optimal coding is given by the K-L divergence. As cross entropy is just the negated logarithm of likelihood, minimizing cross entropy means maximizing likelihood, and thus, cross entropy is widely used now for optimization in machine learning. K-L divergence also stands independently as a widely used metric for measuring the difference between two distributions, and it appears in many theoretical results. As examples, the definition of mutual information between two variables is a K-L divergence, and the absolute difference of K-L divergences with respect to a third distribution was recently used to define a family of metrics [5].

Recently, order invariance for inequalities between means was presented [6]. Since likelihood is the weighted geometric mean of the representing distribution weighted by the data distribution, likelihood and thus cross entropy will share the order invariance properties too. Order invariance properties are characterized by the stochastic order between weights or data distributions [7]. Here, we further develop this idea in order to compare representations of different data by the same distribution, and we extend it to a new order, K-L dominance, that allows us to define inequalities between representations of the same data by different distributions. Finally, we establish the connection between the two orders. We also give numerical examples to illustrate the theoretical results.

This paper is organized as follows. After this introduction, in Section 2, we recapitulate the meaning of cross entropy and its relationship to likelihood and K-L divergence, and we obtain the first theoretical results based on the rearrangement inequality. Next, in Section 3, we apply the invariance results for weighted means to cross entropy. In Section 4, we introduce a new order, K-L dominance, that allows us to establish inequalities between K-L divergences and give the relationship between stochastic and K-L order. Finally, in Section 5, we present our conclusions and future work.

2. Cross Entropy as a Measure of Goodness of Representation

Consider two probability distributions, ${\left\{{\alpha }_k\right\}}_{k=1}^n$, ${\left\{x_k\right\}}_{k=1}^n$, for all k ${\alpha }_k\ge 0$, $x_k>0$, ${\sum }_k{\alpha }_k={\sum }_k{x}_k=1$ (when not explicit, the sum limits will be understood to be between 1 and n). Cross entropy $CE({\alpha }_k,x_k)$ (to avoid cluttered notation we will write $CE({\alpha }_k,x_k)$ instead of $CE(\left\{{\alpha }_k\right\},\left\{x_k\right\})$) is defined as $CE({\alpha }_k,x_k)={\sum }_k{\alpha }_{k}log\frac{1}{x_k}=-{\sum }_k{\alpha }_{k}log{x}_k$ and can be written as $CE({\alpha }_k,x_k)=H\left({\alpha }_k\right)+D_{\mathrm{KL}}({\alpha }_k,x_k)$ (similar notation to $CE({\alpha }_k,x_k)$), where $H\left({\alpha }_k\right)=-{\sum }_k{\alpha }_{k}log{\alpha }_k$ is the entropy of distribution $\left\{{\alpha }_k\right\}$, and $D_{\mathrm{KL}}({\alpha }_k,x_k)={\sum }_k{\alpha }_{k}log\frac{{\alpha }_k}{x_k}\ge 0$ is the Kullback–Leibler divergence of $\left\{{\alpha }_k\right\}$ and $\left\{x_k\right\}$ distributions. It represents the average length of code per symbol needed to represent $\left\{{\alpha }_k\right\}$ using $\left\{x_k\right\}$. In the context of coding, we do not let any component of $\left\{x_k\right\}$ be zero, as the corresponding component of $\left\{{\alpha }_k\right\}$ would not be coded; thus, $D_{\mathrm{KL}}({\alpha }_k,x_k)$ is well defined. The minimum code length to code $\left\{{\alpha }_k\right\}$ is obtained by taking $\left\{{\alpha }_k\right\}\equiv \left\{x_k\right\}$, because $D_{\mathrm{KL}}({\alpha }_k,{\alpha }_k)=0$. Minimum code length will be between $H\left({\alpha }_k\right)$ and $H\left({\alpha }_k\right)$ + 1 bit (Huffman coding, [8]).

Cross Entropy and Likelihood

Observe that cross entropy is the negated logarithm of likelihood. Let us suppose a distribution with n states, which are guessed to be $\{x_1,x_2,\dots ,x_n\}$, and a relative frequency of realizations $\{{\alpha }_1,{\alpha }_2,\dots ,{\alpha }_n\}$ of the different states. That is, we guess that the distribution $\left\{x_k\right\}$ represents the collected data $\left\{{\alpha }_k\right\}$. Likelihood is then used as a measure of how well the distribution represents the data. The likelihood $\mathcal{L}({\alpha }_k,x_k)$ of $\left\{x_k\right\}$ is the true distribution of the data with relative frequency $\left\{{\alpha }_k\right\}$, and $\mathcal{L}=x_1^{{\alpha }_1}{x}_2^{{\alpha }_2}\dots x_n^{{\alpha }_n}={\mathrm{\Pi }}_k{x}_k^{{\alpha }_k}$. Observe that $\mathcal{L}({\alpha }_k,x_k)$ is the weighted geometric mean of $\{x_1,x_2,\dots ,x_n\}$ with weights $\{{\alpha }_1,{\alpha }_2,\dots ,{\alpha }_n\}$. The likelihood represents the probability of $\left\{x_k\right\}$ being the true distribution of the known data $\left\{{\alpha }_k\right\}$. As $CE({\alpha }_k,x_k)=-log\mathcal{L}({\alpha }_k,x_k)$, maximizing the likelihood is equivalent to minimizing the cross entropy.

Now, let us suppose we take an arbitrary $\left\{x_k\right\}$ distribution to code $\left\{{\alpha }_k\right\}$. Without loss of generality, we will consider two or more sequences to be equally ordered, or in the same order, when, by the same permutation of indexes, we can make it such that all the sequences are increasing. We can state the following theorem.

Theorem 1.

Consider the distribution $\left\{{\alpha }_k\right\}$ and the sequence of strictly positive numbers $\{x_1,x_2,\dots ,x_n\}$, ${\sum }_k{x}_k=1$. When $\{x_1,x_2,\dots ,x_n\}$ is permuted so that $\left\{x_k\right\}$ is ordered in the same order as $\left\{{\alpha }_k\right\}$, then the cross entropy $CE({\alpha }_k,x_k)$ takes a global minimum, and, correspondingly, the likelihood $\mathcal{L}({\alpha }_k,x_k)$ takes a global maximum.

Proof. 

Using the rearrangement inequality [6,9] and the definition of cross entropy, the minimum of the sum of the product ${\mathrm{\Pi }}_k{\alpha }_{k}log\frac{1}{x_k}$ is realized when the two sequences are ordered inversely to each other. That means the $\{log{x}_k\}$ sequence will be ordered as the $\left\{{\alpha }_k\right\}$ sequence, and, by the monotonicity of the logarithmic function, the $\left\{x_k\right\}$ sequence will be ordered as the $\left\{{\alpha }_k\right\}$ sequence. □

As $\left\{{\alpha }_k\right\}$, and thus $H\left({\alpha }_k\right)$, is fixed, $D_{\mathrm{KL}}({\alpha }_k,x_k)$ also takes the global minimum when $\left\{x_k\right\}$ is ordered as $\left\{{\alpha }_k\right\}$. Also, when there are elements repeated in $\left\{{\alpha }_k\right\}$, there can be non-ordered $\left\{x_k\right\}$ distributions with the same minimum cross entropy value. The extreme case is when ${\alpha }_k=1/n$ for all k, and all orderings of $\left\{x_k\right\}$ will give the same result.

Observe that by Theorem 1, if we have a series of distributions ${\left\{{\alpha }_{t,k}\right\}}_{t=1}^m$ that are all ordered in the same order, which we can consider to be increasing, and want to represent them with a single distribution $\left\{x_k\right\}$—to be chosen by permuting the elements in $\{x_1,x_2,\dots ,x_n\}$—then the best distribution will be the one for which we select the elements in $\{x_1,x_2,\dots ,x_n\}$ in increasing order too.

Example 1.

For any increasing sequence $\left\{{\alpha }_k\right\}$, we have that

1. 

$CE(\left\{{\alpha }_k\right\},\{2/9,3/9,4/9\})\le CE(\left\{{\alpha }_k\right\},\{4/9,3/9,2/9\}),$

2. 

$CE(\left\{{\alpha }_k\right\},\{2/9,3/9,4/9\})\le CE(\left\{{\alpha }_k\right\},\{4/9,2/9,3/9\}),$ and so on.

However, what happens when we have to choose between two distributions, $\left\{x_k\right\}$ and $\left\{x_k^{\prime }\right\}$, to represent a set of increasing distributions, ${\left\{{\alpha }_{t,k}\right\}}_{t=1}^m$? Can we find which one is better? Section 4 provides some answers to this. Before answering this question, we introduce the concept of stochastic dominance in Section 3.

3. First Stochastic Order Dominance

When for all increasing distributions $\left\{x_k\right\}:{\sum }_k{\alpha }_k^{\prime }{x}_k\le {\sum }_k{\alpha }_k{x}_k$, we say that $\left\{{\alpha }_k\right\}$ stochastically dominates $\left\{{\alpha }_k^{\prime }\right\}$, and we denote this relation as $\left\{{\alpha }_k\right\}{\succ }_{st}\left\{{\alpha }_k^{\prime }\right\}$ [7]. This is a partial order on the set of distributions.

The necessary and sufficient conditions for a sequence $\left\{{\alpha }_k\right\}$ to stochastically dominate another one $\left\{{\alpha }_k^{\prime }\right\}$ are given in the following theorem [6].

Theorem 2.

Given that the sequences $\left\{{\alpha }_k\right\}$ and $\left\{{\alpha }_k^{\prime }\right\}$ are positive and add up to 1, the following conditions are equivalent:

(1) 

The sequence $\left\{{\alpha }_k\right\}$ stochastically dominates the sequence $\left\{{\alpha }_k^{\prime }\right\}$

(2) 

$$\begin{array}{ccc}\hfill {\alpha }_1^{\prime }& \ge & {\alpha }_1\hfill \\ \hfill {\alpha }_1^{\prime }+{\alpha }_2^{\prime }& \ge & {\alpha }_1+{\alpha }_2\hfill \\ \hfill \dots & \ge & \dots \hfill \\ \hfill {\alpha }_1^{\prime }+{\alpha }_2^{\prime }+\dots +{\alpha }_n^{\prime }& =& {\alpha }_1+{\alpha }_2+\dots +{\alpha }_n\hfill \end{array}$$

(1)

(3) 

$$\begin{array}{ccc}\hfill {\alpha }_n^{\prime }& \le & {\alpha }_n\hfill \\ \hfill {\alpha }_n^{\prime }+{\alpha }_{n-1}^{\prime }& \le & {\alpha }_n+{\alpha }_{n-1}\hfill \\ \hfill \dots & \le & \dots \hfill \\ \hfill {\alpha }_n^{\prime }+{\alpha }_{n-1}^{\prime }+\dots +{\alpha }_1^{\prime }& =& {\alpha }_n+{\alpha }_{n-1}+\dots +{\alpha }_1\hfill \end{array}$$

(2)

(4) 

For all increasing sequences $\left\{x_k\right\}$ and for any $f\left(x\right)$ strictly monotonous function, $f^{-1}\left({\sum }_k{\alpha }_k^{\prime }f\left(x_k\right)\right)\le f^{-1}\left({\sum }_k{\alpha }_{k}f\left(x_k\right)\right)$ (quasi-arithmetic, or Kolmogorov, mean), whenever $f\left(x\right)$ is applicable.

(5) 

There exists a strictly monotonous function $f\left(x\right)$ such that for all increasing sequences $\left\{x_k\right\}$:

$$f^{-1}\left(\sum _k{\alpha }_k^{\prime }f\left(x_k\right)\right)\le f^{-1}\left(\sum _k{\alpha }_{k}f\left(x_k\right)\right)$$

(3)

(quasi-arithmetic, or Kolmogorov, mean), whenever $f\left(x\right)$ is applicable.

A sufficient condition for $\left\{{\alpha }_k\right\}{\succ }_{st}\left\{{\alpha }_k^{\prime }\right\}$ is given by the following theorem [6,10].

Theorem 3.

Given the distributions $\left\{{\alpha }_k\right\},\left\{{\alpha }_k^{\prime }\right\}$, a sufficient condition for $\left\{{\alpha }_k\right\}{\succ }_{st}\left\{{\alpha }_k^{\prime }\right\}$ is that whenever $i\le j$, then $\frac{{\alpha }_i^{\prime }}{{\alpha }_j^{\prime }}\ge \frac{{\alpha }_i}{{\alpha }_j}$.

By Theorem 3, if $\left\{{\alpha }_k\right\}$ is increasing, then $\left\{{\alpha }_k\right\}{\succ }_{st}\{1/n\}$, where $\{1/n\}$ is the uniform distribution; if $\left\{{\alpha }_k\right\}$ is decreasing, then $\{1/n\}{\succ }_{st}\left\{{\alpha }_k\right\}$; if $\left\{{\alpha }_k\right\}$ is increasing and $\left\{{\alpha }_k^{\prime }\right\}$ is decreasing, then $\left\{{\alpha }_k\right\}{\succ }_{st}\left\{{\alpha }_k^{\prime }\right\}$.

Observe now that, by condition (5) of Theorem 2, if Equation (3) applies for one strictly monotonous function, then, for the equivalence between conditions (4) and (5), it applies for any other strictly monotonous function. Then, by the definition of cross-entropy,

Theorem 4.

Given two distributions $\left\{{\alpha }_k\right\}$ and $\left\{{\alpha }_k^{\prime }\right\}$, $\left\{{\alpha }_k\right\}{\succ }_{st}\left\{{\alpha }_k^{\prime }\right\}$ if and only if for all increasing distributions $\left\{x_k\right\}$, $CE({\alpha }_k,x_k)\le CE({\alpha }_k^{\prime },x_k)$ (or $\mathcal{L}({\alpha }_k,x_k)\ge \mathcal{L}({\alpha }_k^{\prime },x_k)$).

Theorem 4 means that, if $\left\{{\alpha }_k\right\}{\succ }_{st}\left\{{\alpha }_k^{\prime }\right\}$, then, taking all possible increasing representations $\left\{x_k\right\}$, the code length for $\left\{{\alpha }_k\right\}$ will be shorter than for $\left\{{\alpha }_k^{\prime }\right\}$, and the likelihood for $\left\{{\alpha }_k\right\}$ will be bigger than the likelihood for $\left\{{\alpha }_k^{\prime }\right\}$. Observe also that if $\left\{{\alpha }_k\right\}$ is an increasing sequence, then $\left\{{\alpha }_k\right\}{\succ }_{st}\{1/n\}$; thus, the representation of $\left\{{\alpha }_k\right\}$ by an increasing distribution $\left\{x_k\right\}$ will be shorter than the representation of uniform distribution $\{1/n\}$, and the likelihood of $\left\{{\alpha }_k\right\}$ will be bigger than that of $\{1/n\}$.

Corollary 1.

For any increasing distributions $\left\{x_k\right\}$ and $\left\{{\alpha }_k\right\}$, we have that $CE({\alpha }_k,x_k)\le CE(1/n,x_k)$ (or $\mathcal{L}({\alpha }_k,x_k)\ge \mathcal{L}(1/n,x_k))$.

Indeed, the above result is valid too if, by an index permutation, we can bring $\left\{x_k\right\}$, $\left\{{\alpha }_k\right\}$ to be in the same order. Thus,

Corollary 2.

For any distributions $\left\{x_k\right\}$ and $\left\{{\alpha }_k\right\}$ that are ordered equally, we have that $CE({\alpha }_k,x_k)\le CE(1/n,x_k)$.

Now, suppose $\left\{{\alpha }_k\right\}$ is decreasing; then, $\{1/n\}{\succ }_{st}\left\{{\alpha }_k\right\}$. Also, for any distribution $\left\{x_k\right\}$, $CE(1/n,x_k)\ge CE(1/n,1/n)=logn$, and thus, the following corollary,

Corollary 3.

For any increasing distribution $\left\{x_k\right\}$ and decreasing distribution $\left\{{\alpha }_k\right\}$, we have that $logn\le CE(1/n,x_k)\le CE({\alpha }_k,x_k)$.

Also, by reordering the indexes, Corollary 3 can be extended to

Corollary 4.

For any distributions $\left\{x_k\right\}$ and $\left\{{\alpha }_k\right\}$ that are inversely ordered to each other, we have that $logn\le CE(1/n,x_k)\le CE({\alpha }_k,x_k)$.

Observe now that if $\left\{{\alpha }_k\right\}$ is increasing and $\left\{{\alpha }_k^{\prime }\right\}$ is decreasing, then, $\left\{{\alpha }_k\right\}{\succ }_{st}\left\{{\alpha }_k^{\prime }\right\}$, and thus,

Corollary 5.

For any increasing distribution $\left\{{\alpha }_k\right\}$ and decreasing distribution $\left\{{\alpha }_k^{\prime }\right\}$, we have that for any increasing distribution $\left\{x_k\right\}$, $CE({\alpha }_k,x_k)\le CE({\alpha }_k^{\prime },x_k)$.

Corollary 5 is also a direct consequence of Corollaries 1 and 3.

Example 2.

We have:

1. 

$\{1/4,1/4,1/2\}{\succ }_{st}\{1/3,1/3,1/3\}$

2. 

$\{1/2,1/4,1/4\}{\cancel{\succ }}_{st}\{1/3,1/3,1/3\}$

3. 

$\{1/3,1/3,1/3\}{\succ }_{st}\{1/2,1/4,1/4\}$

4. 

$\{1/2,1/4,1/4\}{\cancel{\succ }}_{st}\{1/5,3/5,1/5\}$

5. 

$\{1/5,3/5,1/5\}{\cancel{\succ }}_{st}\{1/2,1/4,1/4\}$

In Figure 1, we plot the sums in Equation (1) corresponding to relation 1 and relations 2–3, 4–5 in Example 2. Observe that the sums in Equation (1) correspond to the cumulative distribution function (cdf). A first-order stochastic dominance implies that there is no crossing of the respective cdf’s.

Discussion

We show above how the interpretation of cross entropy as a weighted mean allows for obtaining interesting order invariance results for the cross entropy of two data distributions represented by the same distribution (correspondingly for likelihood). Since, in our results, the distribution of the data is variable while the representative distribution is fixed, the resulting cross entropy depends on both entropy and K-L divergence. In the next section, we consider the data fixed and the representative distribution variable; thus, the comparison of cross entropies of a given data distribution for two representative distributions will be equivalent to the comparison of their K-L divergences.

4. A New Partial Order: K-L Dominance

We extend here the results of Section 2. We first define a new partial order between distributions and call it K-L dominance.

Definition 1.

K-L dominance. We say that distribution $\left\{x_k\right\}$ K-L-dominates distribution $\left\{x_k^{\prime }\right\}$, written symbolically as $\left\{x_k\right\}{\succ }_{\mathrm{KL}}\left\{x_k^{\prime }\right\}$, when, for all increasing distributions $\left\{{\alpha }_k\right\}$, $D_{\mathrm{KL}}({\alpha }_k,x_k)\le D_{\mathrm{KL}}({\alpha }_k,x_k^{\prime })$.

Observe that $\left\{x_k\right\}{\succ }_{\mathrm{KL}}\left\{x_k^{\prime }\right\}\cancel{\iff }\left\{x_k\right\}{\succ }_{st}\left\{x_k^{\prime }\right\}$, because, in general, $D_{\mathrm{KL}}({\alpha }_k,x_k)\le D_{\mathrm{KL}}({\alpha }_k,x_k^{\prime })\cancel{\iff }{D}_{\mathrm{KL}}(x_k,{\alpha }_k)\le D_{\mathrm{KL}}(x_k^{\prime },{\alpha }_k)$.

From the definition of cross entropy, $\left\{x_k\right\}{\succ }_{\mathrm{KL}}\left\{x_k^{\prime }\right\}$ is equivalent to—for all increasing $\left\{{\alpha }_k\right\}$—$CE({\alpha }_k,x_k)\le CE({\alpha }_k,x_k^{\prime })$, and thus, for all increasing $\left\{{\alpha }_k\right\}$ sequences, coding them with $\left\{x_k\right\}$ will always generate a shorter codification than coding them with $\left\{x_k^{\prime }\right\}$.

The following theorem holds:

Theorem 5.

A necessary and sufficient condition for distributions $\left\{x_k\right\}$, $\left\{x_k^{\prime }\right\}$ to hold $\left\{x_k\right\}{\succ }_{\mathrm{KL}}\left\{x_k^{\prime }\right\}$ (equivalent to—for all increasing distributions $\left\{{\alpha }_k\right\}$—$D_{\mathrm{KL}}({\alpha }_k,x_k)\le D_{\mathrm{KL}}({\alpha }_k,x_k^{\prime })$, or $CE({\alpha }_k,x_k)\le CE({\alpha }_k,x_k^{\prime })$) or $\mathcal{L}({\alpha }_k,x_k)\ge \mathcal{L}({\alpha }_k,x_k^{\prime })$ is the following:

$$\begin{array}{ccc}\hfill x_n^{\prime }& \le & x_n\hfill \\ \hfill x_n^{\prime }{x}_{n-1}^{\prime }& \le & x_n{x}_{n-1}\hfill \\ \hfill \cdots & \le & \cdots \hfill \\ \hfill x_n^{\prime }{x}_{n-1}^{\prime }\dots x_1^{\prime }& \le & x_n{x}_{n-1}\dots x_1\hfill \end{array}$$

(4)

Observe that Theorem 1 is a particular case of Theorem 5, as a sequence rearranged in increasing order will always fill the second members of inequalities in Equation (4) with respect to all other rearrangements. Thus, an increasing distribution $\left\{x_k\right\}$ K-L-dominates all the distributions obtained by permuting the values of $\left\{x_k\right\}$. Observe also that Equation (4) condition makes the ${\succ }_{\mathrm{KL}}$ order reflexive, antisymmetric, and transitive, and thus, it is a partial order. To prove Theorem 5, let us consider first the following lemma:

Lemma 1.

Consider the sequences of n numbers $\left\{w_k\right\}$ and $\left\{y_k\right\}$. Then, conditions (1) and (2) are equivalent:

(1) 

for any sequence of n positive numbers in increasing order $\left\{z_k\right\}$, the following inequality holds:

$$\sum _{k=1}^n{w}_k{z}_k\le \sum _{k=1}^n{y}_k{z}_k,$$

(5)

(2) 

(2) the following inequalities hold:

$$\begin{array}{ccc}\hfill w_n& \le & y_n\hfill \\ \hfill w_n+w_{n-1}& \le & y_n+y_{n-1}\hfill \\ \hfill \cdots & \le & \cdots \hfill \\ \hfill w_n+\dots +w_2& \le & y_n+\dots +y_2\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill w_1+\dots +w_{n-1}+w_n& \le & y_1+\dots +y_{n-1}+y_n.\hfill \end{array}$$

(7)

Proof. 

That $\left(1\right)\Rightarrow \left(2\right)$ is immediate, considering, respectively, the $z_k$ sequences $\{0,0,\dots ,1\},\{0,0,\dots ,1,1\},\dots ,\{0,1,\dots ,1\},\{1,1,\dots ,1\}$. Let us see now that $\left(2\right)\Rightarrow \left(1\right)$: Define for $1\le n$, $A_k={\sum }_{j=1}^k{y}_{n-j+1},A_k^{\prime }={\sum }_{j=1}^k{w}_{n-j+1}$, and $A_0=A_0^{\prime }=0$, $z_0=0$. Observe that condition (2) is equivalent to—for all k—$A_k-A_k^{\prime }\ge 0$. Observe also that the $\left\{z_k\right\}$ sequence augmented with $z_0$ is still increasing. We have

$$\begin{array}{ccc}\hfill & & \sum _{k=1}^n{y}_{n-k+1}{z}_{n-k+1}-\sum _{k=1}^n{w}_{n-k+1}{z}_{n-k+1}\hfill \\ \hfill & =& \sum _{k=1}^n(y_{n-k+1}-w_{n-k+1})z_{n-k+1}\hfill \\ \hfill & =& \sum _{k=1}^n(A_k-A_{k-1}-{A^{\prime }}_k+{A^{\prime }}_{k-1})z_{n-k+1}\hfill \\ \hfill & =& \sum _{k=1}^n(A_k-{A^{\prime }}_k)z_{n-k+1}-\sum _{k=1}^n(A_{k-1}-{A^{\prime }}_{k-1})z_{n-k+1}\hfill \\ \hfill & =& \sum _{k=1}^n(A_k-{A^{\prime }}_k)z_{n-k+1}-\sum _{k=0}^{n-1}(A_k-{A^{\prime }}_k)z_{n-k}\hfill \\ \hfill & =& \sum _{k=1}^n(A_k-{A^{\prime }}_k)z_{n-k+1}-\sum _{k=1}^n(A_k-{A^{\prime }}_k)z_{n-k}\hfill \\ \hfill & =& \sum _{k=1}^n(A_k-{A^{\prime }}_k)(z_{n-k+1}-z_{n-k})\ge 0,\hfill \end{array}$$

(8)

as, by hypothesis for all k, $A_k-A_k^{\prime }\ge 0$, and $\left\{z_k\right\}$ is an increasing sequence. □

The proof of Theorem 5 follows immediately from applying Lemma 1 to the inequality ${\sum }_k{\alpha }_{k}log{x}_k^{\prime }\le {\sum }_k{\alpha }_{k}log{x}_k$.

The following corollary follows from observing that, by taking $\left\{{\alpha }_k\right\}\equiv \left\{x_k\right\}$, then, $D_{\mathrm{KL}}(x_k,x_k)=0<D_{\mathrm{KL}}(x_k,x_k^{\prime })$, $\left\{x_k^{\prime }\right\}\cancel{\equiv }\left\{x_k\right\}$.

Corollary 6.

An increasing distribution $\left\{x_k\right\}$ cannot be K-L dominated by any other distribution (except trivially by itself). In particular, the uniform distribution $\left\{x_k\right\}\equiv \{1/n\}$ does not dominate any other increasing distribution.

Observe now that the product $1/n^n$ is always bigger than or equal ${\mathrm{\Pi }}_k{x}_k$, as the maximum value for ${\mathrm{\Pi }}_k{x}_k$ subjected to the condition ${\sum }_k{x}_k=1$ is, for all k, $x_k=1/n$. Thus, from the last inequality in Equation (4), we have the following corollary,

Corollary 7.

For all distributions $\left\{x_k\right\}$, $\left\{x_k\right\}\cancel{\equiv }\{1/n\}$, we have that $\left\{x_k\right\}{\cancel{\succ }}_{\mathrm{KL}}\{1/n\}$.

Observe that Corollary 7 is also a particular case of Corollary 6, considering $\{1/n\}$ as an increasing distribution.

Corollary 8.

The uniform distribution $\{1/n\}$ K-L-dominates all decreasing distributions.

Proof. 

For all $\left\{{\alpha }_k\right\}$ increasing and $\left\{x_k\right\}$ decreasing, we apply Corollary 4, and we have $CE({\alpha }_k,1/n)=logn\le CE({\alpha }_k,x_k)$. ☐

Example 3.

Some examples are:

1. 

$\{1/3,1/3,1/3\}{\succ }_{\mathrm{KL}}\{4/9,3/9,2/9\}$

2. 

$\{1/3,1/3,1/3\}{\succ }_{\mathrm{KL}}\{7/9,1/9,1/9\}$

3. 

$\{1/3,1/3,1/3\}{\cancel{\succ }}_{\mathrm{KL}}\{1/9,1/9,7/9\}$

4. 

$\{1/3,1/3,1/3\}{\succ }_{\mathrm{KL}}\{4/9,2/9,3/9\}$

5. 

$\{1/3,1/3,1/3\}{\cancel{\succ }}_{\mathrm{KL}}\{3/9,2/9,4/9\}$

6. 

$\{1/4,1/4,1/2\}{\succ }_{\mathrm{KL}}\{1/4,1/2,1/4\}$

7. 

$\{1/2,1/4,1/4\}{\succ }_{\mathrm{KL}}\{5/8,2/8,1/8\}$

The first and second relation are examples of Corollary 8. The third is an example of Corollary 6. The sixth is an example of Theorem 1. The fourth and fifth relations mean that the uniform distribution can both K-L dominate and non-dominate non-monotonous sequences, and we have to look for each case at the Condition (4) of Theorem 5. The seventh relation tells us that a decreasing sequence can dominate another decreasing sequence. In Figure 2, we plot the logarithm of products in Equation (4) for relations 1, 3, 6, and 7, respectively. If a sequence K-L-dominates another one, the plots do not cross, and its plot appears over that of the second sequence. Inversely, no dominance means that the plots will cross.

4.1. Discussion

Remember that the equivalence between cross entropy and likelihood means that the likelihood that $\left\{x_k\right\}$ is the true distribution, given the data distribution $\left\{{\alpha }_k\right\}$, and the average length of code using $\left\{x_k\right\}$ to represent $\left\{{\alpha }_k\right\}$ are related by the second quantity being the negated logarithm of the first. If you have one distribution $\left\{{\alpha }_k\right\}$ and have the choice to use $\left\{x_{1,k}\right\}$ or $\left\{x_{2,k}\right\}$ to represent it, you want to choose the one with the minimum length of code, or, alternatively, the one with maximum likelihood. In both cases, the best is the one that gives minimum Kullback–Leibler divergence. Suppose now you want to represent data $\left\{{\alpha }_k\right\}$ and have two alternative distributions to represent it, either $\left\{x_{1,k}\right\}\equiv \{1/n\}$, the uniform distribution, or $\left\{x_{2,k}\right\}$, which is ordered in the same order as $\left\{{\alpha }_k\right\}$. To fix ideas and without loss of generality, let us consider that both $\left\{{\alpha }_k\right\}$ and $\left\{x_{2,k}\right\}$ are in increasing order. It would seem reasonable to represent the data $\left\{{\alpha }_k\right\}$, which is increasing, as distribution $\left\{x_{2,k}\right\}$, which is increasing too; thus, the order is preserved. However, that would work as long as $D_{\mathrm{KL}}({\alpha }_k,x_{2,k})<D_{\mathrm{KL}}({\alpha }_k,1/n)$; otherwise, we should choose $\left\{x_{1,k}\right\}$, the uniform distribution. This is clear from corollary 7, which tells us that, for any sequence, particularly $\left\{x_{2,k}\right\}$, there will always be increasing $\left\{{\alpha }_k\right\}$ sequences such that $D_{\mathrm{KL}}({\alpha }_k,1/n)<D_{\mathrm{KL}}({\alpha }_k,x_{2,k})$. Intuitively, if $\left\{{\alpha }_k\right\}$ has a high increasing gradient, then we would choose $\left\{x_{2,k}\right\}$, the increasing distribution to represent it, but if $\left\{{\alpha }_k\right\}$ becomes smoother, then we would choose $\left\{x_{1,k}\right\}$, the uniform distribution.

As an example, suppose the two distributions $\{0.3,0.7\},\{0.4,0.6\}$ have to be represented by one of the distributions $\{0.33,0.67\},\{0.5,0.5\}$. To preserve the order, we would choose $\{0.33,0.67\}$ for both. It is intuitively a clear choice of $\{0.3,0.7\}$ (and a correct one, as it has a K-L divergence, half the one for $\{0.5,0.5\}$), but we might have some doubts for $\{0.4,0.6\}$, where, actually, the K-L divergence is only slightly smaller for $\{0.5,0.5\}$. To represent distributions $\{0.43,0.57\},\{0.45,0.55\}$, it is intuitively more clear to select the uniform distribution $\{0.5,0.5\}$ rather than $\{0.33,0.67\}$, even if order is lost. In fact, the K-L divergence from both distributions to $\{0.33,0.67\}$ is one and two orders of magnitude greater, respectively, than to the uniform distribution $\{0.5,0.5\}$.

4.2. Relationship between K-L Dominance and First Stochastic Dominance Orders

Summarizing K-L dominance and first stochastic dominance orders, we have

• $\left\{{\alpha }_k\right\}{\succ }_{st}\left\{{\alpha }_k^{\prime }\right\}\leftrightarrow \forall x_k\mathrm{increasing}CE({\alpha }_k,x_k)\le CE({\alpha }_k^{\prime },x_k)$
• $\left\{x_k\right\}{\succ }_{\mathrm{KL}}\left\{x_k^{\prime }\right\}\leftrightarrow \forall {\alpha }_k\mathrm{increasing}CE({\alpha }_k,x_k)\le CE({\alpha }_k,x_k^{\prime })$

Is there a relationship between the two orders? When ${\mathrm{\Pi }}_k{x}_k={\mathrm{\Pi }}_k{x}_k^{\prime }$ or, equivalently, ${\sum }_{k}log{x}_k={\sum }_{k}log{x}_k^{\prime }$ (observe that Theorem 1 is a particular case), we can find a necessary and sufficient condition for $\left\{x_k\right\}{\succ }_{KL}\left\{x_k^{\prime }\right\}$.

Theorem 6.

Given distributions $\left\{x_k\right\}$, $\left\{x_k^{\prime }\right\}$, and ${\mathrm{\Pi }}_k{x}_k={\mathrm{\Pi }}_k{x}_k^{\prime }$, then the following conditions are equivalent:

(1) 

$$\left\{x_k\right\}{\succ }_{\mathrm{KL}}\left\{x_k^{\prime }\right\}$$

(9)

(2) 

$$\left\{\frac{-log{x}_k^{\prime }}{-{\sum }_{k}log{x}_k^{\prime }}\right\}{\succ }_{st}\left\{\frac{-log{x}_k}{-{\sum }_{k}log{x}_k}\right\}$$

(10)

Proof. 

Observe first that $\left\{\frac{-log{x}_k^{\prime }}{-{\sum }_{k}log{x}_k^{\prime }}\right\},\left\{\frac{-log{x}_k}{-{\sum }_{k}log{x}_k}\right\}$ are also distributions (we can indeed use the simpler expressions $\left\{\frac{log{x}_k^{\prime }}{{\sum }_{k}log{x}_k^{\prime }}\right\},\left\{\frac{log{x}_k}{{\sum }_{k}log{x}_k}\right\}$, but, here, we have left the negated ones to remind the reader that both numerator and denominator are negative), because they are positive sequences adding to 1. Observe also that when ${\mathrm{\Pi }}_k{x}_k={\mathrm{\Pi }}_k{x}_k^{\prime }$, taking the logarithms in Equation (4) and changing the sign (which changes the direction of inequalities), we obtain condition (3) in Theorem 2, which is necessary and sufficient for $\left\{\frac{-log{x}_k^{\prime }}{-{\sum }_{k}log{x}_k^{\prime }}\right\}{\succ }_{st}\left\{\frac{-log{x}_k}{-{\sum }_{k}log{x}_k}\right\}$. □

Example 4.

Consider the distributions $\{1/4,1/4,1/2\},\{1/4,1/2,1/4\}$ and their normalized negated logarithms $\{2/5,2/5,1/5\},\{2/5,1/5,2/5\}$, respectively. We have that $\{1/4,1/4,1/2\}{\succ }_{KL}\{1/4,1/2,1/4\}$ and $\{2/5,1/5,2/5\}{\succ }_{st}\{2/5,2/5,1/5\}$.

5. Conclusions and Future Work

In this paper, we present new inequalities for cross entropy and Kullback–Leibler divergence. As cross entropy is the negated logarithm of likelihood, the inequalities also hold for likelihood, changing the sense of the inequality. First, we applied to cross entropy the rearrangement inequality and recent stochastic order invariance results for Kolmogorov weighted means, as likelihood is a weighted geometric mean. Then, we introduced another partial order, K-L dominance, that applies directly to K-L divergences, and we give the relationship between both orders.

In data retrieval, exploration, analysis, and visualization, sorting some data objects into an ordered list or display is a common task performed by users. In the past, the benefit of having a sorted list has been typically articulated qualitatively. We plan to use the theorems presented in this paper to help establish an information-theoretic explanation about the cost–benefit of ordering in data retrieval, exploration, analysis, and visualization. We will also investigate the application to machine learning. We will study whether K-L-dominance order supports invariance properties, in the same way as first stochastic order supports the order invariance for weighted means, and we will look for sufficient conditions for K-L-dominance order, similar to the one in Theorem 3 for first stochastic order. Finally, we will investigate extensions of K-L dominance order: for instance, when $\left\{{\alpha }_k\right\}\equiv \left\{\frac{f\left(k\right)}{{\sum }_{k}f\left(k\right)}\right\}$, $f\left(x\right)$ increasing and concave/convex, in the same way as the extensions of first stochastic order.