# Some Order Preserving Inequalities for Cross Entropy and Kullback–Leibler Divergence

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Cross Entropy as a Measure of Goodness of Representation

#### Cross Entropy and Likelihood

**Theorem**

**1.**

**Proof.**

**Example**

**1.**

- 1.
- $CE(\left\{{\alpha}_{k}\right\},\{2/9,3/9,4/9\})\le CE(\left\{{\alpha}_{k}\right\},\{4/9,3/9,2/9\}),$
- 2.
- $CE(\left\{{\alpha}_{k}\right\},\{2/9,3/9,4/9\})\le CE(\left\{{\alpha}_{k}\right\},\{4/9,2/9,3/9\}),$ and so on.

## 3. First Stochastic Order Dominance

**Theorem**

**2.**

- (1)
- The sequence $\left\{{\alpha}_{k}\right\}$ stochastically dominates the sequence $\left\{{\alpha}_{k}^{\prime}\right\}$
- (2)
- $$\begin{array}{ccc}\hfill {\alpha}_{1}^{\prime}& \ge & {\alpha}_{1}\hfill \\ \hfill {\alpha}_{1}^{\prime}+{\alpha}_{2}^{\prime}& \ge & {\alpha}_{1}+{\alpha}_{2}\hfill \\ \hfill \dots & \ge & \dots \hfill \\ \hfill {\alpha}_{1}^{\prime}+{\alpha}_{2}^{\prime}+\dots +{\alpha}_{n}^{\prime}& =& {\alpha}_{1}+{\alpha}_{2}+\dots +{\alpha}_{n}\hfill \end{array}$$
- (3)
- $$\begin{array}{ccc}\hfill {\alpha}_{n}^{\prime}& \le & {\alpha}_{n}\hfill \\ \hfill {\alpha}_{n}^{\prime}+{\alpha}_{n-1}^{\prime}& \le & {\alpha}_{n}+{\alpha}_{n-1}\hfill \\ \hfill \dots & \le & \dots \hfill \\ \hfill {\alpha}_{n}^{\prime}+{\alpha}_{n-1}^{\prime}+\dots +{\alpha}_{1}^{\prime}& =& {\alpha}_{n}+{\alpha}_{n-1}+\dots +{\alpha}_{1}\hfill \end{array}$$
- (4)
- For all increasing sequences $\left\{{x}_{k}\right\}$ and for any $f\left(x\right)$ strictly monotonous function, ${f}^{-1}\left({\sum}_{k}{\alpha}_{k}^{\prime}f\left({x}_{k}\right)\right)\le {f}^{-1}\left({\sum}_{k}{\alpha}_{k}f\left({x}_{k}\right)\right)$ (quasi-arithmetic, or Kolmogorov, mean), whenever $f\left(x\right)$ is applicable.
- (5)
- There exists a strictly monotonous function $f\left(x\right)$ such that for all increasing sequences $\left\{{x}_{k}\right\}$:$${f}^{-1}\left(\sum _{k}{\alpha}_{k}^{\prime}f\left({x}_{k}\right)\right)\le {f}^{-1}\left(\sum _{k}{\alpha}_{k}f\left({x}_{k}\right)\right)$$

**Theorem**

**3.**

**Theorem**

**4.**

**Corollary**

**1.**

**Corollary**

**2.**

**Corollary**

**3.**

**Corollary**

**4.**

**Corollary**

**5.**

**Example**

**2.**

- 1.
- $\{1/4,1/4,1/2\}{\succ}_{st}\{1/3,1/3,1/3\}$
- 2.
- $\{1/2,1/4,1/4\}{\overline{)\succ}}_{st}\{1/3,1/3,1/3\}$
- 3.
- $\{1/3,1/3,1/3\}{\succ}_{st}\{1/2,1/4,1/4\}$
- 4.
- $\{1/2,1/4,1/4\}{\overline{)\succ}}_{st}\{1/5,3/5,1/5\}$
- 5.
- $\{1/5,3/5,1/5\}{\overline{)\succ}}_{st}\{1/2,1/4,1/4\}$

#### Discussion

## 4. A New Partial Order: K-L Dominance

**Definition**

**1.**

**Theorem**

**5.**

**Lemma**

**1.**

- (1)
- for any sequence of n positive numbers in increasing order $\left\{{z}_{k}\right\}$, the following inequality holds:$$\sum _{k=1}^{n}{w}_{k}{z}_{k}\le \sum _{k=1}^{n}{y}_{k}{z}_{k},$$
- (2)
- (2) the following inequalities hold:$$\begin{array}{ccc}\hfill {w}_{n}& \le & {y}_{n}\hfill \\ \hfill {w}_{n}+{w}_{n-1}& \le & {y}_{n}+{y}_{n-1}\hfill \\ \hfill \cdots & \le & \cdots \hfill \\ \hfill {w}_{n}+\dots +{w}_{2}& \le & {y}_{n}+\dots +{y}_{2}\hfill \end{array}$$$$\begin{array}{ccc}\hfill {w}_{1}+\dots +{w}_{n-1}+{w}_{n}& \le & {y}_{1}+\dots +{y}_{n-1}+{y}_{n}.\hfill \end{array}$$

**Proof.**

**Corollary**

**6.**

**Corollary**

**7.**

**Corollary**

**8.**

**Proof.**

**Example**

**3.**

- 1.
- $\{1/3,1/3,1/3\}{\succ}_{\mathrm{KL}}\{4/9,3/9,2/9\}$
- 2.
- $\{1/3,1/3,1/3\}{\succ}_{\mathrm{KL}}\{7/9,1/9,1/9\}$
- 3.
- $\{1/3,1/3,1/3\}{\overline{)\succ}}_{\mathrm{KL}}\{1/9,1/9,7/9\}$
- 4.
- $\{1/3,1/3,1/3\}{\succ}_{\mathrm{KL}}\{4/9,2/9,3/9\}$
- 5.
- $\{1/3,1/3,1/3\}{\overline{)\succ}}_{\mathrm{KL}}\{3/9,2/9,4/9\}$
- 6.
- $\{1/4,1/4,1/2\}{\succ}_{\mathrm{KL}}\{1/4,1/2,1/4\}$
- 7.
- $\{1/2,1/4,1/4\}{\succ}_{\mathrm{KL}}\{5/8,2/8,1/8\}$

#### 4.1. Discussion

#### 4.2. Relationship between K-L Dominance and First Stochastic Dominance Orders

- $\left\{{\alpha}_{k}\right\}{\succ}_{st}\left\{{\alpha}_{k}^{\prime}\right\}\leftrightarrow \forall {x}_{k}\phantom{\rule{3.33333pt}{0ex}}\mathrm{increasing}\phantom{\rule{3.33333pt}{0ex}}CE({\alpha}_{k},{x}_{k})\le CE({\alpha}_{k}^{\prime},{x}_{k})$
- $\left\{{x}_{k}\right\}{\succ}_{\mathrm{KL}}\left\{{x}_{k}^{\prime}\right\}\leftrightarrow \forall {\alpha}_{k}\phantom{\rule{3.33333pt}{0ex}}\mathrm{increasing}\phantom{\rule{3.33333pt}{0ex}}CE({\alpha}_{k},{x}_{k})\le CE({\alpha}_{k},{x}_{k}^{\prime})$

**Theorem**

**6.**

- (1)
- $$\left\{{x}_{k}\right\}{\succ}_{\mathrm{KL}}\left\{{x}_{k}^{\prime}\right\}$$
- (2)
- $$\left\{\frac{-log{x}_{k}^{\prime}}{-{\sum}_{k}log{x}_{k}^{\prime}}\right\}{\succ}_{st}\left\{\frac{-log{x}_{k}}{-{\sum}_{k}log{x}_{k}}\right\}$$

**Proof.**

**Example**

**4.**

## 5. Conclusions and Future Work

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Hu, B.G.; He, R.; Yuan, X.T. Information-Theoretic Measures for Objective Evaluation of Classifications. Acta Autom. Sin.
**2012**, 38, 1170–1182. [Google Scholar] [CrossRef] - Chen, M.; Golan, A. What may visualization processes optimize? IEEE Trans. Vis. Comput. Graph.
**2016**, 22, 2619–2632. [Google Scholar] [CrossRef] [PubMed] - Chen, M.; Feixas, M.; Viola, I.; Bardera, A.; Sbert, M.; Shen, H.W. Information Theory Tools for Visualization; CRC Press: Boca Raton, FL, USA, 2016. [Google Scholar]
- Sbert, M.; Feixas, M.; Rigau, J.; Chover, M.; Viola, I. Information Theory Tools for Computer Graphics; Synthesis Lectures on Computer Graphics and Animation; Morgan & Claypool Publishers: San Rafael, CA, USA, 2009; Volume 4, pp. 1–153. [Google Scholar]
- Galas, D.; Dewey, G.; Kunert-Graf, J.; Sakhanenko, N. Expansion of the Kullback-Leibler Divergence, and a New Class of Information Metrics. Axioms
**2017**, 6, 8. [Google Scholar] [CrossRef] - Sbert, M.; Poch, J. A necessary and sufficient condition for the inequality of generalized weighted means. J. Inequal. Appl.
**2016**, 2016, 292. [Google Scholar] [CrossRef] - Shaked, M.; Shanthikumar, G. Stochastic Orders; Springer: New York, NY, USA, 2007. [Google Scholar]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley–Interscience: New York, NY, USA, 1991. [Google Scholar]
- Hardy, G.; Littlewood, J.; Pólya, G. Inequalities; Cambridge University Press: Cambridge, UK, 1952. [Google Scholar]
- Belzunce, F.; Martinez-Riquelme, C.; Mulero, J. An Introduction to Stochastic Orders; Academic Press: Cambridge, MA, USA, 2016. [Google Scholar]

**Figure 1.**Plotting the sums in Equation (1) for relations 1–5 in Example 2.

**Figure 2.**Plotting the logarithm of products in Equation (4) for relations 1, 3, 6, and 7 in Example 3.

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Sbert, M.; Chen, M.; Poch, J.; Bardera, A. Some Order Preserving Inequalities for Cross Entropy and Kullback–Leibler Divergence. *Entropy* **2018**, *20*, 959.
https://doi.org/10.3390/e20120959

**AMA Style**

Sbert M, Chen M, Poch J, Bardera A. Some Order Preserving Inequalities for Cross Entropy and Kullback–Leibler Divergence. *Entropy*. 2018; 20(12):959.
https://doi.org/10.3390/e20120959

**Chicago/Turabian Style**

Sbert, Mateu, Min Chen, Jordi Poch, and Anton Bardera. 2018. "Some Order Preserving Inequalities for Cross Entropy and Kullback–Leibler Divergence" *Entropy* 20, no. 12: 959.
https://doi.org/10.3390/e20120959