# A Characterization of Entropy in Terms of Information Loss

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

**MSC**Primary: 94A17; Secondary: 62B10

## 1. Introduction

**information loss**. We also assume that F obeys three axioms. If we call a morphism a “process” (to be thought of as deterministic), we can state these roughly in words as follows. For the precise statement, including all the definitions, see Section 2.

- (i)
**Functoriality**. Given a process consisting of two stages, the amount of information lost in the whole process is the sum of the amounts lost at each stage:$$F(f\circ g)=F\left(f\right)+F\left(g\right)$$- (ii)
**Convex linearity**. If we flip a probability-λ coin to decide whether to do one process or another, the information lost is λ times the information lost by the first process plus $(1-\lambda )$ times the information lost by the second:$$F(\lambda f\oplus (1-\lambda \left)g\right)=\lambda F\left(f\right)+(1-\lambda )F\left(g\right)$$- (iii)
**Continuity**. If we change a process slightly, the information lost changes only slightly: $F\left(f\right)$ is a continuous function of f.

## 2. The Main Result

**Definition 1.**Let $\mathtt{FinProb}$ be the category where an object $(X,p)$ is given by a finite set X equipped with a probability measure p, and where a morphism $f:(X,p)\to (Y,q)$ is a measure-preserving function from $(X,p)$ to $(Y,q)$, that is, a function $f:X\to Y$ such that

**Shannon entropy**of a probability measure p on a finite set X is

**Theorem 2.**Suppose F is any map sending morphisms in $\mathtt{FinProb}$ to numbers in $[0,\infty )$ and obeying these three axioms:

- (i)
**Functoriality**:$$F(f\circ g)=F\left(f\right)+F\left(g\right)$$- (ii)
**Convex linearity**:$$F(\lambda f\oplus (1-\lambda \left)g\right)=\lambda F\left(f\right)+(1-\lambda )F\left(g\right)$$- (iii)
**Continuity**: F is continuous.

**converges**to a morphism $(X,p)\stackrel{f}{\to}(Y,q)$ if:

- for all sufficiently large n, we have ${X}_{n}=X$, ${Y}_{n}=Y$, and ${f}_{n}\left(i\right)=f\left(i\right)$ for all $i\in X$;
- $p\left(n\right)\to p$ and $q\left(n\right)\to q$ pointwise.

**continuous**if $F\left({f}_{n}\right)\to F\left(f\right)$ whenever ${f}_{n}$ is a sequence of morphisms converging to a morphism f.

**Definition 3.**Let $\mathtt{FinMeas}$ be the category whose objects are finite sets equipped with measures and whose morphisms are measure-preserving functions.

- For direct sums, first note that the disjoint union of two finite sets equipped with measures is another object of the same type. We write the disjoint union of $p,q\in \mathtt{FinMeas}$ as $p\oplus q$. Then, given morphisms $f:p\to {p}^{\prime}$, $g:q\to {q}^{\prime}$ there is a unique morphism $f\oplus g:p\oplus q\to {p}^{\prime}\oplus {q}^{\prime}$ that restricts to f on the measure space p and to g on the measure space q.
- For scalar multiplication, first note that we can multiply a measure by a nonnegative real number and get a new measure. So, given an object $p\in \mathtt{FinMeas}$ and a number $\lambda \ge 0$ we obtain an object $\lambda p\in \mathtt{FinMeas}$ with the same underlying set and with ${\left(\lambda p\right)}_{i}=\lambda {p}_{i}$. Then, given a morphism $f:p\to q$, there is a unique morphism $\lambda f:\lambda p\to \lambda q$ that has the same underlying function as f.

**total mass**of $(X,p)$ to be

**Shannon entropy**of p to be $\parallel p\parallel H\left(\overline{p}\right)$. If the total mass of p is zero, we define its Shannon entropy to be zero.

**Corollary 4.**Suppose F is any map sending morphisms in $\mathtt{FinMeas}$ to numbers in $[0,\infty )$ and obeying these four axioms:

- (i)
**Functoriality**:$$F(f\circ g)=F\left(f\right)+F\left(g\right)$$- (ii)
**Additivity**:$$F(f\oplus g)=F\left(f\right)+F\left(g\right)$$- (iii)
**Homogeneity**:$$F\left(\lambda f\right)=\lambda F\left(f\right)$$- (iv)
- Continuity: F is continuous.

## 3. Why Shannon Entropy Works

## 4. Faddeev’s Theorem

**Theorem 5.**

**(Faddeev)**Suppose I is a map sending any probability measure on any finite set to a nonnegative real number. Suppose that:

- (i)
- I is invariant under bijections.
- (ii)
- I is continuous.
- (iii)
- For any probability measure p on a set of the form $\{1,\cdots ,n\}$, and any number $0\le t\le 1$,$$I\left((t{p}_{1},(1-t){p}_{1},{p}_{2},\cdots ,{p}_{n})\right)=I\left(({p}_{1},\cdots ,{p}_{n})\right)+{p}_{1}I\left((t,1-t)\right)$$

**strong additivity**formula:

**Theorem 6.**Suppose I is a map sending any probability measure on any finite set to a nonnegative real number. Suppose that:

- (i)
- I is invariant under bijections.
- (ii)
- I is continuous.
- (iii)
- $I\left(\right(1\left)\right)=0$, where $\left(1\right)$ is our name for the unique probability measure on the set $\left\{1\right\}$.
- (iv)
- For any probability measure p on the set $\{1,\cdots ,n\}$ and probability measures $q\left(1\right),\dots ,q\left(n\right)$ on finite sets, we have$$I({p}_{1}q\left(1\right)\oplus \cdots \oplus {p}_{n}q\left(n\right))=I\left(p\right)+\sum _{i=1}^{n}{p}_{i}I\left(q\left(i\right)\right)$$

## 5. Proof of the Main Result

## 6. A Characterization of Tsallis Entropy

**Tsallis entropy of order α**of a probability measure p on a finite set X is defined as:

**Theorem 7.**Let $\alpha \in (0,\infty )$. Suppose F is any map sending morphisms in $\mathtt{FinProb}$ to numbers in $[0,\infty )$ and obeying these three axioms:

- (i)
**Functoriality**:$$F(f\circ g)=F\left(f\right)+F\left(g\right)$$- (ii)
**Compatibility with convex combinations**:$$F(\lambda f\oplus (1-\lambda )g)={\lambda}^{\alpha}F\left(f\right)+{(1-\lambda )}^{\alpha}F\left(g\right)$$- (iii)
**Continuity**: F is continuous.

**Corollary 8.**Let $\alpha \in (0,\infty )$. Suppose F is any map sending morphisms in $\mathtt{FinMeas}$ to numbers in $[0,\infty )$, and obeying these four properties:

- (i)
**Functoriality**:$$F(f\circ g)=F\left(f\right)+F\left(g\right)$$- (ii)
**Additivity**:$$F(f\oplus g)=F\left(f\right)+F\left(g\right)$$- (iii)
**Homogeneity of degree**:$$F\left(\lambda f\right)={\lambda}^{\alpha}F\left(f\right)$$- (iv)
**Continuity**: F is continuous.

## Acknowledgements

## References

- Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J.
**1948**, 27, 379–423. [Google Scholar] [CrossRef] - Aczél, J.; Daróczy, Z. On Measures of information and their characterizations. In Mathematics in Science and Engineering; Academic Press: New York, NY, USA, 1975; Volume 15. [Google Scholar]
- Mac Lane, S. Categories for the working mathematician. In Graduate Texts in Mathematics 5; Springer: Berlin, Germany, 1971. [Google Scholar]
- Faddeev, D.K. On the concept of entropy of a finite probabilistic scheme. Uspehi Mat. Nauk (N.S.)
**1956**, 1, 227–231. (In Russian) [Google Scholar] - Rényi, A. Measures of information and entropy. In Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability, Statistical Laboratory of the University of California, Berkeley, CA, USA, June 20–July 30, 1960; University of California Press: Berkeley, CA, USA, 1961; pp. 547–561. [Google Scholar]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley Series in Telecommunications and Signal Processing; Wiley-Interscience: New York, NY, USA, 2006. [Google Scholar]
- Tsallis, C. Possible generalization of Boltzmann-Gibbs statistics. J. Stat. Phys.
**1988**, 52, 479–487. [Google Scholar] [CrossRef] - Havrda, J.; Charvát, F. Quantification method of classification processes: concept of structural α-entropy. Kybernetika
**1967**, 3, 30–35. [Google Scholar] - Patil, G.P.; Taillie, C. Diversity as a concept and its measurement. J. Am. Stat. Assoc.
**1982**, 77, 548–561. [Google Scholar] [CrossRef] - Furuichi, S. On uniqueness theorems for Tsallis entropy and Tsallis relative entropy. IEEE Trans. Infor. Theor.
**2005**, 51, 3638–3645. [Google Scholar] [CrossRef]

© 2011 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

## Share and Cite

**MDPI and ACS Style**

Baez, J.C.; Fritz, T.; Leinster, T.
A Characterization of Entropy in Terms of Information Loss. *Entropy* **2011**, *13*, 1945-1957.
https://doi.org/10.3390/e13111945

**AMA Style**

Baez JC, Fritz T, Leinster T.
A Characterization of Entropy in Terms of Information Loss. *Entropy*. 2011; 13(11):1945-1957.
https://doi.org/10.3390/e13111945

**Chicago/Turabian Style**

Baez, John C., Tobias Fritz, and Tom Leinster.
2011. "A Characterization of Entropy in Terms of Information Loss" *Entropy* 13, no. 11: 1945-1957.
https://doi.org/10.3390/e13111945