On a General Definition of Conditional Rényi Entropies ^†

Abstract

In recent decades, different definitions of conditional Rényi entropy (CRE) have been introduced. Thus, Arimoto proposed a definition that found an application in information theory, Jizba and Arimitsu proposed a definition that found an application in time series analysis and Renner-Wolf, Hayashi and Cachin proposed definitions that are suitable for cryptographic applications. However, there is still no a commonly accepted definition, nor a general treatment of the CRE-s, which can essentially and intuitively be represented as an average uncertainty about a random variable X if a random variable Y is given. In this paper we fill the gap and propose a three-parameter CRE, which contains all of the previous definitions as special cases that can be obtained by a proper choice of the parameters. Moreover, it satisfies all of the properties that are simultaneously satisfied by the previous definitions, so that it can successfully be used in aforementioned applications. Thus, we show that the proposed CRE is positive, continuous, symmetric, permutation invariant, equal to Rényi entropy for independent X and Y, equal to zero for $X = Y$ and monotonic. In addition, as an example for the further usage, we discuss the properties of generalized mutual information, which is defined using proposed CRE.

1. Introduction

Rényi entropy (RE) is a well known one parameter generalization of Shannon entropy. It has successfully been used in a number of different fields, such as statistical physics, quantum mechanics, communication theory and data processing [1,2]. On the other hand, the generalization of conditional Shannon entropy to Rényi entropy case is not uniquely defined. Thus, different definitions of conditional Rényi entropy has been proposed in the context of channel coding [3], secure communication [4,5,6] and multifractal analysis [1]. However, no one of the generalization satisfies a set of basic properties which are satisfied by the Shannon conditional entropy and there is no general agreement about the proper definition, so the choice of the definition depends on application purpose. Essentially, in all of the previous discussions, CRE can be represented as an average uncertainty about a random variable X if a random variable Y is given.

In this paper, we introduce three-parameter CRE which contains previously defined conditional entropies as special cases that can be obtained by a proper choice of the parameters. Moreover, it satisfies all of the properties that are simultaneously satisfied by the previous definitions, so that it can successfully be used in aforementioned applications. Thus, we show that the proposed CRE is positive, continuous, symmetric, permutation invariant, equal to Rényi entropy for independent X and Y, equal to zero for $X = Y$ and monotonic.

One of the most frequent usages of conditional entropies is for the definition of mutual information (MI). The MI represents information transfer between the channel input and output which can be defined as the input uncertainty reduction if the output symbols are known. Thus, the MI which corresponds to the $\alpha$-$\beta$-$\gamma$ entropy measured as a difference between the input RE and the input $\alpha$-$\beta$-$\gamma$ CRE when the output is given. We analyze the properties of $\alpha$-$\beta$-$\gamma$ MI and show that the basic properties of Shannon MI, such as continuity, annulation for independent input and output and reducing to the output entropy for independent events, are also satisfied in the case of $\alpha$-$\beta$-$\gamma$ MI, which further validates the usage of $\alpha$-$\beta$-$\gamma$ CRE.

The paper is organized as follows. In Section 2 we review basic notions about Rényi entropy. The definition of $\alpha$-$\beta$-$\gamma$ CRE is introduced in Section 3 and its properties are considered in Section 4. The $\alpha$-$\beta$-$\gamma$ MI is considered in Section 5.

2. Rényi Entropy

Let X be a discrete random variable taking values from a sample space $\{x_1,\cdots ,x_n\}$ and distributed according to $P_X = (p_1,\cdots ,p_n)$. The Rényi entropy of X of order $\alpha$ ($\alpha >0$) is defined as

$${\mathcal{R}}_{\alpha }\left(X\right) =\frac{1}{1-\alpha }{log}_2\left(\sum _x{P}_X{\left(x\right)}^{\alpha }\right).$$

(1)

For two discrete random variables X and Y, with joint density $P_{XY}$, the joint Rényi entropy is defined with

$${\mathcal{R}}_{\alpha }(X,Y) =\frac{1}{1-\alpha }{log}_2\left(\sum _{x,y}{P}_{XY}{(x,y)}^{\alpha }\right).$$

(2)

The following properties hold [1]:

(A1)

${\mathcal{R}}_{\alpha }\left(X\right)$ is continuous with respect to $P_X$;

(A2)

Adding a zero probability event to the sample space of X does not change ${\mathcal{R}}_{\alpha }\left(X\right)$

(A3)

${\mathcal{R}}_{\alpha }\left(X\right)$ takes its largest value for the uniformly distributed random variable i.e.,

$${\mathcal{R}}_{\alpha }\left(X\right) \le {log}_{2}n,$$

(3)

with equality iff $P_X = \left(1/n,\cdots ,1/n\right)$.

(A4)

If $P_X = (1,0,\cdots ,0)$, then ${\mathcal{R}}_{\alpha }\left(X\right) = 0$;

(A5)

${\mathcal{R}}_{\alpha }\left(X\right)$ is symmetric with respect to $P_X$, i.e., if $X\sim (p_1,\cdots ,p_n)$ and $Y\sim \left((p_{\pi \left(1\right)},\cdots ,p_{\pi \left(n\right)})\right)$, where $\pi$ is any permutation of $\{1,\cdots ,n\}$, than ${\mathcal{R}}_{\alpha }\left(X\right) = {\mathcal{R}}_{\alpha }\left(Y\right)$ for all $P_X$;

(A6)

${\mathcal{R}}_{\alpha }\left(X\right)$ is continuous with respect to $\alpha$ and in the limit case reduces to Shannon entropy $\mathcal{S}\left(X\right)$ [7]:

$$\mathcal{S}\left(X\right) = \underset{\alpha \to 1}{lim}{\mathcal{R}}_{\alpha }\left(X\right) = -\sum _x{P}_X\left(x\right){log}_2{P}_X\left(x\right);$$

(4)

(A7)

Rényi entropy can be represented as quasi-linear mean of Hartley information content $H = -{log}_2{P}_X$ as:

$${\mathcal{R}}_{\alpha }\left(X\right) = g_{\alpha }^{-1}\left(\sum _{x}P\left(x\right)g_{\alpha }\left(H\left(x\right)\right)\right)$$

(5)

where the function $g_{\alpha }$ is defined with

$$g_{\alpha }\left(x\right) =\left\{\begin{array}{cc}{\displaystyle \frac{2^{(1-\alpha )x-1}}{1-\alpha }},\hfill & \mathrm{for}\alpha \ne 1;\hfill \\ x,\hfill & \mathrm{for}\alpha =1.\hfill \end{array}\right.$$

(6)

or any linear function of Equation (6) (it follows from a well known result from mean value theory: if one function is a linear function of another one, they generate the same quasi-linear mean).

3. $\mathbf{\alpha }$-$\mathbf{\beta }$-$\mathbf{\gamma }$ Conditional Rényi Entropy

Previously, several definitions of the CRE has been proposed [8] as a measure of average uncertainty about random variable Y when X is known. In this section, we unify all of this definitions and we define the CRE as three parameter function which can access all of the previous definitions by a special choice of the parameters.

Let $(X,Y)\sim P_{X,Y}$, $X\sim P_X$. The Rényi entropy of conditional random variables $Y|X = x$ distributed according to $P_{Y|X=x}$ is denoted with ${\mathcal{R}}_{\alpha }\left(Y\right|X=x)$. The conditional Rényi entropy is defined with

$$R_{\alpha }^{\beta ,\gamma }\left(Y\right|X) = g_{\gamma }^{-1}\left(\sum _x{P}_X^{\left(\beta \right)}\left(x\right)g_{\gamma }\left({\mathcal{R}}_{\alpha }\left(Y\right|X=x)\right)\right)$$

(7)

where $g_{\gamma }$ is given with Equation (6) and escort distribution of $P_X$ is defined with:

$$P_X^{\left(\beta \right)}\left(x\right) = \frac{P_X{\left(x\right)}^{\beta }}{{\sum }_x{P}_X{\left(x\right)}^{\beta }}.$$

(8)

The definition can straightforwardly be extended to the joint conditional entropy. For random variables X,Y,Z, we define joint conditional entropy as:

$$R_{\alpha }^{\beta ,\gamma }(Y,Z|X) = g_{\gamma }^{-1}\left(\sum _x{P}_X^{\left(\beta \right)}\left(x\right)g_{\gamma }\left({\mathcal{R}}_{\alpha }(Z,Y|X=x)\right)\right)$$

(9)

The definitions extends to the case $\beta = \infty$, by taking a limit $\beta \to \infty$, and by using ${lim}_{\beta \to \infty }{P}_X^{\left(\beta \right)}\left(x\right) = {max}_x{P}_X\left(x\right)$. In the case of $\alpha = \beta = 1$, the definitions reduces to the Shannon case. By choosing appropriate values for $\beta$ and $\gamma$ we get the previously considered definitions as follows:

[C-CRE] $\beta = \gamma = 1$, Cachin [4]

$$R_{\alpha }^C\left(Y\right|X) = \sum _x{P}_X\left(x\right){\mathcal{R}}_{\alpha }\left(Y\right|X=x)$$

(10)

[JA-CRE] $\beta = \gamma = \alpha$, Jizba and Arimitsu [1]

$$R_{\alpha }^{JA}\left(Y\right|X) = \frac{1}{1-\alpha }{log}_2\sum _x{P}_X^{\left(\alpha \right)}\left(x\right)2^{(1-\alpha ){\mathcal{R}}_{\alpha }\left(Y\right|X=x)}$$

(11)

[RW-CRE] $\beta = \infty$ Renner and Wolf [5] (RW-CRE)

$$R_{\alpha }^{RW}\left(Y\right|X) =\left\{\begin{array}{cc}\underset{x}{min}{R}_{\alpha }\left(Y\right|X=x)\hfill & \mathrm{if}\alpha > 1\hfill \\ \underset{x}{max}{R}_{\alpha }\left(Y\right|X=x)\hfill & \mathrm{if}\alpha < 1\hfill \end{array}\right.$$

(12)

[A-CRE] $\beta = 1, \gamma = 2 - {\alpha }^{-1}$ Arimoto [3] (A-CRE)

$$R_{\alpha }^A\left(Y\right|X) = \frac{1}{1-\alpha }{log}_2\sum _x{P}_X\left(x\right)2^{\frac{1-\alpha }{\alpha }{\mathcal{R}}_{\alpha }\left(Y\right|X=x)}$$

(13)

[H-CRE] $\alpha = \gamma , \beta = 1$ Hayashi [6] (H-CRE).

$$R_{\alpha }^H\left(Y\right|X) = \frac{1}{1-\alpha }{log}_2\sum _x{P}_X\left(x\right)2^{(1-\alpha ){\mathcal{R}}_{\alpha }\left(Y\right|X=x)}$$

(14)

4. Properties of $\mathbf{\alpha }$-$\mathbf{\beta }$-$\mathbf{\gamma }$ CRE

The $\alpha$-$\beta$-$\gamma$ CRE satisfies the set of important properties for all $\alpha ,\beta ,\gamma$:

(B1)

$R_{\alpha }^{\beta ,\gamma }\left(Y\right|X) \ge 0$

(B2)

$R_{\alpha }^{\beta ,\gamma }\left(Y\right|X)$ is continuous with respect to $P_{X,Y}$;

(B3)

$R_{\alpha }^{\beta ,\gamma }\left(Y\right|X)$ is symmetric with respect to $P_{Y|X=x}$ for all $P_X\in {\Delta }_n$, $i=1,\cdots ,n$ and $P_{Y|X=x}$;

(B4)

If $P_{Y|X=x}$ is a permutation of $P_{Y|X=x_1}$, for all x, then, and $R_{\alpha }^{\beta ,\gamma }\left(Y\right|X) = \mathcal{R}\left(Y\right|X=x_1)$

(B5)

If X and Y are independent, then $R_{\alpha }^{\beta ,\gamma }\left(Y\right|X) = \mathcal{R}\left(Y\right)$

(B6)

If $X = Y$, then $R_{\alpha }^{\beta ,\gamma }\left(Y\right|X) = 0$

(B7)

In the case of $\alpha = \beta = 1$, the definitions reduces to the Shannon case.

(B8)

Let $X,Y,Z$ be random variables distributed according with joint distribution $P_{X,Y,Z}$ and corresponding marginal distributions $P_X,P_Y,P_{Y,Z}$

$$R_{\alpha }^{\beta ,\gamma }(Y,Z|X) \le R_{\alpha }^{\beta ,\gamma }\left(Y\right|X)$$

(15)

The proofs for the properties B1–B6 straightforwardly follows from the definition Equation (7) of the conditional Rényi entropy and from the properties A1–A5 of Rényi entropy. Here, we give the proof for the property B8.

Proof of the Property 8. 

Let, $P_{Y,Z|x},P_{Z|x}$, $P_{Y|x,z}$ be conditional distributions. Similarly as in [9], we have:

$$\begin{array}{c}\hfill \sum _{y,z}{P}_{Y,Z|x}{(y,z)}^{\alpha } = \sum _z{P}_{Z|x}{\left(z\right)}^{\alpha }\sum _y{P}_{Y|x,z}{\left(y\right)}^{\alpha }\\ \hfill \left\{\begin{array}{c}\le \sum _z{P}_{Z|x}{\left(z\right)}^{\alpha },\mathrm{for}\alpha > 1\hfill \\ \ge \sum _z{P}_{Z|x}{\left(z\right)}^{\alpha },\mathrm{for}\alpha < 1\hfill \end{array}\right.\end{array}$$

(16)

so that we have ${\mathcal{R}}_{\alpha }\left(Y,Z|X=x\right) \le {\mathcal{R}}_{\alpha }\left(Y|X=x\right)$ and result follows from the definition of conditional entropy since $g_{\gamma }$ is increasing. □

Additional properties which are satisfied in the case of Shannon entropy are not satisfied in general and are limited to a special choices of $\beta$ and $\gamma$.

(B9)

Chain rule:

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\alpha }(X,Y)& = {\mathcal{R}}_{\alpha }\left(X\right) + R_{\alpha }^{\beta ,\gamma }\left(Y\right|X)\hfill \\ \hfill & = {\mathcal{R}}_{\alpha }\left(Y\right) + R_{\alpha }^{\beta ,\gamma }\left(X\right|Y)\hfill \end{array}$$

(17)

is satisfied in general only in the case of JA-CRE. Jizba and Arimitsu [1] used it (with an assumption that $g_{\gamma }$ is invertible and positive) as one of the Generalized Shannon-Khinchin axioms, along with the properties A1-A3 of Rényi entropy, and shown that Réni can be characterized as a unique function which satisfies them. It also implies the symmetry of generalized mutual information introduced in the next section.

(B10)

Weak chain rule:

$$R_{\alpha }^{\beta ,\gamma }\left(Y\right|X) \ge {\mathcal{R}}_{\alpha }(X,Y) - {log}_{2}m$$

(18)

(B11)

Conditioning reduces the entropy (CRE),

$$R_{\alpha }^{\beta ,\gamma }\left(Y\right|X) \le {\mathcal{R}}_{\alpha }\left(X\right)$$

(19)

is satisfied in the cases of H-CRE, A-CRE and RW-CRE (for $\alpha \ge 1$). CRE states that an additional knowledge can not increase the information. Although it can intuitively be treated as an ineluctable property, breaking the CRE can still be interpreted using concept of spoiling knowledge as in [4].

(B12)

Monotonicity says that if X, Y, and Z forms Markov chain then

$$R_{\alpha }^{\beta ,\gamma }\left(X\right|Z) \le R_{\alpha }^{\beta ,\gamma }\left(X\right|Y)$$

(20)

It holds in the case of A-CRE and H-CRE and implies data processing inequality (defined in the next section), which is an important property for applications of Rényi entropy in cryptography [5].

The previous discussion is summarized in

Table 1. Properties of different CRE-s (✓ stands for satisfied, ✗ for not satisfied, and ✓${}^{*}$ for satisfied for $\alpha \ge 1$).

	C	$\mathit{JA}$	$\mathit{RW}$	H	A
Chain Rule	✗	✓	✗	✗	✗
Weak Chain Rule	✗	✓	✓	✗	✗
CRE	✗	✗	✓${}^{*}$	✓	✓
Monotonicity	✗	✗	✗	✓	✓

[9].

Since the additional information theoretic properties are not satisfied in general, no one of the previous definitions has not been commonly accepted, and the choice of the definition depends on application purpose. On the other hand, a set of properties B1–B8 which are satisfied for all of them, also hold in case of $\alpha$-$\beta$-$\gamma$ CRE defined by Equation (7), which justifies the definition.

5. $\mathbf{\alpha }$-$\mathbf{\beta }$-$\mathbf{\gamma }$ Mutual Information

A communication channel with input X and output Y described by transition matrix $P_{Y|X}$

$$P_{Y|X}^{(j,i)} = P_{Y|X=x_i}(Y=y_j|X=x_i)$$

The mutual information is a measure of information transfer between input and output of communication channel. Thus, if the uncertainty about the input is measured by ${\mathcal{R}}_{\alpha }\left(X\right)$ and its uncertainty after all symbols are received by ${\mathcal{R}}_{\alpha }^{(\beta ,\gamma )}\left(X\right|Y)$, the $\alpha$-$\beta$-$\gamma$ mutual information between X and Y is defined as

$$I_{\alpha }^{\beta ,\gamma }(X,Y) = {\mathcal{R}}_{\alpha }\left(X\right) - R_{\alpha }^{\beta ,\gamma }\left(X\right|Y).$$

(21)

This definition generalizes the previously introduced one by Arimoto who used A-CRE, and one by Jizba et al. [10] who used JA-CRE.

By usage of the properties of $\alpha$-$\beta$-$\gamma$ CRE, it is easy to conclude that the basic properties are satisfied for all $\alpha ,\beta ,\gamma$:

(D1)

$I_{\alpha }^{\beta ,\gamma }(Y,X)$ is continuous with respect to $P_{X,Y}$;

(D2)

If X and Y are independent, then $I_{\alpha }^{\beta ,\gamma }(X,Y) = 0$

(D3)

If $X = Y$, then $I_{\alpha }^{\beta ,\gamma }(X,Y) = {\mathcal{R}}_{\alpha }\left(X\right)$

However, another properties which hold in the Shannon case are not satisfied in general:

(D4)

Non-Negativity $I_{\alpha }^{\beta ,\gamma }(Y,X) \ge 0$

(D5)

Symmetry: $I_{\alpha }^{\beta ,\gamma }(X,Y) = I_{\alpha }^{\beta ,\gamma }(Y,X)$

(D6)

Data processing inequality (DPI): If X, Y, and Z forms Markov chain then $I_{\alpha }^{\beta ,\gamma }(X,Y) \ge I_{\alpha }^{\beta ,\gamma }(X,Z)$

In

Table 2. Properties of different MI deffinitions (✓ stands for satisfied, ✗ for not satisfied, and ✓${}^{*}$ for satisfied for $\alpha \ge 1$).

	C	$\mathit{JA}$	$\mathit{RW}$	H	A
Non-Negativity	✗	✗	✓${}^{*}$	✓	✓
Symmetry	✗	✓	✗	✗	✗
DPI	✗	✗	✗	✓	✓

we list the properties and their fulfilments for the cases when the mutual information is defined using C, JA, RW, A and H conditional Rényi entropies [9]. Thus, if the MI is defined using any of the previously considered CRE definitions, it fails down to satisfy some of the properties properties D4-D6. On the other hand, the set of properties D1-D3 which is sattisfied for all of them is also satisfied for $\alpha$-$\beta$-$\gamma$ MI, which justify its usage as a measure of information transfer.

6. Conclusions

We introduced $\alpha$-$\beta$-$\gamma$ conditional Rényi entropy (CRE) which contains previously defined conditional entropies as special cases that can be obtained by a proper choice of the parameters [1,3,4,5,6]. It satisfies all of the properties that are simultaneously satisfied by the previous definitions, so that it could be successfully be used in channel coding, secure communication and multifractal analysis.

In addition, we analyzed the properties of mutual information (MI) which is defined using $\alpha$-$\beta$-$\gamma$ CRE. The resulting MI measure satisfies the set of basic properties which further validates the usage of $\alpha$-$\beta$-$\gamma$ CRE.