On the α-q-Mutual Information and the α-q-Capacities

Abstract

The measures of information transfer which correspond to non-additive entropies have intensively been studied in previous decades. The majority of the work includes the ones belonging to the Sharma–Mittal entropy class, such as the Rényi, the Tsallis, the Landsberg–Vedral and the Gaussian entropies. All of the considerations follow the same approach, mimicking some of the various and mutually equivalent definitions of Shannon information measures, and the information transfer is quantified by an appropriately defined measure of mutual information, while the maximal information transfer is considered as a generalized channel capacity. However, all of the previous approaches fail to satisfy at least one of the ineluctable properties which a measure of (maximal) information transfer should satisfy, leading to counterintuitive conclusions and predicting nonphysical behavior even in the case of very simple communication channels. This paper fills the gap by proposing two parameter measures named the $\alpha$-q-mutual information and the $\alpha$-q-capacity. In addition to standard Shannon approaches, special cases of these measures include the $\alpha$-mutual information and the $\alpha$-capacity, which are well established in the information theory literature as measures of additive Rényi information transfer, while the cases of the Tsallis, the Landsberg–Vedral and the Gaussian entropies can also be accessed by special choices of the parameters $\alpha$ and q. It is shown that, unlike the previous definition, the $\alpha$-q-mutual information and the $\alpha$-q-capacity satisfy the set of properties, which are stated as axioms, by which they reduce to zero in the case of totally destructive channels and to the (maximal) input Sharma–Mittal entropy in the case of perfect transmission, which is consistent with the maximum likelihood detection error. In addition, they are non-negative and less than or equal to the input and the output Sharma–Mittal entropies, in general. Thus, unlike the previous approaches, the proposed (maximal) information transfer measures do not manifest nonphysical behaviors such as sub-capacitance or super-capacitance, which could qualify them as appropriate measures of the Sharma–Mittal information transfer.

1. Introduction

In the past, extensive work has been written on defining the information measures which generalize the Shannon entropy [1], such as the one-parameter Rényi entropy [2], the Tsallis entropy [3], the Landsberg–Vedral entropy [4], the Gaussian entropy [5], and the two-parameter Sharma–Mittal entropy [5,6], which reduces to former ones for special choices of the parameters. The Sharma–Mittal entropy can axiomatically be founded as the unique q-additive measure [7,8] which satisfies generalized Shannon–Kihinchin axioms [9,10] and which has widely been explored in different research fields starting from statistics [11] and thermodynamics [12,13] to quantum mechanics [14,15], machine learning [16,17] and cosmology [18,19]. The Sharma–Mittal entropy has also been recognized in the field of information theory, where the measures of conditional Sharma–Mittal entropy [20], Sharma–Mittal divergences [21] and Sharma–Mittal entropy rate [22] have been established and analyzed.

Considerable research has also been done in the field of communication theory in order to analyze information transmission in the presence of noise if, instead of Shannon’s entropy, the information is quantified with (instances of) Sharma–Mittal entropy and, in general, the information transfer is quantified by an appropriately defined measure of mutual information, while the maximal information transfer is considered as a generalized channel capacity. Thus, after Rényi’s proposal for the additive generalization of Shannon entropy [2], several different definitions for Rényi information transfer were proposed by Sibson [23], Arimoto [24], Augustin [25], Csiszar [26], Lapidoth and Pfister [27] and Tomamichel and Hayashi [28]. These measures have been explored thoroughly and their operational characterization in coding theory, hypothesis testing, cryptography and quantum information theory was established, which qualifies them as a reasonable measure of Rényi information transfer [29]. Similar attempts have also been made in the case of non-additive entropies. Thus, starting from the work of Daroczy [30], who introduced a measure for generalized information transfer related to the Tsallis entropy, several attempts followed for the measures which correspond to non-additive particular instances of the Sharma–Mittal entropy, so the definitions for the Rényi information transfer were considered in [24,31], for the Tsallis information transfer in [32] and for the Landsber–Vedral information transfer in [4,33].

In this paper we provide a general treatment of the Sharma–Mittal entropy transfer and a detailed analysis of existing measures, showing that all of the definitions related to non-additive entropies fail to satisfy at least one of the ineluctable properties common to the Shannon case, which we state as axioms, by which the information transfer has to be non-negative, less than the input and output uncertainty, equal to the input uncertainty in the case of perfect transmission and equal to zero, in the case of a totally destructive channel. Thus, breaking some of these axioms implies unexpected and counterintuitive conclusions about the channels, such as achieving super-capacitance or sub-capacitance [4], which could be treated as nonphysical behavior. As an alternative, we propose the $\alpha$-q-mutual information as a measure of Sharma–Mittal information transfer, maximized with the $\alpha$-q-capacity. The $\alpha$-q mutual information generalizes the $\alpha$-mutual information by Arimoto [24], which is defined as a q-difference between the input Sharma–Mittal entropy and the appropriately defined conditional Sharma–Mittal entropy if the output is given, while the $\alpha$-q-capacity represents a generalization of Arimoto’s $\alpha$-capacity in the case of $q=1$. In addition, several other instances can be obtained by specifying the values of parameters $\alpha$ and q, which includes the information transfer measures for the Tsallis, the Landsber–Vedral and the Shannon entropy, as well as the case of the Gaussian entropy which was not considered before in the context of information transmission.

The paper is organized as follows. The basic properties and special instances of the Sharma–Mittal entropy are listed in Section 2. Section 3 reviews the basics of communication theory, introduces the basic communication channels and establishes the set of axioms which information transfer measures should satisfy. The information transfer measures which are defined by Arimoto are introduced in Section 4, and the alternative definitions for Rényi information transfer measures are discussed in Section 5. Finally, the $\alpha$-q-mutual information and the $\alpha$-q-capacities are proposed and their properties analyzed in Section 6 while the previously proposed measures of Sharma–Mittal entropy transfer are discussed in Section 7.

2. Sharma–Mittal Entropy

Let the sets of positive and nonnegative real numbers be denoted with ${\mathbb{R}}^{+}$ and ${\mathbb{R}}_0^{+}$, respectively, and let the mapping ${\eta }_q:\mathbb{R}\to \mathbb{R}$ be defined in

$${\eta }_q\left(x\right)=\left\{\begin{array}{cc}x,& \mathrm{for}q=1\\ \frac{2^{(1-q)x}-1}{(1-q)ln2},& \mathrm{for}q\ne 1\end{array}\right.$$

(1)

so that its inverse is given in

$${\eta }_q^{-1}\left(x\right)=\left\{\begin{array}{cc}x,& \mathrm{for}q=1\\ \frac{1}{1-q}log((1-q)xln2+1),& \mathrm{for}q\ne 1\end{array}\right..$$

(2)

The mapping q and its inverse are increasing continuous (hence invertible) functions such that $0)=0$. The q-logarithm is defined in

$${\mathrm{Log}}_q\left(x\right)={\eta }_q(logx)=\left\{\begin{array}{cc}logx,& \mathrm{for}q=1\\ \frac{x^{(1-q)}-1}{(1-q)ln2},& \mathrm{for}q\ne 1\end{array}\right.,$$

(3)

and its inverse, the q-exponential, is defined in

$${\mathrm{Exp}}_q\left(y\right)=\left\{\begin{array}{cc}{2}^y,& \mathrm{for}q=1\\ {\left(1+(1-q)yln2\right)}^{\frac{1}{1-q}}& \mathrm{for}q\ne 1\end{array}\right.,$$

(4)

for $1+(1-q)yln2>0$. Using η_q, we can define the pseudo-addition operation ${\oplus }_q$ [7,8]

$$x{\oplus }_{q}y={\eta }_q\left({\eta }_q^{-1}\left(x\right)+{\eta }_q^{-1}\left(y\right)\right)=x+y+(1-q)xy;x,y\in \mathbb{R},$$

(5)

and its inverse operation, the pseudo substraction

$$x{\ominus }_{q}y={\eta }_q\left({\eta }_q^{-1}\left(x\right)-{\eta }_q^{-1}\left(y\right)\right)=\frac{x-y}{1+(1-q)yln2};x,y\in \mathbb{R}.$$

(6)

The ${\oplus }_q$ can be rewritten in terms of the generalized logarithm by settings $x=logu$ and $y=logv$ so that

$${\mathrm{Log}}_q(u·v)={\mathrm{Log}}_q\left(u\right){\oplus }_q{\mathrm{Log}}_q\left(v\right);u,v\in {\mathbb{R}}_{+}.$$

(7)

Let the set of all n-dimensional distributions be denoted with

$${\Delta }_n\equiv \left\{(p_1,\dots ,p_n)|p_i\ge 0,\sum _{i=1}^n{p}_i=1\right\};n>1.$$

(8)

Let the function $H_n:{\Delta }_n\to {\mathbb{R}}_0^{+}$ satisfy the following the Shannon–Khinchin axioms, for all $n\in \mathbb{N}$, $n>1$.

GSK1

$H_n$ is continuous in ${\Delta }_n$;

GSK2

$H_n$ takes its largest value for the uniform distribution, $U_n=\left(1/n,\dots ,1/n\right)\in {\Delta }_n$, i.e., $H_n\left(P\right)\le H_n\left(U_n\right)$, for any $P\in {\Delta }_n$;

GSK3

$H_n$ is expandable: $H_{n+1}(p_1,p_2,\dots ,p_n,0)=H_n(p_1,p_2,\dots ,p_n)$ for all $(p_1,\dots ,p_n)\in {\Delta }_n$;

GSK4

Let $P=(p_1,\dots ,p_n)\in {\Delta }_n$, $PQ=(r_{11},r_{12},\dots ,r_{nm})\in {\Delta }_{nm}$, $n,m\in \mathbb{N}$, $n,m>1$ such that $p_i={\sum }_{j=1}^m{r}_{ij}$, and $Q_{|k}=(q_{1|k},\dots ,q_{m|k})\in {\Delta }_m$, where $q_{i|k}=r_{ik}/p_k$ and $\alpha \in {\mathbb{R}}_0^{+}$ are some fixed parameters. Then,

$$H_{nm}\left(PQ\right)=H_n\left(P\right){\oplus }_q{H}_m\left(Q\right|P),\mathrm{where}{H}_m\left(Q\right|P)=f^{-1}\left(\sum _{k=1}^n{p}_k^{\left(\alpha \right)}f\left(H_m\left(Q_{|k}\right)\right)\right),$$

(9)

where f is an invertible continuous function and $P^{\left(\alpha \right)}=(p_1^{\left(\alpha \right)},\dots ,p_n^{\left(\alpha \right)})\in {\Delta }_n$ is the $\alpha$-escort distribution of distribution $P\in {\Delta }_n$ defined in

$$p_k^{\left(\alpha \right)}=\frac{p_k^{\alpha }}{{\sum }_{i=1}^n{p}_i^{\alpha }},k=1,\dots ,n,\alpha >0.$$

(10)

GSK5

$H_2\left(\frac{1}{2},\frac{1}{2}\right)={\mathrm{Log}}_q\left(1\right)$.

As shown in [9], the unique function $H_n$, which satisfies [GSK1]-[GSK5], is Sharma–Mittal entropy [6].

In the following paragraphs we will assume that X and Y are discrete jointly distributed random variables taking values from sample spaces $\{x_1,\dots ,x_n\}$ and $\{y_1,\dots ,y_m\}$, and distributed in accordance to $P_X\in {\Delta }_n$ and $P_Y\in {\Delta }_m$, respectively. In addition, the joint distribution of X and Y will be denoted in $P_{X,Y}\in {\Delta }_{nm}$ and the conditional distribution of X given Y will be denoted in $P_{X|Y}=\frac{P_{X,Y}(x,y)}{P_Y\left(y\right)}\in {\Delta }_m$, provided that $P_Y\left(y\right)>0$. We will identify the entropy of a random variable X with the entropy of its distribution $P_X$ and the Sharma–Mittal entropy will be denoted with $H_{\alpha ,q}\left(X\right)\equiv H_n\left(P_X\right)$.

Thus, for a random variable which is distributed to X, Sharma–Mittal entropy can be expressed in

$$H_{\alpha ,q}\left(X\right)=\frac{1}{1-q}\left({\left(\sum _x{P}_X{\left(x\right)}^{\alpha }\right)}^{\frac{1-q}{1-\alpha }}-1\right),$$

(11)

and it can equivalently be expressed as the η_q transformation of Rényi entropy as in

$$H_{\alpha ,q}\left(X\right)\equiv {\eta }_q\left(R_{\alpha }\left(X\right)\right).$$

(12)

Sharma–Mittal entropy, for $\alpha ,q\in {\mathbb{R}}_0^{+}\backslash 1$, being a continuous function of the parameters and the sums goes over the support of $P_X$. Thus, in the case of $q=1$, $\alpha \ne 1$, Sharma–Mittal reduces to Rényi entropy of order $\alpha$ [2]

$$R_{\alpha }\left(X\right)\equiv H_{\alpha ,1}\left(X\right)=\frac{1}{1-\alpha }log\left(\sum _x{P}_X{\left(x\right)}^{\alpha }\right),$$

(13)

which further reduces to Shannon entropy for $\alpha =1,q=1$,34]

$$S\left(X\right)\equiv H_{1,1}\left(X\right)=\sum _x{P}_X\left(x\right)log{P}_X\left(x\right),$$

(14)

while in the case of $q\ne 1$, $\alpha =1$ it reduces to Gaussian entropy [5]

$$G_q\left(X\right)\equiv H_{1,q}\left(X\right)=\frac{1}{(1-q)ln2}\left(\underset{i=1}{\overset{n}{\Pi }}{P}_X{\left(x\right)}^{P_X\left(x\right)}-1\right).$$

(15)

In addition, Tsallis entropy [3] is obtained for $\alpha =q\ne 1$,

$$T_q\left(X\right)\equiv \frac{1}{(1-q)ln2}\left(\sum _x{P}_X{\left(x\right)}^q-1\right),$$

(16)

while in the case of for $q=2-\alpha$ it reduces to the Landsberg–Vedral entropy [4]

$$L_{\alpha }\left(X\right)\equiv H_{\alpha ,2-\alpha }\left(X\right)=\frac{1}{(\alpha -1)ln2}\left(\frac{1}{{\sum }_x{P}_X{\left(x\right)}^{\alpha }}-1\right).$$

(17)

3. Sharma–Mittal Information Transfer Axioms

One of the main goals of information and communication theories is characterization and analysis of the information transfer between sender X and receiver Y, which communicate through a channel. The sender and receiver are described by probability distributions $P_X$ and $P_Y$ while the communication channel with the input X and the output Y is described by the transition matrix $P_{Y|X}$:

$$P_{Y|X}^{(i,j)}\equiv P_{Y|X}\left(y_j\right|x_i).$$

(18)

We assume that maximum likelihood detection is performed at the receiver, which is defined by the mapping $d:\{y_1,\dots ,y_m\}\to \{x_1,\dots ,x_n\}$ as follows:

$$d\left(y_j\right)=x_i\iff P_{Y|X}\left(y_j\right|x_i)>P_{Y|X}\left(y_j\right|x_k);\mathrm{for}\mathrm{all}k\ne i,$$

(19)

assuming that the inequality in (19) is uniquely satisfied. Thus, if the input symbol $x_i$ is sent and the output symbol $y_j$ is received, the $x_i$ will be detected if $x_i=d\left(y_j\right)$ and a detection error will be made otherwise, and we define the error function functions $\varphi :\{x_1,\dots ,x_m\}\times \{y_1,\dots ,y_m\}\to \{0,1\}$ as in

$$\varphi (x_i,y_j)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if} x_i=d\left(y_j\right)\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(20)

the detection error if a symbol $x_i$ is sent

$$P_{err}\left(x_i\right)=\sum _{y_j}{P}_{Y|X}\left(y_j\right|x_i)\varphi (x_i,y_j);\mathrm{for}\mathrm{all}{x}_i,$$

(21)

as well as the average detection error

$${\overline{P}}_{err}=\sum _{x_i}{P}_X\left(x_i\right)P_{err}\left(x_i\right)=\sum _{x_i,y_j}{P}_{X,Y}(x,y)\varphi (x_i,y_j).$$

(22)

Totally destructive channel: A channel is said to be totally destructive if

$$P_{Y|X}^{(i,j)}=P_{Y|X}\left(y_j\right|x_i)=P_Y\left(y_j\right)=\frac{1}{m};\mathrm{for}\mathrm{all}{x}_i,$$

(23)

i.e., if the sender X and receiver Y are described by independent random variables,

$$X\perp \perp Y\iff P_{X,Y}(x,y)=P_X\left(x\right)P_Y\left(y\right),$$

(24)

where the relationship of independence is denoted in $\perp \perp$. In this case, ${\varphi }_i\left(y_j\right)=1$ for all $y_j$ and the probability of error is $P_{err}\left(x_i\right)=1$; for all $x_i$, as well as the average probability of error ${\overline{P}}_{err}=1$, which means that a correct maximum likelihood detection is not possible.

Perfect communication channel: A channel is said to be perfect if for every $x_i$,

$$P_{Y|X}\left(y_j\right|x_i)>0,\mathrm{for}\mathrm{at}\mathrm{least}\mathrm{one}{y}_j$$

(25)

and for every $y_j$

$$P_{Y|X}\left(y_j\right|x_i)>0,\mathrm{for}\mathrm{exactly}\mathrm{one}{x}_i.$$

(26)

Note that in this case $P_{Y|X}\left(y_j\right|x_i)$ can still take a zero value for some $y_j$ and that ${\varphi }_i\left(y_j\right)=0$ for any non-zero $P_{Y|X}\left(y_j\right|x_i)$. Thus, the error probability is equal to zero $P_{err}\left(x_i\right)=0$; for all $x_i$, as well as the average probability of error ${\overline{P}}_{err}=0$, which means that perfect detection is possible by means of a maximum likelihood detector.

Noisy channel with non-overlapping outputs: A simple example of a perfect transmission channel is the noisy channel with non-overlapping outputs (NOC), which is schematically described in Figure 1. It is a 2-input $m=2k$-output channel ($k\in \mathbb{N}$) defined by the transition matrix:

$$P_{Y|X}=\left[\begin{array}{c}{P}_{Y|X}(·|x_1)\\ P_{Y|X}(·|x_2)\end{array}\right]=\left[\begin{array}{cccccc}\frac{1}{k}& \dots & \frac{1}{k}& 0& \dots & 0\\ 0& \dots & 0& \frac{1}{k}& \dots & \frac{1}{k}\end{array}\right]$$

(27)

(in this and in the following matrices, the symbol “⋯” stands for the k-time repletion). In the case of $k=1$ and $m=2k=2$, the channel reduces to the noiseless channel. Although the channel is noisy, the input can always be recovered from the output (if $y_j$ is received and $j\le k$, the input symbol $x_1$ is sent, otherwise $x_2$ is sent). Thus, it is expected that the information which is passed through the channel is equal to the information that can be generated by the input. Note that for a channel input distributed in accordance with

$$P_X=\left[\begin{array}{c}{P}_X\left(x_1\right)\\ P_X\left(x_2\right)\end{array}\right]=\left[\begin{array}{c}a\\ 1-a\end{array}\right];0\le a\le 1,$$

(28)

the joint probability distribution $P_{X,Y}$ can be expressed as in:

$$P_{X,Y}=\left[\begin{array}{cccccc}\frac{a}{k}& \dots & \frac{a}{k}& 0& \dots & 0\\ 0& \dots & 0& \frac{1-a}{k}& \dots & \frac{1-a}{k}\end{array}\right]$$

(29)

and the output distribution $P_Y$, which can be obtained by the summations over columns, is

$$P_Y={\left[P_Y\left(y_1\right),\dots ,P_Y\left(y_m\right)\right]}^T={\left[\frac{a}{k},\dots ,\frac{a}{k},\frac{1-a}{k},\dots ,\frac{1-a}{k}\right]}^T.$$

(30)

Figure 1. Noisy channel with non-overlapping outputs.

Binary symmetric channels: The binary symmetric channel (BSC) is a two input two output channel described by the transition matrix

$$P_{Y|X}=\left[\begin{array}{c}{P}_{Y|X}{(·|x_1)}^T\\ P_{Y|X}{(·|x_2)}^T\end{array}\right]=\left[\begin{array}{cc}1-p& p\\ p& 1-p\end{array}\right],$$

(31)

which is schematically described in Figure 2. Note that for $p=\frac{1}{2}$ BSC reduces to a totally destructive channel, while in the case of $p=0$ it reduces to a perfect channel.

Figure 2. Binary symmetric channel.

Sharma–Mittal Information Transfer Axioms

In this paper, we search for information theoretical measures of information transfer between sender X and receiver Y, which communicate through a channel if the information is measured with Sharma–Mittal entropy. Thus, we are interested in the information transfer measure, $I_{\alpha ,q}(X,Y)$, which is called the $\alpha$-q-mutual information and its maximum,

$$C=\underset{P_X}{max}{I}_{\alpha ,q}(X,Y),$$

(32)

which is called the $\alpha$-q-capacity and which requires the following set of axioms to be satisfied.

(A₁)

The channel cannot convey negative information, i.e.,

$$C_{\alpha ,q}\left(P_{Y|X}\right)\ge I_{\alpha ,q}(X,Y)\ge 0.$$

(33)

(A₂)

The information transfer is zero in the case of a totally destructive channel, i.e.,

$$P_{Y|X}\left(y\right|x)=\frac{1}{m},\mathrm{for}\mathrm{all}x,y\Rightarrow I_{\alpha ,q}(X,Y)=C_{\alpha ,q}\left(P_{Y|X}\right)=0,$$

(34)

which is consistent with the conclusion that the average probability of error is one, ${\overline{P}}_{err}=1$, in the case of a totally destructive channel.

(A₃)

In the case of perfect transmission, the information transfer is equal to the input information, i.e.,

$$X=Y\Rightarrow I_{\alpha ,q}(X,Y)=H_{\alpha ,q}\left(X\right),C_{\alpha ,q}\left(P_{Y|X}\right)={\mathrm{Log}}_{q}n,$$

(35)

which is consistent with the conclusion that the average probability of error is zero, ${\overline{P}}_{err}=0$, in the case of a perfect transmission channel, so that all the information from the input is conveyed.

(A₄)

The channel cannot transfer more information than it is possible to be sent, i.e.,

$$I_{\alpha ,q}(X,Y)\le C_{\alpha ,q}\left(P_{Y|X}\right)\le {\mathrm{Log}}_{q}n,$$

(36)

which means that a channel cannot add additional information.

(A₅)

The channel cannot transfer more information than it is possible to be received, i.e.,

$$I_{\alpha ,q}(X,Y)\le C_{\alpha ,q}\left(P_{Y|X}\right)\le {\mathrm{Log}}_{q}m,$$

(37)

which means that a channel cannot add additional information.

(A₆)

Consistency with the Shannon case:

$$\underset{q\to 1,\alpha \to 1}{lim}{I}_{\alpha ,q}(X,Y)=I(X,Y),\mathrm{and}\underset{q\to 1,\alpha \to 1}{lim}{C}_{\alpha ,q}\left(P_{Y|X}\right)=C\left(P_{Y|X}\right)$$

(38)

Thus, the axioms (${\overline{A}}_2$) and (${\overline{A}}_3$) ensure that the information measures are consistent with the maximum likelihood detection (19)–(21). On the other hand, the axioms (${\overline{A}}_1$), (${\overline{A}}_4$) and (${\overline{A}}_5$), prevent a situation in which a physical system conveys information in spite of going through a completely destructive channel, or in which the negative information transfer is observed, indicating that the channel adds or removes information by itself, which could be treated as nonphysical behavior without an intuitive explanation. Finally, the property (${\overline{A}}_6$) ensure that the information transfer measures can be considered as generalizations of corresponding Shannon measures. For these reasons, we assume that the satisfaction of the properties (${\overline{A}}_1$)–(${\overline{A}}_5$) is mandatory for any reasonable definition of Sharma–Mittal information transfer measures.

4. The α-Mutual Information and the α-Capacity

One of the first proposals for the Rényi mutual information goes back to Arimoto [24], who considered the following definition of mutual information:

$$I_{\alpha }(X,Y)=\frac{\alpha }{1-\alpha }log\left(\sum _y{\left(\sum _x{P}_X^{\left(\alpha \right)}\left(x\right)P_{Y|X}^{\alpha }\left(y\right|x)\right)}^{\frac{1}{\alpha }}\right),$$

(39)

where the escort distribution $P_{X^{\left(\alpha \right)}}$ is defined as in (10), and he also invented an iterative algorithm for the computation of the $\alpha$-capacity [35], which is defined from the $\alpha$-mutual information:

$$C_{\alpha }\left(P_{Y|X}\right)=\underset{P_X}{max}{I}_{\alpha }(X,Y).$$

(40)

Notably, Arimoto’s mutual information can equivalently be represented using the conditional Rényi entropy

$$R_{\alpha }\left(X\right|Y)=\frac{\alpha }{\alpha -1}{log}_2\sum _y{P}_Y\left(y\right){\left(\sum _x{P}_{X|Y=y}{\left(x\right)}^{\alpha }\right)}^{\frac{1}{\alpha }},$$

(41)

as in

$$I_{\alpha }(X,Y)\equiv R_{\alpha }\left(X\right)-R_{\alpha }\left(X\right|Y),$$

(42)

which can be interpreted as the input uncertainty reduction after the output symbols are received and, in the case of $\alpha \to 1$, the previous definition reduces to the Shannon case. In addition, this measure is directly related to the famous Gallager exponent

$$E_0\left(\rho ,P_X\right)=-log\left(\sum _y{\left(\sum _x{P}_X\left(x\right)P_{Y|X}^{\frac{1}{1+\rho }}\left(y\right|x)\right)}^{1+\rho }\right),$$

(43)

which has been widely used to establish the upper bound of error probability in channel coded communication systems [36] via the relationship [29]

$$I_{\alpha }(X,Y)=\frac{\alpha }{1-\alpha }{E}_0\left(\frac{1}{\alpha }-1,P_X^{\left(\alpha \right)}\right).$$

(44)

In addition, in the case of $\alpha \to 1$, it reduces to

$$I_1(X,Y)=\underset{\alpha \to 1}{lim}{I}_{\alpha }(X,Y)=I(X,Y),$$

(45)

where

$$I(X,Y)=\sum _{x,y}{P}_{X,Y}(x,y)log\frac{P_{X,Y}(x,y)}{P_X\left(x\right)P_Y\left(y\right)}$$

(46)

stands for Shannon’s mutual information [37].

The $\alpha$-mutual information $I_{\alpha }(X,Y)$ and the $\alpha$-capacity $C_{\alpha }\left(P_{Y_X}\right)$ satisfy the axioms (${\overline{A}}_1$)–(${\overline{A}}_6$) for $q=1$ and $\alpha >0$, as stated by the following theorem, which further justifies their usage as the measures of (maximal) information transfer.

Theorem 1.

The mutual information measures $I_{\alpha }$ and $C_{\alpha }$ satisfy the following set of properties:

(A₁)

The channel cannot convey negative information, i.e.,

$$C_{\alpha }\left(P_{Y|X}\right)\ge I_{\alpha }(X,Y)\ge 0.$$

(47)

(A₂)

The (maximal) information transfer is zero in the case of a totally destructive channel, i.e.,

$$P_{Y|X}\left(y\right|x)=\frac{1}{m},\mathit{for}\mathit{all}x,y\Rightarrow I_{\alpha }(X,Y)=C_{\alpha }\left(P_{Y|X}\right)=0.$$

(48)

(A₃)

In the case of perfect transmission, the (maximal) information transfer is equal to the (maximal) input information, i.e.,

$$X=Y\Rightarrow I_{\alpha }(X,Y)=R_{\alpha }\left(X\right),C_{\alpha }\left(P_{Y|X}\right)=logn.$$

(49)

(A₄)

The channel cannot transfer more information than it is possible to be sent, i.e.,

$$I_{\alpha }(X,Y)\le C_{\alpha }\left(P_{Y|X}\right)\le logn;$$

(50)

(A₅)

The channel cannot transfer more information than it is possible to be received, i.e.,

$$I_{\alpha }(X,Y)\le C_{\alpha }\left(P_{Y|X}\right)\le logm.$$

(51)

(A₆)

Consistency with the Shannon case:

$$\underset{\alpha \to 1}{lim}{I}_{\alpha }(X,Y)=I(X,Y),\mathit{and}\underset{\alpha \to 1}{lim}{C}_{\alpha }\left(P_{Y|X}\right)=C\left(P_{Y|X}\right)$$

(52)

Proof. 

As shown in [38], $R_{\alpha }\left(X\right|Y)\le R_{\alpha }\left(X\right)$, and the nonnegativity property ($A_1$) follows from the definition of Arimoto’s mutual information (42). In addition, if $X\perp \perp Y$, then $P_{Y|X}\left(y\right|x)=P_Y\left(y\right)$ so that the definition (61) implies the property ($A_2$). Furthermore, in the case of a perfect transmission channel, the mutual information (61) can be represented in -4.6cm0cm

$$I_{\alpha }(X,Y)=\frac{\alpha }{\alpha -1}log\frac{{\sum }_y{\left({\sum }_x{P}_X{\left(x\right)}^{\alpha }{P}_{Y|X}^{\alpha }\left(y\right|x)\right)}^{\frac{1}{\alpha }}}{{\left({\sum }_x{P}_X^{\left(\alpha \right)}\left(x\right)\right)}^{\frac{1}{\alpha }}}=\frac{\alpha }{\alpha -1}log\frac{{\sum }_y{\left(P_X{\left(d\left(y\right)\right)}^{\alpha }{P}_{Y|X}^{\alpha }(y\mid d\left(y\right))\right)}^{\frac{1}{\alpha }}}{{\left({\sum }_x{P}_X^{\left(\alpha \right)}\left(x\right)\right)}^{\frac{1}{\alpha }}},$$

(53)

and since

$$\begin{array}{c}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\sum _y{\left(P_X{\left(d\left(y\right)\right)}^{\alpha }{P}_{Y|X}^{\alpha }(y\mid d\left(y\right))\right)}^{\frac{1}{\alpha }}=\sum _y{P}_X\left(d\left(y\right)\right)P_{Y|X}(y\mid d\left(y\right))=\\ \hfill \sum _x\sum _{y:d\left(y\right)=x}{P}_X\left(d\left(y\right)\right)P_{Y|X}(y\mid d\left(y\right))=\sum _x{P}_X\left(x\right)\sum _{y:d\left(y\right)=x}{P}_{Y|X}\left(y\right|x)=1,\end{array}$$

(54)

we obtain $I_{\alpha }(X,Y)=R_{\alpha }\left(X\right)$, which proves the property ($A_3$). Moreover, from the definition as shown in [38], Arimoto’s conditional entropy is positive and satisfies the weak chain rule $R_{\alpha }\left(X\right|Y)\ge R_{\alpha }\left(X\right)-logm$, so that the properties ($A_4$) and ($A_5$) follow from the definition of Arimoto’s mutual information (42). Finally, the property ($A_6$) follows directly from the equation (45) and can be approved using L’Hôpital’s rule, which completes the proof of the theorem. □

5. Alternative Definitions of the α-Mutual Information and the α-Channel Capacity

Since Rényi’s proposal, there have been several lines of research to find an appropriate definition and characterization of information transfer measures related to Rényi entropy, which are established by the substitution of the Rényi divergence measure

$$D_{\alpha }\left(P\right|\left|Q\right)=\frac{1}{\alpha -1}log\left(\sum _{x}P{\left(x\right)}^{\alpha }Q{\left(x\right)}^{1-\alpha }\right),$$

(55)

instead of the Kullback–Leibler one,

$$D\left(P\right|\left|Q\right)=D_1\left(P\right|\left|Q\right)=\sum _{x}P\left(x\right)log\frac{P\left(x\right)}{Q\left(x\right)},$$

(56)

in some of the various definitions which are equivalent in the case of Shannon information measures (46) [29]:

$$\begin{array}{cc}\hfill I(X,Y)& =\underset{Q_Y}{min}\mathbb{E}\left[D_{\alpha }\left(P_{Y|X}\parallel Q_Y\right)\right]=\underset{Q_Y}{min}\mathbb{E}\left[D_{\alpha }\left(P_{Y|X}\parallel Q_Y\right)\right]\hfill \\ \hfill & =\underset{Q_X}{min}\underset{Q_Y}{min}{D}_{\alpha }\left(P_{X,Y}\parallel Q_X{Q}_Y\right)=D_{\alpha }\left(P_{X,Y}\parallel P_X{P}_Y\right)=S\left(X\right)-S\left(X\right|Y)\hfill \end{array}$$

(57)

where $S\left(X\right|Y)$ stands for the Shannon conditional entropy,

$$S\left(X\right|Y)=\sum _{x,y}{P}_{X,Y}(x,y)log{P}_{X|Y}\left(x\right|y).$$

(58)

All of these measures are consistent with the Shannon case in view of the property ($A_6$), but their direct usage as measures of Rényi information transfer leads to a breaking of some the properties ($A_1$)–($A_5$), which justifies the usage of Arimoto’s measures from the previous section as appropriate ones in the context of this research. In the following section, we review the alternative definitions.

5.1. Information Transfer Measures by Sibson

Alternative approaches based on Rényi divergence were proposed by Sibson [23] and considered later by several authors in the context of quantum secure communications [39,40,41,42,43,44], who introduced

$$J_{\alpha }^1(X;Y)=\underset{Q_Y}{min}{D}_{\alpha }\left(P_{Y|X}{P}_X\parallel Q_Y{P}_X\right),$$

(59)

which can be represented as in [26]

$$J_{\alpha }^1(X,Y)=\frac{\alpha }{\alpha -1}log\left(\sum _y{\left(\sum _x{P}_X\left(x\right)P_{Y|X}^{\alpha }\left(y\right|x)\right)}^{\frac{1}{\alpha }}\right)$$

(60)

and, in the discrete setting, can be related to the Gallager exponent as in [29]:

$$J_{\alpha }^1(X,Y)=\frac{\alpha }{1-\alpha }{E}_0\left(\frac{1}{\alpha }-1,P_X\right),$$

(61)

which differs from Arimoto’s definition (61) since in this case the escort distribution does not participate in the error exponent, but an ordinary one does. However, in the case of a perfect channel for which $X=Y$, the conditional distribution $P_{Y|X}^{\alpha }\left(y\right|x)=1$ for $x=y$ and zero otherwise, so Sibson’s measure (60) reduces to $R_{1/\alpha }\left(X\right)$, thus breaking the axiom ($A_3$). This disadvantage can be overcome by the reparametrization $\alpha \leftrightarrow 1/\alpha$ so that $J_{1/\alpha }^1(X,Y)$ is used as a measure of Rényi information transfer, and the properties of the resulting measure can be considered in a manner similar to the case of Arimoto.

5.2. Information Transfer Measures by Augustin and Csiszar

An alternative definition of Rényi mutual information was also presented by Augustin [25], and later Csiszar [26], who defined

$$J_{\alpha }^2(X;Y)=\underset{Q_Y}{min}\mathbb{E}\left[D_{\alpha }\left(P_{Y|X}\parallel Q_Y\right)\right],$$

(62)

However, in the case of perfect transmission, for which $X=Y$, the measure reduces to Shannon entropy

$$J_{\alpha }^2(X;Y)=S\left(X\right),$$

(63)

which breaks the axiom ($A_3$).

5.3. Information Transfer Measures by Lapidoth, Pfister, Tomamichel and Hayashi

A similar obstacle to the case of the Augustin–Csiszar measure can be observed in the case of mutual information which was considered by Lapidoth and Pfister [27] and Tomamichel and Hayashi [28], who proposed

$$J_{\alpha }^3(X;Y)=\underset{Q_X}{min}\underset{Q_Y}{min}{D}_{\alpha }\left(P_{X,Y}\parallel Q_X{Q}_Y\right).$$

(64)

As shown in [27] (Lemma 11), if $X=Y$, then

$$J_{\alpha }^3(X;Y)=\left\{\begin{array}{cc}\frac{\alpha }{1-\alpha }\underset{\alpha \to \infty }{lim}{R}_{\alpha }\left(X\right)\hfill & \mathrm{if}\alpha \in \left[0,\frac{1}{2}\right],\hfill \\ R_{\frac{\alpha }{2\alpha -1}}\left(X\right)\hfill & \mathrm{if}\alpha >\frac{1}{2}\hfill \end{array}\right.$$

(65)

so the axiom ($A_3$) is broken in this case, as well.

Remark 1.

Despite the difference between the definitions of information transfer, in the discrete setting, the alternative definitions discussed above reach the same maximum over the set of input probability distributions, $P_X$,26,29,45].

5.4. Information Transfer Measures by Chapeau-Blondeau, Delahaies, Rousseau, Tridenski, Zamir, Ingber and Harremoes

Chapeau-Blondeau, Delahaies and Rousseau [31], and independently Tridenski, Zamir and Ingber [46] and Harremoes [47], defined the Rényi mutual information using the Rényi divergence (55), so that the mutual information defined using the Rényi divergence

$$J_{\alpha }^4(X,Y)=D_{\alpha }\left(P_{X,Y}\parallel P_X{P}_Y\right)$$

(66)

for $\alpha >0$ and $\alpha \ne 1$, while in the case of $\alpha =1$ it reduces to Shannon mutual information. However, the ordinal definition can correspond only to a Rényi entropy of order $2-\alpha$ since in the case of $X=Y$ it reduces to $J_{\alpha }^4(X,Y)=R_{2-\alpha }\left(X\right)$ (see also [47]), which can be overcome by the reparametrization $\alpha =2-q$, similar to the case of Sibson’s measure. This measure has been discussed in the past with various operational characterizations, and could also be considered as a measure of information transfer, although the satisfaction of all of the axioms ($A_1$)–($A_6$) is not self-evident for general channels.

5.5. Information Transfer Measures by Jizba, Kleinert and Shefaat

Finally, we will mention the definition by Jizba, Kleinert and Shefaat [48],

$$J_{\alpha }^4(X,Y)\equiv R_{\alpha }\left(X\right)-{\widehat{R}}_{\alpha }\left(X\right|Y),$$

(67)

which is defined in the same manner as in Arimoto’s case (42), but with another choice of conditional Rényi entropy

$${\widehat{R}}_{\alpha }\left(X\right|Y)=\frac{1}{1-\alpha }log\sum _x{P}_X^{\left(\alpha \right)}\left(x\right)2^{(1-\alpha )R_{\alpha }\left(X\right|Y=y)},$$

(68)

which arises from the Generalized Shannon–Khinchin axiom [GSK4] if the pseudo-additivity in the equation (9) is restricted to an ordinary addition, in which case the GSK axioms uniquely determine Rényi entropy [49]. However, despite its wide applicability in the modeling of causality and financial time series, this mutual information can take negative values which breaks the axiom ($A_1$), which is assumed to be mandatory in this paper. For further discussion of the physicalism of negative mutual information in the domain of financial time series analysis, the reader is referred to [48].

6. The α-q Mutual Information and the α-q-Capacity

In the past several attempts have been done to define an appropriate channel capacity measure which corresponds to instances of the Sharma–Mittal entropy class. All of them follow a similar recipe by which the channel capacity is defined as in (32), as a maximum of appropriately defined mutual information $I_{\alpha ,q}$. However, all of the classes consider only special cases of Sharma–Mittal entropy and all of them fail to satisfy at least one of the properties (${\overline{A}}_1$)–(${\overline{A}}_6$) which an information transfer has to satisfy, as will be discussed Section 7.

In this section we propose a general measures of the $\alpha$-q mutual information and the $\alpha$-q-capacity by the requirement that the axioms (${\overline{A}}_1$)–(${\overline{A}}_6$) are satisfied, which could qualify them as appropriate measures of information transfer, without nonphysical properties. The special instances of the $\alpha$-q (maximal) information transfer measures are also discussed and the analytic expressions for a binary symmetric channel are provided.

6.1. The α-q Information Transfer Measures and Its Instances

The $\alpha$-q-mutual information (42) is defined using the q-subtraction defined in (6), as follows:

$$I_{\alpha ,q}(X,Y)=H_{\alpha ,q}\left(X\right){\ominus }_q{H}_{\alpha ,q}\left(X\right|Y),$$

(69)

where we introduced the conditional Sharma–Mittal entropy $H_{\alpha ,q}\left(Y\right|X)$ as in

$$H_{\alpha ,q}\left(X\right|Y)={\eta }_q\left(R_{\alpha }\left(X\right|Y)\right)=\frac{1}{(1-q)ln2}\left({\left(\sum _y{P}_Y\left(y\right){\left(\sum _x{P}_{X|Y=y}{\left(x\right)}^{\alpha }\right)}^{\frac{1}{\alpha }}\right)}^{\frac{\alpha (1-q)}{\alpha -1}}-1\right),$$

(70)

$R_{\alpha }\left(X\right|Y)$ stands for Arimoto’s definition of the conditional Rényi entropy (41). The expression (69) can also be obtained if the mapping ${\eta }_q$ is applied to both sides of the equality (42), by which Arimoto’s mutual information is defined, so we may establish the relationship

$$I_{\alpha ,q}(X,Y)={\eta }_q\left(I_{\alpha }(X,Y)\right)={\eta }_q\left(\frac{\alpha }{1-\alpha }log\left(\sum _y{\left(\sum _x{P}_X^{\left(\alpha \right)}\left(x\right)P_{Y|X}^{\alpha }\left(y\right|x)\right)}^{\frac{1}{\alpha }}\right)\right),$$

(71)

which can be represented using the Gallager error exponent (43) as in

$$I_{\alpha ,q}(X,Y)={\eta }_q\left(\frac{\alpha }{1-\alpha }{E}_0\left(\frac{1}{\alpha }-1,P_X^{\left(\alpha \right)}\right)\right)=\frac{1}{(1-q)ln2}\left(2^{\frac{\alpha (1-q)}{1-\alpha }{E}_0\left(\frac{1}{\alpha }-1,P_X^{\left(\alpha \right)}\right)}-1\right).$$

(72)

Arimoto’s $\alpha$-q-capacity is now defined in

$$C_{\alpha ,q}=\underset{P_X}{max}{I}_{\alpha ,q}(X,Y),$$

(73)

and using the fact that ${\eta }_q$ is increasing, it can be related with the corresponding $\alpha$-capacity as in

$$C_{\alpha ,q}=\underset{P_X}{max}{I}_{\alpha ,q}(X,Y)=\underset{P_X}{max}{\eta }_q\left(I_{\alpha }(X,Y)\right)={\eta }_q\left(\underset{P_X}{max}{I}_{\alpha }(X,Y)\right)={\eta }_q\left(C_{\alpha }\left(P_{Y|X}\right)\right).$$

(74)

Using the expressions (45) and (71), in the case of $\alpha =1$, the $\alpha$-q mutual information reduces to

$$\begin{array}{cc}\hfill I_{1,q}& =\frac{1}{(1-q)ln2}\left(\underset{x,y}{\Pi }{2}^{P_{X,Y}(x,y)log\frac{P_{X,Y}(x,y)}{P_X\left(x\right)P_Y\left(y\right)}}-1\right)\hfill \\ \hfill & =\frac{1}{(1-q)ln2}\left(\underset{x,y}{\Pi }{\left(\frac{P_{X,Y}(x,y)}{P_X\left(x\right)P_Y\left(y\right)}\right)}^{P_{X,Y}(x,y)}-1\right).\hfill \end{array}$$

(75)

The $\alpha$-q-capacity is given in

$$C_{1,q}=\underset{P_X}{max}\left(\frac{1}{(1-q)ln2}\left(\underset{x,y}{\Pi }{\left(\frac{P_{X,Y}(x,y)}{P_X\left(x\right)P_Y\left(y\right)}\right)}^{P_{X,Y}(x,y)}-1\right)\right)$$

(76)

and these measures can serve as (maximal) information transfer measures corresponding to Gaussian entropy, which was not considered before in the context of information transmission. Naturally, if in addition $q\to 1$, the measures reduce to Shannon’s mutual information and Shannon capacity [37].

Additional special cases of the $\alpha$-q (maximal) information transfer include the $\alpha$-mutual information (42) and the $\alpha$-capacity (40), which are obtained for $q=1$; the measures which correspond to Tsallis entropy can be obtained for $q=\alpha$ and the ones which correspond to Landsberg–Vedral entropy for $q=2-\alpha$. These special instances are listed in Table 1.

Table 1. Instances of the $\alpha$-q-mutual information for different values of the parameters and corresponding expressions for the BSC $\alpha$-q-capacities.

${\mathit{H}}_{\mathit{\alpha },\mathit{q}}$	${\mathit{I}}_{\mathit{\alpha },\mathit{q}}$	${\mathit{C}}_{\mathit{\alpha },\mathit{q}}$
S $\alpha =q=1$	${\sum }_{x,y}{P}_{X,Y}(x,y)log{\displaystyle \frac{P_{X,Y}(x,y)}{P_X\left(x\right)P_Y\left(y\right)}}$	$1+plogp+(1-p)log(1-p)$
$R_{\alpha }$ $q=1$	${\displaystyle \frac{\alpha }{1-\alpha }}{E}_0\left(\frac{1}{\alpha }-1,P_X^{\left(\alpha \right)}\right)$	$1-{\displaystyle \frac{log(p^{\alpha }+{(1-p)}^{\alpha })}{1-\alpha }}$
$T_q$ $q=\alpha$	${\displaystyle \frac{1}{(1-q)ln2}}\left(2^{q{E}_0\left(\frac{1}{q}-1,P_X^{\left(q\right)}\right)}-1\right)$	${\displaystyle \frac{1}{(1-q)ln2}}\left(2^{1-q}{(p^q+{(1-p)}^q)}^{-1}-1\right)$
$L_{\alpha }$ $q=2-\alpha$	${\displaystyle \frac{1}{(\alpha -1)ln2}}\left(2^{-\alpha E_0\left(\frac{1}{\alpha }-1,P_X^{\left(\alpha \right)}\right)}-1\right)$	${\displaystyle \frac{1}{(1-\alpha )ln2}}\left(2^{\alpha -1}(p^{\alpha }+{(1-p)}^{\alpha })-1\right)$
$G_q$ $\alpha =1$	${\displaystyle \frac{1}{(1-q)ln2}}\left({\Pi }_{x,y}{\left({\displaystyle \frac{P_{X,Y}(x,y)}{P_X\left(x\right)P_Y\left(y\right)}}\right)}^{P_{X,Y}(x,y)}-1\right)$	${\displaystyle \frac{1}{(1-q)ln2}}\left(2^{1-q}{p}^{(1-q)p}{(1-p)}^{(1-q)(1-p)}-1\right)$
$E_0\left(\rho ,P_X\right)=-log\left({\sum }_y{\left({\sum }_x{P}_X\left(x\right)P_{Y|X}^{\frac{1}{1+\rho }}\left(y\right|x)\right)}^{1+\rho }\right)$

As discussed in Section 7, previously considered information measures cover only particular special cases and break at least one of the axioms (${\overline{A}}_1$)–(${\overline{A}}_5$), which leads to unexpected and counterintuitive conclusions about the channels, such as negative information transfer and achieving super-capacitance or sub-capacitance [4], which could be treated as a nonphysical behavior. On the other hand, apart from the generality, the $\alpha$-q information transfer measures proposed in this paper overcame the disadvantages which could qualify them as appropriate measures, as stated in the following theorem.

Theorem 2.

The α-q information transfer measures $I_{\alpha ,q}$ and $C_{\alpha ,q}$ satisfy the set of the axioms (${\overline{A}}_1$)–(${\overline{A}}_6$).

Proof. 

The proof is the straightforward application of the mapping ${\eta }_q$ to the equations in the $\alpha$-mutual information properties ($A_1$)–($A_5$), while the (${\overline{A}}_6$) follows from the above discussion. □

Remark 2.

Note that the symmetry $I_{\alpha ,q}(X,Y)=I_{\alpha ,q}(Y,X)$ does not hold in general in the case of the α-q mutual information nor in the case of the α mutual information [50,51] and if the mutual information is defined so that the symmetry is preserved, some of the axioms (${\overline{A}}_1$)–(${\overline{A}}_6$) might be broken. In addition, the alternative definition of the mutual information, $I_{\alpha ,q}(Y,X)=H_{\alpha ,q}\left(Y\right)-H_{\alpha ,q}\left(Y\right|X)$, which uses an ordinary substraction operator instead of ${\ominus }_q$ operation, can also be introduced, but in this case the property (${\overline{A}}_5$) might not hold in general, as discussed in Section 7.

6.2. The α-q-Capacity of Binary Symmetric Channels

As shown by Cai and Verdú [45], the $\alpha$-mutual information of Arimoto’s type $I_{\alpha }$ is maximized for the uniform distribution $P_X=(1/2,1/2)$, and Arimoto’s $\alpha$-capacity has the value

$$C_{\alpha }\left(BSC\right)=1-r_{\alpha }\left(p\right),$$

(77)

where the binary entropy function $r_{\alpha }$ is defined as

$$r_{\alpha }\left(p\right)=R_{\alpha }(p,1-p)=\frac{1}{1-\alpha }log(p^{\alpha }+{(1-p)}^{\alpha }),$$

(78)

for $\alpha >0$, $\alpha \ne 1$, while in the limit of $\alpha \to 1$, the expression (78) reduces to the well-known result for the Shannon capacity (see Fano [52])

$$C_1\left(BSC\right)=\underset{\alpha \to 1}{lim}{C}_{\alpha }\left(BSC\right)=1+plogp+(1-p)log(1-p).$$

(79)

The analytic expressions for the $\alpha$-q-capacities of binary symmetric channel’s can be obtained from the expressions (74) and (77), so that

$$C_{\alpha ,q}\left(BSC\right)=q\left(C_{\alpha }\left(BSC\right)\right)=\frac{1}{(1-q)ln2}\left(2^{1-q}{\left(p^{\alpha }+{(1-p)}^{\alpha }\right)}^{-\frac{1-q}{1-\alpha }}-1\right);$$

(80)

in the case of $q=1$, it reduces to the case of Rényi entropy while, in the case of $\alpha =1$, to the case of Gaussian entropy (77)

$$C_{1,q}\left(BSC\right)=\frac{1}{(1-q)ln2}\left(2p^p{(1-p)}^{1-p}-1\right).$$

(81)

The analytic expressions for BSC $\alpha$-q capacities for other instances can straightforwardly be obtained by specifying the values of the parameters, whose instances are listed in Table 1, while the plots of the BSC $\alpha$-q-capacities, which correspond to the Gaussian and the Tsallis entropies, are shown in Figure 3 and Figure 4.

Figure 3. The $\alpha$-q-capacity of BSC for the Gaussian entropy (the case of $\alpha =1$) as a function of q for various values of the channel parameter p from $0.5$ (totally destructive channel) to 0 (perfect transmission). All of the curves lies between 0 and ${\mathrm{Log}}_{q}2$, which is the maximum value of the Gaussian entropy.

Figure 4. The $\alpha$-q-capacity of BSC for the Tsallis entropy (the case of $\alpha =q$) as a function of q for various values of the channel parameter p from $0.5$ (totally destructive channel) to 0 (perfect transmission). All of the curves lies between 0 and ${\mathrm{Log}}_{q}2$, which is the maximum value of the Tsallis entropy.

The $\alpha$-q-capacity (80) can equivalently be expressed in

$$C_{\alpha ,q}\left(BSC\right)={\mathrm{Log}}_{q}2{\ominus }_q{h}_{\alpha ,q}\left(p\right),$$

(82)

where the Sharma–Mittal binary entropy function is defined in

$$h_{\alpha ,q}\left(p\right)=H_{\alpha ,q}(p,1-p)=\frac{1}{1-q}\left({(p^{\alpha }+{(1-p)}^{\alpha })}^{\frac{1-q}{1-\alpha }}-1\right),$$

(83)

which reduces to the Rényi binary entropy function, in the case of $q=1$,

$$h_{\alpha ,1}\left(p\right)=\underset{q\to 1}{lim}{h}_{\alpha ,q}\left(p\right)=R_{\alpha }(p,1-p)=\frac{1}{1-\alpha }log\left(p^{\alpha }+{(1-p)}^{\alpha })\right),$$

(84)

to the Tsallis binary entropy function, in the case of $\alpha =1$,

$$h_{q,q}\left(p\right)=h_{q,q}\left(p\right)=T_q(p,1-p)=\frac{1}{1-q}\left(p^q+{(1-p)}^q-1\right),$$

(85)

to the Gaussian binary entropy function, in the case of $\alpha =1$,

$$h_{1,q}\left(p\right)=\underset{\alpha \to 1}{lim}{h}_{\alpha ,q}\left(p\right)=G_q(p,1-p)={\displaystyle \frac{1}{(1-q)ln2}}\left(p^{-(1-q)p}{(1-p)}^{-(1-q)(1-p)}-1\right),$$

(86)

and to the Shannon binary entropy function, in the case of $\alpha =q=1$,

$$h_{1,1}\left(p\right)=\underset{q,\alpha \to 1}{lim}{h}_{\alpha ,q}\left(p\right)=S(p,1-p)=-plogp-(1-p)log(1-p).$$

(87)

The expression (82) can be interpreted similarly as in the Shannon case. Thus, a BSC channel with input X and output Y can be modeled with an input–output relation $Y=X\oplus Z$ where ⊕ stands for modulo 2 sum and Z is channel noise taking values from $\{1,0\}$, distributed in accordance with $(p,1-p)$. If we measure the information which is lost per bit during transmission with the Sharma–Mittal entropy $H_{\alpha ,q}\left(Z\right)=h_{\alpha }\left(p\right)$, then $C_{\alpha ,q}$ stands for useful information left over for every bit of information received.

7. An Overview of the Previous Approaches to Sharma–Mittal Information Transfer Measures

In this section, we review the previous attempts at a definition of Sharma–Mittal information transfer measures, which are defined from the basic requirement of consistency with the Shannon measure as given by the axiom (${\overline{A}}_6$). However, as we show in the following paragraphs, all of them break at least one of the axioms (${\overline{A}}_1$)–(${\overline{A}}_5$), which are satisfied in the case of the $\alpha$-q (maximal) information transfer measures (69) and (73), in accordance with the discussion in Section 6.

7.1. Daróczy’s Capacity

The first considerations of generalized channel capacities and generalized mutual information for the q-entropy go back to Daróczy [30], who introduced conditional Tsallis entropy

$${\overline{T}}_q\left(Y\right|X)=\sum _x{P}_X^q\left(x\right)T_q\left(Y\right|X=x),$$

(88)

where the row entropies are defined as in

$$T_q\left(Y\right|X=x)=\frac{1}{(1-q)log\left(2\right)}\left(\sum _x{P}_{Y|X}{\left(y\right|x)}^q-1\right)$$

(89)

and the mutual information is defined as in

$$J_{\alpha ,q}^5(X,Y)=T_q\left(Y\right)-{\overline{T}}_q\left(Y\right|X).$$

(90)

However, in the case of a totally destructive channel, $X\perp \perp Y$, $P_{Y|X}\left(y\right|x)=P_Y\left(y\right)$, $T_q\left(Y\right|X=x)=T_q\left(Y\right)$ and

$$T_q\left(Y\right|X)=T_q\left(Y\right)\sum _x{P}_X{\left(x\right)}^q$$

(91)

so that

$$J_{\alpha ,q}^5(X,Y)=T_q\left(Y\right)\left(1-\sum _x{P}_X{\left(x\right)}^q\right)=\left(1-\sum _x{P}_X{\left(x\right)}^q\right){\mathrm{Log}}_{q}m.$$

(92)

This expression is zero for an input probability distribution $P_X=(1,0,\dots ,0)$ and its permutations, but, in general, it is negative for $q<1$, positive for $q>1$ and 0 only for $q=1$, so the axiom (${\overline{A}}_2$) is broken (see Figure 5). As a result, the channel capacity, which is defined in accordance to (32), is zero for $q\le 1$ and positive for $q>1$, as illustrated in Figure 6 by the example of BSC for which the Daroczy’s channel capacity can be computed as in [30,53]

$$C_q^5\left(BSC\right)=\frac{1-2^{1-q}}{q-1}-\frac{2^{-q}}{q-1}[1-{(1-p)}^q-p^q].$$

(93)

In the same figure, we plotted the graph for the $\alpha$-q channel capacities proposed in this paper, and all of them remain zero in the case of a totally destructive BSC, as expected.

Figure 5. Daróczy’s (solid lines) and Yamano’s (dashed lines) mutual information in the case of a totally destructive BSC as functions of the input distribution parameter a, $P_X={[a,1-a]}^T$ for different values of q, obtaining negative values for $q<1$ and $q>1$, respectively, breaking the axioms (${\overline{A}}_1$) and (${\overline{A}}_2$). The $\alpha$-q-mutual information is zero; for all q, and satisfies (${\overline{A}}_1$) and (${\overline{A}}_2$).

Figure 6. Daróczy’s (solid lines) and Yamano’s (dashed lines) capacities in the case of totally destructive BSC as functions of the parameter q. In the regions of $q<1$ and $q>1$, respectively, the corresponding negative mutual information is maximized for $P_X={[1,0]}^T$ (zero capacity) having the positive values outside the regions and breaking the axiom (${\overline{A}}_2$). The $\alpha$-q-capacity is zero; for all q, and satisfies (${\overline{A}}_2$).

7.2. Yamano Capacities

Similar problems to the ones mentioned above arise in the case of mutual information and corresponding capacity measures considered by Yamano [33], who addressed the information transmission characterized by Landsberg–Vedral entropy $L_q$, given in (17).

Thus, the first proposal is based on the mutual information of the form

$$J_q^6(X,Y)=L_q\left(X\right)+L_q\left(Y\right)-L_q(X,Y),$$

(94)

where the joint entropy is defined in

$$L_q(X,Y)=\frac{1}{q-1}\left(\frac{1}{{\sum }_{x,y}{P}_{X,Y}{(x,y)}^q}-1\right).$$

(95)

However, in the case of a fully destructive channel, $P_Y\left(y\right)=1/m$ and $P_{X,Y}(x,y)=P_X\left(x\right)/m$, so that

$$J_q^6(X,Y)=\frac{1}{q-1}\left(\frac{1}{{\sum }_x{P}_X{\left(x\right)}^q}-1\right)+\frac{1}{q-1}\left(m^{q-1}-1\right)-\frac{1}{q-1}\left(m^{q-1}\frac{1}{{\sum }_x{P}_X{\left(x\right)}^q}-1\right),$$

(96)

which can be simplified to

$$J_q^6(X,Y)=\frac{1-m^{q-1}}{q-1}\left(\frac{1}{{\sum }_x{P}_X{\left(x\right)}^q}-1\right).$$

(97)

Similarly to the case of Daroczy’s capacity, this expression is zero for an input probability distribution $P_X=(1,0,\dots ,0)$ and its permutations but, in general, it is negative for $q>1$, positive for $q<1$ and 0 only for $q=1$, so the axiom (${\overline{A}}_2$) is broken (see Figure 5). In Figure 6 we illustrated the Yamano channel capacity as a function of the parameter q, in the case of two input channels with $P_X=[a,1-a]$, the channel capacity is zero for $q>1$ (which is obtained for $P_X=[1,0]$), and

$$C_q^6\left(BSC\right)=\frac{1}{q-1}\left(2^q-1-2^{2q-2}\right),$$

(98)

for $q>1$ (which is obtained for $P_X=[1/2,1/2]$). In the same Figure, we plotted the graph for the $\alpha$-q channel capacities proposed in this paper, and, as before, all of them remain zero in the case of a totally destructive BSC, as expected.

Further attempts were made in [33], where the mutual information is defined in an analogous manner to (66) and (66), with the generalized divergence measure introduced in [54]. Thus, the alternative measure for mutual information is defined in

$$\begin{array}{c}\hfill J_q^7(X,Y)=\frac{1}{(1-q)ln2}\frac{1}{{\sum }_{x,y}{P}_{X,Y}^q(x,y)}\left[1-\sum _{x,y}{P}_{X,Y}(x,y){\left(\frac{P_X\left(x\right)P_Y\left(y\right)}{P_{X,Y}(x,y)}\right)}^{1-q}\right].\end{array}$$

(99)

However, in the case of the simplest perfect communication channel for which $X=Y$, the mutual information reduces to

$$J_q^7(X,Y)=\frac{1}{(1-q)ln2}\frac{1-{\sum }_x{P}_X{\left(x\right)}^{2-q}}{{\sum }_x{P}_X{\left(x\right)}^q}\ne L_q\left(X\right),$$

(100)

which breaks the axiom (${\overline{A}}_3$).

7.3. Landsber–Vedral Capacities

To avoid these problems, Landsberg and Vedral [4] proposed the mutual information measure and related channel capacities for the Sharma–Mittal entropy class $H_{\alpha ,q}$, particularly considering the choice of $q=\alpha$, which corresponds to Tsallis entropy, $q=2-\alpha$, and the case of $q=1$, which corresponds to the Rényi entropy

$$J_{\alpha ,q}^8(X,Y)=H_{\alpha ,q}\left(Y\right)-\widetilde{H_{\alpha ,q}}\left(Y\right|X),$$

(101)

where the conditional entropy ${\widetilde{H_{\alpha ,q}}}^{LV}\left(Y\right|X)$ is defined as in

$$\widetilde{H_{\alpha ,q}}\left(Y\right|X)=\sum _x{P}_X\left(x\right)H_{\alpha ,q}\left(Y\right|X=x)$$

(102)

and

$$H_{\alpha ,q}\left(Y\right|X=x)=\frac{1}{1-q}\left({\left(\sum _y{P}_{Y|X}{\left(y\right|x)}^{\alpha }\right)}^{\frac{1-q}{1-\alpha }}-1\right).$$

(103)

Although this definition bears some similarities to the $\alpha$-q mutual information proposed in formula (69), several key differences can be observed. First of all, it characterizes the information transfer as the output uncertainty reduction after the input symbols are known, instead of input uncertainty reduction, after the output symbols are known (42). In addition, it uses the ordinary—operation instead of the ${\ominus }_q$ one. In addition, note that the definition of conditional entropy (102) generally differs from the definition proposed in (70).

The definition (101) resolves the issue of the axiom (${\overline{A}}_2$) which appears in the case of the Daroczy capacity, since in the case of a totally destructive channel ($X\perp \perp Y$), $P_{Y|X}\left(y\right|x)=P_Y\left(y\right)$ and $L_q\left(Y\right|X=x)=L_q\left(Y\right)$ and $L_q\left(Y\right|X)=L_q\left(Y\right)$, so that $I_{\alpha ,q}^{lv}(X,Y)=0$. However, the problems remain with the axiom (${\overline{A}}_5$), which can be observed in the case of a noisy channel with non-overlapping outputs if the number of channel inputs is lower than the number of channel outputs $n<m$. Indeed, in the case of a noisy channel with non-overlapping outputs given by the transition matrix (27), both of the row entropies $L_q\left(Y\right|X=x)$ have the same value, which is independent of x

$$H_{\alpha ,q}\left(Y\right|X=x)=\frac{k^{1-q}-1}{(q-1)ln2}={\mathrm{Log}}_{q}k;\mathrm{for}x=x_1,x_2,$$

(104)

and the maximal value of Landsberg–Vedral mutual information (101) is obtained only by maximizing $H_{\alpha ,q}\left(Y\right)$ over $P_X$, which is achieved if X is uniformly distributed, since in this case Y is uniformly distributed, as well as ($a=\frac{1}{2}$ in (28)), so the maximal value of the output entropy is $H_{\alpha ,q}\left(Y\right)={\mathrm{Log}}_q\left(2k\right)$ and the mutual information is maximized for

$$C_{\alpha ,q}^8\left(NOC\right)={\mathrm{Log}}_q\left(2k\right)-{\mathrm{Log}}_q\left(k\right),$$

(105)

which is greater than ${\mathrm{Log}}_q\left(2\right)$ for $k\ge 2$, i.e., for $m\ge 4$ outputs, so the axiom (${\overline{A}}_5$) is broken, which is illustrated in Figure 7.

Figure 7. Landsberg–Vedral capacities for the Tsallis (solid lines) and the Landsberg–Vedral (dashed lines) entropies in the case of a (perfect) noisy channel with non-overlapping outputs with m outputs as functions of q, for different values of m. The axiom (${\overline{A}}_4$) is broken for all $m>2$ and satisfied in the case of corresponding $\alpha$-q-capacities, $C_{q,q}$ and $C_{q,2-q}$.

7.4. Chapeau-Blondeau–Delahaies–Rousseau Capacities

Following a similar approach to the one in Section 5.4, Chapeau-Blondeau, Delahaies and Rousseau considered the definition of mutual information which corresponds to the Tsallis entropy using Tsallis divergence,

$$D_{q,q}\left(P\right|\left|Q\right)=\frac{1}{q-1}\left(\sum _{x}P{\left(x\right)}^{q}Q{\left(x\right)}^{1-q}-1\right),$$

(106)

can be written in

$$\begin{array}{cc}\hfill J_q^9(X,Y)& =D_{q,q}\left(P_{X,Y}\parallel P_X{P}_Y\right)={\eta }_q\left(D_q\left(P_{X,Y}\parallel P_X{P}_Y\right)\right)\hfill \\ \hfill & =\frac{1}{1-q}\left(1-\sum _{x,y}{P}_{X,Y}{(x,y)}^q{P}_X{\left(x\right)}^{1-q}{P}_Y{\left(y\right)}^{1-q}\right).\hfill \end{array}$$

(107)

However, this definition is not directly applicable as a measure of information transfer to the Tsallis entropy with index q, since in the case of $X=Y$ it reduces to $J_q^9(X,Y)=T_{2-q}\left(X\right)$, and requires the reparametrization $q\leftrightarrow 2-q$, similar to Section 5.4, while the satisfaction of the axioms (${\overline{A}}_4$) and (${\overline{A}}_5$) is not self evident.

8. Conclusions and Future Work

A general treatment of the Sharma–Mittal entropy transfer was provided together with the analyses of existing information transfer measures for the non-additive Sharma–Mittal information transfer. It was shown that the existing definitions fail to satisfy at least one of the axioms common to the Shannon case, by which the information transfer has to be non-negative, less than the input and output uncertainty, equal to the input uncertainty in the case of perfect transmission and equal to zero in the case of a totally destructive channel. Thus, breaking some of these axioms implies unexpected and counterintuitive conclusions about the channels, such as achieving super-capacitance or sub-capacitance [4], which could be treated as nonphysical behavior. In this paper, alternative measures of the $\alpha$-q mutual information and the $\alpha$-q channel capacity were proposed so that all of the axioms which are broken in the case of the Sharma–Mittal information transfer measures considered before are satisfied, which could qualify them as physically consistent measures of information transfer.

Taking into account the previous research of non-extensive statistical mechanics [3], where the linear growth of the physical quantities has been recognized as a critical property in non-extensive [55] and non-exponentially growing systems [56], and taking into account the previous research from the field of information theory, where the Sharma–Mittal entropy has been considered an appropriate scaling measure which provides extensive information rates [21], the $\alpha$-q mutual information and the $\alpha$-q channel capacity seem to be promising measures for the characterization of information transmission in the systems where the Shannon entropy rate diverges or disappears in an infinite time limit. In addition, as was shown in this paper, the proposed information transfer measures are compatible with the maximum likelihood detection, which indicates their potential for operational characterization of coding theory and hypothesis testing problems [26].