Error Exponents and α-Mutual Information

Abstract

Over the last six decades, the representation of error exponent functions for data transmission through noisy channels at rates below capacity has seen three distinct approaches: (1) Through Gallager’s $E_0$ functions (with and without cost constraints); (2) large deviations form, in terms of conditional relative entropy and mutual information; (3) through the $\alpha$-mutual information and the Augustin–Csiszár mutual information of order $\alpha$ derived from the Rényi divergence. While a fairly complete picture has emerged in the absence of cost constraints, there have remained gaps in the interrelationships between the three approaches in the general case of cost-constrained encoding. Furthermore, no systematic approach has been proposed to solve the attendant optimization problems by exploiting the specific structure of the information functions. This paper closes those gaps and proposes a simple method to maximize Augustin–Csiszár mutual information of order $\alpha$ under cost constraints by means of the maximization of the $\alpha$-mutual information subject to an exponential average constraint.

1. Introduction

1.1. Phase 1: The MIT School

The capacity C of a stationary memoryless channel is equal to the maximal symbolwise input–output mutual information. Not long after Shannon [1] established this result, Rice [2] observed that, when operating at any encoding rate $R<C$, there exist codes whose error probability vanishes exponentially with blocklength, with a speed of decay that decreases as R approaches C. This early observation moved the center of gravity of information theory research towards the quest for the reliability function, a term coined by Shannon [3] to refer to the maximal achievable exponential decay as a function of R. The MIT information theory school, and most notably, Elias [4], Feinstein [5], Shannon [3,6], Fano [7], Gallager [8,9], and Shannon, Gallager and Berlekamp [10,11], succeeded in upper/lower bounding the reliability function by the sphere-packing error exponent function and the random coding error exponent function, respectively. Fortunately, these functions coincide for rates between C and a certain value, called the critical rate, thereby determining the reliability function in that region. The influential 1968 textbook by Gallager [9] set down the major error exponent results obtained during Phase 1 of research on this topic, including the expurgation technique to improve upon the random coding error exponent lower bound. Two aspects of those early works (and of Dobrushin’s contemporary papers [12,13] on the topic) stand out:

(a)

The error exponent functions were expressed as the result of the Karush-Kuhn-Tucker optimization of ad-hoc functions which, unlike mutual information, carried little insight. In particular, during the first phase, center stage is occupied by the parametrized function of the input distribution $P_X$ and the random transformation (or "channel”) $P_{Y|X}$,

$$\begin{array}{c}\hfill E_0(\rho ,P_X)=-log\sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}{P}_X\left(x\right)P_{Y|X}^{\frac{1}{1+\rho }}\left(y\right|x)\right)}^{1+\rho },\end{array}$$

(1)

introduced by Gallager in [8].

(b)

Despite the large-deviations nature of the setup, none of the tools from that then-nascent field (other than the Chernoff bound) found their way to the first phase of the work on error exponents; in particular, relative entropy, introduced by Kullback and Leibler [14], failed to put in an appearance.

To this date, the reliability function remains open for low rates even for the binary symmetric channel, despite a number of refined converse and achievability results (e.g., [15,16,17,18,19,20,21]) obtained since [9]. Our focus in this paper is not on converse/achievability techniques but on the role played by various information measures in the formulation of error exponent results.

1.2. Phase 2: Relative Entropy

The second phase of the error exponent research was pioneered by Haroutunian [22] and Blahut [23], who infused the expressions for the error exponent functions with meaning by incorporating relative entropy. The sphere-packing error exponent function corresponding to a random transformation $P_{Y|X}$ is given as

$$E_{\mathtt{sp}}\left(R\right)=\underset{P_X}{sup}\underset{\begin{array}{c}{Q}_{Y|X}:\mathcal{A}\to \mathcal{B}\\ I(P_X,Q_{Y|X})\le R\end{array}}{min}D(Q_{Y|X}\parallel P_{Y|X}\left|P_X\right).$$

(2)

Roughly speaking, optimal codes of rate $R<C$ incur in errors due to atypical channel behavior, and large deviations establishes that the overwhelmingly most likely such behavior can be explained as if the channel would be supplanted by the one with mutual information bounded by R which is closest to the true channel in conditional relative entropy $D(Q_{Y|X}\parallel P_{Y|X}|P_X)$. Within the confines of finite-alphabet memoryless channels, this direction opened the possibility of using the combinatorial method of types to obtain refined results robustifying the choice of the optimal code against incomplete knowledge of the channel. The 1981 textbook by Csiszár and Körner [24] summarizes the main results obtained during Phase 2.

1.3. Phase 3: Rényi Information Measures

Entropy and relative entropy were generalized by Rényi [25], who introduced the notions of Rényi entropy and Rényi divergence of order $\alpha$. He arrived at Rényi entropy by relaxing the axioms Shannon proposed in [1], and showed to be satisfied by no measure but entropy. Shortly after [25], Campbell [26] realized the operational role of Rényi entropy in variable-length data compression if the usual average encoding length criterion $\mathbb{E}\left[\ell \right(c\left(X\right)\left)\right]$ is replaced by an exponential average ${\alpha }^{-1}log\mathbb{E}[exp\left(\alpha \ell \left(c\left(X\right)\right)\right]$. Arimoto [27] put forward a generalized conditional entropy inspired by Rényi’s measures (now known as Arimoto-Rényi conditional entropy) and proposed a generalized mutual information by taking the difference between Rényi entropy and the Arimoto-Rényi conditional entropy. The role of the Arimoto-Rényi conditional entropy in the analysis of the error probability of Bayesian M-ary hypothesis testing problems has been recently shown in [28], tightening and generalizing a number of results dating back to Fano’s inequality [29].

Phase 3 of the error exponent research was pioneered by Csiszár [30] where he established a connection between Gallager’s $E_0$ function and Rényi divergence by means of a Bayesian measure of the discrepancy among a finite collection of distributions introduced by Sibson [31]. Although [31] failed to realize its connection to mutual information, Csiszár [30,32] noticed that it could be viewed as a natural generalization of mutual information. Arimoto [27] also observed that the unconstrained maximization of his generalized mutual information measure with respect to the input distribution coincides with a scaled version of the maximal $E_0$ function. This resulted in an extension of the Arimoto-Blahut algorithm useful for the computation of error exponent functions [33] (see also [34]) for finite-alphabet memoryless channels.

Within Haroutunian’s framework [22] applied in the context of the method of types, Poltyrev [35] proposed an alternative to Gallager’s $E_0$ function, defined by means of a cumbersome maximization over a reverse random transformation. This measure turned out to coincide (modulo different parametrizations) with another generalized mutual information introduced four years earlier by Augustin in his unpublished thesis [36], by means of a minimization with respect to an output probability measure.

The key contribution in the development of this third phase is Csiszár’s paper [32] where he makes a compelling case for the adoption of Rényi’s information measures in the large deviations analysis of lossless data compression, hypothesis testing and data transmission. Recall that more than two decades earlier, Csiszár [30] had already established the connection of Gallager’s $E_0$ function and the generalized mutual information inspired by Sibson [31], which, henceforth, we refer to as the $\alpha$-mutual information. Therefore, its relevance to the error exponent analysis of error correcting codes had already been established. Incidentally, more recently, another operational role was found for $\alpha$-mutual information in the context of the large deviations analysis of composite hypothesis testing [37]. In addition to $\alpha$-mutual information, and always working with discrete alphabets, Csiszár [32] considers the generalized mutual informations due to Arimoto [27], and to Augustin [36], which we refer to as the Augustin–Csiszár mutual information of order $\alpha$. Csiszár shows that all those three generalizations of mutual information coincide upon their unconstrained maximization with respect to the input distribution. Further relationships among those Rényi-based generalized mutual informations have been obtained in recent years in [38,39,40,41,42,43,44,45]. In [32] the maximal $\alpha$-mutual information or generalized capacity of order $\alpha$ finds an operational characterization as a generalized cutoff rate–an equivalent way to express the reliability function. This would have been the final word on the topic if it weren’t for its limitation to discrete-alphabet channels, and more importantly, encoding without cost constraints.

1.4. Cost Constraints

If the transmitted codebook is cost-constrained, i.e., every codeword $(c_1,\dots ,c_n)$ is forced to satisfy ${\sum }_{i=1}^n\mathsf{b}\left(c_i\right)\le n\theta$ for some nonnegative cost function $\mathsf{b}(·)$, then the channel capacity is equal to the input–output mutual information maximized over input probability measures restricted to satisfy $\mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta$. Gallager [9] incorporated cost constraints in his treatment of error exponents by generalizing (1) to the function

$$\begin{array}{c}\hfill E_0(\rho ,P_X,r,\theta )=-log\sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}{P}_X\left(x\right)exp\left(r\mathsf{b}\left(x\right)-r\theta \right)P_{Y|X}^{\frac{1}{1+\rho }}\left(y\right|x)\right)}^{1+\rho },\end{array}$$

(3)

with which he was able to prove an achievability result invoking Shannon’s random coding technique [1]. Gallager also suggested in the footnote of page 329 of [9] that the converse technique of [10] is amenable to extension to prove a sphere-packing converse based on (3). However, an important limitation is that that technique only applies to constant-composition codes (all codewords have the same empirical distribution). A more powerful converse circumventing that limitation (at least for symmetric channels) was given by [46] also expressing the upper bound on the reliability function by optimizing (3) with respect to $\rho$, r and $P_X$. A notable success of the approach based on the optimization of (3) was the determination of the reliability function (for all rates below capacity) of the direct detection photon channel [47].

In contrast, the Phase Two expression (2) for the sphere-packing error exponent for cost-constrained channels is much more natural and similar to the way the expression for channel capacity is impacted by cost constraints, namely we simply constrain the maximization in (2) to satisfy $\mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta$. Unfortunately, no general methods to solve the ensuing optimization have been reported.

Once cost constraints are incorporated, the equivalence among the maximal $\alpha$-mutual information, maximal order-$\alpha$ Augustin–Csiszár mutual information, and maximal Arimoto mutual information of order $\alpha$ breaks down. Of those three alternatives, it is the maximal Augustin–Csiszár mutual information under cost constraints that appears in the error exponent functions. The challenge is that Augustin–Csiszár mutual information is much harder to evaluate, let alone maximize, than $\alpha$-mutual information. The Phase 3 effort to encompass cost constraints started by Augustin [36] and was continued recently by Nakiboglu [43]. Their focus was to find a way to express (3) in terms of Rényi information measures. Although, as we explain in Item 62, they did not quite succeed, their efforts were instrumental in developing key properties of the Augustin–Csiszár mutual information.

1.5. Organization

To enhance readability and ease of reference, the rest of this work is organized in 81 items, grouped into Section 13 and an appendix.

Basic notions and notation (including the key concept of $\alpha$-response) are collected in Section 2. Unlike much of the literature on the topic, we do not restrict attention to discrete input/output alphabets, nor do we impose any topological structures on them.

The paper is essentially self-contained. Section 3 covers the required background material on relative entropy, Rényi divergence of order $\alpha$, and their conditional versions, including a key representation of Rényi divergence in terms of relative entropies and a tilted probability measure, and additive decompositions of Rényi divergence involving the $\alpha$-response.

Section 4 studies the basic properties of $\alpha$-mutual information and order-$\alpha$ Augustin–Csiszár mutual information. This includes their variational representations in terms of conventional (non-Rényi) information measures such as conditional relative entropy and mutual information, which are particularly simple to show in the main range of interest in applications to error exponents, namely, $\alpha \in (0,1)$.

The interrelationships between $\alpha$-mutual information and order-$\alpha$ Augustin–Csiszár mutual information are covered in Section 5, which introduces the dual notions of $\alpha$-adjunct and $\langle \alpha \rangle$-adjunct of an input probability measure.

The maximizations with respect to the input distribution of $\alpha$-mutual information and order-$\alpha$ Augustin–Csiszár mutual information account for their role in the fundamental limits in data transmission through noisy channels. Section 6 gives a brief review of the results in [45] for the maximization of $\alpha$-mutual information. For Augustin–Csiszár mutual information, Section 7 covers its unconstrained maximization, which coincides with its $\alpha$-mutual information counterpart. Section 8 proposes an approach to find ${\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)$, the maximal Augustin–Csiszár mutual information of order $\alpha \in (0,1)$ subject to $\mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta$. Instead of trying to identify directly the input distribution that maximizes Augustin–Csiszár mutual information, the method seeks its $\langle \alpha \rangle$-adjunct. This is tantamount to maximizing $\alpha$-mutual information over a larger set of distributions.

Section 9 shows

$$\begin{array}{c}\hfill \rho {\mathbb{C}}_{\frac{1}{1+\rho }}^{\mathtt{c}}\left(\theta \right)=\underset{r\ge 0}{min}\underset{P_X}{max}{E}_0(\rho ,P_X,r,\theta ),\end{array}$$

(4)

where the maximization on the right side is unconstrained. In other words, the minimax of Gallager’s $E_0$ function (3) with cost constraints is shown to be equal to the maximal Augustin–Csiszár mutual information, thereby bridging the existing gap between the Phase 1 and Phase 3 representations alluded to earlier in this introduction.

As in [48], Section 10 defines the sphere-packing and random-coding error exponent functions in the natural canonical form of Phase 2 (e.g., (2)), and gives a very simple proof of the nexus between the Phase 2 and Phase 3 representations, namely,

$$\begin{array}{c}\hfill E_{\mathtt{sp}}\left(R\right)=\underset{\rho \ge 0}{sup}\left\{\rho {\mathbb{C}}_{\frac{1}{1+\rho }}^{\mathtt{c}}\left(\theta \right)-\rho R\right\},\end{array}$$

(5)

with or without cost constraints. In this regard, we note that, although all the ingredients required were already present at the time the revised version of [24] was published three decades after the original, [48] does not cover the role of Rényi’s information measures in channel error exponents.

Examples illustrating the proposed method are given in Section 11 and Section 12 for the additive Gaussian noise channel under a quadratic cost function, and the additive exponential noise channel under a linear cost function, respectively. Simple parametric expressions are given for the error exponent functions, and the least favorable channels that account for the most likely error mechanism (Section 1.2) are identified in both cases.

2. Relative Information and Information Density

We begin with basic terminology and notation required for the subsequent development.

1.

If $(\mathcal{A},\mathcal{F},P)$ is a probability space, $X\sim P$ indicates $\mathbb{P}[X\in \mathcal{F}]=P\left(\mathcal{F}\right)$ for all $\mathcal{F}\in \mathcal{F}$.

2.

If probability measures P and Q defined on the same measurable space $(\mathcal{A},\mathcal{F})$ satisfy $P\left(A\right)=0$ for all $A\in \mathcal{F}$ such that $Q\left(A\right)=0$, we say that P is dominated by Q, denoted as $P\ll Q$. If P and Q dominate each other, we write $P\ll \gg Q$. If there is an event such that $P\left(A\right)=0$ and $Q\left(A\right)=1$, we say that P and Q are mutually singular, and we write $P\perp Q$.

3.

If $P\ll Q$, then $\frac{\mathrm{d}P}{\mathrm{d}Q}$ is the Radon-Nikodym derivative of the dominated measure P with respect to the reference measure Q. Its logarithm is known as the relative information, namely, the random variable

$$\begin{array}{c}\hfill {\mathit{ı}}_{P\parallel Q}\left(a\right)=log\frac{\mathrm{d}P}{\mathrm{d}Q}\left(a\right)\in [-\infty ,+\infty ),a\in \mathcal{A}.\end{array}$$

(6)

As with the Radon-Nikodym derivative, any identity involving relative informations can be changed on a set of measure zero under the reference measure without incurring in any contradiction. If $P\ll Q\ll R$, then the chain rule of Radon-Nikodym derivatives yields

$$\begin{array}{c}\hfill {\mathit{ı}}_{P\parallel Q}\left(a\right)+{\mathit{ı}}_{Q\parallel R}\left(a\right)={\mathit{ı}}_{P\parallel R}\left(a\right),a\in \mathcal{A}.\end{array}$$

(7)

Throughout the paper, the base of exp and log is the same and chosen by the reader unless explicitly indicated otherwise. We frequently define a probability measure P from the specification of ${\mathit{ı}}_{P\parallel Q}$ and $Q\gg P$ since

$$\begin{array}{c}\hfill P\left(A\right)={\int }_{A}exp\left({\mathit{ı}}_{P\parallel Q}\left(a\right)\right)\mathrm{d}Q\left(a\right),A\in \mathcal{F}.\end{array}$$

(8)

If $X\sim P$ and $Y\sim Q$, it is often convenient to write ${\mathit{ı}}_{X\parallel Y}\left(x\right)$ instead of ${\mathit{ı}}_{P\parallel Q}\left(x\right)$. Note that

$$\begin{array}{c}\hfill \mathbb{E}\left[exp\left({\mathit{ı}}_{X\parallel Y}\left(Y\right)\right)\right]=1.\end{array}$$

(9)

Example 1.

If $X\sim \mathcal{N}\left({\mu }_X,{\sigma }_X^2\right)$ (Gaussian with mean ${\mu }_X$ and variance ${\sigma }_X^2$) and $Y\sim \mathcal{N}\left({\mu }_Y,{\sigma }_Y^2\right)$, then,

$$\begin{array}{c}\hfill {\mathit{ı}}_{X\parallel Y}\left(a\right)=\frac{1}{2}log\frac{{\sigma }_Y^2}{{\sigma }_X^2}+\frac{1}{2}\left(\frac{{(a-{\mu }_Y)}^2}{{\sigma }_Y^2}-\frac{{(a-{\mu }_X)}^2}{{\sigma }_X^2}\right)log\mathrm{e}.\end{array}$$

(10)

4.

Let $(\mathcal{A},\mathcal{F})$ and $(\mathcal{B},\mathcal{G})$ be measurable spaces, known as the input and output spaces, respectively. Likewise, $\mathcal{A}$ and $\mathcal{B}$ are referred to as the input and output alphabets respectively. The simplified notation $P_{Y|X}:\mathcal{A}\to \mathcal{B}$ denotes a random transformation from $(\mathcal{A},\mathcal{F})$ to $(\mathcal{B},\mathcal{G})$, i.e., for any $x\in \mathcal{A}$, $P_{Y|X=x}(·)$ is a probability measure on $(\mathcal{B},\mathcal{G})$, and for any $B\in \mathcal{G}$, $P_{Y|X=·}\left(B\right)$ is an $\mathcal{F}$-measurable function.

5.

We abbreviate by ${\mathcal{P}}_{\mathcal{A}}$ the set of probability measures on $(\mathcal{A},\mathcal{F})$, and by ${\mathcal{P}}_{\mathcal{A}\times \mathcal{B}}$ the set of probability measures on $(\mathcal{A}\times \mathcal{B},\mathcal{F}\otimes \mathcal{G})$. If $P\in {\mathcal{P}}_{\mathcal{A}}$ and $P_{Y|X}:\mathcal{A}\to \mathcal{B}$ is a random transformation, the corresponding joint probability measure is denoted by $P{P}_{Y|X}\in {\mathcal{P}}_{\mathcal{A}\times \mathcal{B}}$ (or, interchangeably, $P_{Y|X}P$). The notation $P\to P_{Y|X}\to Q$ simply indicates that the output marginal of the joint probability measure $P{P}_{Y|X}$ is denoted by $Q\in {\mathcal{P}}_{\mathcal{B}}$, namely,

$$\begin{array}{c}\hfill Q\left(B\right)=\int P_{Y|X}\left(B\right|x)\mathrm{d}{P}_X\left(x\right)=\mathbb{E}\left[P_{Y|X}\left(B\right|X)\right],B\in \mathcal{G}.\end{array}$$

(11)

6.

If $P_X\to P_{Y|X}\to P_Y$ and $P_{Y|X=a}\ll P_Y$, the information density ${\mathit{ı}}_{X;Y}:\mathcal{A}\times \mathcal{B}\to [-\infty ,\infty )$ is defined as

$$\begin{array}{c}\hfill {\mathit{ı}}_{X;Y}(a;b)={\mathit{ı}}_{P_{Y|X=a}\parallel P_Y}\left(b\right),(a,b)\in \mathcal{A}\times \mathcal{B}.\end{array}$$

(12)

Following Rényi’s terminology [49], if $P_X{P}_{Y|X}\ll P_X\times P_Y$, the dependence between X and Y is said to be regular, and the information density can be defined on $(x,y)\in \mathcal{A}\times \mathcal{B}$. Henceforth, we assume that $P_{Y|X}$ is such that the dependence between its input and output is regular regardless of the input probability measure. For example, if $X=Y\in \mathbb{R}$, then $P_{Y|X=a}\left(A\right)=1\{a\in A\}$, and their dependence is not regular, since for any $P_X$ with non-discrete components P_XY P_X × P_Y.

7.

Let $\alpha >0$, and $P_X\to P_{Y|X}\to P_Y$. The $\alpha$-response to $P_X\in {\mathcal{P}}_{\mathcal{A}}$ is the output probability measure $P_{Y\left[\alpha \right]}\ll P_Y$ with relative information given by

$$\begin{array}{c}\hfill {\mathit{ı}}_{Y\left[\alpha \right]\parallel Y}\left(y\right)=\frac{1}{\alpha }log\mathbb{E}[exp(\alpha {\mathit{ı}}_{X;Y}(X;y)-{\kappa }_{\alpha })],X\sim P_X,\end{array}$$

(13)

where ${\kappa }_{\alpha }$ is a scalar that guarantees that $P_{Y\left[\alpha \right]}$ is a probability measure. Invoking (9), we obtain

$$\begin{array}{cc}\hfill {\kappa }_{\alpha }& =\alpha log\mathbb{E}\left[{\mathbb{E}}^{\frac{1}{\alpha }}[exp\left(\alpha {\mathit{ı}}_{X;Y}(X;\overline{Y})\right)|\overline{Y}]\right],(X,\overline{Y})\sim P_X\times P_Y.\hfill \end{array}$$

(14)

For brevity, the dependence of ${\kappa }_{\alpha }$ on $P_X$ and $P_{Y|X}$ is omitted. Jensen’s inequality applied to ${(·)}^{\alpha }$ results in ${\kappa }_{\alpha }\le 0$ for $\alpha \in (0,1)$ and ${\kappa }_{\alpha }\ge 0$ for $\alpha >1$. Although the $\alpha$-response has a long record of services to information theory, this terminology and notation were introduced recently in [45]. Alternative terminology and notation were proposed in [42], which refers to the $\alpha$-response as the order $\alpha$ Rényi mean. Note that ${\kappa }_1=0$ and the 1-response to $P_X$ is $P_Y$. If $p_{Y\left[\alpha \right]}$ and $p_{Y|X}$ denote the densities of $P_{Y\left[\alpha \right]}$ and $P_{Y|X}$ with respect to some common dominating measure, then (13) becomes

$$\begin{array}{c}\hfill p_{Y\left[\alpha \right]}\left(y\right)=exp\left(-\frac{{\kappa }_{\alpha }}{\alpha }\right){\mathbb{E}}^{\frac{1}{\alpha }}\left[p_{Y|X}^{\alpha }\left(y\right|X)\right],X\sim P_X.\end{array}$$

(15)

For $\alpha >1$ (resp. $\alpha <1$) we can think of the normalized version of $p_{Y|X}^{\alpha }$ as a random transformation with less (resp. more) "noise" than $p_{Y|X}$.

8.

We will have opportunity to apply the following examples.

Example 2.

If $Y=X+N$, where $X\sim \mathcal{N}\left({\mu }_X,{\sigma }_X^2\right)$ independent of $N\sim \mathcal{N}\left({\mu }_N,{\sigma }_N^2\right)$, then the α-response to $P_X$ is

$$\begin{array}{c}\hfill Y\left[\alpha \right]\sim \mathcal{N}\left({\mu }_X+{\mu }_N,\alpha {\sigma }_X^2+{\sigma }_N^2\right).\end{array}$$

(16)

Example 3.

Suppose that $Y=X+N$, where N is exponential with mean ζ, independent of X, which is a mixed random variable with density

$$\begin{array}{c}\hfill f_X\left(t\right)=\frac{\zeta }{\alpha \mu }\delta \left(t\right)+\left(1-\frac{\zeta }{\alpha \mu }\right)\frac{1}{\mu }{\mathrm{e}}^{-t/\mu }1\{t>0\},\end{array}$$

(17)

with $\alpha \mu \ge \zeta$. Then, $Y\left[\alpha \right]$, the α-response to $P_X$, is exponential with mean $\alpha \mu$.

3. Relative Entropy and Rényi Divergence

Given a pair of probability measures $(P,Q)\in {\mathcal{P}}_{\mathcal{A}}^2$, relative entropy and Rényi divergence gauge the distinctness between P and Q.

9.

Provided $P\ll Q$, the relative entropy is the expectation of the relative information with respect to the dominated measure

$$\begin{array}{cc}\hfill D(P\parallel Q)& =\mathbb{E}\left[{\mathit{ı}}_{P\parallel Q}\left(X\right)\right],X\sim P\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & =\mathbb{E}\left[exp\left({\mathit{ı}}_{P\parallel Q}\left(Y\right)\right){\mathit{ı}}_{P\parallel Q}\left(Y\right)\right],Y\sim Q\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & \ge 0,\hfill \end{array}$$

(20)

with equality if and only if $P=Q$. If P Q, then $D(P\parallel Q)=\infty$. As in Item 3, if $X\sim P$ and $Y\sim Q$, we may write $D(X\parallel Y)$ instead of $D(P\parallel Q)$, in the same spirit that the expectation and entropy of P are written as $\mathbb{E}\left[X\right]$ and $H\left(X\right)$, respectively.

10.

Arising in the sequel, a common optimization in information theory finds, among the probability measures satisfying an average cost constraint, that which is closest to a given reference measure Q in the sense of $D(·\parallel Q)$. For that purpose, the following result proves sufficient. Incidentally, we often refer to unconstrained maximizations over probability distributions. It should be understood that those optimizations are still constrained to the sets ${\mathcal{P}}_{\mathcal{A}}$ or ${\mathcal{P}}_{\mathcal{B}}$. As customary in information theory, we will abbreviate ${max}_{P_X\in {\mathcal{P}}_{\mathcal{A}}}$ by ${max}_X$ or ${max}_{P_X}$.

Theorem 1.

Let $P_Z\in {\mathcal{P}}_{\mathcal{A}}$ and suppose that $g:\mathcal{A}\to [0,\infty )$ is a Borel measurable mapping. Then,

$$\begin{array}{c}\hfill \underset{X}{min}\left\{D(X\parallel Z)+\mathbb{E}\left[g\right(X\left)\right]\right\}=-log\mathbb{E}[exp(-g\left(Z\right))],\end{array}$$

(21)

achieved uniquely by $P_X^{*}\ll \gg P_Z$ defined by

$$\begin{array}{c}\hfill {\mathit{ı}}_{X^{*}\parallel Z}\left(a\right)=-g\left(a\right)-log\mathbb{E}[exp(-g\left(Z\right)\left)\right],a\in \mathcal{A}.\end{array}$$

(22)

Proof. 

Note that since g is nonnegative, $\eta =\mathbb{E}[exp(-g\left(Z\right)\left)\right]\in (0,1]$. Furthermore,

$$\begin{array}{c}\hfill \mathbb{E}\left[g\left(X^{*}\right)\right]=\frac{\int g\left(t\right)exp(-g\left(t\right))\mathrm{d}{P}_Z\left(t\right)}{\mathbb{E}[exp(-g\left(Z\right)\left)\right]}\in \left[0,\frac{1}{\mathrm{e}\eta }\right].\end{array}$$

(23)

Therefore, the subset of ${\mathcal{P}}_{\mathcal{A}}$ for which the term in $\{·\}$ in (21) is finite is nonempty: Fix any $P_X$ from that subset, (which therefore satisfies $P_X\ll P_Z\ll P_X^{*}$) and invoke the chain rule (7) to write

$$\begin{array}{cc}\hfill D(X\parallel Z)+\mathbb{E}\left[g\right(X\left)\right]& =\mathbb{E}\left[{\mathit{ı}}_{X\parallel X^{*}}\left(X\right)+{\mathit{ı}}_{X^{*}\parallel Z}\left(X\right)+g\left(X\right)\right]\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & =D(X\parallel X^{*})-log\mathbb{E}[exp(-g\left(Z\right)\left)\right],X\sim P_X,\hfill \end{array}$$

(25)

which is uniquely minimized by letting $P_X=P_X^{*}$. Note that for typographical convenience we have denoted $X^{*}\sim P_X^{*}$. □

11.

Let p and q denote the Radon-Nikodym derivatives of probability measures P and Q, respectively, with respect to a common dominating $\sigma$-finite measure $\mu$. The Rényi divergence of order $\alpha \in (0,1)\cup (1,\infty )$ between P and Q is defined as [25,50]

$$\begin{array}{cc}\hfill D_{\alpha }(P\parallel Q)& =\frac{1}{\alpha -1}log{\int }_{\mathcal{A}}{p}^{\alpha }{q}^{1-\alpha }\mathrm{d}\mu \hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\mathbb{E}\left[exp\left(\alpha {\mathit{ı}}_{P\parallel R}\left(Z\right)+(1-\alpha ){\mathit{ı}}_{Q\parallel R}\left(Z\right)\right)\right],Z\sim R\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\mathbb{E}\left[exp\left(\alpha {\mathit{ı}}_{P\parallel Q}\left(Y\right)\right)\right],Y\sim Q\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\mathbb{E}\left[exp\left((\alpha -1){\mathit{ı}}_{P\parallel Q}\left(X\right)\right)\right],X\sim P,\hfill \end{array}$$

(29)

where (28) and (29) hold if $P\ll Q$, and in (27), R is a probability measure that dominates both P and Q. Note that (28) and (29) state that $(t-1)D_t(X\parallel Y)$ and $t{D}_{1+t}(X\parallel Y)$ are the cumulant generating functions of the random variables ${\mathit{ı}}_{X\parallel Y}\left(Y\right)$ and ${\mathit{ı}}_{X\parallel Y}\left(X\right)$, respectively. The relative entropy is the limit of $D_{\alpha }(P\parallel Q)$ as $\alpha \uparrow 1$, so it is customary to let $D_1(P\parallel Q)=D(P\parallel Q)$. For any $\alpha >0$, $D_{\alpha }(P\parallel Q)\ge 0$ with equality if and only if $P=Q$. Furthermore, $D_{\alpha }(P\parallel Q)$ is non-decreasing in $\alpha$, satisfies the skew-symmetric property

$$\begin{array}{cc}\hfill (1-\alpha )D_{\alpha }(P\parallel Q)& =\alpha D_{1-\alpha }(Q\parallel P),\alpha \in [0,1],\hfill \end{array}$$

(30)

and

$$\begin{array}{c}\hfill \underset{\alpha \in (0,1)}{inf}{D}_{\alpha }(P\parallel Q)=\infty ⟺P\perp Q⟹\underset{\alpha >1}{inf}{D}_{\alpha }(P\parallel Q)=\infty .\end{array}$$

(31)

12.

The expressions in the following pair of examples will come in handy in Section 11 and Section 12.

Example 4.

Suppose that ${\sigma }_{\alpha }^2=\alpha {\sigma }_1^2+(1-\alpha ){\sigma }_0^2>0$ and $\alpha \in (0,1)\cup (1,\infty )$. Then,

$$\begin{array}{c}\hfill D_{\alpha }\left(\mathcal{N}\left({\mu }_0,{\sigma }_0^2\right)\parallel \mathcal{N}\left({\mu }_1,{\sigma }_1^2\right)\right)=\frac{1}{2}log\frac{{\sigma }_1^2}{{\sigma }_0^2}+\frac{1}{2(\alpha -1)}log\frac{{\sigma }_1^2}{{\sigma }_{\alpha }^2}+\frac{\alpha {({\mu }_1-{\mu }_0)}^2}{2{\sigma }_{\alpha }^2}log\mathrm{e},\end{array}$$

(32)

$$\begin{array}{cc}\hfill D\left(\mathcal{N}\left({\mu }_0,{\sigma }_0^2\right)\parallel \mathcal{N}\left({\mu }_1,{\sigma }_1^2\right)\right)& =\frac{1}{2}log\frac{{\sigma }_1^2}{{\sigma }_0^2}+\frac{1}{2}\left(\frac{{\sigma }_0^2}{{\sigma }_1^2}-1\right)log\mathrm{e}+\frac{{({\mu }_1-{\mu }_0)}^2}{2{\sigma }_1^2}log\mathrm{e}\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill & =\underset{\alpha \to 1}{lim}{D}_{\alpha }\left(\mathcal{N}\left({\mu }_0,{\sigma }_0^2\right)\parallel \mathcal{N}\left({\mu }_1,{\sigma }_1^2\right)\right).\hfill \end{array}$$

(34)

Example 5.

Suppose Z is exponentially distributed with unit mean, i.e., its probability density function is ${\mathrm{e}}^{-t}1\{t\ge 0\}$. For $d_0\ge d_1$ and α such that $(1-\alpha ){\mu }_0+\alpha {\mu }_1>0$ we obtain

$$\begin{array}{cc}\hfill D_{\alpha }\left({\mu }_{0}Z+d_0\parallel {\mu }_{1}Z+d_1\right)& =\frac{d_0-d_1}{{\mu }_1}log\mathrm{e}+log\frac{{\mu }_1}{{\mu }_0}+\frac{1}{1-\alpha }log\left(\alpha +(1-\alpha )\frac{{\mu }_0}{{\mu }_1}\right),\hfill \end{array}$$

$$\begin{array}{cc}\hfill D\left({\mu }_{0}Z+d_0\parallel {\mu }_{1}Z+d_1\right)& =\left(\frac{{\mu }_0}{{\mu }_1}-1+\frac{d_0-d_1}{{\mu }_1}\right)log\mathrm{e}+log\frac{{\mu }_1}{{\mu }_0}\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill & =\underset{\alpha \to 1}{lim}{D}_{\alpha }\left({\mu }_{0}Z+d_0\parallel {\mu }_{1}Z+d_1\right).\hfill \end{array}$$

(36)

13.

Intimately connected with the notion of Rényi divergence is the tilted probability measure $P_{\alpha }$ defined, if $D_{\alpha }(P_{\mathtt{1}}\parallel P_{\mathtt{0}})<\infty$, by

$$\begin{array}{c}\hfill {\mathit{ı}}_{P_{\alpha }\parallel Q}\left(a\right)=\alpha {\mathit{ı}}_{P_{\mathtt{1}}\parallel Q}\left(a\right)+(1-\alpha ){\mathit{ı}}_{P_{\mathtt{0}}\parallel Q}\left(a\right)+(1-\alpha )D_{\alpha }(P_{\mathtt{1}}\parallel P_{\mathtt{0}}),\end{array}$$

(37)

where Q is any probability measure that dominates both $P_{\mathtt{0}}$ and $P_{\mathtt{1}}$. Although (37) is defined in general, our main emphasis is on the range $\alpha \in (0,1)$, in which, as long as P₀ P₁, the tilted probability measure is defined and satisfies $P_{\alpha }\ll P_{\mathtt{0}}$ and $P_{\alpha }\ll P_{\mathtt{1}}$, with corresponding relative informations

$$\begin{array}{cc}\hfill {\mathit{ı}}_{P_{\alpha }\parallel P_{\mathtt{0}}}\left(a\right)& ={\mathit{ı}}_{P_{\alpha }\parallel Q}\left(a\right)-{\mathit{ı}}_{P_{\mathtt{0}}\parallel Q}\left(a\right)\hfill \end{array}$$

(38)

$$\begin{array}{c}\hfill =(1-\alpha )D_{\alpha }(P_{\mathtt{1}}\parallel P_{\mathtt{0}})+\alpha \left({\mathit{ı}}_{P_{\mathtt{1}}\parallel Q}\left(a\right)-{\mathit{ı}}_{P_{\mathtt{0}}\parallel Q}\left(a\right)\right),\end{array}$$

(39)

$$\begin{array}{cc}\hfill {\mathit{ı}}_{P_{\alpha }\parallel P_{\mathtt{1}}}\left(a\right)& ={\mathit{ı}}_{P_{\alpha }\parallel Q}\left(a\right)-{\mathit{ı}}_{P_{\mathtt{1}}\parallel Q}\left(a\right)\hfill \end{array}$$

(40)

$$\begin{array}{c}\hfill =(1-\alpha )D_{\alpha }(P_{\mathtt{1}}\parallel P_{\mathtt{0}})-(1-\alpha )\left({\mathit{ı}}_{P_{\mathtt{1}}\parallel Q}\left(a\right)-{\mathit{ı}}_{P_{\mathtt{0}}\parallel Q}\left(a\right)\right),\end{array}$$

(41)

where we have used the chain rule for $P_{\alpha }\ll P_{\mathtt{0}}\ll Q$ and $P_{\alpha }\ll P_{\mathtt{1}}\ll Q$. Taking a linear combination of (38)–(41) we conclude that, for all $a\in \mathcal{A}$,

$$\begin{array}{cc}\hfill (1-\alpha )D_{\alpha }(P_{\mathtt{1}}\parallel P_{\mathtt{0}})& =(1-\alpha ){\mathit{ı}}_{P_{\alpha }\parallel P_{\mathtt{0}}}\left(a\right)+\alpha {\mathit{ı}}_{P_{\alpha }\parallel P_{\mathtt{1}}}\left(a\right).\hfill \end{array}$$

(42)

Henceforth, we focus particular attention on the case $\alpha \in (0,1)$ since that is the region of interest in the application of Rényi information measures to the evaluation of error exponents in channel coding for codes whose rate is below capacity. In addition, often proofs simplify considerably for $\alpha \in (0,1)$.

14.

Much of the interplay between relative entropy and Rényi divergence hinges on the following identity, which appears, without proof, in (3) of [51].

Theorem 2.

Let $\alpha \in (0,1)$ and assume that P₀ P₁ are defined on the same measurable space. Then, for any $P\ll P_{\mathtt{1}}$ and $P\ll P_{\mathtt{0}}$,

$$\begin{array}{c}\hfill \alpha D(P\parallel P_{\mathtt{1}})+(1-\alpha )D(P\parallel P_{\mathtt{0}})=D(P\parallel P_{\alpha })+(1-\alpha )D_{\alpha }(P_{\mathtt{1}}\parallel P_{\mathtt{0}}),\end{array}$$

(43)

where $P_{\alpha }$ is the tilted probability measure in (37) and (43) holds regardless of whether the relative entropies are finite. In particular,

$$\begin{array}{c}\hfill D(P\parallel P_{\alpha })<\infty ⟺max\{D(P\parallel P_{\mathtt{0}}),D(P\parallel P_{\mathtt{1}})\}<\infty .\end{array}$$

(44)

Proof. 

We distinguish three overlapping cases:

(1)

$D(P\parallel P_{\alpha })<\infty$: Taking expectation of (42) with respect to $a\leftarrow X\sim P$, yields (43) because

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\mathit{ı}}_{P_{\alpha }\parallel P_{\mathtt{0}}}\left(X\right)\right]& =D(P\parallel P_{\mathtt{0}})-D(P\parallel P_{\alpha }),\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\mathit{ı}}_{P_{\alpha }\parallel P_{\mathtt{1}}}\left(X\right)\right]& =D(P\parallel P_{\mathtt{1}})-D(P\parallel P_{\alpha }),\hfill \end{array}$$

(46)

where, thanks to the assumption that $D(P\parallel P_{\alpha })<\infty$, we have invoked Corollary A1 in the Appendix A twice with $(P,Q,R)\leftarrow (P,P_{\alpha },P_{\mathtt{0}})$ and $(P,Q,R)\leftarrow (P,P_{\alpha },P_{\mathtt{1}})$, respectively;

(2)

$max\{D(P\parallel P_{\mathtt{0}}),D(P\parallel P_{\mathtt{1}})\}<\infty$: The proof is identical since we are entitled to invoke Corollary A1 to show (45) (resp., (46)) because $D(P\parallel P_{\mathtt{0}})<\infty$ (resp., $D(P\parallel P_{\mathtt{1}})<\infty$).

(3)

$D(P\parallel P_{\alpha })=\infty$ and $max\{D(P\parallel P_{\mathtt{0}}),D(P\parallel P_{\mathtt{1}})\}=\infty$: both sides of (43) are equal to ∞.

Finally, to show that (44) follows from (43), simply recall from (31) that $D_{\alpha }(P_{\mathtt{1}}\parallel P_{\mathtt{0}})<\infty$. □

15.

Relative entropy and Rényi divergence are related by the following fundamental variational representation.

Theorem 3.

Fix $\alpha \in (0,1)$ and $(P_{\mathtt{1}},P_{\mathtt{0}})\in {\mathcal{P}}_{\mathcal{A}}^2$. Then, the Rényi divergence between $P_{\mathtt{1}}$ and $P_{\mathtt{0}}$ satisfies

$$\begin{array}{c}\hfill (1-\alpha )D_{\alpha }(P_{\mathtt{1}}\parallel P_{\mathtt{0}})=\underset{P}{min}\left\{\alpha D(P\parallel P_{\mathtt{1}})+(1-\alpha )D(P\parallel P_{\mathtt{0}})\right\},\end{array}$$

(47)

where the minimum is over ${\mathcal{P}}_{\mathcal{A}}$. If P₀ P₁, then the right side of (47) is attained by the tilted measure $P_{\alpha }$, and the minimization can be restricted to the subset of probability measures which are dominated by both $P_{\mathtt{1}}$ and $P_{\mathtt{0}}$.

Proof. 

If $P_{\mathtt{0}}\perp P_{\mathtt{1}}$, then both sides of (47) are $+\infty$ since there is no probability measure that is dominated by both $P_{\mathtt{0}}$ and $P_{\mathtt{1}}$. If P₀ P₁, then minimizing both sides of (43) with respect to P yields (47) and the fact that the tilted probability measure attains the minimum therein. □

The variational representation in (47) was observed in [39] in the finite-alphabet case, and, contemporaneously, in full generality in [50]. Unlike Theorem 3, both of those references also deal with $\alpha >1$. The function $d\left(\alpha \right)=(1-\alpha )D_{\alpha }(P_{\mathtt{1}}\parallel P_{\mathtt{0}})$, with $d\left(1\right)={lim}_{\alpha \uparrow 1}d\left(\alpha \right)$, is concave in $\alpha$ because the right side of (47) is a minimum of affine functions of $\alpha$.

16.

Given random transformations $P_{Y|X}:\mathcal{A}\to \mathcal{B}$, $Q_{Y|X}:\mathcal{A}\to \mathcal{B}$, and a probability measure $P_X\in {\mathcal{P}}_{\mathcal{A}}$ on the input space, the conditional relative entropy is

$$\begin{array}{cc}\hfill D(P_{Y|X}\parallel Q_{Y|X}|P_X)& =D(P_{Y|X}{P}_X\parallel Q_{Y|X}{P}_X)\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill & =\mathbb{E}\left[D\left(P_{Y|X}(·|X)\parallel Q_{Y|X}(·|X)\right)\right],X\sim P_X.\hfill \end{array}$$

(49)

Analogously, the conditional Rényi divergence is defined as

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X}\parallel Q_{Y|X}\left|P_X\right)=D_{\alpha }(P_{Y|X}{P}_X\parallel Q_{Y|X}{P}_X).\end{array}$$

(50)

A word of caution: the notation in (50) conforms to that in [38,45] but it is not universally adopted, e.g., [43] uses the left side of (50) to denote the Rényi generalization of the right side of (49). We can express the conditional Rényi divergence as

$$\begin{array}{cc}\hfill & D_{\alpha }(P_{Y|X}\parallel Q_{Y|X}\left|P_X\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\mathbb{E}\left[exp\left((\alpha -1)D_{\alpha }\left(P_{Y|X}(·|X)\parallel Q_{Y|X}(·|X)\right)\right)\right],X\sim P_X,\hfill \end{array}$$

(51)

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\mathbb{E}\left[{\left(\frac{\mathrm{d}{P}_{Y|X}}{\mathrm{d}{Q}_{Y|X}}\left(Y\right|X)\right)}^{\alpha -1}\right],(X,Y)\sim P_X{P}_{Y|X},\hfill \end{array}$$

(52)

where (52) holds if $P_X{P}_{Y|X}\ll P_X{Q}_{Y|X}$. Jensen’s inequality applied to (51) results in

$$\begin{array}{cc}\hfill D_{\alpha }(P_{Y|X}\parallel Q_{Y|X}\left|P_X\right)& \le \mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X)\parallel Q_{Y|X}(·|X))\right],\alpha \in (0,1);\hfill \end{array}$$

(53)

$$\begin{array}{cc}\hfill D_{\alpha }(P_{Y|X}\parallel Q_{Y|X}\left|P_X\right)& \ge \mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X)\parallel Q_{Y|X}(·|X))\right],\alpha >1.\hfill \end{array}$$

(54)

Nevertheless, an immediate and crucial observation we can draw from (51) is that the unconstrained maximizations of the sides of (53) and of (54) over $P_X$ do coincide: for all $\alpha >0$,

$$\begin{array}{cc}\hfill \underset{X}{sup}{D}_{\alpha }(P_{Y|X}\parallel Q_{Y|X}\left|P_X\right)& =\underset{X}{sup}\mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X)\parallel Q_{Y|X}(·|X))\right]\hfill \end{array}$$

(55)

$$\begin{array}{cc}\hfill & =\underset{a\in \mathcal{A}}{sup}{D}_{\alpha }(P_{Y|X=a}\parallel Q_{Y|X=a}).\hfill \end{array}$$

(56)

17.

Conditional Rényi divergence satisfies the following additive decomposition, originally pointed out, without proof, by Sibson [31] in the setting of finite $\mathcal{A}$.

Theorem 4.

Given $P_X\in {\mathcal{P}}_{\mathcal{A}}$, $Q_Y\in {\mathcal{P}}_{\mathcal{B}}$, $P_{Y|X}:\mathcal{A}\to \mathcal{B}$, and $\alpha \in (0,1)\cup (1,\infty )$, we have

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)=D_{\alpha }(P_{Y|X}\parallel P_{Y\left[\alpha \right]}\left|P_X\right)+D_{\alpha }(P_{Y\left[\alpha \right]}\parallel Q_Y).\end{array}$$

(57)

Furthermore, with ${\kappa }_{\alpha }$ as in (14),

$$\begin{array}{c}\hfill D_{\alpha }\left(P_{Y|X}\parallel P_{Y\left[\alpha \right]}|P_X\right)=\frac{{\kappa }_{\alpha }}{\alpha -1}.\end{array}$$

(58)

Proof. Select an arbitrary probability measure $R_Y\in {\mathcal{P}}_{\mathcal{B}}$ that dominates both $Q_Y$ and $P_Y$, and, therefore, $P_{Y\left[\alpha \right]}$ too. Letting $(X,Z)\sim P_X\times R_Y$, we have

$$\begin{array}{cc}\hfill & D_{\alpha }(P_{Y|X}\parallel Q_Y\left|P_X\right)=\frac{1}{\alpha -1}log\mathbb{E}\left[{\left(\frac{\mathrm{d}{P}_{XY}}{\mathrm{d}{P}_X\times R_Y}(X,Z)\right)}^{\alpha }{\left(\frac{\mathrm{d}{Q}_Y}{\mathrm{d}{R}_Y}\left(Z\right)\right)}^{1-\alpha }\right]\hfill \end{array}$$

(59)

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\mathbb{E}\left[\mathbb{E}\left[exp\left(\alpha {\mathit{ı}}_{X;Y}(X;Z)\right)|Z\right]{\left(\frac{\mathrm{d}{P}_Y}{\mathrm{d}{R}_Y}\left(Z\right)\right)}^{\alpha }{\left(\frac{\mathrm{d}{Q}_Y}{\mathrm{d}{R}_Y}\left(Z\right)\right)}^{1-\alpha }\right]\hfill \end{array}$$

(60)

$$\begin{array}{cc}\hfill & =\frac{{\kappa }_{\alpha }}{\alpha -1}+\frac{1}{\alpha -1}log\mathbb{E}\left[{\left(\frac{\mathrm{d}{P}_{Y\left[\alpha \right]}}{\mathrm{d}{P}_Y}\left(Z\right)\right)}^{\alpha }{\left(\frac{\mathrm{d}{P}_Y}{\mathrm{d}{R}_Y}\left(Z\right)\right)}^{\alpha }{\left(\frac{\mathrm{d}{Q}_Y}{\mathrm{d}{R}_Y}\left(Z\right)\right)}^{1-\alpha }\right]\hfill \end{array}$$

(61)

$$\begin{array}{cc}\hfill & =\frac{{\kappa }_{\alpha }}{\alpha -1}+\frac{1}{\alpha -1}log\mathbb{E}\left[{\left(\frac{\mathrm{d}{P}_{Y\left[\alpha \right]}}{\mathrm{d}{R}_Y}\left(Z\right)\right)}^{\alpha }{\left(\frac{\mathrm{d}{Q}_Y}{\mathrm{d}{R}_Y}\left(Z\right)\right)}^{1-\alpha }\right]\hfill \end{array}$$

(62)

$$\begin{array}{cc}\hfill & =\frac{{\kappa }_{\alpha }}{\alpha -1}+D_{\alpha }(P_{Y\left[\alpha \right]}\parallel Q_Y),\hfill \end{array}$$

(63)

where (61) follows from (13), and (62) follows from the chain rule of Radon-Nikodym derivatives applied to $P_{Y\left[\alpha \right]}\ll P_Y\ll R_Y$. Then, (58) follows by specializing $Q_Y=P_{Y\left[\alpha \right]}$, and the proof of (57) is complete, upon plugging (58) into the right side of (63). □

A proof of (57) in the discrete case can be found in Appendix A of [37].

18.

For all $\alpha >0$, given two inputs $(P_X,Q_X)\in {\mathcal{P}}_{\mathcal{A}}^2$ and one random transformation $P_{Y|X}:\mathcal{A}\to \mathcal{B}$, Rényi divergence (and, in particular, relative entropy) satisfies the data processing inequality,

$$\begin{array}{c}\hfill D_{\alpha }(P_X\parallel Q_X)\ge D_{\alpha }(P_Y\parallel Q_Y),\end{array}$$

(64)

where $P_X\to P_{Y|X}\to P_Y$, and $Q_X\to P_{Y|X}\to Q_Y$. The data processing inequality for Rényi divergence was observed by Csiszár [52] in the more general context of f-divergences. More recently it was stated in [39,50]. Furthermore, given one input $P_X\in {\mathcal{P}}_{\mathcal{A}}$ and two transformations $P_{Y|X}:\mathcal{A}\to \mathcal{B}$ and $Q_{Y|X}:\mathcal{A}\to \mathcal{B}$, conditioning cannot decrease Rényi divergence,

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X}\parallel Q_{Y|X}\left|P_X\right)\ge D_{\alpha }(P_Y\parallel Q_Y).\end{array}$$

(65)

Since $D_{\alpha }(P_{Y|X}\parallel Q_{Y|X}\left|P_X\right)=D_{\alpha }(P_X{P}_{Y|X}\parallel P_X{Q}_{Y|X})$, (65) follows by applying (64) to a deterministic transformation which takes an input pair and outputs the second component. Inequalities (53) and (65) imply the convexity of $D_{\alpha }(P\parallel Q)$ in $(P,Q)$ for $\alpha \in (0,1]$.

4. Dependence Measures

In this paper we are interested in three information measures that quantify the dependence between random variables X and Y, such that $P_X\to P_{Y|X}\to P_Y$, namely, mutual information, and two of its generalizations, $\alpha$- mutual information and Augustin–Csiszár mutual information of order $\alpha$.

19.

The mutual information is

$$\begin{array}{cc}\hfill I(X;Y)=I(P_X,P_{Y|X})& =D(P_{Y|X}\parallel P_Y|P_X)\hfill \end{array}$$

(66)

$$\begin{array}{cc}\hfill & =\underset{Q_Y}{min}D(P_{Y|X}\parallel Q_Y\left|P_X\right)\hfill \end{array}$$

(67)

$$\begin{array}{cc}\hfill & =\underset{Q_Y}{min}D(P_{XY}\parallel P_X\times Q_Y).\hfill \end{array}$$

(68)

20.

Given $\alpha \in (0,1)\cup (1,\infty )$, the $\alpha$-mutual information is defined as (see [30,31,32,40,42,45])

$$\begin{array}{cc}\hfill & I_{\alpha }(X;Y)=I_{\alpha }(P_X,P_{Y|X})\hfill \end{array}$$

(69)

$$\begin{array}{cc}\hfill & =\underset{Q_Y}{min}{D}_{\alpha }(P_{Y|X}\parallel Q_Y\left|P_X\right)\hfill \end{array}$$

(70)

$$\begin{array}{cc}\hfill & =\underset{Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel P_X\times Q_Y)\hfill \end{array}$$

(71)

$$\begin{array}{cc}\hfill & =D_{\alpha }\left(P_{Y|X}\parallel P_{Y\left[\alpha \right]}|P_X\right)\hfill \end{array}$$

(72)

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\mathbb{E}\left[exp\left((\alpha -1)D_{\alpha }\left(P_{Y|X}(·|X)\parallel P_{Y\left[\alpha \right]}\right)\right)\right],X\sim P_X\hfill \end{array}$$

(73)

$$\begin{array}{cc}\hfill & =D_{\alpha }\left(P_{Y|X}\parallel P_Y|P_X\right)-D_{\alpha }\left(P_{Y\left[\alpha \right]}\parallel P_Y\right)\hfill \end{array}$$

(74)

$$\begin{array}{cc}\hfill & =\frac{{\kappa }_{\alpha }}{\alpha -1}\hfill \end{array}$$

(75)

$$\begin{array}{cc}\hfill & =\frac{\alpha }{\alpha -1}log\mathbb{E}\left[{\mathbb{E}}^{\frac{1}{\alpha }}[exp\left(\alpha {\mathit{ı}}_{X;Y}(X;\overline{Y})\right)|\overline{Y}]\right],(X,\overline{Y})\sim P_X\times P_Y,\hfill \end{array}$$

(76)

where (72) and (74) follow from (57); (73) is a special case of (51); (75) follows from Theorem 4; and, (76) is (14). In view of (67) and (69), we let $I_1(X;Y)=I(X;Y)$. The notation we use for $\alpha$-mutual information conforms to that used in [40,42,45,53]. Other notations include $K_{\alpha }$ in [32,38,39] and $I_{\alpha }^g$ in [43]. $I_0(X;Y)$ and $I_{\infty }(X;Y)$ are defined by taking the corresponding limits.

21.

Theorem 4 and (72) result in the additive decomposition

$$\begin{array}{c}\hfill I_{\alpha }(X;Y)=D_{\alpha }(P_{Y|X}\parallel Q_Y\left|P_X\right)-D_{\alpha }(P_{Y\left[\alpha \right]}\parallel Q_Y),\end{array}$$

(77)

for any $Q_Y$ with $D_{\alpha }(P_{Y\left[\alpha \right]}\parallel Q_Y)<\infty$, thereby generalizing the well-known decomposition for mutual information,

$$\begin{array}{c}\hfill I(X;Y)=D(P_{Y|X}\parallel Q_Y\left|P_X\right)-D(P_Y\parallel Q_Y),\end{array}$$

(78)

which, in contrast to (77), is a simple consequence of the chain rule whenever the dependence between X and Y is regular, and of Lemma A1 in general.

22.

Example 6.

Additive independent Gaussian noise. If $Y=X+N$, where $X\sim \mathcal{N}\left(0,{\sigma }_X^2\right)$ independent of $N\sim \mathcal{N}\left(0,{\sigma }_N^2\right)$, then, for $\alpha >0$,

$$\begin{array}{cc}\hfill Y\left[\alpha \right]& \sim \mathcal{N}\left(0,\alpha {\sigma }_X^2+{\sigma }_N^2\right),\hfill \end{array}$$

(79)

$$\begin{array}{cc}\hfill I_{\alpha }(X;X+N)& =I_{\alpha }(X+N;X)=\frac{1}{2}log\left(1+\alpha \frac{{\sigma }_X^2}{{\sigma }_N^2}\right).\hfill \end{array}$$

(80)

23.

If $\alpha \in (0,1)$, (47) and (69) result in

$$\begin{array}{c}(1-\alpha )I_{\alpha }(P_X,P_{Y|X})\hfill \\ \hfill =\underset{Q_X{Q}_{Y|X}}{min}\left\{D(Q_X\parallel P_X)+\alpha D(Q_{Y|X}\parallel P_{Y|X}\left|Q_X\right)+(1-\alpha )I(Q_X,Q_{Y|X})\right\}.\end{array}$$

(81)

For $\alpha >1$ a proof of (81) is given in [39] for finite alphabets.

24.

Unlike $I(P_X,P_{Y|X})$, we can express $I_{\alpha }(P_X,P_{Y|X})$ directly in terms of its arguments without involving the corresponding output distribution or the $\alpha$-response to $P_X$. This is most evident in the case of discrete alphabets, in which (76) becomes

$$\begin{array}{cc}\hfill I_{\alpha }(X;Y)& =\frac{\alpha }{\alpha -1}log\sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}{P}_X\left(x\right)P_{Y|X=x}^{\alpha }\left(y\right)\right)}^{\frac{1}{\alpha }},\hfill \end{array}$$

(82)

$$\begin{array}{cc}\hfill I_0(X;Y)& =-log\underset{y\in \mathcal{B}}{max}\sum _{x\in \mathcal{A}}{P}_X\left(x\right)1\{P_{Y|X}\left(y\right|x)>0\},\hfill \end{array}$$

(83)

$$\begin{array}{cc}\hfill I_{\infty }(X;Y)& =log\left(\sum _{b\in \mathcal{Y}}\underset{a:P_X\left(a\right)>0}{sup}{P}_{Y|X}\left(b\right|a)\right).\hfill \end{array}$$

(84)

For example, if X is discrete and $H_{\alpha }\left(X\right)$ denotes the Rényi entropy of order $\alpha$, then for all $\alpha >0$,

$$\begin{array}{c}\hfill H_{\alpha }\left(X\right)=I_{\frac{1}{\alpha }}(X;X).\end{array}$$

(85)

If X and Y are equiprobable with $\mathbb{P}[X\ne Y]=\delta$, then, in bits, $I_{\alpha }(X;Y)=1-h_{\alpha }\left(\delta \right)$, where $h_{\alpha }\left(\delta \right)$ denotes the binary Rényi entropy.

25.

In the main region of interest, namely, $\alpha \in (0,1)$, frequently we use a different parametrization in terms of $\rho >0$, with $\alpha =\frac{1}{1+\rho }$.

Theorem 5.

For any $\rho >0$, we have the upper bound

$$\begin{array}{c}\hfill \rho I_{\frac{1}{1+\rho }}(X;Y)\le \underset{Q_{Y|X}:\mathcal{A}\to \mathcal{B}}{min}\left\{D(Q_{Y|X}\parallel P_{Y|X}\left|P_X\right)+\rho I(P_X,Q_{Y|X})\right\}.\end{array}$$

(86)

Proof. 

Fix $Q_{Y|X}:\mathcal{A}\to \mathcal{B}$, and let $P_X\to Q_{Y|X}\to Q_Y$. Then,

$$\begin{array}{cc}\hfill I_{\frac{1}{1+\rho }}(X;Y)& \le D_{\frac{1}{1+\rho }}(P_{XY}\parallel P_X\times Q_Y)\hfill \end{array}$$

(87)

$$\begin{array}{cc}\hfill & =\frac{1+\rho }{\rho }\underset{R_{XY}}{min}\left\{\frac{1}{1+\rho }D(R_{XY}\parallel P_{XY})+\frac{\rho }{1+\rho }D(R_{XY}\parallel P_X\times Q_Y)\right\}\hfill \end{array}$$

(88)

$$\begin{array}{cc}\hfill & \le \frac{1}{\rho }D(Q_{Y|X}{P}_X\parallel P_{XY})+D(Q_{Y|X}{P}_X\parallel P_X\times Q_Y)\hfill \end{array}$$

(89)

$$\begin{array}{cc}\hfill & =\frac{1}{\rho }D(Q_{Y|X}\parallel P_{Y|X}\left|P_X\right)+I(P_X,Q_{Y|X}),\hfill \end{array}$$

(90)

where (87), (88) and (90) follow from (69), (47) and (66) respectively. □

Just like (53), we will show in Section 7 that (86) becomes an equality upon the unconstrained maximization of both sides.

26.

Before introducing the last dependence measure in this section, recall from Definition 7 and (58) that $P_{Y\left[\alpha \right]}\ll P_Y$, the $\alpha$-response (of $P_{Y|X}$) to $P_X$ defined by

$$\begin{array}{c}\hfill {\mathit{ı}}_{Y\left[\alpha \right]\parallel Y}\left(y\right)=\frac{1}{\alpha }log\mathbb{E}[exp\left(\alpha {\mathit{ı}}_{X;Y}(X;y)+(1-\alpha )D_{\alpha }\left(P_{Y|X}\parallel P_{Y\left[\alpha \right]}|P_X\right)\right)],\end{array}$$

(91)

attains ${min}_{Q_Y}{D}_{\alpha }(P_{Y|X}\parallel Q_Y\left|P_X\right)$, where the expectation is with respect to $X\sim P_X$. We proceed to define $P_{Y\langle \alpha \rangle }\ll P_Y$, the $\langle \alpha \rangle$-response (of $P_{Y|X}$) to $P_X$ by means of

$$\begin{array}{c}\hfill {\mathit{ı}}_{Y\langle \alpha \rangle \parallel Y}\left(y\right)=\frac{1}{\alpha }log\mathbb{E}\left[exp(\alpha {\mathit{ı}}_{X;Y}(X;y)+(1-\alpha )D_{\alpha }\left(P_{Y|X}(·|X)\parallel P_{Y\langle \alpha \rangle }\right)\right],\end{array}$$

(92)

with $X\sim P_X$. Note that $P_{Y\langle 1\rangle }=P_{Y\left[1\right]}=P_Y$.

27.

In the case of discrete alphabets, (92) becomes the implicit equation

$$\begin{array}{c}\hfill P_{Y\langle \alpha \rangle }^{\alpha }\left(y\right)=\sum _{a\in \mathcal{A}}{P}_X\left(a\right)\frac{P_{Y|X}^{\alpha }\left(y\right|a)}{{\sum }_{b\in \mathcal{B}}{P}_{Y|X}^{\alpha }\left(b\right|a)P_{Y\langle \alpha \rangle }^{1-\alpha }\left(b\right)},y\in \mathcal{B},\end{array}$$

(93)

which coincides with (9.24) in Fano’s 1961 textbook [7], with $s\leftarrow 1-\alpha$, and is also given by Haroutunian in (19) of [22]. For example, if $\mathcal{A}=\mathcal{B}$ is discrete and $Y=X$, then $P_{Y\langle \alpha \rangle }=P_X$, while $P_{Y\left[\alpha \right]}^{\alpha }\left(y\right)=c{P}_X\left(y\right)$, $y\in \mathcal{A}$.

28.

The $\langle \alpha \rangle$-response satisfies the following identity, which can be regarded as the counterpart of (57) satisfied by the $\alpha$-response.

Theorem 6.

Fix $P_X\in {\mathcal{P}}_{\mathcal{A}}$, $P_{Y|X}:\mathcal{A}\to \mathcal{B}$ and $Q_Y\in {\mathcal{P}}_{\mathcal{B}}$. Then,

$$\begin{array}{c}{D}_{\alpha }(P_{Y\langle \alpha \rangle }\parallel Q_Y)\hfill \\ \hfill =\frac{1}{\alpha -1}log\mathbb{E}\left[exp\left((1-\alpha )\left(D_{\alpha }(P_{Y|X}(·|X)\parallel P_{Y\langle \alpha \rangle })-D_{\alpha }(P_{Y|X}(·|X)\parallel Q_Y)\right)\right)\right].\end{array}$$

(94)

Proof. 

For brevity we assume $Q_Y\ll P_Y$. Otherwise, the proof is similar adopting a reference measure that dominates both $Q_Y$ and $P_Y$. The definition of unconditional Rényi divergence in Item 11 implies that we can write $(\alpha -1)$ times the exponential of the left side of (94) as

$$\begin{array}{cc}\hfill & exp\left((\alpha -1)D_{\alpha }(P_{Y\langle \alpha \rangle }\parallel Q_Y)\right)=\mathbb{E}\left[{\left(\frac{\mathrm{d}{P}_{Y\langle \alpha \rangle }}{\mathrm{d}{P}_Y}\left(Y\right)\right)}^{\alpha }{\left(\frac{\mathrm{d}{Q}_Y}{\mathrm{d}{P}_Y}\left(Y\right)\right)}^{1-\alpha }\right]\hfill \end{array}$$

(95)

$$\begin{array}{cc}\hfill & =\mathbb{E}\left[exp\left(\alpha {\mathit{ı}}_{X;Y}(X;Y)+(1-\alpha )D_{\alpha }\left(P_{Y|X}(·|X)\parallel P_{Y\langle \alpha \rangle }\right)\right){\left(\frac{\mathrm{d}{Q}_Y}{\mathrm{d}{P}_Y}\left(Y\right)\right)}^{1-\alpha }\right]\hfill \\ \hfill & =\mathbb{E}\left[\mathbb{E}\left[exp\left(\alpha {\mathit{ı}}_{X;Y}(X;Y)+(1-\alpha )\left({\mathit{ı}}_{Q_Y\parallel P_Y}\left(Y\right)+D_{\alpha }\left(P_{Y|X}(·|X)\parallel P_{Y\langle \alpha \rangle }\right)\right)\right)|X\right]\right]\hfill \end{array}$$

(96)

$$\begin{array}{cc}\hfill & =\mathbb{E}\left[exp\left((1-\alpha )\left(D_{\alpha }\left(P_{Y|X}(·|X)\parallel P_{Y\langle \alpha \rangle }\right)-D_{\alpha }\left(P_{Y|X}(·|X)\parallel Q_Y\right)\right)\right)\right],\hfill \end{array}$$

(97)

where $(X,Y)\sim P_X\times P_Y$, (96) follows from (92), and (97) follows from the definition of unconditional Rényi divergence in (27). □

Theorem 7.

If $\alpha \in (0,1]$, then

$$\begin{array}{cc}\hfill D_{\alpha }(P_{Y\langle \alpha \rangle }\parallel Q_Y)& \le \mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X)\parallel Q_Y)\right]-\mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X)\parallel P_{Y\langle \alpha \rangle })\right]\hfill \end{array}$$

(98)

$$\begin{array}{cc}\hfill & \le D(P_{Y\langle \alpha \rangle }\parallel Q_Y).\hfill \end{array}$$

(99)

If $\alpha \ge 1$, inequalities (98) and (99) are reversed.

Proof. 

Assume $\alpha \in (0,1]$. Jensen’s inequality applied to the right side of (94) results in (98). To show (99), again we assume for brevity $Q_Y\ll P_Y$, and define the positive functions $V:\mathcal{A}\times \mathcal{B}\to (0,\infty )$ and $W:\mathcal{A}\times \mathcal{B}\to (0,\infty )$,

$$\begin{array}{cc}\hfill V(x,y)& =exp\left(\alpha {\mathit{ı}}_{X;Y}(x;y)+(1-\alpha ){\mathit{ı}}_{Y\langle \alpha \rangle \parallel Y}\left(y\right)\right),\hfill \end{array}$$

(100)

$$\begin{array}{cc}\hfill W(x,y)& =exp\left(\alpha {\mathit{ı}}_{X;Y}(x;y)+(1-\alpha ){\mathit{ı}}_{Q_Y\parallel P_Y}\left(y\right)\right).\hfill \end{array}$$

(101)

Note that, with $(X,Y)\sim P_X\times P_Y$, and $(x,y)\in \mathcal{A}\times \mathcal{B}$,

$$\begin{array}{cc}\hfill \mathbb{E}\left[V\right(x,Y\left)\right]& =exp\left((\alpha -1)D_{\alpha }(P_{Y|X=x}\parallel P_{Y\langle \alpha \rangle })\right),\hfill \end{array}$$

(102)

$$\begin{array}{cc}\hfill \mathbb{E}\left[W\right(x,Y\left)\right]& =exp\left((\alpha -1)D_{\alpha }(P_{Y|X=x}\parallel Q_Y)\right),\hfill \\ \hfill \mathbb{E}\left[\frac{V(X,y)}{\mathbb{E}\left[V\right(X,Y\left)\right|X]}\right]& =exp\left((1-\alpha ){\mathit{ı}}_{Y\langle \alpha \rangle \parallel Y}\left(y\right)\right)·\hfill \end{array}$$

(103)

$$\begin{array}{cc}\hfill & \phantom{=}·\mathbb{E}\left[exp\left(\alpha {\mathit{ı}}_{X;Y}(X;y)+(1-\alpha )D_{\alpha }(P_{Y|X}(·|X)\parallel P_{Y\langle \alpha \rangle })\right)\right]\hfill \end{array}$$

(104)

$$\begin{array}{cc}\hfill & =\frac{\mathrm{d}{P}_{Y\langle \alpha \rangle }}{\mathrm{d}{P}_Y}\left(y\right).\hfill \end{array}$$

(105)

where (104) uses (100) and (102) and (105) follows from (92). Then,

$$\begin{array}{cc}\hfill & D_{\alpha }(P_{Y|X=x}\parallel Q_Y)-D_{\alpha }(P_{Y|X=x}\parallel P_{Y\langle \alpha \rangle })\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =\frac{1}{1-\alpha }log\frac{\mathbb{E}\left[V\right(x,Y\left)\right]}{\mathbb{E}\left[W\right(x,Y\left)\right]}\hfill \end{array}$$

(106)

$$\begin{array}{cc}\hfill & \le \frac{1}{1-\alpha }\mathbb{E}\left[\frac{V(x,Y)}{\mathbb{E}\left[V\right(x,Y\left)\right]}log\frac{V(x,Y)}{W(x,Y)}\right]\hfill \end{array}$$

(107)

$$\begin{array}{cc}\hfill & =\mathbb{E}\left[\frac{V(x,Y)}{\mathbb{E}\left[V\right(x,Y\left)\right]}\left({\mathit{ı}}_{Y\langle \alpha \rangle \parallel Y}\left(Y\right)-{\mathit{ı}}_{Q_Y\parallel P_Y}\left(Y\right)\right)\right],\hfill \end{array}$$

(108)

where the expectations are with respect to $Y\sim P_Y$, and

• (107) follows from the log-sum inequality for integrable non-negative random variables,

  $$\begin{array}{c}\hfill \mathbb{E}\left[V\right]log\frac{\mathbb{E}\left[V\right]}{\mathbb{E}\left[W\right]}\le \mathbb{E}\left[Vlog\frac{V}{W}\right];\end{array}$$

  (109)
• (108) ⇐ (100) and (101).

Taking expectation with respect to $X\sim P_X$ of (106)–(108) yields (99) because of Lemma A1 and (105). If $\alpha \ge 1$, then Jensen’s inequality applied to the right side of (94) results in (98) but with the opposite inequality. Moreover, (107) is reversed and the remainder of the proof holds verbatim. □

In the case of finite input-alphabets, a different proof of (99) is given in Appendix B of [54].

29.

Introduced in the unpublished dissertation [36] and rescued from oblivion in [32], the Augustin–Csiszár mutual information of order $\alpha$ is defined for $\alpha >0$ as

$$\begin{array}{cc}\hfill I_{\alpha }^{\mathtt{c}}(X;Y)=I_{\alpha }^{\mathtt{c}}(P_X,P_{Y|X})=& \underset{Q_Y}{min}\mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X)\parallel Q_Y)\right]\hfill \end{array}$$

(110)

$$\begin{array}{cc}\hfill & =\mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X)\parallel P_{Y\langle \alpha \rangle })\right],\hfill \end{array}$$

(111)

where (111) follows from (98) if $\alpha \in (0,1]$, and from the reverse of (99) if $\alpha \ge 1$. We conform to the notation in [40], where $I_{\alpha }^{\mathtt{a}}$ was used to denote the difference between entropy and Arimoto-Rényi conditional entropy. In [32,39,43] the Augustin–Csiszár mutual information of order $\alpha$ is denoted by $I_{\alpha }$. In Augustin’s original notation [36], $I^{\rho }\left(P_X\right)$ means $I_{1-\rho }^{\mathtt{c}}(P_X,P_{Y|X})$, $\rho \in (0,1)$. Independently of [36], Poltyrev [35] introduced a functional (expressed as a maximization over a reverse random transformation) which turns out to be $\rho I_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y)$ and which he denoted by $E_0(\rho ,P_X)$, although in Gallager’s notation that corresponds to $\rho I_{\frac{1}{1+\rho }}(X;Y)$, as we will see in (233). $I_0^{\mathtt{c}}(X;Y)$ and $I_{\infty }^{\mathtt{c}}(X;Y)$ are defined by taking the corresponding limits.

30.

In the discrete case, (110) boils down to

$$\begin{array}{c}\hfill I_{\alpha }^{\mathtt{c}}(X;Y)=\underset{Q_Y}{min}\frac{1}{\alpha -1}\sum _{x\in \mathcal{A}}{P}_X\left(x\right)log\sum _{y\in \mathcal{B}}{P}_{Y|X}^{\alpha }\left(y\right|x)Q_Y^{1-\alpha }\left(y\right),\end{array}$$

(112)

which can be juxtaposed with the much easier expression in (82) for $I_{\alpha }(X;Y)$ involving no further optimization. Minimizing the Lagrangian, we can verify that the minimizer in (112) satisfies (93). With $(X,\overline{Y})\sim P_X\times Q_Y$, we have

$$\begin{array}{cc}\hfill I_0^{\mathtt{c}}(X;Y)& =\underset{Q_Y}{min}\mathbb{E}\left[log\frac{1}{\mathbb{P}[P_{Y|X}\left(\overline{Y}\right|X)>0\mid X]}\right],\hfill \end{array}$$

(113)

$$\begin{array}{cc}\hfill I_{\infty }^{\mathtt{c}}(X;Y)& =\underset{Q_Y}{min}\mathbb{E}\left[log{∥\frac{P_{Y|X}\left(\overline{Y}\right|X)}{Q_Y\left(\overline{Y}\right)}∥}_{\infty }\right],\hfill \end{array}$$

(114)

where the expectations are with respect to X.

31.

The respective minimizers of (72) and (110), namely, the $\alpha$-response and the $\langle \alpha \rangle$-response, are quite different. Most notably, in contrast to Item 7, an explicit expression for $P_{Y\langle \alpha \rangle }$ is unknown. Instead of defining $P_{Y\langle \alpha \rangle }$ through (92), [36] defines it, equivalently, as the fixed point of the operator (dubbed the Augustin operator in [43]) which maps the set of probability measures on the output space to itself,

$$\begin{array}{c}\hfill \frac{\mathrm{d}{\mathbb{T}}_{\alpha }\left(Q\right)}{\mathrm{d}Q}\left(y\right)=\mathbb{E}\left[{\left(\frac{\mathrm{d}{P}_{Y|X}}{\mathrm{d}Q}\left(y\right|X)\right)}^{\alpha }exp\left((1-\alpha )D_{\alpha }(P_{Y|X}(·|X)\parallel Q)\right)\right],\end{array}$$

(115)

where $X\sim P_X$. Although we do not rely on them, Lemma 34.2 of ($\alpha \in (0,1)$) and Lemma 13 of [43] ($\alpha >1$) claim that the minimizer in (110), referred to in [43] as the Augustin mean of order $\alpha$, is unique and is a fixed point of the operator ${\mathbb{T}}_{\alpha }$ regardless of $P_X$. Moreover, Lemma 13(c) of [43] establishes that for $\alpha \in (0,1)$ and finite input alphabets, repeated iterations of the operator ${\mathbb{T}}_{\alpha }$ with initial argument $P_{Y\left[\alpha \right]}$ converge to $P_{Y\langle \alpha \rangle }$.

32.

It is interesting to contrast the next example with the formulas in Examples 2 and 6.

Example 7.

Additive independent Gaussian noise.If $Y=X+N$, where $X\sim \mathcal{N}\left(0,{\sigma }_X^2\right)$ independent of $N\sim \mathcal{N}\left(0,{\sigma }_N^2\right)$, then

$$\begin{array}{cc}\hfill Y\langle \alpha \rangle & \sim \mathcal{N}\left(0,\frac{{\sigma }_N^2}{2}\left(2-\frac{1}{\alpha }+\Delta +\mathsf{snr}\right)\right),\hfill \end{array}$$

(116)

$$\begin{array}{cc}\hfill \mathsf{snr}& =\frac{{\sigma }_X^2}{{\sigma }_N^2},\hfill \end{array}$$

(117)

$$\begin{array}{cc}\hfill \Delta & =\sqrt{4\mathsf{snr}+{\left(\frac{1}{\alpha }-\mathsf{snr}\right)}^2}.\hfill \end{array}$$

(118)

This result can be obtained by postulating a zero-mean Gaussian distribution with variance $v_{\alpha }^2$ as $P_{Y\langle \alpha \rangle }$ and verifying that (92) is indeed satisfied if $v_{\alpha }^2$ is chosen as in (116). The first step is to invoke (32), which yields

$$\begin{array}{cc}\hfill D_{\alpha }\left(P_{Y|X=x}\parallel P_{Y\langle \alpha \rangle }\right)& =\frac{{\lambda }_{\alpha }}{2}+\frac{\alpha x^2}{2s_{\alpha }^2}log\mathrm{e},\hfill \end{array}$$

(119)

$$\begin{array}{cc}\hfill {\lambda }_{\alpha }& =log\frac{v_{\alpha }^2}{{\sigma }_N^2}+\frac{1}{\alpha -1}log\frac{v_{\alpha }^2}{s_{\alpha }^2},\hfill \end{array}$$

(120)

where we have denoted $s_{\alpha }^2=\alpha v_{\alpha }^2+(1-\alpha ){\sigma }_N^2$. Since $Y\sim \mathcal{N}\left(0,{\sigma }_X^2+{\sigma }_N^2\right)$,

$$\begin{array}{cc}\hfill {\mathit{ı}}_{X;Y}(x;y)& =\frac{1}{2}log\frac{{\sigma }_X^2+{\sigma }_N^2}{{\sigma }_N^2}+\frac{1}{2}\left(\frac{y^2}{{\sigma }_X^2+{\sigma }_N^2}-\frac{{(y-x)}^2}{{\sigma }_N^2}\right)log\mathrm{e},\hfill \end{array}$$

(121)

$$\begin{array}{cc}\hfill {\mathit{ı}}_{Y\langle \alpha \rangle \parallel Y}\left(y\right)& =\frac{1}{2}log\frac{{\sigma }_X^2+{\sigma }_N^2}{v_{\alpha }^2}+\frac{1}{2}\left(\frac{y^2}{{\sigma }_X^2+{\sigma }_N^2}-\frac{y^2}{v_{\alpha }^2}\right)log\mathrm{e}.\hfill \end{array}$$

(122)

Assembling (120) and (121), the right side of (92) becomes

$$\begin{array}{cc}\hfill & \frac{1}{\alpha }log\mathbb{E}\left[exp(\alpha {\mathit{ı}}_{X;Y}(X;y)+(1-\alpha )D_{\alpha }\left(P_{Y|X}(·|X)\parallel P_{Y\langle \alpha \rangle }\right)\right]\hfill \\ \hfill & =\frac{1}{2}log\frac{{\sigma }_X^2+{\sigma }_N^2}{{\sigma }_N^2}+\frac{1}{2}\frac{y^{2}log\mathrm{e}}{{\sigma }_X^2+{\sigma }_N^2}+\frac{1-\alpha }{2\alpha }{\lambda }_{\alpha }\hfill \\ \hfill & +\frac{1}{\alpha }log\mathbb{E}\left[{exp}_{\mathrm{e}}\left(-\frac{\alpha {(y-X)}^2}{2{\sigma }_N^2}+\frac{\alpha (1-\alpha )X^2}{2s_{\alpha }^2}\right)\right]\hfill \end{array}$$

(123)

$$\begin{array}{cc}\hfill & =\frac{1}{2}log\frac{{\sigma }_X^2+{\sigma }_N^2}{{\sigma }_N^2}+\frac{1-\alpha }{2\alpha }{\lambda }_{\alpha }+\frac{y^{2}log\mathrm{e}}{2}\left(\frac{1}{{\sigma }_X^2+{\sigma }_N^2}-\frac{s_{\alpha }^2-\alpha (1-\alpha ){\sigma }_X^2}{{\sigma }_N^2{s}_{\alpha }^2+{\alpha }^2{v}_{\alpha }^2{\sigma }_X^2}\right)\hfill \\ \hfill & +\frac{1}{2\alpha }log\frac{{\sigma }_N^2{s}_{\alpha }^2}{{\sigma }_N^2{s}_{\alpha }^2+{\alpha }^2{v}_{\alpha }^2{\sigma }_X^2}\hfill \end{array}$$

(124)

$$\begin{array}{cc}\hfill & =\frac{1}{2}log\frac{{\sigma }_X^2+{\sigma }_N^2}{v_{\alpha }^2}+\frac{1}{2}\left(\frac{y^2}{{\sigma }_X^2+{\sigma }_N^2}-\frac{y^2}{v_{\alpha }^2}\right)log\mathrm{e},\hfill \end{array}$$

(125)

where (124) follows by Gaussian integration, and the marvelous simplification in (125) is satisfied provided that we choose

$$\begin{array}{c}\hfill s_{\alpha }^2=\frac{\alpha {\sigma }_X^2{v}_{\alpha }^2}{v_{\alpha }^2-{\sigma }_N^2}.\end{array}$$

(126)

Comparing (122) and (125), we see that (92) is indeed satisfied with $Y\langle \alpha \rangle \sim \mathcal{N}\left(0,v_{\alpha }^2\right)$ if $v_{\alpha }^2$ satisfies the quadratic Equation (126), whose solution is in (116)–(118). Invoking (32) and (116), we obtain

$$\begin{array}{cc}\hfill I_{\alpha }^{\mathtt{c}}(X;X+N)& =\frac{\alpha \mathsf{snr}}{1+\alpha \Delta +\alpha \mathsf{snr}}log\mathrm{e}+\frac{1}{2}log\left(1+\frac{1}{2}\left(\Delta +\mathsf{snr}-\frac{1}{\alpha }\right)\right)\hfill \\ \hfill & -\frac{1}{2(1-\alpha )}log\frac{2-\frac{1}{\alpha }+\Delta +\mathsf{snr}}{1+\alpha \Delta +\alpha \mathsf{snr}}.\hfill \end{array}$$

(127)

Beyond its role in evaluating the Augustin–Csiszár mutual information for Gaussian inputs, the Gaussian distribution in (116) has found some utility in the analysis of finite blocklength fundamental limits for data transmission [55].

33.

This item gives a variational representation for the Augustin–Csiszár mutual information in terms of mutual information and conditional relative entropy (i.e., non-Rényi information measures). As we will see in Section 10, this representation accounts for the role played by Augustin–Csiszár mutual information in expressing error exponent functions.

Theorem 8.

For $\alpha \in (0,1)$, the Augustin–Csiszár mutual information satisfies the variational representation in terms of conditional relative entropy and mutual information,

$$\begin{array}{c}\hfill (1-\alpha )I_{\alpha }^{\mathtt{c}}(P_X,P_{Y|X})=\underset{Q_{Y|X}}{min}\left\{\alpha D(Q_{Y|X}\parallel P_{Y|X}\left|P_X\right)+(1-\alpha )I(P_X,Q_{Y|X})\right\},\end{array}$$

(128)

where the minimum is over all the random transformations from the input to the output spaces.

Proof. 

Invoking (47) with $(P_{\mathtt{1}},P_{\mathtt{0}})\leftarrow (P_{Y|X=x},Q_Y)$ we obtain

$$\begin{array}{cc}\hfill (1-\alpha )D_{\alpha }(P_{Y|X=x}\parallel Q_Y)& =\underset{R_Y}{min}\left\{\alpha D(R_Y\parallel P_{Y|X=x})+(1-\alpha )D(R_Y\parallel Q_Y)\right\}\hfill \end{array}$$

(129)

$$\begin{array}{cc}\hfill & =\underset{R_{Y|X=x}}{min}\left\{\alpha D(R_{Y|X=x}\parallel P_{Y|X=x})+(1-\alpha )D(R_{Y|X=x}\parallel Q_Y)\right\}.\hfill \end{array}$$

(130)

Averaging over $x\sim P_X$, followed by minimization with respect to $Q_Y$ yields (128) upon recalling (67). □

In the finite-alphabet case with $\alpha \in (0,1)\cup (1,\infty )$, the representation in (128) is implicit in the appendix of [32], and stated explicitly in [39], where it is shown by means of a minimax theorem. This is one of the instances in which the proof of the result is considerably easier for $\alpha \in (0,1)$; we can take the following route to show (128) for $\alpha >1$. Neglecting to emphasize its dependence on $P_X$, denote

$$\begin{array}{c}\hfill f_{\alpha }(Q_Y,R_{Y|X})=\frac{\alpha }{1-\alpha }D(R_{Y|X}\parallel P_{Y|X}|P_X)+D(R_{Y|X}\parallel Q_Y\left|P_X\right).\end{array}$$

(131)

Invoking (47) we obtain

$$\begin{array}{cc}\hfill D_{\alpha }(P_{Y|X=x}\parallel Q_Y)& =\underset{R_{Y|X=x}}{max}\left\{\frac{\alpha }{1-\alpha }D(R_{Y|X=x}\parallel P_{Y|X=x})+D(R_{Y|X=x}\parallel Q_Y)\right\}.\hfill \end{array}$$

(132)

Averaging (132) with respect to $P_X$ followed by minimization over $Q_Y$, results in

$$\begin{array}{cc}\hfill I_{\alpha }^{\mathtt{c}}(P_X,P_{Y|X})& =\underset{Q_Y}{min}\underset{R_{Y|X}}{max}{f}_{\alpha }(Q_Y,R_{Y|X})\hfill \end{array}$$

(133)

$$\begin{array}{cc}\hfill & \ge \underset{R_{Y|X}}{max}\underset{Q_Y}{min}{f}_{\alpha }(Q_Y,R_{Y|X})\hfill \end{array}$$

(134)

$$\begin{array}{cc}\hfill & =\underset{R_{Y|X}}{max}\left\{\frac{\alpha }{1-\alpha }D(R_{Y|X}\parallel P_{Y|X}\left|P_X\right)+I(P_X,R_{Y|X})\right\},\hfill \end{array}$$

(135)

which shows ≥ in (128). If a minimax theorem can be invoked to show equality in (134), then (128) is established for $\alpha >1$. For that purpose, for fixed $R_{Y|X}$, $f(·,R_{Y|X})$ is convex and lower semicontinuous in $Q_Y$ on the set where it is finite. Rewriting

$$\begin{array}{c}f(Q_Y,R_{Y|X})\hfill \\ \hfill =\frac{1}{1-\alpha }D(R_{Y|X}\parallel P_{Y|X}|P_X)+D(R_{Y|X}\parallel Q_Y|P_X)-D(R_{Y|X}\parallel P_{Y|X}\left|P_X\right),\end{array}$$

(136)

it can be seen that $f(Q_Y,·)$ is upper semicontinuous and concave (if $\alpha >1$). A different, and considerably more intricate route is taken in Lemma 13(d) of [43], which also gives (128) for $\alpha >1$ assuming finite input alphabets.

34.

Unlike mutual information, neither $I_{\alpha }(X;Y)=I_{\alpha }(Y;X)$ nor $I_{\alpha }^{\mathtt{c}}(X;Y)=I_{\alpha }^{\mathtt{c}}(Y;X)$ hold in general.

Example 8.

Erasure transformation. Let $\mathcal{A}=\{\mathtt{0},\mathtt{1}\},\mathcal{B}=\{\mathtt{0},\mathtt{1},\mathtt{e}\}$,

$$P_{Y|X}\left(b\right|a)=\left\{\begin{array}{cc}1-\delta ,\hfill & a=b;\hfill \\ \delta ,\hfill & b=\mathtt{e};\hfill \\ 0,\hfill & a\ne b\ne \mathtt{e},\hfill \end{array}\right.$$

(137)

with $\delta \in (0,1)$, and $P_X\left(\mathtt{0}\right)=\frac{1}{2}$. Then, we obtain, for $\alpha \in (0,1)\cup (1,\infty )$,

$$\begin{array}{cc}\hfill I_{\alpha }(X;Y)& =I_{\alpha }^{\mathtt{c}}(X;Y)=\frac{\alpha }{\alpha -1}log\left(\delta +(1-\delta )2^{\left(1-\frac{1}{\alpha }\right)}\right),\hfill \end{array}$$

(138)

$$\begin{array}{cc}\hfill I_{\alpha }(Y;X)& =\frac{1}{\alpha -1}log\left(\delta +(1-\delta )2^{\alpha -1}\right),\hfill \end{array}$$

(139)

$$\begin{array}{cc}\hfill I_{\alpha }^{\mathtt{c}}(Y;X)& =I(X;Y)=1-\delta bits.\hfill \end{array}$$

(140)

35.

It was shown in Theorem 5.2 of [38] that $\alpha$-mutual information satisfies the data processing lemma, namely, if X and Z are conditionally independent given Y, then

$$\begin{array}{cc}\hfill I_{\alpha }(X;Z)& \le min\left\{I_{\alpha }(X;Y),I_{\alpha }(Y;Z)\right\},\hfill \end{array}$$

(141)

$$\begin{array}{cc}\hfill I_{\alpha }(Z;X)& \le min\left\{I_{\alpha }(Z;Y),I_{\alpha }(Y;X)\right\}.\hfill \end{array}$$

(142)

As shown by Csiszár [32] using the data processing inequality for Rényi divergence, the data processing lemma also holds for $I_{\alpha }^{\mathtt{c}}$.

36.

From (53), (54) and the monotonicity of $D_{\alpha }(P\parallel Q)$ in $\alpha$, we obtain the ordering

$$\begin{array}{cc}\hfill I_{\beta }(X;Y)\le I_{\alpha }(X;Y)& \le I_{\alpha }^{\mathtt{c}}(X;Y)\le I_{\nu }^{\mathtt{c}}(X;Y)\le I(X;Y),0<\beta \le \alpha \le \nu <1;\hfill \end{array}$$

(143)

$$\begin{array}{cc}\hfill I(X;Y)\le I_{\nu }^{\mathtt{c}}(X;Y)& \le I_{\alpha }^{\mathtt{c}}(X;Y)\le I_{\alpha }(X;Y)\le I_{\beta }(X;Y),1<\nu \le \alpha \le \beta .\hfill \end{array}$$

(144)

37.

The convexity/concavity properties of the generalized mutual informations are summarized next.

Theorem 9.

(a)

$\rho I_{\frac{1}{1+\rho }}(X;Y)$ and $\rho I_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y)$ are concave and monotonically non-decreasing in $\rho \ge 0$.

(b)

$I(·,P_{Y|X})$ and $I_{\alpha }^{\mathtt{c}}(·,P_{Y|X})$ are concave functions. The same holds for $I_{\alpha }(·,P_{Y|X})$ if $\alpha >1$.

(c)

If $\alpha \in (0,1)$, then $I(P_X,·)$, $I_{\alpha }(P_X,·)$ and $I_{\alpha }^{\mathtt{c}}(P_X,·)$ are convex functions.

Proof. 

(a)

According to (81) and (128), respectively, with $\alpha =\frac{1}{1+\rho }\in (0,1)$, $\rho I_{\frac{1}{1+\rho }}(X;Y)$ and $\rho I_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y)$ are the infima of affine functions with nonnegative slopes.

(b)

For mutual information the result goes back to [56] in the finite-alphabet case. In general, it holds since (67) is the infimum of linear functions of $P_X$. The same reasoning applies to Augustin–Csiszár mutual information in view of (110). For $\alpha$-mutual information with $\alpha >1$, notice from (51) that $D_{\alpha }(P_{Y|X}\parallel Q_Y\left|P_X\right)$ is concave in $P_X$ if $\alpha >1$. Therefore,

$$\begin{array}{cc}\hfill & I_{\alpha }(\lambda P_X^1+(1-\lambda )P_X^0,P_{Y|X})\hfill \end{array}$$

(145)

$$\begin{array}{cc}\hfill & =\underset{Q_Y}{inf}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|\lambda P_X^1+(1-\lambda )P_X^0)\hfill \end{array}$$

(146)

$$\begin{array}{cc}\hfill & \ge \underset{Q_Y}{inf}\lambda D_{\alpha }(P_{Y|X}\parallel Q_Y|P_X^1)+(1-\lambda )D_{\alpha }(P_{Y|X}\parallel Q_Y\left|P_X^0\right)\hfill \end{array}$$

(147)

$$\begin{array}{cc}\hfill & \ge \lambda I_{\alpha }(P_X^1,P_{Y|X})+(1-\lambda )I_{\alpha }(P_X^0,P_{Y|X}).\hfill \end{array}$$

(148)

(c)

The convexity of $I(P_X,·)$ and $I_{\alpha }(P_X,·)$ follow from the convexity of $D_{\alpha }(P\parallel Q)$ in $(P,Q)$ for $\alpha \in (0,1]$ as we saw in Item 18. To show convexity of $I_{\alpha }^{\mathtt{c}}(P_X,·)$ if $\alpha \in (0,1)$, we apply (169) in Item 45 with $P_{Y|X}=\lambda P_{Y|X}^1+(1-\lambda )P_{Y|X}^0$, and invoke the convexity of $I_{\alpha }(P_X,·)$:

$$\begin{array}{cc}\hfill & (1-\alpha )I_{\alpha }^{\mathtt{c}}(P_X,P_{Y|X})\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =\underset{Q_X}{max}\left\{(1-\alpha )I_{\alpha }(Q_X,\lambda P_{Y|X}^1+(1-\lambda )P_{Y|X}^0)-D(P_X\parallel Q_X)\right\},\hfill \\ \hfill & \le \underset{Q_X}{max}\left\{\lambda \left(1-\alpha )I_{\alpha }(Q_X,P_{Y|X}^1)-D(P_X\parallel Q_X)\right)\right.\hfill \end{array}$$

(149)

$$\begin{array}{cc}\hfill & \left.+(1-\lambda )\left(1-\alpha )I_{\alpha }(Q_X,P_{Y|X}^0)-D(P_X\parallel Q_X)\right)\right\}\hfill \end{array}$$

(150)

$$\begin{array}{cc}\hfill & \le (1-\alpha )\left(\lambda I_{\alpha }^{\mathtt{c}}(P_X,P_{Y|X}^1)+(1-\lambda )I_{\alpha }^{\mathtt{c}}(P_X,P_{Y|X}^0)\right).\hfill \end{array}$$

(151)

□

Although not used in the sequel, we note, for completeness, that if $\alpha \in (0,1)\cup (1,\infty )$, [38] (see corrected version in [41]) shows that $exp\left(\left(1-\frac{1}{\alpha }\right)I_{\alpha }(·,P_{Y|X})\right)/(\alpha -1)$ is concave.

5. Interplay between ${\mathit{I}}_{\mathit{\alpha }}({\mathit{P}}_{\mathit{X}},{\mathit{P}}_{\mathit{Y}|\mathit{X}})$ and ${\mathit{I}}_{\mathit{\alpha }}^{\mathtt{c}}({\mathit{P}}_{\mathit{X}},{\mathit{P}}_{\mathit{Y}|\mathit{X}})$

In this section we study the interplay between both notions of mutual informations of order $\alpha$, and, in particular, various variational representations of these information measures.

38.

For given $\alpha \in (0,1)\cup (1,\infty )$ and $P_{Y|X}:\mathcal{A}\to \mathcal{B}$, define $Q_{X\left[\alpha \right]}\ll \gg P_X$, the $\alpha$-adjunct of $P_X$ by

$$\begin{array}{c}\hfill {\mathit{ı}}_{Q_{X\left[\alpha \right]}\parallel P_X}\left(x\right)=(\alpha -1)D_{\alpha }\left(P_{Y|X=x}\parallel P_{Y\left[\alpha \right]}\right)-{\kappa }_{\alpha },\end{array}$$

(152)

with ${\kappa }_{\alpha }$ the constant in (14) and $P_{Y\left[\alpha \right]}$, the $\alpha$-response to $P_X$.

39.

Example 9.

Let $Y=X+N$ with $X\sim \mathcal{N}\left(0,{\sigma }_X^2\right)$ independent of $N\sim \mathcal{N}\left(0,{\sigma }_N^2\right)$, and $\mathsf{snr}=\frac{{\sigma }_X^2}{{\sigma }_N^2}$. The α-adjunct of the input is

$$\begin{array}{c}\hfill Q_{X\left[\alpha \right]}=\mathcal{N}\left(0,{\sigma }_X^2\frac{1+{\alpha }^2\mathsf{snr}}{1+\alpha \mathsf{snr}}\right).\end{array}$$

(153)

40.

Theorem 10.

The $\langle \alpha \rangle$-response to $Q_{X\left[\alpha \right]}$ is $P_{Y\left[\alpha \right]}$, the α-response to $P_X$.

Proof. 

We just need to verify that (92) is satisfied if we substitute $Y\langle \alpha \rangle$ by $Y\left[\alpha \right]$, and instead of taking the expectation in the right side with respect to $X\sim P_X$ we take it with respect to $\tilde{X}\sim Q_{X\left[\alpha \right]}$. Then,

$$\begin{array}{cc}\hfill & \mathbb{E}\left[exp(\alpha {\mathit{ı}}_{X;Y}(\tilde{X};y)+(1-\alpha )D_{\alpha }\left(P_{Y|X}(·|\tilde{X})\parallel P_{Y\left[\alpha \right]}\right)\right]\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =\mathbb{E}\left[exp\left({\mathit{ı}}_{Q_{X\left[\alpha \right]}\parallel P_X}\left(X\right)+\alpha {\mathit{ı}}_{X;Y}(X;y)+(1-\alpha )D_{\alpha }\left(P_{Y|X}(·|X)\parallel P_{Y\left[\alpha \right]}\right)\right)\right]\hfill \end{array}$$

(154)

$$\begin{array}{cc}\hfill & =\mathbb{E}\left[exp(\alpha {\mathit{ı}}_{X;Y}(X;y)-{\kappa }_{\alpha })\right]\hfill \end{array}$$

(155)

$$\begin{array}{cc}\hfill & =exp\left(\alpha {\mathit{ı}}_{Y\left[\alpha \right]\parallel Y}\left(y\right)\right),\hfill \end{array}$$

(156)

where (154) is by change of measure, (155) follows by substitution of (152), and (156) is the same as (13). □

41.

For given $\alpha \in (0,1)\cup (1,\infty )$ and $P_{Y|X}:\mathcal{A}\to \mathcal{B}$, we define $Q_{X\langle \alpha \rangle }\ll \gg P_X$, the $\langle \alpha \rangle$-adjunct of an input probability measure $P_X$ through

$$\begin{array}{c}\hfill {\mathit{ı}}_{Q_{X\langle \alpha \rangle }\parallel P_X}\left(x\right)=(1-\alpha )D_{\alpha }\left(P_{Y|X=x}\parallel P_{Y\langle \alpha \rangle }\right)+{\upsilon }_{\alpha },\end{array}$$

(157)

where $P_{Y\langle \alpha \rangle }$ is the $\langle \alpha \rangle$-response to $P_X$ and ${\upsilon }_{\alpha }$ is a normalizing constant so that $Q_{X\langle \alpha \rangle }$ is a probability measure. According to (9), we must have

$$\begin{array}{c}\hfill \mathbb{E}\left[exp\left({\mathit{ı}}_{Q_{X\langle \alpha \rangle }\parallel P_X}\left(X\right)\right)\right]=1,X\sim P_X.\end{array}$$

(158)

Hence,

$$\begin{array}{cc}\hfill {\upsilon }_{\alpha }& =(\alpha -1)D_{\alpha }\left(P_{Y|X}\parallel P_{Y\langle \alpha \rangle }|Q_{X\langle \alpha \rangle }\right).\hfill \end{array}$$

(159)

42.

With the aid of the expression in Example 7, we obtain

Example 10.

Let $Y=X+N$ with $X\sim \mathcal{N}\left(0,{\sigma }_X^2\right)$ independent of $N\sim \mathcal{N}\left(0,{\sigma }_N^2\right)$, and $\mathsf{snr}=\frac{{\sigma }_X^2}{{\sigma }_N^2}$. Then, the $\langle \alpha \rangle$-adjunct of the input is

$$\begin{array}{c}\hfill Q_{X\langle \alpha \rangle }=\mathcal{N}\left(0,{\sigma }_X^2\frac{1+\alpha (\Delta +\mathsf{snr})}{1+\alpha (\Delta -\mathsf{snr})+2{\alpha }^2\mathsf{snr}}\right),\end{array}$$

(160)

which, in contrast to $Q_{X\left[\alpha \right]}$, has larger variance than ${\sigma }_X^2$ if $\alpha \in (0,1)$.

43.

The following result is the dual of Theorem 10.

Theorem 11.

The α-response to $Q_{X\langle \alpha \rangle }$ is $P_{Y\langle \alpha \rangle }$, the $\langle \alpha \rangle$-response to $P_X$. Therefore,

$$\begin{array}{cc}\hfill {\upsilon }_{\alpha }& =(\alpha -1)I_{\alpha }\left(Q_{X\langle \alpha \rangle },P_{Y|X}\right).\hfill \end{array}$$

(161)

Proof. 

The proof is similar to that of Theorem 10. We just need to verify that we obtain the right side of (92) if on the right side of (91) we substitute $P_X$ by $Q_{X\langle \alpha \rangle }$ and $P_{Y\left[\alpha \right]}$ by $P_{Y\langle \alpha \rangle }$. Let $\overline{X}\sim Q_{X\langle \alpha \rangle }$. Then,

$$\begin{array}{cc}\hfill & \frac{1}{\alpha }log\mathbb{E}\left[exp\left(\alpha {\mathit{ı}}_{X;Y}(\overline{X};y)+(1-\alpha )D_{\alpha }\left(P_{Y|X}\parallel P_{Y\langle \alpha \rangle }|Q_{X\langle \alpha \rangle }\right)\right)\right]\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha }log\mathbb{E}\left[exp\left({\mathit{ı}}_{Q_{X\langle \alpha \rangle }\parallel P_X}\left(X\right)+\alpha {\mathit{ı}}_{X;Y}(X;y)-{\upsilon }_{\alpha }\right)\right]\hfill \end{array}$$

(162)

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha }log\mathbb{E}\left[exp\left(\alpha {\mathit{ı}}_{X;Y}(X;y)+(1-\alpha )D_{\alpha }\left(P_{Y|X}(·|X)\parallel P_{Y\langle \alpha \rangle }\right)\right)\right]\hfill \end{array}$$

(163)

$$\begin{array}{cc}\hfill & ={\mathit{ı}}_{Y\langle \alpha \rangle \parallel Y}\left(y\right),\hfill \end{array}$$

(164)

where (162)–(164) follow by change of measure, (157), and (92), respectively. □

44.

By recourse to a minimax theorem, the following representation is given for $\alpha \in (0,1)\cup (1,\infty )$ in the case of finite alphabets in [39], and dropping the restriction on the finiteness of the output space in [43]. As we show, a very simple and general proof is possible for $\alpha \in (0,1)$.

Theorem 12.

Fix $\alpha \in (0,1)$, $P_X\in {\mathcal{P}}_{\mathcal{A}}$ and $P_{Y|X}:\mathcal{A}\to \mathcal{B}$. Then,

$$\begin{array}{c}\hfill (1-\alpha )I_{\alpha }(X;Y)=\underset{Q_X}{min}\left\{(1-\alpha )I_{\alpha }^{\mathtt{c}}(Q_X,P_{Y|X})+D(Q_X\parallel P_X)\right\},\end{array}$$

(165)

where the minimum is attained by $Q_{X\left[\alpha \right]}$, the α-adjunct of $P_X$ defined in (152).

Proof. 

The variational representations in (81) and (128) result in (165). To show that the minimum is indeed attained by $Q_{X\left[\alpha \right]}$, recall from Theorem 10 that the $\langle \alpha \rangle$-response to $Q_{X\left[\alpha \right]}$ is $P_{Y\left[\alpha \right]}$. Therefore, evaluating the term in $\left\{\right\}$ in (165) for $Q_X\leftarrow Q_{X\left[\alpha \right]}$ yields, with $\tilde{X}\sim Q_{X\left[\alpha \right]}$,

$$\begin{array}{cc}\hfill & (1-\alpha )I_{\alpha }^{\mathtt{c}}(Q_{X\left[\alpha \right]},P_{Y|X})+D(Q_{X\left[\alpha \right]}\parallel P_X)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =(1-\alpha )\mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|\tilde{X})\parallel P_{Y\left[\alpha \right]})\right]+D(Q_{X\left[\alpha \right]}\parallel P_X)\hfill \end{array}$$

(166)

$$\begin{array}{cc}\hfill & =-{\kappa }_{\alpha }\hfill \end{array}$$

(167)

$$\begin{array}{cc}\hfill & =(1-\alpha )I_{\alpha }(X;Y),\hfill \end{array}$$

(168)

where (167) follows from (152) and (168) results from (69)–(75). □

45.

For finite-input alphabets, Lemma 18(b) of [43] (earlier Theorem 3.4 of [35] gave an equivalent variational characterization assuming, in addition, finite output alphabets) established the following dual to Theorem 12.

Theorem 13.

Fix $\alpha \in (0,1)$, $P_X\in {\mathcal{P}}_{\mathcal{A}}$ and $P_{Y|X}:\mathcal{A}\to \mathcal{B}$. Then,

$$\begin{array}{c}\hfill (1-\alpha )I_{\alpha }^{\mathtt{c}}(X;Y)=\underset{Q_X}{max}\left\{(1-\alpha )I_{\alpha }(Q_X,P_{Y|X})-D(P_X\parallel Q_X)\right\}.\end{array}$$

(169)

The maximum is attained by $Q_{X\langle \alpha \rangle }$, the $\langle \alpha \rangle$-adjunct of $P_X$ defined by (157).

Proof. 

First observe that (165) implies that ≥ holds in (169). Second, the term in $\left\{\right\}$ on the right side of (169) evaluated at $Q_X\leftarrow Q_{X\langle \alpha \rangle }$ becomes

$$\begin{array}{cc}\hfill & (1-\alpha )I_{\alpha }(Q_{X\langle \alpha \rangle },P_{Y|X})-D(P_X\parallel Q_{X\langle \alpha \rangle })\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =(1-\alpha )I_{\alpha }(Q_{X\langle \alpha \rangle },P_{Y|X})+(1-\alpha )I_{\alpha }^{\mathtt{c}}(P_X,P_{Y|X})+{\upsilon }_{\alpha }\hfill \end{array}$$

(170)

$$\begin{array}{cc}\hfill & =(1-\alpha )I_{\alpha }^{\mathtt{c}}(P_X,P_{Y|X}),\hfill \end{array}$$

(171)

where (170) follows by taking the expectation of minus (157) with respect to $P_X$. Therefore, ≤ also holds in (169) and the maximum is attained by $Q_{X\langle \alpha \rangle }$, as we wanted to show. □

Hinging on Theorem 8, Theorems 12 and 13 are given for $\alpha \in (0,1)$ which is the region of interest in the analysis of error exponents. Whenever, as in the finite-alphabet case, (128) holds for $\alpha >1$, Theorems 12 and 13 also hold for $\alpha >1$.

Notice that since the definition of $Q_{X\langle \alpha \rangle }$ involves $P_{Y\langle \alpha \rangle }$, the fact that it attains the maximum in (169) does not bring us any closer to finding $I_{\alpha }^{\mathtt{c}}(X;Y)$ for a specific input probability measure $P_X$. Fortunately, as we will see in Section 8, (169) proves to be the gateway to the maximization of $I_{\alpha }^{\mathtt{c}}(X;Y)$ in the presence of input-cost constraints.

46.

Focusing on the main range of interest, $\alpha \in (0,1)$, we can express (169) as

$$\begin{array}{cc}\hfill I_{\alpha }^{\mathtt{c}}(P_X,P_{Y|X})& =\underset{Q_X}{max}\left\{I_{\alpha }(Q_X,P_{Y|X})-\frac{1}{1-\alpha }D(P_X\parallel Q_X)\right\}\hfill \end{array}$$

(172)

$$\begin{array}{cc}\hfill & =\underset{\xi \ge 0}{max}\left\{\mathbb{I}\left(\xi \right)-\frac{\xi }{1-\alpha }\right\}\hfill \end{array}$$

(173)

$$\begin{array}{cc}\hfill & =\mathbb{I}\left({\xi }_{\alpha }\right)-\frac{{\xi }_{\alpha }}{1-\alpha },\hfill \end{array}$$

(174)

where we have defined the function (dependent on $\alpha$, $P_X$, and $P_{Y|X}$)

$$\mathbb{I}\left(\xi \right)=\underset{\begin{array}{c}{Q}_X:\\ D(P_X\parallel Q_X)\le \xi \end{array}}{max}{I}_{\alpha }(Q_X,P_{Y|X}),$$

(175)

and ${\xi }_{\alpha }$ is the solution to

$$\begin{array}{c}\hfill \dot{\mathbb{I}}\left({\xi }_{\alpha }\right)=\frac{1}{1-\alpha }.\end{array}$$

(176)

Recall that the maxima over the input distribution in (172) and (175) are attained by the $\langle \alpha \rangle$-adjunct $Q_{X\langle \alpha \rangle }$ defined in Item 41.

47.

At this point it is convenient to summarize the notions of input and output probability measures that we have defined for a given $\alpha$, random transformation $P_{Y|X}$, and input probability measure $P_X$:

• $P_Y$: The familiar output probability measure $P_X\to P_{Y|X}\to P_Y$, defined in Item 5.
• $P_{Y\left[\alpha \right]}$: The $\alpha$-response to $P_X$, defined in Item 7. It is the unique achiever of the minimization in the definition of $\alpha$-mutual information in (67).
• $P_{Y\langle \alpha \rangle }$: The $\langle \alpha \rangle$-response to $P_X$ defined in Item 26. It is the unique achiever of the minimization in the definition of Augustin–Csiszár $\alpha$-mutual information in (110).
• $Q_{X\left[\alpha \right]}$: The $\alpha$-adjunct of $P_X$, defined in (152). The $\langle \alpha \rangle$-response to $Q_{X\left[\alpha \right]}$ is $P_{Y\left[\alpha \right]}$. Furthermore, $Q_{X\left[\alpha \right]}$ achieves the minimum in (165).
• $Q_{X\langle \alpha \rangle }$: The $\langle \alpha \rangle$-adjunct of $P_X$, defined in (157). The $\alpha$-response to $Q_{X\langle \alpha \rangle }$ is $P_{Y\langle \alpha \rangle }$. Furthermore, $Q_{X\langle \alpha \rangle }$ achieves the maximum in (169).

6. Maximization of ${\mathit{I}}_{\mathit{\alpha }}(\mathit{X};\mathit{Y})$

Just like the maximization of mutual information with respect to the input distribution yields the channel capacity (of course, subject to conditions [57]), the maximization of $I_{\alpha }(X;Y)$ and of $I_{\alpha }^{\mathtt{c}}(X;Y)$ arises in the analysis of error exponents, as we will see in Section 10. A recent in-depth treatment of the maximization of $\alpha$-mutual information is given in [45]. As we see most clearly in (82) for the discrete case, when it comes to its optimization, one advantage of $I_{\alpha }(X;Y)$ over $I(X;Y)$ is that the input distribution does not affect the expression through its influence on the output distribution.

48.

The maximization of $\alpha$-mutual information is facilitated by the following result.

Theorem 14 ([45]).

Given $\alpha \in (0,1)\cup (1,\infty )$; a random transformation $P_{Y|X}:\mathcal{A}\to \mathcal{B}$; and, a convex set $\mathcal{P}\subset {\mathcal{P}}_{\mathcal{A}}$, the following are equivalent.

(a) 

$P_X^{*}\in \mathcal{P}$ attains the maximal α-mutual information on $\mathcal{P}$,

$$\begin{array}{c}\hfill I_{\alpha }(P_X^{*},P_{Y|X})=\underset{P\in \mathcal{P}}{max}{I}_{\alpha }(P,P_{Y|X})<\infty .\end{array}$$

(177)

(b) 

For any $P_X\in \mathcal{P}$, and any output distribution $Q_Y\in {\mathcal{P}}_{\mathcal{B}}$,

$$\begin{array}{cc}\hfill D_{\alpha }(P_{Y|X}\parallel P_{Y\left[\alpha \right]}^{*}\left|P_X\right)& \le D_{\alpha }(P_{Y|X}\parallel P_{Y\left[\alpha \right]}^{*}\left|P_X^{*}\right)\hfill \end{array}$$

(178)

$$\begin{array}{cc}\hfill & \le D_{\alpha }(P_{Y|X}\parallel Q_Y\left|P_X^{*}\right),\hfill \end{array}$$

(179)

where $P_{Y\left[\alpha \right]}^{*}$ is the α-response to $P_X^{*}$.

Moreover, if $P_{Y\left[\alpha \right]}$ denotes the α-response to $P_X$, then

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y\left[\alpha \right]}\parallel P_{Y\left[\alpha \right]}^{*})\le I_{\alpha }(P_X^{*},P_{Y|X})-I_{\alpha }(P_X,P_{Y|X})<\infty .\end{array}$$

(180)

Note that, while $I_{\alpha }(·,P_{Y|X})$ may not be maximized by a unique (or, in fact, by any) input distribution, the resulting $\alpha$-response $P_{Y\left[\alpha \right]}^{*}$ is indeed unique. If $\mathcal{P}$ is such that none of its elements attain the maximal $I_{\alpha }$, it is known [42,45] that the $\alpha$-response to any asymptotically optimal sequence of input distributions converges to $P_{Y\left[\alpha \right]}^{*}$. This is the counterpart of a result by Kemperman [58] concerning mutual information.

49.

The following example appears in [45].

Example 11.

Let $Y=X+N$ where $N\sim \mathcal{N}\left(0,{\sigma }_N^2\right)$ independent of X. Fix $\alpha \in (0,1)$ and $P>0$. Suppose that the set, $\mathcal{P}\subset {\mathcal{P}}_{\mathcal{A}}$, of allowable input probability measures consists of those that satisfy the constraint

$$\begin{array}{c}\hfill \mathbb{E}\left[{exp}_{\mathrm{e}}\left(-\frac{\alpha (1-\alpha )X^2}{2\left({\alpha }^{2}P+{\sigma }_N^2\right)}\right)\right]\ge \sqrt{\frac{{\alpha }^{2}P+{\sigma }_N^2}{\alpha P+{\sigma }_N^2}}.\end{array}$$

(181)

We can readily check that $X^{*}\sim \mathcal{N}\left(0,P\right)$ satisfies (181) with equality, and as we saw in Example 2, its α-response is $P_{Y\left[\alpha \right]}^{*}=\mathcal{N}(0,\alpha P+{\sigma }^2)$. Theorem 14 establishes that $P_X^{*}$ does indeed maximize the α-mutual information among all the distributions in $\mathcal{P}$, yielding (recall Example 6)

$$\begin{array}{c}\hfill \underset{P_X\in \mathcal{P}}{max}{I}_{\alpha }(X;Y)=\frac{1}{2}log\left(1+\frac{\alpha P}{{\sigma }^2}\right).\end{array}$$

(182)

Curiously, if, instead of $\mathcal{P}$ defined by the constraint (181), we consider the more conventional $\mathcal{P}=\{X:\mathbb{E}\left[X^2\right]\le P\}$, then the left side of (182) is unknown at present. Numerical evidence shows that it can exceed the right side by employing non-Gaussian inputs.

50.

Recalling (56) and (178) implies that if $P_X^{*}$ attains the finite maximal unconstrained $\alpha$-mutual information and its $\alpha$-response is denoted by $P_{Y\left[\alpha \right]}^{*}$, then,

$$\begin{array}{c}\hfill \underset{X}{max}{I}_{\alpha }(X;Y)=\underset{P\in \mathcal{P}}{max}{I}_{\alpha }(P,P_{Y|X})=\underset{a\in \mathcal{A}}{max}{D}_{\alpha }(P_{Y|X=a}\parallel P_{Y\left[\alpha \right]}^{*}),\end{array}$$

(183)

which requires that $P_X^{*}\left({\mathcal{A}}_{\alpha }^{*}\right)=1$, with

$$\begin{array}{c}\hfill {\mathcal{A}}_{\alpha }^{*}=\left\{x\in \mathcal{A}:D_{\alpha }(P_{Y|X=x}\parallel P_{Y\left[\alpha \right]}^{*})=\underset{a\in \mathcal{A}}{max}{D}_{\alpha }(P_{Y|X=a}\parallel P_{Y\left[\alpha \right]}^{*})\right\}.\end{array}$$

(184)

For discrete alphabets, this requires that if $x\notin {\mathcal{A}}_{\alpha }^{*}$, then $P_X^{*}\left(x\right)=0$, which is tantamount to

$$\begin{array}{cc}\hfill \sum _{y\in \mathcal{B}}{P}_{Y|X}^{\alpha }\left(y\right|x){\mathbb{E}}^{\frac{1-\alpha }{\alpha }}\left[P_{Y|X}^{\alpha }\left(y\right|X^{*})\right]& \ge exp\left(\frac{\alpha -1}{\alpha }{I}_{\alpha }(X^{*};Y^{*})\right),\hfill \end{array}$$

(185)

with equality for all $x\in \mathcal{A}$ such that $P_X^{*}\left(x\right)>0$. For finite-alphabet random transformations this observation is equivalent to Theorem 5.6.5 in [9].

51.

Getting slightly ahead of ourselves, we note that, in view of (128), an important consequence of Theorem 15 below, is that, as anticipated in Item 25, the unconstrained maximization of $I_{\alpha }(X;Y)$ for $\alpha \in (0,1)$ can be expressed in terms of the solution to an optimization problem involving only conventional mutual information and conditional relative entropy. For $\rho \ge 0$,

$$\begin{array}{c}\hfill \rho \underset{X}{sup}{I}_{\frac{1}{1+\rho }}(X;Y)=\underset{X}{sup}\underset{Q_{Y|X}:\mathcal{A}\to \mathcal{B}}{min}\left\{D(Q_{Y|X}\parallel P_{Y|X}\left|P_X\right)+\rho I(P_X,Q_{Y|X})\right\}.\end{array}$$

(186)

7. Unconstrained Maximization of ${\mathit{I}}_{\mathit{\alpha }}^{\mathtt{c}}(\mathit{X};\mathit{Y})$

52.

In view of the fact that it is much easier to determine the $\alpha$-mutual information than the order-$\alpha$ Augustin–Csiszár information, it would be advantageous to show that the unconstrained maximum of $I_{\alpha }^{\mathtt{c}}(X;Y)$ equals the unconstrained maximum of $I_{\alpha }(X;Y)$. In the finite-alphabet setting, in which it is possible to invoke a "minisup” theorem (e.g., see Section 7.1.7 of [59]), Csiszár [32] showed this result for $\alpha >0$. The assumption of finite output alphabets was dropped in Theorem 1 of [42], and further generalized in Theorem 3 of the same reference. As we see next, for $\alpha \in (0,1)$, it is possible to give an elementary proof without restrictions on the alphabets.

Theorem 15.

Let $\alpha \in (0,1)$. If the suprema are over ${\mathcal{P}}_{\mathcal{A}}$, the set of all probability measures defined on the input space, then

$$\begin{array}{c}\hfill \underset{X}{sup}{I}_{\alpha }^{\mathtt{c}}(X;Y)=\underset{X}{sup}{I}_{\alpha }(X;Y).\end{array}$$

(187)

Proof. In view of (143), ≥ holds in (187). To show ≤, we assume ${sup}_X{I}_{\alpha }(X;Y)<\infty$ as, otherwise, there is nothing left to prove. The unconstrained maximization identity in (183) implies

$$\begin{array}{cc}\hfill \underset{X}{sup}{I}_{\alpha }(X;Y)& =\underset{a\in \mathcal{A}}{sup}{D}_{\alpha }(P_{Y|X=a}\parallel P_{Y\left[\alpha \right]}^{*})\hfill \end{array}$$

(188)

$$\begin{array}{cc}\hfill & =\underset{P_X\in \mathcal{P}}{sup}\mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X)\parallel P_{Y\left[\alpha \right]}^{*})\right]\hfill \end{array}$$

(189)

$$\begin{array}{cc}\hfill & \ge \underset{Q\in \mathcal{Q}}{inf}\underset{P_X\in \mathcal{P}}{sup}\mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X)\parallel Q)\right]\hfill \end{array}$$

(190)

$$\begin{array}{cc}\hfill & \ge \underset{P_X\in \mathcal{P}}{sup}\underset{Q\in \mathcal{Q}}{inf}\mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X)\parallel Q)\right]\hfill \end{array}$$

(191)

$$\begin{array}{cc}\hfill & =\underset{X}{sup}{I}_{\alpha }^{\mathtt{c}}(X;Y),\hfill \end{array}$$

(192)

where $P_{Y\left[\alpha \right]}^{*}$ is the unique $\alpha$-response to any input that achieves the maximal $\alpha$-mutual information, and if there is no such input, it is the limit of the $\alpha$-responses to any asymptotically optimal input sequence (Item 48). □

Furthermore, if $\left\{X_n\right\}$ is asymptotically optimal for $I_{\alpha }$, i.e., ${lim}_{n\to \infty }{I}_{\alpha }(X_n;Y_n)={sup}_X{I}_{\alpha }(X;Y)$, then $\left\{X_n\right\}$ is also asymptotically optimal for $I_{\alpha }^{\mathtt{c}}$ because for any $\delta >0$, we can find N, such that for all $n>N$,

$$\begin{array}{cc}\hfill I_{\alpha }(X_n;Y_n)+\delta & \ge \underset{a\in \mathcal{A}}{sup}{D}_{\alpha }(P_{Y|X=a}\parallel P_{Y\left[\alpha \right]}^{*})\hfill \end{array}$$

(193)

$$\begin{array}{cc}\hfill & \ge \mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X_n)\parallel P_{Y\left[\alpha \right]}^{*})\right]\hfill \end{array}$$

(194)

$$\begin{array}{cc}\hfill & \ge I_{\alpha }^{\mathtt{c}}(X_n;Y_n)\hfill \end{array}$$

(195)

$$\begin{array}{cc}\hfill & \ge I_{\alpha }(X_n;Y_n).\hfill \end{array}$$

(196)

8. Maximization of ${\mathit{I}}_{\mathit{\alpha }}^{\mathtt{c}}(\mathit{X};\mathit{Y})$ Subject to Average Cost Constraints

This section is at the heart of the relevance of Rényi information measures to error exponent functions.

53.

Given $\alpha \in (0,1)$, $P_{Y|X}:\mathcal{A}\to \mathcal{B}$, a cost function $\mathsf{b}:\mathcal{A}\to [0,\infty )$ and real scalar $\theta \ge 0$, the objective is to maximize the Augustin–Csiszár mutual information allowing only those probability measures that satisfy $\mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta$, namely,

$${\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)=\underset{\begin{array}{c}{P}_X:\\ \mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta \end{array}}{sup}{I}_{\alpha }^{\mathtt{c}}(P_X,P_{Y|X}).$$

(197)

Unfortunately, identity (187) no longer holds when the maximizations over the input probability measure are cost-constrained, and, in general, we can only claim

$${\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)\ge \underset{\begin{array}{c}{P}_X:\\ \mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta \end{array}}{sup}{I}_{\alpha }(P_X,P_{Y|X}).$$

(198)

A conceptually simple approach to solve for ${\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)$ is to

(a)

postulate an input probability measure $P_X^{*}$ that achieves the supremum in (197);

(b)

solve for its $\langle \alpha \rangle$-response $P_Y^{*}$ using (92);

(c)

show that $(P_X^{*},P_Y^{*})$ is a saddle point for the game with payoff function

$$\begin{array}{c}\hfill B(P_X,Q_Y)=\int D_{\alpha }\left(P_{Y|X=x}\parallel Q_Y\right)\mathrm{d}{P}_X,\end{array}$$

(199)

where $Q_Y\in {\mathcal{P}}_{\mathcal{A}}$ and $P_X$ is chosen from the convex subset of ${\mathcal{P}}_{\mathcal{A}}$ of probability measures which satisfy $\mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta$.

Since $P_Y^{*}$ is already known, by definition, to be the $\langle \alpha \rangle$-response to $P_X^{*}$, verifying the saddle point is tantamount to showing that $B(P_X,P_Y^{*})$ is maximized by $P_X^{*}$ among $\{P_X\in {\mathcal{P}}_{\mathcal{A}}:\mathbb{E}\left[\mathsf{b}\left(X\right)\right]\le \theta \}$. Theorem 1 of [43] guarantees the existence of a saddle point in the case of finite input alphabets. In addition to the fact that it is not always easy to guess the optimum input $P_X^{*}$ (see e.g., Section 12), the main stumbling block is the difficulty in determining the $\langle \alpha \rangle$-response to any candidate input distribution, although sometimes this is indeed feasible as we saw in Example 7.

54.

Naturally, Theorem 15 implies

$$\begin{array}{c}\hfill {\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)\le \underset{X}{sup}{I}_{\alpha }(X;Y).\end{array}$$

(200)

If the unconstrained maximization of $I_{\alpha }^{\mathtt{c}}(·,P_{Y|X})$ is achieved by an input distribution $X^{☆}$ that satisfies $\mathbb{E}\left[\mathsf{b}\left(X^{☆}\right)\right]\le \theta$, then equality holds in (200), which, in turn, is equal to $I_{\alpha }^{\mathtt{c}}(P_X^{☆},P_{Y|X})$. In that case, the average cost constraint is said to be inactive. For most cost functions and random transformations of practical interest, the cost constraint is active for all $\theta >0$. To ascertain whether it is, we simply verify whether there exists an input achieving the right side of (200), which happens to satisfy the constraint. If so, ${\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)$ has been found. The same holds if we can find a sequence $\left\{X_n\right\}$ such that $\mathbb{E}\left[\mathsf{b}\left(X_n\right)\right]\le \theta$ and $I_{\alpha }(X_n;Y_n)\to {sup}_X{I}_{\alpha }(X;Y)$. Otherwise, we proceed with the method described below. Thus, henceforth, we assume that the cost constraint is active.

55.

The approach proposed in this paper to solve for ${\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)$ for $\alpha \in (0,1)$ hinges on the variational representation in (172), which allows us to sidestep having to find any $\langle \alpha \rangle$-response. Note that once we set out to maximize $I_{\alpha }^{\mathtt{c}}(P_X,P_{Y|X})$ over $\mathcal{P}=\{P_X\in {\mathcal{P}}_{\mathcal{A}}:\mathbb{E}\left[\mathsf{b}\left(X\right)\right]\le \theta \}$, the allowable $Q_X$ in the maximization in (175) range over a $\xi$-blow-up of $\mathcal{P}$ defined by

$$\begin{array}{c}\hfill {\mathsf{\Gamma }}_{\xi }\left(\mathcal{P}\right)=\left\{Q_X\in {\mathcal{P}}_{\mathcal{A}}:\exists P_X\in \mathcal{P},suchthatD(P_X\parallel Q_X)\le \xi \right\}.\end{array}$$

(201)

As we show in Item 56, we can accomplish such an optimization by solving an unconstrained maximization of the sum of $\alpha$-mutual information and a term suitably derived from the cost function.

56.

It will not be necessary to solve for (176), as our goal is to further maximize (172) over $P_X$ subject to an average cost constraint. The Lagrangian corresponding to the constrained optimization in (197) is

$$\begin{array}{c}\hfill {\mathbb{L}}_{\alpha }(\nu ,P_X)=I_{\alpha }^{\mathtt{c}}(X;Y)-\nu \mathbb{E}\left[\mathsf{b}\left(X\right)\right]+\nu \theta ,\end{array}$$

(202)

where on the left side we have omitted, for brevity, the dependence on $\theta$ stemming from the last term on the right side. The Lagrange multiplier method (e.g., [60]) implies that if $X^{*}$ achieves the supremum in (197), then there exists ${\nu }^{*}\ge 0$ such that for all $P_X$ on $\mathcal{A}$ and $\nu \ge 0$,

$$\begin{array}{c}\hfill {\mathbb{L}}_{\alpha }({\nu }^{*},P_X)\le {\mathbb{L}}_{\alpha }({\nu }^{*},P_X^{*})\le {\mathbb{L}}_{\alpha }(\nu ,P_X^{*}).\end{array}$$

(203)

Note from (202) that the right inequality in (203) can only be achieved if

$$\begin{array}{c}\hfill \mathbb{E}\left[\mathsf{b}\left(X^{*}\right)\right]=\theta ,\end{array}$$

(204)

and, consequently,

$$\begin{array}{c}\hfill {\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)={\mathbb{L}}_{\alpha }({\nu }^{*},P_X^{*})=\underset{\nu \ge 0}{min}\underset{P_X}{max}{\mathbb{L}}_{\alpha }(\nu ,P_X)=\underset{P_X}{max}\underset{\nu \ge 0}{min}{\mathbb{L}}_{\alpha }(\nu ,P_X).\end{array}$$

(205)

The pivotal result enabling us to obtain ${\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)$ without the need to deal with Augustin–Csiszár mutual information is the following.

Theorem 16.

Given $\alpha \in (0,1)$, $\nu \ge 0$, $P_{Y|X}:\mathcal{A}\to \mathcal{B}$, and $\mathsf{b}:\mathcal{A}\to [0,\infty )$, denote the function

$$\begin{array}{c}\hfill {\mathbb{A}}_{\alpha }\left(\nu \right)=\underset{X}{max}\left\{I_{\alpha }(X;Y)+\frac{1}{1-\alpha }log\mathbb{E}\left[exp\left(-(1-\alpha )\nu \mathsf{b}\left(X\right)\right)\right]\right\}.\end{array}$$

(206)

Then,

$$\begin{array}{c}\hfill \underset{P_X\in {\mathcal{P}}_{\mathcal{A}}}{sup}{\mathbb{L}}_{\alpha }(\nu ,P_X)=\nu \theta +{\mathbb{A}}_{\alpha }\left(\nu \right),\end{array}$$

(207)

and

$$\begin{array}{c}\hfill {\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)=\underset{\nu \ge 0}{min}\left\{\nu \theta +{\mathbb{A}}_{\alpha }\left(\nu \right)\right\}.\end{array}$$

(208)

Proof. 

Plugging (172) into (197) we obtain, with $X\sim P_X$, and $\widehat{X}\sim Q_X$,

$$\begin{array}{cc}\hfill & \underset{P_X\in {\mathcal{P}}_{\mathcal{A}}}{sup}{\mathbb{L}}_{\alpha }(\nu ,P_X)=\underset{P_X}{sup}\left\{I_{\alpha }^{\mathtt{c}}(X;Y)-\nu \mathbb{E}\left[\mathsf{b}\left(X\right)\right]+\nu \theta \right\}\hfill \end{array}$$

(209)

$$\begin{array}{cc}\hfill & =\underset{P_X\in {\mathcal{P}}_{\mathcal{A}}}{sup}\left\{\underset{Q_X\in {\mathcal{P}}_{\mathcal{A}}}{max}\left\{I_{\alpha }(Q_X,P_{Y|X})-\frac{1}{1-\alpha }D(P_X\parallel Q_X)\right\}-\nu \mathbb{E}\left[\mathsf{b}\left(X\right)\right]+\nu \theta \right\}\hfill \end{array}$$

(210)

$$\begin{array}{cc}\hfill & =\nu \theta +\underset{Q_X\in {\mathcal{P}}_{\mathcal{A}}}{max}\left\{I_{\alpha }(Q_X,P_{Y|X})-\frac{1}{1-\alpha }\underset{P_X}{inf}\left\{D(P_X\parallel Q_X)+\nu (1-\alpha )\mathbb{E}\left[\mathsf{b}\left(X\right)\right]\right\}\right\}\hfill \end{array}$$

(211)

$$\begin{array}{cc}\hfill & =\nu \theta +\underset{Q_X\in {\mathcal{P}}_{\mathcal{A}}}{max}\left\{I_{\alpha }(Q_X,P_{Y|X})+\frac{1}{1-\alpha }log\mathbb{E}\left[exp\left(-\nu (1-\alpha )\mathsf{b}\left(\widehat{X}\right)\right)\right]\right\}\hfill \end{array}$$

(212)

$$\begin{array}{cc}\hfill & =\nu \theta +{\mathbb{A}}_{\alpha }\left(\nu \right),\hfill \end{array}$$

(213)

where (209) and (213) follow from (202) and (206), respectively, and (212) follows by invoking Theorem 1 with $Z\sim Q_X$ and

$$\begin{array}{c}\hfill g\left(a\right)=(1-\alpha )\nu \mathsf{b}\left(a\right),\end{array}$$

(214)

which is nonnegative since $\alpha \in (0,1)$ and $\nu >0$. Finally, (208) follows from (205) and (207). □

In conclusion, we have shown that the maximization of Augustin–Csiszár mutual information of order $\alpha$ subject to $\mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta$ boils down to the unconstrained maximization of a Lagrangian consisting of the sum of $\alpha$-mutual information and an exponential average of the cost function. Circumventing the need to deal with $\langle \alpha \rangle$-responses and with Augustin–Csiszár mutual information of order $\alpha$ leads to a particularly simple optimization, as illustrated in Section 11 and Section 12.

57.

Theorem 16 solves for the maximal Augustin–Csiszár mutual information of order $\alpha$ under an average cost constraint without having to find out the input probability measure $P_X^{*}$ that attains it nor its $\langle \alpha \rangle$-response $P_Y^{*}$ (using the notation in Item 53). Instead, it gives the solution as

$$\begin{array}{c}\hfill {\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)=\underset{\nu \ge 0}{min}\left\{\nu \theta +\underset{X}{max}\left\{I_{\alpha }(X;Y)+\frac{1}{1-\alpha }log\mathbb{E}\left[exp\left(-(1-\alpha )\nu \mathsf{b}\left(X\right)\right)\right]\right\}\right\}.\end{array}$$

(215)

Although we are not going to invoke a minimax theorem, with the aid of Theorem 9-(b) we can see that the functional within the inner brackets is concave in $P_X$; Furthermore, if $V\in (0,1]$, then $log\mathbb{E}\left[V^{\nu }\right]$ is easily seen to be convex in $\nu$ with the aid of the Cauchy-Schwarz inequality. Before we characterize the saddle point $({\nu }^{*},Q_X^{*})$ of the game in (215) we note that $(P_X^{*},P_Y^{*})$ can be readily obtained from $({\nu }^{*},Q_X^{*})$.

Theorem 17.

Fix $\alpha \in (0,1)$. Let ${\nu }^{*}>0$ denote the minimizer on the right side of (215), and $Q_X^{*}$ the input probability measure that attains the maximum in (206) (or (215)) for $\nu ={\nu }^{*}$. Then,

(a)

$Q_X^{*}$ is the $\langle \alpha \rangle$-adjunct of $P_X^{*}$.

(b)

$P_Y^{*}=Q_{Y\left[\alpha \right]}^{*}$, the α-response to $Q_X^{*}$.

(c)

$P_X^{*}\ll \gg Q_X^{*}$ with

$$\begin{array}{c}\hfill {\mathit{ı}}_{P_X^{*}\parallel Q_X^{*}}\left(a\right)=-(1-\alpha ){\nu }^{*}\mathsf{b}\left(a\right)+{\tau }_{\alpha },a\in \mathcal{A},\end{array}$$

(216)

where ${\tau }_{\alpha }$ is a normalizing constant ensuring that $P_X^{*}$ is a probability measure.

Proof. 

(a)

We had already established in Theorem 13 that the maximum on the right side of (210) is achieved by the $\langle \alpha \rangle$-adjunct of $P_X$. In the special case $\nu ={\nu }^{*}$, such $P_X$ is $P_X^{*}$. Therefore, $Q_X^{*}$, the argument that achieves the maximum in (206) for $\nu ={\nu }^{*}$, is the $\langle \alpha \rangle$-adjunct of $P_X^{*}$.

(b)

According to Theorem 11, the $\alpha$-response to $Q_X^{*}$ is the $\langle \alpha \rangle$-response to $P_X^{*}$, which is $P_Y^{*}$ by definition.

(c)

For $\nu ={\nu }^{*}$, $P_X^{*}$ achieves the supremum in (209) and the infimum in (211). Therefore, (216) follows from Theorem 1 with $Z\sim Q_X^{*}$ and $g(·)$ given by (214) particularized to $\nu ={\nu }^{*}$.

□

The saddle point of (215) admits the following characterization.

Theorem 18.

If $\alpha \in (0,1)$, the saddle point $({\nu }^{*},Q_X^{*})$ of (215) satisfies

$$\begin{array}{cc}\hfill \mathbb{E}\left[\mathsf{b}\left({\overline{X}}^{*}\right)exp\left(-(1-\alpha ){\nu }^{*}\mathsf{b}\left({\overline{X}}^{*}\right)\right)\right]& =\theta \mathbb{E}\left[exp\left(-(1-\alpha ){\nu }^{*}\mathsf{b}\left({\overline{X}}^{*}\right)\right)\right],{\overline{X}}^{*}\sim Q_X^{*};\hfill \end{array}$$

(217)

$$\begin{array}{cc}\hfill D_{\alpha }\left(P_{Y|X=a}\parallel Q_{Y\left[\alpha \right]}^{*}\right)& ={\nu }^{*}\mathsf{b}\left(a\right)+c_{\alpha }\left({\nu }^{*}\right),a\in \mathcal{A},\hfill \end{array}$$

(218)

where $Q_{Y\left[\alpha \right]}^{*}$ is the α-response to $Q_X^{*}$, and $c_{\alpha }\left({\nu }^{*}\right)$ does not depend on $a\in \mathcal{A}$. Furthermore,

$$\begin{array}{cc}\hfill {\mathbb{A}}_{\alpha }\left({\nu }^{*}\right)& =c_{\alpha }\left({\nu }^{*}\right),\hfill \end{array}$$

(219)

$$\begin{array}{cc}\hfill {\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)& ={\nu }^{*}\theta +c_{\alpha }\left({\nu }^{*}\right).\hfill \end{array}$$

(220)

Proof. First, we show that the scalar ${\nu }^{*}\ge 0$ that minimizes

$$\begin{array}{c}\hfill f\left(\nu \right)=\nu \theta +I_{\alpha }(Q_X^{*},P_{Y|X})+\frac{1}{1-\alpha }log\mathbb{E}\left[exp\left(-(1-\alpha )\nu \mathsf{b}\left({\overline{X}}^{*}\right)\right)\right]\end{array}$$

(221)

satisfies (217). If we abbreviate $V=exp\left(-(1-\alpha )\mathsf{b}\left({\overline{X}}^{*}\right)\right)\in (0,1]$, then the dominated convergence theorem results in

$$\begin{array}{c}\hfill \frac{\mathrm{d}}{\mathrm{d}\nu }\left\{\nu \theta +\frac{1}{1-\alpha }log\mathbb{E}\left[V^{\nu }\right]\right\}=\theta +\frac{1}{1-\alpha }\frac{\mathbb{E}\left[V^{\nu }logV\right]}{\mathbb{E}\left[V^{\nu }\right]}.\end{array}$$

(222)

Therefore, (217) is equivalent to $\dot{f}\left({\nu }^{*}\right)=0$, which is all we need on account of the convexity of $f(·)$. To show (218), notice that for all $a\in \mathcal{A}$,

$$\begin{array}{cc}\hfill (1-\alpha ){\nu }^{*}\mathsf{b}\left(a\right)-{\tau }_{\alpha }& ={\mathit{ı}}_{Q_X^{*}\parallel P_X^{*}}\left(a\right)\hfill \end{array}$$

(223)

$$\begin{array}{cc}\hfill & =(1-\alpha )D_{\alpha }(P_{Y|X=a}\parallel P_Y^{*})+{\upsilon }_{\alpha },\hfill \end{array}$$

(224)

where (223) is (216) and (224) is (157) with $P_{Y\langle \alpha \rangle }\leftarrow P_Y^{*}$ in view of Theorem 17-(b). In conclusion, (218) holds with

$$\begin{array}{c}\hfill c_{\alpha }\left({\nu }^{*}\right)=\frac{{\upsilon }_{\alpha }+{\tau }_{\alpha }}{\alpha -1}.\end{array}$$

(225)

Finally, (206) implies

$$\begin{array}{cc}\hfill {\mathbb{A}}_{\alpha }\left({\nu }^{*}\right)& =I_{\alpha }(Q_X^{*},P_{Y|X})+\frac{1}{1-\alpha }log\mathbb{E}\left[exp\left(-(1-\alpha ){\nu }^{*}\mathsf{b}\left({\overline{X}}^{*}\right)\right)\right]\hfill \\ \hfill & =\frac{1}{\alpha -1}log\mathbb{E}\left[exp\left((\alpha -1)D_{\alpha }\left(P_{Y|X}(·|{\overline{X}}^{*})\parallel P_Y^{*}\right)\right)\right]\hfill \end{array}$$

(226)

$$\begin{array}{cc}\hfill & +\frac{1}{1-\alpha }log\mathbb{E}\left[exp\left((\alpha -1){\nu }^{*}\mathsf{b}\left({\overline{X}}^{*}\right)\right)\right]\hfill \\ \hfill & =\frac{1}{\alpha -1}log\mathbb{E}\left[exp\left((\alpha -1)\left({\nu }^{*}\mathsf{b}\left({\overline{X}}^{*}\right)+c_{\alpha }\left({\nu }^{*}\right)\right)\right)\right]\hfill \end{array}$$

(227)

$$\begin{array}{cc}\hfill & +\frac{1}{1-\alpha }log\mathbb{E}\left[exp\left((\alpha -1){\nu }^{*}\mathsf{b}\left({\overline{X}}^{*}\right)\right)\right]\hfill \end{array}$$

(228)

$$\begin{array}{cc}\hfill & =c_{\alpha }\left({\nu }^{*}\right),\hfill \end{array}$$

(229)

where (227) follows from the definition of $\alpha$-mutual information and Theorem 17-(b), and (228) follows from (218). Plugging (219) into (208) results in (220). □

58.

Typically, the application of Theorem 18 involves

(a)

guessing the form of the auxiliary input $Q_X^{*}$ (modulo some unknown parameter),

(b)

obtaining its $\alpha$-response $Q_{Y\left[\alpha \right]}^{*}$, and

(c)

verifying that (217) and (218) are satisfied for some specific choice of the unknown parameter.

With the same approach, we can postulate, for every $\nu \ge 0$, an input distribution $R_X^{\nu }$, whose $\alpha$-response $R_{Y\left[\alpha \right]}^{\nu }$ satisfies

$$\begin{array}{c}\hfill D_{\alpha }\left(P_{Y|X=a}\parallel R_{Y\left[\alpha \right]}^{\nu }\right)=\nu \mathsf{b}\left(a\right)+c_{\alpha }\left(\nu \right),a\in \mathcal{A},\end{array}$$

(230)

where the only condition we place on $c_{\alpha }\left(\nu \right)$ is that it not depend on $a\in \mathcal{A}$. If this is indeed the case, then the same derivation in (226)–(229) results in

$$\begin{array}{c}\hfill {\mathbb{A}}_{\alpha }\left(\nu \right)=c_{\alpha }\left(\nu \right),\end{array}$$

(231)

and we determine ${\nu }^{*}$ as the solution to $\theta =-{\dot{c}}_{\alpha }\left({\nu }^{*}\right)$, in lieu of (217). Section 11 and Section 12 illustrate the effortless nature of this approach to solve for ${\mathbb{A}}_{\alpha }\left(\nu \right)$. Incidentally, (230) can be seen as the $\alpha$-generalization of the condition in Problem 8.2 of [48], elaborated later in [61].

9. Gallager’s ${\mathit{E}}_{\mathbf{0}}$ Functions and the Maximal Augustin–Csiszár Mutual Information

In keeping with Gallager’s setting [9], we stick to discrete alphabets throughout this section.

59.

In his derivation of an achievability result for discrete memoryless channels, Gallager [8] introduced the function (1), which we repeat for convenience,

$$\begin{array}{c}\hfill E_0(\rho ,P_X)=-log\sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}{P}_X\left(x\right)P_{Y|X}^{\frac{1}{1+\rho }}\left(y\right|x)\right)}^{1+\rho }.\end{array}$$

(232)

Comparing (82) and (232), we obtain

$$\begin{array}{c}\hfill E_0(\rho ,P_X)=\rho I_{\frac{1}{1+\rho }}(X;Y),\end{array}$$

(233)

which, as we mentioned in Section 1, is the observation by Csiszár in [30] that triggered the third phase in the representation of error exponents. Popularized in [9], the $E_0$ function was employed by Shannon, Gallager and Berlekamp [10] for $\rho \ge 0$ and by Arimoto [62] for $\rho \in (-1,0)$ in the derivation of converse results in data transmission, the latter of which considers rates above capacity, a region in which error probability increases with blocklength, approaching one at an exponential rate. For the achievability part, [8] showed upper bounds on the error probability involving $E_0(\rho ,P_X)$ for $\rho \in [0,1]$. Therefore, for rates below capacity, the $\alpha$-mutual information only enters the picture for $\alpha \in (0,1)$. One exception in which Rényi divergence of order greater than 1 plays a role at rates below capacity was found by Sason [63], where a refined achievability result is shown for binary linear codes for output symmetric channels (a case in which equiprobable $P_X$ maximizes (233)), as a function of their Hamming weight distribution.

Although Gallager did not have the benefit of the insight provided by the Rényi information measures, he did notice certain behaviors of $E_0$ reminiscent of mutual information. For example, the derivative of (233) with respect to $\rho$, at $\rho \leftarrow 0$ is equal to $I(X;Y)$. As pointed out by Csiszár in [32], in the absence of cost constraints, Gallager’s $E_0$ function in (232) satisfies

$$\begin{array}{c}\hfill \underset{P_X}{max}{E}_0(\rho ,P_X)=\rho \underset{X}{max}{I}_{\frac{1}{1+\rho }}(X;Y)=\rho \underset{X}{max}{I}_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y),\end{array}$$

(234)

in view of (233) and (187).

Recall that Gallager’s modified $E_0$ function in the case of cost constraints is

$$\begin{array}{c}\hfill E_0(\rho ,P_X,r,\theta )=-log\sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}{P}_X\left(x\right)exp\left(r\mathsf{b}\left(x\right)-r\theta \right)P_{Y|X}^{\frac{1}{1+\rho }}\left(y\right|x)\right)}^{1+\rho },\end{array}$$

(235)

which, like (232) he introduced in order to show an achievability result. Up until now, no counterpart to (234) has been found with cost constraints and (235). This is accomplished in the remainder of this section.

60.

In the finite alphabet case the following result is useful to obtain a numerical solution for the functional in (206). More importantly, it is relevant to the discussion in Item 61.

Theorem 19.

In the special case of discrete alphabets, the function in (206) is equal to

$$\begin{array}{c}\hfill {\mathbb{A}}_{\alpha }\left(\nu \right)=\underset{G}{max}\frac{\alpha }{\alpha -1}log\sum _{y\in \mathcal{B}}{\left(\sum _{a\in \mathcal{A}}G\left(a\right)P_{Y|X}^{\alpha }\left(y\right|a)\right)}^{\frac{1}{\alpha }},\end{array}$$

(236)

where the maximization is over all $G:\mathcal{A}\to [0,\infty )$ such that

$$\begin{array}{c}\hfill \sum _{a\in \mathcal{A}}G\left(a\right)exp\left(-(1-\alpha )\nu \mathsf{b}\left(a\right)\right)=1.\end{array}$$

(237)

Proof. 

Recalling (82) we have

$$\begin{array}{cc}\hfill & I_{\alpha }(X;Y)+\frac{1}{1-\alpha }log\mathbb{E}\left[exp\left(-(1-\alpha )\nu \mathsf{b}\left(X\right)\right)\right]\hfill \\ \hfill & =\frac{\alpha }{\alpha -1}log\sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}{P}_X\left(x\right)P_{Y|X=x}^{\alpha }\left(y\right)\right)}^{\frac{1}{\alpha }}\hfill \end{array}$$

$$\begin{array}{cc}\hfill & +\frac{1}{1-\alpha }log\mathbb{E}\left[exp\left(-(1-\alpha )\nu \mathsf{b}\left(X\right)\right)\right]\hfill \end{array}$$

(238)

$$\begin{array}{cc}\hfill & =\frac{\alpha }{\alpha -1}log\sum _{y\in \mathcal{B}}{\left(\frac{\mathbb{E}\left[P_{Y|X}^{\alpha }\left(y\right|X)\right]}{\mathbb{E}\left[exp\left(-(1-\alpha )\nu \mathsf{b}\left(X\right)\right)\right]}\right)}^{\frac{1}{\alpha }}\hfill \end{array}$$

(239)

$$\begin{array}{cc}\hfill & =\frac{\alpha }{\alpha -1}log\sum _{y\in \mathcal{B}}{\left(\sum _{a\in \mathcal{A}}G\left(a\right)P_{Y|X}^{\alpha }\left(y\right|a)\right)}^{\frac{1}{\alpha }},\hfill \end{array}$$

(240)

where

$$\begin{array}{c}\hfill G\left(x\right)=\frac{P_X\left(x\right)}{{\sum }_{a\in \mathcal{A}}{P}_X\left(a\right)exp\left(-(1-\alpha )\nu \mathsf{b}\left(a\right)\right)}.\end{array}$$

(241)

□

61.

We can now proceed to close the circle between the maximization of Augustin–Csiszár mutual information subject to average cost constraints (Phase 3 in Section 1) and Gallager’s approach (Phase 1 in Section 1).

Theorem 20.

In the discrete alphabet case, recalling the definitions in (202) and (235), for $\rho >0$,

$$\begin{array}{cc}\hfill \underset{P_X}{max}{E}_0(\rho ,P_X,r,\theta )& =\rho \underset{P_X}{max}{\mathbb{L}}_{\frac{1}{1+\rho }}\left(r+\frac{r}{\rho },P_X\right),r>0;\hfill \end{array}$$

(242)

$$\begin{array}{cc}\hfill \underset{r\ge 0}{min}\underset{P_X}{max}{E}_0(\rho ,P_X,r,\theta )& =\rho {\mathbb{C}}_{\frac{1}{1+\rho }}^{\mathtt{c}}\left(\theta \right),\hfill \end{array}$$

(243)

where the maximizations are over ${\mathcal{P}}_{\mathcal{A}}$.

Proof. 

With

$$\begin{array}{c}\hfill \alpha =\frac{1}{1+\rho }and\nu =r\frac{1+\rho }{\rho }=\frac{r}{1-\alpha },\end{array}$$

(244)

the maximization of (235) with the respect to the input probability measure yields

$$\begin{array}{cc}\hfill & \underset{P_X}{max}{E}_0(\rho ,P_X,r,\theta )\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =\underset{P_X}{max}\left\{(1+\rho )r\theta -log\sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}{P}_X\left(x\right)exp\left(r\mathsf{b}\left(x\right)\right)P_{Y|X}^{\frac{1}{1+\rho }}\left(y\right|x)\right)}^{1+\rho }\right\}\hfill \end{array}$$

(245)

$$\begin{array}{cc}\hfill & =\rho \nu \theta +\rho \underset{P_X}{max}\frac{\alpha }{\alpha -1}log\sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}{P}_X\left(x\right)exp\left((1-\alpha )\nu \mathsf{b}\left(x\right)\right)P_{Y|X}^{\alpha }\left(y\right|x)\right)}^{\frac{1}{\alpha }}\hfill \end{array}$$

(246)

$$\begin{array}{cc}\hfill & =\rho \nu \theta +\rho \underset{G}{max}\frac{\alpha }{\alpha -1}log\sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}G\left(x\right)P_{Y|X}^{\alpha }\left(y\right|x)\right)}^{\frac{1}{\alpha }}\hfill \end{array}$$

(247)

$$\begin{array}{cc}\hfill & =\rho \nu \theta +\rho {\mathbb{A}}_{\alpha }\left(\nu \right)\hfill \end{array}$$

(248)

$$\begin{array}{cc}\hfill & =\rho \underset{P_X}{max}{\mathbb{L}}_{\alpha }(\nu ,P_X),\hfill \end{array}$$

(249)

where

• the maximization on the right side of (247) is over all $G:\mathcal{A}\to [0,\infty )$ that satisfy (237), since that constraint is tantamount to enforcing the constraint that $P_X\in {\mathcal{P}}_{\mathcal{A}}$ on the left side of (247);
• (248) ⟸ Theorem 19;
• (249) ⟸ Theorem 16.

The proof of (242) is complete once (244) is invoked to substitute $\alpha$ and $\nu$ from the right side of (249). If we now minimize the outer sides of (245)–(249) with respect to r we obtain, using (205) and (244),

$$\begin{array}{cc}\hfill \underset{r\ge 0}{min}\underset{P_X}{max}{E}_0(\rho ,P_X,r,\theta )& =\rho \underset{r\ge 0}{min}\underset{P_X}{max}{\mathbb{L}}_{\alpha }\left(\frac{r}{1-\alpha },P_X\right)\hfill \end{array}$$

(250)

$$\begin{array}{cc}\hfill & =\rho \underset{\nu \ge 0}{min}\underset{P_X}{max}{\mathbb{L}}_{\alpha }\left(\nu ,P_X\right)\hfill \end{array}$$

(251)

$$\begin{array}{cc}\hfill & =\rho {\mathbb{C}}_{\frac{1}{1+\rho }}^{\mathtt{c}}\left(\theta \right).\hfill \end{array}$$

(252)

□

In p. 329 of [9], Gallager poses the unconstrained maximization (i.e., over $P_X\in {\mathcal{P}}_{\mathcal{A}}$) of the Lagrangian

$$\begin{array}{c}\hfill E_0(\rho ,P_X,r,\theta )+\gamma \sum _{a\in \mathcal{A}}{P}_X\left(a\right)\mathsf{b}\left(a\right)-\gamma \theta .\end{array}$$

(253)

Note the apparent discrepancy between the optimizations in (243) and (253): the latter is parametrized by r and $\gamma$ (in addition to $\rho$ and $\theta$), while the maximization on the right side of (243) does not enforce any average cost constraint. In fact, there is no disparity since Gallager loc. cit. finds serendipitously that $\gamma =0$ regardless of r and $\theta$, and, therefore, just one parameter is enough.

62.

The raison d’être for Augustin’s introduction of $I_{\alpha }^{\mathtt{c}}$ in [36] was his quest to view Gallager’s approach with average cost constraints under the optic of Rényi information measures. Contrasting (232) and (235) and inspired by the fact that, in the absence of cost constraints, (232) satisfies a variational characterization in view of (69) and (233), Augustin [36] dealt, not with (235), but with

$$\underset{Q_Y}{min}{D}_{\alpha }({\tilde{P}}_{Y|X}\parallel Q_Y\left|P_X\right),where{\tilde{P}}_{Y|X=x}=P_{Y|X=x}exp\left(r^{\prime }\mathsf{b}\left(x\right)\right).$$

Assuming finite alphabets, Augustin was able to connect this quantity with the maximal $I_{\alpha }^{\mathtt{c}}(X;Y)$ under cost constraints in an arcane analysis that invokes a minimax theorem. This line of work was continued in Section 5 of [43], which refers to ${min}_{Q_Y}{D}_{\alpha }({\tilde{P}}_{Y|X}\parallel Q_Y\left|P_X\right)$ as the Rényi-Gallager information. Unfortunately, since ${\tilde{P}}_{Y|X}$ is not a random transformation, the conditional pseudo-Rényi divergence $D_{\alpha }({\tilde{P}}_{Y|X}\parallel Q_Y\left|P_X\right)$ need not satisfy the key additive decomposition in Theorem 4 so the approach of [36,43] fails to establish an identity equating the maximization of Gallager’s function (235) with the maximization of Augustin–Csiszár mutual information, which is what we have accomplished through a crisp and elementary analysis.

10. Error Exponent Functions

The central objects of interest in the error exponent analysis of data transmission are the functions $E_{\mathtt{sp}}(R,P_X)$ and $E_{\mathtt{r}}(R,P_X)$ of a random transformation $P_{Y|X}:\mathcal{A}\to \mathcal{B}$. Reflecting the three different phases referred to in Section 1, there is no unanimity in the definition of those functions. Following [48], we adopt the standard canonical Phase 2 (Section 1.2) definitions of those functions, which are given in Items 63 and 67.

63.

If $R\ge 0$ and $P_X\in {\mathcal{P}}_{\mathcal{A}}$, the sphere-packing error exponent function is (e.g., (10.19) of [48])

$$E_{\mathtt{sp}}(R,P_X)=\underset{\begin{array}{c}{Q}_{Y|X}:\mathcal{A}\to \mathcal{B}\\ I(P_X,Q_{Y|X})\le R\end{array}}{min}D(Q_{Y|X}\parallel P_{Y|X}\left|P_X\right).$$

(254)

64.

As a function of $R\ge 0$, the basic properties of (254) for fixed $(P_X,P_{Y|X})$ are as follows.

(a)

If $R\ge I(P_X,P_{Y|X})$, then $E_{\mathtt{sp}}(R,P_X)=0$;

(b)

If $R<I(P_X,P_{Y|X})$, then $E_{\mathtt{sp}}(R,P_X)>0$;

(c)

The infimum of the arguments for which the sphere-packing error exponent function is finite is denoted by $R_{\infty }\left(P_X\right)$;

(d)

On the interval $R\in (R_{\infty }\left(P_X\right),I(P_X,P_{Y|X}))$, $E_{\mathtt{sp}}(R,P_X)$ is convex, strictly decreasing, continuous, and equal to (254) where the constraint is satisfied with equality. This implies that for R belonging to that interval, we can find ${\rho }_R\ge 0$ so that for all $r\ge 0$,

$$\begin{array}{c}\hfill E_{\mathtt{sp}}(r,P_X)\ge E_{\mathtt{sp}}(R,P_X)-{\rho }_{R}r+{\rho }_{R}R.\end{array}$$

(255)

65.

In view of Theorem 8 and its definition in (254), it is not surprising that $E_{\mathtt{sp}}(R,P_X)$ is intimately related to the Augustin–Csiszár mutual information, through the following key identity.

Theorem 21.

$$\begin{array}{cc}\hfill E_{\mathtt{sp}}(R,P_X)& =\underset{\rho \ge 0}{sup}\left\{\rho I_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y)-\rho R\right\},R\ge 0;\hfill \end{array}$$

(256)

$$\begin{array}{cc}\hfill R_{\infty }\left(P_X\right)& =I_0^{\mathtt{c}}(X;Y).\hfill \end{array}$$

(257)

Proof. 

First note that ≥ holds in (256) because from (128) we obtain, for all $\rho \ge 0$,

$$\begin{array}{cc}\hfill & \rho I_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y)=\underset{Q_{Y|X}}{min}\left\{D(Q_{Y|X}\parallel P_{Y|X}\left|P_X\right)+\rho I(P_X,Q_{Y|X})\right\}\hfill \end{array}$$

(258)

$$\le \underset{\begin{array}{c}{Q}_{Y|X}:\\ I(P_X,Q_{Y|X})\le R\end{array}}{min}\left\{D(Q_{Y|X}\parallel P_{Y|X}\left|P_X\right)+\rho I(P_X,Q_{Y|X})\right\}$$

(259)

$$\begin{array}{cc}\hfill & \le E_{\mathtt{sp}}(R,P_X)+\rho R,\hfill \end{array}$$

(260)

where (260) follows from the definition in (254). To show ≤ in (256) for those R such that $0<E_{\mathtt{sp}}(R,P_X)<\infty$, Property (d) in Item 64 allows us to write

$$\begin{array}{cc}\hfill \underset{Q_{Y|X}}{min}\left\{D(Q_{Y|X}\parallel P_{Y|X}\left|P_X\right)+{\rho }_{R}I(P_X,Q_{Y|X})\right\}& =\underset{r\ge 0}{min}\left\{E_{\mathtt{sp}}(r,P_X)+{\rho }_{R}r\right\}\hfill \end{array}$$

(261)

$$\begin{array}{cc}\hfill & \ge E_{\mathtt{sp}}(R,P_X)+{\rho }_{R}R,\hfill \end{array}$$

(262)

where (262) follows from (255).

To determine the region where the sphere-packing error exponent is infinite and show (257), first note that if $R<I_0^{\mathtt{c}}(X;Y)={lim}_{\alpha \downarrow 0}{I}_{\alpha }^{\mathtt{c}}(X;Y)$, then $E_{\mathtt{sp}}(R,P_X)=\infty$ because for any $\rho \ge 0$, the function in $\left\{\right\}$ on the right side of (256) satisfies

$$\begin{array}{cc}\hfill \rho I_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y)-\rho R& =\rho I_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y)-\rho I_0^{\mathtt{c}}(X;Y)+\rho I_0^{\mathtt{c}}(X;Y)-\rho R\hfill \end{array}$$

(263)

$$\begin{array}{cc}\hfill & \ge \rho I_0^{\mathtt{c}}(X;Y)-\rho R,\hfill \end{array}$$

(264)

where (264) follows from the monotonicity of $I_{\alpha }^{\mathtt{c}}(X;Y)$ in $\alpha$ we saw in (143). Conversely, if $I_0^{\mathtt{c}}(X;Y)<R<\infty$, there exists $ϵ\in (0,1)$ such that $I_{ϵ}^{\mathtt{c}}(X;Y)<R$, which implies that in the minimization

$$\begin{array}{c}\hfill I_{ϵ}^{\mathtt{c}}(X;Y)=\underset{Q_{Y|X}}{min}\left\{\frac{ϵ}{1-ϵ}D(Q_{Y|X}\parallel P_{Y|X}\left|P_X\right)+I(P_X,Q_{Y|X})\right\}\end{array}$$

(265)

we may restrict to those $Q_{Y|X}$ such that $I(P_X,Q_{Y|X})\le R$, and consequently, $I_{ϵ}^{\mathtt{c}}(X;Y)\ge \frac{ϵ}{1-ϵ}{E}_{\mathtt{sp}}(R,P_X)$. Therefore, to avoid a contradiction, we must have $E_{\mathtt{sp}}(R,P_X)<\infty$.

The remaining case is $I_0^{\mathtt{c}}(X;Y)=\infty$. Again, the monotonicity of the Augustin–Csiszár mutual information implies that $I_{\alpha }^{\mathtt{c}}(X;Y)=\infty$ for all $\alpha >0$. So, (128) prescribes $D(Q_{Y|X}\parallel P_{Y|X}|P_X)=\infty$ for any $Q_{Y|X}$ is such that $I(P_X,Q_{Y|X})<\infty$. Therefore, $E_{\mathtt{sp}}(R,P_X)=\infty$ for all $R\ge 0$, as we wanted to show. □

Augustin [36] provided lower bounds on error probability for codes of type $P_X$ as a function of $I_{\alpha }^{\mathtt{c}}(X;Y)$ but did not state (256); neither did Csiszár in [32] as he was interested in a non-conventional parametrization (generalized cutoff rates) of the reliability function. As pointed out in p. 5605 of [64], the ingredients for the proof of (256) were already present in the hint of Problem 23 of Section II.5 of [24]. In the discrete case, an exponential lower bound on error probability for codes with constant composition $P_X$ is given as a function of $I_{\frac{1}{1+\rho }}^{\mathtt{c}}(P_X,P_{Y|X})$ in [44,64]. As in [64], Nakiboglu [65] gives (256) as the definition of the sphere-packing function and connects it with (254) in Lemma 3 therein, within the context of discrete input alphabets.

In the discrete case, (257) is well-known (e.g., [66]), and given by (83). As pointed out in [40], ${max}_X{I}_0^{\mathtt{c}}(X;Y)$ is the zero-error capacity with noiseless feedback found by Shannon [67], provided there is at least a pair $(a_1,a_2)\in {\mathcal{A}}^2$ such that $P_{Y|X=a_1}\perp P_{Y|X=a_2}$. Otherwise, the zero-error capacity with feedback is zero.

66.

The critical rate, $R_{\mathtt{c}}\left(P_X\right)$, is defined as the smallest abscissa at which the convex function $E_{\mathtt{sp}}(·,P_X)$ meets its supporting line of slope $-1$. According to (256),

$$\begin{array}{c}\hfill I_{\frac{1}{2}}^{\mathtt{c}}(X;Y)=R_{\mathtt{c}}\left(P_X\right)+E_{\mathtt{sp}}(R_{\mathtt{c}}\left(P_X\right),P_X).\end{array}$$

(266)

67.

If $R\ge 0$ and $P_X\in {\mathcal{P}}_{\mathcal{A}}$, the random-coding exponent function is (e.g., (10.15) of [48])

$$\begin{array}{c}\hfill E_{\mathtt{r}}(R,P_X)=\underset{Q_{Y|X}:\mathcal{A}\to \mathcal{B}}{min}\left\{D(Q_{Y|X}\parallel P_{Y|X}\left|P_X\right)+{[I(P_X,Q_{Y|X})-R]}^{+}\right\},\end{array}$$

(267)

with ${\left[t\right]}^{+}=max\{0,t\}$.

68.

The random-coding error exponent function is determined by the sphere-packing error exponent function through the following relation, illustrated in Figure 1.

Theorem 22.

$$\begin{array}{cc}\hfill E_{\mathtt{r}}(R,P_X)& =\underset{r\ge R}{min}\left\{E_{\mathtt{sp}}(r,P_X)+r-R\right\}\hfill \end{array}$$

(268)

$$=\left\{\begin{array}{cc}0,\hfill & R\ge I(P_X,P_{Y|X});\hfill \\ E_{\mathtt{sp}}(R,P_X),\hfill & R\in [R_{\mathtt{c}}\left(P_X\right),I(P_X,P_{Y|X})];\hfill \\ I_{\frac{1}{2}}^{\mathtt{c}}(X;Y)-R,\hfill & R\in [0,R_{\mathtt{c}}\left(P_X\right)].\hfill \end{array}\right.$$

(269)

$$\begin{array}{cc}\hfill & =\underset{\rho \in [0,1]}{sup}\left\{\rho I_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y)-\rho R\right\}.\hfill \end{array}$$

(270)

Proof. 

Identities (268) and (269) are well-known (e.g., Lemma 10.4 and Corollary 10.4 in [48]). To show (270), note that (256) expresses $E_{\mathtt{sp}}(·,P_X)$ as the supremum of supporting lines parametrized by their slope $-\rho$. By definition of critical rate (for brevity, we do not show explicitly its dependence on $P_X$), if $R\in [R_{\mathtt{c}},I(P_X,P_{Y|X})]$, then $E_{\mathtt{sp}}(R,P_X)$ can be obtained by restricting the optimization in (256) to $\rho \in [0,1]$. In that segment of values of R, $E_{\mathtt{sp}}(R,P_X)=E_r(R,P_X)$ according to (269). Moreover, on the interval $R\in [0,R_{\mathtt{c}}]$, we have

$$\begin{array}{cc}\hfill \underset{\rho \in [0,1]}{max}\left\{\rho I_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y)-\rho R\right\}& =I_{\frac{1}{2}}^{\mathtt{c}}(X;Y)-R\hfill \end{array}$$

(271)

$$\begin{array}{cc}\hfill & =E_{\mathtt{sp}}(R_{\mathtt{c}},P_X)+R_{\mathtt{c}}-R\hfill \end{array}$$

(272)

$$\begin{array}{cc}\hfill & =E_{\mathtt{r}}(R,P_X),\hfill \end{array}$$

(273)

where we have used (266) and (269). □

The first explicit connection between $E_{\mathtt{r}}(R,P_X)$ and the Augustin–Csiszár mutual information was made by Poltyrev [35] although he used a different form for $I_{\alpha }^{\mathtt{c}}(X;Y)$, as we discussed in (29).

69.

The unconstrained maximizations over the input distribution of the sphere-packing and random coding error exponent functions are denoted, respectively, by

$$\begin{array}{cc}\hfill E_{\mathtt{sp}}\left(R\right)& =\underset{P_X}{sup}{E}_{\mathtt{sp}}(R,P_X),\hfill \end{array}$$

(274)

$$\begin{array}{cc}\hfill E_{\mathtt{r}}\left(R\right)& =\underset{P_X}{sup}{E}_{\mathtt{r}}(R,P_X).\hfill \end{array}$$

(275)

Coding theorems [8,9,10,22,48] have shown that when these functions coincide they yield the reliability function (optimum speed at which the error probability vanishes with blocklength) as a function of the rate $R<{max}_{X}I(X;Y)$. The intuition is that, for the most favorable input distribution, errors occur when the channel behaves so atypically that codes of rate R are not reliable. There are many ways in which the channel may exhibit such behavior and they are all unlikely, but the most likely among them is the one that achieves (254).

It follows from (187), (256) and (270) that (274) and (275) can be expressed as

$$\begin{array}{cc}\hfill E_{\mathtt{sp}}\left(R\right)& =\underset{\rho \ge 0}{sup}\left\{\rho \underset{X}{sup}{I}_{\frac{1}{1+\rho }}(X;Y)-\rho R\right\},\hfill \end{array}$$

(276)

$$\begin{array}{cc}\hfill E_{\mathtt{r}}\left(R\right)& =\underset{\rho \in [0,1]}{sup}\left\{\rho \underset{X}{sup}{I}_{\frac{1}{1+\rho }}(X;Y)-\rho R\right\}.\hfill \end{array}$$

(277)

Therefore, we can sidestep working with the Augustin–Csiszár mutual information in the absence of cost constraints.

70.

Shannon [1] showed that, operating at rates below maximal mutual information, it is possible to find codes whose error probability vanishes with blocklength; for the converse, instead of error probability, Shannon measured reliability by the conditional entropy of the message given the channel output. That alternative reliability measure, as well as its generalization to Arimoto-Rényi conditional entropy, is also useful analyzing the average performance over code ensembles. It turns out (see e.g., [28,68]) that, below capacity, those conditional entropies also vanish exponentially fast in much the same way as error probability with bounds that are governed by $E_{\mathtt{sp}}\left(R\right)$ and $E_{\mathtt{r}}\left(R\right)$ thereby lending additional operational significance to those functions.

71.

We now introduce a cost function $\mathsf{b}:\mathcal{A}\to [0,\infty )$ and real scalar $\theta \ge 0$, and reexamine the optimizations in (274) and (275) allowing only those probability measures that satisfy $\mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta .$ With a patent, but unavoidable, abuse of notation we define

$$\begin{array}{cc}\hfill & E_{\mathtt{sp}}(R,\theta )=\underset{\begin{array}{c}{P}_X:\\ \mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta \end{array}}{sup}{E}_{\mathtt{sp}}(R,P_X)\hfill \end{array}$$

(278)

$$\begin{array}{cc}\hfill & =\underset{\rho \ge 0}{sup}\left\{\rho \underset{\begin{array}{c}{P}_X:\\ \mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta \end{array}}{sup}{I}_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y)-\rho R\right\}\hfill \end{array}$$

(279)

$$\begin{array}{cc}\hfill & =\underset{\rho \ge 0}{sup}\left\{\rho {\mathbb{C}}_{\frac{1}{1+\rho }}^{\mathtt{c}}\left(\theta \right)-\rho R\right\}\hfill \end{array}$$

(280)

$$\begin{array}{cc}\hfill & =\underset{\rho \ge 0}{sup}\left\{-\rho R+\rho \underset{\nu \ge 0}{min}\left\{\nu \theta +{\mathbb{A}}_{\frac{1}{1+\rho }}\left(\nu \right)\right\}\right\}\hfill \\ \hfill & =\underset{\rho \ge 0}{sup}\left\{-\rho R+\underset{\nu \ge 0}{min}\left\{\rho \nu \theta \right.\right.\hfill \end{array}$$

(281)

$$\begin{array}{cc}\hfill & \left.\left.+\underset{X}{max}\left\{\rho I_{\frac{1}{1+\rho }}(X;Y)+(1+\rho )log\mathbb{E}\left[exp\left(-\frac{\rho \nu }{1+\rho }\mathsf{b}\left(X\right)\right)\right]\right\}\right\}\right\},\hfill \end{array}$$

(282)

where (279), (281) and (282) follow from (256), (208) and (206), respectively.

72.

In parallel to (278)–(281),

$$\begin{array}{cc}\hfill E_{\mathtt{r}}(R,\theta )& =\underset{\begin{array}{c}{P}_X:\\ \mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta \end{array}}{sup}{E}_{\mathtt{r}}(R,P_X)\hfill \end{array}$$

(283)

$$\begin{array}{cc}\hfill & =\underset{\rho \in [0,1]}{sup}\left\{\rho \underset{\begin{array}{c}{P}_X:\\ \mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta \end{array}}{sup}{I}_{\frac{1}{1+\rho }}^{\mathtt{c}}(X;Y)-\rho R\right\}\hfill \end{array}$$

(284)

$$\begin{array}{cc}\hfill & =\underset{\rho \in [0,1]}{sup}\left\{\rho {\mathbb{C}}_{\frac{1}{1+\rho }}^{\mathtt{c}}\left(\theta \right)-\rho R\right\},\hfill \end{array}$$

(285)

where (284) follows from (270). In particular, if we define the critical rate and the cutoff rate as

$$\begin{array}{cc}\hfill R_{\mathtt{c}}& =\underset{\begin{array}{c}{P}_X:\\ \mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta \end{array}}{sup}{R}_{\mathtt{c}}\left(P_X\right),\hfill \end{array}$$

(286)

$$\begin{array}{cc}\hfill R_0& =\underset{\begin{array}{c}{P}_X:\\ \mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \theta \end{array}}{sup}{I}_{\frac{1}{2}}^{\mathtt{c}}(X;Y),\hfill \end{array}$$

(287)

respectively, then it follows from (270) that

$$\begin{array}{c}\hfill E_{\mathtt{r}}\left(R\right)=R_0-R,R\in [0,R_{\mathtt{c}}].\end{array}$$

(288)

Summarizing, the evaluation of $E_{\mathtt{sp}}(R,\theta )$ and $E_{\mathtt{r}}(R,\theta )$ can be accomplished by the method proposed in Section 8, at the heart of which is the maximization in (206) involving $\alpha$-mutual information instead of Augustin–Csiszár mutual information. In Section 11 and Section 12, we illustrate the evaluation of the error exponent functions with two important additive-noise examples.

11. Additive Independent Gaussian Noise; Input Power Constraint

We illustrate the procedure in Item 58 by taking Example 6 considerably further.

73.

Suppose $\mathcal{A}=\mathcal{B}=\mathbb{R}$, $\mathsf{b}\left(x\right)=x^2$, and $P_{Y|X=a}=\mathcal{N}\left(a,{\sigma }_N^2\right)$. We start by testing whether we can find $R_X^{\nu }\in {\mathcal{P}}_{\mathcal{A}}$ such that its $\alpha$-response satisfies (230). Naturally, it makes sense to try $R_X^{\nu }=\mathcal{N}\left(0,{\sigma }^2\right)$ for some yet to be determined ${\sigma }^2$. As we saw in Example 6, this choice implies that its $\alpha$-response is $R_{Y\left[\alpha \right]}^{\nu }=\mathcal{N}\left(0,\alpha {\sigma }^2+{\sigma }_N^2\right)$. Specializing Example 4, we obtain

$$\begin{array}{cc}\hfill & D_{\alpha }\left(P_{Y|X=x}\parallel R_{Y\left[\alpha \right]}^{\nu }\right)=D_{\alpha }\left(\mathcal{N}\left(x,{\sigma }_N^2\right)\parallel \mathcal{N}\left(0,\alpha {\sigma }^2+{\sigma }_N^2\right)\right)\hfill \end{array}$$

(289)

$$\begin{array}{cc}\hfill & =\frac{1}{2}log\left(1+\frac{\alpha {\sigma }^2}{{\sigma }_N^2}\right)-\frac{1}{2(1-\alpha )}log\left(1+\frac{\alpha (1-\alpha ){\sigma }^2}{{\alpha }^2{\sigma }^2+{\sigma }_N^2}\right)+\frac{1}{2}\frac{\alpha x^2}{{\alpha }^2{\sigma }^2+{\sigma }_N^2}log\mathrm{e}.\hfill \end{array}$$

(290)

Therefore, (230) is indeed satisfied with

$$\begin{array}{cc}\hfill c_{\alpha }\left(\nu \right)& =\frac{1}{2}log\left(1+\frac{\alpha {\sigma }^2}{{\sigma }_N^2}\right)-\frac{1}{2(1-\alpha )}log\left(1+\frac{\alpha (1-\alpha ){\sigma }^2}{{\alpha }^2{\sigma }^2+{\sigma }_N^2}\right),\hfill \end{array}$$

(291)

$$\begin{array}{cc}\hfill \nu & =\frac{1}{2}\frac{\alpha }{{\alpha }^2{\sigma }^2+{\sigma }_N^2}log\mathrm{e},\hfill \end{array}$$

(292)

where (292) follows if we choose the variance of the auxiliary input as

$$\begin{array}{cc}\hfill {\sigma }^2& =\frac{log\mathrm{e}}{2\alpha \nu }-\frac{{\sigma }_N^2}{{\alpha }^2}\hfill \end{array}$$

(293)

$$\begin{array}{cc}\hfill & =\frac{{\sigma }_N^2}{{\alpha }^2}\left(\frac{\alpha }{\lambda }-1\right).\hfill \end{array}$$

(294)

In (294) we have introduced an alternative, more convenient, parametrization for the Lagrange multiplier

$$\begin{array}{c}\hfill \lambda =\frac{2\nu {\sigma }_N^2}{log\mathrm{e}}\in (0,\alpha ).\end{array}$$

(295)

In conclusion, with the choice in (293), $\mathcal{N}\left(0,{\sigma }^2\right)$ attains the maximum in (206), and in view of (231), ${\mathbb{A}}_{\alpha }\left(\nu \right)$ is given by the right side of (291) substituting ${\sigma }^2$ by (293). Therefore, we have

$$\begin{array}{cc}\hfill & \nu \theta +{\mathbb{A}}_{\alpha }\left(\nu \right)=\frac{\lambda }{2}\mathsf{snr}log\mathrm{e}+c_{\alpha }\left(\frac{\lambda log\mathrm{e}}{2{\sigma }_N^2}\right)\hfill \end{array}$$

(296)

$$\begin{array}{cc}\hfill & =\frac{\lambda }{2}\mathsf{snr}log\mathrm{e}+\frac{1}{2}log\left(1+\frac{1}{\lambda }-\frac{1}{\alpha }\right)-\frac{1}{2(1-\alpha )}log\left(\alpha -\lambda (1-\alpha )\right)+\frac{log\alpha }{1-\alpha },\hfill \end{array}$$

(297)

where we denoted $\mathsf{snr}=\frac{\theta }{{\sigma }_N^2}.$

In accordance with Theorem 16 all that remains is to minimize (297) with respect to $\nu$, or equivalently, with respect to $\lambda$. Differentiating (297) with respect to $\lambda$, the minimum is achieved at ${\lambda }^{*}$ satisfying

$$\begin{array}{c}\hfill \mathsf{snr}=\frac{1}{{\lambda }^{*}}\frac{\alpha -{\lambda }^{*}}{\alpha -{\lambda }^{*}+\alpha {\lambda }^{*}},\end{array}$$

(298)

whose only valid root (obtained by solving a quadratic equation) is

$$\begin{array}{cc}\hfill {\lambda }^{*}& =\frac{1+\alpha \mathsf{snr}-\alpha \Delta }{2\mathsf{snr}(1-\alpha )}\in (0,\alpha ),\hfill \end{array}$$

(299)

with $\Delta$ defined in (118). So, for $\alpha \in (0,1)$, (208) becomes

$$\begin{array}{cc}\hfill {\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\mathsf{snr}{\sigma }_N^2\right)& =\frac{1+\alpha \mathsf{snr}-\alpha \Delta }{4(1-\alpha )}log\mathrm{e}+\frac{1}{2}log\left(1+\frac{2\mathsf{snr}(1-\alpha )}{1+\alpha \mathsf{snr}-\alpha \Delta }-\frac{1}{\alpha }\right)\hfill \\ \hfill & -\frac{1}{2(1-\alpha )}log\left(\frac{\alpha \mathsf{snr}+\alpha \Delta -1}{2\mathsf{snr}{\alpha }^2}\right).\hfill \end{array}$$

(300)

Letting $\alpha =\frac{1}{1+\rho }$, we obtain

$$\begin{array}{cc}\hfill {\mathbb{C}}_{\frac{1}{1+\rho }}^{\mathtt{c}}\left(\mathsf{snr}{\sigma }_N^2\right)& =\frac{\mathsf{snr}}{2\rho }\left(1-\beta \right)log\mathrm{e}+\frac{1}{2}log(1+\beta \mathsf{snr})-\frac{1+\rho }{2\rho }log\left((1+\rho )\beta \right),\hfill \end{array}$$

(301)

with

$$\begin{array}{c}\hfill \beta =\frac{1}{2}\left(1-\frac{1}{\alpha \mathsf{snr}}+\frac{\Delta }{\mathsf{snr}}\right)=\frac{1}{2}\left(1-\frac{1+\rho }{\mathsf{snr}}+\sqrt{\frac{4}{\mathsf{snr}}+{\left(\frac{1+\rho }{\mathsf{snr}}-1\right)}^2}\right).\end{array}$$

(302)

74.

Alternatively, it is instructive to apply Theorem 18 to the current Gaussian/quadratic cost setting. Suppose we let $Q_X^{*}=\mathcal{N}\left(0,{\sigma }^{*2}\right)$, where ${\sigma }^{*2}$ is to be determined. With the aid of the formulas

$$\begin{array}{cc}\hfill \mathbb{E}\left[X^2{\mathrm{e}}^{-\mu X^2}\right]& =\frac{{\sigma }^2}{{\left(1+2\mu {\sigma }^2\right)}^{\frac{3}{2}}},\hfill \end{array}$$

(303)

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\mathrm{e}}^{-\mu X^2}\right]& =\frac{1}{\sqrt{1+2\mu {\sigma }^2}},\hfill \end{array}$$

(304)

where $\mu \ge 0$, and $X\sim \mathcal{N}\left(0,{\sigma }^2\right)$, (217) becomes

$$\begin{array}{c}\hfill \frac{1}{\mathsf{snr}}=\frac{{\sigma }_N^2}{{\sigma }^{*2}}+(1-\alpha ){\lambda }^{*},\end{array}$$

(305)

upon substituting ${\sigma }^2\leftarrow {\sigma }^{*2}$ and

$$\begin{array}{c}\hfill \mu \leftarrow {\nu }^{*}\frac{1-\alpha }{log\mathrm{e}}={\lambda }^{*}\frac{1-\alpha }{2{\sigma }_N^2}.\end{array}$$

(306)

Likewise (218) translates into (291) and (292) with $(\nu ,{\sigma }^2)\leftarrow ({\nu }^{*},{\sigma }^{*2})$, namely,

$$\begin{array}{cc}\hfill c_{\alpha }\left({\nu }^{*}\right)& =\frac{1}{2}log\left(1+\frac{\alpha {\sigma }^{*2}}{{\sigma }_N^2}\right)-\frac{1}{2(1-\alpha )}log\left(1+\frac{\alpha (1-\alpha ){\sigma }^{*2}}{{\alpha }^2{\sigma }^{*2}+{\sigma }_N^2}\right),\hfill \end{array}$$

(307)

$$\begin{array}{cc}\hfill {\lambda }^{*}& =\frac{\alpha {\sigma }_N^2}{{\alpha }^2{\sigma }^{*2}+{\sigma }_N^2}.\hfill \end{array}$$

(308)

Eliminating ${\sigma }^{*2}$ from (305) by means of (308) results in (299) and the same derivation that led to (300) shows that it is equal to ${\nu }^{*}\theta +c_{\alpha }\left({\nu }^{*}\right)$.

75.

Applying Theorem 17, we can readily find the input distribution, $P_X^{*}$, that attains ${\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)$ as well as its $\langle \alpha \rangle$-response $P_Y^{*}$ (recall the notation in Item 53). According to Example 2, $P_Y^{*}$, the $\alpha$-response to $Q_X^{*}$ is Gaussian with zero mean and variance

$$\begin{array}{cc}\hfill {\sigma }_N^2+\alpha {\sigma }^{*2}& ={\sigma }_N^2\left(1+\frac{1}{{\lambda }^{*}}-\frac{1}{\alpha }\right)\hfill \end{array}$$

(309)

$$\begin{array}{cc}\hfill & =\frac{{\sigma }_N^2}{2}\left(2-\frac{1}{\alpha }+\Delta +\mathsf{snr}\right),\hfill \end{array}$$

(310)

where (309) follows from (308) and (310) follows by using the expression for $\Delta$ in (118). Note from Example 7 that $P_Y^{*}$ is nothing but the $\langle \alpha \rangle$-response to $\mathcal{N}\left(0,\mathsf{snr}{\sigma }_N^2\right)$. We can easily verify from Theorem 17 that indeed $P_X^{*}=\mathcal{N}\left(0,\mathsf{snr}{\sigma }_N^2\right)$ since in this case (216) becomes

$$\begin{array}{c}\hfill {\mathit{ı}}_{P_X^{*}\parallel Q_X^{*}}\left(a\right)=-(1-\alpha ){\nu }^{*}{a}^2+{\tau }_{\alpha },\end{array}$$

(311)

which can only be satisfied by $P_X^{*}=\mathcal{N}\left(0,\mathsf{snr}{\sigma }_N^2\right)$ in view of (305). As an independent confirmation, we can verify, after some algebra, that the right sides of (127) and (300) are identical.

In fact, in the current Gaussian setting, we could start by postulating that the distribution that maximizes the Augustin–Csiszár mutual information under the second moment constraint does not depend on $\alpha$ and is given by $P_X^{*}=\mathcal{N}\left(0,\theta \right)$. Its $\langle \alpha \rangle$-response $P_{Y\langle \alpha \rangle }^{*}$ was already obtained in Example 7. Then, an alternative method to find ${\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)$, given in Section 6.2 of [43], is to follow the approach outlined in Item 53. To validate the choice of $P_X^{*}$ we must show that it maximizes $B(P_X,P_{Y\langle \alpha \rangle }^{*})$ (in the notation introduced in (199)) among the subset of ${\mathcal{P}}_{\mathcal{A}}$ which satisfies $\mathbb{E}\left[X^2\right]\le \theta$. This follows from the fact that $D_{\alpha }\left(P_{Y|X=x}\parallel P_{Y\langle \alpha \rangle }^{*}\right)$ is an affine function of $x^2$.

76.

Let’s now use the result in Item 73 to evaluate, with a novel parametrization, the error exponent functions for the Gaussian channel under an average power constraint.

Theorem 23.

Let $\mathcal{A}=\mathcal{B}=\mathbb{R}$, $\mathsf{b}\left(x\right)=x^2$, and $P_{Y|X=a}=\mathcal{N}\left(a,{\sigma }_N^2\right)$. Then, for $\beta \in [0,1]$,

$$\begin{array}{cc}\hfill E_{\mathtt{sp}}(R,\mathsf{snr}{\sigma }_N^2)& =\frac{\mathsf{snr}}{2}(1-\beta )log\mathrm{e}-\frac{1}{2}log\left(1+\mathsf{snr}\beta (1-\beta )\right),\hfill \end{array}$$

(312)

$$\begin{array}{cc}\hfill R& =\frac{1}{2}log\left(1+\frac{{\beta }^2}{\beta (1-\beta )+\frac{1}{\mathsf{snr}}}\right).\hfill \end{array}$$

(313)

The critical rate and cutoff rate are, respectively,

$$\begin{array}{cc}\hfill R_{\mathsf{c}}& =\frac{1}{2}log\left(\frac{1}{2}+\frac{\mathsf{snr}}{4}+\frac{1}{2}\sqrt{1+\frac{{\mathsf{snr}}^2}{4}}\right),\hfill \end{array}$$

(314)

$$\begin{array}{cc}\hfill R_0& =\frac{1}{2}\left(1+\frac{\mathsf{snr}}{2}-\sqrt{1+\frac{{\mathsf{snr}}^2}{4}}\right)log\mathrm{e}+\frac{1}{2}log\left(\frac{1}{2}+\frac{1}{2}\sqrt{1+\frac{{\mathsf{snr}}^2}{4}}\right).\hfill \end{array}$$

(315)

Proof. 

Expression (315) for the cutoff rate follows by letting $\rho =1$ in (301) and (302). The supremum in (281) is attained by ${\rho }^{*}\ge 0$ that satisfies (recall the concavity result in Theorem 9-(a))

$$\begin{array}{cc}\hfill R& =\frac{\mathrm{d}}{\mathrm{d}\rho }\rho {\mathbb{C}}_{\frac{1}{1+\rho }}^{\mathtt{c}}\left(\mathsf{snr}{\sigma }_N^2\right){|}_{\rho \leftarrow {\rho }^{*}}\hfill \end{array}$$

(316)

$$\begin{array}{cc}\hfill & =\frac{1}{2}log\left(\mathsf{snr}+\frac{1}{\beta }\right)-\frac{1}{2}log\left(1+{\rho }^{*}\right),\hfill \end{array}$$

(317)

obtained after a dose of symbolic computation working with (301). In particular, letting ${\rho }^{*}=1$, we obtain the critical rate in (314). Note that if in (302) we substitute $\rho \leftarrow {\rho }^{*}$, with ${\rho }^{*}$ given as a function of R, $\mathsf{snr}$ and $\beta$ by (317), we end up with an equation involving R, $\mathsf{snr}$, and $\beta$. We proceed to verify that that equation is, in fact, (312). By solving a quadratic equation, we can readily check that (302) is the positive root of

$$\begin{array}{c}\hfill 1+\rho =\mathsf{snr}(1-\beta )+\frac{1}{\beta }.\end{array}$$

(318)

If we particularize (318) to $\rho \leftarrow {\rho }^{*}$, with ${\rho }^{*}$ given by (317), namely,

$$\begin{array}{c}\hfill {\rho }^{*}=-1+exp(-2R)\left(\mathsf{snr}+\frac{1}{\beta }\right),\end{array}$$

(319)

we obtain

$$\begin{array}{c}\hfill exp\left(2R\right)=\frac{\mathsf{snr}\beta +1}{\mathsf{snr}\beta (1-\beta )+1},\end{array}$$

(320)

which is (313). Notice that the right side of (320) is monotonic increasing in $\beta >0$ ranging from 1 (for $\beta =0$) to $1+\mathsf{snr}$ (for $\beta =1$). Therefore, $\beta \in [0,1]$ spans the whole gamut of values of R of interest.

Assembling (281), (301) and (317), we obtain

$$\begin{array}{cc}\hfill & E_{\mathtt{sp}}(R,\mathsf{snr}{\sigma }_N^2)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =-{\rho }^{*}R+\frac{\mathsf{snr}}{2}\left(1-\beta \right)log\mathrm{e}+\frac{{\rho }^{*}}{2}log(1+\beta \mathsf{snr})-\frac{1+{\rho }^{*}}{2}log\left((1+{\rho }^{*})\beta \right)\hfill \\ \hfill & =-{\rho }^{*}R+\frac{\mathsf{snr}}{2}\left(1-\beta \right)log\mathrm{e}+\frac{{\rho }^{*}}{2}log(1+\beta \mathsf{snr})-\frac{1+{\rho }^{*}}{2}log\beta \hfill \end{array}$$

(321)

$$\begin{array}{cc}\hfill & +(1+{\rho }^{*})R-\frac{1+{\rho }^{*}}{2}log\left(\mathsf{snr}+\frac{1}{\beta }\right)\hfill \end{array}$$

(322)

$$\begin{array}{cc}\hfill & =R+\frac{\mathsf{snr}}{2}\left(1-\beta \right)log\mathrm{e}-\frac{1}{2}log\left(1+\beta \mathsf{snr}\right)\hfill \end{array}$$

(323)

$$\begin{array}{cc}\hfill & =\frac{\mathsf{snr}}{2}(1-\beta )log\mathrm{e}-\frac{1}{2}log\left(1+\mathsf{snr}\beta (1-\beta )\right),\hfill \end{array}$$

(324)

where (324) follows by substituting (313) on the left side. □

Note that the parametric expression in (312) and (313) (shown in Figure 2) is, in fact, a closed-form expression for $E_{\mathtt{sp}}(R,\mathsf{snr}{\sigma }_N^2)$ since we can invert (313) to obtain

$$\begin{array}{c}\hfill \beta =\frac{1}{2}\left(1-exp(-2R)\right)\left(1+\sqrt{1+\frac{4}{\mathsf{snr}(1-exp(-2R\left)\right)}}\right).\end{array}$$

(325)

The random coding error exponent is

$$\begin{array}{c}\hfill E_{\mathsf{r}}(R,\theta )=\left\{\begin{array}{cc}{E}_{\mathtt{sp}}(R,\theta ),\hfill & R\in (R_{\mathsf{c}},\frac{1}{2}log(1+\mathsf{snr}));\hfill \\ R_0-R,\hfill & R\in [0,R_{\mathsf{c}}],\hfill \end{array}\right.\end{array}$$

(326)

with the critical rate $R_{\mathsf{c}}$ and cutoff rate $R_0$ in (314) and (315), respectively. It can be checked that (326) coincides with the expression given by Gallager [9] (p. 340) where he optimizes (235) with respect to $\rho$ and r, but not $P_X$, which he just assumes to be $P_X=\mathcal{N}\left(0,\theta \right)$. The expression for $R_{\mathsf{c}}$ in (314) can be found in (7.4.34) of [9]; $R_0$ in (314) is implicit in p. 340 of [9], and explicit in e.g., [69].

77.

The expression for $E_{\mathtt{sp}}(R,\theta )$ in Theorem 23 has more structure than meets the eye. The analysis in Item 73 has shown that $E_{\mathtt{sp}}(R,P_X)$ is maximized over $P_X$ with second moment not exceeding $\theta$ by $P_X^{*}=\mathcal{N}\left(0,\theta \right)$ regardless of $R\in \left(0,\frac{1}{2}log(1+\mathsf{snr})\right)$. The fact that we have found a closed-form expression for (254) when evaluated at such input probability measure and $P_{Y|X=a}=\mathcal{N}\left(a,{\sigma }_N^2\right)$ is indicative that the minimum therein is attained by a Gaussian random transformation $Q_{Y|X}^{*}$. This is indeed the case: define the random transformation

$$\begin{array}{cc}\hfill Q_{Y|X=a}^{*}& =\mathcal{N}\left(\beta a,{\sigma }_1^2\right),\hfill \end{array}$$

(327)

$$\begin{array}{cc}\hfill \frac{{\sigma }_1^2}{{\sigma }_N^2}& =1+\mathsf{snr}\beta (1-\beta ).\hfill \end{array}$$

(328)

In comparison with the nominal random transformation $P_{Y|X=a}=\mathcal{N}\left(a,{\sigma }_N^2\right)$, this channel attenuates the input and contaminates it with a more powerful noise. Then,

$$\begin{array}{cc}\hfill I(P_X^{*},Q_{Y|X}^{*})& =\frac{1}{2}log\left(1+\frac{{\beta }^2}{\beta (1-\beta )+\frac{1}{\mathsf{snr}}}\right)=R.\hfill \end{array}$$

(329)

Furthermore, invoking (33), we get

$$\begin{array}{cc}\hfill D(Q_{Y|X}^{*}\parallel P_{Y|X}|P_X^{*})& =\mathbb{E}\left[D\left(\mathcal{N}\left(\beta X^{*},{\sigma }_1^2\right)\parallel \mathcal{N}\left(X^{*},{\sigma }_N^2\right)\right)\right]\hfill \end{array}$$

(330)

$$\begin{array}{cc}\hfill & =\frac{1}{2}\left({(\beta -1)}^2\mathsf{snr}+\frac{{\sigma }_1^2}{{\sigma }_N^2}-1\right)log\mathrm{e}-\frac{1}{2}log\frac{{\sigma }_1^2}{{\sigma }_N^2}\hfill \end{array}$$

(331)

$$\begin{array}{cc}\hfill & =\frac{\mathsf{snr}}{2}(1-\beta )log\mathrm{e}-\frac{1}{2}log\left(1+\mathsf{snr}\beta (1-\beta )\right)\hfill \end{array}$$

(332)

$$\begin{array}{cc}\hfill & =E_{\mathtt{sp}}(R,\mathsf{snr}{\sigma }_N^2),\hfill \end{array}$$

(333)

where (333) is (312). Therefore, $Q_{Y|X}^{*}$ does indeed achieve the minimum in (254) if $P_{Y|X=a}=\mathcal{N}\left(a,{\sigma }_N^2\right)$ and $P_X^{*}=\mathcal{N}\left(0,\theta \right)$. So, the most likely error mechanism is the result of atypically large noise strength and an attenuated received signal. Both effects cannot be combined into additional noise variance: there is no ${\sigma }^2>0$ such that $Q_{Y|X=a}=\mathcal{N}\left(a,{\sigma }^2\right)$ achieves the minimum in (254).

12. Additive Independent Exponential Noise; Input-Mean Constraint

This section finds the sphere-packing error exponent for the additive independent exponential noise channel under an input-mean constraint.

78.

Suppose that $\mathcal{A}=\mathcal{B}=[0,\infty )$, $\mathsf{b}\left(x\right)=x$, and

$$\begin{array}{c}\hfill Y=X+N,\end{array}$$

(334)

where N is exponentially distributed, independent of X, and $\mathbb{E}\left[N\right]=\zeta$. Therefore $P_{Y|X=a}$ has density

$$\begin{array}{c}\hfill p_{Y|X=a}\left(t\right)=\frac{1}{\zeta }{\mathrm{e}}^{-\frac{t-a}{\zeta }}1\{t\ge a\}.\end{array}$$

(335)

It is shown in [70,71] that

$$\begin{array}{cc}\hfill \underset{X:\mathbb{E}\left[X\right]\le \theta }{max}I(X;X+N)& =log\left(1+\mathsf{snr}\right),\hfill \end{array}$$

(336)

$$\begin{array}{cc}\hfill \mathsf{snr}& =\frac{\theta }{\zeta },\hfill \end{array}$$

(337)

achieved by a mixed random variable with density

$$\begin{array}{c}\hfill f_X^{*}\left(t\right)=\frac{\zeta }{\zeta +\theta }\delta \left(t\right)+\frac{\theta }{{(\zeta +\theta )}^2}{\mathrm{e}}^{-t/(\zeta +\theta )}1\{t>0\}.\end{array}$$

(338)

To determine ${\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\mathsf{snr}\zeta \right)$, $\alpha \in (0,1)$, we invoke Theorem 18. A sensible candidate for the auxiliary input distribution $Q_X^{*}$ is a mixed random variable with density

$$\begin{array}{cc}\hfill q_X^{*}\left(t\right)& ={\mathsf{\Gamma }}^{*}\delta \left(t\right)+\left(1-{\mathsf{\Gamma }}^{*}\right)\frac{1}{\mu }{\mathrm{e}}^{-t/\mu }1\{t>0\},\hfill \end{array}$$

(339)

$$\begin{array}{cc}\hfill \mu & =\frac{\zeta }{\alpha {\mathsf{\Gamma }}^{*}},\hfill \end{array}$$

(340)

where ${\mathsf{\Gamma }}^{*}\in (0,1)$ is yet to be determined. This is an attractive choice because its $\alpha$-response, $Q_{Y\left[\alpha \right]}^{*}$, is particularly simple: exponential with mean $\alpha \mu =\frac{\zeta }{{\mathsf{\Gamma }}^{*}}$, as we can verify using Laplace transforms. Then, if Z is exponential with unit mean, with the aid of Example 5, we can write

$$\begin{array}{cc}\hfill D_{\alpha }\left(P_{Y|X=x}\parallel Q_{Y\left[\alpha \right]}^{*}\right)& =D_{\alpha }(\zeta Z+x\parallel \alpha \mu Z)\hfill \end{array}$$

(341)

$$\begin{array}{cc}\hfill & =\frac{x}{\alpha \mu }log\mathrm{e}+log\frac{\alpha \mu }{\zeta }+\frac{1}{1-\alpha }log\left(\alpha +(1-\alpha )\frac{\zeta }{\alpha \mu }\right)\hfill \end{array}$$

(342)

$$\begin{array}{cc}\hfill & =\frac{{\mathsf{\Gamma }}^{*}x}{\zeta }log\mathrm{e}-log{\mathsf{\Gamma }}^{*}+\frac{1}{1-\alpha }log\left(\alpha +(1-\alpha ){\mathsf{\Gamma }}^{*}\right).\hfill \end{array}$$

(343)

So, (218) is satisfied with

$$\begin{array}{cc}\hfill {\nu }^{*}& =\frac{{\mathsf{\Gamma }}^{*}}{\zeta }log\mathrm{e},\hfill \end{array}$$

(344)

$$\begin{array}{cc}\hfill c_{\alpha }\left({\nu }^{*}\right)& =\frac{1}{1-\alpha }log\left(\alpha +(1-\alpha ){\mathsf{\Gamma }}^{*}\right)-log{\mathsf{\Gamma }}^{*}.\hfill \end{array}$$

(345)

To evaluate (217), it is useful to note that if $\gamma >-1$, then

$$\begin{array}{cc}\hfill \mathbb{E}\left[Z{\mathrm{e}}^{-\gamma Z}\right]& =\frac{1}{{(1+\gamma )}^2},\hfill \end{array}$$

(346)

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\mathrm{e}}^{-\gamma Z}\right]& =\frac{1}{1+\gamma }.\hfill \end{array}$$

(347)

Therefore, the left side of (217) specializes to, with ${\overline{X}}^{*}\sim Q_X^{*}$,

$$\begin{array}{cc}\hfill \mathbb{E}\left[\mathsf{b}\left({\overline{X}}^{*}\right)exp\left(-(1-\alpha ){\nu }^{*}\mathsf{b}\left({\overline{X}}^{*}\right)\right)\right]& =\frac{\mu (1-{\mathsf{\Gamma }}^{*})}{{\left(1+\mu (1-\alpha )\frac{{\nu }^{*}}{log\mathrm{e}}\right)}^2}\hfill \end{array}$$

(348)

$$\begin{array}{cc}\hfill & =\zeta \alpha \left(\frac{1}{{\mathsf{\Gamma }}^{*}}-1\right),\hfill \end{array}$$

(349)

while the expectation on the right side of (217) is given by

$$\begin{array}{cc}\hfill \mathbb{E}\left[exp\left(-(1-\alpha ){\nu }^{*}\mathsf{b}\left({\overline{X}}^{*}\right)\right)\right]& =\alpha +{\mathsf{\Gamma }}^{*}-\alpha {\mathsf{\Gamma }}^{*}.\hfill \end{array}$$

(350)

Therefore, (217) yields

$$\begin{array}{c}\hfill \mathsf{snr}=\frac{1}{{\mathsf{\Gamma }}^{*}}-\frac{1}{\alpha +(1-\alpha ){\mathsf{\Gamma }}^{*}}\end{array}$$

(351)

whose solution is

$$\begin{array}{c}\hfill {\mathsf{\Gamma }}^{*}=\frac{1}{2\rho \mathsf{snr}}\left(\sqrt{{\left(1+\mathsf{snr}\right)}^2+4\rho \mathsf{snr}}-1-\mathsf{snr}\right),\end{array}$$

(352)

with $\rho =\frac{1-\alpha }{\alpha }$. So, finally, (220), (344) and (345) give the closed-form expression

$$\begin{array}{c}\hfill {\mathbb{C}}_{\alpha }^{\mathtt{c}}\left(\theta \right)=\mathsf{snr}{\mathsf{\Gamma }}^{*}log\mathrm{e}-log{\mathsf{\Gamma }}^{*}+\frac{1}{1-\alpha }log\left(\alpha +(1-\alpha ){\mathsf{\Gamma }}^{*}\right).\end{array}$$

(353)

As in Item 73, we can postulate an auxiliary distribution that satisfies (230) for every $\nu \ge 0$. This is identical to what we did in (341)–(343) except that now (344) and (345) hold for generic $\nu$ and $\mathsf{\Gamma }$. Then, (351) is the result of solving $\theta =-{\dot{c}}_{\alpha }\left({\nu }^{*}\right)$, which is, in fact, somewhat simpler than obtaining it through (217).

79.

We proceed to get a very simple parametric expression for $E_{\mathtt{sp}}(R,\theta )$.

Theorem 24.

Let $\mathcal{A}=\mathcal{B}=[0,\infty )$, $\mathsf{b}\left(x\right)=x$, and $Y=X+N$, with N exponentially distributed, independent of X, and $\mathbb{E}\left[N\right]=\zeta$. Then, under the average cost constraint $\mathbb{E}\left[\mathsf{b}\right(X\left)\right]\le \zeta \mathsf{snr}$,

$$\begin{array}{cc}\hfill E_{\mathtt{sp}}(R,\zeta \mathsf{snr})& =\left(\frac{1}{\eta }-1\right)log\mathrm{e}+log\eta ,\hfill \end{array}$$

(354)

$$\begin{array}{cc}\hfill R& =log(1+\eta \mathsf{snr}),\hfill \end{array}$$

(355)

where $\eta \in (0,1]$.

Proof. 

Rewriting (353), results in

$$\begin{array}{c}\hfill \rho {\mathbb{C}}_{\frac{1}{1+\rho }}^{\mathtt{c}}\left(\theta \right)=\rho \mathsf{snr}{\mathsf{\Gamma }}^{*}log\mathrm{e}-\rho log{\mathsf{\Gamma }}^{*}+(1+\rho )log\frac{1+\rho {\mathsf{\Gamma }}^{*}}{1+\rho },\end{array}$$

(356)

which is monotonically decreasing with $\rho$. With ${\dot{\mathsf{\Gamma }}}^{*}=\frac{\partial }{\partial \rho }{\mathsf{\Gamma }}^{*}(\rho ,\mathsf{snr})$, the counterpart of (317) is now

$$\begin{array}{cc}\hfill R& =\frac{\mathrm{d}}{\mathrm{d}\rho }\rho {\mathbb{C}}_{\frac{1}{1+\rho }}^{\mathtt{c}}\left(\theta \right){|}_{\rho \leftarrow {\rho }^{*}}\hfill \end{array}$$

(357)

$$\begin{array}{cc}\hfill & =({\mathsf{\Gamma }}^{*}+{\rho }^{*}{\dot{\mathsf{\Gamma }}}^{*})\left(\mathsf{snr}-\frac{1}{{\mathsf{\Gamma }}^{*}}+\frac{1+{\rho }^{*}}{1+{\rho }^{*}{\mathsf{\Gamma }}^{*}}\right)log\mathrm{e}+log\frac{1+{\rho }^{*}{\mathsf{\Gamma }}^{*}}{{\mathsf{\Gamma }}^{*}+{\rho }^{*}{\mathsf{\Gamma }}^{*}}\hfill \end{array}$$

(358)

$$\begin{array}{cc}\hfill & =({\mathsf{\Gamma }}^{*}+{\rho }^{*}{\dot{\mathsf{\Gamma }}}^{*})\left(\mathsf{snr}+\frac{1}{{\mathsf{\Gamma }}^{*}}\frac{{\mathsf{\Gamma }}^{*}-1}{1+{\rho }^{*}{\mathsf{\Gamma }}^{*}}\right)log\mathrm{e}+log\frac{1+{\rho }^{*}{\mathsf{\Gamma }}^{*}}{{\mathsf{\Gamma }}^{*}+{\rho }^{*}{\mathsf{\Gamma }}^{*}}\hfill \end{array}$$

(359)

$$\begin{array}{cc}\hfill & =log\frac{1+{\rho }^{*}{\mathsf{\Gamma }}^{*}}{{\mathsf{\Gamma }}^{*}+{\rho }^{*}{\mathsf{\Gamma }}^{*}},\hfill \end{array}$$

(360)

where the drastic simplification in (360) occurs because, with the current parametrization, (351) becomes

$$\begin{array}{c}\hfill 1-{\mathsf{\Gamma }}^{*}=(1+{\rho }^{*}{\mathsf{\Gamma }}^{*}){\mathsf{\Gamma }}^{*}\mathsf{snr}.\end{array}$$

(361)

Now we go ahead and express both ${\rho }^{*}$ and ${\mathsf{\Gamma }}^{*}$ as functions of $\mathsf{snr}$ and R exclusively. We may rewrite (357)–(360) as

$$\begin{array}{c}\hfill {\rho }^{*}{\mathsf{\Gamma }}^{*}=\frac{exp(-R)-{\mathsf{\Gamma }}^{*}}{1-exp(-R)},\end{array}$$

(362)

which, when plugged in (361), results in

$$\begin{array}{cc}\hfill {\mathsf{\Gamma }}^{*}& =\frac{1}{\mathsf{snr}}\left(1-exp(-R)\right)<1,\hfill \end{array}$$

(363)

$$\begin{array}{cc}\hfill {\rho }^{*}& =\frac{(1+\mathsf{snr})exp(-R)-1}{{\left(1-exp(-R)\right)}^2}>0,\hfill \end{array}$$

(364)

where the inequalities in (363) and (364) follow from $R<log(1+\mathsf{snr})$. So, in conclusion,

$$\begin{array}{cc}\hfill E_{\mathtt{sp}}(R,\theta )& =\underset{\rho \ge 0}{max}\left\{\rho {\mathbb{C}}_{\frac{1}{1+\rho }}^{\mathtt{c}}\left(\theta \right)-\rho R\right\}\hfill \end{array}$$

(365)

$$\begin{array}{cc}\hfill & ={\rho }^{*}{\mathbb{C}}_{\frac{1}{1+{\rho }^{*}}}^{\mathtt{c}}\left(\theta \right)-{\rho }^{*}R\hfill \end{array}$$

(366)

$$\begin{array}{cc}\hfill & ={\rho }^{*}\mathsf{snr}{\mathsf{\Gamma }}^{*}log\mathrm{e}-{\rho }^{*}log{\mathsf{\Gamma }}^{*}+(1+{\rho }^{*})log\frac{1+{\rho }^{*}{\mathsf{\Gamma }}^{*}}{1+{\rho }^{*}}-{\rho }^{*}R\hfill \end{array}$$

(367)

$$\begin{array}{cc}\hfill & ={\rho }^{*}\mathsf{snr}{\mathsf{\Gamma }}^{*}log\mathrm{e}-{\rho }^{*}log{\mathsf{\Gamma }}^{*}+(1+{\rho }^{*})(R+log{\mathsf{\Gamma }}^{*})-{\rho }^{*}R\hfill \end{array}$$

(368)

$$\begin{array}{cc}\hfill & ={\rho }^{*}\mathsf{snr}{\mathsf{\Gamma }}^{*}log\mathrm{e}+log{\mathsf{\Gamma }}^{*}+R\hfill \end{array}$$

(369)

$$\begin{array}{cc}\hfill & =\left(\frac{\mathsf{snr}}{exp\left(R\right)-1}-1\right)log\mathrm{e}+log\frac{exp\left(R\right)-1}{\mathsf{snr}}\hfill \end{array}$$

(370)

$$\begin{array}{cc}\hfill & =\left(\frac{1}{\eta }-1\right)log\mathrm{e}+log\eta ,\hfill \end{array}$$

(371)

where we have introduced

$$\begin{array}{c}\hfill \eta =\frac{exp\left(R\right)-1}{\mathsf{snr}}=\frac{{\mathsf{\Gamma }}^{*}}{1-\mathsf{snr}{\mathsf{\Gamma }}^{*}}.\end{array}$$

(372)

Evidently, the left identity in (372) is the same as (355). □

The critical rate and the cutoff rate are obtained by particularizing (360) and (356) to ${\rho }^{*}=1$ and $\rho =1$, respectively. This yields

$$\begin{array}{cc}\hfill R_{\mathsf{c}}& =log\frac{1+{\mathsf{\Gamma }}_1^{*}}{2{\mathsf{\Gamma }}_1^{*}},\hfill \end{array}$$

(373)

$$\begin{array}{cc}\hfill R_0& =\mathsf{snr}{\mathsf{\Gamma }}_1^{*}log\mathrm{e}-log\left(4{\mathsf{\Gamma }}_1^{*}\right)+2log\left(1+{\mathsf{\Gamma }}_1^{*}\right),\hfill \end{array}$$

(374)

$$\begin{array}{cc}\hfill {\mathsf{\Gamma }}_1^{*}& =\frac{\sqrt{{\left(1+\mathsf{snr}\right)}^2+4\mathsf{snr}}-1-\mathsf{snr}}{2\mathsf{snr}}.\hfill \end{array}$$

(375)

As in (326), the random coding error exponent is

$$\begin{array}{c}\hfill E_{\mathsf{r}}(R,\zeta \mathsf{snr})=\left\{\begin{array}{cc}{E}_{\mathtt{sp}}(R,\zeta \mathsf{snr}),\hfill & R\in (R_{\mathsf{c}},log(1+\mathsf{snr}));\hfill \\ R_0-R,\hfill & R\in [0,R_{\mathsf{c}}],\hfill \end{array}\right.\end{array}$$

(376)

with the critical rate $R_{\mathsf{c}}$ and cutoff rate $R_0$ in (373) and (375), respectively. This function is shown along with $E_{\mathtt{sp}}(R,\zeta \mathsf{snr})$ in Figure 3 for $\mathsf{snr}=3$.

80.

In parallel to Item 77, we find the random transformation that explains the most likely mechanism to produce errors at every rate R, namely the minimizer of (254) when $P_X=P_X^{*}$, the maximizer of the Augustin–Csiszár mutual information of order $\alpha$. In this case, $P_X^{*}$ is not as trivial to guess as in Section 11, but since we already found $Q_X^{*}$ in (339) with $\mathsf{\Gamma }={\mathsf{\Gamma }}^{*}$, we can invoke Theorem 17 to show that the density of $P_X^{*}$ achieving the maximal order-$\alpha$ Augustin–Csiszár mutual information is

$$\begin{array}{c}\hfill p_X^{*}\left(t\right)=\frac{{\mathsf{\Gamma }}^{*}}{\alpha +(1-\alpha ){\mathsf{\Gamma }}^{*}}\delta \left(t\right)+\frac{1-{\mathsf{\Gamma }}^{*}}{\alpha +(1-\alpha ){\mathsf{\Gamma }}^{*}}\frac{\alpha {\mathsf{\Gamma }}^{*}}{\zeta }{\mathrm{e}}^{-t{\mathsf{\Gamma }}^{*}/\zeta }1\{t>0\},\end{array}$$

(377)

whose mean is, as it should,

$$\begin{array}{c}\hfill \frac{\alpha \zeta }{{\mathsf{\Gamma }}^{*}}\frac{1-{\mathsf{\Gamma }}^{*}}{\alpha +(1-\alpha ){\mathsf{\Gamma }}^{*}}=\zeta \mathsf{snr}=\theta .\end{array}$$

(378)

Let $Q_Y^{*}$ be exponential with mean $\theta +\kappa$, and $Q_{Y|X=a}^{*}$ have density

$$\begin{array}{c}\hfill q_{Y|X=a}^{*}\left(t\right)=\frac{1}{\kappa }{\mathrm{e}}^{-\frac{t-a}{\kappa }}1\{t\ge a\},\end{array}$$

(379)

with

$$\begin{array}{c}\hfill \kappa =\frac{\zeta }{\eta },\end{array}$$

(380)

and $\eta$ as defined in (372). Using Laplace transforms, we can verify that $P_X^{*}\to Q_{Y|X}^{*}\to Q_Y^{*}$ where $P_X^{*}$ is the probability measure with density in (377). Let Z be unit-mean exponentially distributed. Writing mutual information as the difference between the output differential entropy and the noise differential entropy we get

$$\begin{array}{cc}\hfill I(P_X^{*},Q_{Y|X}^{*})& =h\left(\right(\theta +\kappa \left)Z\right)-h\left(\kappa Z\right)\hfill \end{array}$$

(381)

$$\begin{array}{cc}\hfill & =log\left(1+\frac{\theta }{\kappa }\right)\hfill \end{array}$$

(382)

$$\begin{array}{cc}\hfill & =R,\hfill \end{array}$$

(383)

in view of (363). Furthermore, using (335) and (379),

$$\begin{array}{cc}\hfill D(Q_{Y|X}^{*}\parallel P_{Y|X}|P_X^{*})& =log\frac{\zeta }{\kappa }+\left(\frac{\kappa }{\zeta }-1\right)log\mathrm{e}\hfill \end{array}$$

(384)

$$\begin{array}{cc}\hfill & =log\eta +\left(\frac{1}{\eta }-1\right)log\mathrm{e}\hfill \end{array}$$

(385)

$$\begin{array}{cc}\hfill & =E_{\mathtt{sp}}(R,\zeta \mathsf{snr}),\hfill \end{array}$$

(386)

where we have used (380) and (354). Therefore, we have shown that $Q_{Y|X}^{*}$ is indeed the minimizer of (254). In this case, the most likely mechanism for errors to happen is that the channel adds independent exponential noise with mean $\zeta /\eta$, instead of the nominal mean $\zeta$. In this respect, the behavior is reminiscent of that of the exponential timing channel for which the error exponent is dominated (at least above critical rate) by an exponential server which is slower than the nominal [72].

13. Recap

81.

The analysis of the fundamental limits of noisy channels in the regime of vanishing error probability with blocklength growing without bound expresses channel capacity in terms of a basic information measure: the input–output mutual information maximized over the input distribution. In the regime of fixed nonzero error probability, the asymptotic fundamental limit is a function of not only capacity but channel dispersion [73], which is also expressible in terms of an information measure: the variance of the information density obtained with the capacity-achieving distribution. In the regime of exponentially decreasing error probability (at fixed rate below capacity) the analysis of the fundamental limits has gone through three distinct phases. No information measures were involved during the first phase and any optimization with respect to various auxiliary parameters and input distribution had to rely on standard convex optimization techniques, such as Karush-Kuhn-Tucker conditions, which not only are cumbersome to solve in this particular setting, but shed little light on the structure of the solution. The second phase firmly anchored the problem in a large deviations foundation, with the fundamental limits expressed in terms of conditional relative entropy as well as mutual information. Unfortunately, the associated maximinimization in (2) did not immediately lend itself to analytical progress. Thanks to Csiszár’s realization of the relevance of Rényi’s information measures to this problem, the third phase has found a way to, not only express the error exponent functions as a function of information measures, but to solve the associated optimization problems in a systematic way. While, in the absence of cost constraints, the problem reduces to finding the maximal $\alpha$-mutual information, cost constraints make the problem much more challenging because of the difficulty in determining the order-$\alpha$ Augustin–Csiszár mutual information. Fortunately, thanks to the introduction of an auxiliary input distribution (the $\langle \alpha \rangle$-adjunct of the distribution that maximizes $I_{\alpha }^{\mathtt{c}}$), we have shown that $\alpha$-mutual information also comes to the rescue in the maximization of the order-$\alpha$ Augustin–Csiszár mutual information in the presence of average cost constraints. We have also finally ended the isolation of Gallager’s $E_0$ function with cost constraints from the representations in Phases 2 and 3. The pursuit of such a link is what motivated Augustin in 1978 to define a generalized mutual information measure. Overall, the analysis has given yet another instance of the benefits of variational representations of information measures, leading to solutions based on saddle points. However, we have steered clear of off-the-shelf minimax theorems and their associated topological constraints.

We have worked out two channels/cost constraints (additive Gaussian noise with quadratic cost, and additive exponential noise with a linear cost) that admit closed-form error-exponent functions, most easily expressed in parametric form. Furthermore, in Items 77 and 80 we have illuminated the structure of those closed-form expressions by identifying the anomalous channel behavior responsible for most errors at every given rate. In the exponential noise case, the solution is simply a noisier exponential channel, while in the Gaussian case it is the result of both a noisier Gaussian channel and an attenuated input.

These observations prompt the question of whether there might be an alternative general approach that eschews Rényi’s information measures to arrive at not only the most likely anomalous channel behavior, but the error exponent functions themselves.