Conditional Rényi Divergence Saddlepoint and the Maximization of α-Mutual Information

Abstract

Rényi-type generalizations of entropy, relative entropy and mutual information have found numerous applications throughout information theory and beyond. While there is consensus that the ways A. Rényi generalized entropy and relative entropy in 1961 are the “right” ones, several candidates have been put forth as possible mutual informations of order $\alpha$. In this paper we lend further evidence to the notion that a Bayesian measure of statistical distinctness introduced by R. Sibson in 1969 (closely related to Gallager’s $E_0$ function) is the most natural generalization, lending itself to explicit computation and maximization, as well as closed-form formulas. This paper considers general (not necessarily discrete) alphabets and extends the major analytical results on the saddle-point and saddle-level of the conditional relative entropy to the conditional Rényi divergence. Several examples illustrate the main application of these results, namely, the maximization of $\alpha$-mutual information with and without constraints.

1. Introduction

The Rényi divergence of order $\alpha$ between two probability measures defined on the same measurable space,

$$\begin{array}{c}\hfill D_{\alpha }(P\parallel Q)=\frac{1}{\alpha -1}log\int {\left(\frac{\mathrm{d}P}{\mathrm{d}Q}(x)\right)}^{\alpha }\mathrm{d}Q(x),\end{array}$$

(1)

is a useful generalization of the relative entropy $D(P\parallel Q)$ introduced by Rényi [1] in the discrete case $({lim}_{\alpha \uparrow 1}{D}_{\alpha }(P\parallel Q)=D(P\parallel Q)$). Many of the properties satisfied by relative entropy hold for Rényi divergence, such as nonnegativity, convexity, lower semicontinuity, data processing inequality, and additivity for product measures. $D_{\alpha }(P\parallel Q)$ can be defined in more generality without requiring $P\ll Q$. A comprehensive survey of the properties satisfied by Rényi divergence can be found in [2]. Just as $D(P\parallel Q)$, $D_{\alpha }(P\parallel Q)$ provides a useful gauge of the distinctness of P and Q, which has found applications in large deviations problems (such as the asymptotic analysis of hypothesis testing [3,4,5]), lossless data compression [4,6,7], data transmission through noisy channels [8,9,10], and statistical physics [11]. If $P_1\ll P_0$, then Rényi divergence of order $\alpha \in (0,1)\cup (1,\infty )$ can be expressed in terms of relative entropy through [5]

$$\begin{array}{c}\hfill (1-\alpha )D_{\alpha }(P_1\parallel P_0)=\underset{P\ll P_1}{min}\left\{\alpha D(P\parallel P_1)+(1-\alpha )D(P\parallel P_0)\right\}.\end{array}$$

(2)

Although not an f-divergence, there is a one-to-one correspondence between Rényi divergence and Hellinger divergence ${\mathcal{H}}_{\alpha }(P\parallel Q)$ (e.g., [12])

$$\begin{array}{c}\hfill D_{\alpha }(P\parallel Q)=\frac{1}{\alpha -1}log\left(1+(\alpha -1){\mathcal{H}}_{\alpha }(P\parallel Q)\right).\end{array}$$

(3)

One of the major applications of relative entropy is to quantify statistical dependence in a joint probability measure by means of the mutual information

$$\begin{array}{c}\hfill I(X;Y)=D(P_{XY}\parallel P_X\times P_Y).\end{array}$$

(4)

The corresponding straight generalization replacing relative entropy by Rényi divergence is also a measure of dependence but has found scant utility so far (see [6,13]). To explore the generalization that we study in this paper, namely $\alpha$-mutual information, we need to consider the conditional versions of relative entropy and Rényi divergence. These are defined in general for two random transformations $P_{Y|X}$ and $Q_{Y|X}$ and an unconditional probability measure $P_X$ simply as

$$\begin{array}{cc}\hfill D(P_{Y|X}\parallel Q_{Y|X}|P_X)& =D(P_{Y|X}{P}_X\parallel Q_{Y|X}{P}_X),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill D_{\alpha }(P_{Y|X}\parallel Q_{Y|X}|P_X)& =D_{\alpha }(P_{Y|X}{P}_X\parallel Q_{Y|X}{P}_X).\hfill \end{array}$$

(6)

A major difference between those conditional measures is that while $D(P_{Y|X}\parallel Q_{Y|X}|P_X)$ is plainly the expectation $\int D(P_{Y|X=x}\parallel Q_{Y|X=x})\mathrm{d}{P}_X(x)$, the conditional Rényi divergence depends on the function $D_{\alpha }(P_{Y|X=x}\parallel Q_{Y|X=x})$ in a more involved way. In this paper, the use of the conditional information measures will be circumscribed to the special case in which $Q_{Y|X}$ is actually an unconditional measure. In fact, a more productive way to express mutual information than (4) is the asymmetric expression

$$\begin{array}{cc}\hfill I(X;Y)& =D(P_{Y|X}\parallel P_Y|P_X)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & =\underset{Q}{min}D(P_{Y|X}\parallel Q|P_X).\hfill \end{array}$$

(8)

Equation (8) follows from the key additive decomposition formula

$$\begin{array}{c}\hfill D(P_{Y|X}\parallel P_Y|P_X)=D(P_{Y|X}\parallel Q_Y|P_X)-D(P_Y\parallel Q_Y),\end{array}$$

(9)

where $Q_Y$ is an arbitrary measure dominating $P_Y$. We see that (8) is a Bayesian measure of the distinctness of the constellation of probability measures $\{P_{Y|X=x},x\in \mathcal{A}\}$, sometimes referred to as information radius, where the center of gravity of the constellation is none other than $P_Y$. Equation (8) has proven to be very fertile, particularly when it comes to supremize $I(X;Y)$ with respect to $P_X$ since the ensuing $supmin$ optimization has a saddle-point if and only if there is an input distribution that attains the maximal mutual information. The convexity of $D(P_{Y|X}\parallel Q|P_X)$ in Q and concavity (linearity) in $P_X$, along with the minimax theorem ensures the existence of the saddle-point whenever the set of allowed input distributions is compact. The Arimoto-Blahut algorithm [14,15] for finding $maxI(X;Y)$ in finite alphabet settings is also inspired by (8).

Mutual information $I(X;Y)=I(P_X,P_{Y|X})$ also possesses a saddle point (assuming convexity and compactness of the corresponding feasible sets) since it is concave in $P_X$ (to see that, nothing better than (9)) and is convex in $P_{Y|X}$. This property has found rich applications in information theory (e.g., [16,17,18]) but neither it nor its generalization to $\alpha$-mutual information will not concern us in this paper.

Even if a saddle-point for the conditional relative entropy does not exist, Kemperman [19] showed that $supmin$ can be swapped, thereby establishing the existence of a saddle value.

Another well-known application of (8) and the conditional relative entropy saddle-point is the so-called channel capacity-minimax redundancy theorem due to Gallager [20] and Ryabko [21] (see also [22,23]), which shows that the maximal mutual information obtained with a finite constellation $\{P_{Y|X=x},x\in \mathcal{A}\}$ is equal to the minimax redundancy in universal lossless data compression of an unknown source selected from $\{P_{Y|X=x},x\in \mathcal{A}\}$. Notable generalizations of this result to infinite alphabets without requiring that a distribution maximizing mutual information exists are due to Kemperman [19] and Haussler [24]. Recently, the Rényi counterpart of the channel capacity-minimax redundancy result has also been considered, under various restrictions, in [2,25,26].

The main purpose of this paper is to generalize the saddle-point property of conditional relative entropy and its applications to the maximization of mutual information when relative entropy is replaced by Rényi divergence. Towards that end, we recall the various directions in which mutual information has been generalized using Rényi divergence (see also [27]):

• As aforementioned, the straight generalization $D_{\alpha }(P_{XY}\parallel P_X\times P_Y)$ has not yet found wide applicability.
• In the discrete case and $\alpha \in (0,1)\cup (1,\infty )$, Arimoto [28] proposed the definition of the nonnegative quantity

  $$\begin{array}{c}\hfill I^{\mathtt{a}}(X;Y)=H_{\alpha }(X)-H_{\alpha }^{\mathtt{a}}(X|Y),\end{array}$$

  (10)

  where the Rényi entropy [1] and Arimoto-Rényi conditional entropy [28] are

  $$\begin{array}{cc}\hfill H_{\alpha }(X)& =\frac{\alpha }{1-\alpha }log{\parallel P_X\parallel }_{\alpha },\hfill \end{array}$$

  (11)

  $$\begin{array}{cc}\hfill H_{\alpha }^{\mathtt{a}}(X|Y)& =\frac{\alpha }{1-\alpha }log\mathbb{E}[\parallel P_{X|Y}(·|Y){\parallel }_{\alpha }]\hfill \end{array}$$

  (12)

  with the $\alpha$-norm of a probability mass function denoted as ${\parallel P\parallel }_{\alpha }={\left({\sum }_{x\in \mathcal{A}}{P}^{\alpha }(x)\right)}^{\frac{1}{\alpha }}$. Arimoto extended his algorithm in [14] to compute what he called the capacity of order $\alpha$,

  $$\begin{array}{c}\hfill C_{\alpha }^{\mathtt{a}}=\underset{X}{max}{I}^{\mathtt{a}}(X;Y),\end{array}$$

  (13)

  for finite-alphabet random transformations and showed that there exist codes of rate R and blocklength n whose error probability is upper bounded by

  $$\underset{\alpha \in (\frac{1}{2},1)}{inf}exp\left(\frac{\alpha -1}{\alpha }(C_{\alpha }^{\mathtt{a}}-R)\right).$$
• Augustin [29] and, later, Csiszár [4] defined

  $$\begin{array}{c}\hfill I_{\alpha }^{\mathtt{c}}(X;Y)=\underset{Q_Y}{min}\mathbb{E}\left[D_{\alpha }(P_{Y|X}(·|X)\parallel Q_Y)\right].\end{array}$$

  (14)

  $C_{\alpha }^{\mathtt{c}}={max}_X{I}_{\alpha }^{\mathtt{c}}(X;Y)$ is dubbed the Augustin capacity of order $\alpha$ in [30]. Csiszár [4] showed that for $\alpha \in (\frac{1}{2},1)$, $I_{\alpha }^{\mathtt{c}}(X;Y)$ is the intercept on the R-axis of a supporting line of slope $1-\frac{1}{\alpha }$ of the error exponent function for codes of rate R with constant-composition $P_X$. Unfortunately, the minimization in (14) is not amenable to explicit solution.
• For the purpose of coming up with a measure of the similarity among a finite collection of probability measures $\{P_{Y|X=x},x\in \mathcal{A}\}$, weighted by $P_X$ on $\mathcal{A}$, Sibson [31] proposed the information radius of order $\alpha$ as

  $$\begin{array}{c}\hfill I_{\alpha }(X;Y)=\underset{Q}{min}{D}_{\alpha }(P_{Y|X}\parallel Q|P_X).\end{array}$$

  (15)

  As we will see, the minimization in (15) can be solved explicitly. This is the generalization of mutual information we adopt in this paper and which, as in [27], we refer to as $\alpha$-mutual information. A word of caution is that in [4], the symbols $I_{\alpha }(X;Y)$ and $K_{\alpha }(X;Y)$ are used in lieu of what we denote $I_{\alpha }^{\mathtt{c}}(X;Y)$ and $I_{\alpha }(X;Y)$, respectively. $C_{\alpha }={max}_X{I}_{\alpha }(X;Y)$ is dubbed the Rényi capacity of order $\alpha$ in [26].
• Independently, Lapidoth-Pfister [32] and Tomamichel-Hayashi [33] proposed

  $$\begin{array}{c}\hfill I_{\alpha }^{\mathtt{l}}(X;Y)=\underset{Q_X}{min}\underset{Q_Y}{min}{D}_{\alpha }(P_{XY}\parallel Q_X\times Q_Y).\end{array}$$

  (16)

  and showed that it determines the performance of composite hypothesis tests for independence where the hypothesized joint distribution is known but under the independence hypothesis the marginals are unknown. It was shown in [34] that

  $$\begin{array}{c}\hfill I_{\alpha }^{\mathtt{c}}(X;Y)\le I_{\alpha }^{\mathtt{l}}(X;Y)\le I_{\alpha }(X;Y).\end{array}$$

  (17)

Despite the difference in the definitions of the various versions, it was shown in the discrete setting that [4,28].

$$\begin{array}{c}\hfill C_{\alpha }^{\mathtt{a}}=C_{\alpha }^{\mathtt{c}}=C_{\alpha }\end{array}$$

(18)

Therefore, solving for ${max}_X{I}_{\alpha }(X;Y)$ carries added significance, whenever one of the other definitions is adopted. Note that (17) and (18) imply that $C_{\alpha }={max}_{P_X}{I}_{\alpha }^{\mathtt{l}}(P_X,P_{Y|X})$. A major application for the maximization of $I_{\alpha }(X;Y)$ is in the large deviation analysis of optimal data transmission codes since the sphere-packing error exponent function and the random-coding error exponent function

$$\begin{array}{cc}\hfill E_{\mathtt{sp}}(R)& =\underset{\rho \ge 0}{sup}\left\{\rho C_{\frac{1}{1+\rho }}-\rho R\right\},\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill E_{\mathtt{r}}(R)& =\underset{\rho \in [0,1]}{sup}\left\{\rho C_{\frac{1}{1+\rho }}-\rho R\right\},\hfill \end{array}$$

(20)

popularized in [35] and [36], respectively, are upper and lower bounds to the channel reliability function, respectively. A function similar to (20) has recently been shown [37] to yield the large deviations behavior of random coding in the setting of channel resolvability.

The organization of the paper is as follows. Section 2 states the definitions and properties of the various information measures that are used throughout the paper. In particular, we introduce the key notion of $\alpha$-response to an input probability measure through a given random transformation. In Section 3 we present the main results (with proofs relegated to Section 5) related to the saddle-point and saddle-value of the conditional Rényi divergence, allowing the optimization to be circumscribed to any convex set of input probability measures. The equivalence of the existence of a probability measure that maximizes $\alpha$-mutual information and the existence of a saddle point is shown and several illustrative examples of the use of this result in the computation of $C_{\alpha }$ are also given. The fact that a saddle-level exists (i.e., $supmin$ commute) even if there is no input probability measure that achieves the supremum $\alpha$-mutual information is established, thereby generalizing Kemperman’s [19] saddle-level result to Rényi divergence through a different route than that followed in [26].

2. Notation, Definitions and Properties

• If $(\mathcal{A},\mathcal{F},P)$ is a probability space, $X\sim P$ indicates $\mathbb{P}[X\in \mathcal{F}]=P(\mathcal{F})$ for all $\mathcal{F}\in \mathcal{F}$.
• Let $(\mathcal{A},\mathcal{F})$ and $(\mathcal{B},\mathcal{G})$ be measurable spaces, which we refer to as the input and output spaces, respectively, with $\mathcal{A}$ and $\mathcal{B}$ referred to as the input and output alphabets respectively. $P_{Y|X}:\mathcal{A}\to \mathcal{B}$ denotes a random transformation from $\mathcal{A}$ to $\mathcal{B}$, i.e., for any $x\in \mathcal{A}$, $P_{Y|X=x}(·)$ is a probability measure on $(\mathcal{B},\mathcal{G})$, and for any $B\in \mathcal{G}$, $P_{Y|X=·}(B)$ is an $\mathcal{F}$-measurable function. For brevity, we will usually drop mention of the underlying $\sigma$-fields. If P is a probability measure on $\mathcal{A}$ and $P_{Y|X}:\mathcal{A}\to \mathcal{B}$ is a random transformation, the corresponding joint probability measure on $\mathcal{A}\times \mathcal{B}$ is denoted by $P{P}_{Y|X}$ (or, interchangeably, $P_{Y|X}P$). The notation $P\to P_{Y|X}\to Q$ indicates that the output marginal of the joint probability measure $P{P}_{Y|X}$ is denoted by Q.
• The relative information ${ı}_{P\parallel Q}(x)$ between two probability measures P and Q on the same measurable space such that $P\ll Q$ is defined as

  $$\begin{array}{c}\hfill {ı}_{P\parallel Q}(x)=log\frac{\mathrm{d}P}{\mathrm{d}Q}(x),\end{array}$$

  (21)

  where $\frac{\mathrm{d}P}{\mathrm{d}Q}$ is the Radon-Nikodym derivative of P with respect to Q. The relative entropy is

  $$\begin{array}{c}\hfill D(P\parallel Q)=\mathbb{E}\left[{ı}_{P_X\parallel Q_X}(X)\right],X\sim P.\end{array}$$

  (22)
• Given $P_X\to P_{Y|X}\to P_Y$, the information density is defined as

  $$\begin{array}{c}\hfill {ı}_{X;Y}(a;b)={ı}_{P_{Y|X=a}\parallel P_Y}(b),(a,b)\in \mathcal{A}\times \mathcal{B}.\end{array}$$

  (23)
• Fix $\alpha >0$, $P_{Y|X}:\mathcal{A}\to \mathcal{B}$, and a probability measure $P_X$ on $\mathcal{A}$. Then, the output probability measure $P_{Y_{\left[\alpha \right]}}$ is called the $\alpha$-response to $P_X$ if

  $$\begin{array}{c}\hfill {ı}_{Y_{\left[\alpha \right]}\parallel Y}(y)=\frac{1}{\alpha }log\mathbb{E}[exp(\alpha {ı}_{X;Y}(X;y)-{\kappa }_{\alpha })],X\sim P_X,\end{array}$$

  (24)

  where $P_X\to P_{Y|X}\to P_Y$, and ${\kappa }_{\alpha }$ is a scalar that guarantees that $P_{Y_{\left[\alpha \right]}}$ is a probability measure. For notational convenience, we omit the dependence of ${\kappa }_{\alpha }$ on $P_X$ and $P_{Y|X}$. Equivalently, if $p_{Y_{\left[\alpha \right]}}$ and $p_{Y|X}$ denote the densities with respect to some dominating measure, then (24) becomes

  $$\begin{array}{c}\hfill p_{Y_{\left[\alpha \right]}}(y)=exp\left(-\frac{{\kappa }_{\alpha }}{\alpha }\right){\mathbb{E}}^{\frac{1}{\alpha }}\left[p_{Y|X}^{\alpha }(y|X)\right],X\sim P_X.\end{array}$$

  (25)

  In particular, the 1-response to $P_X$ is $P_Y$. In [26], the $\alpha$-response to $P_X$ is dubbed the order $\alpha$ Rényi mean for prior $P_X$.
• Given two probability measures P and Q on the same measurable space and a scalar $\alpha \in (0,1)\cup (1,\infty )$, the Rényi divergence of order $\alpha$ between P and Q is defined as [1]

  $$\begin{array}{c}\hfill D_{\alpha }(P\parallel Q)=\frac{1}{\alpha -1}log{\int }_{\mathcal{A}}{p}^{\alpha }{q}^{1-\alpha }\mathrm{d}\mu ,\end{array}$$

  (26)

  where p and q are the Radon-Nikodym derivatives of P and Q, respectively, with respect to a common dominating $\sigma$-finite measure $\mu$. We define $D_1(P\parallel Q)=D(P\parallel Q)$ as this coincides with the limit from the left at $\alpha =1$. It is also the limit from the right whenever $D_{\alpha }(P\parallel Q)<\infty$ for some $\alpha >1$. The cases $\alpha =0$ and $\alpha =\infty$ can be defined by taking the corresponding limits. In this work, we only focus on the simple orders of $\alpha$, i.e., $\alpha \in (0,1)\cup (1,\infty )$. As we saw in (1), if $P\ll Q$, then (26) becomes

  $$\begin{array}{cc}\hfill D_{\alpha }(P\parallel Q)& =\frac{1}{\alpha -1}log\left(\mathbb{E}\left[exp\left(\alpha {ı}_{P\parallel Q}(W)\right)\right]\right),W\sim Q\hfill \end{array}$$

  (27)

  $$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\left(\mathbb{E}\left[exp\left((\alpha -1){ı}_{P\parallel Q}(V)\right)\right]\right),V\sim P\hfill \end{array}$$

  (28)
• If $\alpha \in (0,1)\cup (1,\infty )$, then the binary Rényi divergence of order $\alpha$ is given by

  $$\begin{array}{cc}\hfill d_{\alpha }(p\parallel q)& =D_{\alpha }([p1-p]\parallel [q1-q])\hfill \end{array}$$

  (29)

  $$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\left(p^{\alpha }{q}^{1-\alpha }+{(1-p)}^{\alpha }{(1-q)}^{1-\alpha }\right).\hfill \end{array}$$

  (30)

  Note that

  $$\begin{array}{c}\hfill \underset{\alpha \to 1}{lim}{d}_{\alpha }\left(p\parallel {\displaystyle \frac{1}{2}}\right)=log2-h(p),\end{array}$$

  (31)

  where the usual binary entropy function is denoted by $h(x)=xlog\frac{1}{x}+(1-x)log\frac{1}{1-x}$. Given $(p_0,p_1)\in {(0,1)}^2$, $p_0\ne p_1$, the solution to $d_{\alpha }(p_0\parallel q)=d_{\alpha }(p_1\parallel q)$ is

  $$\begin{array}{c}\hfill q={\left(1+{\left(\frac{p_0^{\alpha }-p_1^{\alpha }}{{\left(1-p_1\right)}^{\alpha }-{\left(1-p_0\right)}^{\alpha }}\right)}^{\frac{1}{1-\alpha }}\right)}^{-1}.\end{array}$$

  (32)
• $D_{\alpha }(P\parallel Q)\ge 0$, with equality only if $P=Q$.
• $D_{\alpha }(P\parallel Q)$ is monotonically increasing with $\alpha$.
• While we may have $D(P\parallel Q)=\infty$ and $P\ll Q$ simultaneously, $D_{\alpha }(P\parallel Q)=\infty$ for any $\alpha \in (0,1)$ is equivalent to P and Q being orthogonal. Conversely, if for some $\alpha >1$, $D_{\alpha }(P\parallel Q)<\infty$, then $P\ll Q$.
• The Rényi divergence satisfies the data-processing inequality. If $P_X\to P_{Y|X}\to P_Y$ and $Q_X\to P_{Y|X}\to Q_Y$, then

  $$\begin{array}{c}\hfill D_{\alpha }(P_X\parallel Q_X)\ge D_{\alpha }(P_Y\parallel Q_Y).\end{array}$$

  (33)
• Gilardoni [38] gave a strengthened Pinsker’s inequality upper bounding the square of the total variation distance by

  $$\begin{array}{cc}\hfill {|P-Q|}^2& \le \underset{\alpha \in (0,1]}{inf}\frac{1}{2\alpha }{D}_{\alpha }(P\parallel Q)\hfill \end{array}$$

  (34)

  $$\begin{array}{cc}\hfill & \le \underset{\alpha >0}{inf}\frac{1}{2min\{\alpha ,1\}}{D}_{\alpha }(P\parallel Q),\hfill \end{array}$$

  (35)

  where we have used the monotonicity in $\alpha$ of the Rényi divergence.
• The Rényi divergence is lower semicontinuous in the topology of setwise convergence, i.e., if for every event $A\in \mathcal{F}$, $P_n(A)\to P(A)$, and $Q_n(A)\to Q(A)$, then

  $$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}{D}_{\alpha }(P_n\parallel Q_n)\ge D_{\alpha }(P\parallel Q),\alpha \in (0,\infty ].\end{array}$$

  (36)

  In particular, note that (36) holds if $|P_n-Q_n|\to 0$.
• In the theory of robust lossless source coding [22,25] the following scalar, called the $\alpha$-minimax redundancy of $P_{Y|X}$, is an important measure of the worst-case redundancy penalty that ensues when the encoder only knows that the data is generated according to one of the probability measures in the collection $\{P_{Y|X=x},x\in \mathcal{A}\}$:

  $$\begin{array}{c}\hfill R_{\alpha }=\underset{Q_Y}{inf}\underset{x\in \mathcal{A}}{sup}{D}_{\alpha }(P_{Y|X=x}\parallel Q_Y),\end{array}$$

  (37)

  where the infimum is over all the probability measures on $\mathcal{B}$.
• Given input distribution $P_X$ and random transformations $P_{Y|X},Q_{Y|X}$, the conditional Rényi divergence of order $\alpha \in (0,1)\cup (1,\infty )$ is

  $$\begin{array}{cc}\hfill D_{\alpha }(P_{Y|X}\parallel Q_{Y|X}|P_X)& =D_{\alpha }(P_X{P}_{Y|X}\parallel P_X{Q}_{Y|X}).\hfill \end{array}$$

  (38)
  Although (38) also holds for the familiar $\alpha =1$ case, in general the conditional Rényi divergence is not the arithmetic average of $D(P_{Y|X=x}\parallel Q_Y)$ with respect to $P_X$ if $\alpha \ne 1$. Instead it’s a generalized mean, or a scaled cumulant generating function evaluated at $\alpha -1$. Specifically, if $X\sim P_X$, then

  $$\begin{array}{cc}\hfill D_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)& =\frac{1}{\alpha -1}log\mathbb{E}\left[exp\left((\alpha -1)D_{\alpha }(P_{Y|X}(·|X)\parallel Q_Y)\right)\right].\hfill \end{array}$$

  (39)
  Regardless of whether $\alpha \in (0,1)$ or $\alpha \in (1,\infty )$, (39) implies that

  $$\begin{array}{cc}\hfill D_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)& \le \underset{x\in \mathcal{A}}{sup}{D}_{\alpha }(P_{Y|X=x}\parallel Q_Y)\hfill \end{array}$$

  (40)

  $$\begin{array}{cc}\hfill & =\underset{P_X}{sup}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)\hfill \end{array}$$

  (41)

  with the supremum in (41) over all input probability measures.
• The key additive decomposition formula for the mutual information (9) has a nice counterpart for the $\alpha$-mutual information [27]. Let $P_X\to P_{Y|X}\to P_Y$ and $Q_Y$ be an arbitrary probability measure on $\mathcal{B}$ such that $P_Y\ll Q_Y$. Then, it is easy to verify that

  $$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}|P_X)=D_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)-D_{\alpha }(P_{Y_{\left[\alpha \right]}}\parallel Q_Y),\end{array}$$

  (42)

  a relationship noted by Sibson [31] in the discrete case.
• Given $\alpha >0$, $P_X$ and $P_{Y|X}$, the α-mutual information is [27,31]

  $$\begin{array}{cc}\hfill I_{\alpha }(X;Y)& =\underset{Q}{min}{D}_{\alpha }(P_{Y|X}\parallel Q|P_X)\hfill \end{array}$$

  (43)

  $$\begin{array}{cc}\hfill & =D_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}|P_X)\hfill \end{array}$$

  (44)

  $$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\mathbb{E}\left[exp\left((\alpha -1)D_{\alpha }(P_{Y|X}(·|X)\parallel P_{Y_{\left[\alpha \right]}})\right)\right]\hfill \end{array}$$

  (45)

  $$\begin{array}{cc}\hfill & =D_{\alpha }(P_{Y|X}\parallel P_Y|P_X)-D_{\alpha }(P_{Y_{\left[\alpha \right]}}\parallel P_Y),\hfill \end{array}$$

  (46)

  where $P_X\to P_{Y|X}\to P_Y$. It can be checked that the constant in (24) is equal to

  $$\begin{array}{cc}\hfill {\kappa }_{\alpha }& =(\alpha -1)I_{\alpha }(X;Y).\hfill \end{array}$$

  (47)
  Note that $I_1(X;Y)=I(X;Y)$ but, in general, $I_{\alpha }(X;Y)\ne I_{\alpha }(Y;X)$.
• An alternative expression for $\alpha$-mutual information, which will come in handy in our analysis and which does not involve either $P_Y$ or $P_{Y_{\left[\alpha \right]}}$ is obtained by introducing an auxiliary probability measure $P_{\overline{Y}}$ dominating the collection $\{P_{Y|X=u},u\in \mathcal{A}\}$ [27]:

  $$\begin{array}{cc}\hfill I_{\alpha }(X;Y)& =\frac{\alpha }{\alpha -1}log\mathbb{E}\left[{\mathbb{E}}^{\frac{1}{\alpha }}[exp(\alpha {ı}_{X;\overline{Y}}(X;\overline{Y}))|\overline{Y}]\right],(X,\overline{Y})\sim P_X\times P_{\overline{Y}},\hfill \end{array}$$

  (48)

  where

  $$\begin{array}{c}\hfill {ı}_{X;\overline{Y}}(x;y)=log\frac{\mathrm{d}{P}_{Y|X=x}}{\mathrm{d}{P}_{\overline{Y}}}(y).\end{array}$$

  (49)
  As usual, sometimes it is convenient to fix $\sigma$-finite measures ${\mu }_X$ and ${\mu }_Y$ on the input and output spaces which dominate $P_X$ and $\{P_{Y|X=x}:x\in \mathcal{A}\}$, respectively, and denote their densities with respect to the reference measures by

  $$\begin{array}{cc}\hfill p_{Y|X}(y|x)& =\frac{\mathrm{d}{P}_{Y|X=x}}{\mathrm{d}{\mu }_Y}(y),\hfill \end{array}$$

  (50)

  $$\begin{array}{cc}\hfill p_X(x)& =\frac{\mathrm{d}{P}_X}{\mathrm{d}{\mu }_X}(x).\hfill \end{array}$$

  (51)
  Then, we can write $\alpha$-mutual information as

  $$\begin{array}{cc}\hfill I_{\alpha }(P_X,P_{Y|X})& =\frac{\alpha }{\alpha -1}log{\int }_{\mathcal{B}}{\left({\int }_{\mathcal{A}}{p}_{Y|X}^{\alpha }(y|x)p_X(x)\mathrm{d}{\mu }_X(x)\right)}^{\frac{1}{\alpha }}\mathrm{d}{\mu }_Y(y).\hfill \end{array}$$

  (52)
• In the special case of discrete alphabets,

  $$\begin{array}{c}\hfill E_0(\rho ,P_X,P_{Y|X})=\rho I_{\frac{1}{1+\rho }}(X;Y),\end{array}$$

  (53)

  where the left side is the familiar Gallager function defined in [36] for $\rho \in (0,1)$ as

  $$\begin{array}{c}\hfill E_0(\rho ,P_X,P_{Y|X})=-log\sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}{P}_X(x)P_{Y|X}^{\frac{1}{1+\rho }}(y|x)\right)}^{1+\rho }.\end{array}$$

  (54)
• Fix $\alpha >0$, $P_{Y|X}:\mathcal{A}\to \mathcal{B}$, and a collection $\mathcal{P}$ of probability measures on the input space. Then, we denote

  $$\begin{array}{c}\hfill C_{\alpha }(\mathcal{P})=\underset{P_X\in \mathcal{P}}{sup}{I}_{\alpha }(P_X,P_{Y|X}).\end{array}$$

  (55)
  When $\mathcal{P}$ is the set of all input measures we write simply $C_{\alpha }$, dubbed the Rényi capacity in [26]. $C_{\alpha }$ is a measure of the similarity of the family $\{P_{Y|X=x},x\in \mathcal{A}\}$, which plays an important role in the analysis of the fundamental limits of information transmission through noisy channels, particularly in the regime of exponentially small error probability. For a long time (e.g., [39]) the cutoff rate $C_{\frac{1}{2}}$ was conjectured to be the maximal rate for which reliable codes with manageable decoding complexity can be found. The zero-error capacity of the discrete memoryless channel with feedback is equal to either zero or [40]

  $$\begin{array}{cc}\hfill C_{0\mathsf{f}}& =\underset{X}{max}{I}_0(X;Y),\hfill \end{array}$$

  (56)

  depending on whether there is $(a_1,a_2)\in {\mathcal{A}}^2$ such that $P_{\mathsf{Y}|\mathsf{X}}(·|a_1)\perp P_{\mathsf{Y}|\mathsf{X}}(·|a_2)$.
• The related quantity ${max}_{P_X}{I}_{\frac{1}{\alpha }}(P_X,P_{Y|X}^{\alpha })$ arises in the study of the fundamental limits of guessing and task completion under mismatch [41,42].
• While $D(P\parallel Q)$ is convex in the pair $(P,Q)$, the picture for Rényi divergence is somewhat more nuanced:

  (a)

  If $\alpha \in (0,1)$, then $D_{\alpha }(P\parallel Q)$ is convex in $(P,Q)$.

  (b)

  If $\alpha >0$, then $D_{\alpha }(P\parallel Q)$ is convex in Q for all P, (see [4]).

• For any fixed pair $(P_{Y|X},Q_{Y|X})$, $D_{\alpha }(P_{Y|X}\parallel Q_{Y|X}|P_X)$ is concave (resp. convex) in $P_X$ if $\alpha \ge 1$ (resp. $\alpha \in (0,1]$) (see [43]).
• The $\alpha$-mutual information $I_{\alpha }(P_X,P_{Y|X})$ is concave in $P_X$ for any fixed $P_{Y|X}$ and $\alpha >1$ (see [43]). If $\alpha \in (0,1)\cup (1,\infty )$, then the following monotonically increasing function of $I_{\alpha }(P_X,P_{Y|X})$ is concave in $P_X$

  $$\begin{array}{c}\hfill {\Gamma }_{\alpha }\left(I_{\alpha }(P_X,P_{Y|X})\right)=\frac{1}{\alpha -1}{\phi }_{\frac{1}{\alpha }}\left(I_{\alpha }(P_X,P_{Y|X})\right),\end{array}$$

  (57)

  where ${\phi }_{\alpha }(z)=exp(z-\alpha z)$ (see [10,43]).

3. Conditional Rényi Divergence Game

As can be expected from (43), when maximizing $\alpha$-mutual information, for fixed $P_{Y|X}$, with respect to the input probability measure, it is interesting to consider a zero-sum game with payoff function

$$D_{\alpha }(P_{Y|X}\parallel Q|P_X)$$

such that one player tries to maximize it by choosing $P_X\in \mathcal{P}$, where $\mathcal{P}$ is a given collection of input probability measures, and the other player tries to minimize it by choosing the probability measure $Q\in \mathcal{Q}$ on the output space. Balancing simplicity and generality and motivated by applications, while we allow $\mathcal{P}$ to be a proper subset of the set of all input probability measures, we assume that there are no restrictions in the choice of the output probability measure, and therefore $\mathcal{Q}$ stands for the whole collection of probability measures on the output space. This game also arises in the determination of the worst-case redundancy in (37). In Section 3.1 we consider the important special case in which there exists an input distribution that attains the supremum in (55). In the more general scenario in which the supremum may not be achieved, we cannot identify a saddle point but we can indeed swap sup and min as we show in Section 3.2.

3.1. Saddle point

We begin by showing that the maximal $\alpha$-mutual information input distribution and its $\alpha$-response form a saddle point.

Theorem 1.

Let $\mathcal{P}$ be a convex set of probability distributions on $\mathcal{A}$ and $\mathcal{Q}$ be the set of all probability distributions on $\mathcal{B}$. Let $\alpha \in (0,1)\cup (1,\infty )$. Suppose that there exists some $P_X^{\ast }\in \mathcal{P}$ such that

$$\begin{array}{c}\hfill I_{\alpha }(P_X^{\ast },P_{Y|X})=\underset{P_X\in \mathcal{P}}{max}{I}_{\alpha }(P_X,P_{Y|X})<\infty ,\end{array}$$

(58)

and denote the α-response to $P_X^{\ast }$ by $P_{Y_{\left[\alpha \right]}}^{\ast }$. Then, for any $(P_X,Q_Y)\in \mathcal{P}\times \mathcal{Q}$,

$$\begin{array}{cc}\hfill D_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }|P_X)& \le D_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }|P_X^{\ast })\hfill \end{array}$$

(59)

$$\begin{array}{cc}\hfill & \le D_{\alpha }(P_{Y|X}\parallel Q_Y|P_X^{\ast }).\hfill \end{array}$$

(60)

Conversely, if $(P_X^{\ast },P_{Y_{\left[\alpha \right]}}^{\ast })$ is a saddle point of $D_{\alpha }(P_{Y|X}\parallel ·|·)$, namely, (59)–(60) are satisfied, then $P_X^{\ast }$ maximizes the α-mutual information.

Remark 1.

Assuming that $\mathcal{P}$ includes ${\delta }_x$ (unit mass at $x\in \mathcal{A}$), (59) implies that for any $x\in \mathcal{A}$,

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X=x}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })\le \underset{P_X\in \mathcal{P}}{max}{I}_{\alpha }(P_X,P_{Y|X}).\end{array}$$

(61)

We can easily obtain corollaries to Theorem 1 that elucidate useful properties of the saddle point.

Corollary 1.

Let $\alpha \in (0,1)\cup (1,\infty )$. Under the assumptions in Theorem 1, for any $P_X\in \mathcal{P}$, we have

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y_{\left[\alpha \right]}}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })\le I_{\alpha }(P_X^{\ast },P_{Y|X})-I_{\alpha }(P_X,P_{Y|X})<\infty ,\end{array}$$

(62)

where $P_{Y_{\left[\alpha \right]}}$ is the α-response to $P_X$. Moreover, $P_{Y_{\left[\alpha \right]}}=P_{Y_{\left[\alpha \right]}}^{\ast }$ if, in addition to $P_X^{\ast }$, $P_X$ also attains $C_{\alpha }(\mathcal{P})={max}_{P_X\in \mathcal{P}}{I}_{\alpha }(P_X,P_{Y|X})$.

Proof of Corollary 1.

For any $P_X\in \mathcal{P}$,

$$\begin{array}{cc}\hfill I_{\alpha }(P_X,P_{Y|X})& =D_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}|P_X)\hfill \end{array}$$

(63)

$$\begin{array}{cc}\hfill & =D_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }|P_X)-D_{\alpha }(P_{Y_{\left[\alpha \right]}}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })\hfill \end{array}$$

(64)

$$\begin{array}{cc}\hfill & \le D_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }|P_X^{\ast })-D_{\alpha }(P_{Y_{\left[\alpha \right]}}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })\hfill \end{array}$$

(65)

$$\begin{array}{cc}\hfill & =I_{\alpha }(P_X^{\ast },P_{Y|X})-D_{\alpha }(P_{Y_{\left[\alpha \right]}}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }),\hfill \end{array}$$

(66)

where (64) and (65) follow from (42) and (59), respectively. Since Rényi divergence is nonnegative, $D_{\alpha }(P_{Y_{\left[\alpha \right]}}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })=0$ if $P_X$ also attains $C_{\alpha }(\mathcal{P})$.  □

Therefore, Corollary 1 implies that the $\alpha$-responses to all the maximal $\alpha$-mutual information input distributions must be identical. Moreover, if $\alpha >1$, then every $\alpha$-response to any input distribution satisfies $P_{Y_{\left[\alpha \right]}}\ll P_{Y_{\left[\alpha \right]}}^{\ast }$.

If $\mathcal{P}$ is the space of all probability distributions on $\mathcal{A}$, then we can get the following corollary.

Corollary 2.

Unconstrained maximization of α-mutual information. Suppose that $\alpha \in (0,1)\cup (1,\infty )$ and $\mathcal{P}$ contains all probability mass functions on the discrete alphabet $\mathcal{A}$. Fix $P_{Y|X}:\mathcal{A}\to \mathcal{B}$. For any input distribution ${\overline{P}}_X$, denote its support by ${\overline{\mathcal{A}}}_X\subset \mathcal{A}$ and the corresponding α-response by ${\overline{P}}_{Y_{\left[\alpha \right]}}$.

A necessary and sufficient condition for ${\overline{P}}_X$ to achieve ${max}_{X}I(X;Y)<\infty$ is

$$\begin{array}{c}\hfill \underset{a\in {\overline{\mathcal{A}}}_X}{max}{D}_{\alpha }(P_{Y|X=a}\parallel {\overline{P}}_{Y_{\left[\alpha \right]}})=\underset{a\in {\overline{\mathcal{A}}}_X}{min}{D}_{\alpha }(P_{Y|X=a}\parallel {\overline{P}}_{Y_{\left[\alpha \right]}})\ge \underset{a\in {\overline{\mathcal{A}}}_X^c}{max}{D}_{\alpha }(P_{Y|X=a}\parallel {\overline{P}}_{Y_{\left[\alpha \right]}}).\end{array}$$

(67)

Proof of Corollary 2.

• ${max}_X{I}_{\alpha }(X;Y)=I_{\alpha }({\overline{P}}_X,P_{Y|X})\Rightarrow (67)$: Regardless of whether $\alpha >1$ or $\alpha <1$, we see from (45) that if there exists some $x_0\in {\overline{\mathcal{A}}}_X$ such that

  $$\begin{array}{c}\hfill D_{\alpha }\left(P_{Y|X=x_0}\parallel {\overline{P}}_{Y_{\left[\alpha \right]}}\right)<\underset{P_X\in \mathcal{P}}{max}{I}_{\alpha }(P_X,P_{Y|X}),\end{array}$$

  (68)

  then $I_{\alpha }({\overline{P}}_X,P_{Y|X})<{max}_{P_X\in \mathcal{P}}{I}_{\alpha }(P_X,P_{Y|X})$, which contradicts the assumed optimality of ${\overline{P}}_X$. Moreover, if there exists some $x_0\in {\overline{\mathcal{A}}}_X^c$ such that (68) holds with the strict inequality reversed, then (59) would be violated, again contradicting the assumed optimality of ${\overline{P}}_X$.
• $(67)\Rightarrow {max}_X{I}_{\alpha }(X;Y)=I_{\alpha }({\overline{P}}_X,P_{Y|X})$: Again, we see from (45) that if (67) is satisfied, then (59) is satisfied. Since ${\overline{P}}_{Y_{\left[\alpha \right]}}$ is the $\alpha$-response to ${\overline{P}}_X$, (59) is also satisfied, and the converse part in Theorem 1 results in the optimality of ${\overline{P}}_X$.

 □

Remark 2.

According to Corollary 2, if some input distribution $P_X^{\ast }$ achieves $C_{\alpha }$, we know the α-response output distribution $P_{Y_{\left[\alpha \right]}}^{\ast }$ is equidistant in $D_{\alpha }(·\parallel P_{Y_{\left[\alpha \right]}}^{\ast })$ to any of the output distributions in the collection

$$\begin{array}{c}\hfill \mathcal{S}=\{P_{Y|X=x},P_X^{\ast }(x)>0\}\subset \mathcal{Q}.\end{array}$$

(69)

Moreover, we know that the optimal α-response output distribution is actually unique even if there exist several optimal input distributions. So the key is to find the unique centroid of $\mathcal{S}$ when the distance is measured by the Rényi divergence. In contrast to the maximization of the mutual information, the optimal α-response output distribution is no longer a mixture of the conditional output distributions.

Remark 3.

Corollary 2 enables us to recover Gallager’s finite alphabet result in Theorem 5.6.5 of [44], which characterizes the maximal α-mutual information input distribution if $\alpha \in (0,1)$ when both $\mathcal{A}$ and $\mathcal{B}$ are finite. The optimal input distribution $P_X^{\ast }$ must satisfy the following condition:

$$\begin{array}{c}\hfill \sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}{P}_{Y|X}^{\alpha }(y|x)P_X^{\ast }(x)\right)}^{\frac{1-\alpha }{\alpha }}{P}_{Y|X}^{\alpha }(y|u)\ge \sum _{y\in \mathcal{B}}{\left(\sum _{x\in \mathcal{A}}{P}_{Y|X}^{\alpha }(y|x)P_X^{\ast }(x)\right)}^{\frac{1}{\alpha }},\end{array}$$

(70)

with equality for all u such that $P_X^{\ast }(u)>0$. To verify this condition, note that Corollary 2 requires that

$$\begin{array}{c}\hfill exp({\kappa }_{\alpha }^{\ast })=exp\left((\alpha -1)C_{\alpha }\right)\le \sum _{y\in \mathcal{B}}{P}_{Y|X=u}^{\alpha }(y){\left(P_{Y_{\left[\alpha \right]}}^{\ast }(y)\right)}^{1-\alpha },\end{array}$$

(71)

with equality if $P_X^{\ast }(u)>0$, and where ${\kappa }_{\alpha }^{\ast }$ stands for the normalizing constant in (24) with $P_X\leftarrow P_X^{\ast }$. Upon substitution of (25) with $P_X\leftarrow P_X^{\ast }$, we obtain (70). The assumption of finite output alphabet can be easily dispensed with to obtain the more general optimality condition

$$\begin{array}{c}\hfill \mathbb{E}\left[{\Psi }_{\alpha }^{1-\alpha }(Y^{\ast })exp\left(\alpha {ı}_{X;Y}^{\ast }(u;Y^{\ast })\right)\right]\ge \mathbb{E}\left[{\Psi }_{\alpha }(Y^{\ast })\right]\end{array}$$

(72)

with equality for all u such that $P_X^{\ast }(u)>0$. In (72), ${ı}_{X;Y}^{\ast }$ stands for the information density corresponding to $P_X^{\ast }\to P_{Y|X}\to P_Y^{\ast }$ and

$$\begin{array}{c}\hfill {\Psi }_{\alpha }(y)={\mathbb{E}}^{\frac{1}{\alpha }}\left[exp\left(\alpha {ı}_{X;Y}^{\ast }(X^{\ast };y)\right)\right],X^{\ast }\sim P_X^{\ast }\end{array}$$

(73)

If $\alpha >1$, condition (72) holds with the inequality reversed.

Remark 4.

When $\mathcal{B}$ is finite, it was shown in [2,4,25] that for any $\alpha \in [0,\infty ]$,

$$\begin{array}{c}\hfill C_{\alpha }=R_{\alpha },\end{array}$$

(74)

where $R_{\alpha }$ is defined in (37). This is now established without imposing finiteness conditions, as long as there is an input that achieves the maximal α-mutual information because

$$\begin{array}{cc}\hfill R_{\alpha }& \le C_{\alpha }\hfill \end{array}$$

(75)

$$\begin{array}{cc}\hfill & =\underset{P_X\in \mathcal{P}}{max}\underset{Q_Y\in \mathcal{Q}}{min}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)\hfill \end{array}$$

(76)

$$\begin{array}{cc}\hfill & \le \underset{Q_Y\in \mathcal{Q}}{min}\underset{P_X\in \mathcal{P}}{max}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)\hfill \end{array}$$

(77)

$$\begin{array}{cc}\hfill & \le R_{\alpha },\hfill \end{array}$$

(78)

where (75) follows from particularizing (59) to deterministic $P_X$, and () follows from (40).

3.2. Minimax identity

In this section we drop the assumption that there exists an input probability measure that attains the maximal $\alpha$-mutual information and show that the conditional Rényi divergence still satisfies a minimax identity, even if a saddle point does not exist.

Theorem 2.

Let $\mathcal{P}$ be a convex set of probability distributions on $\mathcal{A}$ and $\mathcal{Q}$ be the set of all probability distributions on $\mathcal{B}$. We have the minimax equality:

$$\begin{array}{cc}\hfill C_{\alpha }(\mathcal{P})& =\underset{P_X\in \mathcal{P}}{sup}\underset{Q_Y\in \mathcal{Q}}{min}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)\hfill \end{array}$$

(79)

$$\begin{array}{cc}\hfill & =\underset{Q_Y\in \mathcal{Q}}{min}\underset{P_X\in \mathcal{P}}{sup}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X).\hfill \end{array}$$

(80)

Furthermore, if $C_{\alpha }(\mathcal{P})<\infty$, then there exists a unique element in $\mathcal{Q}$ attaining the minimum in (80).

The assumption of convexity in Theorem 2 is not superfluous, as the following example illustrates.

Example 1.

Let $\mathcal{A}=\mathcal{B}=\mathbb{N}$ and $Y=X+N$, where N is a geometric random variable on the nonnegative integers with positive mean and independent of X. Let $\mathcal{P}$ be the non-convex set of all the deterministic probability measures on $\mathcal{A}$. In this case, the left side of (80) is zero, while the right side is infinity. To see this, note that for any $Q_Y\in \mathcal{Q}$ and $n\in \mathbb{N}$, it follows from the data processing inequality applied to the binary deterministic transformation $1\{Y\ge n\}$ that

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X=n}\parallel Q_Y)\ge d_{\alpha }\left(1\parallel Q_Y(\{n+1,\dots \})\right),\end{array}$$

(81)

whose right side diverges as $n\to \infty$. Therefore, for any $Q_Y\in \mathcal{Q}$, (39) results in

$$\begin{array}{c}\hfill \underset{P_X\in \mathcal{P}}{sup}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)=\infty .\end{array}$$

(82)

Continuing with the theme in Remark 4, Theorem 2 extends the validity of $R_{\alpha }=C_{\alpha }$ without requiring the existence of the maximal $\alpha$-mutual information input distribution. It was conjectured in [2] (and proved in [26]) that for $\alpha \in (0,\infty )$, if $R_{\alpha }<\infty$ and $\mathcal{B}$ is finite or countable, there exists a unique redundancy-achieving distribution

$$\begin{array}{c}\hfill Q_Y^{\ast }=arg\underset{Q_Y}{min}\underset{x\in \mathcal{A}}{sup}{D}_{\alpha }(P_{Y|X=x}\parallel Q_Y)\end{array}$$

(83)

and for all probability measures $Q_Y$ on the output space,

$$\begin{array}{c}\hfill \underset{x\in \mathcal{A}}{sup}{D}_{\alpha }(P_{Y|X=x}\parallel Q_Y)\ge C_{\alpha }+D_{\alpha }(Q_Y^{\ast }\parallel Q_Y).\end{array}$$

(84)

We can prove the conjecture easily with the help of Theorem 2.

Proof. 

Let $\mathcal{P}$ be the convex set of all probability measures on $\mathcal{A}$. Since $C_{\alpha }=R_{\alpha }<\infty$, by Theorem 2, we know there exists a unique $P_{Y_{\left[\alpha \right]}}^{\ast }$ such that ${sup}_{P_X\in \mathcal{P}}{D}_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }|P_X)=C_{\alpha }$, which implies that $P_{Y_{\left[\alpha \right]}}^{\ast }$ is precisely the unique redundancy-achieving distribution in (83). Moreover, as shown in the proof of Theorem 2, we can find a sequence ${\left\{P_{X_n}\right\}}_{n\ge 1}$ in $\mathcal{P}$ such that $I_{\alpha }(P_{X_n},P_{Y|X})\to C_{\alpha }$ as $n\to \infty$ and such that the corresponding $\alpha$-responses $P_{Y_{n\left[\alpha \right]}}$ converge to $P_{Y_{\left[\alpha \right]}}^{\ast }$ in the total variation metric. Pick an arbitrary $Q_Y\in \mathcal{Q}$. If $\alpha >1$ and $P_{Y|X=x}\cancel{\ll }{Q}_Y$ for some $x\in \mathcal{A}$, then ${sup}_{x\in \mathcal{A}}{D}_{\alpha }(P_{Y|X=x}\parallel Q_Y)=\infty$ and (84) holds. Otherwise, by (42) we always have

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)=D_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }|P_X)+D_{\alpha }(P_{Y_{\left[\alpha \right]}}^{\ast }\parallel Q_Y).\end{array}$$

(85)

For any $n\ge 1$, since $\mathcal{P}$ includes all probability measures on $\mathcal{A}$, we have

$$\begin{array}{cc}\hfill \underset{x\in \mathcal{A}}{sup}{D}_{\alpha }(P_{Y|X=x}\parallel Q_Y)& =\underset{P_X\in \mathcal{P}}{sup}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)\hfill \end{array}$$

(86)

$$\begin{array}{cc}\hfill & \ge D_{\alpha }(P_{Y|X}\parallel Q_Y|P_{X_n})\hfill \end{array}$$

(87)

$$\begin{array}{cc}\hfill & =D_{\alpha }(P_{Y|X}\parallel P_{Y_{n\left[\alpha \right]}}|P_{X_n})+D_{\alpha }(P_{Y_{n\left[\alpha \right]}}\parallel Q_Y)\hfill \end{array}$$

(88)

$$\begin{array}{cc}\hfill & =I_{\alpha }(P_{X_n},P_{Y|X})+D_{\alpha }(P_{Y_{n\left[\alpha \right]}}\parallel Q_Y),\hfill \end{array}$$

(89)

where (88) is due to (42). Taking the limit as $n\to \infty$, the lower-semicontinuity of the Rényi divergence ensures that

$$\begin{array}{c}\hfill \underset{x\in \mathcal{A}}{sup}{D}_{\alpha }(P_{Y|X=x}\parallel Q_Y)\ge C_{\alpha }+D_{\alpha }(P_{Y_{\left[\alpha \right]}}^{\ast }\parallel Q_Y),\end{array}$$

(90)

and therefore the sought-after $Q_Y^{\ast }$ is none other that $P_{Y_{\left[\alpha \right]}}^{\ast }$, the unique maximal $\alpha$-mutual information output distribution.  □

4. Finding ${\mathit{C}}_{\mathit{\alpha }}$

In this section, we present a number of examples to illustrate how the results in Section 3 can be used to maximize the $\alpha$-mutual information with respect to the input distribution. It is instructive to contrast the present approach with the maximization of $\alpha$-mutual information invoking the KKT conditions, which is feasible in both the case $\alpha >1$ in which the functional is concave with respect to the input distribution, and the case $\alpha \in (0,1)$ in which a monotonically increasing function of $\alpha$-mutual information is concave. Simple finite-alphabet examples of such approach can be found in [44] when dealing with the $E_0$ functional in (54). Thanks to Theorem 1 it is possible to avoid taking derivatives of any functionals.

Example 2 (Binary symmetric channel).

Let the input and output alphabet be $\mathcal{A}=\mathcal{B}=\{0,1\}$ and the random transformation be

$$\begin{array}{c}\hfill P_{Y|X}=\left[\begin{array}{cc}1-\delta & \delta \\ \delta & 1-\delta \end{array}\right],\delta \in [0,1].\end{array}$$

(91)

Let’s try the input distribution $P_X^{\ast }(0)=P_X^{\ast }(1)=0.5$. Then, according to (25), the α-response output distribution is also equiprobable $P_{Y_{\left[\alpha \right]}}^{\ast }(0)=P_{Y_{\left[\alpha \right]}}^{\ast }(1)=0.5$. Since

$$\begin{array}{cc}\hfill D_{\alpha }(P_{Y|X=0}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })& =D_{\alpha }(P_{Y|X=1}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })=d_{\alpha }\left(\delta \parallel \frac{1}{2}\right)\hfill \end{array}$$

(92)

the conditions of Corollary 2 are met, $P_X^{\ast }$ attains the maximal α-mutual information and therefore,

$$\begin{array}{c}\hfill C_{\alpha }=d_{\alpha }\left(\delta \parallel \frac{1}{2}\right)=log2+\frac{1}{\alpha -1}log\left({\delta }^{\alpha }+{(1-\delta )}^{\alpha }\right),\end{array}$$

(93)

which satisfies, according to (31)

$$\begin{array}{cc}\hfill \underset{\alpha \to 1}{lim}{C}_{\alpha }& =log2-h(\alpha ),\hfill \end{array}$$

(94)

$$\begin{array}{cc}\hfill \underset{\alpha \to \infty }{lim}{C}_{\alpha }& =log\left(2max\{\delta ,1-\delta \}\right).\hfill \end{array}$$

(95)

Example 3 (Binary erasure channel.).

Let the input/output alphabets be $\mathcal{A}=\{0,1\}$ and $\mathcal{B}=\{0,\mathrm{e},1\}$, and the random transformation be

$$\begin{array}{c}\hfill P_{Y|X}=\left[\begin{array}{cc}1-\delta & 0\\ \delta & \delta \\ 0& 1-\delta \end{array}\right],\delta \in [0,1].\end{array}$$

(96)

(Departing from usual practice, columns/rows represent input/output letters respectively, i.e., probability vectors are column vectors, although for typographical convenience we show them as row vectors in the text.)

The α-response output distribution to $P_X^{\ast }(0)=P_X^{\ast }(1)=0.5$ is

$$\begin{array}{cc}\hfill P_{Y_{\left[\alpha \right]}}^{\ast }(0)=P_{Y_{\left[\alpha \right]}}^{\ast }(1)& =\frac{1-\delta }{\delta 2^{\frac{1}{\alpha }}+2(1-\delta )},\hfill \end{array}$$

(97)

$$\begin{array}{cc}\hfill P_{Y_{\left[\alpha \right]}}^{\ast }(\mathrm{e})& =\frac{\delta 2^{\frac{1}{\alpha }}}{\delta 2^{\frac{1}{\alpha }}+2(1-\delta )}.\hfill \end{array}$$

(98)

By symmetry,

$$\begin{array}{cc}\hfill C_{\alpha }& =D_{\alpha }(P_{Y|X=0}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })=D_{\alpha }(P_{Y|X=1}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })\hfill \end{array}$$

(99)

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\left({(1-\delta )}^{\alpha }{\left(\frac{(1-\delta )}{\delta 2^{\frac{1}{\alpha }}+2(1-\delta )}\right)}^{1-\alpha }+{\delta }^{\alpha }{\left(\frac{\delta 2^{\frac{1}{\alpha }}}{\delta 2^{\frac{1}{\alpha }}+2(1-\delta )}\right)}^{1-\alpha }\right)\hfill \end{array}$$

(100)

$$\begin{array}{c}=\frac{1}{\alpha -1}log\left(\frac{1-\delta +2^{\frac{1-\alpha }{\alpha }}\delta }{{\left(\delta 2^{\frac{1}{\alpha }}+2(1-\delta )\right)}^{1-\alpha }}\right)\hfill \end{array}$$

(101)

$$\begin{array}{c}=\frac{1}{\alpha -1}log\left(\frac{2^{\frac{\alpha -1}{\alpha }}\left(1-\delta \right)+\delta }{{\left(\delta +2^{\frac{\alpha -1}{\alpha }}(1-\delta )\right)}^{1-\alpha }}\right)\hfill \end{array}$$

(102)

$$\begin{array}{c}=\frac{\alpha }{\alpha -1}log\left((1-\delta )2^{\left(1-\frac{1}{\alpha }\right)}+\delta \right),\hfill \end{array}$$

(103)

which satisfies (in bits)

$$\begin{array}{cc}\hfill \underset{\alpha \to 1}{lim}{C}_{\alpha }& =1-\delta ,\hfill \end{array}$$

(104)

$$\begin{array}{cc}\hfill \underset{\alpha \to \infty }{lim}{C}_{\alpha }& ={log}_2(2-\delta ).\hfill \end{array}$$

(105)

Example 4 (Binary asymmetric channel).

Let the input and output alphabet be $\mathcal{A}=\mathcal{B}=\{0,1\}$ and the random transformation be

$$\begin{array}{c}\hfill P_{Y|X}=\left[\begin{array}{cc}1-{\delta }_0& {\delta }_1\\ {\delta }_0& 1-{\delta }_1\end{array}\right],({\delta }_0,{\delta }_1)\in {[0,1]}^2.\end{array}$$

(106)

If ${\delta }_0+{\delta }_1=1$, then $I_{\alpha }(X;Y)=0$ for any input distribution. We will assume ${\delta }_0+{\delta }_1<1$. Otherwise, we can just relabel the output alphabet $(0,1)\leftarrow (1,0)$, or equivalently $({\delta }_0,{\delta }_1)\leftarrow (1-{\delta }_0,1-{\delta }_1)$. The condition

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X=0}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })=D_{\alpha }(P_{Y|X=1}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })\end{array}$$

(107)

is now $d_{\alpha }\left(1-{\delta }_0\parallel P_{Y_{\left[\alpha \right]}}^{\ast }(0)\right)=d_{\alpha }\left({\delta }_1\parallel P_{Y_{\left[\alpha \right]}}^{\ast }(0)\right)$, which, in view of (32) yields

$$\begin{array}{c}\hfill P_{Y_{\left[\alpha \right]}}^{\ast }(0)={\left(1+{\left(\frac{{(1-{\delta }_0)}^{\alpha }-{\delta }_1^{\alpha }}{{(1-{\delta }_1)}^{\alpha }-{\delta }_0^{\alpha }}\right)}^{\frac{1}{1-\alpha }}\right)}^{-1}.\end{array}$$

(108)

We can verify from (25) that this corresponds to the α-response to $P_X^{\ast }(1)=p^{\ast }=1-P_X^{\ast }(0)$, where $p^{\ast }\in [0,1]$ is the solution to

$$\begin{array}{c}\hfill \frac{{\delta }_0^{\alpha }(1-p)+{(1-{\delta }_1)}^{\alpha }p}{{(1-{\delta }_0)}^{\alpha }(1-p)+{\delta }_1^{\alpha }p}={\left(\frac{{(1-{\delta }_0)}^{\alpha }-{\delta }_1^{\alpha }}{{(1-{\delta }_1)}^{\alpha }-{\delta }_0^{\alpha }}\right)}^{\frac{\alpha }{1-\alpha }}.\end{array}$$

(109)

Then,

$$\begin{array}{cc}\hfill C_{\alpha }& =D_{\alpha }(P_{Y|X=0}\parallel P_{Y_{\left[\alpha \right]}}^{\ast })\hfill \end{array}$$

(110)

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha -1}log\left({\left(1-{\delta }_0\right)}^{\alpha }{\left(1-{\delta }_1\right)}^{\alpha }-{\delta }_0^{\alpha }{\delta }_1^{\alpha }\right)+log\left({\left({\left(1-{\delta }_0\right)}^{\alpha }-{\delta }_1^{\alpha }\right)}^{\frac{1}{1-\alpha }}+{\left({\left(1-{\delta }_1\right)}^{\alpha }-{\delta }_0^{\alpha }\right)}^{\frac{1}{1-\alpha }}\right).\hfill \end{array}$$

(111)

which satisfies

$$\begin{array}{cc}\hfill \underset{\alpha \to 1}{lim}{C}_{\alpha }& =log\left(exp\left(\frac{h({\delta }_0)}{1-{\delta }_1-{\delta }_0}\right)+exp\left(\frac{h({\delta }_1)}{1-{\delta }_1-{\delta }_0}\right)\right)-\frac{(1-{\delta }_1)h({\delta }_0)+(1-{\delta }_0)h({\delta }_1)}{1-{\delta }_1-{\delta }_0},\hfill \end{array}$$

(112)

and

$$\begin{array}{cc}\hfill \underset{\alpha \to \infty }{lim}{C}_{\alpha }& =log(2-{\delta }_0-{\delta }_1).\hfill \end{array}$$

(113)

Example 5 (Z channel).

Let the input and output alphabet be $\mathcal{A}=\mathcal{B}=\{0,1\}$ and the random transformation be

$$\begin{array}{c}\hfill P_{Y|X}=\left[\begin{array}{cc}1& \delta \\ 0& 1-\delta \end{array}\right],\delta \in [0,1].\end{array}$$

(114)

Since this is the special case $({\delta }_0,{\delta }_1)=(0,\delta )$ of the binary asymmetric channel we obtain

$$\begin{array}{c}\hfill C_{\alpha }=log\left(1+{\left(\frac{1-{\delta }^{\alpha }}{{(1-\delta )}^{\alpha }}\right)}^{\frac{1}{1-\alpha }}\right).\end{array}$$

(115)

The limit

$$\begin{array}{c}\hfill \underset{\alpha \to 1}{lim}{C}_{\alpha }=log\left(1-{\delta }^{\frac{1}{1-\delta }}+{\delta }^{\frac{\delta }{1-\delta }}\right)\end{array}$$

(116)

coincides with the capacity of the Z-channel originally derived in [45].

The next example illustrates a case in which there are multiple optimal input distributions.

Example 6.

Let $\mathcal{A}=\{0,1,2\}$, $\mathcal{B}=\{0,1,2,3\}$ and the random transformation be

$$\begin{array}{c}\hfill P_{Y|X}=\left[\begin{array}{ccc}\frac{1}{2}-\delta & \delta & \frac{1}{2}-\delta \\ \delta & \frac{1}{2}-\delta & \delta \\ \delta & \frac{1}{2}-\delta & \delta \\ \frac{1}{2}-\delta & \delta & \frac{1}{2}-\delta \end{array}\right],\delta \in \left[0,\frac{1}{2}\right].\end{array}$$

(117)

Let $P_X^0=\left[\frac{1}{2}\frac{1}{2}0\right]$ and $P_X^1=\left[0\frac{1}{2}\frac{1}{2}\right]$. Its easy to verify that the corresponding α-responses are the equiprobable distribution on $\mathcal{B}$. To verify that $P_X^0$ and $P_X^1$ attain the maximal α-mutual information, denote $P_{Y_{\left[\alpha \right]}}^{\ast }=\left[\frac{1}{4}\frac{1}{4}\frac{1}{4}\frac{1}{4}\right]$. For all $x\in \mathcal{A}$, we have

$$\begin{array}{cc}\hfill D_{\alpha }\left(P_{Y|X=x}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }\right)& =D_{\alpha }\left(\left[\delta {\displaystyle \frac{1}{2}}-\delta {\displaystyle \frac{1}{2}}-\delta \delta \right]\parallel \left[{\displaystyle \frac{1}{4}}{\displaystyle \frac{1}{4}}{\displaystyle \frac{1}{4}}{\displaystyle \frac{1}{4}}\right]\right)\hfill \end{array}$$

(118)

$$\begin{array}{cc}\hfill & =\frac{2\alpha -1}{\alpha -1}log2+log\left({\delta }^{\alpha }+{\left(\frac{1}{2}-\delta \right)}^{\alpha }\right)\hfill \end{array}$$

(119)

$$\begin{array}{c}=C_{\alpha },\hfill \end{array}$$

(120)

where (120) follows from Corollary 2.

In the next example $C_{\alpha }$ is constant in $\alpha$.

Example 7 (Additive phase noise).

Let the input and output alphabet be $\mathcal{A}=\mathcal{B}=[0,2\pi )$ and the random transformation be $Y=\left(X+N\right)\mathrm{mod}2\pi$, where N is independent of X and is uniform on the interval $[-{\theta }_0,{\theta }_0]$ with ${\theta }_0\in (0,\pi ]$. Suppose $P_X^{\ast }$ is uniform on $[0,2\pi )$, it is easy to verify that $P_{Y_{\left[\alpha \right]}}^{\ast }$ is also uniform on $[0,2\pi )$. Invoking (26), we obtain

$$\begin{array}{c}\hfill D_{\alpha }\left(P_{Y|X=x}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }\right)=log\frac{\pi }{{\theta }_0},x\in \mathcal{A},\end{array}$$

(121)

which according to Theorem 1 must be equal to $C_{\alpha }$ attained by $P_X^{\ast }$.

Example 8 (Additive Gaussian noise).

Let $\mathcal{A}=\mathcal{B}=\mathbb{R}$, $Y=X+N$, where $N\sim \mathcal{N}(0,{\sigma }^2)$ is independent of X. Fix $\alpha \in (0,1)$ and $P>0$. Suppose that the set, $\mathcal{P}$, of allowable input probability measures on $\mathcal{A}$ consists of those that satisfy

$$\begin{array}{c}\hfill \mathbb{E}\left[{exp}_{\mathrm{e}}\left(-\frac{\alpha (1-\alpha )X^2}{2\left({\alpha }^{2}P+{\sigma }^2\right)}\right)\right]\ge \sqrt{\frac{{\alpha }^{2}P+{\sigma }^2}{\alpha P+{\sigma }^2}}.\end{array}$$

(122)

By completing the square, it is easy to verify that $P_X^{\ast }\sim \mathcal{N}(0,P)$ satisfies (122) with equality. Furthermore, its α-response is $P_{Y_{\left[\alpha \right]}}^{\ast }\sim \mathcal{N}(0,\alpha P+{\sigma }^2)$. To show that $P_X^{\ast }$ does indeed attain $C_{\alpha }(\mathcal{P})$, first we compute

$$\begin{array}{c}\hfill D_{\alpha }\left(P_{Y|X=x}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }\right)=\frac{1}{2}log\left(1+\frac{\alpha P}{{\sigma }^2}\right)-\frac{1}{2\left(1-\alpha \right)}log\left(\frac{\alpha P+{\sigma }^2}{{\alpha }^{2}P+{\sigma }^2}\right)+\frac{1}{2}\frac{\alpha x^2}{{\alpha }^{2}P+{\sigma }^2}log\mathrm{e}.\end{array}$$

(123)

With (122) and (123), it is straightforward to see that if $P_X\in \mathcal{P}$ then

$$\begin{array}{c}\hfill D_{\alpha }\left(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }|P_X\right)\le D_{\alpha }\left(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }|P_X^{\ast }\right).\end{array}$$

(124)

Consequently, Theorem 1 establishes that $P_X^{\ast }$ achieves the maximal α-mutual information, which using (39) is given by

$$\begin{array}{cc}\hfill C_{\alpha }(\mathcal{P})& =\frac{1}{2}log\left(1+\frac{\alpha P}{{\sigma }^2}\right).\hfill \end{array}$$

(125)

5. Proofs

5.1. Proof of Theorem 1

We deal with the converse statement first. If (59)–(60) are satisfied then $(P_X^{\ast },P_{Y_{\left[\alpha \right]}}^{\ast })$ is a saddle point, and therefore,

$$\begin{array}{cc}\hfill I_{\alpha }(P_X^{\ast },P_{Y|X})& =\underset{P_X\in \mathcal{P}}{max}\underset{Q_Y\in \mathcal{Q}}{min}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)\hfill \end{array}$$

(126)

$$\begin{array}{cc}\hfill & =\underset{Q_Y\in \mathcal{Q}}{min}\underset{P_X\in \mathcal{P}}{max}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X).\hfill \end{array}$$

(127)

which, by definition of $\alpha$-mutual information, implies that $P_X^{\ast }$ attains ${max}_{P_X\in \mathcal{P}}{I}_{\alpha }(P_X,P_{Y|X})$. To show that the optimal input and its $\alpha$-response must form a saddle point, first we deal with the case $\alpha >1$, in which we can use the concavity of the conditional Rényi divergence. Choose arbitrary $\nu \in (0,1)$ and $P_X\in \mathcal{P}$. Let $P_{\nu }=\nu P_X+(1-\nu )P_X^{\ast }$ and denote its $\alpha$-response by

$$\begin{array}{c}\hfill Q_{Y_{\left[\alpha \right]}}^{(\nu )}=\underset{Q_Y\in \mathcal{Q}}{arg\; min}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_{\nu }).\end{array}$$

(128)

Therefore, $Q_{Y_{\left[\alpha \right]}}^{(0)}=P_{Y_{\left[\alpha \right]}}^{\ast }$. We have

$$\begin{array}{cc}\hfill I_{\alpha }(P_X^{\ast },P_{Y|X})& \ge I_{\alpha }(P_{\nu },P_{Y|X})\hfill \end{array}$$

(129)

$$\begin{array}{cc}\hfill & =D_{\alpha }(P_{Y|X}\parallel Q_{Y_{\left[\alpha \right]}}^{(\nu )}|P_{\nu })\hfill \end{array}$$

(130)

$$\begin{array}{cc}\hfill & \ge \nu D_{\alpha }(P_{Y|X}\parallel Q_{Y_{\left[\alpha \right]}}^{(\nu )}|P_X)+(1-\nu )D_{\alpha }(P_{Y|X}\parallel Q_{Y_{\left[\alpha \right]}}^{(\nu )}|P_X^{\ast })\hfill \end{array}$$

(131)

$$\begin{array}{cc}\hfill & \ge \nu D_{\alpha }(P_{Y|X}\parallel Q_{Y_{\left[\alpha \right]}}^{(\nu )}|P_X)+(1-\nu )\underset{Q_Y\in \mathcal{Q}}{min}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X^{\ast })\hfill \end{array}$$

(132)

$$\begin{array}{cc}\hfill & =\nu D_{\alpha }(P_{Y|X}\parallel Q_{Y_{\left[\alpha \right]}}^{(\nu )}|P_X)+(1-\nu )I_{\alpha }(P_X^{\ast },P_{Y|X}),\hfill \end{array}$$

(133)

where (129) is due to the assumption of the optimality of $P_X^{\ast }$, (131) holds because $D_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)$ is concave in $P_X$ for $\alpha >1$ and (130) and (133) are due to the definition of $\alpha$-mutual information.

It follows that

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X}\parallel Q_{Y_{\left[\alpha \right]}}^{(\nu )}|P_X)\le I_{\alpha }(P_X^{\ast },P_{Y|X})=D_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }|P_X^{\ast }).\end{array}$$

(134)

Since $|Q_{Y_{\left[\alpha \right]}}^{(\nu )}-Q_{Y_{\left[\alpha \right]}}^{(0)}|\to 0$ as $\nu \to 0$, the lower semicontinuity of Rényi divergence and (134) imply (59).

We now show the desired result for $\alpha \in (0,1)$. In this case, the method of proof is easy to adapt to the $\alpha >1$ case but it is more cumbersome than the foregoing proof, which is able to capitalize on the concavity of the conditional Rényi divergence. The starting point is the expression in (52) which we write as

$$\begin{array}{cc}\hfill I_{\alpha }(P_X,P_{Y|X})& =\frac{\alpha }{\alpha -1}logf(p_X),\hfill \end{array}$$

(135)

$$\begin{array}{cc}\hfill f(r)& ={\int }_{\mathcal{B}}{\left({\int }_{\mathcal{A}}{p}_{Y|X}^{\alpha }(y|x)r(x)\mathrm{d}{\mu }_X(x)\right)}^{\frac{1}{\alpha }}\mathrm{d}{\mu }_Y(y),\hfill \end{array}$$

(136)

where we have defined the functional f on the convex cone $\overline{\mathcal{P}}$ of nonnegative functions r on the input space such that $r(x)=\beta \frac{\mathrm{d}P}{\mathrm{d}{\mu }_X}(x)$ for some $\beta \ge 0$ and $P\in \mathcal{P}$. Recall from (48) that when the argument is a density, then we have

$$\begin{array}{c}\hfill f(p_X)=\mathbb{E}\left[{\mathbb{E}}^{\frac{1}{\alpha }}[exp(\alpha {ı}_{X;\overline{Y}}(X;\overline{Y}))|\overline{Y}]\right],(X,\overline{Y})\sim P_X\times P_{\overline{Y}}.\end{array}$$

(137)

By virtue of the convexity of ${(·)}^{\frac{1}{\alpha }}$, f is a convex functional. Its directional (Gateaux) derivative is given by (note that the assumed finiteness of $I_{\alpha }(X;Y)$ allows swapping of differentiation and integration by means of the dominated convergence theorem)

$$\begin{array}{cc}\hfill f^{\prime }(r;q)& =\frac{\mathrm{d}}{\mathrm{d}\delta }{f(r+\delta q)|}_{\delta =0}\hfill \end{array}$$

(138)

$$\begin{array}{cc}\hfill & =\frac{1}{\alpha }{\int }_{\mathcal{B}}{\left({\int }_{\mathcal{A}}{p}_{Y|X}^{\alpha }(y|x)r(x)\mathrm{d}{\mu }_X(x)\right)}^{\frac{1-\alpha }{\alpha }}\left({\int }_{\mathcal{A}}{p}_{Y|X}^{\alpha }(y|x)q(x)\mathrm{d}{\mu }_X(x)\right)\mathrm{d}{\mu }_Y(y).\hfill \end{array}$$

(139)

Define the Lagrangian

$$\begin{array}{c}\hfill L(r,\lambda )=f(r)-\lambda \left({\int }_{\mathcal{A}}r(x)\mathrm{d}{\mu }_X(x)-1\right).\end{array}$$

(140)

Since f is convex and (135) is maximized by $P_X^{\ast }$ among all probability measures on the convex set $\mathcal{P}$, there exists some ${\lambda }_0\ge 0$ such that

$$\begin{array}{c}\hfill \underset{\lambda \ge 0}{max}\underset{r}{min}L(r,\lambda )=L(p_X^{\ast },{\lambda }_0),\end{array}$$

(141)

where the minimization is over the convex cone $\overline{\mathcal{P}}$. It follows from standard convex optimization (e.g., see p. 227 in Reference [46]) that the Gateaux derivative of $L(·,{\lambda }_0)$ at $p_X^{\ast }$ in the direction of any $q\in \overline{\mathcal{P}}$ satisfies

$$\begin{array}{cc}\hfill L^{\prime }(p_X^{\ast };q,{\lambda }_0)& \ge 0,\hfill \end{array}$$

(142)

with equality if $q=p_X^{\ast }$. Invoking (139), we obtain

$$\begin{array}{cc}\hfill L^{\prime }(p_X^{\ast };q,{\lambda }_0)=& \frac{1}{\alpha }{\int }_{\mathcal{B}}{\left({\int }_{\mathcal{A}}{p}_{Y|X}^{\alpha }(y|x)p_X^{\ast }(x)\mathrm{d}{\mu }_X(x)\right)}^{\frac{1-\alpha }{\alpha }}\left({\int }_{\mathcal{A}}{p}_{Y|X}^{\alpha }(y|x)q(x)\mathrm{d}{\mu }_X(x)\right)\mathrm{d}{\mu }_Y(y)\hfill \\ \hfill & -{\lambda }_0\left({\int }_{\mathcal{A}}{p}_X^{\ast }(x)\mathrm{d}{\mu }_X(x)-1\right).\hfill \end{array}$$

(143)

Specializing (142) and its condition for equality to $q\leftarrow p_X$ we obtain

$$\begin{array}{c}\hfill L^{\prime }(p_X^{\ast };p_X,{\lambda }_0)\ge L^{\prime }(p_X^{\ast };p_X^{\ast },{\lambda }_0)\end{array}$$

(144)

which, upon substitution of (143), becomes

$$\begin{array}{c}\hfill f(p_X^{\ast })\le {\int }_{\mathcal{A}}{\int }_{\mathcal{B}}{p}_{Y|X}^{\alpha }(y|x){\left({\int }_{\mathcal{A}}{p}_{Y|X}^{\alpha }(y|x)p_X^{\ast }(x)\mathrm{d}{\mu }_X(x)\right)}^{\frac{1-\alpha }{\alpha }}{p}_X(x)\mathrm{d}{\mu }_Y(y)\mathrm{d}{\mu }_X(x).\end{array}$$

(145)

Taking $\frac{1}{\alpha -1}log(·)$ of both sides of (145), invoking (25) and (47), the inequality is reversed and we obtain

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }|P_X)+\frac{1-\alpha }{\alpha }{I}_{\alpha }(X^{\ast };Y^{\ast })\le \frac{1}{\alpha }{I}_{\alpha }(X^{\ast };Y^{\ast }),\end{array}$$

(146)

which upon rearranging is the sought-after inequality (59).

5.2. Proof of Theorem 2

In order to show

$$\begin{array}{c}\hfill \underset{Q_Y\in \mathcal{Q}}{inf}\underset{P_X\in \mathcal{P}}{sup}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)\le \underset{P_X\in \mathcal{P}}{sup}\underset{Q_Y\in \mathcal{Q}}{min}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X)=C_{\alpha }(\mathcal{P}),\end{array}$$

(147)

we construct $Q_Y^{\ast }\in \mathcal{Q}$ such that

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X}\parallel Q_Y^{\ast }|P_X)\le C_{\alpha }(\mathcal{P}),\forall P_X\in \mathcal{P}.\end{array}$$

(148)

Moreover, $Q_Y^{\ast }$ is indeed the minimizer in the leftmost side of (147) and we may replace the inf with min therein.

The construction of $Q_Y^{\ast }$ follows a Cauchy-sequence approach in the proof of Kemperman’s result in [47]. Let ${\left\{P_{X_n}\right\}}_{n\ge 1}$ be a sequence of probability distributions in $\mathcal{P}$ such that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{I}_{\alpha }(P_{X_n},P_{Y|X})=C_{\alpha }(\mathcal{P}).\end{array}$$

(149)

Fix an arbitrary $P_X\in \mathcal{P}$ and let ${\mathcal{P}}_n\subset \mathcal{P}$ denote the convex hull of $\{P_X,P_{X_1},\cdots ,P_{X_n}\}$, which is a compact set. Although $I_{\alpha }(·,P_{Y|X})$ may not be concave for $\alpha \in (0,1)$, recall from (57) that the monotonically increasing function ${\Gamma }_{\alpha }$ is such that ${\Gamma }_{\alpha }(I_{\alpha }(·,P_{Y|X}))$ is concave. So there exists some $P_{X_n}^{\ast }\in {\mathcal{P}}_n$ that attains $C_{\alpha }({\mathcal{P}}_n)$. Thus for any $n\ge 1$,

$$\begin{array}{c}\hfill I_{\alpha }(P_{X_n},P_{Y|X})\le I_{\alpha }(P_{X_n}^{\ast },P_{Y|X})\le C_{\alpha }(\mathcal{P})\end{array}$$

(150)

by the definition of $C_{\alpha }(\mathcal{P})$. The asymptotic optimality of the sequence $X_n$ implies that $I_{\alpha }(P_{X_n}^{\ast },P_{Y|X})=C_{\alpha }({\mathcal{P}}_n)$ also converges to $C_{\alpha }(\mathcal{P})$.

Denote by $P_{Y_{n\left[\alpha \right]}}^{\ast }$ the $\alpha$-response to $P_{X_n}^{\ast }$. Then for any $m\ge n\ge 1$, we have

$$\begin{array}{cc}\hfill I_{\alpha }(P_{X_n}^{\ast },P_{Y|X})& =D_{\alpha }(P_{Y|X}\parallel P_{Y_{n\left[\alpha \right]}}^{\ast }|P_{X_n}^{\ast })\hfill \end{array}$$

(151)

$$\begin{array}{cc}\hfill & =D_{\alpha }(P_{Y|X}\parallel P_{Y_{m\left[\alpha \right]}}^{\ast }|P_{X_n}^{\ast })-D_{\alpha }(P_{Y_{n\left[\alpha \right]}}^{\ast }\parallel P_{Y_{m\left[\alpha \right]}}^{\ast })\hfill \end{array}$$

(152)

$$\begin{array}{cc}\hfill & \le D_{\alpha }(P_{Y|X}\parallel P_{Y_{m\left[\alpha \right]}}^{\ast }|P_{X_m}^{\ast })-D_{\alpha }(P_{Y_{n\left[\alpha \right]}}^{\ast }\parallel P_{Y_{m\left[\alpha \right]}}^{\ast })\hfill \end{array}$$

(153)

$$\begin{array}{cc}\hfill & =I_{\alpha }(P_{X_m}^{\ast },P_{Y|X})-D_{\alpha }(P_{Y_{n\left[\alpha \right]}}^{\ast }\parallel P_{Y_{m\left[\alpha \right]}}^{\ast }),\hfill \end{array}$$

(154)

where (151) and (154) are due to the definition of $\alpha$-mutual information, (152) follows from (42), and (153) holds because of Theorem 1 applied to ${\mathcal{P}}_m$ since $P_{X_n}^{\ast }\in {\mathcal{P}}_m$ as ${\mathcal{P}}_n\subset {\mathcal{P}}_m$ for $m\ge n$. Rearranging the end-to-end inequality in (151)–(154) results in

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y_{n\left[\alpha \right]}}^{\ast }\parallel P_{Y_{m\left[\alpha \right]}}^{\ast })\le I_{\alpha }(P_{X_m}^{\ast },P_{Y|X})-I_{\alpha }(P_{X_n}^{\ast },P_{Y|X}).\end{array}$$

(155)

But since $I_{\alpha }(P_{X_n}^{\ast },P_{Y|X})$ converges to $C_{\alpha }(\mathcal{P})$, it is a Cauchy sequence, i.e.

$$\begin{array}{c}\hfill \left|I_{\alpha }(P_{X_n}^{\ast },P_{Y|X})-I_{\alpha }(P_{X_m}^{\ast },P_{Y|X})\right|\to 0,n,m\to \infty .\end{array}$$

(156)

Hence, (155) ensures that $P_{Y_{n\left[\alpha \right]}}^{\ast }$ is also a Cauchy sequence in the sense that $D_{\alpha }(P_{Y_{n\left[\alpha \right]}}^{\ast }\parallel P_{Y_{m\left[\alpha \right]}}^{\ast })\to 0$ as $n,m\to \infty$. By the generalized Pinkser’s inequality (35), $P_{Y_{n\left[\alpha \right]}}^{\ast }$ is also a Cauchy sequence in total variation distance, i.e., $|P_{Y_{n\left[\alpha \right]}}^{\ast }-P_{Y_{m\left[\alpha \right]}}^{\ast }|\to 0$ as $n,m\to \infty$. Since the space of probability measures is complete in the total variation distance, ${\left\{P_{Y_{n\left[\alpha \right]}}^{\ast }\right\}}_n$ must possess a limit point, which we denote by $P_{Y_{\left[\alpha \right]}}^{\ast }$.

Now, by Theorem 1 applied to ${\mathcal{P}}_n$, we have

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X}\parallel P_{Y_{n\left[\alpha \right]}}^{\ast }|P_X)\le D_{\alpha }(P_{Y|X}\parallel P_{Y_{n\left[\alpha \right]}}^{\ast }|P_{X_n}^{\ast })\le C_{\alpha }(\mathcal{P}),\end{array}$$

(157)

and since $D_{\alpha }(P\parallel Q)$ is lower-semicontinuous in $(P,Q)$ for $\alpha >0$, taking limits as $n\to \infty$ of (157), we obtain

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y|X}\parallel P_{Y_{\left[\alpha \right]}}^{\ast }|P_X)\le C_{\alpha }(\mathcal{P}).\end{array}$$

(158)

Next we show that (158) holds for all $P_X\in \mathcal{P}$, in other words, the limit point $P_{Y_{\left[\alpha \right]}}^{\ast }$ does not depend on the initial choice of $P_X\in \mathcal{P}$. Choose an arbitrary distribution $Q_X\in \mathcal{P},Q_X\ne P_X$, and introduce the following notation: ${\mathcal{P}}_n^{\prime }$ is the convex hull of $\{Q_X,P_X,P_{X_1},\cdots ,P_{X_n}\}$; $Q_{X_n}^{\ast }$ is a maximizer of $I_{\alpha }(P_X,P_{Y|X})$ in ${\mathcal{P}}_n^{\prime }$; its $\alpha$-response is $Q_{Y_{n\left[\alpha \right]}}^{\ast }$; and $Q_{Y_{\left[\alpha \right]}}^{\ast }$ is the limit of the sequence ${\left\{Q_{Y_{n\left[\alpha \right]}}^{\ast }\right\}}_n$. Then we have

$$\begin{array}{cc}\hfill D_{\alpha }(P_{Y_{n\left[\alpha \right]}}^{\ast }\parallel Q_{Y_{n\left[\alpha \right]}}^{\ast })& =D_{\alpha }(P_{Y|X}\parallel Q_{Y_{n\left[\alpha \right]}}^{\ast }|P_{X_n}^{\ast })-D_{\alpha }(P_{Y|X}\parallel P_{Y_{n\left[\alpha \right]}}^{\ast }|P_{X_n}^{\ast })\hfill \end{array}$$

(159)

$$\begin{array}{cc}\hfill & \le D_{\alpha }(P_{Y|X}\parallel Q_{Y_{n\left[\alpha \right]}}^{\ast }|Q_{X_n}^{\ast })-D_{\alpha }(P_{Y|X}\parallel P_{Y_{n\left[\alpha \right]}}^{\ast }|P_{X_n}^{\ast })\hfill \end{array}$$

(160)

$$\begin{array}{cc}\hfill & =I_{\alpha }(Q_{X_n}^{\ast },P_{Y|X})-I_{\alpha }(P_{X_n}^{\ast },P_{Y|X}),\hfill \end{array}$$

(161)

where (159) holds because of (42); (160) is due to Theorem 1 applied to ${\mathcal{P}}_n^{\prime }$ and the fact that $P_{X_n}^{\ast }\in {\mathcal{P}}_n\subset {\mathcal{P}}_n^{\prime }$ for any $n\ge 1$; and (161) is because of the definition of the $\alpha$-mutual information.

The same argument that led to the conclusion that $I_{\alpha }(P_{X_n}^{\ast },P_{Y|X})\to C_{\alpha }(\mathcal{P})$ establishes that $I_{\alpha }(Q_{X_n}^{\ast },P_{Y|X})\to C_{\alpha }(\mathcal{P})$. Therefore, taking limits as $n\to \infty$ in (159)–(161) and applying the lower-semicontinuity of $D_{\alpha }(P\parallel Q)$ again, we obtain

$$\begin{array}{c}\hfill D_{\alpha }(P_{Y_{\left[\alpha \right]}}^{\ast }\parallel Q_{Y_{\left[\alpha \right]}}^{\ast })=0,\end{array}$$

(162)

and therefore $P_{Y_{\left[\alpha \right]}}^{\ast }=Q_{Y_{\left[\alpha \right]}}^{\ast }.$ Since the limiting output distribution is the same whether we use ${\mathcal{P}}_n$ or ${\mathcal{P}}_n^{\prime }$ and according to the latter the roles of $P_X$ and $Q_X$ are identical, we conclude that had we defined ${\mathcal{P}}_n$ with $Q_X$ instead of $P_X$, we would have reached the same limiting output distribution and (158) holds for all $P_X\in \mathcal{P}$. So we have constructed $Q_Y^{\ast }$ satisfying (148).

Finally, we show that $P_{Y_{\left[\alpha \right]}}^{\ast }$ is the only element that achieves

$$\underset{Q_Y\in \mathcal{Q}}{inf}\underset{P_X\in \mathcal{P}}{sup}{D}_{\alpha }(P_{Y|X}\parallel Q_Y|P_X).$$

Arguing by contradiction, suppose that there exists another $\widehat{P_Y}$ such that

$$\begin{array}{c}\hfill \underset{P_X\in \mathcal{P}}{sup}{D}_{\alpha }(P_{Y|X}\parallel \widehat{P_Y}|P_X)=C_{\alpha }(\mathcal{P}).\end{array}$$

(163)

As earlier in the proof, let ${\{P_{X_n}\in \mathcal{P}\}}_n$ be a sequence satisfying (149), and denote the corresponding $\alpha$-responses by $P_{Y_{n\left[\alpha \right]}}$. Then, invoking (42) again we have

$$\begin{array}{cc}\hfill D_{\alpha }(P_{Y|X}\parallel P_{Y_{n\left[\alpha \right]}}|P_{X_n})+D_{\alpha }(P_{Y_{n\left[\alpha \right]}}\parallel \widehat{P_Y})& =D_{\alpha }(P_{Y|X}\parallel \widehat{P_Y}|P_{X_n})\hfill \end{array}$$

(164)

$$\begin{array}{cc}\hfill & \le C_{\alpha }(\mathcal{P})<\infty ,\hfill \end{array}$$

(165)

where the inequality follows from (163). Using (149) we obtain

$$\begin{array}{cc}\hfill D_{\alpha }(P_{Y_{n\left[\alpha \right]}}\parallel \widehat{P_Y})& \le C_{\alpha }(\mathcal{P})-D_{\alpha }(P_{Y|X}\parallel P_{Y_{n\left[\alpha \right]}}|P_{X_n})\hfill \end{array}$$

(166)

$$\begin{array}{c}\to 0,\hfill \end{array}$$

(167)

and by (35), it follows that

$$\begin{array}{c}\hfill |P_{Y_{n\left[\alpha \right]}}-\widehat{P_Y}|\to 0.\end{array}$$

(168)

Furthermore, we established above that

$$\begin{array}{c}\hfill |P_{Y_{n\left[\alpha \right]}}-P_{Y_{\left[\alpha \right]}}^{\ast }|\to 0.\end{array}$$

(169)

So, by the triangle inequality, we conclude that $\widehat{P_Y}=P_{Y_{\left[\alpha \right]}}^{\ast }$.

6. Conclusions

The supremization of $\alpha$-mutual information with respect to the input distribution plays an important role in various information theoretic settings, most notably in the error exponent of optimal codes operating below capacity. We show that the optimal (if it exists) input distribution, together with its $\alpha$-response, form a saddle-point of the conditional Rényi divergence, and vice versa, the existence of the saddle point ensure the existence of a maximal $\alpha$-mutual information input distribution. The application of this result to various discrete and non-discrete settings illustrates the power and generality of this tool, which mirrors a similar result enjoyed by conditional relative entropy; However, the proof of the latter result is much easier due to the more convenient structure of the objective function. Regardless of whether there exists an input distribution maximizing $\alpha$-mutual information, there always exists a unique optimal output distribution, which is the limit of the $\alpha$-responses of any asymptotically optimal sequence of input distributions. Furthermore, a saddle-value exists and

$$\begin{array}{c}\hfill \underset{P_X}{sup}\underset{Q}{min}{D}_{\alpha }(P_{Y|X}\parallel Q|P_X)=\underset{Q}{min}\underset{P_X}{sup}{D}_{\alpha }(P_{Y|X}\parallel Q|P_X)\end{array}$$

(170)

even if we restrict the feasible set of input distributions to be an arbitrary convex subset. These results lend further evidence to the notion that, out of all the available Rényi-generalizations of mutual information, the $\alpha$-mutual information defined as in (43) is the most convenient and insightful, although $I_{\alpha }^{\mathtt{c}}(X;Y)$ is also of considerable interest particularly in the error exponent analysis of channels with cost constraints.