Generalizations of Fano’s Inequality for Conditional Information Measures via Majorization Theory ^†

Abstract

Fano’s inequality is one of the most elementary, ubiquitous, and important tools in information theory. Using majorization theory, Fano’s inequality is generalized to a broad class of information measures, which contains those of Shannon and Rényi. When specialized to these measures, it recovers and generalizes the classical inequalities. Key to the derivation is the construction of an appropriate conditional distribution inducing a desired marginal distribution on a countably infinite alphabet. The construction is based on the infinite-dimensional version of Birkhoff’s theorem proven by Révész [Acta Math. Hungar. 1962, 3, 188–198], and the constraint of maintaining a desired marginal distribution is similar to coupling in probability theory. Using our Fano-type inequalities for Shannon’s and Rényi’s information measures, we also investigate the asymptotic behavior of the sequence of Shannon’s and Rényi’s equivocations when the error probabilities vanish. This asymptotic behavior provides a novel characterization of the asymptotic equipartition property (AEP) via Fano’s inequality.

1. Introduction

Inequalities relating probabilities to various information measures are fundamental tools for proving various coding theorems in information theory. Fano’s inequality [1] is one such paradigmatic example of an information-theoretic inequality; it elucidates the interplay between the conditional Shannon entropy $H(X\mid Y)$ and the error probability $\mathbb{P}\{X\ne Y\}$. Denoting by $h_2:u\mapsto -ulogu-(1-u)log(1-u)$ the binary entropy function on $[0,1]$ with the conventional hypothesis that $h_2\left(0\right)=h_2\left(1\right)=0$, Fano’s inequality can be written as

$$\begin{array}{c}\hfill \underset{(X,Y):\mathbb{P}\{X\ne Y\}\le \epsilon }{max}H(X\mid Y)=h_2\left(\epsilon \right)+\epsilon log(M-1)\end{array}$$

(1)

for every $0\le \epsilon \le 1-1/M$, where log stands for the natural logarithm, and the maximization is taken over the jointly distributed pairs of $\{1,\cdots ,M\}$-valued random variables (r.v.’s) X and Y satisfying $\mathbb{P}\{X\ne Y\}\le \epsilon$. An important consequence of Fano’s inequality is that if the error probabilities vanish, so do the normalized equivocations. In other words,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\mathbb{P}\{X^n\ne Y^n\}=0⟹\underset{n\to \infty }{lim}\frac{1}{n}H(X^n\mid Y^n)=0,\end{array}$$

(2)

where both $X^n=(X_1,\cdots ,X_n)$ and $Y^n=(Y_1,\cdots ,Y_n)$ are random vectors in which each component is a $\{1,\cdots ,M\}$-valued r.v. This is the key in proving weak converse results in various communication models (cf. [2,3,4]). Moreover, Fano’s inequality also shows that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\mathbb{P}\{X_n\ne Y_n\}=0⟹\underset{n\to \infty }{lim}H(X_n\mid Y_n)=0,\end{array}$$

(3)

where $X_n$ and $Y_n$ are $\{1,\cdots ,M\}$-valued r.v.’s for each $n\ge 1$. This implication is used, for example, to prove that various Shannon’s information measures are continuous in the error metric $\mathbb{P}\{X_n\ne Y_n\}$ or the variational distance (cf. [5,6,7]).

1.1. Main Contributions

In this study, we consider general maximization problems that can be specialized to the left-hand side of (1); we generalize Fano’s inequality in the following four ways:

(i)

the alphabet $\mathcal{X}$ of a discrete r.v. X to be estimated is countably infinite,

(ii)

the marginal distribution $P_X$ of X is fixed,

(iii)

the inequality is established on a general class of conditional information measures, and

(iv)

the decoding rule is a list decoding scheme in contrast to a unique decoding scheme.

Specifically, given an $\mathcal{X}$-valued r.v. X with a countably infinite alphabet $\mathcal{X}$ and a $\mathcal{Y}$-valued r.v. Y with an abstract alphabet $\mathcal{Y}$, this study considers a generalized conditional information measure defined by

$$\begin{array}{c}\hfill {\mathsf{H}}_{\varphi }(X\mid Y):=\mathbb{E}\left[\varphi \left(P_{X|Y}\right)\right],\end{array}$$

(4)

where $P_{X|Y}\left(x\right)$ stands for a version of the conditional probability $\mathbb{P}\{X=x\mid Y\}$ for each $x\in \mathcal{X}$, and $\mathbb{E}\left[Z\right]$ stands for the expectation of the real-valued r.v. Z. Here, this function $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ defined on the set $\mathcal{P}\left(\mathcal{X}\right)$ of discrete probability distributions on $\mathcal{X}$ plays the role of an information measure of a discrete probability distribution. When $\mathcal{Y}$ is a countable alphabet, the right-hand side of (4) can be written as

$$\begin{array}{c}\hfill {\mathsf{H}}_{\varphi }(X\mid Y)=\sum _{\begin{array}{c}y\in \mathcal{Y}:\\ P_Y\left(y\right)>0\end{array}}{P}_Y\left(y\right)\varphi \left(P_{X|Y=y}\right),\end{array}$$

(5)

where $P_Y=\mathbb{P}\circ Y^{-1}$ denotes the probability law of Y, and $P_{X|Y=y}\left(x\right):=\mathbb{P}\{X=x\mid Y=y\}$ denotes the conditional probability for each $(x,y)\in \mathcal{X}\times \mathcal{Y}$. In this study, we impose some postulates on $\varphi$ for technical reasons. Choosing $\varphi$ appropriately, we can specialize ${\mathsf{H}}_{\varphi }(X\mid Y)$ to the conditional Shannon entropy $H(X\mid Y)$, Arimoto’s and Hayashi’s conditional Rényi entropies [8,9], and so on. For example, if $\varphi$ is given as

$$\begin{array}{c}\hfill \varphi \left(P\right)=\sum _{x\in \mathcal{X}}P\left(x\right)log\frac{1}{P\left(x\right)},\end{array}$$

(6)

then ${\mathsf{H}}_{\varphi }(X\mid Y)$ coincides with the conditional Shannon entropy $H(X\mid Y)$. Denoting by $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)$ the minimum average probability of list decoding error with a list size L, the principal maximization problem considered in this study can be written as

$$\begin{array}{c}\hfill {\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y}):=\underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon ,P_X=Q}{\sup }{\mathsf{H}}_{\varphi }(X\mid Y),\end{array}$$

(7)

where the supremum is taken over the pairs $(X,Y)$ satisfying $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon$ and fixing the $\mathcal{X}$-marginal $P_X$ to a given distribution Q. The feasible region of systems $(Q,L,\epsilon ,\mathcal{Y})$ will be characterized in this paper to ensure that ${\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})$ is well-defined. Under some mild conditions on a given system $(Q,L,\epsilon ,\mathcal{Y})$, especially on the cardinality of $\mathcal{Y}$, we derive explicit formulas of ${\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})$; otherwise, we establish tight upper bounds on ${\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})$. As ${\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})$ can be thought of as a generalization of the maximization problem stated in (1), we call these results Fano-type inequalities in this paper. These Fano-type inequalities are formulated by the considered information measures $\varphi \left(P_{\mathrm{type}-\ast }\right)$ of certain (extremal) probability distributions $P_{\mathrm{type}-\ast }$ depending only on the system $(Q,L,\epsilon ,\mathcal{Y})$.

In this study, we provide Fano-type inequalities via majorization theory [10]. A proof outline to obtain our Fano-type inequalities is as follows.

• First, we show that a generalized conditional information measure ${\mathsf{H}}_{\varphi }(X\mid Y)$ can be bounded from above by ${\mathsf{H}}_{\varphi }(U\mid V)$ with a certain pair $(U,V)$ in which the conditional distribution $P_{U|V}$ of U given V can be thought of as a so-called uniformly dispersive channel [11,12] (see also Section II-A of [13]). We prove this fact via Jensen’s inequality (cf. Proposition A-2 of [14]) and the symmetry of the considered information measures $\varphi$. Moreover, we establish a novel characterization of uniformly dispersive channels via a certain majorization relation; we show that the output distribution of a uniformly dispersive channel is majorized by its transition probability distribution for any fixed input symbol. This majorization relation is used to obtain a sharp upper bound via the Schur-concavity property of the considered information measures $\varphi$.
• Second, we ensure the existence of a joint distribution $P_{X,Y}$ of $(X,Y)$ which satisfies all constraints in our maximization problems ${\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})$ stated in (7) and the conditional distribution $P_{X|Y}$ is uniformly dispersive. Here, a main technical difficulty is to maintain a marginal distribution $P_X$ of X over a countably infinite alphabet $\mathcal{X}$; see (ii) above. Using a majorization relation for a uniformly dispersive channel, we express a desired marginal distribution $P_X$ by the multiplication of a doubly stochastic matrix and a uniformly dispersive $P_{X|Y}$. This characterization of the majorization relation via a doubly stochastic matrix was proven by Hardy–Littleweed–Pólya [15] in the finite-dimensional case, and by Markus [16] in the infinite-dimensional case. From this doubly stochastic matrix, we construct a marginal distribution $P_Y$ of Y so that the joint distribution $P_{X,Y}=P_{X|Y}{P}_Y$ has the desired marginal distribution $P_X$. The construction of $P_Y$ is based on the infinite-dimensional version of Birkhoff’s theorem, which was posed by Birkhoff [17] and was proven by Révész [18] via Kolmogorov’s extension theorem. Although the finite-dimensional version of Birkhoff’s theorem [19] (also known as the Birkhoff–von Neumann decomposition) is well-known, the application of the infinite-dimensional version of Birkhoff’s theorem in information theory appears to be novel; its application aids in dealing with communication systems over countably infinite alphabets.
• Third, we introduce an extremal distribution $P_{\mathrm{type}-\ast }$ on a countably infinite alphabet $\mathcal{X}$. Showing that $P_{\mathrm{type}-\ast }$ is the infimum of a certain class of discrete probability distributions with respect to the majorization relation, our maximization problems can be bounded from above by the considered information measure $\varphi \left(P_{\mathrm{type}-\ast }\right)$. Namely, our Fano-type inequality is expressed by a certain information measure of the extremal distribution. When the cardinality of the alphabet of Y is large enough, by constructing a joint distribution $P_{X,Y}$ achieving equality in our generalized Fano-type inequality, we say that the inequality is sharp.

When the alphabet of Y is finite, we further tighten our Fano-type inequality. To do so, we prove a reduction lemma for the principal maximization problem from an infinite- to a finite-dimensional feasible region. Therefore, when the alphabet of Y is finite, we do not have to employ technical tools in infinite-dimensional majorization theory, e.g., the infinite-dimensional version of Birkhoff’s theorem. This reduction lemma is useful not only to tighten our Fano-type inequality but also to characterize a sufficient condition of the considered information measure $\varphi$ in which ${\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})$ is finite if and only if $\varphi \left(Q\right)$ is also finite. In fact, Shannon’s and Rényi’s information measures meet this sufficient condition.

We show that our Fano-type inequalities can be specialized to some known generalizations of Fano’s inequality [20,21,22,23] on Shannon’s and Rényi’s information measures. Therefore, one of our technical contributions is a unified proof of Fano’s inequality for conditional information measures via majorization theory. Generalizations of Erokhin’s function [20] from the ordinary mutual information to Sibson’s and Arimoto’s $\alpha$-mutual information [8,24] are also discussed.

Via our generalized Fano-type inequalities, we investigate sufficient conditions on a general source $\mathbf{X}={\{X_n=(Z_1^{\left(n\right)},\cdots ,Z_n^{\left(n\right)})\}}_{n=1}^{\infty }$ in which vanishing error probabilities implies vanishing equivocations (cf. (2) and (3)). We show that the asymptotic equipartition property (AEP) as defined by Verdú–Han [25] is indeed such a sufficient condition. In other words, if a general source $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$ satisfies the AEP and $H\left(X_n\right)=\mathsf{\Omega }\left(1\right)$ as $n\to \infty$, then we prove that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{P}_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n)=\underset{n\to \infty }{lim}\frac{log{L}_n}{H\left(X_n\right)}=0⟹\underset{n\to \infty }{lim}\frac{H(X_n\mid Y_n)}{H\left(X_n\right)}=0,\end{array}$$

(8)

where ${\left\{L_n\right\}}_{n=1}^{\infty }$ is an arbitrary sequence of list sizes. This is a generalization of (2) and (3) and, to the best of the author’s knowledge, a novel connection between the AEP and Fano’s inequality. We prove this connection by using the splitting technique of a probability distribution; this technique was used to derive limit theorems of Markov processes by Nummelin [26] and Athreya–Ney [27]. Note that there are also many applications of the splitting technique in information theory (cf. [21,28,29,30,31,32]). In addition, we extend Ho–Verdú’s sufficient conditions (See Section V of [21]) and Sason–Verdú’s sufficient conditions (see Theorem 4 of [23]) on a general source $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$ in which equivocations vanish if the error probabilities vanish.

1.2. Related Works

1.2.1. Information Theoretic Tools on Countably Infinite Alphabets

As the right-hand side of (1) diverges as M goes to infinity whenever $\epsilon >0$ is fixed, the classical Fano inequality is applicable only if X is supported on a finite alphabet (see also Chapter 1 of [33]). In fact, if both $X_n$ and $Y_n$ are supported on the same countably infinite alphabet for each $n\ge 1$, one can construct a somewhat pathological example so that $\mathbb{P}\{X_n\ne Y_n\}=\mathrm{o}\left(1\right)$ as $n\to \infty$ but $H(X_n\mid Y_n)=\infty$ for every $n\ge 1$ (cf. Example 2.49 of [4]).

Usually, it is not straightforward to generalize information theoretic tools for systems defined on a finite alphabet to systems defined on a countably infinite alphabet. Ho–Yeung [34] showed that Shannon’s information measures defined on countably infinite alphabets are not continuous with respect to the following distances; the ${\chi }^2$-divergence, the relative entropy, and the variational distance. Continuity issues of Rényi’s information measures defined on countably infinite alphabets were explored by Kovačević–Stanojević–Šenk [35]. In addition, although weak typicality (cf. Chapter 3 of [2] that is also known as the entropy-typical sequences (cf. Problem 2.5 of [6]) is a convenient tool in proving achievability theorems for sources and channels with defined on countably infinite (or even uncountable) alphabets, strong typicality [6] is only amenable in situations with finite alphabets. To ameliorate this issue, Ho–Yeung [36] proposed a notion known as unified typicality that ensures that the desirable properties of weak and strong typicality are retained when one is working with countably infinite alphabets.

Recently, Madiman–Wang–Woo [37] investigated relations between majorization and the strong Sperner property [38] of posets together with applications to the Rényi entropy power inequality for sums of independent and integer-valued r.v.’s, i.e., supported on countably infinite alphabets.

To the best of the author’s knowledge, a generalization of Fano’s inequality to the case when X is supported on a countably infinite alphabet was initiated by Erokhin [20]. Given a discrete probability distribution Q on a countably infinite alphabet $\mathcal{X}=\{1,2,\cdots \}$, Erokhin established in Equation (11) of [20] an explicit formula of the function:

$$\begin{array}{c}\hfill \mathbb{I}(Q,\epsilon ):=\underset{(X,Y):\mathbb{P}\{X\ne Y\}\le \epsilon ,P_X=Q}{min}I(X\wedge Y),\end{array}$$

(9)

where the minimization is taken over the pairs of $\mathcal{X}$-valued r.v.’s X and Y satisfying $\mathbb{P}\{X\ne Y\}\le \epsilon$ and $\mathbb{P}\{X=x\}=Q\left(x\right)$ for each $x\in \mathcal{X}$, and $I(X\wedge Y)$ stands for the mutual information between X and Y. Note that Erokhin’s function $\mathbb{I}(Q,\epsilon )$ is the rate-distortion function with Hamming distortion measures (cf. [39,40]). As the well-known identity $I(X\wedge Y)=H\left(X\right)-H(X\mid Y)$ implies that

$$\begin{array}{c}\hfill \mathbb{I}(Q,\epsilon )=H\left(X\right)-\underset{(X,Y):\mathbb{P}\{X\ne Y\}\le \epsilon ,P_X=Q}{max}H(X\mid Y),\end{array}$$

(10)

Erokhin’s function $\mathbb{I}(Q,\epsilon )$ can be naturally thought of as a generalization of the classical Fano inequality stated in (1), where $H\left(X\right)$ stands for the Shannon entropy of X, and the probability distribution of X is given by $\mathbb{P}\{X=x\}=Q\left(x\right)$ for each $x\in \mathcal{X}$. Kostina–Polyanskiy–Verdú [41] derived a second-order asymptotic expansion of $\mathbb{I}(Q^n,\epsilon )$ as $n\to \infty$, where $Q^n$ stands for the n-fold product of Q. Their asymptotic expansion is closely related to the second-order asymptotics of the variable-length compression allowing errors; see ([41], Theorem 4).

Ho–Verdú [21] gave an explicit formula of the maximization in the right-hand side of (10); they proved it via the additivity of Shannon’s information measures. Note that Ho–Verdú’s formula (cf. Theorem 1 of [21]) coincides with Erokhin’s formula (cf. Equation (11) of [20]) via the identity stated in (10). In Theorems 2 and 4 of [21], Ho–Verdú also tightened the maximization in the right-hand side of (10) when Y is supported on a proper subalphabet of $\mathcal{X}$. Moreover, they provided in Section V of [21] some sufficient conditions on a general source in which vanishing error probabilities (i.e., $\mathbb{P}\{X_n\ne Y_n\}=\mathrm{o}\left(1\right)$) implies vanishing unnormalized or normalized equivocations (i.e., $H(X_n\mid Y_n)=\mathrm{o}\left(1\right)$ or $H(X_n\mid Y_n)=\mathrm{o}\left(n\right)$).

1.2.2. Fano’s Inequality with List Decoding

Fano’s inequality with list decoding was initiated by Ahlswede–Gács–Körner [42]. By a minor extension of the usual proof (see, e.g., Lemma 3.8 of [6]), one can see that

$$\begin{array}{c}\hfill \underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon }{max}H(X\mid Y)=h_2\left(\epsilon \right)+(1-\epsilon )logL+\epsilon log(M-L)\end{array}$$

(11)

for every integers $1\le L<M$ and every real number $0\le \epsilon \le 1-L/M$, where the maximization is taken over the pairs of a $\{1,\cdots ,M\}$-valued r.v. X and a $\mathcal{Y}$-valued r.v. Y satisfying $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon$. Note that the right-hand side of (11) coincides with the Shannon entropy of the extremal distribution of type-0 defined by

$$\begin{array}{c}\hfill P_{type-0}\left(x\right)=P_{type-0}^{(M,L,\epsilon )}\left(x\right):=\left\{\begin{array}{cc}\frac{1-\epsilon }{L}\hfill & \mathrm{if}1\le x\le L,\hfill \\ \frac{\epsilon }{M-L}\hfill & \mathrm{if}L<x\le M,\hfill \\ 0\hfill & \mathrm{if}M<x<\infty \hfill \end{array}\right.\end{array}$$

(12)

for each integer $x\ge 1$. A graphical representation of this extremal distribution is plotted in Figure 1.

Figure 1. Each bar represents a probability mass of the extremal distribution of type-0 defined in (12), where $M=8$ and $L=3$.

Combining (11) and the blowing-up technique (cf. Chapter 5 of [6] or Section 3.6.2 of [43]), Ahlswede–Gács–Körner [42] proved the strong converse property (in Wolfowitz’s sense [44]) of degraded broadcast channels under the maximum error probability criterion. Extending the proof technique in [42] together with the wringing technique, Dueck [45] proved the strong converse property of multiple-access channels under the average error probability criterion. As these proofs rely on a combinatorial lemma (cf. Lemma 5.1 of [6]), they work only when the channel output alphabet is finite; but see recent work by Fong–Tan [46,47] in which such techniques have been extended to Gaussian channels. On the other hand, Kim–Sutivong–Cover [48] investigated a trade-off between the channel coding rate and the state uncertainty reduction of a channel with state information available only at the sender, and derived its trade-off region in the weak converse regime by employing (11).

1.2.3. Fano’s Inequality for Rényi’s Information Measures

So far, many researchers have considered various directions for generalizing Fano’s inequality. An interesting study involves reversing the usual Fano inequality. In this regard, lower bounds on $H(X\mid Y)$ subject to $\mathbb{P}\{X\ne Y\}=\epsilon$ were independently established by Kovalevsky [49], Chu–Cheuh [50], and Tebbe–Dwyer [51] (see also Feder–Merhav’s study [52]). Prasad [53] provided several refinements of the reverse/forward Fano inequalities for Shannon’s information measures.

In [54], Ben-Bassat–Raviv explored several inequalities between the (unconditional) Rényi entropy and the error probability. Generalizations of Fano’s inequality from the conditional Shannon entropy $H(X\mid Y)$ to Arimoto’s conditional Rényi entropy $H_{\alpha }^{\mathrm{Arimoto}}(X\mid Y)$ introduced in [8] were recently and independently investigated by Sakai–Iwata [22] and Sason–Verdú [23]. Specifically, Sakai–Iwata [22] provided sharp upper/lower bounds on $H_{\alpha }^{\mathrm{Arimoto}}(X\mid Y)$ for fixed $H_{\beta }^{\mathrm{Arimoto}}(X\mid Y)$ with two distinct orders $\alpha \ne \beta$. In other words, they gave explicit formulas of the following minimization and maximization,

$$\begin{array}{cc}\hfill f_{min}(\alpha ,\beta ,\gamma )& :=\underset{(X,Y):H_{\beta }^{\mathrm{Arimoto}}(X\mid Y)=\gamma }{min}{H}_{\alpha }^{\mathrm{Arimoto}}(X\mid Y),\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill f_{max}(\alpha ,\beta ,\gamma )& :=\underset{(X,Y):H_{\beta }^{\mathrm{Arimoto}}(X\mid Y)=\gamma }{max}{H}_{\alpha }^{\mathrm{Arimoto}}(X\mid Y),\hfill \end{array}$$

(14)

respectively. As $H_{\beta }^{\mathrm{Arimoto}}(X\mid Y)$ is a strictly monotone function of the minimum average probability of error if $\beta =\infty$, both functions $f_{min}(\alpha ,\infty ,\gamma )$ and $f_{max}(\alpha ,\infty ,\gamma )$ can be thought of as reverse and forward Fano inequalities on $H_{\alpha }^{\mathrm{Arimoto}}(X\mid Y)$, respectively (cf. Section V in the arXiv paper [22]). Sason–Verdú [23] also gave generalizations of the forward and reverse Fano’s inequalities on $H_{\alpha }^{\mathrm{Arimoto}}(X\mid Y)$. Moreover, in the forward Fano inequality pertaining to $H_{\alpha }^{\mathrm{Arimoto}}(X\mid Y)$, they generalized in Theorem 8 of [23] the decoding rules from unique decoding to list decoding as follows:

$$\begin{array}{c}\hfill \underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon }{max}{H}_{\alpha }^{\mathrm{Arimoto}}(X\mid Y)=\frac{1}{1-\alpha }log\left(L^{1-\alpha }(1-\epsilon )+{(M-L)}^{1-\alpha }{\epsilon }^{\alpha }\right)\end{array}$$

(15)

for every $0\le \epsilon \le 1-L/M$ and $\alpha \in (0,1)\cup (1,\infty )$, where the maximization is taken as with (11). Similar to (11), the right-hand side of (15) coincides with the Rényi entropy [55] of the extremal distribution of type-0. Note that the reverse Fano inequality proven in [22,23] does not require that $\mathcal{X}$ is finite. On the other hand, the forward Fano inequality proven in [22,23] is applicable only when $\mathcal{X}$ is finite.

1.2.4. Lower Bounds on Mutual Information

Han–Verdú [56] generalized Fano’s inequality on a countably infinite alphabet $\mathcal{X}$ by investigating lower bounds on the mutual information, i.e.,

$$\begin{array}{c}\hfill I(X\wedge Y)\ge \mathbb{P}\{X\ne Y\}log\frac{\mathbb{P}\{X\ne Y\}}{\mathbb{P}\{\overline{X}\ne \overline{Y}\}}+\mathbb{P}\{X=Y\}log\frac{\mathbb{P}\{X=Y\}}{\mathbb{P}\{\overline{X}=\overline{Y}\}},\end{array}$$

(16)

via the data processing lemma without additional constraints on the r.v.’s X and Y, where $\overline{X}$ and $\overline{Y}$ are independent r.v.’s having marginals as X and Y respectively. Polyanskiy–Verdú [57] showed a lower bound on Sibson’s $\alpha$-mutual information by using the data processing lemma for the Rényi divergence. Recently, Sason [58] generalized Fano’s inequality with list decoding via the strong data processing lemma for the f-divergences.

Liu–Verdú [59] showed that

$$\begin{array}{c}\hfill I(X^n\wedge Y^n)\ge log{M}_n+\mathrm{O}\left(\sqrt{n}\right)\end{array}$$

(17)

as $n\to \infty$, provided that the geometric average probability of error, which is a weaker and a stronger criteria than the maximum and the average error criteria, respectively, satisfies

$$\begin{array}{c}\hfill {\left(\prod _{m=1}^{M_n}\mathbb{P}\{Y^n\in {\mathcal{D}}_{m,n}\mid X^n={\mathbf{c}}_{m,n}\}\right)}^{1/M_n}\ge 1-\epsilon \end{array}$$

(18)

for sufficiently large n, where $X^n$ is a r.v. uniformly distributed on the codeword set ${\left\{{\mathbf{c}}_{m,n}\right\}}_{m=1}^{M_n}$, $Y^n$ is a r.v. induced by the n-fold product of a discrete memoryless channel with the input $X^n$, $M_n$ is a positive integer denoting the message size, ${\left\{D_{m,n}\right\}}_{m=1}^{M_n}$ is a collection of disjoint subsets playing the role of decoding regions, and $0<\epsilon <1$ is a tolerated probability of error. This is a second-order asymptotic estimate on the mutual information, and is derived by using the Donsker–Varadhan lemma (cf. Equation (3.4.67) of [43]) and the so-called pumping-up argument.

1.3. Paper Organization

The rest of this paper is organized as follows. Section 2 introduces basic notations and definitions to understand our generalized conditional information measure ${\mathsf{H}}_{\varphi }(X\mid Y)$ and the principal maximization problem ${\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})$. Section 3 presents the main results: our Fano-type inequalities. Section 4 specializes our Fano-type inequalities to Shannon’s and Rényi’s information measures, and discusses generalizations of Erokhin’s function from the ordinary mutual information to Sibson’s and Arimoto’s $\alpha$-mutual information. Section 5 investigates several conditions on a general source in which the vanishing error probabilities implies the vanishing equivocations; a novel characterization of the AEP via Fano’s inequality is also presented. Section 6 proves our Fano-type inequalities stated in Section 3, and contains most technical contributions in this study. Section 7 proves the asymptotic behaviors stated in Section 5. Finally, Section 8 concludes this study with some remarks.

2. Preliminaries

2.1. A General Class of Conditional Information Measures

This subsection introduces some notions in majorization theory [10] and a rigorous definition of generalized conditional information measure ${\mathsf{H}}_{\varphi }(X\mid Y)$ defined in (4). Let $\mathcal{X}=\{1,2,\cdots \}$ be a countably infinite alphabet. A discrete probability distribution P on $\mathcal{X}$ is a map $P:\mathcal{X}\to [0,1]$ satisfying ${\sum }_{x\in \mathcal{X}}P\left(x\right)=1$. In this paper, motivated to consider the joint probability distributions on $\mathcal{X}\times \mathcal{Y}$, it is called an $\mathcal{X}$-marginal. Given an $\mathcal{X}$-marginal P, a decreasing rearrangement of P is denoted by $P^{\downarrow }$, i.e., it fulfills

$$\begin{array}{c}\hfill P^{\downarrow }\left(1\right)\ge P^{\downarrow }\left(2\right)\ge P^{\downarrow }\left(3\right)\ge P^{\downarrow }\left(4\right)\ge P^{\downarrow }\left(5\right)\ge \cdots .\end{array}$$

(19)

The following definition gives us the notion of majorization for $\mathcal{X}$-marginals.

Definition 1

(Majorization [10]). An $\mathcal{X}$-marginal P is said to majorize another $\mathcal{X}$-marginal Q if

$$\begin{array}{c}\hfill \sum _{i=1}^k{P}^{\downarrow }\left(i\right)\ge \sum _{i=1}^k{Q}^{\downarrow }\left(i\right)\end{array}$$

(20)

for every $k\ge 1$. This relation is denoted by $P\succ Q$ or $Q\prec P$.

Let $\mathcal{P}\left(\mathcal{X}\right)$ be the set of $\mathcal{X}$-marginals. The following definitions are important postulates on a function $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ playing the role of an information measure of an $\mathcal{X}$-marginal.

Definition 2.

A function $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ is said to be symmetric if it is invariant for any permutation of probability masses, i.e., $\varphi \left(P\right)=\varphi \left(P^{\downarrow }\right)$ for every $P\in \mathcal{P}\left(\mathcal{X}\right)$.

Definition 3.

A function $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ is said to be lower semicontinuous if for any $P\in \mathcal{P}\left(\mathcal{X}\right)$, it holds that ${lim\; inf}_n\varphi \left(P_n\right)\ge \varphi \left(P\right)$ for every pointwise convergent sequence $P_n\to P$, where the pointwise convergence $P_n\to P$ means that $P_n\left(x\right)\to P\left(x\right)$ as $n\to \infty$ for every $x\in \mathcal{X}$.

Definition 4.

A function $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ is said to be convex if $\varphi \left(R\right)\le \lambda \varphi \left(P\right)+(1-\lambda )\varphi \left(Q\right)$ with $R=\lambda P+(1-\lambda )Q$ for every $P,Q\in \mathcal{P}\left(\mathcal{X}\right)$ and $0\le \lambda \le 1$.

Definition 5.

A function $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ is said to be quasiconvex if the sublevel set $\{P\in \mathcal{P}(\mathcal{X})\mid \varphi (P)\le c\}$ is convex for every $P\in \mathcal{P}\left(\mathcal{X}\right)$ and $c\in [0,\infty )$.

Definition 6.

A function $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ is said to be Schur-convex if $P\prec Q$ implies that $\varphi \left(P\right)\le \varphi \left(Q\right)$.

In Definitions 4–6, each term or its suffix convex is replaced by concave if $-\varphi$ fulfills the condition. In Definition 3, note that the pointwise convergence of $\mathcal{X}$-marginals is equivalent to the convergence in the variational distance topology (see, e.g., Lemma 3.1 of [60] or Section III-D of [61]).

Let X be an $\mathcal{X}$-valued r.v. and Y a $\mathcal{Y}$-valued r.v., where $\mathcal{Y}$ is an abstract alphabet. Unless stated otherwise, assume that the measurable space of $\mathcal{Y}$ with a certain $\sigma$-algebra is standard Borel, where a measurable space is said to be standard Borel if its $\sigma$-algebra is the Borel $\sigma$-algebra generated by a Polish topology on the space. Assuming that $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ is a symmetric, concave, and lower semicontinuous function, the generalized conditional information measure ${\mathsf{H}}_{\varphi }(X\mid Y)$ is defined by (4). The postulates on $\varphi$ we have imposed here are useful for technical reasons to employ majorization theory; see the following lemma.

Proposition 1.

Every symmetric and quasiconvex function $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ is Schur-convex.

Proof of Proposition 1.

In Proposition 3.C.3 of [10], the assertion of Proposition 1 was proved in the case where the dimension of the domain of $\varphi$ is finite. Employing Theorem 4.2 of [16] instead of Corollary 2.B.3 of [10], the proof of Proposition 3.C.3 of [10] can be directly extended to infinite-dimensional domains. □

To employ the Schur-concavity property in the sequel, Proposition 1 suggests assuming that $\varphi$ is symmetric and quasiconcave. In addition, to apply Jensen’s inequality on the function $\varphi$, it suffices to assume that $\varphi$ is concave and lower semicontinuous, because the domain $\mathcal{P}\left(\mathcal{X}\right)$ forms a closed convex bounded set in the variational distance topology (cf. Proposition A-2 of [14]). Motivated by these properties, we impose the three postulates (corresponding to Definitions 2–4) on $\varphi$ in this study.

2.2. Minimum Average Probability of List Decoding Error

Consider a certain communication model for which a $\mathcal{Y}$-valued r.v. Y plays the role of the side-information of an $\mathcal{X}$-valued r.v. X. A list decoding scheme with a list size $1\le L<\infty$ is a decoding scheme producing L candidates for realizations of X when we observe a realization of Y. The minimum average error probability under list decoding is defined by

$$\begin{array}{c}\hfill P_{\mathrm{e}}^{\left(L\right)}(X\mid Y):=\underset{f:\mathcal{Y}\to \left(\genfrac{}{}{0pt}{}{\mathcal{X}}{L}\right)}{min}\mathbb{P}\{X\notin f\left(Y\right)\},\end{array}$$

(21)

where the minimization is taken over all set-valued functions $f:\mathcal{Y}\to \left(\genfrac{}{}{0pt}{}{\mathcal{X}}{L}\right)$ with the decoding range

$$\begin{array}{cc}\hfill \left(\genfrac{}{}{0pt}{}{\mathcal{X}}{L}\right)& :=\{\mathcal{D}\subset \mathcal{X}\mid |\mathcal{D}|=L\},\hfill \end{array}$$

(22)

and $|·|$ stands for the cardinality of a set. If $\mathcal{S}$ is an infinite set, then we assume that $\left|\mathcal{S}\right|=\infty$ as usual. If $L=1$, then (21) coincides with the average error probability of the maximum a posteriori (MAP) decoding scheme. For the sake of brevity, we write

$$\begin{array}{c}\hfill P_{\mathrm{e}}(X\mid Y):=P_{\mathrm{e}}^{\left(1\right)}(X\mid Y).\end{array}$$

(23)

It is clear that

$$\begin{array}{c}\hfill \mathbb{P}\{X\notin f\left(Y\right)\}\le \epsilon ⟹P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon \end{array}$$

(24)

for any list decoder $f:\mathcal{Y}\to \left(\genfrac{}{}{0pt}{}{\mathcal{X}}{L}\right)$ and any tolerated probability of error $\epsilon \ge 0$. Therefore, it suffices to consider the constraint $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon$ rather than $\mathbb{P}\{X\notin f(Y\left)\right\}\le \epsilon$ in our subsequent analyses.

The following proposition is an elementary formula of $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)$ as in the MAP decoding.

Proposition 2.

It holds that

$$\begin{array}{c}\hfill P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)=1-\mathbb{E}\left[\sum _{x=1}^L{P}_{X|Y}^{\downarrow }\left(x\right)\right].\end{array}$$

(25)

Proof of Proposition 2.

See Appendix A. □

Remark 1.

It follows from Proposition 2 that ${\mathsf{H}}_{\varphi }(X\mid Y)$ defined in (4) can be specialized to $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)$ with

$$\begin{array}{c}\hfill \varphi \left(P\right)=1-\sum _{x=1}^L{P}^{\downarrow }\left(x\right).\end{array}$$

(26)

The following proposition characterizes the feasible region of systems $(Q,L,\epsilon ,\mathcal{Y})$ considered in our principal maximization problem ${\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})$ stated in (7).

Proposition 3.

If $P_X=Q$, then

$$\begin{array}{c}\hfill 1-\sum _{x=1}^{L·\left|\mathcal{Y}\right|}{Q}^{\downarrow }\left(x\right)\le P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le 1-\sum _{x=1}^L{Q}^{\downarrow }\left(x\right).\end{array}$$

(27)

Moreover, both inequalities are sharp in the sense that there exist pairs of r.v.’s X and Y achieving the equalities while respecting the constraint $P_X=Q$.

Proof of Proposition 3.

See Appendix B. □

The minimum average error probability for list decoding concerning $X\sim Q$ without any side-information is denoted by

$$\begin{array}{c}\hfill P_{\mathrm{e}}^{\left(L\right)}\left(Q\right):=1-\sum _{x=1}^L{Q}^{\downarrow }\left(x\right).\end{array}$$

(28)

Then, the second inequality in (27) is obvious, and it is similar to the property that conditioning reduces uncertainty (cf. [2], Theorem 2.8.1). Proposition 3 ensures that when we have to consider the constraints $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon$ and $P_X=Q$, it suffices to consider a system $(Q,L,\epsilon ,\mathcal{Y})$ satisfying

$$\begin{array}{c}\hfill 1-\sum _{x=1}^{L·\left|\mathcal{Y}\right|}{Q}^{\downarrow }\left(x\right)\le \epsilon \le 1-\sum _{x=1}^L{Q}^{\downarrow }\left(x\right).\end{array}$$

(29)

3. Main Results: Fano-Type Inequalities

Let $(Q,L,\epsilon ,\mathcal{Y})$ be a system satisfying (29), and $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ a symmetric, concave, and lower semicontinuous function. The main aim of this study is to find an explicit formula or a tight upper bound on ${\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})$ defined in (7). Now, define the extremal distribution of type-1 by the following $\mathcal{X}$-marginal,

$$\begin{array}{c}\hfill P_{type-1}\left(x\right)=P_{type-1}^{(Q,L,\epsilon )}\left(x\right):=\left\{\begin{array}{cc}{Q}^{\downarrow }\left(x\right)\hfill & \mathrm{if}1\le x<J\mathrm{or}{K}_1<x<\infty ,\hfill \\ \mathcal{V}\left(J\right)\hfill & \mathrm{if}J\le x\le L,\hfill \\ \mathcal{W}\left(K_1\right)\hfill & \mathrm{if}L<x\le K_1,\hfill \end{array}\right.\end{array}$$

(30)

for each $x\in \mathcal{X}$, the weight $\mathcal{V}\left(j\right)$ is defined by

$$\begin{array}{c}\hfill \mathcal{V}\left(j\right)={\mathcal{V}}^{(Q,L,\epsilon )}\left(j\right):=\left\{\begin{array}{cc}\frac{(1-\epsilon )-{\sum }_{x=1}^{j-1}{Q}^{\downarrow }\left(x\right)}{L-j+1}\hfill & \mathrm{if}1\le j\le L,\hfill \\ 1\hfill & \mathrm{if}j>L\hfill \end{array}\right.\end{array}$$

(31)

for each $j\ge 1$, the weight $\mathcal{W}\left(k\right)$ is defined by

$$\begin{array}{c}\hfill \mathcal{W}\left(k\right)={\mathcal{W}}^{(Q,L,\epsilon )}\left(k\right):=\left\{\begin{array}{cc}-1\hfill & \mathrm{if}k=L,\hfill \\ \frac{{\sum }_{x=1}^k{Q}^{\downarrow }\left(x\right)-(1-\epsilon )}{k-L}\hfill & \mathrm{if}L<k<\infty ,\hfill \\ 0\hfill & \mathrm{if}k=\infty \hfill \end{array}\right.\end{array}$$

(32)

for each $k\ge L$, the integer J is chosen so that

$$\begin{array}{cc}\hfill J=J(Q,L,\epsilon )& :=min\{1\le j<\infty \mid Q^{\downarrow }\left(j\right)<\mathcal{V}\left(j\right)\},\hfill \end{array}$$

(33)

and $K_1$ is chosen so that

$$\begin{array}{cc}\hfill K_1=K_1(Q,L,\epsilon )& :=\sup \{L\le k<\infty \mid \mathcal{W}\left(k\right)<Q^{\downarrow }\left(k\right)\}.\hfill \end{array}$$

(34)

A graphical representation of $P_{type-1}$ is shown in Figure 2. Under some mild conditions, the following theorem gives an explicit formula of ${\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})$.

Figure 2. Plot of making the extremal distribution of type-1 defined in (30) from an $\mathcal{X}$-marginal Q, where $L=3$. Each bar represents a probability mass with decreasing rearrangement $Q^{\downarrow }$.

Theorem 1.

Suppose that $\epsilon >0$ and the cardinality of $\mathcal{Y}$ is at least countably infinite. Then, it holds that

$$\begin{array}{c}\hfill {\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})=\varphi \left(P_{type-1}\right).\end{array}$$

(35)

Proof of Theorem 1.

See Section 6.1. □

The Fano-type inequality stated in (35) of Theorem 1 is formulated by the extremal distribution $P_{type-1}$ defined in (30). The following proposition summarizes basic properties of $P_{type-1}$.

Proposition 4.

The extremal distribution of type-1 defined in (30) satisfies the following,

• the probability masses are nonincreasing in $x\in \mathcal{X}$, i.e.,

  $$\begin{array}{c}\hfill P_{type-1}\left(1\right)\ge P_{type-1}\left(2\right)\ge P_{type-1}\left(3\right)\ge P_{type-1}\left(4\right)\ge P_{type-1}\left(5\right)\ge \cdots ,\end{array}$$

  (36)
• the sum of first L probability masses of is equal to $1-\epsilon$, i.e.,

  $$\begin{array}{c}\hfill \sum _{x=1}^L{P}_{type-1}\left(x\right)=1-\epsilon ,\end{array}$$

  (37)

  consequently, it holds that

  $$\begin{array}{c}\hfill P_{\mathrm{e}}^{\left(L\right)}\left(P_{type-1}\right)=\epsilon ,\end{array}$$

  (38)
• the first $J-1$ probability masses are equal to that of $Q^{\downarrow }$, i.e.,

  $$\begin{array}{c}\hfill P_{type-1}\left(x\right)=Q^{\downarrow }\left(x\right)(\mathrm{for}1\le x\le J-1),\end{array}$$

  (39)
• the probability masses for $J\le x\le L$ are equal to $\mathcal{V}\left(J\right)$, i.e.,

  $$\begin{array}{c}\hfill P_{type-1}\left(x\right)=\mathcal{V}\left(J\right)(\mathrm{for}J\le x\le L),\end{array}$$

  (40)
• the probability masses for $L+1\le x\le K_1$ are equal to $\mathcal{W}\left(K_1\right)$, i.e.,

  $$\begin{array}{c}\hfill P_{type-1}\left(x\right)=\mathcal{W}\left(K_1\right)(\mathrm{for}L+1\le x\le K_1),\end{array}$$

  (41)
• the probability masses for $x\ge K_1+1$ are equal to that of $Q^{\downarrow }$, i.e.,

  $$\begin{array}{c}\hfill P_{type-1}\left(x\right)=Q^{\downarrow }\left(x\right)(\mathrm{for}x\ge K_1+1),\end{array}$$

  (42)

  and
• it holds that $P_{type-1}$ majorizes Q.

Proof of Proposition 4.

See Appendix C. □

Although positive tolerated probabilities of error (i.e., $\epsilon >0$) are highly interesting in most of the lossless communication systems, the scenario in which the error events with positive probabilities are not allowed (i.e., $\epsilon =0$) is also important to deal with the error-free communication systems. The following theorem is an error-free version of Theorem 1.

Theorem 2.

Suppose that $\epsilon =0$ and $\mathcal{Y}$ is at least countably infinite. Then, it holds that

$$\begin{array}{c}\hfill {\mathbb{H}}_{\varphi }(Q,L,0,\mathcal{Y})\le \varphi \left(P_{type-1}\right)\end{array}$$

(43)

with equality if $supp\left(Q\right):=\{x\in \mathcal{X}\mid Q(x)>0\}$ is finite or $J=L$. Moreover, if the cardinality of $\mathcal{Y}$ is at least the cardinality of the continuum $\mathbb{R}$, then there exists a σ-algebra on $\mathcal{Y}$ satisfying (43) with equality.

Proof of Theorem 2.

See Section 6.2. □

Remark 2.

Note that $J=L$ holds under the unique decoding rule (i.e., $L=1$); that is, we see from Theorem 2 that (43) holds with equality if $L=1$. The inequality $J<L$ occurs only if a non-unique decoding rule (i.e., $L>1$) is considered. In Theorem 2, the existence of a σ-algebra on an uncountably infinite alphabet $\mathcal{Y}$ in which (43) holds with equality is due to Révész’s generalization of the Birkhoff–von Neumann decomposition via Kolmogorov’s extension theorem; see Section 6.1 and Section 6.2 for technical details.

Consider the case where $\mathcal{Y}$ is a finite alphabet. Define the extremal distribution of type-2 as the following $\mathcal{X}$-marginal,

$$\begin{array}{c}\hfill P_{type-2}\left(x\right)=P_{type-2}^{(Q,L,\epsilon ,\mathcal{Y})}\left(x\right):=\left\{\begin{array}{cc}{Q}^{\downarrow }\left(x\right)\hfill & \mathrm{if}1\le x<J\mathrm{or}{K}_2<x<\infty ,\hfill \\ \mathcal{V}\left(J\right)\hfill & \mathrm{if}J\le x\le L,\hfill \\ \mathcal{W}\left(K_2\right)\hfill & \mathrm{if}L<x\le K_2\hfill \end{array}\right.\end{array}$$

(44)

for each $x\in \mathcal{X}$, where the three quantities $\mathcal{V}(·)$, $\mathcal{W}(·)$, and J are defined in (31), (32), and (33), respectively, and $K_2$ is chosen so that

$$\begin{array}{cc}\hfill K_2=K_2(Q,L,\epsilon ,\mathcal{Y})& :=max\{L\le k\le L·|\mathcal{Y}|\mid \mathcal{W}\left(k\right)<Q^{\downarrow }\left(k\right)\}.\hfill \end{array}$$

(45)

Moreover, define the integer D by

$$\begin{array}{c}\hfill D=D(Q,L,\epsilon ,\mathcal{Y}):=min\left\{\left(\genfrac{}{}{0pt}{}{K_2-J+1}{L-J+1}\right),{(K_2-J)}^2+1\right\},\end{array}$$

(46)

where $\left(\genfrac{}{}{0pt}{}{a}{b}\right):=\frac{a!}{b!(a-b)!}$ stands for the binomial coefficient for two integers $0\le b\le a$. A graphical representation of $P_{type-2}$ is illustrated in Figure 3. When $\mathcal{Y}$ is finite, the Fano-type inequality stated in Theorems 1 and 2 can be tightened as follows:

Figure 3. Plot of making the extremal distribution of type-2 defined in (44) from an $\mathcal{X}$-marginal Q, where $L=3$ and $\left|\mathcal{Y}\right|=2$. Each bar represents a probability mass of the decreasing rearrangement $Q^{\downarrow }$.

Theorem 3.

Suppose that $\mathcal{Y}$ is finite. Then, it holds that

$$\begin{array}{c}\hfill {\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})\le \varphi \left(P_{type-2}\right)\end{array}$$

(47)

with equality if $\epsilon =P_{\mathrm{e}}^{\left(L\right)}\left(Q\right)$ or $\left|\mathcal{Y}\right|\ge D$.

Proof of Theorem 3.

See Section 6.3. □

Similar to Theorems 1 and 2, the Fano-type inequality stated in (47) of Theorem 3 is formulated by the extremal distribution $P_{type-2}$ defined in (44). The difference between $P_{type-1}$ and $P_{type-2}$ is only the difference between $K_1$ and $K_2$ defined in (34) and (45), respectively.

Remark 3.

In contrast to Theorems 1 and 2, Theorem 3 holds in both cases: $\epsilon >0$ and $\epsilon =0$. By Lemma 5 stated in Section 6.1, it can be verified that $P_{type-2}$ majorizes $P_{type-1}$, and it follows from Proposition 1 that

$$\begin{array}{c}\hfill \varphi \left(P_{type-2}\right)\le \varphi \left(P_{type-1}\right).\end{array}$$

(48)

Namely, the Fano-type inequalities stated in Theorems 1 and 2 also holds for finite $\mathcal{Y}$. In other words, it holds that

$$\begin{array}{c}\hfill {\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})\le \varphi \left(P_{\mathrm{type}-1}\right)\end{array}$$

(49)

for every nonempty alphabet $\mathcal{Y}$, provided that $(Q,L,\epsilon ,\mathcal{Y})$ satisfies (29). As $\left|\mathcal{Y}\right|\ge D$ if $L=1$ (see (46)), another benefit of Theorem 3 is that the Fano-type inequality is always sharp under a unique decoding rule (i.e., $L=1$).

So far, it is assumed that the probability law $P_X$ of the $\mathcal{X}$-valued r.v. X is fixed to a given $\mathcal{X}$-marginal Q. When we assume that X is supported on a finite subalphabet of $\mathcal{X}$, we can loosen and simplify our Fano-type inequalities by removing the constraint that $P_X=Q$. Let L and M be two integers satisfying $1\le L<M$, $\epsilon$ a real number satisfying $0\le \epsilon \le 1-L/M$, and $\mathcal{Y}$ a nonempty alphabet. Consider the following maximization,

$$\begin{array}{c}\hfill {\mathbb{H}}_{\varphi }(M,L,\epsilon ,\mathcal{Y}):=\underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon }{max}{\mathsf{H}}_{\varphi }(X\mid Y),\end{array}$$

(50)

where the maximization is taken over the pairs $(X,Y)$ of r.v.’s satisfying (i) X is $\{1,\cdots ,M\}$-valued, (ii) Y is $\mathcal{Y}$-valued, and (iii) $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon$.

Theorem 4.

It holds that

$$\begin{array}{c}\hfill {\mathbb{H}}_{\varphi }(M,L,\epsilon ,\mathcal{Y})=\varphi \left(P_{type-0}\right),\end{array}$$

(51)

where $P_{type-0}$ is defined in (12).

Proof of Theorem 4.

See Section 6.4. □

Remark 4.

Although Theorems 1–3 depend on the cardinality of $\mathcal{Y}$, the Fano-type inequality stated in Theorem 4 does not depend on it whenever $\mathcal{Y}$ is nonempty.

4. Special Cases: Fano-Type Inequalities on Shannon’s and Rényi’s Information Measures

In this section, we specialize our Fano-type inequalities stated in Theorems 1–4 from general conditional information measures ${\mathsf{H}}_{\varphi }(X\mid Y)$ to Shannon’s and Rényi’s information measures. We then recover several known results such as those in [1,20,21,22,23] along the way.

4.1. On Shannon’s Information Measures

The conditional Shannon entropy [62] of an $\mathcal{X}$-valued r.v. X given a $\mathcal{Y}$-valued r.v. Y is defined by

$$\begin{array}{c}\hfill H(X\mid Y):=\mathbb{E}\left[H\left(P_{X|Y}\right)\right]=\mathbb{E}\left[\sum _{x\in \mathcal{X}}{P}_{X|Y}\left(x\right)log\frac{1}{P_{X|Y}\left(x\right)}\right],\end{array}$$

(52)

where the (unconditional) Shannon entropy of an $\mathcal{X}$-marginal P is defined by

$$\begin{array}{c}\hfill H\left(P\right):=\sum _{x\in \mathcal{X}}P\left(x\right)log\frac{1}{P\left(x\right)}.\end{array}$$

(53)

Remark 5.

It can be verified by the monotone convergence theorem (cf. [63] Theorem 10.1.7) that

$$\begin{array}{c}\hfill H(X\mid Y)=\mathbb{E}\left[log\frac{1}{P_{X|Y}\left(X\right)}\right],\end{array}$$

(54)

provided that the right-hand side of (54) is finite. In some cases, it is convenient to define the conditional Shannon entropy $H(X\mid Y)$ by the right-hand side of (54) (see, e.g., [64]).

The following proposition is a well-known property of Shannon’s information measures.

Proposition 5

(Topsøe [60]). The Shannon entropy $H(·)$ is symmetric, concave, and lower semicontinuous.

Namely, the conditional Shannon entropy $H(X\mid Y)$ is a special case of ${\mathsf{H}}_{\varphi }(X\mid Y)$ with $\varphi =H$. Therefore, defining the quantity

$$\begin{array}{c}\hfill \mathbb{H}(Q,L,\epsilon ,\mathcal{Y}):={\mathbb{H}}_H(Q,L,\epsilon ,\mathcal{Y})=\underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon ,P_X=Q}{\sup }H(X\mid Y),\end{array}$$

(55)

we readily observe the following corollary.

Corollary 1.

Suppose that $\epsilon >0$ and the cardinality of $\mathcal{Y}$ is at least countably infinite. Then, it holds that

$$\begin{array}{cc}\hfill \mathbb{H}(Q,L,\epsilon ,\mathcal{Y})& =H\left(P_{type-1}\right)\hfill \\ \hfill & =(J-L+1)\mathcal{V}\left(J\right)log\frac{1}{\mathcal{V}\left(J\right)}+(K_1-L)\mathcal{W}\left(K_1\right)log\frac{1}{\mathcal{W}\left(K_1\right)}+\sum _{\begin{array}{c}x=1:\\ x<J\mathrm{or}x>K_1\end{array}}^{\infty }{Q}^{\downarrow }\left(x\right)log\frac{1}{Q^{\downarrow }\left(x\right)}.\hfill \end{array}$$

(56)

Proof of Corollary 1.

Corollary 1 is a direct consequence of Theorem 1 and Proposition 5. □

Remark 6.

Applying Theorem 2 instead of Theorem 1, an error-free version (i.e., $\epsilon =0$) of Corollary 1 can be considered.

Remark 7.

Note that Corollary 1 coincides with Theorem 1 of [21] if $L=1$ and $\mathcal{Y}=\mathcal{X}$. Moreover, we observe from (10) and Corollary 1 that

$$\begin{array}{cc}\hfill \mathbb{I}(Q,\epsilon )& =H\left(Q\right)-\mathbb{H}(Q,1,\epsilon ,\mathcal{X})\hfill \\ \hfill & =\sum _{x=1}^{K_1}{Q}^{\downarrow }\left(x\right)log\frac{1}{Q^{\downarrow }\left(x\right)}+\mathcal{V}\left(1\right)log\mathcal{V}\left(1\right)+(K_1-1)\mathcal{W}\left(K_1\right)log\mathcal{W}\left(K_1\right)\hfill \end{array}$$

(57)

for every $\mathcal{X}$-marginal Q and every tolerated probability of error $0\le \epsilon \le 1-Q^{\downarrow }\left(1\right)$, where Erokhin’s function $\mathbb{I}(Q,\epsilon )$ is defined in (9). See Section 4.3 for details of generalizing of Erokhin’s function. Kostina–Polyanskiy–Verdú showed in Theorem 4 and Remark 3 of [41] that

$$\begin{array}{c}\hfill \mathbb{I}(Q^n,\epsilon )=n(1-\epsilon )H\left(Q\right)-\sqrt{\frac{nV\left(Q\right)}{2\pi }}{\mathrm{e}}^{-{\mathsf{\Phi }}^{-1}{\left(\epsilon \right)}^2/2}+\mathrm{O}(logn)(\mathrm{as}n\to \infty ),\end{array}$$

(58)

where $V\left(P\right)$ is defined by

$$\begin{array}{c}\hfill V\left(P\right):=\sum _{x\in \mathcal{X}}P\left(x\right){\left(log\frac{1}{P\left(x\right)}-H\left(P\right)\right)}^2\end{array}$$

(59)

and ${\mathsf{\Phi }}^{-1}(·)$ stands for the inverse of the Gaussian cumulative distribution function

$$\begin{array}{c}\hfill \mathsf{\Phi }\left(u\right):=\frac{1}{\sqrt{2\pi }}{\int }_{-\infty }^u{\mathrm{e}}^{-t^2/2}\mathrm{d}t.\end{array}$$

(60)

If $\mathcal{Y}$ is finite, then a tighter version of the Fano-type inequality than Corollary 1 can be obtained as follows:

Corollary 2.

Suppose that $\mathcal{Y}$ is finite. Then, it holds that

$$\begin{array}{cc}\hfill \mathbb{H}(Q,L,\epsilon ,\mathcal{Y})& \le H\left(P_{type-2}\right)\hfill \\ \hfill & =(J-L+1)\mathcal{V}\left(J\right)log\frac{1}{\mathcal{V}\left(J\right)}+(K_2-L)\mathcal{W}\left(K_2\right)log\frac{1}{\mathcal{W}\left(K_2\right)}+\sum _{\begin{array}{c}x=1:\\ x<J\mathrm{or}x>K_2\end{array}}^{\infty }{Q}^{\downarrow }\left(x\right)log\frac{1}{Q^{\downarrow }\left(x\right)},\hfill \end{array}$$

(61)

with equality if $\epsilon =P_{\mathrm{e}}^{\left(L\right)}\left(Q\right)$ or $\left|\mathcal{Y}\right|\ge D$.

Proof of Corollary 2.

Corollary 2 is a direct consequence of Theorem 3 and Proposition 5. □

Remark 8.

The inequality in (61) holds with equality if $L=1$ (cf. Remark 3). In fact, when $L=1$, Corollary 2 coincides with Ho–Verdú’s refinement of Erokhin’s function $\mathbb{I}(Q,\epsilon )$ with finite $\mathcal{Y}$ (see Theorem 4 of [21]).

Similar to (50) and (55), we can define

$$\begin{array}{c}\hfill \mathbb{H}(M,L,\epsilon ,\mathcal{Y}):={\mathbb{H}}_H(M,L,\epsilon ,\mathcal{Y})=\underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon }{max}H(X\mid Y),\end{array}$$

(62)

and can give an explicit formula of $\mathbb{H}(M,L,\epsilon ,\mathcal{Y})$ as follows.

Corollary 3.

It holds that

$$\begin{array}{c}\hfill \mathbb{H}(M,L,\epsilon ,\mathcal{Y})=H\left(P_{type-0}\right)=h_2\left(\epsilon \right)+(1-\epsilon )logL+\epsilon log(M-L).\end{array}$$

(63)

Proof of Corollary 3.

(Corollary 3 is a direct consequence of Theorem 4 and Proposition 5. □

Remark 9.

Indeed, Corollary 3 states the classical Fano inequality with list decoding; see (11).

4.2. On Rényi’s Information Measures

Although the choices of Shannon’s information measures are unique based on a set of axioms (see, e.g., Theorem 3.6 of [6] and Chapter 3 of [4]), there are several different definitions of conditional Rényi entropies (cf. [65,66,67]). Among them, this study focuses on Arimoto’s and Hayashi’s conditional Rényi entropies [8,9]. Arimoto’s conditional Rényi entropy of X given Y is defined by

$$\begin{array}{c}\hfill H_{\alpha }^{\mathrm{Arimoto}}(X\mid Y):=\frac{\alpha }{1-\alpha }log\mathbb{E}[\parallel P_{X|Y}{\parallel }_{\alpha }]=\frac{\alpha }{1-\alpha }log\mathbb{E}\left[{\left(\sum _{x\in \mathcal{X}}{P}_{X|Y}{\left(x\right)}^{\alpha }\right)}^{1/\alpha }\right]\end{array}$$

(64)

for each order $\alpha \in (0,1)\cup (1,\infty )$, where the ${\ell }_{\alpha }$-norm of an $\mathcal{X}$-marginal P is defined by

$$\begin{array}{c}\hfill {\parallel P\parallel }_{\alpha }:={\left(\sum _{x\in \mathcal{X}}P{\left(x\right)}^{\alpha }\right)}^{1/\alpha }.\end{array}$$

(65)

Here, note that the (unconditional) Rényi entropy [55] of an $\mathcal{X}$-marginal P can be defined by

$$\begin{array}{c}\hfill H_{\alpha }\left(P\right):=\frac{\alpha }{1-\alpha }log{\parallel P\parallel }_{\alpha }=\frac{1}{1-\alpha }log\sum _{x\in \mathcal{X}}P{\left(x\right)}^{\alpha },\end{array}$$

(66)

i.e., it is a monotone function of the ${\ell }_{\alpha }$-norm. Basic properties of the ${\ell }_{\alpha }$-norm can be found in the following proposition.

Proposition 6.

The ${\ell }_{\alpha }$-norm ${\parallel ·\parallel }_{\alpha }$ is symmetric and lower semicontinuous. Moreover, it is concave (resp. convex) if $0<\alpha \le 1$ (resp. if $\alpha \ge 1$).

Proof of Proposition 6.

The symmetry is obvious. The lower semicontinuity was proven by Kovačević–Stanojević–Šenk in Theorem 5 of [35]. The concavity (resp. convexity) property can be verified by the reverse (resp. forward) Minkowski inequality. □

Proposition 6 implies that $H_{\alpha }^{\mathrm{Arimoto}}(X\mid Y)$ is a monotone function of ${\mathsf{H}}_{\varphi }(X\mid Y)$ with $\varphi ={\parallel ·\parallel }_{\alpha }$, i.e.,

$$\begin{array}{c}\hfill H_{\alpha }^{\mathrm{Arimoto}}(X\mid Y)=\frac{\alpha }{1-\alpha }log\left({\mathsf{H}}_{{\parallel ·\parallel }_{\alpha }}(X\mid Y)\right).\end{array}$$

(67)

On the other hand, Hayashi’s conditional Rényi entropy of X given Y is defined by

$$\begin{array}{c}\hfill H_{\alpha }^{\mathrm{Hayashi}}(X\mid Y):=\frac{1}{1-\alpha }log\mathbb{E}[\parallel P_{X|Y}{\parallel }_{\alpha }^{\alpha }]=\frac{1}{1-\alpha }log\mathbb{E}\left[\sum _{x\in \mathcal{X}}{P}_{X|Y}{\left(x\right)}^{\alpha }\right]\end{array}$$

(68)

for each order $\alpha \in (0,1)\cup (1,\infty )$. It is easy to see that ${\parallel ·\parallel }_{\alpha }^{\alpha }:\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ also admits the same properties as those stated in Proposition 6. Therefore, Hayashi’s conditional Rényi entropy $H_{\alpha }^{\mathrm{Hayashi}}(X\mid Y)$ is also a monotone function of ${\mathsf{H}}_{\varphi }(X\mid Y)$ with $\varphi ={\parallel ·\parallel }_{\alpha }^{\alpha }$, i.e.,

$$\begin{array}{c}\hfill H_{\alpha }^{\mathrm{Hayashi}}(X\mid Y)=\frac{1}{1-\alpha }log\left({\mathsf{H}}_{{\parallel ·\parallel }_{\alpha }^{\alpha }}(X\mid Y)\right).\end{array}$$

(69)

It can be verified by Jensen’s inequality (see, e.g., Proposition 1 of [66]) that

$$\begin{array}{c}\hfill H_{\alpha }^{\mathrm{Hayashi}}(X\mid Y)\le H_{\alpha }^{\mathrm{Arimoto}}(X\mid Y).\end{array}$$

(70)

Similar to (7), we now define

$$\begin{array}{c}\hfill {\mathbb{H}}_{\alpha }^{†}(Q,L,\epsilon ,\mathcal{Y}):=\underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon ,P_X=Q}{\sup }{H}_{\alpha }^{†}(X\mid Y)\end{array}$$

(71)

for each $†\in \{\mathrm{Arimoto},\mathrm{Hayashi}\}$ and each $\alpha \in (0,1)\cup (1,\infty )$. Then, we can establish the Fano-type inequality on Rényi’s information measures as follows.

Corollary 4.

Suppose that $\epsilon >0$ and the cardinality of $\mathcal{Y}$ is at least countably infinite. For every $†\in \{\mathrm{Arimoto},\mathrm{Hayashi}\}$ and $\alpha \in (0,1)\cup (1,\infty )$, it holds that

$$\begin{array}{cc}\hfill {\mathbb{H}}_{\alpha }^{†}(Q,L,\epsilon ,\mathcal{Y})& =H_{\alpha }\left(P_{type-1}\right)\hfill \\ \hfill & =\frac{1}{1-\alpha }log\left((J-L+1)\mathcal{V}{\left(J\right)}^{\alpha }+(K_1-L)\mathcal{W}{\left(K_1\right)}^{\alpha }+\sum _{\begin{array}{c}x=1:\\ x<J\mathrm{or}x>K_1\end{array}}^{\infty }{Q}^{\downarrow }{\left(x\right)}^{\alpha }\right).\hfill \end{array}$$

(72)

Proof of Corollary 4.

Let $†=\mathrm{Arimoto}$. It follows from Theorem 1 and Proposition 6 that

$$\begin{array}{cc}\hfill 0<\alpha \le 1& ⟹\underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon ,P_X=Q}{\sup }\mathbb{E}\left[\parallel P_{X|Y}{\parallel }_{\alpha }\right]={\parallel P_{type-1}\parallel }_{\alpha },\hfill \end{array}$$

(73)

$$\begin{array}{cc}\hfill \alpha \ge 1& ⟹\underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon ,P_X=Q}{inf}\mathbb{E}\left[\parallel P_{X|Y}{\parallel }_{\alpha }\right]={\parallel P_{type-1}\parallel }_{\alpha }.\hfill \end{array}$$

(74)

As the mapping $u\mapsto (\alpha /(1-\alpha \left)\right)logu$ is strictly increasing (resp. strictly decreasing) if $0<\alpha <1$ (resp. if $\alpha >1$), it follows from (66), (67), (73), and (74) that

$$\begin{array}{c}\hfill \underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon ,P_X=Q}{\sup }{H}_{\alpha }^{\mathrm{Arimoto}}(X\mid Y)=H_{\alpha }\left(P_{type-1}\right).\end{array}$$

(75)

The proof for the case when $†=\mathrm{Hayashi}$ is the same as above, proving Corollary 4. □

Remark 10.

Applying Theorem 2 instead of Theorem 1, an error-free version (i.e., $\epsilon =0$) of Corollary 4 can be considered.

Remark 11.

Although Hayashi’s conditional Rényi entropy is smaller than Arimoto’s one in general (see (70)), Corollary 4 implies that the maximization problem ${\mathbb{H}}_{\alpha }^{†}(Q,L,\epsilon ,\mathcal{Y})$ results in the same Rényi entropy $H_{\alpha }\left(P_{type-1}\right)$ for each $†\in \{\mathrm{Arimoto},\mathrm{Hayashi}\}$.

When $\mathcal{Y}$ is finite, a tighter Fano-type inequality than Corollary 4 can be obtained as follows.

Corollary 5.

Suppose that $\mathcal{Y}$ is finite. For any $†\in \{\mathrm{Arimoto},\mathrm{Hayashi}\}$ and $\alpha \in (0,1)\cup (1,\infty )$, it holds that

$$\begin{array}{cc}\hfill {\mathbb{H}}_{\alpha }^{†}(Q,L,\epsilon ,\mathcal{Y})& \le H_{\alpha }\left(P_{type-2}\right)\hfill \\ \hfill & =\frac{1}{1-\alpha }log\left((J-L+1)\mathcal{V}{\left(J\right)}^{\alpha }+(K_2-L)\mathcal{W}{\left(K_2\right)}^{\alpha }+\sum _{\begin{array}{c}x=1:\\ x<J\mathrm{or}x>K_2\end{array}}^{\infty }{Q}^{\downarrow }{\left(x\right)}^{\alpha }\right),\hfill \end{array}$$

(76)

with equality if $\epsilon =P_{\mathrm{e}}^{\left(L\right)}\left(Q\right)$ or $\left|\mathcal{Y}\right|\ge D$.

Proof of Corollary 5.

The proof is the same as the proof of Corollary 4 by replacing Theorem 1 by Theorem 3. □

Similar to (50) and (71), define

$$\begin{array}{c}\hfill {\mathbb{H}}_{\alpha }^{†}(M,L,\epsilon ,\mathcal{Y}):=\underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon }{max}{H}_{\alpha }^{†}(X\mid Y)\end{array}$$

(77)

for each $†\in \{\mathrm{Arimoto},\mathrm{Hayashi}\}$ and each $\alpha \in (0,1)\cup (1,\infty )$.

Corollary 6.

For every $†\in \{\mathrm{Arimoto},\mathrm{Hayashi}\}$ and $\alpha \in (0,1)\cup (1,\infty )$, it holds that

$$\begin{array}{c}\hfill {\mathbb{H}}_{\alpha }^{†}(M,L,\epsilon ,\mathcal{Y})=H_{\alpha }\left(P_{type-0}\right)=\frac{1}{1-\alpha }log\left(L^{1-\alpha }(1-\epsilon )+{(M-L)}^{1-\alpha }{\epsilon }^{\alpha }\right).\end{array}$$

(78)

Proof of Corollary 6.

The proof is the same as the proof of Corollary 4 by replacing Theorem 1 by Theorem 4. □

Remark 12.

When $†=\mathrm{Arimoto}$, Corollary 6 coincides with Sason–Verdú’s generalization (cf. Theorem 8 of [23]) of Fano’s inequality for Rényi’s information measures with list decoding (see (15)).

Remark 13.

It follows by l’Hôpital’s rule that

$$\begin{array}{cc}\hfill \underset{\alpha \to 1}{lim}{H}_{\alpha }\left(P_{type-0}\right)& =H\left(P_{type-0}\right),\hfill \end{array}$$

(79)

$$\begin{array}{cc}\hfill \underset{\alpha \to 1}{lim}{H}_{\alpha }\left(P_{type-1}\right)& =H\left(P_{type-1}\right),\hfill \end{array}$$

(80)

$$\begin{array}{cc}\hfill \underset{\alpha \to 1}{lim}{H}_{\alpha }\left(P_{type-2}\right)& =H\left(P_{type-2}\right).\hfill \end{array}$$

(81)

Therefore, our Fano-type inequalities stated in Corollaries 1–6 satisfy the continuity of Shannon’s and Rényi’s information measures with respect to the order $0<\alpha <\infty$.

4.3. Generalization of Erokhin’s Function to α-Mutual Information

Erokhin’s function $\mathbb{I}(Q,\epsilon )$ defined in (9) can be generalized to the $\alpha$-mutual information (cf. [68]) as follows: Let X be an $\mathcal{X}$-valued r.v. and Y a $\mathcal{Y}$-valued r.v. Sibson’s $\alpha$-mutual information [24] (see also Equation (32) of [68], Equation (13) of [69], and Definition 7 of [70]) is defined by

$$\begin{array}{c}\hfill I_{\alpha }^{\mathrm{Sibson}}(X\wedge Y):=\underset{Q_Y}{inf}{D}_{\alpha }(P_{X,Y}\parallel P_X\times Q_Y)\end{array}$$

(82)

for each $0<\alpha <\infty$, where $P_{X,Y}$ (resp. $P_X$) denotes the probability measure on $\mathcal{X}\times \mathcal{Y}$ (resp. $\mathcal{X}$) induced by the pair $(X,Y)$ of r.v.’s (resp. the r.v. X), the infimum is taken over the probability measures $Q_Y$ on $\mathcal{Y}$, and the Rényi divergence [55] between two probability measures $\mu$ and $\nu$ on $\mathcal{A}$ is defined by

$$\begin{array}{c}\hfill D_{\alpha }(\mu \parallel \nu ):=\left\{\begin{array}{cc}\frac{1}{\alpha -1}log\left({\int }_{\mathcal{A}}{\left(\frac{\mathrm{d}\mu }{\mathrm{d}\nu }\right)}^{\alpha }\mathrm{d}\nu \right)\hfill & \mathrm{if}\mu \ll \nu \mathrm{and}\alpha \ne 1,\hfill \\ {\int }_{\mathcal{A}}\left(log\frac{\mathrm{d}\mu }{\mathrm{d}\nu }\right)\mathrm{d}\mu \hfill & \mathrm{if}\mu \ll \nu \mathrm{and}\alpha =1,\hfill \\ \infty \hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(83)

for each $0<\alpha <\infty$. Note that Sibson’s $\alpha$-mutual information coincides with the ordinary mutual information when $\alpha =1$, i.e., it holds that $I(X\wedge Y)=I_1(X\wedge Y)$. Similar to (7) and (9), given a system $(Q,L,\epsilon ,\mathcal{Y})$ satisfying (29), define

$$\begin{array}{c}\hfill {\mathbb{I}}_{\alpha }^{\mathrm{Sibson}}(Q,L,\epsilon ,\mathcal{Y}):=\underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon ,P_X=Q}{inf}{I}_{\alpha }^{\mathrm{Sibson}}(X\wedge Y),\end{array}$$

(84)

where the infimum is taken over the pairs of r.v.’s X and Y in which (i) X is $\mathcal{X}$-valued, (ii) Y is $\mathcal{Y}$-valued, (iii) $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon$, and (iv) $P_X=Q$. By convention, we denote by

$$\begin{array}{c}\hfill \mathbb{I}(Q,L,\epsilon ,\mathcal{Y}):={\mathbb{I}}_1^{\mathrm{Sibson}}(Q,L,\epsilon ,\mathcal{Y}).\end{array}$$

(85)

It is clear that this definition can be specialized to Erokhin’s function $\mathbb{I}(Q,\epsilon )$ defined in (9); in other words, it holds that

$$\begin{array}{c}\hfill \mathbb{I}(Q,1,\epsilon ,\mathcal{X})=\mathbb{I}(Q,\epsilon );\end{array}$$

(86)

see Remark 7.

Corollary 7

(When $\alpha =1$). Suppose that $\epsilon >0$ and the cardinality of $\mathcal{Y}$ is at least countably infinite. Then, it holds that

$$\begin{array}{cc}\hfill \mathbb{I}(Q,L,\epsilon ,\mathcal{Y})& =H\left(Q\right)-\mathbb{H}(Q,L,\epsilon ,\mathcal{Y})\hfill \\ \hfill & =\sum _{x=J}^{K_1}{Q}^{\downarrow }\left(x\right)log\frac{1}{Q^{\downarrow }\left(x\right)}+(J-L+1)\mathcal{V}\left(J\right)log\mathcal{V}\left(J\right)+(K_1-L)\mathcal{W}\left(K_1\right)log\mathcal{W}\left(K_1\right).\hfill \end{array}$$

(87)

Proof of Corollary 7.

The equality in (87) is trivial from the well-known identity $I(X\wedge Y)=H\left(X\right)-H(X\mid Y)$. The inequality in (87) follows from Corollary 1, completing the proof. □

Corollary 8

(Sibson, when $\alpha \ne 1$). Suppose that $\epsilon >0$ and $\mathcal{Y}$ is countably infinite. For every $\alpha \in (0,1)\cup (1,\infty )$, it holds that

$$\begin{array}{cc}\hfill {\mathbb{I}}_{\alpha }^{\mathrm{Sibson}}(Q,L,\epsilon ,\mathcal{Y})& =H_{\alpha }\left(Q^{(1/\alpha )}\right)-{\mathbb{H}}_{\alpha }^{\mathrm{Arimoto}}(Q^{(1/\alpha )},L,\epsilon ,\mathcal{Y})\hfill \\ \hfill & =\frac{1}{\alpha -1}log\left(1-\sum _{x=J^{(1/\alpha )}}^{K_1^{(1/\alpha )}}{Q}^{\downarrow }\left(x\right)\right.\hfill \\ \hfill & \left.\phantom{\sum _{\begin{array}{c}x=1:\\ x<J\mathrm{or}x>K_1\end{array}}^{\infty }}+\left((J^{(1/\alpha )}-L+1){\mathcal{V}}^{(1/\alpha )}{\left(J^{(1/\alpha )}\right)}^{\alpha }+(K_1^{(1/\alpha )}-L){\mathcal{W}}^{(1/\alpha )}{\left(K_1^{(1/\alpha )}\right)}^{\alpha }\right){\left(\sum _{x\in \mathcal{X}}Q{\left(x\right)}^{1/\alpha }\right)}^{\alpha }\right),\hfill \end{array}$$

(88)

where $Q^{\left(s\right)}$ stands for the s-tilted distribution of Q with real parameter $0<s<\infty$, i.e.,

$$\begin{array}{c}\hfill Q^{\left(s\right)}\left(x\right):=\frac{Q{\left(x\right)}^s}{{\sum }_{x^{\prime }\in \mathcal{X}}Q{\left(x^{\prime }\right)}^s}\end{array}$$

(89)

for each $x\in \mathcal{X}$, and ${\mathcal{V}}^{\left(s\right)}(·)$, $W^{\left(s\right)}(·)$, $J^{\left(s\right)}$, and $K_1^{\left(s\right)}$ are defined as in (31), (32), (33), and (34), respectively, by replacing the $\mathcal{X}$-marginal Q by the s-tilted distribution $Q^{\left(s\right)}$.

Proof of Corollary 8.

As Sibson’s identity [24] (see also [69], Equation (12)) states that

$$\begin{array}{c}\hfill D_{\alpha }(P_{X,Y}\parallel P_X\times Q_Y)=D_{\alpha }(P_{X,Y}\parallel P_X\times Q_{\alpha })+D_{\alpha }(Q_{\alpha }\parallel Q_Y),\end{array}$$

(90)

where $Q_{\alpha }$ stands for the probability distribution on $\mathcal{Y}$ given as

$$\begin{array}{c}\hfill Q_{\alpha }\left(y\right)={\left(\sum _{x\in \mathcal{X}}{P}_{X,Y}{(x,y)}^{\alpha }{P}_X{\left(x\right)}^{1-\alpha }\right)}^{1/\alpha }{\left(\sum _{y^{\prime }\in \mathcal{Y}}{\left(\sum _{x^{\prime }\in \mathcal{X}}{P}_{X,Y}{(x^{\prime },y^{\prime })}^{\alpha }{P}_X{\left(x^{\prime }\right)}^{1-\alpha }\right)}^{1/\alpha }\right)}^{-1}\end{array}$$

(91)

for each $y\in \mathcal{Y}$, we observe that

$$\begin{array}{c}\hfill I_{\alpha }^{\mathrm{Sibson}}(X\wedge Y)=\frac{\alpha }{\alpha -1}log\sum _{y\in \mathcal{Y}}{\left(\sum _{x\in \mathcal{X}}{P}_{X,Y}{(x,y)}^{\alpha }{P}_X{\left(x\right)}^{1-\alpha }\right)}^{1/\alpha }\end{array}$$

(92)

for every $\alpha \in (0,1)\cup (1,\infty )$, provided that $\mathcal{Y}$ is countable. On the other hand, it follows from ([8] Equation (13)) that

$$\begin{array}{c}\hfill I_{\alpha }^{\mathrm{Arimoto}}(X\wedge Y)=\frac{\alpha }{\alpha -1}log\sum _{y\in \mathcal{Y}}{\left(\sum _{x\in \mathcal{X}}\frac{P_{X,Y}{(x,y)}^{\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}{P}_X{\left(x^{\prime }\right)}^{\alpha }}\right)}^{1/\alpha }\end{array}$$

(93)

for every $\alpha \in (0,1)\cup (1,\infty )$, provided that $\mathcal{Y}$ is countable. Combining (92) and (93), we have the first equality in (88). Finally, the second equality in (88) follows from Corollary 4 after some algebra. This completes the proof of Corollary 8. □

In contrast to (82), Arimoto defined the $\alpha$-mutual information ([8], Equation (15)) by

$$\begin{array}{c}\hfill I_{\alpha }^{\mathrm{Arimoto}}(X\wedge Y):=H_{\alpha }\left(X\right)-H_{\alpha }^{\mathrm{Arimoto}}(X\mid Y)\end{array}$$

(94)

for every $\alpha \in (0,1)\cup (1,\infty )$. Similar to (84), one can define

$$\begin{array}{cc}\hfill {\mathbb{I}}_{\alpha }^{\mathrm{Arimoto}}(Q,L,\epsilon ,\mathcal{Y})& :=\underset{(X,Y):P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon ,P_X=Q}{inf}{I}_{\alpha }^{\mathrm{Arimoto}}(X\wedge Y),\hfill \end{array}$$

(95)

and a counterpart of Corollary 8 can be stated as follows.

Corollary 9

(Arimoto, when $\alpha \ne 1$). Suppose that $\epsilon >0$ and the cardinality of $\mathcal{Y}$ is at least countably infinite. For every $\alpha \in (0,1)\cup (1,\infty )$, it holds that

$$\begin{array}{cc}\hfill {\mathbb{I}}_{\alpha }^{\mathrm{Arimoto}}(Q,L,\epsilon ,\mathcal{Y})& =H_{\alpha }\left(Q\right)-{\mathbb{H}}_{\alpha }^{\mathrm{Arimoto}}(Q,L,\epsilon ,\mathcal{Y})\hfill \\ \hfill & =\frac{1}{\alpha -1}log\left(1-\sum _{x=J}^{K_1}{\left(Q^{\left(\alpha \right)}\right)}^{\downarrow }\left(x\right)\right.\hfill \\ \hfill & \left.+\left((J-L+1)\mathcal{V}{\left(J\right)}^{\alpha }+(K_1-L)\mathcal{W}{\left(K_1\right)}^{\alpha }\right){\left(\sum _{x\in \mathcal{X}}Q{\left(x\right)}^{\alpha }\right)}^{-1}\right).\hfill \end{array}$$

(96)

Proof of Corollary 9.

The first equality in (96) is obvious from the definition. The second equality in (96) follows from Corollary 4 after some algebra, completing the proof. □

When $\mathcal{Y}$ is finite, then the inequalities stated in Corollaries 7–9 can be tightened by Theorem 3 as in Corollaries 2 and 5. We omit to explicitly state these tightened inequalities in this paper.

5. Asymptotic Behaviors on Equivocations

In information theory, the equivocation or the remaining uncertainty of an r.v. X relative to a correlated r.v. Y has an important role in establishing fundamental limits of the optimal transmission ratio and/or rate in several communication models. Shannon’s equivocation $H(X\mid Y)$ is a well-known measure in the formulation of the notion of perfect secrecy of symmetric-key encryption in information-theoretic cryptography [71]. Iwamoto–Shikata [66] considered the extension of such a secrecy criterion by generalizing Shannon’s equivocation to Rényi’s equivocation by showing various desired properties of the latter. Recently, Hayashi–Tan [72] and Tan–Hayashi [73] studied the asymptotics of Shannon’s and Rényi’s equivocations when the side-information about the source is given via a various class of random hash functions with a fixed rate.

In this section, we assume that certain error probabilities vanish and we then establish asymptotic behaviors on Shannon’s, or sometimes on Rényi’s, equivocations via the Fano-type inequalities stated in Section 4.

5.1. Fano’s Inequality Meets the AEP

We consider a general form of the asymptotic equipartition property (AEP) as follows.

Definition 7

([25]). We say that a sequence of $\mathcal{X}$-valued r.v.’s $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$ satisfies the AEP if

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\mathbb{P}\left\{log\frac{1}{P_{X_n}\left(X_n\right)}\le (1-\delta )H\left(X_n\right)\right\}=0\end{array}$$

(97)

for every fixed $\delta >0$.

In the literature, the r.v. $X_n$ is commonly represented as a random vector $X_n=(Z_1^{\left(n\right)},\cdots ,Z_n^{\left(n\right)})$. The formulation without reference to random vectors means that $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$ is a general source in the sense of Page 100 of [33].

Let ${\left\{L_n\right\}}_{n=1}^{\infty }$ be a sequence of positive integers, ${\left\{{\mathcal{Y}}_n\right\}}_{n=1}^{\infty }$ a sequence of nonempty alphabets, and ${\left\{(X_n,Y_n)\right\}}_{n=1}^{\infty }$ a sequence of pairs of r.v.’s, where $X_n$ (resp. $Y_n$) is $\mathcal{X}$-valued (resp. ${\mathcal{Y}}_n$-valued) for each $n\ge 1$. As

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\mathbb{P}\{X_n\notin f_n\left(Y_n\right)\}=0⟹\underset{n\to \infty }{lim}{P}_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n)=0\end{array}$$

(98)

for any sequence of list decoders ${\{f_n:\mathcal{Y}\to \left(\genfrac{}{}{0pt}{}{\mathcal{X}}{L_n}\right)\}}_{n=1}^{\infty }$, it suffices to assume that $P_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n)=\mathrm{o}\left(1\right)$ as $n\to \infty$ in our analysis. The following theorem is a novel characterization of the AEP via Fano’s inequality.

Theorem 5.

Suppose that a general source $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$ satisfies the AEP, and $H\left(X_n\right)=\mathsf{\Omega }\left(1\right)$ as $n\to \infty$. Then, it holds that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{P}_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n)=0⟹|H(X_n\mid Y_n)-log{L}_n{|}^{+}=\mathrm{o}\left(H\left(X_n\right)\right),\end{array}$$

(99)

where ${\left|u\right|}^{+}:=max\{0,u\}$ for $u\in \mathbb{R}$. Consequently, it holds that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{P}_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n)=\underset{n\to \infty }{lim}\frac{log{L}_n}{H\left(X_n\right)}=0⟹\underset{n\to \infty }{lim}\frac{H(X_n\mid Y_n)}{H\left(X_n\right)}=0.\end{array}$$

(100)

Proof of Theorem 5.

See Section 7.1. □

The following three examples are particularizations of Theorem 5.

Example 1.

Let ${\left\{Z_n\right\}}_{n=1}^{\infty }$ be an i.i.d. source on a countably infinite alphabet $\mathcal{X}$ with finite Shannon entropy $H\left(Z_1\right)<\infty$. Suppose that $X_n=(Z_1,\cdots ,Z_n)$ and ${\mathcal{Y}}_n={\mathcal{X}}^n$ for each $n\ge 1$. Then, Theorem 5 states that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\mathbb{P}\{X_n\ne Y_n\}=0⟹\underset{n\to \infty }{lim}\frac{1}{n}H(X_n\mid Y_n)=0.\end{array}$$

(101)

This result is commonly referred to as the weak converse property of the source ${\left\{Z_n\right\}}_{n=1}^{\infty }$ in the unique decoding setting.

Example 2.

Let $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$ be a source as described in Example 1. Even if the list decoding setting, Theorem 5 states that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{P}_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n)=\underset{n\to \infty }{lim}\frac{1}{n}log{L}_n=0⟹\underset{n\to \infty }{lim}\frac{1}{n}H(X_n\mid Y_n)=0,\end{array}$$

(102)

similarly to Example 1. This is a key observation in Ahlswede–Gács–Körner’s proof of the strong converse property of degraded broadcast channels; see Chapter 5 of [42] (see also Section 3.6.2 of [43] and Lemma 1 of [48]).

Example 3.

Consider the Poisson source $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$ with growing mean ${\lambda }_n=\omega \left(1\right)$ as $n\to \infty$, i.e.,

$$\begin{array}{c}\hfill P_{X_n}\left(k\right)=\frac{{\lambda }_n^{k-1}{\mathrm{e}}^{-{\lambda }_n}}{(k-1)!}\mathrm{for}k\in \mathcal{X}=\{1,2,\cdots \}.\end{array}$$

(103)

It is known that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{H\left(X_n\right)}{(1/2)log{\lambda }_n}=1,\end{array}$$

(104)

and the Poisson source $\mathbf{X}$ satisfies the AEP (see [25]). Therefore, it follows from Theorem 5 that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{P}_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n)=0⟹{|H(X_n\mid Y_n)-log{L}_n|}^{+}=\mathrm{o}(log{\lambda }_n).\end{array}$$

(105)

The following example shows a general source that satisfies neither the AEP nor (99).

Example 4.

Let $L\ge 1$ be an integer, $\gamma >0$ a positive real, and ${\left\{{\delta }_n\right\}}_{n=1}^{\infty }$ a sequence of reals satisfying ${\delta }_n=\mathrm{o}\left(1\right)$ and $0<{\delta }_n<1$ for each $n\ge 1$. As $p\mapsto h_2\left(p\right)/p$ is continuous on $(0,1]$ and $h_2\left(p\right)/p\to \infty$ as $p\to 0^{+}$, one can find a sequence of reals ${\left\{p_n\right\}}_{n=1}^{\infty }$ satisfying $0<p_n\le min\{1,(1-{\delta }_n)/\left({\delta }_{n}L\right)\}$ for each $n\ge 1$ and

$$\begin{array}{c}\hfill \frac{{\delta }_n{h}_2\left(p_n\right)}{p_n}=\gamma \mathrm{for}\mathrm{sufficiently}\mathrm{large}n.\end{array}$$

(106)

Consider a general source $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$ whose component distributions are given by

$$\begin{array}{c}\hfill P_{X_n}\left(x\right)=\left\{\begin{array}{cc}\frac{1-{\delta }_n}{L}\hfill & \mathrm{if}1\le x\le L,\hfill \\ {\delta }_n{p}_n{(1-p_n)}^{x-(L+1)}\hfill & \mathrm{if}x\ge L+1\hfill \end{array}\right.\end{array}$$

(107)

for each $n\ge 1$. Suppose that $X_n⫫Y_n$ for each $n\ge 1$. After some algebra, we have

$$\begin{array}{cc}\hfill P_{\mathrm{e}}^{\left(L\right)}(X_n\mid Y_n)& =P_{\mathrm{e}}^{\left(L\right)}\left(X_n\right)={\delta }_n,\hfill \end{array}$$

(108)

$$\begin{array}{cc}\hfill H(X_n\mid Y_n)& =H\left(X_n\right)=h_2\left({\delta }_n\right)+(1-{\delta }_n)logL+\frac{{\delta }_n{h}_2\left(p_n\right)}{p_n}\hfill \end{array}$$

(109)

for each $n\ge 1$. Therefore, we observe that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{P}_{\mathrm{e}}^{\left(L\right)}(X_n\mid Y_n)=0\end{array}$$

(110)

holds, but

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{|H(X_n\mid Y_n){-logL|}^{+}}{H\left(X_n\right)}=0\end{array}$$

(111)

does not hold. In fact, it holds that $H\left(X_n\right)\to \gamma +logL$ as $n\to \infty$ and

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{P}_{X_n}\left(x\right)=\left\{\begin{array}{cc}\frac{1}{L}\hfill & \mathrm{if}1\le x\le L,\hfill \\ 0\hfill & \mathrm{if}x\ge L.\hfill \end{array}\right.\end{array}$$

(112)

Consequently, we also see that $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$ does not satisfy the AEP.

Example 4 implies that the AEP has an important role in Theorem 5.

5.2. Vanishing Unnormalized Rényi’s Equivocations

Let X be an $\mathcal{X}$-valued r.v. satisfying $H\left(X\right)<\infty$, ${\left\{L_n\right\}}_{n=1}^{\infty }$ a sequence of positive integers, ${\left\{{\mathcal{Y}}_n\right\}}_{n=1}^{\infty }$ a sequence of nonempty alphabets, and ${\left\{(X_n,Y_n)\right\}}_{n=1}^{\infty }$ a sequence of $\mathcal{X}\times {\mathcal{Y}}_n$-valued r.v.’s. The following theorem provides four conditions on a general source $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$ such that vanishing error probabilities implies vanishing unnormalized Shannon’s and Rényi’s equivocations.

Theorem 6.

Let $\alpha \ge 1$ be an order. Suppose that any one of the following four conditions hold,

(a) 

the order α is strictly larger than 1, i.e., $\alpha >1$,

(b) 

the sequence ${\left\{X_n\right\}}_{n=1}^{\infty }$ satisfies the AEP and $H\left(X_n\right)=\mathrm{O}\left(1\right)$ as $n\to \infty$,

(c) 

there exists an $n_0\ge 1$ such that $P_{X_n}$ majorizes $P_X$ for every $n\ge n_0$,

(d) 

the sequence ${\left\{X_n\right\}}_{n=1}^{\infty }$ converges in distribution to X and $H\left(X_n\right)\to H\left(X\right)$ as $n\to \infty$.

Then, it holds that for each $†\in \{\mathrm{Arimoto},\mathrm{Hayashi}\}$,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{P}_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n)=0⟹\underset{n\to \infty }{lim}|H_{\alpha }^{†}(X_n\mid Y_n)-log{L}_n{|}^{+}=0.\end{array}$$

(113)

Proof of Theorem 6.

See Section 7.2. □

In contrast to Condition (b) of Theorem 6, Conditions (a), (c), and (d) of Theorem 6 do not require the AEP to hold. Interestingly, Condition (a) of Theorem 6 states that (113) holds for every $\alpha >1$ and $†\in \{\mathrm{Arimoto},\mathrm{Hayashi}\}$ without any other conditions on the general source $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$.

Remark 14.

If $L_n=1$ for each $n\ge 1$, then Conditions (c) and (d) of Theorem 6 coincide with Ho–Verdú’s result stated in Theorem 18 of [21]. Moreover, if $L_n=1$ for each $n\ge 1$, and if $X_n$ is $\{1,\cdots ,M^n\}$-valued for each $n\ge 1$, then Condition (a) of Theorem 6 coincides with Sason–Verdú’s result stated in Assertion (a) of Theorem 4 of [23].

5.3. Under the Symbol-Wise Error Criterion

Let $\mathbf{L}={\left\{L_n\right\}}_{n=1}^{\infty }$ be a sequence of positive integers, ${\left\{{\mathcal{Y}}_n\right\}}_{n=1}^{\infty }$ a sequence of nonempty alphabets, and ${\left\{(X_n,Y_n)\right\}}_{n=1}^{\infty }$ a sequence of $\mathcal{X}\times {\mathcal{Y}}_n$-valued r.v.’s satisfying $H\left(X_n\right)<\infty$ for every $n\ge 1$. In this subsection, we focus on the minimum arithmetic-mean probability of symbol-wise list decoding error defined as

$$\begin{array}{c}\hfill P_{\mathrm{e},\mathrm{sym}.}^{\left(\mathbf{L}\right)}(X^n\mid Y^n):=\frac{1}{n}\sum _{i=1}^n{P}_{\mathrm{e}}^{\left(L_i\right)}(X_i\mid Y_i),\end{array}$$

(114)

where $X^n=(X_1,X_2,\cdots ,X_n)$ and $Y^n=(Y_1,Y_2,\cdots ,Y_n)$. Now, let X be an $\mathcal{X}$-valued r.v. satisfying $H\left(X\right)<\infty$. Under this symbol-wise error criterion, the following theorem holds.

Theorem 7.

Suppose that $P_{X_n}$ majorizes $P_X$ for sufficiently large n. Then, it holds that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{P}_{\mathrm{e},\mathrm{sym}.}^{\left(\mathbf{L}\right)}(X^n\mid Y^n)=0⟹\underset{n\to \infty }{lim\; sup}\frac{1}{n}H(X^n\mid Y^n)\le \underset{n\to \infty }{lim\; sup}log{L}_n.\end{array}$$

(115)

Proof of Theorem 7.

See Section 7.3. □

It is known that the classical Fano inequality stated in (1) can be extended from the average error criterion $\mathbb{P}\{X^n\ne Y^n\}$ to the symbol-wise error criterion $(1/n)\mathbb{E}\left[d_{\mathrm{H}}(X^n,Y^n)\right]$ (see Corollary 3.8 of [6]), where

$$\begin{array}{c}\hfill d_{\mathrm{H}}(x^n,y^n):=\left|\{1\le i\le n\mid x_i\ne y_i\}\right|\end{array}$$

(116)

stands for the Hamming distance between two strings $x^n=(x_1,\cdots ,n_n)$ and $y^n=(y_1,\cdots ,y_n)$. In fact, Theorem 7 states that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n}\mathbb{E}\left[d_{\mathrm{H}}(X^n,Y^n)\right]=0⟹\underset{n\to \infty }{lim}\frac{1}{n}H(X^n\mid Y^n)=0,\end{array}$$

(117)

provided that $P_{X_n}$ majorizes $P_X$ for sufficiently large n.

However, in the list decoding setting, we observe that $P_{\mathrm{e},\mathrm{sym}.}^{\left(\mathbf{L}\right)}(X^n\mid Y^n)=\mathrm{o}\left(1\right)$ does not imply $H(X^n\mid Y^n)=\mathrm{o}\left(n\right)$ in general. A counterexample can be readily constructed.

Example 5.

Let ${\left\{X_n\right\}}_{n=1}^{\infty }$ be uniformly distributed Bernoulli r.v.’s, and ${\left\{Y_n\right\}}_{n=1}^{\infty }$ arbitrary r.v.’s. Suppose that $(X_n,Y_n)⫫(X_m,Y_m)$ if $n\ne m$, $X_n⫫Y_n$ for each $n\ge 1$, and $L_n=2$ for each $n\ge 1$. Then, we observe that

$$\begin{array}{c}\hfill P_{\mathrm{e},\mathrm{sym}.}^{\left(\mathbf{L}\right)}(X^n\mid Y^n)=0\end{array}$$

(118)

for every $n\ge 1$, but

$$\begin{array}{c}\hfill H(X^n\mid Y^n)=nlog2\end{array}$$

(119)

for every $n\ge 1$.

6. Proofs of Fano-Type Inequalities

In this section, we prove Theorems 1–4 via majorization theory [10].

6.1. Proof of Theorem 1

We shall relax the feasible regions of the supremum in (7) via some lemmas, i.e., our preliminary results. Define a notion of symmetry for the conditional distribution $P_{X|Y}$ as follows.

Definition 8.

A jointly distributed pair $(X,Y)$ is said to be connected uniform-dispersively if $P_{X|Y}^{\downarrow }$ is almost surely constant.

Remark 15.

The term introduced in Definition 8 is inspired by uniformly dispersive channels named by Massey (see Page 77 of [12]). In fact, if $\mathcal{Y}$ is countable and X (resp. Y) denotes the output (resp. input) of a channel $P_{X|Y}$, then the channel $P_{X|Y}$ can be thought of as a uniformly dispersive channel, provided that $(X,Y)$ is connected uniform-dispersively. Initially, Fano said such channels to be uniform from the input; see Page 127 of [11]. Refer to Section II-A of [13] for several symmetry notions of channels.

Although an almost surely constant $P_{X|Y}$ implies the independence $X⫫Y$, note also that an almost surely constant $P_{X|Y}^{\downarrow }$ does not imply the independence. We now give the following lemma.

Lemma 1.

If a jointly distributed pair $(X,Y)$ is connected uniform-dispersively, then $P_{X|Y}$ majorizes $P_X$ a.s.

Proof of Lemma 1.

Let k be a positive integer. Choose a collection ${\left\{x_i\right\}}_{i=1}^k$ of k distinct elements in $\mathcal{X}$ so that

$$\begin{array}{c}\hfill P_X\left(x_i\right)=P_X^{\downarrow }\left(i\right)\end{array}$$

(120)

for every $1\le i\le k$. As

$$\begin{array}{c}\hfill \sum _{i=1}^k{P}_{X|Y}\left(x_i\right)\le \sum _{x=1}^k{P}_{X|Y}^{\downarrow }\left(x\right)(\mathrm{a}.\mathrm{s}.)\end{array}$$

(121)

and

$$\begin{array}{c}\hfill P_X\left(x\right)=\mathbb{E}\left[P_{X|Y}\left(x\right)\right]\end{array}$$

(122)

for each $x\in \mathcal{X}$, we observe that

$$\begin{array}{c}\hfill \sum _{x=1}^k{P}_X^{\downarrow }\left(x\right)=\mathbb{E}\left[\sum _{i=1}^k{P}_{X|Y}\left(x_i\right)\right]\le \mathbb{E}\left[\sum _{x=1}^k{P}_{X|Y}^{\downarrow }\left(x\right)\right].\end{array}$$

(123)

If $(X,Y)$ is connected uniform-dispersively (see Definition 8), then (123) implies that

$$\begin{array}{c}\hfill \sum _{x=1}^k{P}_X^{\downarrow }\left(x\right)\le \sum _{x=1}^k{P}_{X|Y}^{\downarrow }\left(x\right)(\mathrm{a}.\mathrm{s}.),\end{array}$$

(124)

which is indeed the majorization relation stated Definition 1, completing the proof of Lemma 1. □

Remark 16.

Lemma 1 is can be thought of as a novel characterization of uniformly dispersive channels via the majorization relation; see Remark 15. More precisely, given an input distribution P on $\mathcal{X}$ and a uniformly dispersive channel $W:\mathcal{X}\to \mathcal{Y}$ with countable output alphabet $\mathcal{Y}$, it holds that $W(·\mid x)$ majorizes the output distribution $PW$ for every $x\in \mathcal{X}$, where $PW$ is given by

$$\begin{array}{c}\hfill PW\left(y\right):=\sum _{x\in \mathcal{X}}P\left(x\right)W(y\mid x)\end{array}$$

(125)

for each $y\in \mathcal{Y}$.

Definition 9.

Let $\mathcal{A}$ be a collection of jointly distributed pairs of an $\mathcal{X}$-valued r.v. and a $\mathcal{Y}$-valued r.v. We say that $\mathcal{A}$ has balanced conditional distributions if $(X,Y)\in \mathcal{A}$ implies that there exists $(U,V)\in \mathcal{A}$ satisfying

$$\begin{array}{c}\hfill P_{U|V}^{\downarrow }\left(x\right)=\mathbb{E}\left[P_{X|Y}^{\downarrow }\left(x\right)\right](\mathrm{a}.\mathrm{s}.)\end{array}$$

(126)

for every $x\in \mathcal{X}$.

For such a collection $\mathcal{A}$, the following lemma holds.

Lemma 2.

Suppose that $\mathcal{A}$ has balanced conditional distributions. For any $(X,Y)\in \mathcal{A}$, there exists a pair $(U,V)\in \mathcal{A}$ connected uniform-dispersively such that

$$\begin{array}{c}\hfill {\mathsf{H}}_{\varphi }(U\mid V)\ge {\mathsf{H}}_{\varphi }(X\mid Y).\end{array}$$

(127)

Proof of Lemma 2.

For any $(X,Y)\in \mathcal{A}$, it holds that

$$\begin{array}{cc}\hfill {\mathsf{H}}_{\varphi }(X\mid Y)& \stackrel{\left(\mathrm{a}\right)}{=}\mathbb{E}\left[\varphi \left(P_{X|Y}^{\downarrow }\right)\right]\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{\le }\varphi \left(\mathbb{E}\left[P_{X|Y}^{\downarrow }\right]\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{=}\varphi \left(P_{U|V}^{\downarrow }\right)(\mathrm{a}.\mathrm{s}.)\hfill \\ \hfill & \stackrel{\left(\mathrm{d}\right)}{=}\mathbb{E}\left[\varphi \left(P_{U|V}\right)\right](\mathrm{a}.\mathrm{s}.)\hfill \\ \hfill & ={\mathsf{H}}_{\varphi }(U\mid V),\hfill \end{array}$$

(128)

where

• (a) follows by the symmetry of $\varphi$,
• (b) follows by Jensen’s inequality (see [14], Proposition A-2),
• (c) follows by the existence of a pair $(U,V)\in \mathcal{A}$ connected uniform-dispersively (see (126)), and
• (d) follows by the symmetry of $\varphi$ again.

This completes the proof of Lemma 2. □

For a system $(Q,L,\epsilon ,\mathcal{Y})$ satisfying (29), we now define a collection of pairs of r.v.’s as follows,

$$\begin{array}{c}\hfill \mathcal{R}(Q,L,\epsilon ,\mathcal{Y}):=\left\{(X,Y)|\begin{array}{c}X\mathrm{is}\mathcal{X}-\mathrm{valued},\hfill \\ Y\mathrm{is}\mathcal{Y}-\mathrm{valued},\hfill \\ P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon ,\hfill \\ P_X=Q\hfill \end{array}\right\}.\end{array}$$

(129)

Note that this is the feasible region of the supremum in (7). The main idea of proving Theorem 1 is to apply Lemma 2 for this collection. The collection $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y})$ does not, however, have balanced conditional distributions in general. More specifically, there exists a measurable space $\mathcal{Y}$ such that $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y})$ does not have balanced conditional distributions even if $\mathcal{Y}$ is standard Borel. Fortunately, the following lemma can avoid this issue by blowing-up the collection $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y})$ via the infinite-dimensional version of Birkhoff’s theorem [18].

Lemma 3.

If the cardinality of $\mathcal{Y}$ is at least the cardinality of the continuum $\mathbb{R}$, then there exists a σ-algebra on $\mathcal{Y}$ such that the collection $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y})$ has balanced conditional distributions.

Proof of Lemma 3.

First, we shall choose an appropriate alphabet $\mathcal{Y}$ so that its cardinality is the cardinality of the continuum. Denote by $\mathsf{\Psi }$ the set of $\infty \times \infty$ permutation matrices, where an $\infty \times \infty$ permutation matrix is a real matrix $\mathsf{\Pi }={\left\{{\pi }_{i,j}\right\}}_{i,j=1}^{\infty }$ satisfying either ${\pi }_{i,j}=0$ or ${\pi }_{i,j}=1$ for each $1\le i,j<\infty$, and

$$\begin{array}{cc}\hfill \sum _{j=1}^{\infty }{\pi }_{i,j}& =1\mathrm{for}\mathrm{each}1\le i<\infty ,\hfill \end{array}$$

(130)

$$\begin{array}{cc}\hfill \sum _{i=1}^{\infty }{\pi }_{i,j}& =1\mathrm{for}\mathrm{each}1\le j<\infty .\hfill \end{array}$$

(131)

For an $\infty \times \infty$ permutation matrix $\mathsf{\Pi }={\left\{{\pi }_{i,j}\right\}}_{i,j}\in \mathsf{\Psi }$, define the permutation ${\psi }_{\mathsf{\Pi }}$ on $\mathcal{X}=\{1,2,\cdots \}$ by

$$\begin{array}{c}\hfill {\psi }_{\mathsf{\Pi }}\left(i\right):=\sum _{j=1}^{\infty }{\pi }_{i,j}j.\end{array}$$

(132)

It is known that there is a one-to-one correspondence between the permutation matrices $\mathsf{\Pi }$ and the bijections ${\psi }_{\mathsf{\Pi }}$; and thus, the cardinality of $\mathsf{\Psi }$ is the cardinality of the continuum. Therefore, in this proof, we may assume without loss of generality that $\mathcal{Y}=\mathsf{\Psi }$.

Second, we shall construct an appropriate $\sigma$-algebra on $\mathcal{Y}$ via the infinite-dimensional version of Birkhoff’s theorem (cf. Theorem 2 of [18]) for $\infty \times \infty$ doubly stochastic matrices, where an $\infty \times \infty$ doubly stochastic matrix is a real matrix $\mathbf{M}={\left\{m_{i,j}\right\}}_{i,j=1}^{\infty }$ satisfying $0\le m_{i,j}\le 1$ for each $1\le i,j<\infty$, and

$$\begin{array}{cc}\hfill \sum _{j=1}^{\infty }{m}_{i,j}& =1\mathrm{for}\mathrm{each}1\le i<\infty ,\hfill \end{array}$$

(133)

$$\begin{array}{cc}\hfill \sum _{i=1}^{\infty }{m}_{i,j}& =1\mathrm{for}\mathrm{each}1\le j<\infty .\hfill \end{array}$$

(134)

Similar to $\mathsf{\Psi }$, denote by ${\mathsf{\Psi }}_{i,j}$ the set of $\infty \times \infty$ permutation matrices in which the entry in the ith row and the jth column is 1, where note that ${\mathsf{\Psi }}_{i,j}\subset \mathcal{Y}$. Then, the following lemma holds.

Lemma 4

(infinite-dimensional version of Birkhoff’s theorem; cf. Theorem 2 of [18]). There exists a σ-algebra Γ on $\mathcal{Y}$ such that (i) ${\mathsf{\Psi }}_{i,j}\in \Gamma$ for every $1\le i,j<\infty$ and (ii) for any $\infty \times \infty$ doubly stochastic matrix $\mathbf{M}={\left\{m_{i,j}\right\}}_{i,j=1}^{\infty }$, there exists a probability measure μ on $(\mathcal{Y},\Gamma )$ such that $\mu \left({\mathsf{\Psi }}_{i,j}\right)=m_{i,j}$ for every $1\le i,j<\infty$.

Remark 17.

In the original statement of Theorem 2 of [18], it is written that a probability space $(\mathcal{Y},\Gamma ,\mu )$ exists for a given $\infty \times \infty$ doubly stochastic matrix $\mathbf{M}$, namely, the σ-algebra Γ may depend on $\mathbf{M}$. However, the construction of Γ is independent of $\mathbf{M}$ (see Page 196 of [18]); and we can restate Theorem 2 of [18] as Lemma 4.

This is a probabilistic description of an $\infty \times \infty$ doubly stochastic matrix via a probability measure on the $\infty \times \infty$ permutation matrices. The existence of the probability measure $\mu$ is due to Kolmogorov’s extension theorem. We employ this $\sigma$-algebra $\Gamma$ on $\mathcal{Y}$ in the proof.

Thirdly, we shall show that under this measurable space $(\mathcal{Y},\Gamma )$, the collection $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y})$ has balanced conditional distributions defined in (126). In other words, for a given pair $(X,Y)\in \mathcal{R}(Q,L,\epsilon ,\mathcal{Y})$, it suffices to construct another pair $(U,V)$ of r.v.’s satisfying (126) and $(U,V)\in \mathcal{R}(Q,L,\epsilon ,\mathcal{Y})$. At first, construct its conditional distribution $P_{U|V}$ by

$$\begin{array}{c}\hfill P_{U|V}\left(x\right)=\mathbb{E}\left[P_{X|Y}^{\downarrow }\left({\psi }_V\left(x\right)\right)|V\right](\mathrm{a}.\mathrm{s}.)\end{array}$$

(135)

for each $x\in \mathcal{X}$, where $\mathbb{E}[Z\mid W]$ stands for the conditional expectation of a real-valued r.v. Z given the sub-$\sigma$-algebra $\sigma \left(W\right)$ generated by a r.v. W, and ${\varphi }_V$ is given as in (132). As ${\psi }_V\left(x\right)$ is $\sigma \left(V\right)$-measurable for each $x\in \mathcal{X}$, it is clear that

$$\begin{array}{c}\hfill P_{U|V}^{\downarrow }\left(x\right)=\mathbb{E}\left[P_{X|Y}^{\downarrow }\left(x\right)|V\right]=\mathbb{E}\left[P_{X|Y}^{\downarrow }\left({\psi }_V\left({\psi }_V^{-1}\left(x\right)\right)\right)|V\right]=P_{U|V}\left({\psi }_V^{-1}\left(x\right)\right)(\mathrm{a}.\mathrm{s}.)\end{array}$$

(136)

for every $x\in \mathcal{X}$. Thus, we readily see that (126) holds, and $(U,V)$ is connected uniform-dispersively. Thus, by (123) and the hypothesis that $P_X=Q$, we see that $P_{U|V}$ majorizes Q a.s. Therefore, it follows from the well-known characterization of the majorization relation via $\infty \times \infty$ doubly stochastic matrices (see Lemma 3.1 of [16] or Page 25 of [10]) that one can find an $\infty \times \infty$ doubly stochastic matrix $\mathbf{M}={\left\{m_{i,j}\right\}}_{i,j=1}^{\infty }$ satisfying

$$\begin{array}{c}\hfill Q\left(i\right)=\sum _{j=1}^{\infty }{m}_{i,j}{P}_{U|V}^{\downarrow }\left(j\right)(\mathrm{a}.\mathrm{s}.)\end{array}$$

(137)

for every $i\ge 1$. By Lemma 4, we can construct an induced probability measure $P_V$ so that $P_V\left({\mathsf{\Psi }}_{i,j}\right)=m_{i,j}$ for each $1\le i,j<\infty$. Now, the pair of $P_{U|V}$ and $P_V$ can define the probability law of $(U,V)$. To ensure that $(U,V)$ belongs to $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y})$, it remains to verity that $P_{\mathrm{e}}^{\left(L\right)}(U\mid V)\le \epsilon$ and $P_U=Q$.

As ${\psi }_{\mathsf{\Pi }}$ is a permutation defined in (132), we have

$$\begin{array}{cc}\hfill P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)& \stackrel{\left(\mathrm{a}\right)}{=}1-\mathbb{E}\left[\sum _{x=1}^L{P}_{X|Y}^{\downarrow }\left(x\right)\right]\hfill \\ \hfill & =1-\mathbb{E}\left[\sum _{x=1}^L\mathbb{E}[P_{X|Y}^{\downarrow }\left(x\right)\mid V]\right]\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{=}1-\mathbb{E}\left[\sum _{x=1}^L{P}_{U|V}^{\downarrow }\left(x\right)\right]\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{=}{P}_{\mathrm{e}}^{\left(L\right)}(U\mid V),\hfill \end{array}$$

(138)

where

• (a) and (c) follow from Proposition 2, and
• and (b) follows from (136).

Therefore, we see that $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le \epsilon$ is equivalent to $P_{\mathrm{e}}^{\left(L\right)}(U\mid V)\le \epsilon$. Furthermore, we observe that

$$\begin{array}{cc}\hfill Q\left(i\right)& \stackrel{\left(\mathrm{a}\right)}{=}\sum _{j=1}^{\infty }{m}_{i,j}{P}_{U|V}^{\downarrow }\left(j\right)(\mathrm{a}.\mathrm{s}.)\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{=}\sum _{j=1}^{\infty }\mathbb{E}\left[{\mathbf{1}}_{\{V\in {\mathsf{\Psi }}_{i,j}\}}\right]P_{U|V}^{\downarrow }\left(j\right)(\mathrm{a}.\mathrm{s}.)\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{=}\sum _{j=1}^{\infty }\mathbb{E}\left[{\mathbf{1}}_{\{V\in {\mathsf{\Psi }}_{i,j}\}}{P}_{U|V}^{\downarrow }\left(j\right)\right](\mathrm{a}.\mathrm{s}.)\hfill \\ \hfill & =\sum _{j=1}^{\infty }\mathbb{E}\left[\mathbb{E}\left[{\mathbf{1}}_{\{V\in {\mathsf{\Psi }}_{i,j}\}}{P}_{U|V}^{\downarrow }\left(j\right)|V\right]\right]\hfill \\ \hfill & \stackrel{\left(\mathrm{d}\right)}{=}\sum _{j=1}^{\infty }\mathbb{E}\left[\mathbb{E}\left[{\mathbf{1}}_{\{V\in {\mathsf{\Psi }}_{i,j}\}}{P}_{U|V}\left({\psi }_V^{-1}\left(j\right)\right)|V\right]\right]\hfill \\ \hfill & \stackrel{\left(\mathrm{e}\right)}{=}\sum _{j=1}^{\infty }\mathbb{E}\left[\mathbb{E}\left[{\mathbf{1}}_{\{V\in {\mathsf{\Psi }}_{i,j}\}}{P}_{U|V}\left(\sum _{k=1}^{\infty }{\mathbf{1}}_{\{V\in {\mathsf{\Psi }}_{k,j}\}}k\right)|V\right]\right]\hfill \\ \hfill & =\sum _{j=1}^{\infty }\mathbb{E}\left[{\mathbf{1}}_{\{V\in {\mathsf{\Psi }}_{i,j}\}}{P}_{U|V}\left(\sum _{k=1}^{\infty }{\mathbf{1}}_{\{V\in {\mathsf{\Psi }}_{k,j}\}}k\right)\right]\hfill \\ \hfill & \stackrel{\left(\mathrm{f}\right)}{=}\mathbb{E}\left[\sum _{j=1}^{\infty }{\mathbf{1}}_{\{V\in {\mathsf{\Psi }}_{i,j}\}}{P}_{U|V}\left(\sum _{k=1}^{\infty }{\mathbf{1}}_{\{V\in {\mathsf{\Psi }}_{k,j}\}}k\right)\right]\hfill \\ \hfill & \stackrel{\left(\mathrm{g}\right)}{=}\mathbb{E}\left[P_{U|V}\left(i\right)\right]\hfill \\ \hfill & =P_U\left(i\right)\hfill \end{array}$$

(139)

for every $i\ge 1$, where

• (a) follows from (137),
• (b) follows by the identity $m_{i,j}=\mathbb{P}\{V\in {\mathsf{\Psi }}_{i,j}\}$,
• (c) follows from the fact that $(X,Y)$ is connected uniform-dispersively,
• (d) follows from (136),
• (e) follows by the definition of ${\mathsf{\Psi }}_{i,j}$,
• (f) follows by the Fubini–Tonelli theorem, and
• (f) follows from the fact that the inverse of a permutation matrix is its transpose.

Therefore, we have $P_U=Q$, and the assertion of Lemma 3 is proved in the case where the cardinality of $\mathcal{Y}$ is the cardinality of the continuum.

Finally, even if the cardinality of $\mathcal{Y}$ is larger than the cardinality of continuum, the assertion of Lemma 3 can be immediately proved by considering the trace of the space $\mathcal{Y}$ on $\mathsf{\Psi }$ (cf. [74], p. 23). This completes the proof of Lemma 3. □

Finally, we show that the Fano-type distribution of type-1 defined in (30) is the infimum of a certain class of $\mathcal{X}$-marginals with respect to the majorization relation ≺.

Lemma 5.

Suppose that the system $(Q,L,\epsilon )$ satisfies the right-hand inequality in (29). For every $\mathcal{X}$-marginal R in which R majorizes Q and $P_{\mathrm{e}}^{\left(L\right)}\left(R\right)\le \epsilon$, it holds that R majorizes $P_{type-1}$ as well.

Proof of Lemma 5.

We first give an elementary fact of the weak majorization on the finite-dimensional real vectors.

Lemma 6.

Let $\mathit{p}={\left(p_i\right)}_{i=1}^n$ and $\mathit{q}={\left(q_i\right)}_{i=1}^n$ be n-dimensional real vectors satisfying $p_1\ge p_2\ge \cdots \ge p_n\ge 0$ and $q_1\ge q_2\ge \cdots \ge q_n\ge 0$, respectively. Consider an integer $1\le k\le n$ satisfying $q_k=q_i$ for every $i=k,k+1,\cdots ,n$. If

$$\begin{array}{cc}\hfill \sum _{i=1}^j{p}_i& \ge \sum _{i=1}^j{q}_i\mathrm{for}j=1,2,\cdots ,k-1,\hfill \end{array}$$

(140)

$$\begin{array}{cc}\hfill \sum _{i=1}^n{p}_i& \ge \sum _{i=1}^n{q}_i\hfill \end{array}$$

(141)

then it holds that

$$\begin{array}{c}\hfill \sum _{i=1}^j{p}_i\ge \sum _{i=1}^j{q}_i\mathrm{for}j=1,2,\cdots ,n.\end{array}$$

(142)

Proof of Lemma 6.

See Appendix E. □

Since $P_{type-1}=P_{type-1}^{\downarrow }$ (see Proposition 4), it suffices to prove that

$$\begin{array}{c}\hfill \sum _{x=1}^k{P}_{type-1}\left(x\right)\le \sum _{x=1}^k{R}^{\downarrow }\left(x\right)\end{array}$$

(143)

for every $k\ge 1$.

As $P_{type-1}\left(x\right)=Q^{\downarrow }\left(x\right)$ for each $1\le x<J$ (see Proposition 4), it follows by the majorization relation $Q\prec R$ that (143) holds for each $1\le k<J$. Moreover, as $P_{\mathrm{e}}^{\left(L\right)}\left(P_{type-1}\right)=\epsilon$ (see Proposition 4), it follows from (28) and the hypothesis $P_{\mathrm{e}}^{\left(L\right)}\left(R\right)\le \epsilon$ that

$$\begin{array}{c}\hfill \sum _{x=J}^L{P}_{type-1}\left(x\right)\le \sum _{x=J}^L{R}^{\downarrow }\left(x\right).\end{array}$$

(144)

In addition, as (143) holds for each $1\le k<J$ and $P_{type-1}\left(x\right)=\mathcal{V}\left(J\right)$ for each $J\le x\le L$ (see Proposition 4), it follows from Lemma 6 and (144) that (143) also holds for each $1\le k\le L$.

Now, suppose that $K_1=\infty$. Then, it follows that

$$\begin{array}{c}\hfill P_{type-1}\left(x\right)=\mathcal{W}(\infty )=0\end{array}$$

(145)

for each $x\ge L+1$ (see Proposition 4). Thus, Inequality (143) holds for every $k\ge 1$; therefore, we have that R majorizes $P_{type-1}$, provided that $K_1=\infty$.

Finally, suppose that $K_1<\infty$. Since $P_{type-1}\left(x\right)=Q^{\downarrow }\left(x\right)$ for each $x\ge K_1+1$ (see Proposition 4), it follows by the majorization relation $Q\prec R$ that (143) holds for every $k\ge K_1$. Moreover, since (143) holds for every $1\le k\le L$ and every $k\ge K_1$, we observe that

$$\begin{array}{c}\hfill \sum _{x=L+1}^{K_1}{P}_{type-1}\left(x\right)\le \sum _{x=L+1}^{K_1}{R}^{\downarrow }\left(x\right).\end{array}$$

(146)

Finally, as (143) holds for $1\le k\le L$ and $P_{type-1}\left(x\right)=\mathcal{W}\left(K_1\right)$ for $L<x\le K_1$ (see Proposition 4), it follows by Lemma 6 and (146) that (143) holds for every $1\le k\le K_1$. Therefore, Inequality (143) holds for every $k\ge 1$, completing the proof of Lemma 5. □

Using the above lemmas, we can prove Theorem 1 as follows.

Proof of Theorem 1.

Let $\epsilon >0$. For the sake of brevity, we write

$$\begin{array}{cc}\hfill \mathcal{R}& =\mathcal{R}(Q,L,\epsilon ,\mathcal{Y})\hfill \end{array}$$

(147)

in the proof. Let $\mathsf{{\rm Y}}$ be a $\sigma$-algebra on $\mathcal{Y}$, $\mathsf{\Psi }$ an alphabet in which its cardinality is the cardinality of the continuum, and $\Gamma$ a $\sigma$-algebra on $\mathsf{\Psi }$ so that $\mathcal{R}(Q,L,\epsilon ,\mathsf{\Psi })$ has balanced conditional distributions (see Lemma 3). Now, we define the collection

$$\begin{array}{c}\hfill \overline{\mathcal{R}}:=\mathcal{R}(Q,L,\epsilon ,\mathcal{Y}\cup \mathsf{\Psi }),\end{array}$$

(148)

where the $\sigma$-algebra on $\mathcal{Y}\cup \mathsf{\Psi }$ is given by the smallest $\sigma$-algebra $\mathsf{{\rm Y}}\vee \Gamma$ containing $\mathsf{{\rm Y}}$ and $\Gamma$. It is clear that $\mathcal{R}\subset \overline{\mathcal{R}}$, and $\overline{\mathcal{R}}$ has balanced conditional distributions as well (see the last paragraph in the proof of Lemma 3). Then, we have

$$\begin{array}{cc}\hfill {\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})& \stackrel{\left(\mathrm{a}\right)}{=}\underset{(X,Y)\in \mathcal{R}}{\sup }{\mathsf{H}}_{\varphi }(X\mid Y)\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{\le }\underset{(X,Y)\in \overline{\mathcal{R}}}{\sup }{\mathsf{H}}_{\varphi }(X\mid Y)\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{=}\underset{\begin{array}{c}(X,Y)\in \overline{\mathcal{R}}:\\ (X,Y)\mathrm{is}\mathrm{connected}\mathrm{uniform}-\mathrm{dispersively}\end{array}}{\sup }{\mathsf{H}}_{\varphi }(X\mid Y)\hfill \\ \hfill & \stackrel{\left(\mathrm{d}\right)}{=}\underset{\begin{array}{c}(X,Y)\in \overline{\mathcal{R}}:\\ P_{\mathrm{e}}^{\left(L\right)}\left(P_{X|Y}\right)\le \epsilon \mathrm{a}.\mathrm{s}.,\\ (X,Y)\mathrm{is}\mathrm{connected}\mathrm{uniform}-\mathrm{dispersively}\end{array}}{\sup }\varphi \left(P_{X|Y}\right)(\mathrm{a}.\mathrm{s}.)\hfill \\ \hfill & \stackrel{\left(\mathrm{e}\right)}{\le }\underset{\begin{array}{c}R\in \mathcal{P}\left(\mathcal{X}\right):\\ Q\prec R\mathrm{and}{P}_{\mathrm{e}}^{\left(L\right)}\left(R\right)\le \epsilon \end{array}}{\sup }\varphi \left(R\right)(\mathrm{a}.\mathrm{s}.)\hfill \\ \hfill & \stackrel{\left(\mathrm{f}\right)}{\le }\varphi \left(P_{type-1}\right),\hfill \end{array}$$

(149)

where

• (a) follows by the definition of $\mathcal{R}$ stated in (129),
• (b) follows by the inclusion $\mathcal{R}\subset \overline{\mathcal{R}}$,
• (c) follows from Lemma 2 and the fact that $\overline{\mathcal{R}}$ has balanced conditional distributions,
• (d) follows by the symmetry of both $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ and $P_{\mathrm{e}}^{\left(L\right)}:\mathcal{P}\left(\mathcal{X}\right)\to [0,1]$,
• (e) follows from Lemma 1, and
• (f) follows from Proposition 1 and Lemma 5.

Inequalities (149) are indeed the Fano-type inequality stated in (35) of Theorem 1. If $\epsilon =P_{\mathrm{e}}^{\left(L\right)}\left(Q\right)$, then it can be verified by the definition of $P_{type-1}$ stated in (30) that $P_{type-1}=Q^{\downarrow }$ (see also Proposition 4). In such a case, the supremum in (7) can be achieved by a pair $(X,Y)$ satisfying $P_X=Q$ and $X⫫Y$.

Finally, we shall construct a jointly distributed pair $(X,Y)$ satisfying

$$\begin{array}{cc}\hfill {\mathsf{H}}_{\varphi }(X\mid Y)& =\varphi \left(P_{type-1}\right),\hfill \end{array}$$

(150)

$$\begin{array}{cc}\hfill P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)& =\epsilon ,\hfill \end{array}$$

(151)

$$\begin{array}{cc}\hfill P_X\left(x\right)& =Q^{\downarrow }\left(x\right)(\mathrm{for}x\in \mathcal{X}).\hfill \end{array}$$

(152)

For the sake of brevity, suppose that $\mathcal{Y}$ is the index set of the set of permutation matrices on $\{J,J+1,\cdots ,K_1\}$. Namely, denote by ${\mathsf{\Pi }}^{\left(y\right)}={\left\{{\pi }_{i,j}^{\left(y\right)}\right\}}_{i,j=J}^{K_1}$ a permutation matrix for each index $y\in \mathcal{Y}$. By the definition of $P_{type-1}$ stated in (30) (see also Proposition 4), we observe that

$$\begin{array}{cc}\hfill \sum _{x=J}^k{Q}^{\downarrow }\left(x\right)& \le \sum _{x=J}^k{P}_{type-1}\left(x\right)\mathrm{for}J\le k\le K_1,\hfill \end{array}$$

(153)

and

$$\begin{array}{cc}\hfill \sum _{x=J}^{K_1}{Q}^{\downarrow }\left(x\right)& =\sum _{x=J}^{K_1}{P}_{type-1}\left(x\right).\hfill \end{array}$$

(154)

Noting that $K_1<\infty$ if $\epsilon >0$ (see (34)), Equations (153) and (154) are indeed a majorization relation between two finite-dimensional real vectors; and thus, it follows from the Hardy–Littlewood–Pólya theorem (see Theorem 8 of [15] or Theorem 2.B.2 [10]) that there exists a $(K_1-J+1)\times (K_1-J+1)$ doubly stochastic matrix $\mathbf{M}={\left\{m_{i,j}\right\}}_{i,j=J}^{K_1}$ satisfying

$$\begin{array}{c}\hfill Q^{\downarrow }\left(i\right)=\sum _{j=J}^{K_1}{m}_{i,j}{P}_{type-1}\left(j\right)\end{array}$$

(155)

for each $J\le i\le K_1$. Moreover, it follows from the finite dimensional version of Birkhoff’s theorem [19] (see also Theorems 2.A.2 and 2.C.2 of [10]) that for such a doubly stochastic matrix $\mathbf{M}={\left\{m_{i,j}\right\}}_{i,j=J}^{K_1}$, there exists a probability vector $\lambda ={\left({\lambda }_y\right)}_{y\in \mathcal{Y}}$ satisfying

$$\begin{array}{c}\hfill m_{i,j}=\sum _{y\in \mathcal{Y}}{\lambda }_y{\pi }_{i,j}^{\left(y\right)}\end{array}$$

(156)

for every $J\le i,j\le K_1$, where a nonnegative vector is called a probability vector if the sum of the elements is unity. Using them, we construct a pair $(X,Y)$ via the following distributions,

$$\begin{array}{cc}\hfill P_{X|Y=y}\left(x\right)& =\left\{\begin{array}{cc}{P}_{type-1}\left(x\right)\hfill & \mathrm{if}1\le x<J\mathrm{or}{K}_1<x<\infty ,\hfill \\ P_{type-1}\left({\tilde{\psi }}_y\left(x\right)\right)\hfill & \mathrm{if}J\le x\le K_1,\hfill \end{array}\right.\hfill \end{array}$$

(157)

$$\begin{array}{cc}\hfill P_Y\left(y\right)& ={\lambda }_y,\hfill \end{array}$$

(158)

where the permutation ${\tilde{\psi }}_y$ on $\{J,J+1,\cdots ,K_1\}$ is defined by

$$\begin{array}{c}\hfill {\tilde{\psi }}_y\left(i\right):=\sum _{j=J}^{K_1}{\pi }_{i,j}^{\left(y\right)}j\end{array}$$

(159)

for each $y\in \mathcal{Y}$. Then, it follows from (155) and (156) that (152) holds. Moreover, it is easy to see that $P_{X|Y=y}^{\downarrow }=P_{type-1}$ for every $y\in \mathcal{Y}$. Thus, we observe that (150) and (151) hold as well. This implies together with (149) that the constructed pair $(X,Y)$ achieves the supremum in (7), completes the proof of Theorem 1. □

6.2. Proof of Theorem 2

Even if $\epsilon =0$, the inequalities in (149) hold as well; that is, the Fano-type inequality stated in (43) of Theorem 2 holds. In this proof, we shall verify the equality conditions of (43).

If $supp\left(Q\right)$ is finite, then it follows by the definition of $K_1$ stated in (34) that $K_1<\infty$. Thus, the same construction of a jointly distributed pair $(X,Y)$ as the last paragraph of Section 6.1 proves that (43) holds with equality if $supp\left(Q\right)$ is finite.

Consider the case where $supp\left(Q\right)$ is infinite and $J=L$. Since $\epsilon =0$, we readily see that $K_1=\infty$, $\mathcal{V}\left(J\right)>0$, and $\mathcal{W}\left(K_1\right)=0$. Suppose that

$$\begin{array}{c}\hfill \mathcal{Y}=\{L,L+1,L+2,\cdots \}.\end{array}$$

(160)

We then construct a pair $(X,Y)$ via the following distributions,

$$\begin{array}{cc}\hfill P_{X|Y=y}\left(x\right)& =\left\{\begin{array}{cc}{Q}^{\downarrow }\left(x\right)\hfill & \mathrm{if}1\le x<L,\hfill \\ \mathcal{V}\left(J\right)\hfill & \mathrm{if}L\le x<\infty \mathrm{and}x=y,\hfill \\ 0\hfill & \mathrm{if}L\le x<\infty \mathrm{and}x\ne y,\hfill \end{array}\right.\hfill \end{array}$$

(161)

$$\begin{array}{cc}\hfill P_Y\left(y\right)& =\frac{Q\left(y\right)}{\mathcal{V}\left(J\right)}.\hfill \end{array}$$

(162)

We readily see that $P_{X|Y=y}^{\downarrow }=P_{type-1}$ for every $y\in \mathcal{Y}$; therefore, we have that (150)–(152) hold. This implies that the constructed pair $(X,Y)$ achieves the supremum in (7).

Finally, suppose that the cardinality of $\mathcal{Y}$ is at least the cardinality of the continuum. Assume without loss of generality that $\mathcal{Y}$ is the set of $\infty \times \infty$ permutation matrices. Consider the measurable space $(\mathcal{Y},\Gamma )$ given in the infinite-dimensional version of Birkhoff’s theorem (see Lemma 4). In addition, consider a jointly distributed pair $(X,Y)$ satisfying $P_{X|Y}^{\downarrow }=P_{type-1}$ a.s. Then, it is easy to see that (150) and (151) hold for any induced probability measure $P_Y$ on $\mathcal{Y}$. Similar to the construction of the probability measure $P_V$ on $\mathcal{Y}$ below (137), we can find an induced probability measure $P_Y$ satisfying (152). Therefore, it follows from (43) that this pair $(X,Y)$ achieves the supremum in (7). This completes the proof of Theorem 2.

6.3. Proof of Theorem 3

To prove Theorem 3, we need some more preliminary results. Throughout this subsection, assume that the alphabet $\mathcal{Y}$ is finite and nonempty. In this case, given a pair $(X,Y)$, one can define

$$\begin{array}{c}\hfill P_{X|Y=y}\left(x\right)=\mathbb{P}\{X=x\mid Y=y\},\end{array}$$

(163)

provided that $P_Y\left(y\right)>0$.

For a subset $\mathcal{Z}\subset \mathcal{X}$, define

$$\begin{array}{c}\hfill P_{\mathrm{e}}^{\left(L\right)}(X\mid Y\parallel \mathcal{Z}):=\underset{f:\mathcal{Y}\to \left(\genfrac{}{}{0pt}{}{\mathcal{Z}}{L}\right)}{min}\mathbb{P}\{X\notin f\left(Y\right)\}.\end{array}$$

(164)

Note that the difference between $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)$ and $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y\parallel \mathcal{Z})$ is the restriction of the decoding range $\mathcal{Z}\subset \mathcal{X}$, and the inequality $P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le P_{\mathrm{e}}^{\left(L\right)}(X\mid Y\parallel \mathcal{Z})$ is trivial from these definitions stated in (21) and (164), respectively. The following propositions are easy consequences of the proofs of Propositions 2 and 3, and so we omit those proofs in this paper.

Proposition 7.

It holds that

$$\begin{array}{c}\hfill P_{\mathrm{e}}^{\left(L\right)}(X\mid Y\parallel \mathcal{Z})=1-\mathbb{E}\left[\underset{\mathcal{D}\in \left(\genfrac{}{}{0pt}{}{\mathcal{Z}}{L}\right)}{min}\sum _{x\in \mathcal{D}}{P}_{X|Y}\left(x\right)\right].\end{array}$$

(165)

Proposition 8.

Let $\beta :\{1,\cdots ,|\mathcal{Z}\left|\right\}\to \mathcal{Z}$ be a bijection satisfying $P_X\left(\beta \left(i\right)\right)\ge P_X\left(\beta \left(j\right)\right)$ if $i<j$. It holds that

$$\begin{array}{c}\hfill 1-\sum _{x\in \mathcal{Z}}{P}_X\left(x\right)\le P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)\le 1-\sum _{x=1}^L{P}_X\left(\beta \left(x\right)\right).\end{array}$$

(166)

For a finite subset $\mathcal{Z}\subset \mathcal{X}$, denote by $\mathsf{\Psi }\left(\mathcal{Z}\right)$ the set of $\left|\mathcal{Z}\right|\times \left|\mathcal{Z}\right|$ permutation matrices in which both rows and columns are indexed by the elements in $\mathcal{Z}$. The main idea of proving Theorem 3 is the following lemma.

Lemma 7.

For any $\mathcal{X}\times \mathcal{Y}$-valued r.v. $(X,Y)$, there exist a subset $\mathcal{Z}\subset \mathcal{X}$ and an $\mathcal{X}\times \mathsf{\Psi }\left(\mathcal{Z}\right)$-valued r.v. $(U,W)$ such that

$$\begin{array}{cc}\hfill \left|\mathcal{Z}\right|& =L·\left|\mathcal{Y}\right|,\hfill \end{array}$$

(167)

$$\begin{array}{cc}\hfill P_U\left(x\right)& =P_X\left(x\right)\mathrm{for}x\in \mathcal{X},\hfill \end{array}$$

(168)

$$\begin{array}{cc}\hfill P_{\mathrm{e}}^{\left(L\right)}(U\mid W)& \le P_{\mathrm{e}}^{\left(L\right)}(U\mid W\parallel \mathcal{Z})=P_{\mathrm{e}}^{\left(L\right)}(X\mid Y),\hfill \end{array}$$

(169)

$$\begin{array}{cc}\hfill {\mathsf{H}}_{\varphi }(U\mid W)& \ge {\mathsf{H}}_{\varphi }(X\mid Y),\hfill \end{array}$$

(170)

$$\begin{array}{cc}\hfill P_{U|W=w}\left(x\right)& =P_X\left(x\right)\mathrm{for}x\in \mathcal{X}\setminus \mathcal{Z}\mathrm{and}w\in \mathsf{\Psi }\left(\mathcal{Z}\right).\hfill \end{array}$$

(171)

Proof of Lemma 7.

Suppose without loss of generality that

$$\begin{array}{c}\hfill \mathcal{Y}=\{0,1,\cdots ,N-1\}\end{array}$$

(172)

for some positive integer N. By the definition of cardinality, one can find a subset $\mathcal{Z}\subset \mathcal{X}$ satisfying (i) $\left|\mathcal{Z}\right|=LN$, and (ii) for each $x\in \{1,2,\cdots ,L\}$ and $y\in \mathcal{Y}$, there exists $z\in \mathcal{Z}$ satisfying

$$\begin{array}{c}\hfill P_{X|Y=y}\left(z\right)=P_{X|Y=y}^{\downarrow }\left(x\right).\end{array}$$

(173)

For each $\mathsf{\Pi }={\left\{{\pi }_{i,j}\right\}}_{i,j\in \mathcal{Z}}\in \mathsf{\Psi }\left(\mathcal{Z}\right)$, define the permutation ${\phi }_{\mathsf{\Pi }}:\mathcal{Z}\to \mathcal{Z}$ by

$$\begin{array}{c}\hfill {\phi }_{\mathsf{\Pi }}\left(z\right):=\sum _{w\in \mathcal{Z}}{\pi }_{z,w}w,\end{array}$$

(174)

as in (132) and (159). It is clear that for each $y\in \mathcal{Y}$, there exists at least one $\mathsf{\Pi }\in \mathsf{\Psi }\left(\mathcal{Z}\right)$ such that

$$\begin{array}{c}\hfill P_{X|Y=y}\left({\phi }_{\mathsf{\Pi }}\left(x_1\right)\right)\ge P_{X|Y=y}\left({\phi }_{\mathsf{\Pi }}\left(x_2\right)\right)\end{array}$$

(175)

for every $x_1,x_2\in \mathcal{Z}$ satisfying $x_1\le x_2$, which implies that the permutation ${\phi }_{\mathsf{\Pi }}$ plays the role of a decreasing rearrangement of $P_{X|Y=y}$ on $\mathcal{Z}$. To denote such a correspondence between $\mathcal{Y}$ and $\mathsf{\Psi }\left(\mathcal{Z}\right)$, one can choose an injection $\iota :\mathcal{Y}\to \mathsf{\Psi }\left(\mathcal{Z}\right)$ appropriately. In other words, one can find an injection $\iota$ so that

$$\begin{array}{c}\hfill P_{X|Y=y}\left({\phi }_{\iota \left(y\right)}\left(x_1\right)\right)\ge P_{X|Y=y}\left({\phi }_{\iota \left(y\right)}\left(x_2\right)\right)\end{array}$$

(176)

for every $y\in \mathcal{Y}$ and $x_1,x_2\in \mathcal{Z}$ satisfying $x_1\le x_2$. We now construct an $\mathcal{X}\times \mathcal{Y}\times \mathsf{\Psi }\left(\mathcal{Z}\right)$-valued r.v. $(U,V,W)$ as follows: The conditional distribution $P_{U|V,W}$ is given by

$$\begin{array}{cc}\hfill P_{U|V=v,W=w}\left(u\right)& =\left\{\begin{array}{cc}{P}_{X|Y=v}({\phi }_{\iota \left(v\right)}\circ {\phi }_w\left(u\right))\hfill & \mathrm{if}u\in \mathcal{Z},\hfill \\ P_{X|Y=v}\left(u\right)\hfill & \mathrm{if}u\in \mathcal{X}\setminus \mathcal{Z},\hfill \end{array}\right.\hfill \end{array}$$

(177)

where ${\sigma }_1\circ {\sigma }_2$ stands for the composition of two bijections ${\sigma }_1$ and ${\sigma }_2$. The induced probability distribution $P_V$ of V is given by $P_V=P_Y$. Suppose that the independence $V⫫W$ holds. As

$$\begin{array}{c}\hfill P_{U,V,W}=P_{U|V,W}{P}_V{P}_W,\end{array}$$

(178)

it remains to determine the induced probability distribution $P_W$ of W, and we defer to determine it until the last paragraph of this proof. A direct calculation shows

$$\begin{array}{cc}\hfill P_{U|W=w}\left(u\right)& =\sum _{v\in \mathcal{Y}}{P}_{V|W=w}\left(v\right)P_{U|V=v,W=w}\left(u\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{a}\right)}{=}\sum _{v\in \mathcal{Y}}{P}_Y\left(v\right)P_{U|V=v,W=w}\left(u\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{=}\left\{\begin{array}{cc}\omega (u,w)\hfill & \mathrm{if}u\in \mathcal{Z},\hfill \\ P_X\left(u\right)\hfill & \mathrm{if}u\in \mathcal{X}\setminus \mathcal{Z},\hfill \end{array}\right.\hfill \end{array}$$

(179)

where

• (a) follows by the independence $V⫫W$ and $P_V=P_Y$, and
• (b) follows by (177) and defining $\omega (u,w)$ so that

  $$\begin{array}{cc}\hfill \omega (u,w)& :=\sum _{v\in \mathcal{Y}}{P}_Y\left(v\right)P_{X|Y=v}({\phi }_{\iota \left(v\right)}\circ {\phi }_w\left(u\right))\hfill \end{array}$$

  (180)

  for each $x\in \mathcal{Z}$ and $w\in \mathsf{\Psi }\left(\mathcal{Z}\right)$.

Now, we readily see from (179) that (171) holds for any induced probability distribution $P_W$ of W. Therefore, to complete the proof, it suffices to show that $(U,W)$ satisfies (169) and (170) with an arbitrary choice of $P_W$, and $(U,W)$ satisfies (168) with an appropriate choice of $P_W$.

Firstly, we shall prove (169). For each $w\in \mathsf{\Psi }\left(\mathcal{Z}\right)$, denote by $\mathcal{D}\left(w\right)\in \left(\genfrac{}{}{0pt}{}{\mathcal{Z}}{L}\right)$ the set satisfying

$$\begin{array}{c}\hfill {\phi }_w\left(k\right)<{\phi }_w\left(x\right)\end{array}$$

(181)

for every $k\in \mathcal{D}\left(w\right)$ and $x\in \mathcal{Z}\setminus \mathcal{D}\left(w\right)$, i.e., it stands for the set of first L elements in $\mathcal{Z}$ under the permutation rule $w\in \mathsf{\Psi }\left(\mathcal{Z}\right)$. Then, we have

$$\begin{array}{cc}\hfill P_{\mathrm{e}}^{\left(L\right)}(U\mid W)& \stackrel{\left(\mathrm{a}\right)}{\le }{P}_{\mathrm{e}}^{\left(L\right)}(U\mid W\parallel \mathcal{Z})\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{=}1-\sum _{w\in \mathsf{\Psi }\left(\mathcal{Z}\right)}{P}_W\left(w\right)\underset{\mathcal{D}\in \left(\genfrac{}{}{0pt}{}{\mathcal{Z}}{L}\right)}{min}\sum _{u\in \mathcal{D}}{P}_{U|W=w}\left(u\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{=}1-\sum _{w\in \mathsf{\Psi }\left(\mathcal{Z}\right)}{P}_W\left(w\right)\underset{\mathcal{D}\in \left(\genfrac{}{}{0pt}{}{\mathcal{Z}}{L}\right)}{min}\sum _{u\in \mathcal{D}}\omega (u,w)\hfill \\ \hfill & \stackrel{\left(\mathrm{d}\right)}{=}1-\sum _{w\in \mathsf{\Psi }\left(\mathcal{Z}\right)}{P}_W\left(w\right)\underset{\mathcal{D}\in \left(\genfrac{}{}{0pt}{}{\mathcal{Z}}{L}\right)}{min}\sum _{u\in \mathcal{D}}\sum _{v\in \mathcal{Y}}{P}_Y\left(v\right)P_{X|Y=v}({\phi }_{\iota \left(v\right)}\circ {\phi }_w\left(u\right))\hfill \\ \hfill & \stackrel{\left(\mathrm{e}\right)}{=}1-\sum _{w\in \mathsf{\Psi }\left(\mathcal{Z}\right)}{P}_W\left(w\right)\sum _{u\in \mathcal{D}\left(w\right)}\sum _{v\in \mathcal{Y}}{P}_Y\left(v\right)P_{X|Y=v}({\phi }_{\iota \left(v\right)}\circ {\phi }_w\left(u\right))\hfill \\ \hfill & \stackrel{\left(\mathrm{f}\right)}{=}1-\sum _{w\in \mathsf{\Psi }\left(\mathcal{Z}\right)}{P}_W\left(w\right)\sum _{u=1}^L\sum _{v\in \mathcal{Y}}{P}_Y\left(v\right)P_{X|Y=v}^{\downarrow }\left(u\right)\hfill \\ \hfill & =1-\sum _{y\in \mathcal{Y}}{P}_Y\left(y\right)\sum _{x=1}^L{P}_{X|Y=y}^{\downarrow }\left(x\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{g}\right)}{=}{P}_{\mathrm{e}}^{\left(L\right)}(X\mid Y),\hfill \end{array}$$

(182)

where

• (a) is an obvious inequality (see the definitions stated in (21) and (164)),
• (b) follows from Proposition 7,
• (c) follows from (179),
• (d) follows from the definition of $\omega (u,w)$ stated in (180),
• (e) follows from (176) and (181),
• (f) follows from (173), (176), and (181), and
• (g) follows from Proposition 2.

Therefore, we obtain (169).

Secondly, we shall prove (170). We get

$$\begin{array}{cc}\hfill {\mathsf{H}}_{\varphi }(X\mid Y)& =\sum _{y\in \mathcal{Y}}{P}_Y\left(y\right)\varphi \left(P_{X|Y=y}\right)\hfill \\ \hfill & =\sum _{w\in \mathsf{\Psi }\left(\mathcal{Z}\right)}{P}_W\left(w\right)\sum _{y\in \mathcal{Y}}{P}_Y\left(y\right)\varphi \left(P_{X|Y=y}\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{a}\right)}{=}\sum _{w\in \mathsf{\Psi }\left(\mathcal{Z}\right)}{P}_W\left(w\right)\sum _{y\in \mathcal{Y}}{P}_Y\left(y\right)\varphi \left(P_{U|V=y,W=w}\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{=}\sum _{w\in \mathsf{\Psi }\left(\mathcal{Z}\right)}{P}_W\left(w\right)\sum _{v\in \mathcal{Y}}{P}_V\left(v\right)\varphi \left(P_{U|V=v,W=w}\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{\le }\sum _{w\in \mathsf{\Psi }\left(\mathcal{Z}\right)}{P}_W\left(w\right)\varphi \left(\sum _{v\in \mathcal{Y}}{P}_V\left(v\right)P_{U|V=v,W=w}\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{d}\right)}{=}\sum _{w\in \mathsf{\Psi }\left(\mathcal{Z}\right)}{P}_W\left(w\right)\varphi \left(P_{U|W=w}\right)\hfill \\ \hfill & ={\mathsf{H}}_{\varphi }(U\mid W),\hfill \end{array}$$

(183)

where

• (a) follows by the symmetry of $\varphi$ and (177),
• (b) follows by $P_V=P_Y$,
• (c) follows by Jensen’s inequality, and
• (d) follows by the independence $U⫫W$.

Therefore, we obtain (170).

Finally, we shall prove that there exists an induced probability distribution $P_W$ satisfying (168). If we denote by $I\in \mathsf{\Psi }\left(\mathcal{Z}\right)$ the identity matrix, then it follows from (180) that

$$\begin{array}{c}\hfill P_{U|W=I}\left(u\right)=P_{U|W=w}\left({\phi }_w^{-1}\left(u\right)\right)\end{array}$$

(184)

for every $(u,w)\in \mathcal{Z}\times \mathsf{\Psi }\left(\mathcal{Z}\right)$. It follows from (179) that

$$\begin{array}{c}\hfill \sum _{x\in \mathcal{Z}}{P}_X\left(x\right)=\sum _{u\in \mathcal{Z}}{P}_{U|W=I}\left(u\right).\end{array}$$

(185)

Now, denote by ${\beta }_1:\{1,2,\cdots ,LN\}\to \mathcal{Z}$ and ${\beta }_2:\{1,2,\cdots ,LN\}\to \mathcal{Z}$ two bijections satisfying $P_X\left({\beta }_1\left(i\right)\right)\ge P_X\left({\beta }_1\left(j\right)\right)$ and ${\beta }_2\left(i\right)<{\beta }_2\left(j\right)$, respectively, provided that $i<j$. That is, the bijection ${\beta }_1$ and ${\beta }_2$ play roles of decreasing rearrangements of $P_X$ and $P_{U|W=I}$, respectively, on $\mathcal{Z}$. Using those bijections, one can rewrite (185) as

$$\begin{array}{c}\hfill \sum _{i=1}^{LN}{P}_X\left({\beta }_1\left(i\right)\right)=\sum _{i=1}^{LN}{P}_{U|W=I}\left({\beta }_2\left(i\right)\right).\end{array}$$

(186)

In the same way as (123), it can be verified from (180) by induction that

$$\begin{array}{c}\hfill \sum _{i=1}^k{P}_X\left({\beta }_1\left(i\right)\right)\le \sum _{i=1}^k{P}_{U|W=I}\left({\beta }_2\left(i\right)\right)\end{array}$$

(187)

for each $k=1,2,\cdots ,LN$. Equations (186) and (187) are indeed a majorization relation between two finite-dimensional real vectors, because ${\beta }_1$ plays a role of a decreasing rearrangement of $P_X$ on $\mathcal{Z}$. Combining (184) and this majorization relation, it follows from the Hardy–Littlewood–Pólya theorem derived in Theorem 8 of [15] (see also Theorem 2.B.2 of [10]) and the finite-dimensional version of Birkhoff’s theorem [19] (see also Theorem 2.A.2 of [10]) that there exists an induced probability distribution $P_W$ satisfying $P_U=P_X$, i.e., Equation (168) holds, as in (153)–(158). This completes the proof of Lemma 7. □

Remark 18.

Lemma 7 can restrict the feasible region of the supremum in (7) from a countably infinite alphabet $\mathcal{X}$ to a finite alphabet $\mathcal{Z}$ in the sense of (171). Specifically, if $\mathcal{Y}$ is finite, it suffices to vary at most $\left|\mathcal{Z}\right|=L·\left|\mathcal{Y}\right|$ probability masses ${\left\{P_{X|Y=y}\left(x\right)\right\}}_{x\in \mathcal{Z}}$ for each $y\in \mathcal{Y}$. Lemma 7 is useful not only to prove Theorem 3 but also to prove Proposition 9 of Section 8.1 (see Appendix D for the proof).

As with (129), for a subset $\mathcal{Z}\subset \mathcal{X}$, we define

$$\begin{array}{c}\hfill \mathcal{R}(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z}):=\left\{(X,Y)|\begin{array}{c}X\mathrm{is}\mathcal{X}-\mathrm{valued},\hfill \\ Y\mathrm{is}\mathcal{Y}-\mathrm{valued},\hfill \\ P_{\mathrm{e}}^{\left(L\right)}(X\mid Y\parallel \mathcal{Z})\le \epsilon ,\hfill \\ P_X=Q,\hfill \\ P_{X|Y=y}\left(x\right)=Q\left(x\right)\forall (x,y)\in (\mathcal{X}\setminus \mathcal{Z})\times \mathcal{Y}\hfill \end{array}\right\},\end{array}$$

(188)

provided that $\mathcal{Y}$ is finite. It is clear that (188) coincides with (129) if $\mathcal{Z}=\mathcal{X}$, i.e., it holds that

$$\begin{array}{c}\hfill \mathcal{R}(Q,L,\epsilon ,\mathcal{Y},\mathcal{X})=\mathcal{R}(Q,L,\epsilon ,\mathcal{Y}).\end{array}$$

(189)

Note from Lemma 7 that for each system $(Q,L,\epsilon ,\mathcal{Y})$ satisfying (29), there exists a subset $\mathcal{Z}\subset \mathcal{X}$ such that $\left|\mathcal{Z}\right|=L·\left|\mathcal{Y}\right|$ and $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})$ is nonempty, provided that $\mathcal{Y}$ is finite.

Another important idea of proving Theorem 3 is to apply Lemma 2 for this collection of r.v.’s. The correction $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})$ does not, however, have balanced conditional distributions of (126) in general, as with (129). Fortunately, similar to Lemma 3, the following lemma can avoid this issue by blowing-up the collection $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})$ via the finite-dimensional version of Birkhoff’s theorem [19].

Lemma 8.

Suppose that $\mathcal{Z}\subset \mathcal{X}$ is finite and $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})$ is nonempty. If $\left|\mathcal{Z}\right|\le \left|\mathcal{Y}\right|!<\infty$, then the collection $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})$ has balanced conditional distributions.

Proof of Lemma 8.

Lemma 8 can be proven in a similar fashion to the proof of Lemma 3. As this proof is slightly long as with Lemma 3, we only give a sketch of the proof as follows.

As $\left|\mathsf{\Psi }\right(\mathcal{Z}\left)\right|=\left|\mathcal{Z}\right|!$, we may assume without loss of generality that $\mathcal{Y}=\mathsf{\Psi }\left(\mathcal{Z}\right)$. For the sake of brevity, we write

$$\begin{array}{c}\hfill \tilde{\mathcal{R}}=\mathcal{R}(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})\end{array}$$

(190)

in this proof. For a pair $(X,Y)\in \tilde{\mathcal{R}}$, construct another $\mathcal{X}\times \mathcal{Y}$-valued r.v. $(U,V)$, as in (135), so that $P_{U|V=y}\left(x\right)=Q\left(x\right)$ for every $(x,y)\in (\mathcal{X}\setminus \mathcal{Z})\times \mathcal{Y}$. By such a construction of (135), the condition stated in (126) is obviously satisfied. In the same way as (138), we can verify that

$$\begin{array}{c}\hfill P_{\mathrm{e}}^{\left(L\right)}(U\mid V\parallel \mathcal{Z})=P_{\mathrm{e}}^{\left(L\right)}(X\mid Y\parallel \mathcal{Z}).\end{array}$$

(191)

Moreover, employing the finite-dimensional version of Birkhoff’s theorem [19] (also known as the Birkhoff–von Neumann decomposition) instead of Lemma 4, we can also find an induced probability distribution $P_V$ of V so that $P_U=Q$ in the same way as (139). Therefore, for any $(X,Y)\in \tilde{\mathcal{R}}$, one can find $(U,V)\in \tilde{\mathcal{R}}$ satisfying (126). This completes the proof of Lemma 8. □

Let $\mathcal{Z}\subset \mathcal{X}$ be a subset. Consider a bijection $\beta :\{1,2,\cdots ,|\mathcal{Z}\left|\right\}\to \mathcal{Z}$ satisfying $Q\left(\beta \right(i\left)\right)\ge Q\left(\beta \right(j\left)\right)$ whenever $i<j$, i.e., it plays a role of a decreasing rearrangement of Q on $\mathcal{Z}$. Thereforeforth, suppose that $(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})$ satisfies

$$\begin{array}{c}\hfill 1-\sum _{x\in \mathcal{Z}}Q\left(x\right)\le \epsilon \le 1-\sum _{x=1}^{L}Q\left(\beta \left(x\right)\right).\end{array}$$

(192)

Define the extremal distribution of type-3 by the following $\mathcal{X}$-marginal:

$$\begin{array}{c}\hfill P_{type-3}\left(x\right)=P_{type-3}^{(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})}\left(x\right):=\left\{\begin{array}{cc}{\mathcal{V}}_3\left(J_3\right)\hfill & \mathrm{if}x\in \mathcal{Z}\mathrm{and}{J}_3\le {\beta }_1^{-1}\left(x\right)\le L,\hfill \\ {\mathcal{W}}_3\left(K_3\right)\hfill & \mathrm{if}x\in \mathcal{Z}\mathrm{and}L<{\beta }_1^{-1}\left(x\right)\le K_3,\hfill \\ Q\left(x\right)\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(193)

where the weight ${\mathcal{V}}_3\left(j\right)$ is defined by

$$\begin{array}{c}\hfill {\mathcal{V}}_3\left(j\right)={\mathcal{V}}_3^{(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})}\left(j\right):=\frac{(1-\epsilon )-{\sum }_{x=1}^{j-1}Q\left({\beta }_1\left(x\right)\right)}{L-j+1}\end{array}$$

(194)

for each integer $1\le j\le L$, the weight ${\mathcal{W}}_3\left(k\right)$ is defined by

$$\begin{array}{c}\hfill {\mathcal{W}}_3\left(k\right)={\mathcal{W}}_3^{(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})}\left(k\right):=\left\{\begin{array}{cc}-1\hfill & \mathrm{if}k=L,\hfill \\ \frac{{\sum }_{x=1}^{k}Q\left({\beta }_1\left(x\right)\right)-(1-\epsilon )}{k-L}\hfill & \mathrm{if}k>L\hfill \end{array}\right.\end{array}$$

(195)

for each integer $L\le k\le L·\left|\mathcal{Y}\right|$, the integer $J_3$ is chosen so that

$$\begin{array}{c}\hfill J_3=J_3(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z}):=min\{1\le j\le L\mid Q\left({\beta }_1\left(j\right)\right)\le {\mathcal{V}}_3\left(j\right)\},\end{array}$$

(196)

and the integer $K_3$ is chosen so that

$$\begin{array}{c}\hfill K_3=K_3(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z}):=max\{L\le k\le L·|\mathcal{Y}|\mid {\mathcal{W}}_3\left(k\right)\le P_X\left({\beta }_1\left(k\right)\right)\}.\end{array}$$

(197)

Remark 19.

The extremal distribution of type-3 can be specialized to both extremal distribution of type-2 defined in (44) and Ho–Verdú’s truncated distribution defined in Equation (17) of [21], respectively.

The following lemma shows a relation between the type-2 and the type-3.

Lemma 9.

Suppose that $\left|\mathcal{Z}\right|=L·\left|\mathcal{Y}\right|$. Then, it holds that

$$\begin{array}{c}\hfill P_{type-2}^{(Q,L,\epsilon ,\mathcal{Y})}\prec P_{type-3}^{(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})}.\end{array}$$

(198)

Proof of Lemma 9.

We readily see that

$$\begin{array}{c}\hfill P_{type-2}=P_{type-3},\end{array}$$

(199)

provided that $\mathcal{Z}=\{1,2,\cdots ,L·|\mathcal{Y}\left|\right\}$ and $Q=Q^{\downarrow }$, because $\beta :\{1,2,\cdots ,|\mathcal{Z}\left|\right\}\to \mathcal{Z}$ used in (193) is the identity mapping in this case. Actually, we may assume without loss of generality that $Q=Q^{\downarrow }$.

Although

$$\begin{array}{c}\hfill P_{type-2}=P_{type-2}^{\downarrow }\end{array}$$

(200)

does not hold in general, we can see from the definition of $P_{type-2}$ stated in (44) that

$$\begin{array}{c}\hfill P_{type-2}\left(x\right)=P_{type-2}^{\downarrow }\left(x\right)\end{array}$$

(201)

for each $x=1,2,\cdots ,L$. Therefore, as

$$\begin{array}{c}\hfill P_{type-2}\left(x\right)=Q\left(x\right)\le P_{type-3}\left(x\right)\end{array}$$

(202)

for each $x=1,2,\cdots ,J-1$, it follows that

$$\begin{array}{cc}\hfill \sum _{x=1}^k{P}_{type-2}^{\downarrow }\left(x\right)& \le \sum _{x=1}^k{P}_{type-3}^{\downarrow }\left(x\right)\hfill \end{array}$$

(203)

for each $k=1,2,\cdots ,J-1$. By the definitions (31), (33), (194), and (196), it can be verified that

$$\begin{array}{cc}\hfill J& \ge J_3,\hfill \end{array}$$

(204)

$$\begin{array}{cc}\hfill \mathcal{V}\left(J\right)& \le {\mathcal{V}}_3\left(J_3\right).\hfill \end{array}$$

(205)

Thus, as

$$\begin{array}{c}\hfill P_{type-2}^{\downarrow }\left(x\right)=\mathcal{V}\left(J\right)\end{array}$$

(206)

for each $x=J,J+1,\cdots ,L$, it follows that

$$\begin{array}{c}\hfill P_{type-3}^{\downarrow }\left(x\right)\ge {\mathcal{V}}_3\left(J_3\right)\end{array}$$

(207)

for each $x=J,J+1,\cdots ,L$; which implies that (203) also holds for each $k=J,J+1,\cdots ,L$. Therefore, we observe that $P_{type-3}$ majorizes $P_{type-2}$ over the subset $\{1,2,\cdots ,L\}\subset \mathcal{X}$.

We prove the rest of the majorization relation by contradiction. Namely, assume that

$$\begin{array}{cc}\hfill \sum _{x=1}^l{P}_{type-2}^{\downarrow }\left(x\right)& >\sum _{x=1}^l{P}_{type-3}^{\downarrow }\left(x\right)\hfill \end{array}$$

(208)

for some integer $l\ge L+1$. By the definitions stated in (32), (45), (195), and (197), it can be verified that

$$\begin{array}{cc}\hfill K_2& \le K_3,\hfill \end{array}$$

(209)

$$\begin{array}{cc}\hfill \mathcal{W}\left(K_2\right)& \ge {\mathcal{W}}_3\left(K_3\right).\hfill \end{array}$$

(210)

Thus, as

$$\begin{array}{cccc}\hfill P_{type-2}\left(x\right)& =\mathcal{W}\left(K_2\right)\le Q\left(x\right)\hfill & \hfill & (\mathrm{for}x=L+1,L+2,\cdots ,K_2),\hfill \end{array}$$

(211)

$$\begin{array}{cccc}\hfill P_{type-3}\left(x\right)& ={\mathcal{W}}_3\left(K_3\right)\le Q\left(x\right)\hfill & \hfill & (\mathrm{for}x={\beta }_1(L+1),{\beta }_1(L+2),\cdots ,{\beta }_1\left(K_3\right)),\hfill \end{array}$$

(212)

it follows that

$$\begin{array}{c}\hfill P_{type-2}\left(x\right)\ge P_{type-3}\left(x\right)\end{array}$$

(213)

for every $x=l,l+1,\cdots$, which implies together with the hypothesis (208) that

$$\begin{array}{c}\hfill \sum _{x=l}^{\infty }{P}_{type-2}^{\downarrow }\left(x\right)>\sum _{x=l}^{\infty }{P}_{type-3}^{\downarrow }\left(x\right).\end{array}$$

(214)

This, however, contradicts to the definition of probability distributions, i.e., the sum of probability masses is strictly larger than one. This completes the proof of Lemma 9. □

Similar to (164), we now define

$$\begin{array}{c}\hfill P_{\mathrm{e}}^{\left(L\right)}(X\parallel \mathcal{Z}):=\underset{\mathcal{D}\in \left(\genfrac{}{}{0pt}{}{\mathcal{Z}}{L}\right)}{min}\mathbb{P}\{X\in \mathcal{D}\}.\end{array}$$

(215)

As with Proposition 8, we can verify that

$$\begin{array}{cc}\hfill P_{\mathrm{e}}^{\left(L\right)}(X\parallel \mathcal{Z})& =1-\underset{\mathcal{D}\in \left(\genfrac{}{}{0pt}{}{\mathcal{Z}}{L}\right)}{min}\sum _{x\in \mathcal{D}}{P}_X\left(x\right)=1-\sum _{x=1}^L{P}_X\left(\beta \left(x\right)\right).\hfill \end{array}$$

(216)

Therefore, the restriction stated in (192) comes from the same observation as (29) (see Propositions 3 and 8). In view of (216), we write $P_{\mathrm{e}}^{\left(L\right)}(Q\parallel \mathcal{Z})=P_{\mathrm{e}}^{\left(L\right)}(X\parallel \mathcal{Z})$ if $P_X=Q$. As in Lemma 5, the following lemma holds.

Lemma 10.

Suppose that an $\mathcal{X}$-marginal R satisfies that (i) R majorizes Q, (ii) $P_{\mathrm{e}}^{\left(L\right)}(R\parallel \mathcal{Z})\le \epsilon$, and (iii) $R\left(k\right)=Q\left(k\right)$ for each $k\in \mathcal{X}\setminus \mathcal{Z}$. Then, it holds that R majorizes $P_{type-3}$ as well.

Proof of Lemma 10.

Since

$$\begin{array}{c}\hfill R\left(x\right)=P_{type-3}\left(x\right)=Q\left(x\right)\end{array}$$

(217)

for every $x\in \mathcal{X}\setminus \mathcal{Z}$, it suffices to verify the majorization relation over $\mathcal{Z}$. Denote by ${\beta }_1:\{1,2,\cdots ,L·|\mathcal{Y}\left|\right\}\to \mathcal{Z}$ and ${\beta }_2:\{1,2,\cdots ,L·|\mathcal{Y}\left|\right\}\to \mathcal{Z}$ two bijection satisfying $R\left({\beta }_1\left(i\right)\right)\ge R\left({\beta }_1\left(j\right)\right)$ and ${\beta }_2\left(i\right)\le {\beta }_2\left(j\right)$, respectively, whenever $i<j$. In other words, two bijections ${\beta }_1$ and ${\beta }_2$ play roles of decreasing rearrangements of R and $P_3$, respectively, on $\mathcal{Z}$. That is, we shall prove that

$$\begin{array}{c}\hfill \sum _{x=1}^k{P}_{type-3}\left({\beta }_2\left(x\right)\right)\le \sum _{x=1}^{k}R\left({\beta }_1\left(x\right)\right)\end{array}$$

(218)

for every $k=1,2,\cdots ,\left|\mathcal{Z}\right|$.

As R majorizes Q, it follows from (193) that (218) holds for each $k=1,2,\cdots ,J_3-1$. Moreover, we readily see from (193) that

$$\begin{array}{cc}\hfill \sum _{x=1}^L{P}_{type-3}\left({\beta }_2\left(x\right)\right)& =1-\epsilon .\hfill \end{array}$$

(219)

Therefore, it follows from Lemma 6 and the hypothesis $P_{\mathrm{e}}^{\left(L\right)}(R\parallel \mathcal{Z})\le \epsilon$ that (218) holds for each $k=J_3,J_3+1,\cdots ,L$. Similarly, since (218) holds with equality if $k=\left|\mathcal{Z}\right|$, it also follows from Lemma 6 that (218) holds for each $k=L+1,L+2,\cdots \left|\mathcal{Z}\right|$. Therefore, we observe that R majorizes $P_{type-3}$. This completes the proof of Lemma 10. □

Finally, we can prove Theorem 3 by using the above lemmas.

Proof of Theorem 3.

For the sake of brevity, we define

$$\begin{array}{cc}\hfill {\mathcal{R}}_1& :=\mathcal{R}(Q,L,\epsilon ,\mathcal{Y}),\hfill \end{array}$$

(220)

$$\begin{array}{cc}\hfill {\mathcal{R}}_2& :=\bigcup _{\mathcal{Z}\subset \mathcal{X}:\left|\mathcal{Z}\right|=L·\left|\mathcal{Y}\right|}\mathcal{R}(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z}),\hfill \end{array}$$

(221)

$$\begin{array}{cc}\hfill {\mathcal{R}}_3& :=\bigcup _{\mathcal{Z}\subset \mathcal{X}:\left|\mathcal{Z}\right|=L·\left|\mathcal{Y}\right|}\mathcal{R}(Q,L,\epsilon ,\mathcal{Y}\cup \mathsf{\Psi }\left(\mathcal{Z}\right),\mathcal{Z}),\hfill \end{array}$$

(222)

$$\begin{array}{cc}\hfill {\mathcal{P}}_4& :=\left\{R\in \mathcal{P}\left(\mathcal{X}\right)|\begin{array}{c}\hfill \exists \mathcal{Z}\subset \mathcal{X}\mathrm{s}.\mathrm{t}.\left|\mathcal{Z}\right|=L·\left|\mathcal{Y}\right|,\\ \hfill P_{\mathrm{e}}^{\left(L\right)}(R\parallel \mathcal{Z})\le \epsilon ,\\ \hfill R\left(x\right)=Q\left(x\right)\mathrm{for}x\in \mathcal{X}\setminus \mathcal{Z}\end{array}\right\}.\hfill \end{array}$$

(223)

Then, we have

$$\begin{array}{cc}\hfill {\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})& \stackrel{\left(\mathrm{a}\right)}{=}\underset{(X,Y)\in {\mathcal{R}}_1}{\sup }{\mathsf{H}}_{\varphi }(X\mid Y)\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{=}\underset{(X,Y)\in {\mathcal{R}}_2}{\sup }{\mathsf{H}}_{\varphi }(X\mid Y)\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{\le }\underset{(X,Y)\in {\mathcal{R}}_3}{\sup }{\mathsf{H}}_{\varphi }(X\mid Y)\hfill \\ \hfill & \stackrel{\left(\mathrm{d}\right)}{=}\underset{\begin{array}{c}(X,Y)\in {\mathcal{R}}_3:\\ (X,Y)\mathrm{is}\mathrm{connected}\mathrm{uniform}-\mathrm{dispersively}\end{array}}{\sup }{\mathsf{H}}_{\varphi }(X\mid Y)\hfill \\ \hfill & \stackrel{\left(\mathrm{e}\right)}{\le }\underset{R\in {\mathcal{P}}_4}{\sup }\varphi \left(R\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{f}\right)}{\le }\underset{\mathcal{Z}\subset \mathcal{X}:\left|\mathcal{Z}\right|=L·\left|\mathcal{Y}\right|}{\sup }\varphi \left(P_{type-3}\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{g}\right)}{\le }\varphi \left(P_{type-2}\right),\hfill \end{array}$$

(224)

where

• (a) follows from the definition of $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y})$ stated in (129),
• (b) follows from Lemma 7 and the definition of $\mathcal{R}(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})$ stated in (188),
• (c) follows from the inclusion relation

  $$\begin{array}{c}\hfill \mathcal{R}(Q,L,\epsilon ,\mathcal{Y},\mathcal{Z})\subset \mathcal{R}(Q,L,\epsilon ,\mathcal{Y}\cup \mathsf{\Psi }(\mathcal{Z}),\mathcal{Z}),\end{array}$$

  (225)
• (d) follows from Lemmas 2 and 8,
• (e) follows from Lemma 1,
• (f) follows from Lemma 10, and
• (g) follows from Proposition 1 and Lemma 9.

Inequalities (224) are indeed the Fano-type inequality stated in (47) of Theorem 3.

Finally, supposing that $\left|\mathcal{Y}\right|\ge {(K_2-J)}^2+1$, we shall construct a jointly distributed pair $(X,Y)$ satisfying

$$\begin{array}{cc}\hfill {\mathsf{H}}_{\varphi }(X\mid Y)& =\varphi \left(P_{type-2}\right),\hfill \end{array}$$

(226)

$$\begin{array}{cc}\hfill P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)& =\epsilon ,\hfill \end{array}$$

(227)

$$\begin{array}{cc}\hfill P_X\left(x\right)& =Q^{\downarrow }\left(x\right)(\mathrm{for}x\in \mathcal{X}).\hfill \end{array}$$

(228)

Similar to (153) and (154), we see that

$$\begin{array}{cc}\hfill \sum _{x=J}^k{Q}^{\downarrow }\left(x\right)& \le \sum _{x=J}^k{P}_{type-2}\left(x\right)\mathrm{for}J\le k\le K_2,\hfill \end{array}$$

(229)

and

$$\begin{array}{cc}\hfill \sum _{x=J}^{K_2}{Q}^{\downarrow }\left(x\right)& =\sum _{x=J}^{K_2}{P}_{type-2}\left(x\right).\hfill \end{array}$$

(230)

This is a majorization relation between two $(K_2-J+1)$-dimensional real vectors; and thus, it follows from the Hardy–Littlewood–Pólya theorem ([15] Theorem 8) (see also [10], Theorem 2.B.2) that there exists a $(K_2-J+1)\times (K_2-J+1)$ doubly stochastic matrix $\mathbf{M}={\left\{m_{i,j}\right\}}_{i,j=J}^{K_2}$ satisfying

$$\begin{array}{c}\hfill Q^{\downarrow }\left(i\right)=\sum _{j=J}^{K_2}{m}_{i,j}{P}_{type-2}\left(j\right)\end{array}$$

(231)

for each $J\le i\le K_2$. Moreover, it follows from Marcus–Ree’s or Farahat–Mirsky’s refinement of the finite-dimensional version of Birkhoff’s theorem derived in [75] or Theorem 3 of [76], respectively (see also Theorem 2.F.2 of [10]), that there exists a pair of a probability vector $\lambda ={\left({\lambda }_y\right)}_{y\in \mathcal{Y}}$ and a collection ${\left\{{\left\{{\pi }_{i,j}^{\left(y\right)}\right\}}_{i,j=J}^{K_2}\right\}}_{y\in \mathcal{Y}}$ of $(K_2-J+1)\times (K_2-J+1)$ permutation matrices such that

$$\begin{array}{c}\hfill m_{i,j}=\sum _{y\in \mathcal{Y}}{\lambda }_y{\pi }_{i,j}^{\left(y\right)}\end{array}$$

(232)

for every $J\le i,j\le K_2$. Using them, construct a pair $(X,Y)$ via the following distributions,

$$\begin{array}{cc}\hfill P_{X|Y=y}\left(x\right)& =\left\{\begin{array}{cc}{P}_{type-2}\left(x\right)\hfill & \mathrm{if}1\le x<J\mathrm{or}{K}_2<x<\infty ,\hfill \\ P_{type-2}\left({\tilde{\psi }}_y\left(x\right)\right)\hfill & \mathrm{if}J\le x\le K_2,\hfill \end{array}\right.\hfill \end{array}$$

(233)

$$\begin{array}{cc}\hfill P_Y\left(y\right)& ={\lambda }_y,\hfill \end{array}$$

(234)

where ${\tilde{\psi }}_y$ is defined as in (159). Similar to Section 6.1, we now observe that (226)–(228) hold. This implies together with (224) that the constructed pair $(X,Y)$ achieves the supremum in (7). Furthermore, since $P_{type-2}$ and $Q^{\downarrow }$ differ at most $\left(\genfrac{}{}{0pt}{}{K_2-J+1}{L-J+1}\right)$ probability masses, it follows that the collection ${\left\{P_{X|Y=y}\right\}}_{y\in \mathcal{Y}}$ consists of at most $\left(\genfrac{}{}{0pt}{}{K_2-J+1}{L-J+1}\right)$ distinct distributions. Namely, the condition that $\left|\mathcal{Y}\right|\ge \left(\genfrac{}{}{0pt}{}{K_2-J+1}{L-J+1}\right)$ is also sufficient to construct a jointly distributed pair $(X,Y)$ satisfying (226)–(228). This completes the proof of Theorem 3. □

Remark 20.

Step (b) in (224) is a key of proving Theorem 3; it is the reduction step from infinite to finite-dimensional settings via Lemma 7 (see also Remark 18). Note that this proof technique is not applicable when $\mathcal{Y}$ is infinite, while the proof of Theorem 1 works well for infinite $\mathcal{Y}$.

6.4. Proof of Theorem 4

It is known that every discrete probability distribution on $\{1,\cdots ,M\}$ majorizes the uniform distribution on $\{1,\cdots ,M\}$. Thus, since

$$\begin{array}{c}\hfill P_{type-0}^{(M,L,\epsilon )}=P_{type-1}^{({\mathrm{Unif}}_M,L,\epsilon )}\end{array}$$

(235)

with the uniform distribution ${\mathrm{Unif}}_M$ on $\{1,\cdots ,M\}$, it follows from Lemma 5 that

$$\begin{array}{c}\hfill P_{type-0}^{(M,L,\epsilon )}\prec P_{type-1}^{(Q,L,\epsilon )}\end{array}$$

(236)

if $supp\left(Q\right)\subset \{1,\cdots ,M\}$. Therefore, it follows from Proposition 1 and Theorems 1 and 2 that

$$\begin{array}{c}\hfill {\mathbb{H}}_{\varphi }(M,L,\epsilon ,\mathcal{Y})\le \varphi \left(P_{type-0}\right).\end{array}$$

(237)

Finally, it is easy to see that

$$\begin{array}{cc}\hfill {\mathsf{H}}_{\varphi }(X\mid Y)& =\varphi \left(P_{type-0}\right),\hfill \end{array}$$

(238)

$$\begin{array}{cc}\hfill P_{\mathrm{e}}^{\left(L\right)}(X\mid Y)& =\epsilon ,\hfill \end{array}$$

(239)

provided that

$$\begin{array}{c}\hfill P_{X|Y}\left(x\right)=P_{type-0}\left(x\right)(\mathrm{a}.\mathrm{s}.)\end{array}$$

(240)

for every $1\le x\le M$. This implies the existence of a pair $(X,Y)$ achieving the maximum in (50); and therefore, the equality (237) holds. This completes the proof of Theorem 4.

7. Proofs of Asymptotic Behaviors on Equivocations

In this section, we prove Theorems 5–7.

7.1. Proof of Theorem 5

Defining the variational distance between two $\mathcal{X}$-marginals P and Q by

$$\begin{array}{c}\hfill d(P,Q):=\frac{1}{2}\sum _{x\in \mathcal{X}}|P\left(x\right)-Q\left(x\right)|,\end{array}$$

(241)

we now introduce the following lemma, which is useful to prove Theorem 5.

Lemma 11

([77], Theorem 3). Let Q be an $\mathcal{X}$-marginal, and $0\le \delta \le 1-Q^{\downarrow }\left(1\right)$ a real number. Then, it holds that

$$\begin{array}{c}\hfill \underset{R\in \mathcal{P}\left(\mathcal{X}\right):d(Q,R)\le \delta }{min}H\left(R\right)=H\left(S^{(Q,\delta )}\right),\end{array}$$

(242)

where the $\mathcal{X}$-marginal $S^{(Q,\delta )}$ is defined by

$$\begin{array}{c}\hfill S^{(Q,\delta )}\left(x\right):=\left\{\begin{array}{cc}{Q}^{\downarrow }\left(x\right)+\delta \hfill & \mathrm{if}x=1,\hfill \\ Q^{\downarrow }\left(x\right)\hfill & \mathrm{if}1<x<B,\hfill \\ {\displaystyle \sum _{k=B}^{\infty }}{Q}^{\downarrow }\left(k\right)-\delta \hfill & \mathrm{if}x=B,\hfill \\ 0\hfill & \mathrm{if}x>B,\hfill \end{array}\right.\end{array}$$

(243)

and the integer B is chosen so that

$$\begin{array}{c}\hfill B:=\sup \left\{b\ge 1|\sum _{k=b}^{\infty }{Q}^{\downarrow }\left(k\right)\ge \delta \right\}.\end{array}$$

(244)

For the sake of brevity, in this proof, we write

$$\begin{array}{cc}\hfill {\epsilon }_n& :=P_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n),\hfill \end{array}$$

(245)

$$\begin{array}{cc}\hfill P_n& :=P_{X_n}^{\downarrow },\hfill \end{array}$$

(246)

$$\begin{array}{cc}\hfill P_{1,n}& :=P_{type-1}^{(P_n,L_n,{\epsilon }_n)}\hfill \end{array}$$

(247)

for each $n\ge 1$. Suppose that ${\epsilon }_n=\mathrm{o}\left(1\right)$ as $n\to \infty$. By Corollary 1, instead of (99), it suffices to verify that

$$\begin{array}{c}\hfill |H\left(P_{1,n}\right)-log{L}_n{|}^{+}=\mathrm{o}\left(H\left(X_n\right)\right).\end{array}$$

(248)

As $supp\left(P_{1,n}\right)=\{1,\cdots ,L_n\}$ if ${\epsilon }_n=0$, we may assume without loss of generality that $0<{\epsilon }_n<1$.

Define two $\mathcal{X}$-marginals $Q_n^{\left(1\right)}$ and $Q_n^{\left(2\right)}$ by

$$\begin{array}{cc}\hfill Q_n^{\left(1\right)}\left(x\right)& =\left\{\begin{array}{cc}\frac{P_{1,n}\left(x\right)}{1-{\epsilon }_n}\hfill & \mathrm{if}1\le x\le L_n,\hfill \\ 0\hfill & \mathrm{if}x\ge L_n+1,\hfill \end{array}\right.\hfill \end{array}$$

(249)

$$\begin{array}{cc}\hfill Q_n^{\left(2\right)}\left(x\right)& =\left\{\begin{array}{cc}0\hfill & \mathrm{if}1\le x\le L_n,\hfill \\ \frac{P_{1,n}\left(x\right)}{{\epsilon }_n}\hfill & \mathrm{if}x\ge L_n+1\hfill \end{array}\right.\hfill \end{array}$$

(250)

for each $n\ge 1$. As $Q_n^{\left(1\right)}$ majorizes the uniform distribution on $\{1,2,\cdots ,L_n\}$, it is clear from the Schur-concavity property of the Shannon entropy that

$$\begin{array}{c}\hfill H\left(Q_n^{\left(1\right)}\right)\le log{L}_n.\end{array}$$

(251)

Thus, since

$$\begin{array}{c}\hfill P_{1,n}=(1-{\epsilon }_n)Q_n^{\left(1\right)}+{\epsilon }_n{Q}_n^{\left(2\right)},\end{array}$$

(252)

it follows by strong additivity of the Shannon entropy (cf. Property (1.2.6) of [78]) that

$$\begin{array}{cc}\hfill H\left(P_{1,n}\right)& =h_2\left({\epsilon }_n\right)+(1-{\epsilon }_n)H\left(Q_n^{\left(1\right)}\right)+{\epsilon }_{n}H\left(Q_n^{\left(2\right)}\right)\hfill \\ \hfill & \le h_2\left({\epsilon }_n\right)+(1-{\epsilon }_n)log{L}_n+{\epsilon }_{n}H\left(Q_n^{\left(2\right)}\right).\hfill \end{array}$$

(253)

Thus, since $h_2\left({\epsilon }_n\right)=\mathrm{o}\left(1\right)$, it suffices to verify the asymptotic behavior of the third term in the right-hand side of (253), i.e., whether

$$\begin{array}{c}\hfill {\epsilon }_{n}H\left(Q_n^{\left(2\right)}\right)=\mathrm{o}\left(H\left(X_n\right)\right)\end{array}$$

(254)

holds or not.

Consider the $\mathcal{X}$-marginal $Q_n^{\left(3\right)}$ given by

$$\begin{array}{c}\hfill Q_n^{\left(3\right)}\left(x\right)=\frac{P_n\left(x\right)-{\epsilon }_n{Q}_n^{\left(2\right)}\left(x\right)}{1-{\epsilon }_n}\end{array}$$

(255)

for each $n\ge 1$. As

$$\begin{array}{c}\hfill P_n={\epsilon }_n{Q}_n^{\left(2\right)}+(1-{\epsilon }_n)Q_n^{\left(3\right)},\end{array}$$

(256)

it follows by the concavity of the Shannon entropy that

$$\begin{array}{c}\hfill H\left(X_n\right)\ge {\epsilon }_{n}H\left(Q_n^{\left(2\right)}\right)+(1-{\epsilon }_n)H\left(Q_n^{\left(3\right)}\right)\end{array}$$

(257)

for each $n\ge 1$. A direct calculations shows

$$\begin{array}{cc}\hfill d(P_n,Q_n^{\left(3\right)})& =\frac{1}{2}\sum _{x=1}^{\infty }|P_n\left(x\right)-Q_n^{\left(3\right)}\left(x\right)|\hfill \\ \hfill & =\frac{1}{2}\sum _{x=1}^{\infty }|P_n\left(x\right)-\frac{P_n\left(x\right)-{\epsilon }_n{Q}_n^{\left(3\right)}\left(x\right)}{1-{\epsilon }_n}|\hfill \\ \hfill & =\frac{1}{2}\frac{{\epsilon }_n}{1-{\epsilon }_n}\sum _{x=1}^{\infty }|P_n\left(x\right)-Q_n^{\left(2\right)}\left(x\right)|\hfill \\ \hfill & =\frac{{\epsilon }_n}{1-{\epsilon }_n}d(P_n,Q_n^{\left(2\right)})\hfill \\ \hfill & \le \frac{{\epsilon }_n}{1-{\epsilon }_n}\hfill \\ \hfill & =:{\delta }_n\hfill \end{array}$$

(258)

for each $n\ge 1$, where note that ${\epsilon }_n=\mathrm{o}\left(1\right)$ implies ${\delta }_n=\mathrm{o}\left(1\right)$ as well. Thus, it follows from Lemma 11 that

$$\begin{array}{cc}\hfill H\left(Q_n^{\left(3\right)}\right)& \ge H\left(S^{(P_n,{\delta }_n)}\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{a}\right)}{=}(P_n\left(1\right)+{\delta }_n)log\frac{1}{P_n\left(1\right)+{\delta }_n}+\sum _{x=2}^{B_n-1}{P}_n\left(x\right)log\frac{1}{P_n\left(x\right)}-\left(\sum _{k=B_n}^{\infty }{P}_n\left(k\right)-{\delta }_n\right)log\left(\sum _{k=B_n}^{\infty }{P}_n\left(k\right)-{\delta }_n\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{\ge }\sum _{x=1}^{B_n}{P}_n\left(x\right)log\frac{1}{P_n\left(x\right)}-2{\gamma }_n\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{=}\sum _{x\in {\mathcal{B}}^{\left(n\right)}}{P}_{X_n}\left(x\right)log\frac{1}{P_{X_n}\left(x\right)}-2{\gamma }_n\hfill \\ \hfill & \stackrel{\left(\mathrm{d}\right)}{\ge }\sum _{x\in {\mathcal{A}}_{ϵ}^{\left(n\right)}\cap {\mathcal{B}}^{\left(n\right)}}{P}_{X_n}\left(x\right)log\frac{1}{P_{X_n}\left(x\right)}-2{\gamma }_n\hfill \\ \hfill & \stackrel{\left(\mathrm{e}\right)}{\ge }\sum _{x\in {\mathcal{A}}_{ϵ}^{\left(n\right)}\cap {\mathcal{B}}^{\left(n\right)}}{P}_{X_n}\left(x\right)(1-ϵ)H\left(X_n\right)-2{\gamma }_n\hfill \\ \hfill & =\mathbb{P}\{X_n\in {\mathcal{A}}_{ϵ}^{\left(n\right)}\cap {\mathcal{B}}^{\left(n\right)}\}(1-ϵ)H\left(X_n\right)-2{\gamma }_n\hfill \end{array}$$

(259)

for every $ϵ>0$ and each $n\ge 1$, where

• (a) follows by the definition

  $$\begin{array}{c}\hfill B_n:=\sup \left\{b\ge 1|\sum _{k=b}^{\infty }{P}_n\left(k\right)\ge {\delta }_n\right\}\end{array}$$

  (260)

  for each $n\ge 1$,
• (b) follows by the continuity of the map $u\mapsto -ulogu$ and the fact that ${\delta }_n=\mathrm{o}\left(1\right)$ as $n\to \infty$, i.e., there exists a sequence ${\left\{{\gamma }_n\right\}}_{n=1}^{\infty }$ of positive reals satisfying ${\gamma }_n=\mathrm{o}\left(1\right)$ as $n\to \infty$ and

  $$\begin{array}{cc}\hfill \left|P_n\left(1\right)log\frac{1}{P_n\left(1\right)}-(P_n\left(1\right)+{\delta }_n)log\frac{1}{P_n\left(1\right)+{\delta }_n}\right|& \le {\gamma }_n,\hfill \end{array}$$

  (261)

  $$\begin{array}{cc}\hfill \left|P_n\left(B_n\right)log\frac{1}{P_n\left(B_n\right)}+\left(\sum _{k=B_n}^{\infty }{P}_n\left(k\right)-{\delta }_n\right)log\left(\sum _{k=B_n}^{\infty }{P}_n\left(k\right)-{\delta }_n\right)\right|& \le {\gamma }_n\end{array}$$

  (262)

  for each $n\ge 1$,
• (c) follows by constructing the subset ${\mathcal{B}}^{\left(n\right)}\subset \mathcal{X}$ so that

  $$\begin{array}{c}\hfill |{\mathcal{B}}^{\left(n\right)}|=\underset{\begin{array}{c}\mathcal{B}\subset \mathcal{X}:\\ \mathbb{P}\{X_n\in \mathcal{B}\}\ge 1-{\delta }_n\end{array}}{min}\left|\mathcal{B}\right|\end{array}$$

  (263)

  for each $n\ge 1$,
• (d) follows by defining the typical set ${\mathcal{A}}_{ϵ}^{\left(n\right)}\subset \mathcal{X}$ so that

  $$\begin{array}{c}\hfill {\mathcal{A}}_{ϵ}^{\left(n\right)}:=\left\{x\in \mathcal{X}|log\frac{1}{P_{X_n}\left(x\right)}\le (1-ϵ)H\left(X_n\right)\right\}\end{array}$$

  (264)

  with some $ϵ>0$ for each $n\ge 1$, and
• (e) follows by the definition of ${\mathcal{A}}_{ϵ}^{\left(n\right)}$.

As ${\left\{X_n\right\}}_{n=1}^{\infty }$ satisfies the AEP and

$$\begin{array}{cc}\hfill \mathbb{P}\{X_n\in {\mathcal{B}}^{\left(n\right)}\}& \ge 1-{\delta }_n,\hfill \end{array}$$

(265)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim}{\delta }_n& =0,\hfill \end{array}$$

(266)

it is clear that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\mathbb{P}\{X_n\notin {\mathcal{A}}_{ϵ}^{\left(n\right)}\cap {\mathcal{B}}^{\left(n\right)}\}=0\end{array}$$

(267)

(see, e.g., Problem 3.11 of [2]). Thus, since $ϵ>0$ can be arbitrarily small and ${\epsilon }_n=\mathrm{o}\left(1\right)$ as $n\to \infty$, it follows from (259) that there exists a sequence ${\left\{{\lambda }_n\right\}}_{n=1}^{\infty }$ of positive real numbers satisfying ${\lambda }_n=\mathrm{o}\left(1\right)$ as $n\to \infty$ and

$$\begin{array}{c}\hfill (1-{\epsilon }_n)H\left(Q_n^{\left(3\right)}\right)\ge (1-{\lambda }_n)H\left(X_n\right)-\frac{2{\gamma }_n}{1-{\epsilon }_n}\end{array}$$

(268)

for each $n\ge 1$. Combining (257) and (268), we observe that

$$\begin{array}{c}\hfill {\lambda }_{n}H\left(X_n\right)+\frac{2{\gamma }_n}{1-{\epsilon }_n}\ge {\epsilon }_{n}H\left(Q_n^{\left(2\right)}\right)\end{array}$$

(269)

for each $n\ge 1$. Therefore, Equation (254) is indeed valid, which proves (248) together with (253). This completes the proof of Theorem 5.

Remark 21.

The construction of $Q_n^{\left(3\right)}$ defined in (255) is a special case of the splitting technique; it was used to derive limit theorems of Markov processes by Nummelin [26] and Athreya–Ney [27]. This technique has many applications in information theory [21,28,29,30,31,32] and to the Markov chain Monte Carlo (MCMC) algorithm [79].

7.2. Proof of Theorem 6

Condition (b) is a direct consequence of Theorem 5; and we shall verify Conditions (a), (c), and (d) in the proof. For the sake of brevity, in the proof, we write

$$\begin{array}{cc}\hfill {\epsilon }_n& :=P_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n),\hfill \end{array}$$

(270)

$$\begin{array}{cc}\hfill P_n& :=P_{X_n}^{\downarrow },\hfill \end{array}$$

(271)

$$\begin{array}{cc}\hfill P& :=P_X^{\downarrow },\hfill \end{array}$$

(272)

$$\begin{array}{cc}\hfill P_{1,n}& :=P_{type-1}^{(P_n,L_n,{\epsilon }_n)}\hfill \end{array}$$

(273)

for each $n\ge 1$. By Corollary 4, instead on (113), it suffices to verify that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}|H_{\alpha }\left(P_{1,n}\right)-log{L}_n{|}^{+}=0\end{array}$$

(274)

under any one of Conditions (a), (b), and (c). Similar to the proof of Theorem 5, we may assume without loss of generality that $0<{\epsilon }_n<1$.

Firstly, we shall verify Condition (a). Let $Q_n$ be an $\mathcal{X}$-marginal given by

$$\begin{array}{c}\hfill Q_n\left(x\right)=\left\{\begin{array}{cc}\frac{1-{\epsilon }_n}{L_n}\hfill & \mathrm{if}1\le x\le L_n,\hfill \\ P_{\mathrm{type}5,\mathrm{n}}\left(x\right)\hfill & \mathrm{if}x\ge L_n+1\hfill \end{array}\right.\end{array}$$

(275)

for each $n\ge 1$. As $P_{1,n}$ majorizes $Q_n$, it follows by the Schur-concavity property of the Rényi entropy that

$$\begin{array}{cc}\hfill H_{\alpha }\left(P_{1,n}\right)& \le H_{\alpha }\left(Q_n\right)\hfill \\ \hfill & =\frac{1}{1-\alpha }log\left({(1-{\epsilon }_n)}^{\alpha }{L}_n^{1-\alpha }+\sum _{x=L_n}^{\infty }{P}_{1,n}{\left(x\right)}^{\alpha }\right)\hfill \\ \hfill & \le \frac{1}{1-\alpha }log\left({(1-{\epsilon }_n)}^{\alpha }{L}_n^{1-\alpha }\right)\hfill \\ \hfill & =log{L}_n+\frac{\alpha }{1-\alpha }log(1-{\epsilon }_n),\hfill \end{array}$$

(276)

where the second inequality follows by the hypothesis that $\alpha >1$, i.e., by Condition (a). These inequalities immediately ensure (274) under Condition (a).

Second, we shall verify Condition (d) of Theorem 6. As X and ${\left\{X_n\right\}}_n$ are discrete r.v.’s, note that the convergence in distribution $X_n\stackrel{d}{\to }X$ is equivalent to $P_n\left(x\right)\to P\left(x\right)$ as $n\to \infty$ for each $x\in \mathcal{X}$, i.e., the pointwise convergence $P_n\to P$ as $n\to \infty$. It is well-known that the Rényi entropy $\alpha \mapsto H_{\alpha }\left(P\right)$ is nonincreasing for $\alpha \ge 0$; hence, it suffices to verify (274) with $\alpha =1$, i.e.,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}|H\left(P_{1,n}\right)-log{L}_n{|}^{+}=0.\end{array}$$

(277)

We now define two $\mathcal{X}$-marginals $Q_n^{\left(1\right)}$ and $Q_n^{\left(2\right)}$ in the same ways as (249) and (250), respectively, for each $n\ge 1$. By (253), it suffices to verify whether the third term in the right-hand side of (253) approaches to zero, i.e.,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{\epsilon }_{n}H\left(Q_n^{\left(2\right)}\right)=0.\end{array}$$

(278)

This can be verified in a similar fashion to the proof of Lemma 3 of [21] as follows: Consider the $\mathcal{X}$-marginal $Q_n^{\left(3\right)}$ defined in (255) for each $n\ge 1$. Since $Q_n^{\left(2\right)}\left(1\right)=0$ and ${\epsilon }_n{Q}_n^{\left(2\right)}\left(x\right)\le {\epsilon }_n$ for each $x\ge 2$, we observe that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{\epsilon }_n{Q}_n^{\left(2\right)}\left(x\right)=0\end{array}$$

(279)

for every $x\ge 1$; therefore,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{Q}_n^{\left(3\right)}\left(x\right)=\underset{n\to \infty }{lim}{P}_{X_n}^{\downarrow }\left(x\right)\end{array}$$

(280)

for every $x\ge 1$. Therefore, since $P_n$ converges pointwise to P as $n\to \infty$, we see that $Q_n^{\left(3\right)}$ also converges pointwise to $P_X^{\downarrow }$ as ${\epsilon }_n$ vanishes. Therefore, by the lower semicontinuity property of the Shannon entropy, we observe that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}H\left(Q_n^{\left(3\right)}\right)\ge H\left(X\right),\end{array}$$

(281)

and we then have

$$\begin{array}{cc}\hfill H\left(X\right)& =\underset{n\to \infty }{lim}H\left(X_n\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{a}\right)}{\ge }\underset{n\to \infty }{lim\; sup}\left({\epsilon }_{n}H\left(Q_n^{\left(2\right)}\right)+(1-{\epsilon }_n)H\left(Q_n^{\left(3\right)}\right)\right)\hfill \\ \hfill & \ge \underset{n\to \infty }{lim\; sup}\left({\epsilon }_{n}H\left(Q_n^{\left(2\right)}\right)\right)+\underset{n\to \infty }{lim\; inf}\left((1-{\epsilon }_n)H\left(Q_n^{\left(3\right)}\right)\right)\hfill \\ \hfill & =\underset{n\to \infty }{lim\; sup}\left({\epsilon }_{n}H\left(Q_n^{\left(2\right)}\right)\right)+\underset{n\to \infty }{lim\; inf}H\left(Q_n^{\left(3\right)}\right)\hfill \\ \hfill & \ge \underset{n\to \infty }{lim\; sup}\left({\epsilon }_{n}H\left(Q_n^{\left(2\right)}\right)\right)+H\left(X\right),\hfill \end{array}$$

(282)

where (a) follows from (257). Thus, it follows from (282), the hypothesis $H\left(X\right)<\infty$, and the nonnegativity of the Shannon entropy that (278) is valid, which proves (277) together with (253).

Finally, we shall verify Condition (c) of Theorem 6. Define the $\mathcal{X}$-marginal ${\tilde{Q}}_n^{\left(2\right)}$ by

$$\begin{array}{c}\hfill {\tilde{Q}}_n^{\left(2\right)}\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}1\le x\le L_n,\hfill \\ \frac{{\tilde{P}}_{1,n}\left(x\right)}{{\epsilon }_n}\hfill & \mathrm{if}x\ge L_n+1,\hfill \end{array}\right.\end{array}$$

(283)

for each $n\ge 1$, where ${\tilde{P}}_{1,n}=P_{type-1}^{(P,L_n,{\epsilon }_n)}$. Note that the difference between $Q_n^{\left(2\right)}$ and ${\tilde{Q}}_n^{\left(2\right)}$ is the difference between $P_n$ and P. It can be verified by the same way as (282) that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\left({\epsilon }_{n}H\left({\tilde{Q}}_n^{\left(2\right)}\right)\right)=0.\end{array}$$

(284)

It follows by the same manner as Lemma 1 of [21] that if $P_n$ majorizes P, then $Q_n^{\left(2\right)}$ majorizes ${\tilde{Q}}_n^{\left(2\right)}$ as well. Therefore, it follows from the Schur-concavity property of the Shannon entropy that if $P_n$ majorizes P for sufficiently large n, then

$$\begin{array}{c}\hfill H\left(Q_n^{\left(2\right)}\right)\le H\left({\tilde{Q}}_n^{\left(2\right)}\right)\end{array}$$

(285)

for sufficiently large n. Combining (284) and (285), Equation (278) also holds under Condition (c). This completes the proof of Theorem 6.

7.3. Proof of Theorem 7

To prove Theorem 7, we now give the following lemma.

Lemma 12.

If $H\left(Q\right)<\infty$, then the map $\epsilon \mapsto H\left(P_{type-1}^{(Q,L,\epsilon )}\right)$ is concave in the interval (29) with $\left|\mathcal{Y}\right|=\infty$.

Proof of Lemma 12.

It is well-known that for a fixed $P_X$, the conditional Shannon entropy $H(X\mid Y)$ is concave in $P_{Y|X}$ (cf. [2], Theorem 2.7.4). Defining the distortion measure $d:\mathcal{X}\times \left(\genfrac{}{}{0pt}{}{\mathcal{X}}{L}\right)\to \{0,1\}$ by

$$\begin{array}{c}\hfill d(x,\widehat{x})=\left\{\begin{array}{cc}1\hfill & \mathrm{if}x\notin \widehat{x},\hfill \\ 0\hfill & \mathrm{if}x\in \widehat{x},\hfill \end{array}\right.\end{array}$$

(286)

the average probability of list decoding error is equal to the average distortion, i.e.,

$$\begin{array}{c}\hfill \mathbb{P}\{X\notin f(Y\left)\right\}=\mathbb{E}\left[d\right(X,f\left(Y\right)\left)\right]\end{array}$$

(287)

for any list decoder $f:\mathcal{Y}\to \left(\genfrac{}{}{0pt}{}{\mathcal{X}}{L}\right)$. Therefore, by following Theorem 1, the concavity property of Lemma 12 can be proved by the same argument as the proof of the convexity of the rate-distortion function (cf. Lemma 10.4.1 of [2]). □

For the sake of brevity, we write

$$\begin{array}{cc}\hfill P& =P_X,\hfill \end{array}$$

(288)

$$\begin{array}{cc}\hfill P_n& =P_{X_n},\hfill \end{array}$$

(289)

$$\begin{array}{cc}\hfill {\epsilon }_n& =P_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n),\hfill \end{array}$$

(290)

$$\begin{array}{cc}\hfill P_{1,n}& =P_{type-1}^{(P_n,L_n,{\epsilon }_n)},\hfill \end{array}$$

(291)

$$\begin{array}{cc}\hfill {\overline{P}}_{1,n}& =P_{type-1}^{(P_n,{\overline{L}}_n,{\epsilon }_n)}\hfill \end{array}$$

(292)

in this proof. Define

$$\begin{array}{c}\hfill \overline{L}:=\underset{n\to \infty }{lim\; sup}{L}_n.\end{array}$$

(293)

If $\overline{L}=\infty$, then (115) is a trivial inequality. Therefore, it suffices to consider the case where $\overline{L}<\infty$.

It is clear that there exists an integer $n_0\ge 1$ such that $L_n\le \overline{L}$ for every $n\ge n_0$. Then, we can verify that $P_{1,n}$ majorizes ${\overline{P}}_{1,n}$ for every $n\ge n_0$ as follows. Let $J_n$ and $J_3$ be given by (33) with $(Q,L,\epsilon )=(P_n,L_n,{\epsilon }_n)$ and $(Q,L,\epsilon )=(P_n,\overline{L},{\epsilon }_n)$, respectively. Similarly, let $K_n$ and $K_3$ be given by (34) with $(Q,L,\epsilon )=(P_n,L_n,{\epsilon }_n)$ and $(Q,L,\epsilon )=(P_n,\overline{L},{\epsilon }_n)$, respectively. As $L_n\le \overline{L}$ implie that $J_n\le J_3$ and $K_n\le K_3$, it can be seen from (30) that

$$\begin{array}{cc}\hfill P_{1,n}\left(x\right)& ={\overline{P}}_{1,n}\left(x\right)\mathrm{for}1\le x<J_n\mathrm{or}x\ge K_3,\hfill \end{array}$$

(294)

$$\begin{array}{cc}\hfill P_{1,n}\left(x\right)& \ge {\overline{P}}_{1,n}\left(x\right)\mathrm{for}{J}_n\le x\le L_n\mathrm{or}\overline{L}<x\le K_3,\hfill \end{array}$$

(295)

$$\begin{array}{cc}\hfill P_{1,n}\left(x\right)& \le {\overline{P}}_{1,n}\left(x\right)\mathrm{for}{L}_n<x\le \overline{L}.\hfill \end{array}$$

(296)

Therefore, noting that

$$\begin{array}{c}\hfill \sum _{x=1}^{L_n}{P}_{1,n}\left(x\right)=\sum _{x=1}^{\overline{L}}{\overline{P}}_{1,n}\left(x\right)=1-{\epsilon }_n,\end{array}$$

(297)

we obtain the majorization relation $P_{1,n}\succ {\overline{P}}_{1,n}$ for every $n\ge n_0$.

By hypothesis, there exists an integer $n_1\ge 1$ such that $P_n$ majorizes P for every $n\ge n_1$. Letting $n_2=max\{n_0,n_1\}$, we observe that

$$\begin{array}{cc}\hfill \frac{1}{n}H(X^n\mid Y^n)& \le \frac{1}{n}\sum _{i=1}^{n}H(X_i\mid Y_i)\hfill \\ \hfill & \le \frac{1}{n}\sum _{i=1}^{n_2-1}H(X_i\mid Y_i)+\frac{1}{n}\sum _{j=n_2}^{n}H(X_i\mid Y_i)\hfill \\ \hfill & \stackrel{\left(\mathrm{a}\right)}{\le }\frac{n_2-1}{n}\left(\underset{1\le i<n_2}{max}H\left(X_i\right)\right)+\frac{1}{n}\sum _{j=n_2}^{n}H\left({\overline{P}}_{1,j}\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{\le }\frac{n_2-1}{n}\left(\underset{1\le i<n_2}{max}H\left(X_i\right)\right)+\frac{1}{n}\sum _{j=n_2}^{n}H\left(P_{type-1}^{(P,\overline{L},{\epsilon }_j)}\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{\le }\frac{n_2-1}{n}\left(\underset{1\le i<n_2}{max}H\left(X_i\right)\right)+\frac{n-n_2+1}{n}H\left(P_{type-1}^{(P,\overline{L},{\overline{\epsilon }}_n)}\right)\hfill \end{array}$$

(298)

for every $n\ge n_2$, where

• (a) follows by Corollary 4 and $P_{1,n}\succ {\overline{P}}_{1,n}$,
• (b) follows by Condition (b) of Theorem 6 and the same manner as ([21], Lemma 1), and
• (c) follows by Lemma 12 together with the following definition

  $$\begin{array}{c}\hfill {\overline{\epsilon }}_n:=\frac{1}{n-n_2+1}\sum _{j=n_2}^n{\epsilon }_j=\frac{1}{n-n_2+1}\sum _{j=n_2}^n{P}_{\mathrm{e}}^{\left(L_j\right)}(X_j\mid Y_j).\end{array}$$

  (299)

Note that the Schur-concavity property of the Shannon entropy is used in both (b) and (c) of (298). As

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{P}_{\mathrm{e},\mathrm{sym}.}^{\left(\mathbf{L}\right)}(X^n\mid Y^n)=0\iff \underset{n\to \infty }{lim}{\overline{\epsilon }}_n=0,\end{array}$$

(300)

it follows from (274) that there exists an integer $n_3\ge 1$ such that

$$\begin{array}{c}\hfill H\left(P_{type-1}^{(P,\overline{L},{\overline{\epsilon }}_n)}\right)\le log\overline{L}\end{array}$$

(301)

for every $n\ge n_3$. Therefore, it follows from (298) that

$$\begin{array}{cc}\hfill \frac{1}{n}H(X^n\mid Y^n)& \le \frac{n_2-1}{n}\left(\underset{1\le i<n_2}{max}H\left(X_i\right)\right)+\frac{n-n_2+1}{n}log\overline{L}\hfill \end{array}$$

(302)

for every $n\ge max\{n_2,n_3\}$. Therefore, letting $n\to \infty$ in (302), we have (115). This completes the proof of Theorem 7.

8. Concluding Remarks

8.1. Impossibility of Establishing Fano-Type Inequality

In Section 3, we explored the principal maximization problem ${\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})$ defined in (7) without any explicit form of $\varphi$ under the three postulates: $\varphi$ is symmetric, concave, and lower semicontinuous. If $\epsilon >0$ and we impose another postulate on $\varphi$, then we can also avoid the (degenerate) case in which $\varphi \left(Q\right)=\infty$. The following proposition shows this fact.

Proposition 9.

Let $g_1:[0,1]\to [0,\infty )$ be a function satisfying $g_1\left(0\right)=0$, and $g_2:[0,\infty ]\to [0,\infty ]$ a function satisfying $g_2\left(u\right)=\infty$ only if $u=\infty$. Suppose that $\epsilon >0$ and $\varphi :\mathcal{P}\left(\mathcal{X}\right)\to [0,\infty ]$ is of the form

$$\begin{array}{cc}\hfill \varphi \left(Q\right)& =g_2\left(\sum _{x\in \mathcal{X}}{g}_1\left(Q\left(x\right)\right)\right).\hfill \end{array}$$

(303)

Then, it holds that

$$\begin{array}{c}\hfill {\mathbb{H}}_{\varphi }(Q,L,\epsilon ,\mathcal{Y})<\infty \iff \varphi \left(Q\right)<\infty .\end{array}$$

(304)

Proof of Proposition 9.

See Appendix D. □

As seen in Section 4, the conditional Shannon and Rényi entropies can be expressed by ${\mathsf{H}}_{\varphi }(X\mid Y)$; and then $\varphi$ must satisfy (303). Proposition 9 shows that we cannot establish an effective Fano-type inequality based on the conditional information measure ${\mathsf{H}}_{\varphi }(X\mid Y)$ subject to our original postulates in Section 2.1, provided that (i) $\varphi$ satisfies the additional postulate of (303), (ii) $\epsilon >0$, and (iii) $\varphi \left(Q\right)=\infty$. This generalizes a pathological example given in Example 2.49 of [4], which states issues of the interplay between conditional information measures and error probabilities over countably infinite alphabets $\mathcal{X}$; see Section 1.2.1.

8.2. Postulational Characterization of Conditional Information Measures

Our Fano-type inequalities were stated in terms of the general conditional information ${\mathsf{H}}_{\varphi }(X\mid Y)$ defined in Section 2.1. As shown in Section 4, the quantity ${\mathsf{H}}_{\varphi }(X\mid Y)$ can be specialized to Shannon’s and Rényi’s information measures. Moreover, the quantity ${\mathsf{H}}_{\varphi }(X\mid Y)$ can be further specialized to the following quantities:

• If $\varphi ={\parallel ·\parallel }_{1/2}$, then ${\mathsf{H}}_{\varphi }(X\mid Y)$ coincides with the (unnormalized) Bhattacharyya parameter (cf. Definition 17 of [80] and Section 4.2.1 of [81]) defined by

  $$\begin{array}{c}\hfill B(X\mid Y):=\mathbb{E}\left[\sum _{x,x^{\prime }\in \mathcal{X}}\sqrt{P_{X|Y}\left(x\right)P_{X|Y}\left(x^{\prime }\right)}\right].\end{array}$$

  (305)

  Note that the Bhattacharyya parameter is often defined so that $Z(X\mid Y):=\left(B\right(X\mid Y)-1)/(M-1)$ to normalize as $0\le Z(X\mid Y)\le 1$, provided that X is $\{0,1,\cdots ,M-1\}$-valued. When X takes values in a finite alphabet with a certain algebraic structure, the Bhattacharyya parameter $B(X\mid Y)$ is useful in analyzing the speed of polarization for non-binary polar codes (cf. [80,81]). Note that $B(X\mid Y)$ is a monotone function of Arimoto’s conditional Rényi entropy (64) of order $\alpha =1/2$.
• If $\varphi ={1-\parallel ·\parallel }_2^2$, then ${\mathsf{H}}_{\varphi }(X\mid Y)$ coincides with the conditional quadratic entropy [82] defined by

  $$\begin{array}{cc}\hfill H_{\mathrm{o}}(X\mid Y)& :=\mathbb{E}\left[\sum _{x\in \mathcal{X}}{P}_{X|Y}\left(x\right)\left(1-P_{X|Y}\left(x\right)\right)\right],\hfill \end{array}$$

  (306)

  which is used in the analysis of stochastic decoding (see, e.g., [83]). Note that $H_{\mathrm{o}}(X\mid Y)$ is a monotone function of Hayashi’s conditional Rényi entropy (69) of order $\alpha =2$.
• If X is $\{1,2,\cdots ,M\}$-valued, then one can define the following (variational distance-like) conditional quantity:

  $$\begin{array}{cc}\hfill K(X\mid Y)& :=\mathbb{E}\left[\frac{1}{2(M-1)}\sum _{x=1}^M\sum _{x^{\prime }=1}^M|P_{X|Y}\left(x\right)-P_{X|Y}\left(x^{\prime }\right)|\right].\hfill \end{array}$$

  (307)

  Note that $0\le K(X\mid Y)\le 1$. This quantity $K(X\mid Y)$ was introduced by Shuval–Tal [84] to analyze the speed of polarization of non-binary polar codes for sources with memory. When we define the function $\overline{d}:\mathcal{P}\left(\{1,2,\cdots ,M\}\right)\to [0,1]$ by

  $$\begin{array}{cc}\hfill \overline{d}\left(P\right)& :=\frac{1}{2(M-1)}\sum _{x=1}^M\sum _{x^{\prime }=1}^M|P\left(x\right)-P\left(x^{\prime }\right)|,\hfill \end{array}$$

  (308)

  it holds that $K(X\mid Y)={\mathsf{H}}_{\overline{d}}(X\mid Y)$. Clearly, the function $\overline{d}$ is symmetric, convex, and continuous.

On the other hand, the quantity ${\mathsf{H}}_{\varphi }(X\mid Y)$ has the following properties that are appealing in information theory:

• As $\varphi$ is concave, lower bounded, and lower semicontinuous, it follows from Jensen’s inequality for an extended real-valued function on a closed, convex, and bounded subset of a Banach space ([14], Proposition A-2) that

  $$\begin{array}{c}\hfill {\mathsf{H}}_{\varphi }(X\mid Y)\le \varphi \left(P_X\right).\end{array}$$

  (309)

  This bound is analogous to the property that conditioning reduces entropy (cf. [2], Theorem 2.6.5).
• It is easy to check that for any (deterministic) mapping $g:\mathcal{X}\to \mathcal{A}$ with $\mathcal{A}\subset \mathcal{X}$, the conditional distribution $P_{g\left(X\right)|Y}$ majorizes $P_{X|Y}$ a.s. Thus, it follows from Proposition 1 that for any mapping $g:\mathcal{X}\to \mathcal{A}$,

  $$\begin{array}{c}\hfill {\mathsf{H}}_{\varphi }(g\left(X\right)\mid Y)\le {\mathsf{H}}_{\varphi }(X\mid Y),\end{array}$$

  (310)

  which is a counterpart of the data processing inequality (cf. Equations (26)–(28) of [72]).
• As shown in Section 3, the quantity ${\mathsf{H}}_{\varphi }(X\mid Y)$ also satisfies appropriate generalizations of Fano’s inequality.

Therefore, similar to the family of f-divergences [85,86], the quantity ${\mathsf{H}}_{\varphi }(X\mid Y)$ is a generalization of various information-theoretic conditional quantities that also admit certain desirable properties. In addition, we can establish Fano-type inequalities based on ${\mathsf{H}}_{\varphi }(X\mid Y)$; this characterization provides insights on how to measure conditional information axiomatically.

8.3. When Does Vanishing Error Probabilities Imply Vanishing Equivocations?

In the list decoding setting, the rate of a block code with codeword length n, message size $M_n$, and list size $L_n$ can be defined as $(1/n)log(M_n/L_n)$ (cf. [87]). Motivated by this, we established asymptotic behaviors of this quantity in Theorems 5 and 6. We would like to emphasize that Example 2 shows that Ahlswede–Gács–Körner’s proof technique described in Chapter 5 of [42] (see also Section 3.6.2 of [43]) works for an i.i.d. source on a countably infinite alphabet, provided that the alphabets ${\left\{{\mathcal{Y}}_n\right\}}_{n=1}^{\infty }$ are finite.

Theorem 5 states that the asymptotic growth of $H(X_n\mid Y_n)-log{L}_n$ is strictly slower than $H\left(X_n\right)$, provided that the general source $\mathbf{X}={\left\{X_n\right\}}_{n=1}^{\infty }$ satisfies the AEP and the error probabilities vanish (i.e., $P_{\mathrm{e}}^{\left(L_n\right)}(X_n\mid Y_n)=\mathrm{o}\left(1\right)$ as $n\to \infty$). This is a novel characterization of the AEP via Fano’s inequality. An instance of this characterization using the Poisson source (cf. Example 4 of [25]) was provided in Example 3.

8.4. Future Works

• While there are various studies of the reverse Fano inequalities [22,23,49,50,51,52], this study has focused only on the forward Fano inequality. Generalizing the reverse Fano inequality in the same spirit as was done in this study would be of interest.
• Important technical tools used in our analysis include the finite- and infinite-dimensional versions of Birkhoff’s theorem; they were employed to satisfy the constraint that $P_X=Q$. As a similar constraint is imposed in many information-theoretic problems, e.g., coupling problems (cf. [7,88,89]), finding further applications of the infinite-dimensional version of Birkhoff’s theorems would refine technical tools, and potentially results, when we are dealing with communication systems on countably infinite alphabets.
• We have described a novel connection between the AEP and Fano’s inequality in Theorem 5; its role in the classifications of sources and channels and its applications to other coding problems are of interest.