Some Useful Integral Representations for Information-Theoretic Analyses

Abstract

This work is an extension of our earlier article, where a well-known integral representation of the logarithmic function was explored and was accompanied with demonstrations of its usefulness in obtaining compact, easily-calculable, exact formulas for quantities that involve expectations of the logarithm of a positive random variable. Here, in the same spirit, we derive an exact integral representation (in one or two dimensions) of the moment of a nonnegative random variable, or the sum of such independent random variables, where the moment order is a general positive non-integer real (also known as fractional moments). The proposed formula is applied to a variety of examples with an information-theoretic motivation, and it is shown how it facilitates their numerical evaluations. In particular, when applied to the calculation of a moment of the sum of a large number, n, of nonnegative random variables, it is clear that integration over one or two dimensions, as suggested by our proposed integral representation, is significantly easier than the alternative of integrating over n dimensions, as needed in the direct calculation of the desired moment.

1. Introduction

In mathematical analyses associated with many problems in information theory and related fields, one is often faced with the need to compute expectations of logarithmic functions of composite random variables (see, e.g., [1,2,3,4,5,6,7,8]), or moments of such random variables, whose order may be a general positive real, not even necessarily an integer (see, e.g., [9,10,11,12,13,14,15,16,17,18,19,20,21,22]).

In the case of the logarithmic function, the common practice is either to resort to approximate evaluations, provided by upper and lower bounds on the desired expression (for example, by using Jensen’s inequality) or to approximate the calculations by using the Taylor series expansion of the function $lnx$. More recently, it has become popular to use the replica trick (see, e.g., Chapter 8 in [23]), which is a non-rigorous, but useful technique, borrowed from statistical physics.

In our earlier work [6], we demonstrated how the following well-known integral representation of the logarithmic function,

$$\begin{array}{c}\hfill lnx={\int }_0^{\infty }\left(e^{-u}-e^{-ux}\right)\frac{\mathrm{d}u}{u},x>0,\end{array}$$

(1)

can be useful in a variety of application areas in the field of information theory, including both source and channel coding, as well as other aspects of this field. To calculate the expectation, $\mathbb{E}\{lnX\}$, where X is a positive random variable, the idea is simply to invoke the integral representation (1) and to commute the expectation and integration operators, i.e.,

$$\begin{array}{c}\hfill \mathbb{E}\{lnX\}={\int }_0^{\infty }(e^{-u}-\mathbb{E}\left\{e^{-uX}\right\})\frac{\mathrm{d}u}{u},\end{array}$$

(2)

thereby replacing the calculation of $\mathbb{E}\{lnX\}$ by the calculation of the moment-generating function (MGF), $M_X(u):=\mathbb{E}\{e^{uX}\}$ for all $u\le 0$, which is often much easier to express in closed form. Moreover, in frequently encountered situations where X is given by the sum of n independently identically distributed (i.i.d.) random variables, the MGF of X is given by the $n^{\mathrm{th}}$ power of the MGF of a single random variable in the sum that forms X. This reduces the dimension of the integration from n (in the original expression) to a single dimension of the integration over u. Interestingly, this integral representation has also been used in the statistical physics literature (see, e.g., [23] (p. 140) [24,25]), but not as much as the replica trick.

In this paper, we proceed in the same spirit as in [6], and we extend the scope to propose an integral representation of a general moment of a nonnegative random variable, X, namely the expectation, $\mathbb{E}\{X^{\rho }\}$ for a given real $\rho >0$. Obviously, when $\rho$ is an integer, this moment is simply given by the ${\rho }^{\mathrm{th}}$ order derivative of the MGF of X, calculated at the origin, as is very well known. However, the integral representation we propose, in this work, applies to any non-integer, positive $\rho$, and here too, it replaces the direct calculation of $\mathbb{E}\{X^{\rho }\}$ by integration of an expression that involves the MGF of X. We refer to this representation as an extension of (2), as the latter can be obtained as a special case of the formula for $\mathbb{E}\{X^{\rho }\}$, by invoking one of the equivalent identities

$$\begin{array}{c}\hfill \mathbb{E}\{lnX\}=\underset{\rho \to 0}{lim}\frac{\mathbb{E}\{X^{\rho }\}-1}{\rho },\mathbb{E}\{lnX\}=\underset{\rho \to 0}{lim}\frac{ln\left[\mathbb{E}\{X^{\rho }\}\right]}{\rho }.\end{array}$$

(3)

While the proposed integral representation of $\mathbb{E}\{X^{\rho }\}$ can be readily obtained from [26] (p. 363, Identity (3.434.1)) in the range $\rho \in (0,1)$, the nontrivial extension we propose for a non-integer and real $\rho >1$ is new to the best of our knowledge.

Fractional moments have been considered in the mathematical literature (see, e.g., [27,28,29,30]). A relationship between fractional and integer-order moments was considered in [27] by expressing a fractional moment as an infinite series, which depends on all the positive integer-order moments, followed by an algorithm for numerical calculations of fractional moments.

As in [6], the proposed integral representation is applied to a variety of examples with an information-theoretic motivation, and it is shown how it facilitates the numerical evaluations. In particular, similar to the case of the logarithmic function, when applied to the calculation of a moment of the sum of a large number, n, of nonnegative random variables, it is clear that integration over one or two dimensions, as suggested by our proposed integral representation, is significantly easier than the alternative of integrating over n dimensions, as needed in the direct calculation of the desired moment. Furthermore, single- or double-dimensional integrals can be instantly and accurately calculated using built-in numerical integration procedures.

Fractional moments have been considered in the mathematical literature (see, e.g., [27,28,29,30]). A relationship between fractional and integer-order moments was considered in [27] by expressing a fractional moment as an infinite series that depends on all the positive integer-order moments, which was followed by an algorithm for numerical calculations of fractional moments.

The outline of the remainder part of this paper is as follows. In Section 2, we provide the mathematical background associated with the integral representation in general. In Section 3, we demonstrate this integral representation in applications, including: moments of guesswork, moments of estimation errors, differential Rényi entropies of generalized multivariate Cauchy distributions, and mutual information calculations of a certain model of a jammed channel. Each one of these examples occupies one subsection of Section 3. The integral representations in this paper are not limited to the examples in Section 3, and such representations can be proven useful in other information-theoretic problems (see, e.g., [6] and the references therein for the integral representation of the logarithmic expectation and some of its information-theoretic applications).

2. Statistical Moments of Arbitrary Positive Orders

It is well known that integer-order moments of a random variable X are calculable from its MGF

$$\begin{array}{c}\hfill M_X(u):=\mathbb{E}\left\{e^{uX}\right\},u\in \mathbb{R},\end{array}$$

(4)

by using its derivatives, calculated at $u=0$, i.e.,

$$\begin{array}{c}\hfill \mathbb{E}\{X^{\rho }\}=M_X^{(\rho )}(0),\rho \in \mathbb{N}.\end{array}$$

(5)

Quite often, however, there is a theoretical and practical interest to calculate fractional moments of nonnegative random variables. We next obtain a closed-form integral expression of the ${\rho }^{\mathrm{th}}$ moment of a nonnegative random variable X, as a functional of its MGF, for any positive real $\rho$. Before we proceed, it should be noted that for $\rho \in (0,1)$, such an expression is available in handbooks of standard tables of integrals, for example in [26] (p. 363, Identity (3.434.1)). The first innovation here, however, is in a nontrivial extension of this formula for all $\rho >0$ as an expression that involves a one-dimensional integral. It should be noted that although the definition of a fractional moment of a random variable (RV) is also given by a one-dimensional integral (or a sum, depending on whether the RV is discrete or continuous), the utility of our formula is, e.g., in expressing the ${\rho }^{\mathrm{th}}$ moment of a sum of nonnegative and independent random variables as a one-dimensional integral, instead of an n-dimensional integral, which is obtained by the direct definition. This new formula serves as the basic building block in all of our information-theoretic applications throughout this paper.

We first define the Beta and Gamma functions (see, e.g., Section 8.3 in [26] and Chapter 5 in [31]):

$$\begin{array}{cc}\hfill B(u,v)& :={\int }_0^1{t}^{u-1}{(1-t)}^{v-1}\mathrm{d}t,u,v>0,\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill \mathsf{\Gamma }(u)& :={\int }_0^{\infty }{t}^{u-1}{e}^{-t}\mathrm{d}t,u>0,\hfill \end{array}$$

(7)

where these functions are related by the equality

$$\begin{array}{c}\hfill B(u,v)=\frac{\mathsf{\Gamma }(u)\mathsf{\Gamma }(v)}{\mathsf{\Gamma }(u+v)},u,v>0.\end{array}$$

(8)

Theorem 1.

Let X be a nonnegative random variable, and let $\rho >0$ be a non-integer real. Then,

$$\begin{array}{cc}\hfill \mathbb{E}\{X^{\rho }\}& =\frac{1}{1+\rho }\sum _{\ell =0}^{\lfloor \rho \rfloor }\frac{{\alpha }_{\ell }}{B(\ell +1,\rho +1-\ell )}\hfill \\ \hfill & +\frac{\rho sin(\pi \rho )\mathsf{\Gamma }(\rho )}{\pi }{\int }_0^{\infty }\frac{1}{u^{\rho +1}}\left(\sum _{j=0}^{\lfloor \rho \rfloor }\left\{\frac{{(-1)}^j{\alpha }_j}{j!}{u}^j\right\}{e}^{-u}-M_X(-u)\right)\mathrm{d}u,\hfill \end{array}$$

(9)

where, for all $j\in \{0,1,\dots ,\}$,

$$\begin{array}{c}{\alpha }_j:=\mathbb{E}\{{(X-1)}^j\}\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & =\frac{1}{j+1}\sum _{\ell =0}^j\frac{{(-1)}^{j-\ell }{M}_X^{(\ell )}(0)}{B(\ell +1,j-\ell +1)}.\hfill \end{array}$$

(11)

Proof. 

See Appendix A. □

Remark 1.

The proof of (9) in Appendix A does not apply to $\rho \in \mathbb{N}$ (see (A7) and (A8) etc., where the denominators vanish for $\rho \in \mathbb{N}$). In the latter case, by referring to the second term on the right-hand side of (9), we get $sin(\pi \rho )=0$, and also, the integral diverges (specifically, for $\rho \in \mathbb{N}$, the integrand scales like $\frac{1}{u}$ for u that is sufficiently close to zero), yielding an expression of the type $0·\infty$. However, taking a limit in (9) where we let ρ tend to an integer and applying L’Hôpital’s rule can reproduce the well-known result in (5).

Corollary 1.

For any $\rho \in (0,1)$,

$$\begin{array}{cc}\hfill \mathbb{E}\{X^{\rho }\}& =1+\frac{\rho }{\mathsf{\Gamma }(1-\rho )}{\int }_0^{\infty }\frac{e^{-u}-M_X(-u)}{u^{1+\rho }}\mathrm{d}u.\hfill \end{array}$$

(12)

Proof. 

Equation (12) is due to Theorem 1 and by using (A20) and (A22) (see Appendix A), and ${\alpha }_0:=1$, which give

$$\begin{array}{cc}\hfill \mathsf{\Gamma }(\rho )\mathsf{\Gamma }(1-\rho )& =\frac{\pi }{sin(\pi \rho )},\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill \frac{1}{1+\rho }\frac{{\alpha }_0}{B(1,\rho +1)}& =\frac{1}{1+\rho }\frac{\mathsf{\Gamma }(\rho +2)}{\mathsf{\Gamma }(\rho +1)}\hfill \\ \hfill & =1.\hfill \end{array}$$

(14)

 □

Remark 2.

Corollary 1 also follows from [26] (p. 363, Identity (3.434.1)) (see Section 4 in [6]).

Corollary 2.

[6] Let X be a positive random variable. Then,

$$\begin{array}{c}\hfill \mathbb{E}\{lnX\}={\int }_0^{\infty }\frac{e^{-u}-M_X(-u)}{u}\mathrm{d}u.\end{array}$$

(15)

A proof of (15) was presented in Section 2 in [6], based on the integral representation of the logarithmic function in (1), and by interchanging the integration and the expectation (due to Fubini’s theorem). It can be alternatively proven by using Corollary 1, the identity

$$\begin{array}{c}\hfill lnx=\underset{\rho \to 0}{lim}\frac{x^{\rho }-1}{\rho },x>0,\end{array}$$

(16)

and swapping the order of the expectation and limit by the dominated convergence theorem.

Identity (15) has many useful information-theoretic applications in its own right, as demonstrated in [6], and here, we add even some more. The current work is an extension and further development of [6], whose main theme is exploiting Theorem 1 and studying its information-theoretic applications, as well as some more applications of the logarithmic expectation.

3. Applications

In this section, we exemplify the usefulness of the integral representation of the ${\rho }^{\mathrm{th}}$ moment in Theorem 1 and the logarithmic expectation in several problem areas in information theory and statistics. These include analyses of randomized guessing, estimation errors, Rényi entropy of n-dimensional generalized Cauchy distributions, and finally, calculations of the mutual information for channels with a certain jammer model. To demonstrate the direct computability of the relevant quantities, we also present graphs of their numerical calculations.

3.1. Moments of Guesswork

Consider the problem of guessing the realization of a random variable, which takes on values in a finite alphabet, using a sequence of yes/no questions of the form “Is $X=x_1$?”, “Is $X=x_2$?”, etc., until a positive response is provided by a party that observes the actual realization of X. Given a distribution of X, a commonly used performance metric for this problem is the expected number of guesses or, more generally, the ${\rho }^{\mathrm{th}}$ moment of the number of guesses until X is guessed successfully. When it comes to guessing random vectors, say, of length n, minimizing the moments of the number of guesses by different (deterministic or randomized) guessing strategies has several applications and motivations in information theory, such as sequential decoding, guessing passwords, etc., and it is also strongly related to lossless source coding (see, e.g., [9,10,11,12,13,19,20,21,22,32,33,34]). In this vector case, the moments of the number of guesses behave as exponential functions of the vector dimension, n, at least asymptotically, as n grows without bound. For random vectors with i.i.d. components, the best achievable asymptotic exponent of the ${\rho }^{\mathrm{th}}$ guessing moment is expressed in [9] by using the Rényi entropy of X of order $\tilde{\rho }:=\frac{1}{1+\rho }$. Arikan assumed in [9] that the distribution of X is known and analyzed the optimal deterministic guessing strategy, which orders the guesses according to nonincreasing probabilities. Refinements of the exponential bounds in [9] with tight upper and lower bounds on the guessing moments for optimal deterministic guessing were recently derived in [19]. In the sequel, we refer to randomized guessing strategies, rather than deterministic strategies, and we aim to derive exact, calculable expressions for their associated guessing moments (as is later explained in this subsection).

Let the random variable X take on values in a finite alphabet $\mathcal{X}$. Consider a random guessing strategy where the guesser sequentially submits a sequence of independently drawn random guesses according to a certain probability distribution, $\tilde{P}(·)$, defined on $\mathcal{X}$. Randomized guessing strategies have the advantage that they can be used by multiple asynchronous agents, which submit their guesses concurrently (see [33,34]).

In this subsection, we consider the setting of randomized guessing and obtain an exact representation of the guessing moment in the form of a one-dimensional integral. Let $x\in \mathcal{X}$ be any realization of X, and let the guessing distribution, $\tilde{P}$, be given. The random number, G, of independent guesses until success has the geometric distribution

$$\begin{array}{c}\hfill \mathrm{Pr}\{G=k|x\}=[1-\tilde{P}(x){]}^{k-1}\tilde{P}(x),k\in \mathbb{N},\end{array}$$

(17)

and so, the corresponding MGF is equal to

$$\begin{array}{cc}\hfill M_G(u|x)& =\sum _{k=1}^{\infty }{e}^{ku}\mathrm{Pr}\{G=k|x\}\hfill \\ \hfill & =\frac{\tilde{P}(x)}{e^{-u}-(1-\tilde{P}(x))},u<ln\frac{1}{1-\tilde{P}(x)}.\hfill \end{array}$$

(18)

In view of (9)–(11) and (18), for $x\in \mathcal{X}$ and non-integer $\rho >0$,

$$\begin{array}{cc}\hfill \mathbb{E}\{G^{\rho }|x\}& =\frac{1}{1+\rho }\sum _{\ell =0}^{\lfloor \rho \rfloor }\frac{{\alpha }_{\ell }}{B(\ell +1,\rho +1-\ell )}\hfill \\ \hfill & +\frac{\rho sin(\pi \rho )\mathsf{\Gamma }(\rho )}{\pi }{\int }_0^{\infty }\frac{1}{u^{\rho +1}}\left(\sum _{j=0}^{\lfloor \rho \rfloor }\left\{\frac{{(-1)}^j{\alpha }_j}{j!}{u}^j\right\}{e}^{-u}-\frac{\tilde{P}(x)}{e^u-(1-\tilde{P}(x))}\right)\mathrm{d}u,\hfill \end{array}$$

(19)

with ${\alpha }_0:=1$, and for all $j\in \mathbb{N}$

$$\begin{array}{cc}\hfill {\alpha }_j& :=\mathbb{E}\left\{{\left(G-1\right)}^j|X=x\right\}\hfill \\ \hfill & =\sum _{k=1}^{\infty }{(k-1)}^j{\left(1-\tilde{P}(x)\right)}^{k-1}\tilde{P}(x)\hfill \\ \hfill & =\tilde{P}(x){\mathrm{Li}}_{-j}\left(1-\tilde{P}(x)\right).\hfill \end{array}$$

(20)

In (20), ${\mathrm{Li}}_{-j}(·)$ is a polylogarithm (see, e.g., Section 25.12 in [31]), which is given by

$$\begin{array}{c}\hfill {\mathrm{Li}}_{-j}(x)={\left(x\frac{\mathrm{d}}{\mathrm{d}x}\right)}^j\frac{x}{1-x},\forall j\in \mathbb{N}\cup \{0\},\end{array}$$

(21)

with ${\left(x\frac{\mathrm{d}}{\mathrm{d}x}\right)}^j$ denoting differentiation with respect to x and multiplication of the derivative by x, repeatedly j times. In particular, we have

$$\begin{array}{c}\hfill {\mathrm{Li}}_0(x)=\frac{x}{1-x},{\mathrm{Li}}_{-1}(x)=\frac{x}{{(1-x)}^2},{\mathrm{Li}}_{-2}(x)=\frac{x(1+x)}{{(1-x)}^3},\end{array}$$

(22)

and so on. The function ${\mathrm{Li}}_{-j}(x)$ is a built-in function in the MATLAB and Mathematica software, which is expressed as $\mathrm{polylog}(-j,x)$. By Corollary 1, if $\rho \in (0,1)$, then (19) is simplified to

$$\begin{array}{cc}\hfill \mathbb{E}\{G^{\rho }|x\}& =1+\frac{\rho }{\mathsf{\Gamma }(1-\rho )}{\int }_0^{\infty }\frac{e^{-u}-e^{-2u}}{u^{\rho +1}[{(1-\tilde{P}(x))}^{-1}-e^{-u}]}\mathrm{d}u.\hfill \end{array}$$

(23)

Let P denote the distribution of X. Averaging over X to get the unconditional ${\rho }^{\mathrm{th}}$ moment using (23), one obtains for all $\rho \in (0,1)$,

$$\begin{array}{c}\hfill \mathbb{E}\{G^{\rho }\}=1+\frac{\rho }{\mathsf{\Gamma }(1-\rho )}{\int }_0^1\frac{1-z}{{(-lnz)}^{\rho +1}}\sum _{x\in \mathcal{X}}\frac{P(x)(1-\tilde{P}(x))}{1-z(1-\tilde{P}(x))}\mathrm{d}z,\end{array}$$

(24)

where (24) is obtained by using the substitution $z:=e^{-u}$. A suitable expression of such an integral is similarly obtained, for all $\rho >0$, by averaging (19) over X. In comparison, a direct calculation of the ${\rho }^{\mathrm{th}}$ moment gives

$$\begin{array}{c}\hfill \mathbb{E}\{G^{\rho }\}=\sum _{x\in \mathcal{X}}P(x)\mathbb{E}\{G^{\rho }|x\}=\sum _{k=1}^{\infty }\sum _{x\in \mathcal{X}}{k}^{\rho }{(1-\tilde{P}(x))}^{k-1}\tilde{P}(x)P(x).\end{array}$$

(25)

The double sum in (25) involves a numerical computation of an infinite series, where the number of terms required to obtain a good approximation increases with $\rho$ and needs to be determined. The right-hand side of (24), on the other hand, involves integration over $[0,1]$. For every practical purpose, however, definite integrals in one or two dimensions can be calculated instantly using built-in numerical integration procedures in MATLAB, Maple, Mathematica, or any other mathematical software tools, and the computational complexity of the integral in (24) is not affected by $\rho$.

As a complement to (19) (which applies to a non-integral and positive $\rho$), we obtain that the ${\rho }^{\mathrm{th}}$ moment of the number of randomized guesses, with $\rho \in \mathbb{N}$, is equal to

$$\begin{array}{cc}\hfill \mathbb{E}\{G^{\rho }|x\}& =\sum _{j=0}^{\rho }\left(\genfrac{}{}{0pt}{}{\rho }{j}\right)\mathbb{E}\{{\left(G-1\right)}^j|x\}\hfill \\ \hfill & =\sum _{j=0}^{\rho }\left(\genfrac{}{}{0pt}{}{\rho }{j}\right){\alpha }_j\hfill \\ \hfill & =1+\tilde{P}(x)\sum _{j=1}^{\rho }\left\{\left(\genfrac{}{}{0pt}{}{\rho }{j}\right){\mathrm{Li}}_{-j}(1-\tilde{P}(x))\right\},\hfill \end{array}$$

(26)

where (26) follows from (20) and since ${\alpha }_0=1$. By averaging over X,

$$\begin{array}{c}\hfill \mathbb{E}\{G^{\rho }\}=1+\sum _{x\in cX}\left\{P(x)\tilde{P}(x)\sum _{j=1}^{\rho }\left\{\left(\genfrac{}{}{0pt}{}{\rho }{j}\right){\mathrm{Li}}_{-j}(1-\tilde{P}(x))\right\}\right\}.\end{array}$$

(27)

To conclude, (19) and its simplification in (23) for $\rho \in (0,1)$ give calculable one-dimensional integral expressions for the ${\rho }^{\mathrm{th}}$ guessing moment with any $\rho >0$. This refers to a randomized guessing strategy whose practical advantages were further explained in [33,34]. This avoids the need for numerical calculations of infinite sums. A further simplification for $\rho \in \mathbb{N}$ is provided in (26) and (27), expressed in closed form as a function of polylogarithms.

3.2. Moments of Estimation Errors

Let $X_1,\dots ,X_n$ be i.i.d. random variables with an unknown expectation $\theta$ to be estimated, and consider the simple estimator,

$$\begin{array}{c}\hfill {\widehat{\theta }}_n=\frac{1}{n}\sum _{i=1}^n{X}_i.\end{array}$$

(28)

For given $\rho >0$, we next derive an easily-calculable expression of the ${\rho }^{\mathrm{th}}$ moment of the estimation error.

Let $D_n:={({\widehat{\theta }}_n-\theta )}^2$ and ${\rho }^{\prime }:=\frac{\rho }{2}$. By Theorem 1, if $\rho >0$ is a non-integral multiple of two, then

$$\begin{array}{cc}\hfill \mathbb{E}\left\{{\left|{\widehat{\theta }}_n-\theta \right|}^{\rho }\right\}& =\mathbb{E}\left\{D_n^{{\rho }^{\prime }}\right\}\hfill \\ \hfill & =\frac{2}{2+\rho }\sum _{\ell =0}^{\lfloor \rho /2\rfloor }\frac{{\alpha }_{\ell }}{B\left(\ell +1,\rho /2+1-\ell \right)}\hfill \end{array}$$

(29)

$$\begin{array}{cc}& +\frac{\rho }{2\pi }sin\left(\frac{\pi \rho }{2}\right)\mathsf{\Gamma }\left(\frac{\rho }{2}\right){\int }_0^{\infty }\frac{1}{u^{\rho /2+1}}\left(\sum _{j=0}^{\lfloor \rho /2\rfloor }\left\{\frac{{(-1)}^j{\alpha }_j}{j!}{u}^j\right\}{e}^{-u}-M_{D_n}(-u)\right)\mathrm{d}u,\hfill \end{array}$$

(30)

where

$$\begin{array}{c}\hfill M_{D_n}(-u)=\mathbb{E}\{exp(-u{({\widehat{\theta }}_n-\theta )}^2)\},\forall u\ge 0,\end{array}$$

(31)

${\alpha }_0:=1$, and for all $j\in \mathbb{N}$ (see (11))

$$\begin{array}{c}\hfill {\alpha }_j=\frac{1}{j+1}\sum _{\ell =0}^j\frac{{(-1)}^{j-\ell }{M}_{D_n}^{(\ell )}(0)}{B(\ell +1,j-\ell +1)}.\end{array}$$

(32)

By Corollary 1 and (29), if in particular $\rho \in (0,2)$, then the right-hand side of (30) is simplified to

$$\begin{array}{c}\hfill \mathbb{E}\{{|{\widehat{\theta }}_n-\theta |}^{\rho }\}=1+\frac{\rho }{2\mathsf{\Gamma }(1-\frac{1}{2}\rho )}{\int }_0^{\infty }{u}^{-(1+\frac{1}{2}\rho )}\left[e^{-u}-M_{D_n}(-u)\right]\mathrm{d}u,\end{array}$$

(33)

and, for all $k\in \mathbb{N}$,

$$\begin{array}{cc}\hfill & \mathbb{E}\{|{\widehat{\theta }}_n{-\theta |}^{2k}\}=M_{D_n}^{(k)}(0).\hfill \end{array}$$

(34)

In view of (29)–(34), obtaining a closed-form expression for the ${\rho }^{\mathrm{th}}$ moment of the estimation error, for an arbitrary $\rho >0$, hinges on the calculation of the right side of (31) for all $u\ge 0$. To this end, we invoke the identity

$$\begin{array}{c}\hfill e^{-u{z}^2}=\frac{1}{2\sqrt{\pi u}}{\int }_{-\infty }^{\infty }{e}^{-j\omega z-{\omega }^2/(4u)}\mathrm{d}\omega ,\forall u>0,z\in \mathbb{R},\end{array}$$

(35)

which is the MGF of a zero-mean Gaussian random variable with variance $\frac{1}{2u}$. Together with (31), it gives (see Appendix B.1)

$$\begin{array}{c}\hfill M_{D_n}(-u)=\frac{1}{2\sqrt{\pi u}}{\int }_{-\infty }^{\infty }{e}^{-j\omega \theta }{\varphi }_X^n\left(\frac{\omega }{n}\right)e^{-{\omega }^2/(4u)}\mathrm{d}\omega ,\forall u>0,\end{array}$$

(36)

where X is a generic random variable with the same distribution as $X_i$ for all i.

The combination of (30)–(34) enables calculating exactly the ${\rho }^{\mathrm{th}}$ moment $\mathbb{E}\{|{\widehat{\theta }}_n{-\theta |}^{\rho }\}$, for any given $\rho >0$, in terms of a two-dimensional integral. Combining (33) and (36) yields, for all $\rho \in (0,2)$,

$$\begin{array}{cc}\hfill & \mathbb{E}\{{|{\widehat{\theta }}_n-\theta |}^{\rho }\}\hfill \\ \hfill & =1+\frac{\rho }{2\mathsf{\Gamma }(1-\frac{1}{2}\rho )}{\int }_0^{\infty }{\int }_{-\infty }^{\infty }{u}^{-(\rho /2+1)}\left[\frac{1}{2}{e}^{-u-\left|\omega \right|}-\frac{1}{2\sqrt{\pi u}}{\varphi }_X^n\left(\frac{\omega }{n}\right)e^{-j\omega \theta -{\omega }^2/(4u)}\right]\mathrm{d}\omega \mathrm{d}u,\hfill \end{array}$$

(37)

where we have used the identity ${\int }_{-\infty }^{\infty }\frac{1}{2}{e}^{-\left|\omega \right|}\mathrm{d}\omega =1$ in the derivation of the first term of the integral on the right-hand side of (37).

As an example, consider the case where ${\{X_i\}}_{i=1}^n$ are i.i.d. Bernoulli random variables with

$$\begin{array}{c}\hfill \mathbb{P}\{X_1=1\}=\theta ,\mathbb{P}\{X_1=0\}=1-\theta \end{array}$$

(38)

where the characteristic function is given by

$$\begin{array}{c}\hfill {\varphi }_X(u):=\mathbb{E}\left\{e^{juX}\right\}=1+\theta \left(e^{ju}-1\right),u\in \mathbb{R}.\end{array}$$

(39)

Thanks to the availability of the exact expression, we can next compare the exact ${\rho }^{\mathrm{th}}$ moment of the estimation error $|{\widehat{\theta }}_n-\theta |$, with the following closed-form upper bound (see Appendix B.2) and thereby assess its tightness:

$$\begin{array}{c}\hfill \mathbb{E}\{{|{\widehat{\theta }}_n-\theta |}^{\rho }\}\le K(\rho ,\theta )·n^{-\rho /2},\end{array}$$

(40)

which holds for all $n\in \mathbb{N}$, $\rho >0$, and $\theta \in [0,1]$, with

$$\begin{array}{c}\hfill K(\rho ,\theta ):=\rho \mathsf{\Gamma }\left(\frac{\rho }{2}\right){\left(2\theta (1-\theta )\right)}^{\rho /2}.\end{array}$$

(41)

Figure 1 and Figure 2 display plots of $\mathbb{E}|{\widehat{\theta }}_n-\theta |$ as a function of $\theta$ and n, in comparison to the upper bound (40). The difference in the plot of Figure 1 is significant except for the boundaries of the interval $[0,1]$, where both the exact value and the bound vanish. Figure 2 indicates that the exact value of $\mathbb{E}|{\widehat{\theta }}_n-\theta |$, for large n, scales like $\sqrt{n}$; this is reflected by the apparent parallelism of the curves in both graphs and by the upper bound (40).

Figure 1. $\mathbb{E}\left|{\widehat{\theta }}_n-\theta \right|$ (see (37) and (39)) versus its upper bound in (40) as functions of $\theta \in [0,1]$ with $n=1000$.

Figure 2. $\mathbb{E}\left|{\widehat{\theta }}_n-\theta \right|$ (see (37) and (39)) versus its upper bound in (40) as functions of n with $\theta =\frac{1}{4}$.

To conclude, this subsection provides an exact, double-integral expression for the ${\rho }^{\mathrm{th}}$ moment of the estimation error of the expectation of n i.i.d. random variables. In other words, the dimension of the integral does not increase with n, and it is a calculable expression. We further compare our expression with an upper bound that stems from concentration inequalities. Although the scaling of the bound as a polynomial of n is correct, the difference between the exact expression and the bound is significant (see Figure 1 and Figure 2).

3.3. Rényi Entropy of Extended Multivariate Cauchy Distributions

Generalized Cauchy distributions, their mathematical properties, and applications are of interest (see, e.g., [6,35,36,37]). The Shannon differential entropy of a family of generalized Cauchy distributions was derived in Proposition 1 in [36], and also, a lower bound on the differential entropy of a family of extended multivariate Cauchy distributions (cf. Equation (42) in [37]) was derived in Theorem 6 in [37]. Furthermore, an exact single-letter expression for the differential entropy of the different family of extended multivariate Cauchy distributions was recently derived in Section 3.1 in [6]. Motivated by these studies, as well as the various information-theoretic applications of Rényi information measures, we apply Theorem 1 to obtain the Rényi (differential) entropy of an arbitrary positive order $\alpha$ for the extended multivariate Cauchy distributions in Section 3.1 in [6]. As we shall see in this subsection, the integral representation for the Rényi entropy of the latter family of extended multivariate Cauchy distributions is two-dimensional, irrespective of the dimension n of the random vector.

Let $X^n=(X_1,\dots ,X_n)$ be a random vector whose probability density function is of the form

$$\begin{array}{c}\hfill f(x^n)=\frac{C_n}{{\left[1+{\sum }_{i=1}^{n}g(x_i)\right]}^q},x^n=(x_1,\dots ,x_n)\in {\mathbb{R}}^n,\end{array}$$

(42)

for a certain function $g:\mathbb{R}\to [0,\infty )$ and a positive constant q such that

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^n}\frac{1}{{\left[1+{\sum }_{i=1}^{n}g(x_i)\right]}^q}\mathrm{d}{x}^n<\infty .\end{array}$$

(43)

We refer to this kind of density (see also Section 3.1 in [6]) as a generalized multivariate Cauchy density because the multivariate Cauchy density function is the special case pertaining to the choices $g(x)=x^2$ and $q=\frac{1}{2}(n+1)$. The differential Shannon entropy of the generalized multivariate Cauchy density was derived in Section 3.1 in [6] using the integral representation of the logarithm (1), where it was presented as a two-dimensional integral.

We next extend the analysis of [6] to differential Rényi entropies of an arbitrary positive order $\alpha$ (recall that the differential Rényi entropy is specialized to the differential Shannon entropy at $\alpha =1$ [38]). We show that, for the generalized multivariate Cauchy density, the differential Rényi entropy can be presented as a two-dimensional integral, rather than an n-dimensional integral. Defining

$$\begin{array}{c}\hfill Z(t):={\int }_{-\infty }^{\infty }{e}^{-tg(x)}\mathrm{d}x,t>0,\end{array}$$

(44)

we get from (42) (see Section 3.1 in [6]) that

$$\begin{array}{c}\hfill C_n=\frac{\mathsf{\Gamma }(q)}{{\displaystyle {\int }_0^{\infty }{t}^{q-1}{e}^{-t}{Z}^n(t)\mathrm{d}t}}.\end{array}$$

(45)

For $g(x)={\left|x\right|}^{\theta }$, with a fixed $\theta >0$, (44) implies that

$$\begin{array}{c}\hfill Z(t)=\frac{2\mathsf{\Gamma }(1/\theta )}{\theta t^{1/\theta }}.\end{array}$$

(46)

In particular, for $\theta =2$ and $q=\frac{1}{2}(n+1)$, we get the multivariate Cauchy density from (42). In this case, it follows from (46) that $Z(t)=\sqrt{\frac{\pi }{t}}$ for $t>0$, and from (45)

$$\begin{array}{c}\hfill C_n=\frac{\mathsf{\Gamma }\left(\frac{n+1}{2}\right)}{{\pi }^{(n+1)/2}}.\end{array}$$

(47)

For $\alpha \in (0,1)\cup (1,\infty )$, the (differential) Rényi entropy of order $\alpha$ is given by

$$\begin{array}{cc}\hfill h_{\alpha }(X^n)& :=\frac{1}{1-\alpha }log{\int }_{{\mathbb{R}}^n}{f}^{\alpha }(x^n)\mathrm{d}{x}^n\hfill \\ \hfill & =\frac{1}{1-\alpha }log\mathbb{E}\left[f^{\alpha -1}(X^n)\right].\hfill \end{array}$$

(48)

Using the Laplace transform relation,

$$\begin{array}{c}\hfill \frac{1}{s^q}=\frac{1}{\mathsf{\Gamma }(q)}{\int }_0^{\infty }{t}^{q-1}{e}^{-st}\mathrm{d}t,\forall q>0,\mathrm{Re}(s)>0,\end{array}$$

(49)

we obtain that, for $\alpha >1$ (see Appendix C),

$$\begin{array}{cc}\hfill h_{\alpha }(X^n)& =\frac{\alpha }{\alpha -1}log{\int }_0^{\infty }{t}^{q-1}{e}^{-t}{Z}^n(t)\mathrm{d}t+\frac{log\mathsf{\Gamma }\left(q(\alpha -1)\right)}{\alpha -1}-log\mathsf{\Gamma }(q)\hfill \\ \hfill & -\frac{1}{\alpha -1}log{\int }_0^{\infty }{\int }_0^{\infty }{t}^{q(\alpha -1)-1}{u}^{q-1}{e}^{-(t+u)}{Z}^n(t+u)\mathrm{d}u\mathrm{d}t.\hfill \end{array}$$

(50)

If $\alpha \in (0,1)$, we distinguish between the following two cases:

(a)

If $\alpha =1-\frac{m}{q}$ for some $m\in \{1,\dots ,q-1\}$, then

$$\begin{array}{c}\hfill h_{\alpha }(X^n)=\frac{\alpha log{C}_n}{1-\alpha }-\frac{log\mathsf{\Gamma }(q)}{1-\alpha }+\frac{1}{1-\alpha }log\left(\sum _{\ell =0}^m\left\{{(-1)}^{m-\ell }{\int }_0^{\infty }{t}^{q-1}{e}^{-t}{\phi }_n^{(\ell )}(t)\mathrm{d}t\right\}\right),\end{array}$$

(51)

with

$$\begin{array}{c}\hfill {\phi }_n(t):=Z^n(t),\forall t\ge 0.\end{array}$$

(52)

(b)

Otherwise (i.e., if $\rho :=q(1-\alpha )\notin \mathbb{N}$), then

$$\begin{array}{cc}\hfill h_{\alpha }(X^n)& =-log{C}_n+\frac{1}{1-\alpha }log(\frac{1}{1+\rho }\sum _{\ell =0}^{\lfloor \rho \rfloor }\frac{{\beta }_{\ell }(n)}{B(\ell +1,\rho +1-\ell )}+\frac{\rho sin(\pi \rho )\mathsf{\Gamma }(\rho )}{\pi }\hfill \\ \hfill & ·{\int }_0^{\infty }\frac{e^{-u}}{u^{\rho +1}}\left(\sum _{j=0}^{\lfloor \rho \rfloor }\left\{\frac{{(-1)}^j{\beta }_j(n)}{j!}{u}^j\right\}-\frac{C_n}{\mathsf{\Gamma }(q)}{\int }_0^{\infty }{t}^{q-1}{e}^{-t}{Z}^n(t+u)\mathrm{d}t\right)\mathrm{d}u),\hfill \end{array}$$

(53)

where ${\beta }_0:=1$, and for all $j\in \mathbb{N}$,

$$\begin{array}{c}\hfill {\beta }_j(n):=\frac{C_n}{\mathsf{\Gamma }(q)}\sum _{\ell =0}^j\left\{\frac{{(-1)}^{j-\ell }}{B(\ell +1,j-\ell +1)}\sum _{k=0}^{\ell }\left\{{(-1)}^{\ell -k}\left(\genfrac{}{}{0pt}{}{\ell }{k}\right){\int }_0^{\infty }{t}^{q-1}{e}^{-t}{\phi }_n^{(k)}(t)\mathrm{d}t\right\}\right\}.\end{array}$$

(54)

The proof of the integral expressions of the Rényi entropy of order $\alpha \in (0,1)$, as given in (50)–(54), is provided in Appendix C.

Once again, the advantage of these expressions, which do not seem to be very simple (at least on the face of it), is that they only involve one- or two-dimensional integrals, rather than an expression of an n-dimensional integral (as it could have been in the case of an n-dimensional density).

3.4. Mutual Information Calculations for Communication Channels with Jamming

Consider a channel that is fed by an input vector $X^n=(X_1,\dots ,X_n)\in {\mathcal{X}}^n$ and generates an output vector $Y^n=(Y_1,\dots ,Y_n)\in {\mathcal{Y}}^n$, where $\mathcal{X}$ and $\mathcal{Y}$ are either finite, countably infinite, or continuous alphabets, and ${\mathcal{X}}^n$ and ${\mathcal{Y}}^n$ are their $n^{\mathrm{th}}$ order Cartesian powers. Let the conditional probability distribution of the channel be given by

$$\begin{array}{c}\hfill p_{Y^n|X^n}(y^n|x^n)=\frac{1}{n}\sum _{i=1}^n\left\{\prod _{j\ne i}{q}_{Y|X}(y_j|x_j)r_{Y|X}(y_i|x_i)\right\},\end{array}$$

(55)

where $r_{Y|X}(·|·)$ and $q_{Y|X}(·|·)$ are given conditional probability distributions of Y given X, $x^n=(x_1,\dots ,x_n)\in {\mathcal{X}}^n$ and $y^n=(y_1,\dots ,y_n)\in {\mathcal{Y}}^n$. This channel model refers to a discrete memoryless channel (DMC), which is nominally given by

$$\begin{array}{c}\hfill q_{Y^n|X^n}(y^n|x^n)=\prod _{i=1}^n{q}_{Y|X}(y_i|x_i),\end{array}$$

(56)

where one of the transmitted symbols is jammed at a uniformly distributed random time, i, and the transition distribution of the jammed symbol is given by $r_{Y|X}(y_i|x_i)$ instead of $q_{Y|X}(y_i|x_i)$. The restriction to a single jammed symbol is made merely for the sake of simplicity, but it can easily be extended.

We wish to evaluate how the jamming affects the mutual information $I(X^n;Y^n)$. Clearly, when one talks about jamming, the mutual information is decreased, but this is not part of the mathematical model, where the relation between r and q has not been specified. Let the input distribution be given by the product form

$$\begin{array}{c}\hfill p_{X^n}(x^n)=\prod _{i=1}^n{p}_X(x_i),x^n\in {\mathcal{X}}^n.\end{array}$$

(57)

The mutual information (in nats) is given by

$$\begin{array}{ll}\hfill I(X^n;Y^n)& =h(Y^n)-h(Y^n|X^n)\hfill \end{array}$$

(58)

$$\begin{array}{ll}\hfill & ={\int }_{{\mathcal{X}}^n\times {\mathcal{Y}}^n}{p}_{X^n,Y^n}(x^n,y^n)ln{p}_{Y^n|X^n}(y^n|x^n)\mathrm{d}{x}^n\mathrm{d}{y}^n-{\int }_{{\mathcal{Y}}^n}{p}_{Y^n}(y^n)ln{p}_{Y^n}(y^n)\mathrm{d}{y}^n.\hfill \end{array}$$

(59)

For the simplicity of notation, we henceforth omit the domains of integration whenever they are clear from the context. We have,

$$\begin{array}{cc}\hfill & \int p_{X^n,Y^n}(x^n,y^n)ln{p}_{Y^n|X^n}(y^n|x^n)\mathrm{d}{x}^n\mathrm{d}{y}^n\hfill \\ \hfill & =\int p_{X^n,Y^n}(x^n,y^n)ln\left(\frac{p_{Y^n|X^n}(y^n|x^n)}{q_{Y^n|X^n}(y^n|x^n)}\right)\mathrm{d}{x}^n\mathrm{d}{y}^n+\int p_{X^n,Y^n}(x^n,y^n)ln{q}_{Y^n|X^n}(y^n|x^n)\mathrm{d}{x}^n\mathrm{d}{y}^n.\hfill \end{array}$$

(60)

By using the logarithmic expectation in (15) and the following equality (see (55) and (56)):

$$\begin{array}{c}\hfill \frac{p_{Y^n|X^n}(y^n|x^n)}{q_{Y^n|X^n}(y^n|x^n)}=\frac{1}{n}\sum _{i=1}^n\frac{r_{Y|X}(y_i|x_i)}{q_{Y|X}(y_i|x_i)},\end{array}$$

(61)

we obtain (see Appendix D.1)

$$\begin{array}{c}\hfill \int p_{X^n,Y^n}(x^n,y^n)ln\left(\frac{p_{Y^n|X^n}(y^n|x^n)}{q_{Y^n|X^n}(y^n|x^n)}\right)\mathrm{d}{x}^n\mathrm{d}{y}^n={\int }_0^{\infty }\frac{1}{u}\left[e^{-u}-f^{n-1}\left(\frac{u}{n}\right)g\left(\frac{u}{n}\right)\right]\mathrm{d}u,\end{array}$$

(62)

where, for $u\ge 0$,

$$\begin{array}{c}\hfill f(u):=\int p_X(x)q_{Y|X}(y|x)exp\left(-\frac{u{r}_{Y|X}(y|x)}{q_{Y|X}(y|x)}\right)\mathrm{d}x\mathrm{d}y,\end{array}$$

(63)

$$\begin{array}{c}\hfill g(u):=\int p_X(x)r_{Y|X}(y|x)exp\left(-\frac{u{r}_{Y|X}(y|x)}{q_{Y|X}(y|x)}\right)\mathrm{d}x\mathrm{d}y.\end{array}$$

(64)

Moreover, owing to the product form of $q_n$, it is shown in Appendix D.2 that

$$\begin{array}{cc}\hfill & \int p_{X^n,Y^n}(x^n,y^n)ln{q}_{Y^n|X^n}(y^n|x^n)\mathrm{d}{x}^n\mathrm{d}{y}^n\hfill \\ \hfill & =\int p_X(x)r_{Y|X}(y|x)ln{q}_{Y|X}(y|x)\mathrm{d}x\mathrm{d}y+(n-1)\int p_X(x)q_{Y|X}(y|x)ln{q}_{Y|X}(y|x)\mathrm{d}x\mathrm{d}y.\hfill \end{array}$$

(65)

Combining (60), (62), and (65), we express $h(Y^n|X^n)$ as a double integral over $\mathcal{X}\times \mathcal{Y}$, independently of n (rather than an integration over ${\mathcal{X}}^n\times {\mathcal{Y}}^n$):

$$\begin{array}{cc}\hfill h(Y^n|X^n)& ={\int }_0^{\infty }\frac{1}{u}\left[f^{n-1}\left(\frac{u}{n}\right)g\left(\frac{u}{n}\right)-e^{-u}\right]\mathrm{d}u-\int p_X(x)r_{Y|X}(y|x)ln{q}_{Y|X}(y|x)\mathrm{d}x\mathrm{d}y\hfill \\ \hfill & -(n-1)\int p_X(x)q_{Y|X}(y|x)ln{q}_{Y|X}(y|x)\mathrm{d}x\mathrm{d}y.\hfill \end{array}$$

(66)

We next calculate the differential channel output entropy, $h(Y^n)$, induced by $p_{Y^n|X^n}(·|·)$. From Appendix D.3,

$$\begin{array}{c}\hfill p_{Y^n}(y^n)=\prod _{j=1}^{n}v(y_j)·\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)},\end{array}$$

(67)

where, for all $y\in \mathcal{Y}$,

$$\begin{array}{c}\hfill v(y):=\int q_{Y|X}(y|x)p_X(x)\mathrm{d}x,\end{array}$$

(68)

$$\begin{array}{c}\hfill w(y):=\int r_{Y|X}(y|x)p_X(x)\mathrm{d}x.\end{array}$$

(69)

By (1), the following identity holds for every positive random variable Z (see Appendix D.3):

$$\begin{array}{c}\hfill \mathbb{E}\{ZlnZ\}={\int }_0^{\infty }\frac{1}{u}\left[M_Z^{\prime }(0)e^{-u}-M_Z^{\prime }(-u)\right]\mathrm{d}u\end{array}$$

(70)

where $M_Z(u):=\mathbb{E}\{e^{uZ}\}$. By setting $Z:=\frac{1}{n}{\sum }_{i=1}^n\frac{w(V_i)}{v(V_i)}$ where ${\{V_i\}}_{i=1}^n$ are i.i.d. random variables with the density function v, some algebraic manipulations give (see Appendix D.3)

$$\begin{array}{cc}\hfill h(Y^n)& ={\int }_0^{\infty }\frac{1}{u}\left[t^{n-1}\left(\frac{u}{n}\right)s\left(\frac{u}{n}\right)-e^{-u}\right]\mathrm{d}u-\int w(y)lnv(y)\mathrm{d}y-(n-1)\int v(y)lnv(y)\mathrm{d}y,\hfill \end{array}$$

(71)

where

$$\begin{array}{c}\hfill s(u):=\int w(y)exp\left(-\frac{uw(y)}{v(y)}\right)\mathrm{d}y,u\ge 0,\end{array}$$

(72)

$$\begin{array}{c}\hfill t(u):=\int v(y)exp\left(-\frac{uw(y)}{v(y)}\right)\mathrm{d}y,u\ge 0.\end{array}$$

(73)

Combining (58), (66), and (71), we obtain the mutual information for the channel with jamming, which is given by

$$\begin{array}{cc}\hfill I_p(X^n;Y^n)& ={\int }_0^{\infty }\frac{1}{u}\left[t^{n-1}\left(\frac{u}{n}\right)s\left(\frac{u}{n}\right)-f^{n-1}\left(\frac{u}{n}\right)g\left(\frac{u}{n}\right)\right]\mathrm{d}u\hfill \\ \hfill & +\int p_X(x)r_{Y|X}(y|x)ln{q}_{Y|X}(y|x)\mathrm{d}x\mathrm{d}y-\int w(y)lnv(y)\mathrm{d}y\hfill \\ \hfill & +(n-1)\left[\int p_X(x)q_{Y|X}(y|x)ln{q}_{Y|X}(y|x)\mathrm{d}x\mathrm{d}y-\int v(y)lnv(y)\mathrm{d}y\right].\hfill \end{array}$$

(74)

We next exemplify our results in the case where q is a binary symmetric channel (BSC) with crossover probability $\delta \in (0,\frac{1}{2})$ and p is a BSC with a larger crossover probability, $\epsilon \in (\delta ,\frac{1}{2}]$. We assume that the input bits are i.i.d. and equiprobable. The specialization of our analysis to this setup is provided in Appendix D.4, showing that the mutual information of the channel $p_{X^n,Y^n}$, fed by the binary symmetric source, is given by

$$\begin{array}{cc}\hfill I_p(X^n;Y^n)& =nln2-d(\epsilon \parallel \delta )-H_{\mathrm{b}}(\epsilon )-(n-1)H_{\mathrm{b}}(\delta )\hfill \\ \hfill & +{\int }_0^{\infty }\{e^{-u}-{\left[(1-\delta )exp\left(-\frac{(1-\epsilon )u}{(1-\delta )n}\right)+\delta exp\left(-\frac{\epsilon u}{\delta n}\right)\right]}^{n-1}\hfill \\ \hfill & ·\left[(1-\epsilon )exp\left(-\frac{(1-\epsilon )u}{(1-\delta )n}\right)+\epsilon exp\left(-\frac{\epsilon u}{\delta n}\right)\right]\}\frac{\mathrm{d}u}{u},\hfill \end{array}$$

(75)

where $H_{\mathrm{b}}:[0,1]\to [0,ln2]$ is the binary entropy function

$$\begin{array}{c}\hfill H_{\mathrm{b}}(x):=-xln(x)-(1-x)ln(1-x),x\in [0,1],\end{array}$$

(76)

with the convention that $0ln0=0$, and

$$\begin{array}{c}\hfill d(\epsilon \parallel \delta ):=\epsilon ln\left(\frac{\epsilon }{\delta }\right)+(1-\epsilon )ln\left(\frac{1-\epsilon }{1-\delta }\right),(\delta ,\epsilon )\in {[0,1]}^2\end{array}$$

(77)

denotes the binary relative entropy. By the data processing inequality, the mutual information in (75) is smaller than that of the BSC with crossover probability $\delta$:

$$\begin{array}{cc}\hfill I_q(X^n;Y^n)& =n\left(ln2-H_{\mathrm{b}}(\delta )\right).\hfill \end{array}$$

(78)

Figure 3 refers to the case where $\delta ={10}^{-3}$ and $n=128$. Here, $I_q(X^n;Y^n)=87.71\mathrm{nats}$, and $I_p(X^n;Y^n)$ is decreased by 2.88 nats due to the jammer (see Figure 3).

Figure 3. The degradation in mutual information for $n=128$. The jammer-free channel q is a binary symmetric channel (BSC) with crossover probability $\delta ={10}^{-3}$, and r is a BSC with crossover probability $\epsilon \in \left(\delta ,\frac{1}{2}\right]$. The input bits are i.i.d. and equiprobable. The degradation in $I(X^n;Y^n)$ (nats) is displayed as a function of $\epsilon$.

Figure 4 refers to the case where $\delta ={10}^{-3}$ and $\epsilon =\frac{1}{2}$ (referring to complete jamming of a single symbol, which is chosen uniformly at random), and it shows the difference in the mutual information $I(X^n;Y^n)$, as a function of the length n, between the jamming-free BSC with crossover probability $\delta$ and the channel with jamming.

Figure 4. The degradation in mutual information as a function of n. The jammer-free channel $q_{Y|X}$ is a BSC with crossover probability $\delta ={10}^{-3}$, and $r_{Y|X}$ for the jammed symbol is a BSC with crossover probability $\epsilon =\frac{1}{2}$. The input bits are i.i.d. and equiprobable.

To conclude, this subsection studies the change in the mutual information $I(X^n;Y^n)$ due to jamming, relative to the mutual information associated with the nominal channel without jamming. Due to the integral representations provided in our analysis, the calculation of the mutual information finally depends on one-dimensional integrals, as opposed to the original n-dimensional integrals, pertaining to the expressions that define the associated differential entropies.

Appendix B.1. Proof of Equation (36)

For all $u>0$,

$$\begin{array}{cc}\hfill M_{D_n}(-u)& =\mathbb{E}\left\{exp\left(-u{\left({\widehat{\theta }}_n-\theta \right)}^2\right)\right\}\hfill \end{array}$$

(A24)

$$\begin{array}{cc}\hfill & =\mathbb{E}\left\{\frac{1}{2\sqrt{\pi u}}{\int }_{-\infty }^{\infty }{e}^{j\omega ({\widehat{\theta }}_n-\theta )}{e}^{-{\omega }^2/(4u)}\mathrm{d}\omega \right\}\hfill \end{array}$$

(A25)

$$\begin{array}{cc}\hfill & =\frac{1}{2\sqrt{\pi u}}{\int }_{-\infty }^{\infty }{e}^{-j\omega \theta }\mathbb{E}\{e^{j\omega {\widehat{\theta }}_n}\}{e}^{-{\omega }^2/(4u)}\mathrm{d}\omega \hfill \end{array}$$

(A26)

$$\begin{array}{cc}\hfill & =\frac{1}{2\sqrt{\pi u}}{\int }_{-\infty }^{\infty }{e}^{-j\omega \theta }\mathbb{E}\left\{exp\left(\frac{j\omega }{n}\sum _{i=1}^n{X}_i\right)\right\}{e}^{-{\omega }^2/(4u)}\mathrm{d}\omega \hfill \end{array}$$

(A27)

$$\begin{array}{cc}\hfill & =\frac{1}{2\sqrt{\pi u}}{\int }_{-\infty }^{\infty }{e}^{-j\omega \theta }{\varphi }_X^n\left(\frac{\omega }{n}\right)e^{-{\omega }^2/(4u)}\mathrm{d}\omega ,\hfill \end{array}$$

(A28)

where (A24) is (31); (A25) relies on (35); (A26) holds by interchanging expectation and integration (due to Fubini’s theorem); (A27) is due to (28), and (A28) holds by the assumption that $X_1,\dots ,X_n$ are i.i.d.

Appendix B.2. Derivation of the Upper Bound in (40)

For all $\rho >0$,

$$\begin{array}{cc}\hfill \mathbb{E}\{{|{\widehat{\theta }}_n-\theta |}^{\rho }\}& ={\int }_0^{\infty }\mathbb{P}({\left|{\widehat{\theta }}_n-\theta \right|}^{\rho }\ge t)\mathrm{d}t\hfill \\ \hfill & ={\int }_0^{\infty }\mathbb{P}({\left|{\widehat{\theta }}_n-\theta \right|}^{\rho }\ge {\epsilon }^{\rho })\rho {\epsilon }^{\rho -1}\mathrm{d}\epsilon \hfill \\ \hfill & ={\int }_0^{\infty }\mathbb{P}(\left|{\widehat{\theta }}_n-\theta \right|\ge \epsilon )\rho {\epsilon }^{\rho -1}\mathrm{d}\epsilon .\hfill \end{array}$$

(A29)

We next use the Chernoff bound for upper bounding $\mathbb{P}(\left|{\widehat{\theta }}_n-\theta \right|\ge \epsilon )$ for all $\epsilon >0$,

$$\begin{array}{cc}\hfill \mathbb{P}({\widehat{\theta }}_n-\theta \ge \epsilon )& =\mathbb{P}\left(\sum _{i=1}^n(X_i-\theta )\ge n\epsilon \right)\hfill \\ \hfill & \le \underset{s\ge 0}{inf}\left\{e^{-sn\epsilon }\mathbb{E}\left\{exp\left(s\sum _{i=1}^n(X_i-\theta )\right)\right\}\right\}\hfill \\ \hfill & =\underset{s\ge 0}{inf}\left\{e^{-sn\epsilon }\prod _{i=1}^n\mathbb{E}\left\{e^{s(X_i-\theta )}\right\}\right\}\hfill \\ \hfill & =\underset{s\ge 0}{inf}\left\{e^{-sn\epsilon }{\left(\theta e^{s(1-\theta )}+(1-\theta )e^{-s\theta }\right)}^n\right\}\hfill \\ \hfill & =\underset{s\ge 0}{inf}\left\{e^{-ns\epsilon +n{H}_{\theta }(s)}\right\}\hfill \end{array}$$

(A30)

with $\theta \in [0,1]$, and

$$\begin{array}{c}\hfill H_{\theta }(s):=ln\left(\theta e^{s(1-\theta )}+(1-\theta )e^{-s\theta }\right),s\ge 0.\end{array}$$

(A31)

We now use an upper bound on $H_{\theta }(s)$ for every $s\ge 0$. By Theorem 3.2 and Lemma 3.3 in [39] (see also Lemma 2.4.6 in [40]), we have

$$\begin{array}{c}\hfill H_{\theta }(s)\le C(\theta )s^2\end{array}$$

(A32)

with

$$C(\theta ):=\{\begin{array}{cc}0,& \mathrm{if}\theta =0,\hfill \\ \frac{1-2\theta }{4ln\left(\frac{1-\theta }{\theta }\right)},\hfill & \mathrm{if}\theta \in \left(0,\frac{1}{2}\right),\hfill \\ \frac{1}{2}\theta (1-\theta ),\hfill & \mathrm{if}\theta \in \left[\frac{1}{2},1\right].\hfill \end{array}$$

(A33)

Combining (A30) and (A32) yields

$$\begin{array}{cc}\hfill \mathbb{P}({\widehat{\theta }}_n-\theta \ge \epsilon )& \le \underset{s\ge 0}{inf}\left\{e^{-n\epsilon s+nC(\theta )s^2}\right\}\hfill \\ \hfill & =exp\left(-\frac{n{\epsilon }^2}{4C(\theta )}\right).\hfill \end{array}$$

(A34)

Similarly, it is easy to show that the same Chernoff bound applies also to $\mathbb{P}({\widehat{\theta }}_n-\theta \le -\epsilon )$, which overall gives

$$\begin{array}{c}\hfill \mathbb{P}(\left|{\widehat{\theta }}_n-\theta \right|\ge \epsilon )\le 2exp\left(-\frac{n{\epsilon }^2}{4C(\theta )}\right).\end{array}$$

(A35)

Inequality (A35) is a refined version of Hoeffding’s inequality (see Section 2.4.4 in [40]), which is derived for the Bernoulli distribution (see (A30)) and by invoking the Chernoff bound; moreover, (A35) coincides with Hoeffding’s inequality in the special case $\theta =\frac{1}{2}$ (which, from (A33), yields $C(\theta )=\frac{1}{8}$). In view of the fact that (A35) forms a specialization of Theorem 2.4.7 in [40], it follows that the Bernoulli case is the worst one (in the sense of leading to the looser upper bound) among all probability distributions whose support is the interval $[0,1]$ and whose expected value is $\theta \in [0,1]$. However, in the Bernoulli case, a simple symmetry argument applies for improving the bound (A35) as follows. Since $\{X_i\}$ are i.i.d., Bernoulli with mean $\theta$, then obviously, $\{1-X_i\}$ are Bernoulli, i.i.d. with mean $1-\theta$ and (from (28))

$$\begin{array}{c}\hfill {\widehat{\theta }}_n(1-X_1,\dots ,1-X_n)=1-{\widehat{\theta }}_n(X_1,\dots ,X_n),\end{array}$$

(A36)

which implies that the error estimation is identical in both cases. Hence, $\mathbb{P}(|{\widehat{\theta }}_n-\theta |\ge \epsilon )$ is symmetric around $\theta =\frac{1}{2}$. It can be verified that

$$\begin{array}{c}\hfill min\left\{C(\theta ),C(1-\theta )\right\}=\frac{1}{2}\theta (1-\theta ),\forall \theta \in [0,1],\end{array}$$

(A37)

which follows from (A33) and since $C(\theta )>C(1-\theta )$ for all $\theta \in (0,\frac{1}{2})$ (see Figure 2.1 in [40]). In view of (A37) and the above symmetry consideration, the upper bound in (A35) is improved for values of $\theta \in (0,\frac{1}{2})$, which therefore gives

$$\begin{array}{c}\hfill \mathbb{P})|{\widehat{\theta }}_n-\theta |\ge \epsilon )\le 2exp\left(-\frac{n{\epsilon }^2}{2\theta (1-\theta )}\right),\forall \theta \in [0,1],\epsilon >0.\end{array}$$

(A38)

From (28), the probability in (A38) vanishes if $\theta =0$ or $\theta =1$. Consequently, for $\rho >0$,

$$\begin{array}{cc}\hfill \mathbb{E}\left\{{\left|{\widehat{\theta }}_n-\theta \right|}^{\rho }\right\}& ={\int }_0^{\infty }\mathbb{P}\left(\left|{\widehat{\theta }}_n-\theta \right|\ge \epsilon \right)\rho {\epsilon }^{\rho -1}\mathrm{d}\epsilon \hfill \end{array}$$

(A39)

$$\begin{array}{cc}\hfill & \le {\int }_0^{\infty }2exp\left(-\frac{n{\epsilon }^2}{2\theta (1-\theta )}\right)\rho {\epsilon }^{\rho -1}\mathrm{d}\epsilon \hfill \end{array}$$

(A40)

$$\begin{array}{cc}\hfill & =\rho {\left(2\theta (1-\theta )\right)}^{\rho /2}{\int }_0^{\infty }{u}^{\rho /2-1}{e}^{-u}\mathrm{d}u·n^{-\rho /2}\hfill \end{array}$$

(A41)

$$\begin{array}{cc}\hfill & =\rho \mathsf{\Gamma }\left(\frac{\rho }{2}\right){\left(2\theta (1-\theta )\right)}^{\rho /2}·n^{-\rho /2}\hfill \end{array}$$

(A42)

$$\begin{array}{c}=K(\rho ,\theta )·n^{-\rho /2},\hfill \end{array}$$

(A43)

where (A39)–(A43) hold, respectively, due to (A29), (A38), the substitution $u:=\frac{n{\epsilon }^2}{2\theta (1-\theta )}$, (7), and (41).

Appendix D.1. Proof of Equations (62)–(64)

$$\begin{array}{cc}\hfill & \int p_{X^n,Y^n}(x^n,y^n)ln\left(\frac{p_{Y^n|X^n}(y^n|x^n)}{q_{Y^n|X^n}(y^n|x^n)}\right)\mathrm{d}{x}^n\mathrm{d}{y}^n\hfill \\ \hfill & =\int p_{X^n,Y^n}(x^n,y^n)ln\left(\frac{1}{n}\sum _{i=1}^n\frac{r_{Y|X}(y_i|x_i)}{q_{Y|X}(y_i|x_i)}\right)\mathrm{d}{x}^n\mathrm{d}{y}^n\hfill \end{array}$$

$$\begin{array}{cc}\hfill & ={\int }_0^{\infty }\frac{1}{u}\left[e^{-u}-\int p_{X^n,Y^n}(x^n,y^n)exp\left(-\frac{u}{n}\sum _{i=1}^n\frac{r_{Y|X}(y_i|x_i)}{q_{Y|X}(y_i|x_i)}\right)\mathrm{d}{x}^n\mathrm{d}{y}^n\right]\mathrm{d}u\hfill \end{array}$$

(A50)

$$={\int }_0^{\infty }\frac{1}{u}[e^{-u}-\int \frac{1}{n}\sum _{i=1}^n\left\{\prod _{j\ne i}{q}_{Y|X}(y_j|x_j)p_X(x_j)·r_{Y|X}(y_i|x_i)p_X(x_i)\right\}$$

$$\begin{array}{cc}\hfill & ·exp\left(-\frac{u}{n}\sum _{i=1}^n\frac{r_{Y|X}(y_i|x_i)}{q_{Y|X}(y_i|x_i)}\right)\mathrm{d}{x}^n\mathrm{d}{y}^n]\mathrm{d}u\hfill \end{array}$$

(A51)

$$={\int }_0^{\infty }\frac{1}{u}[e^{-u}-\int \frac{1}{n}\sum _{i=1}^n\{\prod _{j\ne i}{q}_{Y|X}(y_j|x_j)p_X(x_j)exp\left(-\frac{u}{n}\frac{r_{Y|X}(y_j|x_j)}{q_{Y|X}(y_j|x_j)}\right)$$

$$\begin{array}{cc}\hfill & ·r_{Y|X}(y_i|x_i)p_X(x_i)exp\left(-\frac{u}{n}\frac{r_{Y|X}(y_i|x_i)}{q_{Y|X}(y_i|x_i)}\right)\}\mathrm{d}{x}^n\mathrm{d}{y}^n]\mathrm{d}u\hfill \end{array}$$

(A52)

$$={\int }_0^{\infty }\frac{1}{u}[e^{-u}-\frac{1}{n}\sum _{i=1}^n\{\prod _{j\ne i}\int q_{Y|X}(y_j|x_j)p_X(x_j)exp\left(-\frac{u}{n}\frac{r_{Y|X}(y_j|x_j)}{q_{Y|X}(y_j|x_j)}\right)\mathrm{d}{x}_j\mathrm{d}{y}_j$$

$$\begin{array}{cc}\hfill & ·\int r_{Y|X}(y_i|x_i)p_X(x_i)exp\left(-\frac{u}{n}\frac{r_{Y|X}(y_i|x_i)}{q_{Y|X}(y_i|x_i)}\right)\mathrm{d}{x}_i\mathrm{d}{y}_i\}]\mathrm{d}u\hfill \end{array}$$

(A53)

$$={\int }_0^{\infty }\frac{1}{u}[e^{-u}-\frac{1}{n}\sum _{i=1}^n\{{\left(\int q_{Y|X}(y|x)p_X(x)exp\left(-\frac{u}{n}\frac{r_{Y|X}(y|x)}{q_{Y|X}(y|x)}\right)\mathrm{d}x\mathrm{d}y\right)}^{n-1}$$

$$\begin{array}{cc}\hfill & ·\int r_{Y|X}(y|x)p_X(x)exp\left(-\frac{u}{n}\frac{r_{Y|X}(y|x)}{q_{Y|X}(y|x)}\right)\mathrm{d}x\mathrm{d}y\}]\mathrm{d}u\hfill \end{array}$$

(A54)

$$={\int }_0^{\infty }\frac{1}{u}[e^{-u}-{\left(\int q_{Y|X}(y|x)p_X(x)exp\left(-\frac{u}{n}\frac{r_{Y|X}(y|x)}{q_{Y|X}(y|x)}\right)\mathrm{d}x\mathrm{d}y\right)}^{n-1}$$

$$\begin{array}{cc}\hfill & ·\int r_{Y|X}(y|x)p_X(x)exp\left(-\frac{u}{n}\frac{r_{Y|X}(y|x)}{q_{Y|X}(y|x)}\right)\mathrm{d}x\mathrm{d}y]\mathrm{d}u\hfill \end{array}$$

(A55)

$$\begin{array}{c}={\int }_0^{\infty }\frac{1}{u}\left[e^{-u}-f^{n-1}\left(\frac{u}{n}\right)g\left(\frac{u}{n}\right)\right]\mathrm{d}u,\hfill \end{array}$$

(A56)

where $f(·)$ and $g(·)$ are defined in (63) and (64), respectively. Consequently, $f(0)=g(0)=1$, and $0\le f(u),g(u)\le 1$ for all $u>0$.

Appendix D.2. Proof of Equation (65)

$$\begin{array}{c}\int p_{X^n,Y^n}(x^n,y^n)ln{q}_{Y^n|X^n}(y^n|x^n)\mathrm{d}{x}^n\mathrm{d}{y}^n\hfill \end{array}$$

$$\begin{array}{c}=\int p_{X^n,Y^n}(x^n,y^n)\sum _{j=1}^{n}ln{q}_{Y|X}(y_j|x_j)\mathrm{d}{x}^n\mathrm{d}{y}^n\hfill \end{array}$$

(A57)

$$\begin{array}{c}=\int \prod _{\ell =1}^n{p}_X(x_{\ell })·\frac{1}{n}\sum _{i=1}^n\left\{\prod _{\ell \ne i}{q}_{Y|X}(y_{\ell }|x_{\ell })r_{Y|X}(y_i|x_i)\right\}\sum _{j=1}^{n}ln{q}_{Y|X}(y_j|x_j)\mathrm{d}{x}^n\mathrm{d}{y}^n\hfill \end{array}$$

(A58)

$$\begin{array}{c}=\int \prod _{\ell =1}^n{p}_X(x_{\ell })·\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^n\left\{\prod _{\ell \ne i}{q}_{Y|X}(y_{\ell }|x_{\ell })·r_{Y|X}(y_i|x_i)ln{q}_{Y|X}(y_j|x_j)\right\}\mathrm{d}{x}^n\mathrm{d}{y}^n\hfill \end{array}$$

(A59)

$$\begin{array}{c}=\frac{1}{n}{\int }_{{\mathcal{X}}^n}\prod _{\ell =1}^n{p}_X(x_{\ell })\left(\sum _{i=1}^n\sum _{j=1}^n{\int }_{{\mathcal{Y}}^n}\prod _{\ell \ne i}{q}_{Y|X}(y_{\ell }|x_{\ell })·r_{Y|X}(y_i|x_i)ln{q}_{Y|X}(y_j|x_j)\mathrm{d}{y}^n\right)\mathrm{d}{x}^n.\hfill \end{array}$$

(A60)

We next calculate the inner integral on the right-hand side of (A60). For $i=j$,

$$\begin{array}{cc}\hfill & {\int }_{{\mathcal{Y}}^n}\prod _{\ell \ne i}{q}_{Y|X}(y_{\ell }|x_{\ell })·r_{Y|X}(y_i|x_i)ln{q}_{Y|X}(y_j|x_j)\mathrm{d}{y}^n\hfill \\ \hfill & =\prod _{\ell \ne i}{\int }_{\mathcal{Y}}{q}_{Y|X}(y_{\ell }|x_{\ell })\mathrm{d}{y}_{\ell }·{\int }_{\mathcal{Y}}{r}_{Y|X}(y_i|x_i)ln{q}_{Y|X}(y_i|x_i)\mathrm{d}{y}_i\hfill \\ \hfill & ={\int }_{\mathcal{Y}}{r}_{Y|X}(y|x_i)ln{q}_{Y|X}(y|x_i)\mathrm{d}y,\hfill \end{array}$$

(A61)

else, if $i\ne j$,

$$\begin{array}{cc}\hfill & {\int }_{{\mathcal{Y}}^n}\prod _{\ell \ne i}{q}_{Y|X}(y_{\ell }|x_{\ell })·r_{Y|X}(y_i|x_i)ln{q}_{Y|X}(y_j|x_j)\mathrm{d}{y}^n\hfill \\ \hfill & =\prod _{\ell \notin \{i,j\}}{\int }_{\mathcal{Y}}{q}_{Y|X}(y_{\ell }|x_{\ell })\mathrm{d}{y}_{\ell }·{\int }_{\mathcal{Y}}{q}_{Y|X}(y_j|x_j)ln{q}_{Y|X}(y_j|x_j)\mathrm{d}{y}_j·{\int }_{\mathcal{Y}}{r}_{Y|X}(y_i|x_i)\mathrm{d}{y}_i\hfill \\ \hfill & ={\int }_{\mathcal{Y}}{q}_{Y|X}(y|x_j)ln{q}_{Y|X}(y|x_j)\mathrm{d}y.\hfill \end{array}$$

(A62)

Hence, from (A60)–(A62),

$$\begin{array}{cc}\hfill & \int p_{X^n,Y^n}(x^n,y^n)ln{q}_{Y^n|X^n}(y^n|x^n)\mathrm{d}{x}^n\mathrm{d}{y}^n\hfill \\ \hfill & =\frac{1}{n}{\int }_{{\mathcal{X}}^n}\prod _{\ell =1}^n{p}_X(x_{\ell })\left(\sum _{i=1}^n{\int }_{\mathcal{Y}}{r}_{Y|X}(y|x_i)ln{q}_{Y|X}(y|x_i)\mathrm{d}y+\sum _{i=1}^n\sum _{j\ne i}{\int }_{\mathcal{Y}}{q}_{Y|X}(y|x_j)ln{q}_{Y|X}(y|x_j)\mathrm{d}y\right)\mathrm{d}{x}^n\hfill \\ \hfill & =\frac{1}{n}[\sum _{i=1}^n\left\{\prod _{\ell \ne i}{\int }_{\mathcal{X}}{p}_X(x_{\ell })\mathrm{d}{x}_{\ell }·{\int }_{\mathcal{X}\times \mathcal{Y}}{r}_{Y|X}(y|x_i)ln{q}_{Y|X}(y|x_i)p_X(x_i)\mathrm{d}{x}_i\mathrm{d}y\right\}\hfill \\ \hfill & +\sum _{i=1}^n\sum _{j\ne i}\left\{\prod _{\ell \ne j}{\int }_{\mathcal{X}}{p}_X(x_{\ell })\mathrm{d}{x}_{\ell }·{\int }_{\mathcal{X}\times \mathcal{Y}}{p}_X(x_j)q_{Y|X}(y|x_j)ln{q}_{Y|X}(y|x_j)\mathrm{d}{x}_j\mathrm{d}y\right\}]\hfill \\ \hfill & =\frac{1}{n}\left[\sum _{i=1}^n{\int }_{\mathcal{X}\times \mathcal{Y}}{r}_{Y|X}(y|x)ln{q}_{Y|X}(y|x)p_X(x)\mathrm{d}x\mathrm{d}y+\sum _{i=1}^n\sum _{j\ne i}{\int }_{\mathcal{X}\times \mathcal{Y}}{p}_X(x)q_{Y|X}(y|x)ln{q}_{Y|X}(y|x)\mathrm{d}x\mathrm{d}y\right]\hfill \\ \hfill & ={\int }_{\mathcal{X}\times \mathcal{Y}}{p}_X(x)r_{Y|X}(y|x)ln{q}_{Y|X}(y|x)\mathrm{d}x\mathrm{d}y+(n-1){\int }_{\mathcal{X}\times \mathcal{Y}}{p}_X(x)q_{Y|X}(y|x)ln{q}_{Y|X}(y|x)\mathrm{d}x\mathrm{d}y.\hfill \end{array}$$

(A63)

Appendix D.3. Proof of Equations (67)–(73)

$$\begin{array}{cc}\hfill p_{Y^n}(y^n)& =\int p_{Y^n|X^n}(y^n|x^n)p_{X^n}(x^n)\mathrm{d}{x}^n\hfill \\ \hfill & =\frac{1}{n}\sum _{i=1}^n\left\{\prod _{j\ne i}\int q_{Y|X}(y_j|x_j)p_X(x_j)\mathrm{d}{x}_j·\int r_{Y|X}(y_i|x_i)P_X(x_i)\mathrm{d}{x}_i\right\}\hfill \\ \hfill & =\frac{1}{n}\sum _{i=1}^n\left\{\prod _{j\ne i}v(y_j)·w(y_i)\right\}\hfill \\ \hfill & =\prod _{j=1}^{n}v(y_j)·\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)},\forall y^n\in {\mathcal{Y}}^n,\hfill \end{array}$$

(A64)

where $v(·)$ and $w(·)$ are probability densities on $\mathcal{Y}$, as defined in (68) and (69), respectively. This proves (67).

We next prove (70), which is used to calculate the entropy of $Y^n$ with the density $p_{Y^n}(·)$ in (A64). In view of the integral representation of the logarithmic function in (1) and by interchanging the order of the integrations (due to Fubini’s theorem), we get that for a positive random variable Z

$$\begin{array}{cc}\hfill \mathbb{E}\left\{ZlnZ\right\}& ={\int }_0^{\infty }\frac{1}{u}·\mathbb{E}\left\{Z\left(e^{-u}-e^{-uZ}\right)\right\}\mathrm{d}u\hfill \\ \hfill & ={\int }_0^{\infty }\frac{\mathbb{E}\left\{Z\right\}{e}^{-u}-\mathbb{E}\left\{Z{e}^{-uZ}\right\}}{u}\mathrm{d}u\hfill \\ \hfill & ={\int }_0^{\infty }\frac{M_Z^{\prime }(0)e^{-u}-M_Z^{\prime }(-u)}{u}\mathrm{d}u,\hfill \end{array}$$

(A65)

which proves (70). Finally, we prove (71). In view of (A64),

$$\begin{array}{cc}\hfill h(Y^n)& =-\int p_{Y^n}(y^n)ln{p}_{Y^n}(y^n)\mathrm{d}{y}^n\hfill \\ \hfill & =-\int \prod _{j=1}^{n}v(y_j)·\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)}·[ln\left(\prod _{j=1}^{n}v(y_j)\right)+ln\left(\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)}\right)]\mathrm{d}{y}^n\hfill \\ \hfill & =-\int \prod _{j=1}^{n}v(y_j)·\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)}·[\sum _{j=1}^{n}lnv(y_j)+ln\left(\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)}\right)]\mathrm{d}{y}^n\hfill \\ \hfill & =-\int \prod _{\ell =1}^{n}v(y_{\ell })·\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^n\frac{w(y_i)lnv(y_j)}{v(y_i)}\mathrm{d}{y}^n\hfill \\ \hfill & -\int \prod _{j=1}^{n}v(y_j)·\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)}·ln\left(\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)}\right)\mathrm{d}{y}^n.\hfill \end{array}$$

(A66)

A calculation of the first integral on the right-hand side of (A66) gives

$$\begin{array}{cc}\hfill \int \prod _{\ell =1}^{n}v(y_{\ell })·\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^n\frac{w(y_i)lnv(y_j)}{v(y_i)}\mathrm{d}{y}^n& =\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^n\int \prod _{\ell =1}^{n}v(y_{\ell })·\frac{w(y_i)lnv(y_j)}{v(y_i)}\mathrm{d}{y}^n\hfill \\ \hfill & =\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^n\int \prod _{\ell \ne i}v(y_{\ell })·w(y_i)lnv(y_j)\mathrm{d}{y}^n.\hfill \end{array}$$

(A67)

For $i=j$, the inner integral on the right-hand side of (A67) satisfies

$$\begin{array}{cc}\hfill \int \prod _{\ell \ne i}v(y_{\ell })·w(y_i)lnv(y_j)\mathrm{d}{y}^n& =\prod _{\ell \ne i}\int v(y_{\ell })\mathrm{d}{y}_{\ell }·\int w(y_i)lnv(y_i)\mathrm{d}{y}_i\hfill \\ \hfill & =\int w(y)lnv(y)\mathrm{d}y,\hfill \end{array}$$

(A68)

and for $i\ne j$,

$$\begin{array}{cc}\hfill \int \prod _{\ell \ne i}v(y_{\ell })·w(y_i)lnv(y_j)\mathrm{d}{y}^n& =\prod _{\ell \ne i,j}\int v(y_{\ell })\mathrm{d}{y}_{\ell }·\int w(y_i)\mathrm{d}{y}_i·\int v(y_j)lnv(y_j)\mathrm{d}{y}_j\hfill \\ \hfill & =\int v(y)lnv(y)\mathrm{d}y.\hfill \end{array}$$

(A69)

Therefore, combining (A67)–(A69) gives

$$\begin{array}{c}\hfill \int \prod _{\ell =1}^{n}v(y_{\ell })·\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^n\frac{w(y_i)lnv(y_j)}{v(y_i)}\mathrm{d}{y}^n=\int w(y)lnv(y)\mathrm{d}y+(n-1)\int v(y)lnv(y)\mathrm{d}y.\end{array}$$

(A70)

Finally, we calculate the second integral on the right-hand side of (A66). Let ${\mu }_n$ be the probability density function defined as

$$\begin{array}{c}\hfill {\mu }_n(y^n):=\prod _{j=1}^{n}v(y_j),y^n\in {\mathcal{Y}}^n,\end{array}$$

(A71)

and let

$$\begin{array}{c}\hfill Z:=\frac{1}{n}\sum _{i=1}^n\frac{w(V_i)}{v(V_i)}\end{array}$$

(A72)

where ${\{V_i\}}_{i=1}^n$ are i.i.d. $\mathcal{Y}$-valued random variables with a probability density function v. Then, in view of (70), the second integral on the right-hand side of (A66) satisfies

$$\begin{array}{cc}\hfill \int \prod _{j=1}^{n}v(y_j)·\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)}·ln\left(\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)}\right)\mathrm{d}{y}^n& =\mathbb{E}\left\{ZlnZ\right\}\hfill \\ \hfill & ={\int }_0^{\infty }\frac{M_Z^{\prime }(0)e^{-u}-M_Z^{\prime }(-u)}{u}\mathrm{d}u.\hfill \end{array}$$

(A73)

The MGF of Z is equal to

$$\begin{array}{cc}\hfill M_Z(u)& ={\int }_{{\mathcal{Y}}^n}\prod _{i=1}^{n}v(r_i)exp\left(\frac{u}{n}\sum _{i=1}^n\frac{w(r_i)}{v(r_i)}\right)\mathrm{d}\underline{r}\hfill \\ \hfill & =\prod _{i=1}^n{\int }_{\mathcal{Y}}v(r_i)exp\left(\frac{u}{n}\frac{w(r_i)}{v(r_i)}\right)\mathrm{d}{r}_i\hfill \\ \hfill & =K^n\left(\frac{u}{n}\right),\hfill \end{array}$$

(A74)

where

$$\begin{array}{c}\hfill K(u):={\int }_{\mathcal{Y}}v(y)exp\left(\frac{uw(y)}{v(y)}\right)\mathrm{d}y,\forall u\in \mathbb{R},\end{array}$$

(A75)

and consequently, (A75) yields

$$\begin{array}{cc}\hfill M_Z^{\prime }(u)& =K^{n-1}\left(\frac{u}{n}\right)K^{\prime }\left(\frac{u}{n}\right)\hfill \\ \hfill & ={\left(\int v(y)exp\left(\frac{uw(y)}{v(y)}\right)\mathrm{d}y\right)}^{n-1}\int w(y)exp\left(\frac{uw(y)}{v(y)}\right)\mathrm{d}y,\hfill \end{array}$$

(A76)

and

$$\begin{array}{c}\hfill M_Z^{\prime }(0)=1.\end{array}$$

(A77)

Therefore, combining (A73)–(A77) gives the following single-letter expression for the second multi-dimensional integral on the right-hand side of (A66):

$$\begin{array}{cc}\hfill \int \prod _{j=1}^{n}v(y_j)·\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)}·ln\left(\frac{1}{n}\sum _{i=1}^n\frac{w(y_i)}{v(y_i)}\right)\mathrm{d}{y}^n& ={\int }_0^{\infty }\frac{1}{u}\left[e^{-u}-t^{n-1}\left(\frac{u}{n}\right)s\left(\frac{u}{n}\right)\right]\mathrm{d}u\hfill \end{array}$$

(A78)

where the functions $s(·)$ and $t(·)$ are defined in (72) and (73), respectively. Combining (A66), (A70), and (A78) gives (71).

Appendix D.4. Specialization to a BSC with Jamming

In the BSC example considered, we have $\mathcal{X}=\mathcal{Y}=\{0,1\}$, and

$$\begin{array}{cc}\hfill & r_{Y|X}(y|x)=\epsilon 1\{x\ne y\}+(1-\epsilon )1\{x=y\},\hfill \end{array}$$

(A79)

$$\begin{array}{cc}\hfill & q_{Y|X}(y|x)=\delta 1\{x\ne y\}+(1-\delta )1\{x=y\},\hfill \end{array}$$

(A80)

where $1\{\mathrm{relation}\}$ is the indicator function that is equal to one if the relation holds, and to zero otherwise. Recall that we assume $0<\delta <\epsilon \le \frac{1}{2}$. Let

$$\begin{array}{c}\hfill p_X(0)=p_X(1)=\frac{1}{2},\end{array}$$

(A81)

be the binary symmetric source (BSS). From (63) and (64), for $u\ge 0$,

$$\begin{array}{cc}\hfill f(u)& =\sum _{x,y}{p}_X(x)q_{Y|X}(y|x)exp\left(-\frac{u{r}_{Y|X}(y|x)}{q_{Y|X}(y|x)}\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =(1-\delta )exp\left(-\frac{(1-\epsilon )u}{1-\delta }\right)+\delta exp\left(-\frac{\epsilon u}{\delta }\right),\hfill \end{array}$$

(A82)

$$\hfill g(u)& =\sum _{x,y}{p}_X(x)r_{Y|X}(y|x)exp\left(-\frac{u{r}_{Y|X}(y|x)}{q_{Y|X}(y|x)}\right)\hfill$$

$$\begin{array}{cc}\hfill & =(1-\epsilon )exp\left(-\frac{(1-\epsilon )u}{1-\delta }\right)+\epsilon exp\left(-\frac{\epsilon u}{\delta }\right).\hfill \end{array}$$

(A83)

Furthermore, we get from (A79), (A80), and (A81) that

$$\begin{array}{c}\hfill -\sum _{x,y}{p}_X(x)r_{Y|X}(y|x)ln{q}_{Y|X}(y|x)=-\epsilon ln\delta -(1-\epsilon )ln(1-\delta )=d(\epsilon \parallel \delta )+H_{\mathrm{b}}(\epsilon ),\end{array}$$

(A84)

and

$$\begin{array}{cc}\hfill & -\sum _{x,y}{p}_X(x)q_{Y|X}(y|x)ln{q}_{Y|X}(y|x)=H_{\mathrm{b}}(\delta ).\hfill \end{array}$$

(A85)

Substituting (A82)–(A85) into (66) (where the integrals in (66) are replaced by sums) gives

$$\begin{array}{cc}\hfill H(Y^n|X^n)& =d(\epsilon \parallel \delta )+H_{\mathrm{b}}(\epsilon )+(n-1)H_{\mathrm{b}}(\delta )+{\int }_0^{\infty }\left[f^{n-1}\left(\frac{u}{n}\right)g\left(\frac{u}{n}\right)-e^{-u}\right]\frac{\mathrm{d}u}{u}.\hfill \end{array}$$

(A86)

Since the input is a BSS, due to the symmetry of the channel (55), the output is also a BSS. This implies that (in units of nats)

$$\begin{array}{c}\hfill H(Y^n)=nln2.\end{array}$$

(A87)

As a sanity check, we verify it by using (71). From (68) and (69), for $y\in \{0,1\}$,

$$\begin{array}{cc}\hfill & v(y)=p_X(0)q_{Y|X}(y|0)+p_X(1)q_{Y|X}(y|1)=\frac{1}{2},\hfill \end{array}$$

(A88)

$$\begin{array}{cc}\hfill & w(y)=p_X(0)r_{Y|X}(y|0)+p_X(1)r_{Y|X}(y|1)=\frac{1}{2},\hfill \end{array}$$

(A89)

and from (72) and (73), it consequently follows that

$$\begin{array}{c}\hfill s(u)=w(0)exp\left(-\frac{uw(0)}{v(0)}\right)+w(1)exp\left(-\frac{uw(1)}{v(1)}\right)=e^{-u},\forall u\ge 0,\end{array}$$

(A90)

and also

$$\begin{array}{c}\hfill t(u)=e^{-u},\forall u\ge 0.\end{array}$$

(A91)

It can be verified that substituting (A88)–(A91) into (71) reproduces (A87). Finally, subtracting (A86) from (A87) gives (75).