An Upper Bound on the Error Induced by Saddlepoint Approximations—Applications to Information Theory ^†

Abstract

This paper introduces an upper bound on the absolute difference between: $\left(a\right)$ the cumulative distribution function (CDF) of the sum of a finite number of independent and identically distributed random variables with finite absolute third moment; and $\left(b\right)$ a saddlepoint approximation of such CDF. This upper bound, which is particularly precise in the regime of large deviations, is used to study the dependence testing (DT) bound and the meta converse (MC) bound on the decoding error probability (DEP) in point-to-point memoryless channels. Often, these bounds cannot be analytically calculated and thus lower and upper bounds become particularly useful. Within this context, the main results include, respectively, new upper and lower bounds on the DT and MC bounds. A numerical experimentation of these bounds is presented in the case of the binary symmetric channel, the additive white Gaussian noise channel, and the additive symmetric $\alpha$-stable noise channel.

1. Introduction

This paper focuses on approximating the cumulative distribution function (CDF) of sums of a finite number of real-valued independent and identically distributed (i.i.d.) random variables with finite absolute third moment. More specifically, let $Y_1$, $Y_2$, ⋯, $Y_n$, with n an integer and $2⩽n<\infty$, be real-valued random variables with probability distribution $P_Y$. Denote by $F_Y$ the CDF associated with $P_Y$, and, if it exists, denote by $f_Y$ the corresponding probability density function (PDF). Let also

$${\displaystyle X_n=\sum _{t=1}^n{Y}_t}$$

(1)

be a random variable with distribution $P_{X_n}$. Denote by $F_{X_n}$ the CDF and if it exists, denote by $f_{X_n}$ the PDF associated with $P_{X_n}$. The objective is to provide a positive function that approximates $F_{X_n}$ and an upper bound on the resulting approximation error. In the following, a positive function $g:\mathbb{R}\to {\mathbb{R}}_{+}$ is said to approximate $F_{X_n}$ with an approximation error that is upper bounded by a function $ϵ:\mathbb{R}\to {\mathbb{R}}_{+}$, if, for all $x\in \mathbb{R}$,

$$|F_{X_n}\left(x\right)-g\left(x\right)|⩽ϵ\left(x\right).$$

(2)

The case in which $Y_1$, $Y_2$, ⋯, $Y_n$ in (1) are stable random variables with $F_Y$ analytically expressible is trivial. This is essentially because the sum $X_n$ follows the same distribution of a random variable $a_{n}Y+b_n$, where $(a_n,b_n)\in {\mathbb{R}}^2$ and Y is a random variable whose CDF is $F_Y$. Examples of this case are random variables following the Gaussian, Cauchy, or Levy distributions [1].

In general, the problem of calculating the CDF of $X_n$ boils down to calculating $n-1$ convolutions. More specifically, it holds that

$${\displaystyle f_{X_n}\left(x\right)={\int }_{-\infty }^{\infty }{f}_{X_{n-1}}\left(x-t\right)f_Y\left(t\right)\mathrm{d}t,}$$

(3)

where $f_{X_1}=f_Y$. Even for discrete random variables and small values of n, the integral in (3) often requires excessive computation resources [2].

When the PDF of the random variable $X_n$ cannot be conveniently obtained but only the r first moments are known, with $r\in \mathbb{N}$, an approximation of the PDF can be obtained by using an Edgeworth expansion. Nonetheless, the resulting relative error in the large deviation regime makes these approximations inaccurate [3].

When the cumulant generating function (CGF) associated with $F_Y$, denoted by $K_Y:\mathbb{R}\to \mathbb{R}$, is known, the PDF $f_{X_n}$ can be obtained via the Laplace inversion lemma [2]. That is, given two reals ${\alpha }_{-}<0$ and ${\alpha }_{+}>0$, if $K_Y$ is analytic for all $z\in \{a+ib\in \mathbb{C}:(a,b)\in {\mathbb{R}}^2\mathrm{and}{\alpha }_{-}⩽a⩽{\alpha }_{+}\}\subset \mathbb{C}$, then,

$${\displaystyle f_{X_n}\left(x\right)=\frac{1}{2\pi i}{\int }_{\gamma -i\infty }^{\gamma +i\infty }exp\left(n{K}_Y\left(z\right)-zx\right)\mathrm{d}z,}$$

(4)

with $i=\sqrt{-1}$ and $\gamma \in ({\alpha }_{-},{\alpha }_{+})$. Note that the domain of $K_Y$ in (4) has been extended to the complex plane and thus it is often referred to as the complex CGF. With an abuse of notation, both the CGF and the complex CGF are identically denoted.

In the case in which n is sufficiently large, an approximation to the Bromwich integral in (4) can be obtained by choosing the contour to include the unique saddlepoint of the integrand as suggested in [4]. The intuition behind this lies on the following observations:

(i) 

the saddlepoint, denoted by $z_0$, is unique, real and $z_0\in ({\alpha }_{-},{\alpha }_{+})$;

(ii) 

within a neighborhood around the saddlepoint of the form $\left|z-z_0\right|<ϵ$, with $z\in \mathbb{C}$ and $ϵ>0$ sufficiently small, $\mathrm{Im}\left[n{K}_Y\left(z\right)-zx\right]=0$ and $\mathrm{Re}\left[n{K}_Y\left(z\right)-zx\right]$ can be assumed constant; and

(iii) 

outside such neighborhood, the integrand is negligible.

From $\left(i\right)$, it follows that the derivative of $n{K}_Y\left(t\right)-tx$ with respect to t, with $t\in \mathbb{R}$, is equal to zero when it is evaluated at the saddlepoint $z_0$. More specifically, for all $t\in \mathbb{R}$,

$$\frac{\mathrm{d}}{\mathrm{d}t}{K}_Y\left(t\right)={\mathbb{E}}_{P_Y}\left[Yexp\left(tY-K_Y\left(t\right)\right)\right],$$

(5)

and thus

$${\mathbb{E}}_{P_Y}\left[Yexp\left(z_{0}Y-K_Y\left(z_0\right)\right)\right]=\frac{x}{n},$$

(6)

which shows the dependence of $z_0$ on both x and n.

A Taylor series expansion of the exponent $n{K}_Y\left(z\right)-zx$ in the neighborhood of $z_0$, leads to the following asymptotic expansion in powers of $\frac{1}{n}$ of the Bromwich integral in (4):

$$\begin{array}{ccc}{f}_{X_n}\left(x\right)\hfill & =& {\widehat{f}}_{X_n}\left(x\right)\left(1+\frac{1}{n}\left(\frac{1}{8}\frac{K_Y^{\left(4\right)}\left(z_0\right)}{{\left(K_Y^{\left(2\right)}\left(z_0\right)\right)}^2}-\frac{5}{24}\frac{{\left(K_Y^{\left(3\right)}\left(z_0\right)\right)}^2}{{\left(K_Y^{\left(2\right)}\left(z_0\right)\right)}^3}\right)+O\left(\frac{1}{n^2}\right)\right),\hfill \end{array}$$

(7)

where ${\widehat{f}}_{X_n}:\mathbb{R}\to {\mathbb{R}}_{+}$ is

$$\begin{array}{ccc}{\widehat{f}}_{X_n}\left(x\right)\hfill & =& \sqrt{\frac{1}{2\pi n{K}_Y^{\left(2\right)}\left(z_0\right)}}exp\left(n{K}_Y\left(z_0\right)-z_{0}x\right),\hfill \end{array}$$

(8)

and for all $k\in \mathbb{N}$ and $t\in \mathbb{R}$, the notation $K_Y^{\left(k\right)}\left(t\right)$ represents the k-th real derivative of the CGF $K_Y$ evaluated at t. The first two derivatives $K_Y^{\left(1\right)}$ and $K_Y^{\left(2\right)}$ play a central role, and thus it is worth providing explicit expressions. That is,

$$\begin{array}{ccc}{K}_Y^{\left(1\right)}\left(t\right)\hfill & \triangleq & {\mathbb{E}}_{P_Y}\left[Yexp\left(tY-K_Y\left(t\right)\right)\right],\mathrm{and}\hfill \end{array}$$

(9)

$$\begin{array}{ccc}{K}_Y^{\left(2\right)}\left(t\right)\hfill & \triangleq & {\mathbb{E}}_{P_Y}\left[{\left|Y-K_Y^{\left(1\right)}\left(t\right)\right|}^{2}exp\left(tY-K_Y\left(t\right)\right)\right].\hfill \end{array}$$

(10)

The function ${\widehat{f}}_{X_n}$ in (8) is referred to as the saddlepoint approximation of the PDF $f_{X_n}$ and was first introduced in [4]. Nonetheless, ${\widehat{f}}_{X_n}$ is not necessarily a PDF as often its integral on $\mathbb{R}$ is not equal to one. A particular exception is observed only in three cases [5]. First, when $f_Y$ is the PDF of a Gaussian random variable, the saddlepoint approximation ${\widehat{f}}_{X_n}$ is identical to $f_{X_n}$, for all $n>0$. Second and third, when $f_Y$ is the PDF associated with a Gamma distribution and an inverse normal distribution, respectively, the saddlepoint approximation ${\widehat{f}}_{X_n}$ is exact up to a normalization constant for all $n>0$.

An approximation to the CDF $F_{X_n}$ can be obtained by integrating the PDF in (4), cf., [6,7,8]. In particular, the result reported in [6] leads to an asymptotic expansion of the CDF of $X_n$, for all $x\in \mathbb{R}$, of the form:

$$\begin{array}{ccc}{F}_{X_n}\left(x\right)\hfill & =& {\widehat{F}}_{X_n}\left(x\right)+O\left(\frac{1}{\sqrt{n}}exp\left(n{K}_Y\left(z_0\right)-x{z}_0\right)\right),\hfill \end{array}$$

(11)

where the function ${\widehat{F}}_{X_n}:\mathbb{R}\to \mathbb{R}$ is the saddlepoint approximation of $F_{X_n}$. That is, for all $x\in \mathbb{R}$,

$$\begin{array}{ccc}{\widehat{F}}_{X_n}\left(x\right)\hfill & =& {\mathbb{1}}_{\left\{z_0>0\right\}}+{(-1)}^{{\mathbb{1}}_{\left\{z_0>0\right\}}}exp\left(n{K}_Y\left(z_0\right)-z_{0}x+\frac{1}{2}{z}_0^{2}n{K}_Y^{\left(2\right)}\left(z_0\right)\right)Q\left(|z_0|\sqrt{n{K}_Y^{\left(2\right)}\left(z_0\right)}\right),\hfill \end{array}$$

(12)

where the function $Q:\mathbb{R}\to [0,1]$ is the complementary CDF of a Gaussian random variable with zero mean and unit variance. That is, for all $t\in \mathbb{R}$,

$$Q\left(t\right)=\frac{1}{\sqrt{2\pi }}{\int }_t^{\infty }exp\left(-\frac{x^2}{2}\right)\mathrm{d}x.$$

(13)

Finally, from the central limit theorem [3], for large values of n and for all $x\in \mathbb{R}$, a reasonable approximation to $F_{X_n}\left(x\right)$ is $1-Q\left(x\right)$. In the following, this approximation is referred to as the normal approximation of $F_{X_n}$.

1.1. Contributions

The main contribution of this work is an upper bound on the error induced by the saddlepoint approximation ${\widehat{F}}_{X_n}$ in (12) (Theorem 3 in Section 2.2). This result builds upon two observations. The first observation is that the CDF $F_{X_n}$ can be written for all $x\in \mathbb{R}$ in the form,

$$\begin{array}{c}{F}_{X_n}\left(x\right)={\mathbb{1}}_{\{z_0⩽0\}}{\mathbb{E}}_{P_{S_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{S}_n\right){\mathbb{1}}_{\{S_n⩽x\}}\right]+{\mathbb{1}}_{\{z_0>0\}}\left(1-{\mathbb{E}}_{P_{S_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{S}_n\right){\mathbb{1}}_{\{S_n>x\}}\right]\right),\hfill \end{array}$$

(14)

where the random variable

$$S_n=\sum _{t=1}^n{Y}_t^{\left(z_0\right)}$$

(15)

has a probability distribution denoted by $P_{S_n}$, and the random variables $Y_1^{\left(z_0\right)}$, $Y_2^{\left(z_0\right)}$, …, $Y_n^{\left(z_0\right)}$ are independent with probability distribution $P_{Y^{\left(z_0\right)}}$. The distribution $P_{Y^{\left(z_0\right)}}$ is an exponentially tilted distribution [9] with respect to the distribution $P_Y$ at the saddlepoint $z_0$. More specifically, the Radon–Nikodym derivative of the distribution $P_{Y^{\left(z_0\right)}}$ with respect to the distribution $P_Y$ satisfies for all $y\in \mathrm{supp}{P}_Y$,

$$\begin{array}{c}\frac{\mathrm{d}{P}_{Y^{\left(z_0\right)}}}{\mathrm{d}{P}_Y}\left(y\right)=exp\left(-\left(K_Y\left(z_0\right)-z_{0}y\right)\right).\hfill \end{array}$$

(16)

The second observation is that the saddlepoint approximation ${\widehat{F}}_{X_n}$ in (12) can be written for all $x\in \mathbb{R}$ in the form,

$$\begin{array}{c}{\widehat{F}}_{X_n}\left(x\right)={\mathbb{1}}_{\{z_0⩽0\}}{\mathbb{E}}_{P_{Z_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{Z}_n\right){\mathbb{1}}_{\{Z_n⩽x\}}\right]+{\mathbb{1}}_{\{z_0>0\}}\left(1-{\mathbb{E}}_{P_{Z_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{Z}_n\right){\mathbb{1}}_{\{Z_n>x\}}\right]\right),\hfill \end{array}$$

(17)

where $Z_n$ is a Gaussian random variable with mean x, variance $n{K}_Y^{\left(2\right)}\left(z_0\right)$, and probability distribution $P_{Z_n}$. Note that the means of the random variable $S_n$ in (14) and $Z_n$ in (17) are equal to $n{K}_Y^{\left(1\right)}\left(z_0\right)$, whereas their variances are equal to $n{K}_Y^{\left(2\right)}\left(z_0\right)$. Note also that, from (6), it holds that $x=n{K}_Y^{\left(1\right)}\left(z_0\right)$.

Using these observations, it holds that the absolute difference between $F_{X_n}$ in (14) and ${\widehat{F}}_{X_n}$ in (17) satisfies for all $x\in \mathbb{R}$,

$$\begin{array}{c}\left|F_{X_n}\left(x\right)-{\widehat{F}}_{X_n}\left(x\right)\right|\hfill \\ ={\mathbb{1}}_{\{z_0⩽0\}}\left|{\mathbb{E}}_{P_{S_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{S}_n\right){\mathbb{1}}_{\{S_n⩽x\}}\right]-{\mathbb{E}}_{P_{Z_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{Z}_n\right){\mathbb{1}}_{\{Z_n⩽x\}}\right]\right|\hfill \\ +{\mathbb{1}}_{\{z_0>0\}}\left|{\mathbb{E}}_{P_{S_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{S}_n\right){\mathbb{1}}_{\{S_n>x\}}\right]-{\mathbb{E}}_{P_{Z_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{Z}_n\right){\mathbb{1}}_{\{Z_n>x\}}\right]\right|.\hfill \end{array}$$

(18)

A step forward (Lemma A1 in Appendix A) is to note that, when x is such that $z_0⩽0$, then,

$$\begin{array}{c}\left|{\mathbb{E}}_{P_{S_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{S}_n\right){\mathbb{1}}_{\{S_n⩽x\}}\right]-{\mathbb{E}}_{P_{Z_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{Z}_n\right){\mathbb{1}}_{\{Z_n⩽x\}}\right]\right|\hfill \\ ⩽exp\left(n{K}_Y\left(z_0\right)-z_{0}x\right)min\left\{1,2\underset{a\in \mathbb{R}}{sup}\left|F_{S_n}\left(a\right)-F_{Z_n}\left(a\right)\right|\right\},\hfill \end{array}$$

(19)

and when x is such that $z_0>0$, it holds that

$$\begin{array}{c}\left|{\mathbb{E}}_{P_{S_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{S}_n\right){\mathbb{1}}_{\{S_n>x\}}\right]-{\mathbb{E}}_{P_{Z_n}}\left[exp\left(n{K}_Y\left(z_0\right)-z_0{Z}_n\right){\mathbb{1}}_{\{Z_n>x\}}\right]\right|\hfill \\ ⩽exp\left(n{K}_Y\left(z_0\right)-z_{0}x\right)min\left\{1,2\underset{a\in \mathbb{R}}{sup}\left|F_{S_n}\left(a\right)-F_{Z_n}\left(a\right)\right|\right\},\hfill \end{array}$$

(20)

where $F_{S_n}$ and $F_{Z_n}$ are the CDFs of the random variables $S_n$ and $Z_n$, respectively. The final result is obtained by observing that ${sup}_{a\in \mathbb{R}}\left|F_{S_n}\left(a\right)-F_{Z_n}\left(a\right)\right|$ can be upper bounded using the Berry–Esseen Theorem (Theorem 1 in Section 2.1). This is essentially due to the fact that the random variable $S_n$ is the sum of n independent random variables, i.e., (15), and $Z_n$ is a Gaussian random variable, and both $S_n$ and $Z_n$ possess identical means and variances. Thus, the main result (Theorem 3 in Section 2.2) is that, for all $x\in \mathbb{R}$,

$$\begin{array}{c}\left|F_{X_n}\left(x\right)-{\widehat{F}}_{X_n}\left(x\right)\right|⩽\frac{2{\xi }_Y\left(z_0\right)}{\sqrt{n}}exp\left(n{K}_Y\left(z_0\right)-z_{0}x\right),\hfill \end{array}$$

(21)

where

$$\begin{array}{c}{\xi }_Y\left(z_0\right)=c_1\left(\frac{{\mathbb{E}}_{P_Y}\left[{\left|Y-K_Y^{\left(1\right)}\left(z_0\right)\right|}^{3}exp\left(z_{0}Y-K_Y\left(z_0\right)\right)\right]}{{\left(K_Y^{\left(2\right)}\left(z_0\right)\right)}^{3/2}}+c_2\right),\hfill \end{array}$$

(22)

with

$$\begin{array}{c}{c}_1\triangleq 0.33554,\mathrm{and}\hfill \end{array}$$

(23a)

$$\begin{array}{c}{c}_2\triangleq 0.415.\hfill \end{array}$$

(23b)

Finally, note that (21) holds for any finite value of n and admits the asymptotic scaling law with respect to n suggested in (11).

1.2. Applications

In the realm of information theory, the normal approximation has played a central role in the calculation of bounds on the minimum decoding error probability (DEP) in point-to-point memoryless channels, cf., [10,11]. Thanks to the normal approximation, simple approximations for the dependence testing (DT) bound, the random coding union bound (RCU) bound, and the meta converse (MC) bound have been obtained in [10,12]. The success of these approximations stems from the fact that they are easy to calculate. Nonetheless, easy computation comes at the expense of loose upper and lower bounds and thus uncontrolled approximation errors.

On the other hand, saddlepoint techniques have been extensively used to approximate existing lower and upper bounds on the minimum DEP. See, for instance, [13,14] in the case of the RCU bound and the MC bound. Nonetheless, the errors induced by saddlepoint approximations are often neglected due to the fact that calculating them involves a large number of optimizations and numerical integrations. Currently, the validation of saddlepoint approximations is carried through Monte Carlo simulations. Within this context, the main objectives of this paper are twofold: $\left(a\right)$ to analytically assess the tightness of the approximation of DT and MC bounds based on the saddlepoint approximation of the CDFs of sums of i.i.d. random variables; $\left(b\right)$ to provide new lower and upper bounds on the minimum DEP by providing a lower bound on the MC bound and an upper bound on the DT bound. Numerical experimentation of these bounds is presented for the binary symmetric channel (BSC), the additive white Gaussian noise (AWGN) channel, and the additive symmetric $\alpha$-stable noise (S$\alpha$S) channel, where the new bounds are tight and obtained at low computational cost.

2. Sums of Independent and Identically Distributed Random Variables

In this section, upper bounds on the absolute error of approximating $F_{X_n}$ by the normal approximation and the saddlepoint approximation are presented.

2.1. Error Induced by the Normal Approximation

Given a random variable Y, let the function ${\xi }_Y:\mathbb{R}\stackrel{}{\to }\mathbb{R}$ be for all $t\in \mathbb{R}$ :

$$\begin{array}{ccc}{\xi }_Y\left(t\right)\hfill & \triangleq & c_1\left(\frac{{\mathbb{E}}_{P_Y}\left[{\left|Y-K_Y^{\left(1\right)}\left(t\right)\right|}^{3}exp\left(tY-K_Y\left(t\right)\right)\right]}{{\left(K_Y^{\left(2\right)}\left(t\right)\right)}^{3/2}}+c_2\right),\hfill \end{array}$$

(24)

where $c_1$ and $c_2$ are defined in (23).

The following theorem, known as the Berry–Esseen theorem [3], introduces an upper bound on the approximation error induced by the normal approximation.

Theorem 1

(Berry–Esseen [15]). Let $Y_1$, $Y_2$, …, $Y_n$ be i.i.d random variables with probability distribution $P_Y$. Let also $Z_n$ be a Gaussian random variable with mean $n{K}_Y^{\left(1\right)}\left(0\right)$, variance $n{K}_Y^{\left(2\right)}\left(0\right)$, and CDF denoted by $F_{Z_n}$. Then, the CDF of the random variable $X_n$=$Y_1$+$Y_2$+…+$Y_n$, denoted by $F_{X_n}$, satisfies

$$\begin{array}{c}\underset{a\in \mathbb{R}}{sup}\left|F_{X_n}\left(a\right)-F_{Z_n}\left(a\right)\right|⩽min\left\{1,\frac{{\xi }_Y\left(0\right)}{\sqrt{n}}\right\},\hfill \end{array}$$

(25)

where the functions $K_Y^{\left(1\right)}$, $K_Y^{\left(2\right)}$ and ${\xi }_Y$ are defined in (9), (10), and (24).

An immediate result from Theorem 1 gives the following upper and lower bounds on $F_{X_n}\left(a\right)$, for all $a\in \mathbb{R}$,

$$\begin{array}{c}{F}_{X_n}\left(a\right)⩽F_{Z_n}\left(a\right)+min\left\{1,\frac{{\xi }_Y\left(0\right)}{\sqrt{n}}\right\}\triangleq \overline{\mathsf{\Sigma }}(a,n),\mathrm{and}\hfill \end{array}$$

(26)

$$\begin{array}{c}{F}_{X_n}\left(a\right)⩾F_{Z_n}\left(a\right)-min\left\{1,\frac{{\xi }_Y\left(0\right)}{\sqrt{n}}\right\}\triangleq \underline{\mathsf{\Sigma }}(a,n).\hfill \end{array}$$

(27)

The main drawback of Theorem 1 is that the upper bound on the approximation error does not depend on the exact value of a. More importantly, for some values of a and n, the upper bound on the approximation error might be particularly big, which leads to irrelevant results.

2.2. Error Induced by the Saddlepoint Approximation

The following theorem introduces an upper bound on the approximation error induced by approximating the CDF $F_{X_n}$ of $X_n$ in (1) by the function ${\eta }_Y:$${\mathbb{R}}^2$×$\mathbb{N}$→$\mathbb{R}$ defined such that for all $(\theta$, a, $n)$∈${\mathbb{R}}^2$×$\mathbb{N}$,

$$\begin{array}{c}{\eta }_Y(\theta ,a,n)\hfill \\ \triangleq {\mathbb{1}}_{\{\theta >0\}}+{(-1)}^{{\mathbb{1}}_{\{\theta >0\}}}exp\left(\frac{1}{2}n{\theta }^2{K}_Y^{\left(2\right)}\left(\theta \right)+n{K}_Y\left(\theta \right)-n\theta K_Y^{\left(1\right)}\left(\theta \right)\right)Q\left({(-1)}^{{\mathbb{1}}_{\{\theta ⩽0\}}}\frac{a+n\theta K_Y^{\left(2\right)}\left(\theta \right)-n{K}_Y^{\left(1\right)}\left(\theta \right)}{\sqrt{n{K}_Y^{\left(2\right)}\left(\theta \right)}}\right),\hfill \end{array}$$

(28)

where the function $Q:\mathbb{R}\to [0,1]$ is the complementary CDF of the standard Gaussian distribution defined in (13). Note that ${\eta }_Y(\theta ,n,a)$ is identical to ${\widehat{F}}_{X_n}\left(a\right)$, when $\theta$ is chosen to satisfy the saddlepoint $K_Y^{\left(1\right)}\left(\theta \right)=\frac{a}{n}$. Note also that ${\eta }_Y(0,n,a)$ is the CDF of a Gaussian random variable with mean $n{K}_Y^{\left(1\right)}\left(0\right)$ and variance $n{K}_Y^{\left(2\right)}\left(0\right)$, which are the mean and the variance of $X_n$ in (1), respectively.

Theorem 2.

Let $Y_1$, $Y_2$, …, $Y_n$ be i.i.d. random variables with probability distribution $P_Y$ and CGF $K_Y$. Let also $F_{X_n}$ be the CDF of the random variable $X_n$=$Y_1$+$Y_2$+…+$Y_n$. Hence, for all a∈$\mathbb{R}$ and for all $\theta \in {\mathsf{\Theta }}_Y$, it holds that

$$\begin{array}{c}\left|F_{X_n}\left(a\right)-{\eta }_Y\left(\theta ,a,n\right)\right|⩽exp\left(n{K}_Y\left(\theta \right)-\theta a\right)min\left\{1,\frac{2{\xi }_Y\left(\theta \right)}{\sqrt{n}}\right\},\hfill \end{array}$$

(29)

where

$$\begin{array}{c}{\mathsf{\Theta }}_Y\triangleq \{t\in \mathbb{R}:K_Y\left(t\right)<\infty \};\hfill \end{array}$$

(30)

and the functions ${\xi }_Y$ and ${\eta }_Y$ are defined in (24) and (28), respectively.

Proof. 

The proof of Theorem 2 is presented in Appendix A. ☐

This result leads to the following upper and lower bounds on $F_{X_n}\left(a\right)$, for all $a\in \mathbb{R}$,

$$\begin{array}{ccc}{F}_{X_n}\left(a\right)\hfill & ⩽& {\eta }_Y\left(\theta ,a,n\right)+exp\left(n{K}_Y\left(\theta \right)-\theta a\right)min\left\{1,\frac{2{\xi }_Y\left(\theta \right)}{\sqrt{n}}\right\},\mathrm{and}\hfill \end{array}$$

(31)

$$\begin{array}{ccc}{F}_{X_n}\left(a\right)\hfill & ⩾& {\eta }_Y\left(\theta ,a,n\right)-exp\left(n{K}_Y\left(\theta \right)-\theta a\right)min\left\{1,\frac{2{\xi }_Y\left(\theta \right)}{\sqrt{n}}\right\},\hfill \end{array}$$

(32)

with $\theta$∈${\mathsf{\Theta }}_Y$.

The advantages of approximating $F_{X_n}$ by using Theorem 2 instead of Theorem 1 are twofold. First, both the approximation ${\eta }_Y$ and the corresponding approximation error depend on the exact value of a. In particular, the approximation can be optimized for each value of a via the parameter $\theta$. Second, the parameter $\theta$ in (29) can be optimized to improve either the upper bound in (31) or the lower bound in (32) for some $a\in \mathbb{R}$. Nonetheless, such optimizations are not necessarily simple.

An alternative to the optimization on $\theta$ in (31) and (32) is to choose $\theta$ such that it minimizes $n{K}_Y\left(\theta \right)-\theta a$. This follows the intuition that, for some values of a and n, the term $exp(n{K}_Y\left(\theta \right)-\theta a)$ is the one that influences the most the value of the right-hand side of (29). To build upon this idea, consider the following lemma.

Lemma 1.

Consider a random variable Y with probability distribution $P_Y$ and CGF $K_Y$. Given $n\in \mathbb{N}$, let the function $h:\mathbb{R}\to \mathbb{R}$ be defined for all $a\in \mathbb{R}$ satisfying $\frac{a}{n}\in \mathrm{int}{\mathcal{C}}_Y$, with $\mathrm{int}{\mathcal{C}}_Y$ denoting the interior of the convex hull of $\mathrm{supp}{P}_{X_n}$, as follows:

$$\begin{array}{c}h\left(a\right)=\underset{\theta \in {\mathsf{\Theta }}_Y}{inf}n{K}_Y\left(\theta \right)-\theta a,\hfill \end{array}$$

(33)

where ${\mathsf{\Theta }}_Y$ is defined in (30). Then, the function h is concave and for all $a\in \mathbb{R}$,

$$\begin{array}{c}h\left(a\right)⩽h\left(n{\mathbb{E}}_{P_Y}\left[Y\right]\right)=0.\hfill \end{array}$$

(34)

Furthermore,

$$\begin{array}{c}h\left(a\right)=n{K}_Y\left({\theta }^{\star }\right)-{\theta }^{\star }a,\hfill \end{array}$$

(35)

where ${\theta }^{\star }$ is the unique solution in θ to

$$\begin{array}{c}n{K}_Y^{\left(1\right)}\left(\theta \right)=a,\hfill \end{array}$$

(36)

with $K_Y^{\left(1\right)}$ is defined in (9).

Proof. 

The proof of Lemma 1 is presented in Appendix B. ☐

Given $(a,n)\in \mathbb{R}\times \mathbb{N}$, the value of $h\left(a\right)$ in (33) is the argument that minimizes the exponential term in (29). An interesting observation from Lemma 1 is that the maximum of h is zero, and it is reached when $a=n{\mathbb{E}}_{P_Y}\left[Y\right]={\mathbb{E}}_{P_{X_n}}\left[X_n\right]$. In this case, ${\theta }^{\star }=0$, and thus, from (31) and (32), it holds that

$$\begin{array}{ccc}{F}_{X_n}\left(a\right)\hfill & ⩽& {\eta }_Y\left(0,a,n\right)+min\left\{1,\frac{2{\xi }_Y\left(0\right)}{\sqrt{n}}\right\}\hfill \end{array}$$

$$\begin{array}{ccc}\hfill & =& F_{Z_n}\left(a\right)+min\left\{1,\frac{2{\xi }_Y\left(0\right)}{\sqrt{n}}\right\},\mathrm{and}\hfill \\ F_{X_n}\left(a\right)\hfill & ⩾& {\eta }_Y\left(0,a,n\right)-min\left\{1,\frac{2{\xi }_Y\left(0\right)}{\sqrt{n}}\right\}\hfill \end{array}$$

(37)

$$\begin{array}{ccc}\hfill & =& F_{Z_n}\left(a\right)-min\left\{1,\frac{2{\xi }_Y\left(0\right)}{\sqrt{n}}\right\},\hfill \end{array}$$

(38)

where $F_{Z_n}$ is the CDF defined in Theorem 1. Hence, the upper bound in (37) and the lower bound in (38) obtained from Theorem 2 are worse than those in (26) and (27) obtained from Theorem 1. In a nutshell, for values of a around the vicinity of $n{\mathbb{E}}_{P_Y}\left[Y\right]={\mathbb{E}}_{P_{X_n}}\left[X_n\right]$, it is more interesting to use Theorem 1 instead of Theorem 2.

Alternatively, given that h is non-positive and concave, when $\left|a-n{\mathbb{E}}_{P_Y}\left[Y\right]\right|$=$|a-$${\mathbb{E}}_{P_{X_n}}\left[X_n\right]|$>$\gamma$, with $\gamma$ sufficiently large, it follows that

$$\begin{array}{c}exp\left(n{K}_Y\left({\theta }^{\star }\right)-{\theta }^{\star }a\right)<min\left\{1,\frac{{\xi }_Y\left(0\right)}{\sqrt{n}}\right\},\hfill \end{array}$$

(39)

with ${\theta }^{\star }$ defined in (36). Hence, in this case, the right-hand side of (29) is always smaller than the right-hand side of (25). That is, for such values of a and n, the upper and lower bounds in (31) and (32) are better than those in (26) and (27), respectively. The following theorem leverages this observation.

Theorem 3.

Let $Y_1$, $Y_2$, …, $Y_n$ be i.i.d. random variables with probability distribution $P_Y$ and CGF $K_Y$. Let also $F_{X_n}$ be the CDF of the random variable $X_n=Y_1+Y_2+\dots +Y_n$. Hence, for all a∈$\mathrm{int}{\mathcal{C}}_{X_n}$, with $\mathrm{int}{\mathcal{C}}_{X_n}$ the interior of the convex hull of $\mathrm{supp}{P}_{X_n}$, it holds that

$$\begin{array}{c}\left|F_{X_n}\left(a\right)-{\widehat{F}}_{X_n}\left(a\right)\right|⩽exp\left(n{K}_Y\left({\theta }^{\star }\right)-{\theta }^{\star }a\right)min\left\{1,\frac{2{\xi }_Y\left({\theta }^{\star }\right)}{\sqrt{n}}\right\},\hfill \end{array}$$

(40)

where ${\theta }^{\star }$ is defined in (36), and the functions ${\widehat{F}}_{X_n}$ and ${\xi }_Y$ are defined in (12), and (24), respectively.

Proof. 

The proof of Theorem 3 is presented in Appendix C. ☐

An immediate result from Theorem 3 gives the following upper and lower bounds on $F_X\left(a\right)$, for all $a\in \mathbb{R}$ ,

$$\begin{array}{ccc}{F}_{X_n}\left(a\right)\hfill & ⩽& {\widehat{F}}_{X_n}\left(a\right)+exp\left(n{K}_Y\left({\theta }^{\star }\right)-{\theta }^{\star }a\right)min\left\{1,\frac{2{\xi }_Y\left({\theta }^{\star }\right)}{\sqrt{n}}\right\}\triangleq \overline{\mathsf{\Omega }}(a,n),\mathrm{and}\hfill \end{array}$$

(41)

$$\begin{array}{ccc}{F}_{X_n}\left(a\right)\hfill & ⩾& {\widehat{F}}_{X_n}\left(a\right)-exp\left(n{K}_Y\left({\theta }^{\star }\right)-{\theta }^{\star }a\right)min\left\{1,\frac{2{\xi }_Y\left({\theta }^{\star }\right)}{\sqrt{n}}\right\}\triangleq \underline{\mathsf{\Omega }}(a,n).\hfill \end{array}$$

(42)

The following section presents two examples that highlight the observations mentioned above.

2.3. Examples

Example 1

(Discrete random variable). Let the random variables $Y_1$, $Y_2$, …, $Y_n$ in (1) be i.i.d. Bernoulli random variables with parameter $p=0.2$ and $n=100$. In this case, ${\mathbb{E}}_{P_{X_n}}\left[X_n\right]=n{\mathbb{E}}_{P_Y}\left[Y\right]=20$. Figure 1 depicts the CDF $F_{X_{100}}$ of $X_{100}$ in (1); the normal approximation $F_{Z_{100}}$ in (25); and the saddlepoint approximation ${\widehat{F}}_{X_{100}}$ in (12). Therein, it is also depicted the upper and lower bounds due to the normal approximation $\overline{\mathsf{\Sigma }}$ in (26) and $\underline{\mathsf{\Sigma }}$ in (27), respectively; and the upper and lower bounds due to the saddlepoint approximation $\overline{\mathsf{\Omega }}$ in (41) and $\underline{\mathsf{\Omega }}$ in (42), respectively. These functions are plotted as a function of a, with a∈$[5,35]$.

Figure 1. Sum of 100 Bernoulli random variables with parameter $p=0.2$. The function $F_{X_{100}}\left(a\right)$ (asterisk markers *) in Example 1; the function $F_{Z_{100}}\left(a\right)$ (star markers ⋆) in (25); the function ${\widehat{F}}_{X_{100}}\left(a\right)$ (diamond markers ⋄) in (12); the function $\overline{\mathsf{\Sigma }}(a,100)$ (circle marker ∘) in (26); the function $\underline{\mathsf{\Sigma }}(a,100)$ (square marker □) in (27); the function $\overline{\mathsf{\Omega }}(a,100)$ (upward-pointing triangle marker ▵) in (41); and the function $\underline{\mathsf{\Omega }}(a,100)$ (downward-pointing triangle marker ▿) in (42) are plotted as functions of a, with $a\in [5,35]$.

Example 2

(Continuous random variable). Let the random variables $Y_1$, $Y_2$, …, $Y_n$ in (1) be i.i.d. chi-squared random variables with parameter $k=1$ and $n=50$. In this case, ${\mathbb{E}}_{P_{X_n}}\left[X_n\right]=n{\mathbb{E}}_{P_Y}\left[Y\right]=50$. Figure 2 depicts the CDF $F_{X_{50}}$ of $X_{50}$ in (1); the normal approximation $F_{Z_{50}}$ in (25); and the saddlepoint approximation ${\widehat{F}}_{X_{50}}$ in (12). Therein, it is also depicted the upper and lower bounds due to the normal approximation $\overline{\mathsf{\Sigma }}$ in (26) and $\underline{\mathsf{\Sigma }}$ in (27), respectively; and the upper and lower bounds due to the saddlepoint approximation $\overline{\mathsf{\Omega }}$ in (41) and $\underline{\mathsf{\Omega }}$ in (42), respectively. These functions are plotted as a function of a, with a∈$[0,100]$.

Figure 2. Sum of 50 Chi-squared random variables with parameter $k=1$. The function $F_{X_{50}}\left(a\right)$ (asterisk markers *) in Example 2; the function $F_{Z_{50}}\left(a\right)$ (star markers ⋆) in (25); the function ${\widehat{F}}_{X_{50}}\left(a\right)$ (diamond markers ⋄) in (12); the function $\overline{\mathsf{\Sigma }}(a,50)$ (circle marker ∘) in (26); the function $\underline{\mathsf{\Sigma }}(a,50)$ (square marker □) in (27); the function $\overline{\mathsf{\Omega }}(a,50)$ (upward-pointing triangle marker ▵) in (41); and the function $\underline{\mathsf{\Omega }}(a,50)$ (downward-pointing triangle marker ▿) in (42) are plotted as functions of a, with $a\in [0,100]$.

3. Application to Information Theory: Channel Coding

This section focuses on the study of the DEP in point-to-point memoryless channels. The problem is formulated in Section 3.1. The main results presented in this section consist of lower and upper bounds on the DEP. The former, which are obtained building upon the existing DT bound [10], are presented in Section 3.2. The latter, which are obtained from the MC bound [10], are presented in Section 3.3.

3.1. System Model

Consider a point-to-point communication in which a transmitter aims at sending information to one receiver through a noisy memoryless channel. Such a channel can be modeled by a random transformation

$$\begin{array}{c}({\mathcal{X}}^n,{\mathcal{Y}}^n,P_{\mathit{Y}|\mathit{X}}),\hfill \end{array}$$

(43)

where $n\in$$\mathbb{N}$ is the blocklength and $\mathcal{X}$ and $\mathcal{Y}$ are the channel input and channel output sets. Given the channel inputs $\mathit{x}$=$(x_1$, $x_2$, …, $x_n)$∈${\mathcal{X}}^n$, the outputs $\mathit{y}$=$(y_1$, $y_2$, …, $y_n)$∈${\mathcal{Y}}^n$ are observed at the receiver with probability

$$\begin{array}{c}{P}_{\mathit{Y}|\mathit{X}}\left(\mathit{y}\right|\mathit{x})=\prod _{t=1}^n{P}_{Y|X}\left(y_t\right|x_t),\hfill \end{array}$$

(44)

where, for all $x\in \mathcal{X}$, $P_{Y|X=x}\in ▵\left(\mathcal{Y}\right)$, with $▵\left(\mathcal{Y}\right)$, the set of all possible probability distributions whose support is a subset of $\mathcal{Y}$. The objective of the communication is to transmit a message index i, which is a realization of a random variable W that is uniformly distributed over the set

$$\begin{array}{c}\mathcal{W}\triangleq \{1,2,\dots ,M\},\hfill \end{array}$$

(45)

with 1 <M<∞. To achieve this objective, the transmitter uses an $(n$, M, $\lambda )$-code, where $\lambda$∈$[0,1]$.

Definition 1

($(n$, M,$\lambda )$-code). Given a tuple $(M$, n, $\lambda )$∈${\mathbb{N}}^2$×$[0,1]$, an $(n$, M, $\lambda )$-code for the random transformation in (43) is a system

$$\begin{array}{c}\left\{\left(\mathit{u}\left(1\right),\mathcal{D}\left(1\right)\right),\left(\mathit{u}\left(2\right),\mathcal{D}\left(2\right)\right),\dots ,\left(\mathit{u}\left(M\right),\mathcal{D}\left(M\right)\right)\right\},\hfill \end{array}$$

(46)

where for all $(j,\ell )\in {\mathcal{W}}^2$, with $j\ne \ell$:

$$\begin{array}{ccc}\mathit{u}\left(j\right)\hfill & =& (u_1\left(j\right),u_2\left(j\right),\dots ,u_n\left(j\right))\in {\mathcal{X}}^n,\hfill \end{array}$$

(47a)

$$\begin{array}{ccc}\mathcal{D}\left(j\right)\cap \mathcal{D}\left(\ell \right)\hfill & =& \varnothing ,\hfill \end{array}$$

(47b)

$$\begin{array}{ccc}\bigcup _{j\in \mathcal{W}}\mathcal{D}\left(j\right)\hfill & \subseteq & {\mathcal{Y}}^n,\mathrm{and}\hfill \end{array}$$

(47c)

$$\begin{array}{ccc}\frac{1}{M}\sum _{i=1}^M{\mathbb{E}}_{P_{\mathit{Y}|\mathit{X}=\mathit{u}(i)}}\left[{\mathbb{1}}_{\left\{Y\notin \mathcal{D}\left(i\right)\right\}}\right]\hfill & ⩽& \lambda .\hfill \end{array}$$

(47d)

To transmit message index i∈$\mathcal{W}$, the transmitter uses the codeword $\mathit{u}\left(i\right)$. For all t∈{ 1,2,…, $n\}$, at channel use t, the transmitter inputs the symbol $u_t\left(i\right)$ into the channel. Assume that, at the end of channel use t, the receiver observes the output $y_t$. After n channel uses, the receiver uses the vector $\mathit{y}$=$(y_1$,$y_2$,…, $y_n)$ and determines that the symbol j was transmitted if $\mathit{y}$∈$\mathcal{D}\left(j\right)$, with j∈$\mathcal{W}$.

Given the $(n$,M,$\lambda )$-code described by the system in (46), the DEP of the message index i can be computed as ${\mathbb{E}}_{P_{\mathit{Y}|\mathit{X}=\mathit{u}(i)}}[$${\mathbb{1}}_{\{Y\notin \mathcal{D}(i\left)\right\}}]$. As a consequence, the average DEP is

$$\frac{1}{M}\sum _{i=1}^M{\mathbb{E}}_{P_{\mathit{Y}|\mathit{X}=\mathit{u}(i)}}\left[{\mathbb{1}}_{\{Y\notin \mathcal{D}(i\left)\right\}}\right].$$

(48)

Note that, from (47d), the average DEP of such an $(n,M,\lambda )$-code is upper bounded by $\lambda$. Given a fixed pair $(n$,$M)$∈${\mathbb{N}}^2$, the minimum $\lambda$ for which an $(n$,M,$\lambda )$-code exists is defined hereunder.

Definition 2.

Given a pair $(n$,$M)$∈${\mathbb{N}}^2$, the minimum average DEP for the random transformation in (43), denoted by ${\lambda }^{*}(n,M)$, is given by

$$\begin{array}{c}{\lambda }^{*}(n,M)=min\left\{\lambda \in [0,1]:\exists (n,M,\lambda )-\mathrm{code}\right\}.\hfill \end{array}$$

(49)

When $\lambda$ is chosen accordingly with the reliability constraints, an $(n,M,\lambda )$-code is said to transmit at an information rate $R=\frac{{\log }_2\left(M\right)}{n}$ bits per channel use.

The remainder of this section introduces the DT and MC bounds. The DT bound is one of the tightest existing upper bounds on ${\lambda }^{*}(n,M)$ in (49), whereas the MC bound is one of the tightest lower bounds.

3.2. Dependence Testing Bound

This section describes an upper bound on ${\lambda }^{*}(n,M)$, for a fixed pair $(n,M)\in {\mathbb{N}}^2$. Given a probability distribution $P_{\mathit{X}}\in ▵\left({\mathcal{X}}^n\right)$, let the random variable $\iota \left(\mathit{X};\mathit{Y}\right)$ satisfy

$$\iota \left(\mathit{X};\mathit{Y}\right)\triangleq \ln \left(\frac{\mathrm{d}{P}_{\mathit{X}\mathit{Y}}}{\mathrm{d}{P}_{\mathit{X}}{P}_{\mathit{Y}}}(\mathit{X},\mathit{Y})\right),$$

(50)

where the function $\frac{\mathrm{d}{P}_{\mathit{X}\mathit{Y}}}{\mathrm{d}{P}_{\mathit{X}}{P}_{\mathit{Y}}}:{\mathcal{X}}^n\times {\mathcal{Y}}^n\to \mathbb{R}$ denotes the Radon–Nikodym derivative of the joint probability measure $P_{\mathit{X}\mathit{Y}}$ with respect to the product of probability measures $P_{\mathit{X}}{P}_{\mathit{Y}}$, with $P_{\mathit{X}\mathit{Y}}=P_{\mathit{X}}{P}_{\mathit{Y}|\mathit{X}}$ and $P_{\mathit{Y}}$ the corresponding marginal. Let the function $T:{\mathbb{N}}^2\times ▵\left({\mathcal{X}}^n\right)\to {\mathbb{R}}_{+}$ be for all $(n$,$M)$∈${\mathbb{N}}^2$ and for all probability distributions $P_{\mathit{X}}\in ▵\left({\mathcal{X}}^n\right)$,

$$\begin{array}{ccc}T(n,M,P_{\mathit{X}})\hfill & =& {\mathbb{E}}_{P_{\mathit{X}}{P}_{\mathit{Y}|\mathit{X}}}\left[{\mathbb{1}}_{\left\{\iota (\mathit{X};\mathit{Y})⩽\ln \left(\frac{M-1}{2}\right)\right\}}\right]+\frac{M-1}{2}{\mathbb{E}}_{P_{\mathit{X}}{P}_{\mathit{Y}}}\left[{\mathbb{1}}_{\left\{\iota (\mathit{X};\mathit{Y})>\ln \left(\frac{M-1}{2}\right)\right\}}\right].\hfill \end{array}$$

(51)

Using this notation, the following lemma states the DT bound.

Lemma 2

(Dependence testing bound [10]). Given a pair $(n$,$M)$∈${\mathbb{N}}^2$, the following holds for all $P_{\mathit{X}}\in ▵\left({\mathcal{X}}^n\right)$, with respect to the random transformation in (43):

$$\begin{array}{c}{\lambda }^{*}(n,M)⩽T(n,M,P_{\mathit{X}}),\hfill \end{array}$$

(52)

with the function T defined in (51).

Note that the input probability distribution $P_{\mathit{X}}$ in Lemma 2 can be chosen among all possible probability distributions $P_{\mathit{X}}\in ▵\left({\mathcal{X}}^n\right)$ to minimize the right-hand side of (52), which improves the bound. Note also that with some loss of optimality, the optimization domain can be restricted to the set of product probability distributions for which for all $\mathit{x}$∈${\mathcal{X}}^n$,

$$\begin{array}{c}{P}_{\mathit{X}}\left(\mathit{x}\right)=\prod _{t=1}^n{P}_X\left(x_t\right),\hfill \end{array}$$

(53)

with $P_X\in ▵\left(\mathcal{X}\right)$. Hence, subject to (44), the random variable $\iota (\mathit{X};\mathit{Y})$ in (50) can be written as the sum of i.i.d. random variables, i.e.,

$$\begin{array}{c}\iota (\mathit{X};\mathit{Y})=\sum _{t=1}^n\iota (X_t;Y_t).\hfill \end{array}$$

(54)

This observation motivates the application of the results of Section 2 to provide upper and lower bounds on the function T in (51), for some given values $(n,M)\in {\mathbb{N}}^2$ and a given distribution $P_{\mathit{X}}\in ▵\left({\mathcal{X}}^n\right)$ for the random transformation in (43) subject to (44). These bounds become significantly relevant when the exact value of $T(n,M,P_{\mathit{X}})$ cannot be calculated with respect to the random transformation in (43). In such a case, providing upper and lower bounds on $T(n,M,P_{\mathit{X}})$ helps in approximating its exact value subject to an error sufficiently small such that the approximation is relevant.

3.2.1. Normal Approximation

This section describes the normal approximation of the function T in (51). That is, the random variable $\iota (\mathit{X};\mathit{Y})$ is assumed to satisfy (54) and to follow a Gaussian distribution. More specifically, for all $P_X\in ▵\left(\mathcal{X}\right)$, let

$$\begin{array}{c}\mu \left(P_X\right)\triangleq {\mathbb{E}}_{P_X{P}_{Y|X}}\left[\iota (X;Y)\right],\hfill \end{array}$$

(55)

$$\begin{array}{c}\sigma \left(P_X\right)\triangleq {\mathbb{E}}_{P_X{P}_{Y|X}}\left[{\left(\iota (X;Y)-\mu \left(P_X\right)\right)}^2\right],\mathrm{and}\hfill \end{array}$$

(56)

$$\begin{array}{c}\xi \left(P_X\right)\triangleq c_1\left(\frac{{\mathbb{E}}_{P_X{P}_{Y|X}}\left[|\iota (X;Y)-\mu \left(P_X\right){|}^3\right]}{\sigma {\left(P_X\right)}^{\frac{3}{2}}}+c_2\right),\hfill \end{array}$$

(57)

with $c_1$ and $c_2$ defined in (23), be functions of the input distribution $P_X$. In particular, $\mu \left(P_X\right)$ and $\sigma \left(P_X\right)$ are respectively the first moment and the second central moment of the random variables $\iota (X_1;Y_1)$, $\iota (X_2;Y_2)$…$\iota (X_n;Y_n)$. Using this notation, consider the functions $D:{\mathbb{N}}^2\times ▵\left(\mathcal{X}\right)\to {\mathbb{R}}_{+}$ and $N:{\mathbb{N}}^2\times ▵\left(\mathcal{X}\right)\to {\mathbb{R}}_{+}$ such that for all $(n,M)\in {\mathbb{N}}^2$ and for all $P_X\in ▵\left(\mathcal{X}\right)$,

$$\begin{array}{c}D(n,M,P_X)=max\left\{0,\alpha \left(n,M,P_X\right)-\frac{\xi \left(P_X\right)}{\sqrt{n}}\right\},\mathrm{and}\hfill \end{array}$$

(58)

$$\begin{array}{c}N(n,M,P_X)=min\left\{1,\alpha \left(n,M,P_X\right)+\frac{5\xi \left(P_X\right)}{\sqrt{n}}+\frac{2\ln \left(2\right)}{\sigma {\left(P_X\right)}^{\frac{1}{2}}\sqrt{2n\pi }}\right\},\hfill \end{array}$$

(59)

where

$$\begin{array}{ccc}\alpha \left(n,M,P_X\right)\hfill & \triangleq & Q\left(\frac{n\mu \left(P_X\right)-\ln \left(\frac{M-1}{2}\right)}{\sqrt{n\sigma \left(P_X\right)}}\right).\hfill \end{array}$$

(60)

Using this notation, the following theorem introduces lower and upper bounds on the function T in (51).

Theorem 4.

Given a pair $(n,M)\in {\mathbb{N}}^2$, for all input distributions $P_{\mathit{X}}\in ▵\left({\mathcal{X}}^n\right)$ subject to (53), the following holds with respect to the random transformation in (43) subject to (44),

$$\begin{array}{c}D(n,M,P_X)⩽T(n,M,P_{\mathit{X}})⩽N(n,M,P_X),\hfill \end{array}$$

(61)

where the functions T, D and N are defined in (51), (58) and (59), respectively.

Proof. 

The proof of Theorem 4 is presented in [12]. Essentially, it relies on Theorem 1 for upper and lower bounding the terms ${\mathbb{E}}_{P_{\mathit{X}}{P}_{\mathit{Y}|\mathit{X}}}\left[{\mathbb{1}}_{\left\{\iota (\mathit{X};\mathit{Y})⩽\ln \left(\frac{M-1}{2}\right)\right\}}\right]$ in (51). The upper bound on ${\mathbb{E}}_{P_{\mathit{X}}{P}_{\mathit{Y}}}\left[{\mathbb{1}}_{\left\{\iota (\mathit{X};\mathit{Y})>\ln \left(\frac{M-1}{2}\right)\right\}}\right]$ in (51) follows from Lemma 47 in [10]. ☐

In [12], the function $\alpha (n,M,P_X)$ in (60) is often referred to as the normal approximation of $T(n,M,P_{\mathit{X}})$, which is indeed a language abuse. In Section 2.1, a comment is given on the fact that the lower and upper bounds, i.e., the functions D in (58) and N in (59), are often too far from the normal approximation $\alpha$ in (60).

3.2.2. Saddlepoint Approximation

This section describes an approximation of the function T in (51) by using the saddlepoint approximation of the CDF of the random variable $\iota (\mathit{X};\mathit{Y})$, as suggested in Section 2.2. Given a distribution $P_X\in ▵\left(\mathcal{X}\right)$, the moment generating function of $\iota (X;Y)$ is

$$\begin{array}{c}\phi (P_X,\theta )\triangleq {\mathbb{E}}_{P_X{P}_{Y|X}}\left[exp\left(\theta \iota (X;Y)\right)\right],\hfill \end{array}$$

(62)

with $\theta \in \mathbb{R}$. For all $P_X\in ▵\left(\mathcal{X}\right)$ and for all $\theta \in \mathbb{R}$, consider the following functions:

$$\begin{array}{c}\mu (P_X,\theta )\triangleq {\mathbb{E}}_{P_X{P}_{Y|X}}\left[\frac{\iota (X;Y)exp\left(\theta \iota (X;Y)\right)}{\phi (P_X,\theta )}\right],\hfill \end{array}$$

(63)

$$\begin{array}{c}V(P_X,\theta )\triangleq {\mathbb{E}}_{P_X{P}_{Y|X}}\left[{\left(\iota (X;Y)-\mu (P_X,\theta )\right)}^2\frac{exp\left(\theta \iota (X;Y)\right)}{\phi (P_X,\theta )}\right],\mathrm{and}\hfill \end{array}$$

(64)

$$\begin{array}{c}\xi (P_X,\theta )\triangleq c_1\left(\frac{{\mathbb{E}}_{P_X{P}_{Y|X}}\left[{\left|\iota (X;Y)-\mu (P_X,\theta )\right|}^3\frac{exp\left(\theta \iota (X;Y)\right)}{\phi (P_X,\theta )}\right]}{{\left(V(P_X,\theta )\right)}^{3/2}}+c_2\right),\hfill \end{array}$$

(65)

where $c_1$ and $c_2$ are defined in (23). Using this notation, consider the functions ${\beta }_1:{\mathbb{N}}^2\times \mathbb{R}\times ▵\left(\mathcal{X}\right)\to {\mathbb{R}}_{+}$ and ${\beta }_2:{\mathbb{N}}^2\times \mathbb{R}\times ▵\left(\mathcal{X}\right)\to {\mathbb{R}}_{+}$:

$$\begin{array}{c}{\beta }_1(n,M,\theta ,P_X)={\mathbb{1}}_{\{\theta >0\}}+{(-1)}^{{\mathbb{1}}_{\{\theta >0\}}}exp\left(n\ln \left(\phi (P_X,\theta )\right)-\theta \ln \left(\frac{M-1}{2}\right)+\frac{1}{2}{\theta }^{2}nV(P_X,\theta )\right)Q\left(\sqrt{nV(P_X,\theta )}\left|\theta \right|\right),\hfill \end{array}$$

(66)

and

$$\begin{array}{c}{\beta }_2(n,M,\theta ,P_X)\hfill \\ ={\mathbb{1}}_{\{\theta ⩽-1\}}+{(-1)}^{{\mathbb{1}}_{\{\theta ⩽-1\}}}exp\left(n\ln \left(\phi (P_X,\theta )\right)-\left(\theta +1\right)\ln \left(\frac{M-1}{2}\right)+\frac{1}{2}{(\theta +1)}^{2}nV(P_X,\theta )\right)Q\left(\sqrt{nV(P_X,\theta )}|\theta +1|\right).\hfill \end{array}$$

(67)

Note that ${\beta }_1$ is the saddlepoint approximation of the CDF of the random variable $\iota (\mathit{X};\mathit{Y})$ in (54) when $\mathit{X}$ and $\mathit{Y}$ follow the distribution $P_{\mathit{X}}{P}_{\mathit{Y}|\mathit{X}}$. Note also that ${\beta }_2$ is the saddlepoint approximation of the complementary CDF of the random variable $\iota (\mathit{X};\mathit{Y})$ in (54) when $\mathit{X}$ and $\mathit{Y}$ follow the distribution $P_{\mathit{X}}{P}_{\mathit{Y}}$.

Consider also the following functions:

$$\begin{array}{c}{G}_1(n,M,\theta ,P_X)={\beta }_1(n,M,\theta ,P_X)-\frac{2\xi (P_X,\theta )}{\sqrt{n}}exp\left(n\ln \left(\phi (P_X,\theta )\right)-\theta \ln \left(\frac{M-1}{2}\right)\right),\hfill \end{array}$$

(68)

$$\begin{array}{c}{G}_2(n,M,\theta ,P_X)={\beta }_2(n,M,\theta ,P_X)-\frac{2\xi (P_X,\theta )}{\sqrt{n}}exp\left(n\ln \left(\phi (P_X,\theta )\right)-(\theta +1)\ln \left(\frac{M-1}{2}\right)\right),\hfill \end{array}$$

(69)

$$\begin{array}{c}G(n,M,\theta ,P_X)=max\left\{0,G_1(n,M,\theta ,P_X)\right\}+\frac{M-1}{2}max\left\{0,G_2(n,M,\theta ,P_X)\right\},\mathrm{and}\hfill \end{array}$$

(70)

$$\begin{array}{c}S(n,M,\theta ,P_X)=min\left\{1,\beta \left(n,M,\theta ,P_X\right)+\frac{4\xi (P_X,\theta )}{\sqrt{n}}exp\left(n\ln \left(\phi (P_X,\theta )\right)-\theta \ln \left(\frac{M-1}{2}\right)\right)\right\},\hfill \end{array}$$

(71)

where,

$$\begin{array}{c}\beta (n,M,\theta ,P_X)={\beta }_1(n,M,\theta ,P_X)+\frac{M-1}{2}{\beta }_2(n,M,\theta ,P_X),\hfill \end{array}$$

(72)

with ${\beta }_1$ in (66) and ${\beta }_2$ in (67). Often, the function $\beta$ in (72) is referred to as the saddlepoint approximation of the function T in (51), which is indeed a language abuse.

The following theorem introduces new lower and upper bounds on the function T in (51).

Theorem 5.

Given a pair $(n,M)\in {\mathbb{N}}^2$, for all input distributions $P_{\mathit{X}}\in ▵\left({\mathcal{X}}^n\right)$ subject to (53), the following holds with respect to the random transformation in (43) subject to (44),

$$\begin{array}{c}G(n,M,\theta ,P_X)⩽T(n,M,P_{\mathit{X}})⩽S(n,M,\theta ,P_X)\hfill \end{array}$$

(73)

where θ is the unique solution in t to

$$\begin{array}{c}n\mu (P_X,t)=\ln \left(\frac{M-1}{2}\right),\hfill \end{array}$$

(74)

and the functions T, G, and S are defined in (51), (70), and (71).

Proof. 

The proof of Theorem 5 is provided in Appendix F. In a nutshell, the proof relies on Theorem 3 for independently bounding the terms ${\mathbb{E}}_{P_{\mathit{X}}{P}_{\mathit{Y}|\mathit{X}}}\left[{\mathbb{1}}_{\left\{\iota (\mathit{X};\mathit{Y})⩽\ln \left(\frac{M-1}{2}\right)\right\}}\right]$ and ${\mathbb{E}}_{P_{\mathit{X}}{P}_{\mathit{Y}}}\left[{\mathbb{1}}_{\left\{\iota (\mathit{X};\mathit{Y})>\ln \left(\frac{M-1}{2}\right)\right\}}\right]$ in (51). ☐

3.3. Meta Converse Bound

This section describes a lower bound on ${\lambda }^{*}(n,M)$, for a fixed pair $(n,M)\in {\mathbb{N}}^2$. Given two probability distributions $P_{\mathit{X}\mathit{Y}}\in ▵\left({\mathcal{X}}^n\times {\mathcal{Y}}^n\right)$ and $Q_{\mathit{Y}}\in ▵\left({\mathcal{Y}}^n\right)$, let the random variable $\tilde{\iota }\left(\mathit{X};\mathit{Y}|Q_{\mathit{Y}}\right)$ satisfy

$$\tilde{\iota }\left(\mathit{X};\mathit{Y}|Q_{\mathit{Y}}\right)\triangleq \ln \left(\frac{\mathrm{d}{P}_{\mathit{X}\mathit{Y}}}{\mathrm{d}{P}_{\mathit{X}}{Q}_{\mathit{Y}}}(\mathit{X},\mathit{Y})\right).$$

(75)

For all $(n$,M,$\gamma )$∈${\mathbb{N}}^2\times \mathbb{R}$ and for all probability distributions $P_{\mathit{X}}\in ▵\left({\mathcal{X}}^n\right)$ and $Q_{\mathit{Y}}\in ▵\left({\mathcal{Y}}^n\right)$, let the function $C:{\mathbb{N}}^2\times ▵\left({\mathcal{X}}^n\right)\times ▵\left({\mathcal{Y}}^n\right)\times {\mathbb{R}}_{+}\to {\mathbb{R}}_{+}$ be

$$\begin{array}{c}C(n,M,P_{\mathit{X}},Q_{\mathit{Y}},\gamma )\triangleq {\mathbb{E}}_{P_{\mathit{X}}{P}_{\mathit{Y}|\mathit{X}}}\left[{\mathbb{1}}_{\left\{\tilde{\iota }\left(\mathit{X};\mathit{Y}|Q_{\mathit{Y}}\right)⩽\ln \left(\gamma \right)\right\}}\right]+\gamma \left({\mathbb{E}}_{P_{\mathit{X}}{Q}_{\mathit{Y}}}\left[{\mathbb{1}}_{\left\{\tilde{\iota }\left(\mathit{X};\mathit{Y}|Q_{\mathit{Y}}\right)>\ln \left(\gamma \right)\right\}}\right]-\frac{1}{M}\right).\hfill \end{array}$$

(76)

Using this notation, the following lemma describes the MC bound.

Lemma 3

(MC Bound [10,13]). Given a pair $(n$,$M)$∈${\mathbb{N}}^2$, the following holds for all $Q_{\mathit{Y}}\in \mathsf{\Delta }\left({\mathcal{Y}}^n\right)$, with respect to the random transformation in (43):

$$\begin{array}{c}{\lambda }^{*}(n,M)⩾\underset{P_{\mathit{X}}\in \mathsf{\Delta }\left({\mathcal{X}}^n\right)}{inf}\underset{\gamma ⩾0}{max}C(n,M,P_{\mathit{X}},Q_{\mathit{Y}},\gamma ),\hfill \end{array}$$

(77)

where the function C is defined in (76).

Note that the output probability distribution $Q_{\mathit{Y}}$ in Lemma 3 can be chosen among all possible probability distributions $Q_{\mathit{Y}}\in ▵\left({\mathcal{Y}}^n\right)$ to maximize the right-hand side of (76), which improves the bound. Note also that, with some loss of optimality, the optimization domain can be restricted to the set of probability distributions for which for all $\mathit{y}$∈${\mathcal{Y}}^n$,

$$\begin{array}{c}{Q}_{\mathit{Y}}\left(\mathit{y}\right)=\prod _{t=1}^n{Q}_Y\left(y_t\right),\hfill \end{array}$$

(78)

with $Q_Y\in ▵\left(\mathcal{Y}\right)$. Hence, subject to (44), for all $\mathit{x}\in {\mathcal{X}}^n$, the random variable $\tilde{\iota }(\mathit{x};\mathit{Y}|Q_{\mathit{Y}})$ in (76) can be written as the sum of the independent random variables, i.e.,

$$\begin{array}{c}\tilde{\iota }(\mathit{x};\mathit{Y}|Q_{\mathit{Y}})=\sum _{t=1}^n\tilde{\iota }(x_t;Y_t|Q_Y).\hfill \end{array}$$

(79)

With some loss of generality, the focus is on a channel transformation of the form in (43) for which the following condition holds: The infimum in (77) is achieved by a product distribution, i.e., $P_{\mathit{X}}$ is of the form in (53), when the probability distribution $Q_{\mathit{Y}}$ satisfies (78). Note that this condition is met by memoryless channels such as the BSC, the AWGN and S$\alpha$S channels with binary antipodal inputs, i.e., input alphabets are of the form $\mathcal{X}=\{a,-a\}$, with $a\in \mathbb{R}$. This follows from the fact that the random variable $\tilde{\iota }(\mathit{x};\mathit{Y}|Q_{\mathit{Y}})$ is invariant of the choice of $\mathit{x}\in {\mathcal{X}}^n$ when the probability distribution $Q_{\mathit{Y}}$ satisfies (78) and for all $y\in \mathcal{Y}$,

$$\begin{array}{c}{Q}_Y\left(y\right)=\frac{P_{Y|X}\left(y\right|-a)+P_{Y|X}\left(y\right|a)}{2}.\hfill \end{array}$$

(80)

Under these conditions, the random variable $\tilde{\iota }(\mathit{X};\mathit{Y}|Q_{\mathit{Y}})$ in (76) can be written as the sum of i.i.d. random variables, i.e.,

$$\begin{array}{c}\tilde{\iota }(\mathit{X};\mathit{Y}|Q_{\mathit{Y}})=\sum _{t=1}^n\tilde{\iota }(X_t;Y_t|Q_Y).\hfill \end{array}$$

(81)

This observation motivates the application of the results of Section 2 to provide upper and lower bounds on the function C in (76), for some given values $(n,M)\in {\mathbb{N}}^2$ and given distributions $P_{\mathit{X}}\in ▵\left({\mathcal{X}}^n\right)$ and $Q_{\mathit{Y}}\in ▵\left({\mathcal{Y}}^n\right)$. These bounds become significantly relevant when the exact value of $C(n,M,P_{\mathit{X}},Q_{\mathit{Y}},\gamma )$ cannot be calculated with respect to the random transformation in (43). In such a case, providing upper and lower bounds on $C(n,M,P_{\mathit{X}},Q_{\mathit{Y}},\gamma )$ helps in approximating its exact value subject to an error sufficiently small such that the approximation is relevant.

3.3.1. Normal Approximation

This section describes the normal approximation of the function C in (76), that is to say, the random variable $\tilde{\iota }(\mathit{X};\mathit{Y}|Q_{\mathit{Y}})$ is assumed to satisfy (81) and to follow a Gaussian distribution. More specifically, for all $(P_X,Q_Y)\in ▵\left(\mathcal{X}\right)\times ▵\left(\mathcal{Y}\right)$, let

$$\begin{array}{c}\tilde{\mu }(P_X,Q_Y)\triangleq {\mathbb{E}}_{P_X{P}_{Y|X}}\left[\tilde{\iota }(X;Y|Q_Y)\right],\hfill \end{array}$$

(82)

$$\begin{array}{c}\tilde{\sigma }(P_X,Q_Y)\triangleq {\mathbb{E}}_{P_X{P}_{Y|X}}\left[{\left(\tilde{\iota }(X;Y|Q_Y)-\tilde{\mu }(P_X,Q_Y)\right)}^2\right],\mathrm{and}\hfill \end{array}$$

(83)

$$\begin{array}{c}\tilde{\xi }(P_X,Q_Y)\triangleq c_1\left(\frac{{\mathbb{E}}_{P_X{P}_{Y|X}}\left[|\tilde{\iota }(X;Y|Q_Y)-\tilde{\mu }(P_X,Q_Y){|}^3\right]}{{\left(\tilde{\sigma }(P_X,Q_Y)\right)}^{3/2}}+c_2\right)\hfill \end{array}$$

(84)

with $c_1$ and $c_2$ defined in (23), be functions of the input and output distributions $P_X$ and $Q_Y$, respectively. In particular, $\tilde{\mu }(P_X,Q_Y)$ and $\tilde{\sigma }(P_X,Q_Y)$ are respectively the first moment and the second central moment of the random variables $\tilde{\iota }(X_1;Y_1|Q_Y),\tilde{\iota }(X_2;Y_2|Q_Y),\dots \tilde{\iota }(X_n;Y_n|Q_Y)$. Using this notation, consider the functions $\tilde{D}:{\mathbb{N}}^2\times ▵\left(\mathcal{X}\right)\times ▵\left(\mathcal{Y}\right)\times {\mathbb{R}}_{+}\to {\mathbb{R}}_{+}$ and $\tilde{N}:{\mathbb{N}}^2\times ▵\left(\mathcal{X}\right)\times ▵\left(\mathcal{Y}\right)\times {\mathbb{R}}_{+}\to {\mathbb{R}}_{+}$ such that, for all $(n,M,\gamma )\in {\mathbb{N}}^2\times {\mathbb{R}}_{+}$ and for all $P_X\in ▵\left(\mathcal{X}\right)$ and for all $Q_Y\in ▵\left(\mathcal{Y}\right)$,

$$\begin{array}{c}\tilde{D}(n,M,P_X,Q_Y,\gamma )=max\left\{0,\tilde{\alpha }\left(n,M,P_X,Q_Y,\gamma \right)-\frac{\tilde{\xi }(P_X,Q_Y)}{\sqrt{n}}\right\},\mathrm{and}\hfill \end{array}$$

(85)

$$\begin{array}{c}\tilde{N}(n,M,P_X,Q_Y,\gamma )=min\left\{1,\tilde{\alpha }\left(n,M,P_X,Q_Y,\gamma \right)+\frac{5\tilde{\xi }(P_X,Q_Y)}{\sqrt{n}}+\frac{2\ln \left(2\right)}{\tilde{\sigma }{(P_X,Q_Y)}^{\frac{1}{2}}\sqrt{2n\pi }}\right\},\hfill \end{array}$$

(86)

where

$$\begin{array}{ccc}\tilde{\alpha }\left(n,M,P_X,Q_Y,\gamma \right)\hfill & \triangleq & Q\left(\frac{n\tilde{\mu }(P_X,Q_Y)-\ln \left(\gamma \right)}{\sqrt{n\tilde{\sigma }(P_X,Q_Y)}}\right)-\frac{\gamma }{M}.\hfill \end{array}$$

(87)

Using this notation, the following theorem introduces lower and upper bounds on the function C in (76).

Theorem 6.

Given a pair $(n,M)\in {\mathbb{N}}^2$, for all input distributions $P_{\mathit{X}}\in ▵\left({\mathcal{X}}^n\right)$ subject to (53), for all output distributions $Q_{\mathit{Y}}\in ▵\left({\mathcal{Y}}^n\right)$ subject to (78), and for all $\gamma ⩾0$, the following holds with respect to the random transformation in (43) subject to (44),

$$\begin{array}{c}\tilde{D}(n,M,P_X,Q_Y,\gamma )⩽C(n,M,P_{\mathit{X}},Q_{\mathit{Y}},\gamma )⩽\tilde{N}(n,M,P_X,Q_Y,\gamma ),\hfill \end{array}$$

(88)

where the functions C, $\tilde{D}$, and $\tilde{N}$ are defined in (76), (85), and (86), respectively.

Proof. 

The proof of Theorem 6 is partially presented in [10]. Essentially, it relies on Theorem 1 for upper and lower bounding the term ${\mathbb{E}}_{P_{\mathit{X}}{P}_{\mathit{Y}|\mathit{X}}}\left[{\mathbb{1}}_{\left\{\tilde{\iota }(\mathit{X};\mathit{Y}|Q_{\mathit{Y}})⩽\ln \left(\gamma \right)\right\}}\right]$ in (76); and using Lemma 47 in [10] for upper bounding the term ${\mathbb{E}}_{P_{\mathit{X}}{Q}_{\mathit{Y}}}\left[{\mathbb{1}}_{\left\{\tilde{\iota }(\mathit{X};\mathit{Y}|Q_{\mathit{Y}})>\ln \left(\gamma \right)\right\}}\right]$ in (76). ☐

The function $\tilde{\alpha }\left(n,M,P_X,Q_Y,\gamma \right)$ in (87) is often referred to as the normal approximation of $C(n,M,P_{\mathit{X}})$, which is indeed a language abuse. In Section 2.1, a comment is given on the fact that the lower and upper bounds on the normal approximation, i.e., the functions $\tilde{D}$ in (85) and $\tilde{N}$ in (86), are often too far from the normal approximation $\tilde{\alpha }$ in (87).

3.3.2. Saddlepoint Approximation

This section describes an approximation of the function C in (76) by using the saddlepoint approximation of the CDF of the random variable $\tilde{\iota }(\mathit{X};\mathit{Y}|Q_{\mathit{Y}})$, as suggested in Section 2.2. Given two distributions $P_X\in ▵\left(\mathcal{X}\right)$ and $Q_Y\in ▵\left(\mathcal{Y}\right)$, let the random variable $\tilde{\iota }(X;Y|Q_Y)$ satisfy

$$\begin{array}{c}\tilde{\iota }(X;Y|Q_Y)\triangleq \ln \left(\frac{\mathrm{d}{P}_X{P}_{Y|X}}{\mathrm{d}{P}_X{Q}_Y}(X,Y)\right),\hfill \end{array}$$

(89)

where $P_{Y|X}$ is in (44). The moment generating function of $\tilde{\iota }(X;Y|Q_Y)$ is

$$\begin{array}{c}\tilde{\phi }(P_X,Q_Y,\theta )\triangleq {\mathbb{E}}_{P_X{P}_{Y|X}}\left[exp\left(\theta \tilde{\iota }(X;Y|Q_Y)\right)\right],\hfill \end{array}$$

(90)

with $\theta \in \mathbb{R}$. For all $P_X\in ▵\left(\mathcal{X}\right)$ and $Q_Y\in ▵\left(\mathcal{Y}\right)$, and for all $\theta \in \mathbb{R}$, consider the following functions:

$$\begin{array}{c}\tilde{\mu }(P_X,Q_Y,\theta )\triangleq {\mathbb{E}}_{P_X{P}_{Y|X}}\left[\frac{\tilde{\iota }(X;Y|Q_Y)exp\left(\theta \tilde{\iota }(X;Y|Q_Y)\right)}{\tilde{\phi }(P_X,Q_Y,\theta )}\right],\hfill \end{array}$$

(91)

$$\begin{array}{c}\tilde{V}(P_X,Q_Y,\theta )\triangleq {\mathbb{E}}_{P_X{P}_{Y|X}}\left[{\left(\tilde{\iota }(X;Y|Q_Y)-\tilde{\mu }(P_X,Q_Y,\theta )\right)}^2\frac{exp\left(\theta \tilde{\iota }(X;Y|Q_Y)\right)}{\tilde{\phi }(P_X,Q_Y,\theta )}\right],\mathrm{and}\hfill \end{array}$$

(92)

$$\begin{array}{c}\tilde{\xi }(P_X,Q_Y,\theta )\triangleq c_1\left(\frac{{\mathbb{E}}_{P_X{P}_{Y|X}}\left[{\left|\tilde{\iota }(X;Y|Q_Y)-\tilde{\mu }(P_X,Q_Y,\theta )\right|}^3\frac{exp\left(\theta \tilde{\iota }(X;Y|Q_Y)\right)}{\tilde{\phi }(P_X,Q_Y,\theta )}\right]}{{\left(\tilde{V}(P_X,Q_Y,\theta )\right)}^{3/2}}+c_2\right),\hfill \end{array}$$

(93)

where $c_1$ and $c_2$ are defined in (23). Using this notation, consider the functions ${\tilde{\beta }}_1:\mathbb{N}\times {\mathbb{R}}_{+}\times \mathbb{R}\times ▵\left(\mathcal{X}\right)\times ▵\left(\mathcal{Y}\right)\to {\mathbb{R}}_{+}$ and ${\tilde{\beta }}_2:\mathbb{N}\times {\mathbb{R}}_{+}\times \mathbb{R}\times ▵\left(\mathcal{X}\right)\times ▵\left(\mathcal{Y}\right)\to {\mathbb{R}}_{+}$:

$$\begin{array}{c}{\tilde{\beta }}_1(n,\gamma ,\theta ,P_X,Q_Y)\hfill \end{array}$$

$$\begin{array}{c}={\mathbb{1}}_{\{\theta >0\}}+{(-1)}^{{\mathbb{1}}_{\{\theta >0\}}}exp\left(n\ln \left(\tilde{\phi }(P_X,Q_Y,\theta )\right)-\theta \ln \left(\gamma \right)+\frac{1}{2}{\theta }^{2}n\tilde{V}(P_X,Q_Y,\theta )\right)Q\left(\sqrt{n\tilde{V}(P_X,Q_Y,\theta )}\left|\theta \right|\right),\mathrm{and}\hfill \\ {\tilde{\beta }}_2(n,\gamma ,\theta ,P_X,Q_Y)\hfill \\ ={\mathbb{1}}_{\{\theta ⩽-1\}}+{(-1)}^{{\mathbb{1}}_{\{\theta ⩽-1\}}}exp\left(n\ln \left(\tilde{\phi }(P_X,Q_Y,\theta )\right)-\left(\theta +1\right)\ln \left(\gamma \right)+\frac{1}{2}{(\theta +1)}^{2}n\tilde{V}(P_X,Q_Y,\theta )\right)\hfill \end{array}$$

(94)

$$\begin{array}{c}Q\left(\sqrt{n\tilde{V}(P_X,Q_Y,\theta )}|\theta +1|\right).\hfill \end{array}$$

(95)

Note that ${\tilde{\beta }}_1$ and ${\tilde{\beta }}_2$ are the saddlepoint approximation of the CDF and the complementary CDF of the random variable $\tilde{\iota }(\mathit{X};\mathit{Y}|Q_{\mathit{Y}})$ in (81) when $\left(\mathit{X},\mathit{Y}\right)$ follows the distribution $P_{\mathit{X}}{P}_{\mathit{Y}|\mathit{X}}$ and $P_{\mathit{X}}{Q}_{\mathit{Y}}$, respectively. Consider also the following functions:

$$\begin{array}{c}{\tilde{G}}_1(n,\gamma ,\theta ,P_X,Q_Y)={\tilde{\beta }}_1(n,\gamma ,\theta ,P_X,Q_Y)-\frac{2\tilde{\xi }(P_X,Q_Y,\theta )}{\sqrt{n}}exp\left(n\ln \left(\tilde{\phi }(P_X,Q_Y,\theta )\right)-\theta \ln \left(\gamma \right)\right),\hfill \end{array}$$

(96)

$$\begin{array}{c}{\tilde{G}}_2(n,\gamma ,\theta ,P_X,Q_Y)={\tilde{\beta }}_2(n,\gamma ,\theta ,P_X,Q_Y)-\frac{2\tilde{\xi }(P_X,Q_Y,\theta )}{\sqrt{n}}exp\left(n\ln \left(\tilde{\phi }(P_X,Q_Y,\theta )\right)-(\theta +1)\ln \left(\gamma \right)\right),\hfill \end{array}$$

(97)

$$\begin{array}{c}\tilde{G}(n,\gamma ,\theta ,P_X,Q_Y,M)=max\left\{0,{\tilde{G}}_1(n,\gamma ,\theta ,P_X,Q_Y)\right\}+\gamma max\left\{0,{\tilde{G}}_2(n,\gamma ,\theta ,P_X,Q_Y)\right\}-\frac{\gamma }{M},\hfill \end{array}$$

(98)

$$\begin{array}{c}\tilde{S}(n,\gamma ,\theta ,P_X,Q_Y,M)=min\left\{1,\tilde{\beta }\left(n,\gamma ,\theta ,P_X,Q_Y,M\right)+\frac{4\tilde{\xi }(P_X,Q_Y,\theta )}{\sqrt{n}}exp\left(n\ln \left(\tilde{\phi }(P_X,Q_Y,\theta )\right)-\theta \ln \left(\gamma \right)\right)\right\},\hfill \end{array}$$

(99)

and

$$\tilde{\beta }(n,\gamma ,\theta ,P_X,Q_Y,M)={\tilde{\beta }}_1(n,\gamma ,\theta ,P_X,Q_Y)+\gamma {\tilde{\beta }}_2(n,\gamma ,\theta ,P_X,Q_Y)-\frac{\gamma }{M}.$$

(100)

The function $\tilde{\beta }(n,\gamma ,\theta ,P_X,Q_Y,M)$ in (100) is referred to as the saddlepoint approximation of the function C in (76), which is indeed a language abuse.

The following theorem introduces new lower and upper bounds on the function C in (76).

Theorem 7.

Given a pair $(n,M)\in {\mathbb{N}}^2$, for all input distributions $P_{\mathit{X}}\in ▵\left({\mathcal{X}}^n\right)$ subject to (53), for all output distributions $Q_{\mathit{Y}}\in ▵\left({\mathcal{Y}}^n\right)$ subject to (81) such that for all $x\in \mathcal{X}$, $P_{Y|X=x}$ is absolutely continuous with respect to $Q_Y$, for all $\gamma ⩾0$, the following holds with respect to the random transformation in (43) subject to (44),

$$\begin{array}{c}\tilde{G}(n,\gamma ,\theta ,P_X,Q_Y,M)⩽C(n,M,P_{\mathit{X}},Q_{\mathit{Y}},\gamma )⩽\tilde{S}(n,\gamma ,\theta ,P_X,Q_Y,M)\hfill \end{array}$$

(101)

where θ is the unique solution in t to

$$\begin{array}{c}n\mu (P_X,t)=\ln \left(\gamma \right),\hfill \end{array}$$

(102)

and the functions C, $\tilde{G}$, and $\tilde{S}$ are defined in (76), (98) and (99).

Proof. 

The proof of Theorem 7 is provided in Appendix G. ☐

Note that, in (101), the parameter $\gamma$ can be optimized as in (77).

3.4. Numerical Experimentation

The normal and the saddlepoint approximations of the DT and MC bounds as well as their corresponding upper and lower bounds presented from Section 3.2.1 to Section 3.3.2 are studied in the cases of the BSC, the AWGN channel, and the S$\alpha$S channel. The latter is defined by the random transformation in (43) subject to (44) and for all $(x,y)\in \mathcal{X}\times \mathcal{Y}$:

$$\begin{array}{c}{P}_{Y|X}\left(y\right|x)=P_Z(y-x),\hfill \end{array}$$

(103)

where $P_Z$ is a probability distribution satisfying for all $t\in \mathbb{R}$,

$$\begin{array}{c}{\mathbb{E}}_{P_Z}\left[exp\left(itZ\right)\right]=exp\left(-{\left|\sigma t\right|}^{\alpha }\right),\hfill \end{array}$$

(104)

with $i=\sqrt{-1}$. The reals $\alpha \in (0,2]$ and $\sigma \in {\mathbb{R}}_{+}$ in (104) are parameters of the S$\alpha$S channel.

In the following figures, Figure 3, Figure 4 and Figure 5, the channel inputs are discrete $\mathcal{X}=\{-1,1\}$, $P_X$ is the uniform distribution, and $\theta$ is chosen to be the unique solution to t in (74) or (102) depending on whether the DT or MC bound is considered. For the results relative to the MC bound, $Q_Y$ is chosen to be equal to the distribution $P_Y$, i.e., the marginal of $P_X{P}_{Y|X}$. The parameter $\gamma$ is chosen to maximize the function $C(n,2^{nR},P_X,Q_Y,\gamma )$ in (76). The plots in Figure 3a, Figure 4a and Figure 5a illustrate the function $T(n,2^{nR},P_X)$ in (51) as well as the bounds in Theorems 4 and 5. Figure 3b, Figure 4b and Figure 5b illustrate the function C in (76) and the bounds in Theorems 6 and 7. The normal approximations, i.e, $\alpha \left(n,2^{nR},P_X\right)$ in (60) and $\tilde{\alpha }\left(n,2^{nR},P_X,Q_Y,\gamma \right)$ in (87), of the DT and MC bounds, respectively, are plotted in black diamonds. The upper bounds, i.e., $N\left(n,2^{nR},P_X\right)$ in (59) and $\tilde{N}\left(n,2^{nR},P_X,Q_Y,\gamma \right)$ in (86), are plotted in blue squares. The lower bounds of the DT and MC bounds, i.e., $D\left(n,M,P_X\right)$ in (58) and $\tilde{D}\left(n,2^{nR},P_X,Q_Y,\gamma \right)$ in (85), are non-positive in these cases, and thus do not appear in the figures. The saddlepoint approximations of the DT and MC bounds, i.e., $\beta \left(n,2^{nR},\theta ,P_X\right)$ in (72) and $\tilde{\beta }\left(n,\gamma ,\theta ,P_X,Q_Y,2^{nR}\right)$ in (100), respectively, are plotted in black stars. The upper bounds, i.e., $S\left(n,2^{nR},\theta ,P_X\right)$ in (71) and $\tilde{S}\left(n,\gamma ,\theta ,P_X,Q_Y,2^{nR}\right)$ in (99), are plotted in blue upward-pointing triangles. The lower bounds, i.e., $G\left(n,2^{nR},\theta ,P_X\right)$ in (70) and $\tilde{G}\left(n,\gamma ,\theta ,P_X,Q_Y,2^{nR}\right)$ in (98), are plotted in red downward-pointing triangles.

Figure 3. Normal and saddlepoint approximations to the functions T (Figure 3a) in (51) and C (Figure 3b) in (76) as functions of the blocklength n for the case of a BSC with cross-over probability $\delta =0.11$. The information rate is $R=0.32$ and $R=0.42$ bits per channel use for Figure 3a,b, respectively. The channel input distribution $P_X$ is chosen to be the uniform distribution, the output distribution $Q_Y$ is chosen to be the channel output distribution $P_Y$, and the parameter $\gamma$ is chosen to maximize C in (76). The parameter $\theta$ is chosen to be respectively the unique solution to t in (74) in Figure 3a and in (102) in Figure 3b.

Figure 4. Normal and saddlepoint approximations to the functions T (Figure 4a) in (51) and C (Figure 4b) in (76) as functions of the blocklength n for the case of a real-valued AWGN channel with discrete channel inputs, $\mathcal{X}=\{-1,1\}$, signal to noise ratio $\mathrm{SNR}=1$, and information rate $R=0.425$ bits per channel use. The channel input distribution $P_X$ is chosen to be the uniform distribution, the output distribution $Q_Y$ is chosen to be the channel output distribution $P_Y$, and the parameter $\gamma$ is chosen to maximize C in (76). The parameter $\theta$ is respectively chosen to be the unique solution to t in (74) in Figure 4a and in (102) in Figure 4b.

Figure 5. Normal and saddlepoint approximation to the functions T (Figure 5a) in (51) and C (Figure 5b) in (76) as functions of the blocklength n for the case of a real-valued symmetric $\alpha$-stable noise channel with discrete channel inputs $\mathcal{X}=\{-1,1\}$, shape parameter $\alpha =1.4$, dispersion parameter $\sigma =0.6$, and information rate $R=0.38$ bits per channel use. The channel input distribution $P_X$ is chosen to be the uniform distribution, the output distribution $Q_Y$ is chosen to be the channel output distribution $P_Y$, and the parameter $\gamma$ is chosen to maximize C in (76). The parameter $\theta$ is respectively chosen to be the unique solution to t in (74) in Figure 5a and in (102) in Figure 5b.

Figure 3 illustrates the case of a BSC with cross-over probability $\delta =0.11$. The information rates are chosen to be $R=0.32$ and $R=0.42$ bits per channel use in Figure 3a,b, respectively. The functions T and C can be calculated exactly and thus they are plotted in magenta asterisks in Figure 3a,b, respectively. In these figures, it can be observed that the saddlepoint approximations of the DT and MC bounds, i.e., $\beta$ and $\tilde{\beta }$, respectively, overlap with the functions T and C. These observations are in line with those reported in [13]. Therein, the saddlepoint approximations of the RCU bound and the MC bound are both shown to be precise approximations. Alternatively, the normal approximations of the DT and MC bounds, i.e., $\alpha$ and $\tilde{\alpha }$, do not overlap with T and C respectively.

In Figure 3, it can be observed that the new bounds on the DT and MC provided in Theorems 5 and 7, respectively, are tighter than those in Theorems 4 and 6. Indeed, the upper-bounds N and $\tilde{N}$ on the DT and MC bounds derived from the normal approximations $\alpha$ and $\tilde{\alpha }$, are several order of magnitude above T and C, respectively. This observation remains valid for AWGN channels in Figure 4 and S$\alpha$S channels in Figure 5, respectively. Note that, in Figure 3a, for $n>1000$, the normal approximation $\alpha$ is below the lower bound G showing that approximating T by $\alpha$ is too optimistic. These results show that the use of the Berry–Esseen Theorem to approximate the DT and MC bounds may lead to erroneous conclusions due to the uncontrolled error made on the approximation.

Figure 4 and Figure 5 illustrate the cases of a real-valued AWGN channel and a S$\alpha$S channel, respectively. The signal-to-noise ratio (SNR) is $\mathrm{SNR}=1$ for the AWGN channel. The information rate is $R=0.425$ bits per channel use for the AWGN channel and $R=0.38$ bits per channel use for the S$\alpha$S channel with $(\alpha ,\sigma )=(1.4,0.6)$. In both cases, the functions T in (51) and C in (76) can not be computed explicitly and hence does not appear in Figure 4 and Figure 5. In addition, the lower bounds $D\left(n,M,P_X\right)$ and $\tilde{D}\left(n,2^{nR},P_X,Q_Y,\gamma \right)$ obtained from Theorems 4 and 6 are non-positive in these cases, and thus, do not appear on these figures.

In Figure 4, note that the saddlepoint approximations, $\beta$ and $\tilde{\beta }$, are well bounded by Theorems 5 and 7 for a large range of blocklengths. Alternatively, the lower bounds D and $\tilde{D}$ based on the normal approximation do not even exist in that case.

In Figure 5, note that the upper bounds S and $\tilde{S}$ on the DT and MC respectively are relatively tight compared to those in AWGN channel case. This characteristic is of a particular importance in a channel such as S$\alpha$S channel, where the DT and MC bounds remain computable only by Monte Carlo simulations.

4. Discussion and Further Work

One of the main results of this work is Theorem 3, which gives an upper bound on the error induced by the saddlepoint approximation of the CDF of a sum of i.i.d. random variables. This result paves the way to study channel coding problems at any finite blocklength and any constraint on the DEP. In particular, Theorem 3 is used to bound the DT and MC bounds in point-to-point memoryless channels. This leads to tighter bounds than those obtained from Berry–Esseen Theorem (Theorem 1), cf., examples in Section 3.4, particularly for the small values of the DEP.

The bound on the approximation error presented in Theorem 2 uses a triangle inequality in the proof of Lemma A1, which is loose. This is essentially the reason why Theorem 2 is not reduced to the Berry–Esseen Theorem when the parameter $\theta$ is equal to zero. An interesting extension of this work is to tighten the inequality in Lemma A1 such that the Berry–Esseen Theorem can be obtained as a special case of Theorem 2, i.e., when $\theta =0$. If such improvement on Theorem 2 is possible, Theorem 3 will be significantly improved and it would be more precise everywhere and in particular in the vicinity of the mean of the sum in (1).