Variable-Length Resolvability for General Sources and Channels ^†

Abstract

We introduce the problem of variable-length (VL) source resolvability, in which a given target probability distribution is approximated by encoding a VL uniform random number, and the asymptotically minimum average length rate of the uniform random number, called the VL resolvability, is investigated. We first analyze the VL resolvability with the variational distance as an approximation measure. Next, we investigate the case under the divergence as an approximation measure. When the asymptotically exact approximation is required, it is shown that the resolvability under two kinds of approximation measures coincides. We then extend the analysis to the case of channel resolvability, where the target distribution is the output distribution via a general channel due to a fixed general source as an input. The obtained characterization of channel resolvability is fully general in the sense that, when the channel is just an identity mapping, it reduces to general formulas for source resolvability. We also analyze the second-order VL resolvability.

1. Introduction

Generating a random number subject to a given probability distribution has a number of applications, such as in information security, statistical machine learning, and computer science. From the viewpoint of information theory, random number generation may be considered to be a transformation (encoding) of sequences emitted from a given source with coin distribution into other sequences with target distribution via a deterministic mapping [1,2,3]. Among others, there have been two major types of problems of random number generation: intrinsic randomness [4,5] and (source) resolvability [6,7]. In the former case, a fixed-length (FL) uniform random number is extracted from an arbitrary coin distribution, and we want to find the maximum achievable rate of such uniform random numbers. In the latter case, in contrast, an FL uniform random number used as a coin distribution is encoded to approximate a given target distribution, and we want to find the minimum achievable rate of such uniform random numbers. Thus, there is a duality between these two problems.

The problem of intrinsic randomness has been extended to the case of variable-length (VL) uniform random numbers, for which the length of random numbers may vary. This problem, referred to as the VL intrinsic randomness, was first introduced by Vembu and Verdú [5] for a finite source alphabet and later extended by Han [4] to a countably infinite alphabet. This problem was actually motivated because, in many practical situations, it is indispensable to consider cases where FL uniform random numbers are not available, and, instead, VL uniform random numbers are available; typically, this is in cases where we work with Elias’ universal random numbers [1]. The use of such uniform random numbers is expected generally to increase the achievable average length rate for intrinsic randomness. Then, the following natural question may be raised: Can we indeed lower the average length rate needed in the “resolvability” problem by using VL random numbers? The answer is “yes”. Despite the duality between these two kinds of problems for random number generation, the VL counterpart in the resolvability problem has not been discussed, where we focus on this problem.

We introduce the problem of VL source/channel resolvability, where a given target probability distribution is to be approximated by encoding a VL uniform random number. Distance measures between the target distribution and an approximated distribution are used to measure the fineness of the approximation. We first analyze the fundamental limit on VL source resolvabilities with the variational distance as an approximation measure in Section 3. We use the smooth Shannon entropy, which is a version of smooth Rényi entropy [8], to characterize the $\delta$-source resolvability, which is defined as the minimum achievable length rate of uniform random numbers with an asymptotic distance of less than or equal to $\delta \in [0,1)$. In the proof of the direct part, we will develop a simple version of information spectrum slicing [2], in which each “sliced” information density quantized to an integer is approximated by an FL uniform random number. Due to the simplicity of the method, the analysis with variational distance is first facilitated. As an important implication of general formulas for the $\delta$-source resolvability, it is shown that the minimum resolvability rate of VL resolvability is equal to $(1-\delta )$ times that of FL resolvability when the source is stationary and memoryless or is even with one-point spectrum (cf. Corollary 1). This result indicates an advantage of the use of a VL uniform random number when $\delta >0$ because we can make the VL resolvability rate strictly smaller than an FV one. We then extend these analyses to the case under the (unnormalized) divergence as an approximation measure in Section 4. When $\delta =0$, that is, when the asymptotically exact approximation is required, it is shown that the 0-source resolvabilities under two kinds of approximation measures coincide with each other.

In Section 5, we then consider the problem of channel resolvability [6,9,10], in which not only a source but also a channel is fixed, and the output distribution via the channel is now the target of approximation. This problem, also referred to as the problem of output approximation, provides a powerful tool to analyze the fundamental limits of various problems in information theory and information security. Some such examples include identification codes [11,12,13], distributed hypothesis testing [14], message authentication [15], secret key generation [16], and coding for secure communication [17,18,19]. We consider two types of problems in which either a general source (mean-channel resolvability) or a VL uniform random number (VL channel resolvability) is used as a coin distribution. It is shown that the formulas established are equal for both coin distributions. In the special case that the channel is the identity mapping, the formulas established reduce to those in source resolvability as established in Section 3 and Section 4.

From Section 3, Section 4 and Section 5, the so-called first-order resolvability rates are analyzed, and the next important step may be the second-order analysis. Second-order analyses for various coding problems were initiated by Strassen [20] and have been studied in the past decade or so (cf. [21,22,23,24,25,26,27,28,29,30,31]). We also analyze the second-order fundamental limits of the VL channel/source resolvability in Section 6. In this paper, it is shown that the VL $\delta$-source resolvability under the variational distance is equal to the minimum achievable rate of fixed-to-variable-length source codes with an error probability of less than or equal to $\delta$. It is demonstrated that this close relationship provides a single-letter characterization for the first- and second-order source resolvability under the variational distance when the source is stationary and memoryless. It is worth noting that second-order analyses for the VL setting are relatively few, compared to those in the FL setting. The second-order formulas established in this paper are of importance from this perspective, too.

The remainder of the paper is organized as follows: Section 2 reviews the problem of FL source resolvability and the relations between the minimum resolvability rate and the minimum coding rate of FL source codes. Section 3 formally introduces the problem of VL source resolvability with variational distance as an approximation measure. Then, Section 4 discusses VL source resolvability with divergence as an approximation measure, and Section 5 generalizes the settings to channel resolvability. Section 6 investigates the second-order fundamental limits of the VL channel/source resolvability. Section 7 concludes the paper with a discussion of possible extensions.

2. FL Resolvability: Review

Let $\mathcal{U}=\{1,2,\dots ,K\}$ be a finite alphabet of size K, and let $\mathcal{X}$ be a finite or countably infinite alphabet. Let $\mathit{X}={\left\{X^n\right\}}_{n=1}^{\infty }$ be a general source [2], where $P_{X^n}$ is a probability distribution on ${\mathcal{X}}^n$. We do not impose any assumptions, such as stationarity or ergodicity. In this paper, we identify $X^n$ with its probability distribution $P_{X^n}$, and these symbols are used interchangeably.

We first review the problem of FL (source) resolvability [2] using the variational distance as an approximation measure. Let $U_{M_n}$ denote the uniform random number, which is a random variable uniformly distributed over ${\mathcal{U}}_{M_n}:=\{1,\dots ,M_n\}$. Consider the problem of approximating the target distribution $P_{X^n}$ by using $U_{M_n}$ as a coin distribution via a deterministic mapping ${\phi }_n:\{1,\dots ,M_n\}\to {\mathcal{X}}^n$. Denoting ${\tilde{X}}^n={\phi }_n\left(U_{M_n}\right)$, we want to make $P_{{\tilde{X}}^n}$ approximate $P_{X^n}$ (cf. Figure 1). A standard choice of the performance measure for approximation is

$$\begin{array}{c}\hfill d(P_{X^n},P_{{\tilde{X}}^n}):=\frac{1}{2}\sum _{\mathit{x}\in {\mathcal{X}}^n}|P_{X^n}\left(\mathit{x}\right)-P_{{\tilde{X}}^n}\left(\mathit{x}\right)|,\end{array}$$

(1)

which is referred to as the variational distance between $P_{X^n}$ and $P_{{\tilde{X}}^n}$. It is easily seen that

$$\begin{array}{cc}\hfill 0& \le d(P_{X^n},P_{{\tilde{X}}^n})\le 1.\hfill \end{array}$$

(2)

Let us now review the problem for source resolvability. Throughout this paper, logarithms are of the base K.

Definition 1

(FL resolvability). A resolution rate $R\ge 0$ is said to be FL achievable or simply f-achievable (under the variational distance) if there exists a deterministic mapping ${\phi }_n:\{1,\dots ,M_n\}\to {\mathcal{X}}^n$ satisfying

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{M}_n& \le R,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim}d(P_{X^n},P_{{\tilde{X}}^n})& =0,\hfill \end{array}$$

(4)

where ${\tilde{X}}^n={\phi }_n\left(U_{M_n}\right)$ and $U_{M_n}$ is the uniform random number over ${\mathcal{U}}_{M_n}$. The infimum of f-achievable rates, i.e.,

$$\begin{array}{c}\hfill S_{\mathrm{f}}\left(\mathit{X}\right):=inf\{R:Risf-achievable\}\end{array}$$

(5)

is called the FL resolvability or simply f-resolvability.

Then, we have the following theorem:

Theorem 1

(Han and Verdú [6]). For any general target source $\mathit{X}$,

$$\begin{array}{c}\hfill S_{\mathrm{f}}\left(\mathit{X}\right)=\overline{H}\left(\mathit{X}\right),\end{array}$$

(6)

where

$$\begin{array}{cc}\hfill \overline{H}\left(\mathit{X}\right)& :=inf\left\{a:\underset{n\to \infty }{lim}Pr\left\{\frac{1}{n}log\frac{1}{P_{X^n}\left(X^n\right)}>a\right\}=0\right\}.\hfill \end{array}$$

(7)

Remark 1.

As a dual counterpart of (7), we may define

$$\begin{array}{cc}\hfill \underline{H}\left(\mathit{X}\right)& :=sup\left\{b:\underset{n\to \infty }{lim}Pr\left\{\frac{1}{n}log\frac{1}{P_{X^n}\left(X^n\right)}<b\right\}=0\right\}.\hfill \end{array}$$

(8)

Sources such that $\overline{H}\left(\mathit{X}\right)=\underline{H}\left(\mathit{X}\right)$ are called one-point spectrum sources (or equivalently, said to satisfy the strong converse property (cf. Han [2])), which includes stationary memoryless sources and stationary ergodic sources, etc. This class of sources is discussed later in Corollary 1.

The following problem is called the δ-resolvability problem [2,7], which relaxes the condition on the variational distance, compared to (4).

Definition 2

(FL $\delta$-resolvability). For a fixed $\delta \in [0,1)$, a resolution rate $R\ge 0$ is said to be FL δ-achievable or simply $f\left(\delta \right)$-achievable (under the variational distance) if there exists a deterministic mapping ${\phi }_n:\{1,\dots ,M_n\}\to {\mathcal{X}}^n$ satisfying

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{M}_n& \le R,\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}d(P_{X^n},P_{{\tilde{X}}^n})& \le \delta ,\hfill \end{array}$$

(10)

where ${\tilde{X}}^n={\phi }_n\left(U_{M_n}\right)$ and $U_{M_n}$ is the uniform random number over ${\mathcal{U}}_{M_n}$. The infimum of all $f\left(\delta \right)$-achievable rates, i.e.,

$$\begin{array}{c}\hfill S_{\mathrm{f}}\left(\delta \right|\mathit{X}):=inf\{R:Ris\mathrm{f}\left(\delta \right)-achievable\}\end{array}$$

(11)

is referred to as the FL δ-resolvability or simply $\mathrm{f}\left(\delta \right)$-resolvability.

Then, a characterization of $S_{\mathrm{f}}\left(\delta \right|\mathit{X})$ is given by

Theorem 2

(Steinberg and Verdú [7]). For any general target source $\mathit{X}$,

$$\begin{array}{c}\hfill S_{\mathrm{f}}\left(\delta \right|\mathit{X})={\overline{H}}_{\delta }\left(\mathit{X}\right)(\delta \in [0,1)),\end{array}$$

(12)

where

$$\begin{array}{cc}\hfill {\overline{H}}_{\delta }\left(\mathit{X}\right)& :=inf\left\{a:\underset{n\to \infty }{lim\; sup}Pr\left\{\frac{1}{n}log\frac{1}{P_{X^n}\left(X^n\right)}>a\right\}\le \delta \right\}.\hfill \end{array}$$

(13)

Remark 2.

The FL resolvability problem is deeply related to the FL source coding problem allowing a probability of a decoding error up to ε. Denoting by $R_{\mathrm{f}}\left(\epsilon \right|\mathit{X})$ the minimum achievable rate for the source $\mathit{X}$, there is the relationship [7]:

$$\begin{array}{c}\hfill R_{\mathrm{f}}\left(\epsilon \right|\mathit{X})={\overline{H}}_{\epsilon }\left(\mathit{X}\right)(\forall \epsilon \in [0,1))\end{array}$$

(14)

and, hence, by Theorem 2,

$$\begin{array}{c}\hfill S_{\mathrm{f}}\left(\delta \right|\mathit{X})=R_{\mathrm{f}}\left(\delta \right|\mathit{X})(\forall \delta \in [0,1)).\end{array}$$

(15)

Formula (14) can also be shown with a smooth Rényi entropy of order zero [32].

3. VL Resolvability: Variational Distance

In this section, we introduce the problem of variable-length (VL) resolvability, where the target probability distribution is approximated by encoding a VL uniform random number. As an initial step, we analyze the fundamental limit on the VL resolvability with the variational distance as an approximation measure.

3.1. Definitions

Let ${\mathcal{U}}^{*}$ denote the set of all sequences $\mathit{u}\in {\mathcal{U}}^m$ over $m=0,1,2,\cdots$, where ${\mathcal{U}}^0=\left\{\lambda \right\}$ ($\lambda$ is the null string). Let $L_n$ denote a random variable which takes a value in $\{0,1,2,\dots \}$. We define the VL uniform random number $U^{\left(L_n\right)}$ so that $U^{\left(m\right)}$ is uniformly distributed over ${\mathcal{U}}^m$ given $L_n=m$. In other words,

$$\begin{array}{cc}\hfill P_{U^{\left(L_n\right)}}(\mathit{u},m)& :=Pr\{U^{\left(L_n\right)}=\mathit{u},L_n=m\}=\frac{Pr\{L_n=m\}}{K^m}(\forall \mathit{u}\in {\mathcal{U}}^m),\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill Pr\{U^{\left(L_n\right)}=\mathit{u}|L_n=m\}& =\frac{P_{U^{\left(L_n\right)}}(\mathit{u},m)}{Pr\{L_n=m\}}=\frac{1}{K^m}(\forall \mathit{u}\in {\mathcal{U}}^m),\hfill \end{array}$$

(17)

where $K=\left|\mathcal{U}\right|$. It should be noticed that the VL sequence $\mathit{u}\in {\mathcal{U}}^m$ is generated with the joint probability $P_{U^{\left(L_n\right)}}(\mathit{u},m)$.

We formally define the $\delta$-resolvability problem under the variational distance using the VL random number, called the VL δ-resolvability or simply $\mathrm{v}\left(\delta \right)$-resolvability.

Definition 3

(VL $\delta$-resolvability: variational distance). A resolution rate $R\ge 0$ is said to be VL δ-achievable (under the variational distance) with $\delta \in [0,1)$ if there exists a VL uniform random number $U^{\left(L_n\right)}$ and a deterministic mapping ${\phi }_n:{\mathcal{U}}^{*}\to {\mathcal{X}}^n$ satisfying

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}\mathbb{E}\left[L_n\right]& \le R,\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}d(P_{X^n},P_{{\tilde{X}}^n})& \le \delta ,\hfill \end{array}$$

(19)

where $\mathbb{E}[·]$ denotes the expected value and ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$. The infimum of all $\mathrm{v}\left(\delta \right)$-achievable rates, i.e.,

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\delta \right|\mathit{X}):=inf\{R:Ris\mathrm{v}\left(\delta \right)-achievable\}\end{array}$$

(20)

is referred to as the VL δ-resolvability or simply $\mathrm{v}\left(\delta \right)$-resolvability.

If $\delta =0$, $\mathrm{v}\left(0\right)$-achievable is said to be VL achievable or simply v-achievable (under the variational distance). The infimum of all v-achievable rates, i.e.,

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\mathit{X}\right):=inf\{R:Risv-achievable\}\end{array}$$

(21)

is called the VL resolvability or simply v-resolvability.

Remark 3.

One may think that condition (18) can be replaced with the condition on the sup-entropy rate:

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}H\left(U^{\left(L_n\right)}\right)& \le R\hfill \end{array}$$

(22)

as in [6], where $H(·)$ denotes the Shannon entropy. Indeed, both conditions yield the same resolvability result. To see this, let us denote by ${\tilde{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X})$ the infimum of v-achievable rates R under constraints (19) and (22). It is easily checked that

$$\begin{array}{cc}\hfill \mathbb{E}\left[L_n\right]& =\sum _{m=1}^{\infty }\sum _{\mathit{u}\in {\mathcal{U}}^m}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)log{K}^m\hfill \\ \hfill & =\sum _{m=1}^{\infty }\sum _{\mathit{u}\in {\mathcal{U}}^m}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)log\frac{Pr\{L_n=m\}}{P_{U^{\left(L_n\right)}}(\mathit{u},m)}\hfill \\ \hfill & =H\left(U^{\left(L_n\right)}\right)-H\left(L_n\right)\le H\left(U^{\left(L_n\right)}\right).\hfill \end{array}$$

(23)

This implies $S_{\mathrm{v}}\left(\delta \right|\mathit{X})\le {\tilde{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X})$. On the other hand, by invoking the well-known relation (cf. ([33], Corollary 3.12)), it holds that

$$\begin{array}{c}\hfill H\left(L_n\right)\le log(e·\mathbb{E}\left[L_n\right]).\end{array}$$

(24)

Consider any resolution rate $R>S_{\mathrm{v}}\left(\delta \right|\mathit{X})$. Then, (18) holds for some $U^{\left(L_n\right)}$ and ${\phi }_n$ and, hence, (24) leads to

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n}H\left(L_n\right)=\underset{n\to \infty }{lim}\frac{1}{n}log(e·\mathbb{E}\left[L_n\right])=0.\end{array}$$

(25)

From this equation, (23) yields that

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}H\left(U^{\left(L_n\right)}\right)=\underset{n\to \infty }{lim\; sup}\frac{1}{n}\mathbb{E}\left[L_n\right]\le R\end{array}$$

(26)

to obtain $R\ge {\tilde{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X})$, implying that $S_{\mathrm{v}}\left(\delta \right|\mathit{X})\ge {\tilde{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X})$. Thus, $S_{\mathrm{v}}\left(\delta \right|\mathit{X})={\tilde{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X})$.

3.2. Smooth Shannon Entropy

To establish a general formula for $S_{\mathrm{v}}\left(\delta \right|\mathit{X})$, we introduce the following quantity for a general source $\mathit{X}$. Let $\mathcal{P}\left({\mathcal{X}}^n\right)$ denote the set of all probability distributions on ${\mathcal{X}}^n$. For $\delta \in [0,1)$, by defining the δ-ball using the variational distance

$$\begin{array}{c}\hfill B_{\delta }\left(X^n\right)=\left\{P_{V^n}\in \mathcal{P}\left({\mathcal{X}}^n\right):d(P_{X^n},P_{V^n})\le \delta \right\},\end{array}$$

(27)

we introduce the smooth Shannon entropy:

$$\begin{array}{cc}\hfill H_{\left[\delta \right]}\left(X^n\right)& :=\underset{P_{V^n}\in B_{\delta }\left(X^n\right)}{inf}\sum _{\mathit{x}\in {\mathcal{X}}^n}{P}_{V^n}\left(\mathit{x}\right)log\frac{1}{P_{V^n}\left(\mathit{x}\right)}\hfill \\ \hfill & =\underset{P_{V^n}\in B_{\delta }\left(X^n\right)}{inf}H\left(V^n\right),\hfill \end{array}$$

(28)

where $H\left(V^n\right)$ denotes the Shannon entropy of $P_{V^n}$. The $H_{\left[\delta \right]}\left(X^n\right)$ is a nonincreasing function of $\delta$. Based on this quantity for a general source $\mathit{X}={\left\{X^n\right\}}_{n=1}^{\infty }$, we define

$$\begin{array}{cc}\hfill H_{\left[\delta \right]}\left(\mathit{X}\right)& =\underset{n\to \infty }{lim\; sup}\frac{1}{n}{H}_{\left[\delta \right]}\left(X^n\right).\hfill \end{array}$$

(29)

Remark 4.

Renner and Wolf [8] have defined the smooth Rény entropy of order $\alpha \in (0,1)\cup (1,\infty )$ as

$$\begin{array}{c}\hfill H_{\left[\delta \right]}^{\alpha }\left(X^n\right)=\underset{P_{V^n}\in B_{\delta }\left(X^n\right)}{inf}\frac{1}{1-\alpha }log\sum _{\mathit{x}\in {\mathcal{X}}^n}{P}_{V^n}{\left(\mathit{x}\right)}^{\alpha }.\end{array}$$

(30)

By letting $\alpha \uparrow 1$, we have

$$\begin{array}{c}\hfill \underset{\alpha \uparrow 1}{lim}{H}_{\left[\delta \right]}^{\alpha }\left(X^n\right)=H_{\left[\delta \right]}\left(X^n\right).\end{array}$$

(31)

As for the proof, see Appendix A.

3.3. General Formula for General $\delta \in [0,1)$

The following main theorem indicates that the $\mathrm{v}\left(\delta \right)$-resolvability $S_{\mathrm{v}}\left(\delta \right|\mathit{X})$ can be characterized by the smooth Shannon entropy for $\mathit{X}$.

Theorem 3.

For any general target source $\mathit{X}$,

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\delta \right|\mathit{X})=\underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ]}\left(\mathit{X}\right)(\delta \in [0,1)).\end{array}$$

(32)

Remark 5.

In Formula (32), the limit ${lim}_{\gamma \downarrow 0}$ of the offset term $+\gamma$ appears in the characterization of $S_{\mathrm{v}}\left(\delta \right|\mathit{X})$. This is because the smooth entropy $H_{\left[\delta \right]}\left(X^n\right)$ for $X^n$ involves the infimum over the non-asymptotic δ-ball $B_{\delta }\left(X^n\right)$ for a given length n. Alternatively, we may consider the asymptotic δ-ball defined as

$$\begin{array}{c}\hfill B_{\delta }\left(\mathit{X}\right)=\left\{\mathit{V}={\left\{V^n\right\}}_{n=1}^{\infty }:\underset{n\to \infty }{lim\; sup}d(P_{X^n},P_{V^n})\le \delta \right\},\end{array}$$

(33)

and then we obtain the alternative formula

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\delta \right|\mathit{X})=\underset{\mathit{V}\in B_{\delta }\left(\mathit{X}\right)}{inf}H\left(\mathit{V}\right)(\delta \in [0,1))\end{array}$$

(34)

without an offset term, where

$$\begin{array}{c}\hfill H\left(\mathit{V}\right):=\underset{n\to \infty }{lim\; sup}\frac{1}{n}H\left(V^n\right)\end{array}$$

(35)

is the sup-entropy rate for $\mathit{V}$ with the Shannon entropy $H\left(V^n\right)$. The proof of (34) is given in Appendix B.

The same remark also applies to general formulas to be established in the subsequent sections.

Remark 6.

Independently of this work, Tomita, Uyematsu, and Matsumoto [34] have investigated the following problem: the coin distribution is given by fair coin-tossing and the average number of coin tosses should be asymptotically minimized as in [3] while the variational distance between the target and approximated distributions should satisfy (19). In this case, the asymptotically minimum average number of coin tosses is also characterized by the right-hand side (r.h.s.) of (32) (cf. [34]). Since the coin distribution is restricted to that given by fair coin-tossing with a stopping algorithm, realizations of $L_n$ must satisfy the Kraft inequality (for prefix codes), whereas the problem addressed in this paper allows the probability distribution of $L_n$ to be an arbitrary discrete one, not necessarily implying prefix codes. In this sense, our problem is more relaxed, while the coin is constrained to be conditionally independent given $L_n$. Theorem 3 indicates that the $v\left(\delta \right)$-resolvability does not differ in both problems. Later, we shall show that, even in the case where the coin distribution may be any general source $\mathit{X}$, the δ-resolvability remains the same (cf. Theorem 5 and Remark 14).

On the other hand, we now define the following information quantity (to be used in Remark 7 below) to discuss the relationship with VL source coding: For $\delta \in [0,1)$ we define

$$\begin{array}{c}\hfill G_{\left[\delta \right]}\left(X^n\right)=\underset{\begin{array}{c}{\mathcal{A}}_n\subseteq {\mathcal{X}}^n:\\ Pr\{X^n\in {\mathcal{A}}_n\}\ge 1-\delta \end{array}}{inf}\sum _{\mathit{x}\in {\mathcal{A}}_n}{P}_{X^n}\left(\mathit{x}\right)log\frac{1}{P_{X^n}\left(\mathit{x}\right)}.\end{array}$$

(36)

The $G_{\left[\delta \right]}\left(X^n\right)$ is a nonincreasing function of $\delta$. Based on this quantity, we define

$$\begin{array}{cc}\hfill G_{\left[\delta \right]}\left(\mathit{X}\right)& =\underset{n\to \infty }{lim\; sup}\frac{1}{n}{G}_{\left[\delta \right]}\left(X^n\right).\hfill \end{array}$$

(37)

Then, we have

Remark 7.

There is a deep relation between the δ-resolvability problem and VL δ-source coding with the error probability asymptotically not exceeding δ. Koga and Yamamoto [35] (also, cf. Han [36]) showed that the minimum average length rate $R_{\mathrm{v}}^{*}\left(\delta \right|\mathit{X})$ of VL δ-source codes is given by

$$\begin{array}{c}\hfill R_{\mathrm{v}}^{*}\left(\delta \right|\mathit{X})=\underset{\gamma \downarrow 0}{lim}{G}_{[\delta +\gamma ]}\left(\mathit{X}\right)(\forall \delta \in [0,1)).\end{array}$$

(38)

Theorem 3 and Proposition 1 (to be shown just below) reveal that

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\delta \right|\mathit{X})=R_{\mathrm{v}}^{*}\left(\delta \right|\mathit{X})(\forall \delta \in [0,1)).\end{array}$$

(39)

The following proposition shows a general relationship between $G_{\left[\delta \right]}\left(\mathit{X}\right)$ and $H_{\left[\delta \right]}\left(\mathit{X}\right)$.

Proposition 1.

For any general source $\mathit{X}$,

$$\begin{array}{c}\hfill H_{\left[\delta \right]}\left(\mathit{X}\right)=G_{\left[\delta \right]}\left(\mathit{X}\right)\le (1-\delta ){\overline{H}}_{\delta -\gamma }\left(\mathit{X}\right)(\forall \delta \in (0,1),\forall \gamma \in (0,\delta ]).\end{array}$$

(40)

In particular,

$$\begin{array}{c}\hfill \underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ]}\left(\mathit{X}\right)=\underset{\gamma \downarrow 0}{lim}{G}_{[\delta +\gamma ]}\left(\mathit{X}\right)\le (1-\delta ){\overline{H}}_{\delta }\left(\mathit{X}\right)(\forall \delta \in [0,1)).\end{array}$$

(41)

(Proof) See Appendix C.

By plugging $\gamma =\delta$ into (40), a looser but sometimes useful bound

$$\begin{array}{c}\hfill H_{\left[\delta \right]}\left(\mathit{X}\right)=G_{\left[\delta \right]}\left(\mathit{X}\right)\le (1-\delta )\overline{H}\left(\mathit{X}\right)\end{array}$$

(42)

can be obtained. Equation (40) has been derived by [21], which improves a bound established in [35,37]. In view of Theorems 2 and 3, (41) in Proposition 1 implies

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\delta \right|\mathit{X})\le (1-\delta )S_{\mathrm{f}}\left(\delta \right|\mathit{X})\end{array}$$

(43)

for all $\delta \in [0,1)$, where $S_{\mathrm{f}}\left(\delta \right|\mathit{X})$ denotes the $\mathrm{f}\left(\delta \right)$-resolvability. This general relationship elucidates the advantage of the use of VL uniform random numbers to make the average length rate lower. The proposition also claims that $G_{\left[\delta \right]}\left(\mathit{X}\right)$ coincides with $H_{\left[\delta \right]}\left(\mathit{X}\right)$ for all $\delta \in [0,1)$ for any general source $\mathit{X}$.

A consequence of Theorem 3 is the following corollary:

Corollary 1.

Let $\mathit{X}={\left\{X^n\right\}}_{n=1}^{\infty }$ be a one-point spectrum source ($\overline{H}\left(\mathit{X}\right)=\underline{H}\left(\mathit{X}\right)$) with $X^n=(X_1,X_2,\dots ,X_n)$, then we have

$$\begin{array}{c}\hfill S_{\mathrm{f}}\left(\delta \right|\mathit{X})={\overline{H}}_{\delta }\left(\mathit{X}\right)=H^{*}\left(\mathit{X}\right)(\forall \delta \in [0,1)),\end{array}$$

(44)

where $H^{*}\left(\mathit{X}\right):=\overline{H}\left(\mathit{X}\right)=\underline{H}\left(\mathit{X}\right)$. Moreover, it holds that

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\delta \right|\mathit{X})=H_{\left[\delta \right]}\left(\mathit{X}\right)=G_{\left[\delta \right]}\left(\mathit{X}\right)=(1-\delta )H^{*}\left(\mathit{X}\right)(\forall \delta \in [0,1)),\end{array}$$

(45)

where we notice that $H^{*}\left(\mathit{X}\right)=H\left(X_1\right)$ (entropy) for stationary memoryless sources.

(Proof) See Appendix D.

Now, we are ready to give the proof of Theorem 3:

Proof of Theorem 3.

(1)

Converse Part:

Let R be $\mathrm{v}\left(\delta \right)$-achievable. Then, there exists $U^{\left(L_n\right)}$ and ${\phi }_n$ satisfying (18) and

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{lim\; sup}{\delta }_n\le \delta ,\hfill \end{array}$$

(46)

where we define ${\delta }_n=d(P_{X^n},P_{{\tilde{X}}^n})$ with ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$. Equation (46) implies that, for any given $\gamma >0$, it holds that ${\delta }_n\le \delta +\gamma$ for all $n\ge n_0$ with some $n_0>0$, and thus we have

$$\begin{array}{c}\hfill H_{[\delta +\gamma ]}\left(X^n\right)\le H_{\left[{\delta }_n\right]}\left(X^n\right)(\forall n\ge n_0),\end{array}$$

(47)

because $H_{\left[\delta \right]}\left(X^n\right)$ is a nonincreasing function of $\delta$. Since $P_{{\tilde{X}}^n}\in B_{{\delta }_n}\left(X^n\right)$, we have

$$\begin{array}{c}\hfill H_{\left[{\delta }_n\right]}\left(X^n\right)\le H\left({\tilde{X}}^n\right).\end{array}$$

(48)

On the other hand, it follows from (23) that

$$\begin{array}{cc}\hfill H\left({\tilde{X}}^n\right)& \le H\left(U^{\left(L_n\right)}\right)=\mathbb{E}\left[L_n\right]+H\left(L_n\right),\hfill \end{array}$$

(49)

where the inequality is due to the fact that ${\phi }_n$ is a deterministic mapping and ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$.

Combining (47)–(49) yields

$$\begin{array}{cc}\hfill H_{[\delta +\gamma ]}\left(\mathit{X}\right)& =\underset{n\to \infty }{lim\; sup}\frac{1}{n}{H}_{[\delta +\gamma ]}\left(X^n\right)\hfill \\ \hfill & \le \underset{n\to \infty }{lim\; sup}\frac{1}{n}\mathbb{E}\left[L_n\right]+\underset{n\to \infty }{lim\; sup}\frac{1}{n}H\left(L_n\right)\le R,\hfill \end{array}$$

(50)

where we have used (18) and (25) for the last inequality. Since $\gamma >0$ is arbitrary, we obtain

$$\begin{array}{c}\hfill \underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ]}\left(\mathit{X}\right)\le R.\end{array}$$

(51)

(2)

Direct Part:

Without loss of generality, we assume that $H^{+}:={lim}_{\gamma \downarrow 0}{H}_{[\delta +\gamma ]}\left(\mathit{X}\right)$ is finite ($H^{+}<+\infty$). Letting $R=H^{+}+3\gamma$, where $\gamma >0$ is an arbitrary constant, we shall show that R is $\mathrm{v}\left(\delta \right)$-achievable. In what follows, we use a simpler form of information spectrum slicing [2], where each piece of sliced information quantized to a positive integer ℓ is approximated by the uniform random number $U^{(\ell )}$ of the length ℓ.

First, we note that

$$\begin{array}{c}\hfill H^{+}\ge H_{[\delta +\gamma ]}\left(\mathit{X}\right)\ge \frac{1}{n}{H}_{[\delta +\gamma ]}\left(X^n\right)-\gamma (\forall n>n_0)\end{array}$$

(52)

because of the monotonicity of $H_{\left[\delta \right]}\left(\mathit{X}\right)$ in $\delta$. Let $V^n$ be a random variable subject to $P_{V^n}\in B_{\delta +\gamma }\left(X^n\right)$, which satisfies

$$\begin{array}{c}\hfill H_{[\delta +\gamma ]}\left(X^n\right)+\gamma \ge H\left(V^n\right).\end{array}$$

(53)

For $\gamma >0$, we can choose a $c_n>0$ so large that

$$\begin{array}{c}\hfill Pr\{V^n\notin T_n\}\le \gamma \end{array}$$

(54)

where

$$\begin{array}{c}\hfill T_n:=\left\{\mathit{x}\in {\mathcal{X}}^n:\frac{1}{n}log\frac{1}{P_{V^n}\left(\mathit{x}\right)}\le c_n\right\}.\end{array}$$

(55)

We also define

$$\begin{array}{c}\hfill \ell \left(\mathit{x}\right):=\left\{\begin{array}{cc}\lceil log\frac{1}{P_{V^n}\left(\mathit{x}\right)}+n\gamma \rceil \hfill & for\mathit{x}\in T_n\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(56)

For $m=0,1,\dots ,{\beta }_n:=\lceil n(c_n+\gamma )\rceil$, set

$$\begin{array}{c}\hfill S_n\left(m\right):=\left\{\mathit{x}\in {\mathcal{X}}^n:\ell \left(\mathit{x}\right)=m\right\},\end{array}$$

(57)

then these sets form a partition of ${\mathcal{X}}^n$, i.e.,

$$\begin{array}{c}\hfill \bigcup _{m=0}^{{\beta }_n}{S}_n\left(m\right)={\mathcal{X}}^{n}and\bigcup _{m=1}^{{\beta }_n}{S}_n\left(m\right)=T_n.\end{array}$$

(58)

We set $L_n$ so that

$$\begin{array}{c}\hfill Pr\{L_n=m\}=Pr\{V^n\in S_n\left(m\right)\},\end{array}$$

(59)

where it is obvious that ${\sum }_{m=0}^{{\beta }_n}Pr\{L_n=m\}=1$, and, hence, the probability distribution of the VL uniform random number $U^{\left(L_n\right)}$ is given as

$$\begin{array}{c}\hfill P_{U^{\left(L_n\right)}}(\mathit{u},m):=Pr\{U^{\left(L_n\right)}=\mathit{u},L_n=m\}=\frac{Pr\{V^n\in S_n\left(m\right)\}}{K^m}(\forall \mathit{u}\in {\mathcal{U}}^m).\end{array}$$

(60)

(a)

Construction of Mapping ${\phi }_n:{\mathcal{U}}^{*}\to {\mathcal{X}}^n$:

Index the elements in $S_n\left(m\right)$ as ${\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_{|S_n\left(m\right)|}(m=1,2,\cdots )$, where

$$\begin{array}{c}\hfill |S_n\left(m\right)|\le K^{m-n\gamma }\end{array}$$

(61)

since for $\mathit{x}\in S_n\left(m\right)$

$$\begin{array}{c}\hfill log\frac{1}{P_{V^n}\left(\mathit{x}\right)}\le m-n\gamma ⟺P_{V^n}\left(\mathit{x}\right)\ge K^{-(m-n\gamma )},\end{array}$$

(62)

and, therefore,

$$\begin{array}{c}\hfill 1\ge \sum _{\mathit{x}\in S_n\left(m\right)}{P}_{V^n}\left(\mathit{x}\right)\ge \sum _{\mathit{x}\in S_n\left(m\right)}{K}^{-(m-n\gamma )}=\left|S_n\left(m\right)\right|K^{-(m-n\gamma )}.\end{array}$$

(63)

For $i=1,2,\dots ,|S_n\left(m\right)|$, define ${\tilde{A}}_i^{\left(m\right)}\subset {\mathcal{U}}^m$ as the set of sequences $\mathit{u}\in {\mathcal{U}}^m$ so that

$$\begin{array}{c}\hfill \sum _{\mathit{u}\in {\tilde{A}}_i^{\left(m\right)}}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)\le P_{V^n}\left({\mathit{x}}_i\right)<\sum _{\mathit{u}\in {\tilde{A}}_i^{\left(m\right)}}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)+\frac{Pr\{V^n\in S_n\left(m\right)\}}{K^m}\end{array}$$

(64)

and

$$\begin{array}{c}\hfill {\tilde{A}}_i^{\left(m\right)}\cap {\tilde{A}}_j^{\left(m\right)}=\varnothing (i\ne j).\end{array}$$

(65)

If

$$\begin{array}{c}\hfill \sum _{i=1}^{|S_n\left(m\right)|}\sum _{\mathit{u}\in {\tilde{A}}_i^{\left(m\right)}}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)<\sum _{i=1}^{|S_n\left(m\right)|}{P}_{V^n}\left({\mathit{x}}_i\right)=Pr\{V^n\in S_n\left(m\right)\},\end{array}$$

(66)

then add a ${\mathit{u}}_i\in {\mathcal{U}}^m\setminus \left({\cup }_j{\tilde{A}}_j^{\left(m\right)}\right)$ to obtain

$$\begin{array}{c}\hfill A_i^{\left(m\right)}={\tilde{A}}_i^{\left(m\right)}\cup \left\{{\mathit{u}}_i\right\}\end{array}$$

(67)

for $i=1,2,\dots$ in order, until it holds that with some $1\le c\le |S_n\left(m\right)|$

$$\begin{array}{c}\hfill \bigcup _{i=1}^c{A}_i^{\left(m\right)}\cup \bigcup _{i=c+1}^{|S_n\left(m\right)|}{\tilde{A}}_i^{\left(m\right)}={\mathcal{U}}^m,\end{array}$$

(68)

where ${\mathit{u}}_1,{\mathit{u}}_2,\cdots$ are selected to be all different. Since $|{\mathcal{U}}^m|=K^m$ and

$$\begin{array}{c}\hfill \sum _{\mathit{u}\in {\mathcal{U}}^m}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)=\sum _{\mathit{u}\in {\mathcal{U}}^m}\frac{Pr\{V^n\in S_n\left(m\right)\}}{K^m}=Pr\{V^n\in S_n\left(m\right)\},\end{array}$$

(69)

such a $1\le c\le |S_n\left(m\right)|$ always exists. For simplicity, we set for $i=c+1,c+2,\dots ,|S_n\left(m\right)|$

$$\begin{array}{c}\hfill A_i^{\left(m\right)}={\tilde{A}}_i^{\left(m\right)}\end{array}$$

(70)

and for $i=1,2,\dots ,|S_n\left(m\right)|$

$$\begin{array}{c}\hfill {\phi }_n\left(\mathit{u}\right)={\mathit{x}}_{i}for\mathit{u}\in A_i^{\left(m\right)},\end{array}$$

(71)

which defines the random variable ${\tilde{X}}^n$ with values in ${\mathcal{X}}^n$ such that

$$\begin{array}{c}\hfill P_{{\tilde{X}}^n}\left({\mathit{x}}_i\right)=\sum _{\mathit{u}\in A_i^{\left(m\right)}}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)({\mathit{x}}_i\in S_n\left(m\right)),\end{array}$$

(72)

that is, ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$, where if ${\mathcal{X}}^n\setminus T_n\ne \varnothing$, we choose some ${\mathit{x}}_0\in {\mathcal{X}}^n\setminus T_n$ and set

$$\begin{array}{c}\hfill P_{{\tilde{X}}^n}\left({\mathit{x}}_0\right)=Pr\{V^n\notin T_n\}\mathrm{and}{\phi }_n\left(\lambda \right)={\mathit{x}}_0.\end{array}$$

(73)

Notice that, by this construction, we have

$$\begin{array}{c}\hfill |P_{{\tilde{X}}^n}\left({\mathit{x}}_i\right)-P_{V^n}\left({\mathit{x}}_i\right)|\le \frac{Pr\{V^n\in S_n\left(m\right)\}}{K^m}\end{array}$$

(74)

for $i=1,2,\dots ,|S_n\left(m\right)|;m=1,2,\dots ,{\beta }_n$, and

$$\begin{array}{c}\hfill Pr\{{\tilde{X}}^n\notin T_n\}=Pr\{V^n\notin T_n\}\le \gamma .\end{array}$$

(75)

(b)

Evaluation of Average Length:

Since $m=0$ does not contribute to the average length $\mathbb{E}\left[L_n\right]$, it is evaluated as follows:

$$\begin{array}{cc}\hfill \mathbb{E}\left[L_n\right]& =\sum _{m=1}^{{\beta }_n}\sum _{\mathit{u}\in {\mathcal{U}}^m}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)·m\hfill \\ \hfill & =\sum _{m=1}^{{\beta }_n}\sum _{i=1}^{|S_n\left(m\right)|}\sum _{\mathit{u}\in A_i^{\left(m\right)}}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)·m\hfill \\ \hfill & =\sum _{m=1}^{{\beta }_n}\sum _{{\mathit{x}}_i\in S_n\left(m\right)}{P}_{{\tilde{X}}^n}\left({\mathit{x}}_i\right)·m,\hfill \end{array}$$

(76)

where we have used ${\mathcal{U}}^m={\bigcup }_{i=1}^{|S_n\left(m\right)|}{A}_i^{\left(m\right)}$ and (72). For ${\mathit{x}}_i\in S_n\left(m\right)$, we obtain from (74)

$$\begin{array}{cc}\hfill P_{{\tilde{X}}^n}\left({\mathit{x}}_i\right)& \le P_{V^n}\left({\mathit{x}}_i\right)+\frac{Pr\{V^n\in S_n\left(m\right)\}}{K^m}\hfill \\ \hfill & \le P_{V^n}\left({\mathit{x}}_i\right)\left(1+\frac{1}{P_{V^n}\left({\mathit{x}}_i\right)K^m}\right)\hfill \\ \hfill & \le P_{V^n}\left({\mathit{x}}_i\right)\left(1+\frac{1}{K^{n\gamma }}\right),\hfill \end{array}$$

(77)

where, to derive the last inequality, we have used (62). Plugging the inequality

$$\begin{array}{c}\hfill m\le log\frac{1}{P_{V^n}\left({\mathit{x}}_i\right)}+n\gamma +1(\forall {\mathit{x}}_i\in S_n\left(m\right))\end{array}$$

(78)

and (77) into (76), we obtain

$$\begin{array}{cc}\hfill \mathbb{E}\left[L_n\right]& \le \left(1+\frac{1}{K^{n\gamma }}\right)\sum _{m=1}^{{\beta }_n}\sum _{{\mathit{x}}_i\in S_n\left(m\right)}{P}_{V^n}\left({\mathit{x}}_i\right)\left(log\frac{1}{P_{V^n}\left({\mathit{x}}_i\right)}+n\gamma +1\right)\hfill \\ \hfill & \le \left(1+\frac{1}{K^{n\gamma }}\right)\left(H\left(V^n\right)+n\gamma +1\right),\hfill \end{array}$$

(79)

which yields

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}\mathbb{E}\left[L_n\right]& \le \underset{n\to \infty }{lim\; sup}\frac{1}{n}H\left(V^n\right)+2\gamma \hfill \\ \hfill & \le \underset{n\to \infty }{lim\; sup}\frac{1}{n}{H}_{[\delta +\gamma ]}\left(X^n\right)+3\gamma \hfill \\ \hfill & =H_{[\delta +\gamma ]}\left(\mathit{X}\right)+3\gamma \hfill \\ \hfill & \le H^{+}+3\gamma =R,\hfill \end{array}$$

(80)

where the second inequality follows from (53) and the last one is due to (52).

(c)

Evaluation of Variational Distance:

From (61) and (74), we have

$$\begin{array}{c}\hfill \sum _{\mathit{x}\in S_n\left(m\right)}|P_{{\tilde{X}}^n}\left(\mathit{x}\right)-P_{V^n}\left(\mathit{x}\right)|\le \frac{|S_n\left(m\right)|Pr\{V^n\in S_n\left(m\right)\}}{K^m}\le \frac{Pr\{V^n\in S_n\left(m\right)\}}{K^{n\gamma }},\end{array}$$

(81)

which, in view of (58), leads to

$$\begin{array}{cc}\hfill d(P_{{\tilde{X}}^n},P_{V^n})& =\frac{1}{2}\sum _{\mathit{x}\in T_n}|P_{{\tilde{X}}^n}\left(\mathit{x}\right)-P_{V^n}\left(\mathit{x}\right)|+\frac{1}{2}\sum _{\mathit{x}\notin T_n}|P_{{\tilde{X}}^n}\left(\mathit{x}\right)-P_{V^n}\left(\mathit{x}\right)|\hfill \\ \hfill & \le \frac{1}{2}\sum _{m=1}^{{\beta }_n}\sum _{\mathit{x}\in S_n\left(m\right)}|P_{{\tilde{X}}^n}\left(\mathit{x}\right)-P_{V^n}\left(\mathit{x}\right)|\hfill \\ \hfill & +\frac{1}{2}\left(Pr\{{\tilde{X}}^n\notin T_n\}+Pr\{V^n\notin T_n\}\right)\hfill \\ \hfill & \le \frac{1}{2}\sum _{m=1}^{{\beta }_n}\frac{Pr\{V^n\in S_n\left(m\right)\}}{K^{n\gamma }}+\gamma \le \frac{1}{2}{K}^{-n\gamma }+\gamma ,\hfill \end{array}$$

(82)

where we have used (75) to obtain the leftmost inequality in (82). By the triangle inequality, we obtain

$$\begin{array}{c}\hfill d(P_{X^n},P_{{\tilde{X}}^n})\le d(P_{X^n},P_{V^n})+d(P_{{\tilde{X}}^n},P_{V^n})\le \delta +2\gamma +\frac{1}{2}{K}^{-n\gamma },\end{array}$$

(83)

where the last inequality follows because $P_{V^n}\in B_{\delta +\gamma }\left(X^n\right)$. Thus, we obtain from (83)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}d(P_{X^n},P_{{\tilde{X}}^n})& \le \delta +2\gamma .\hfill \end{array}$$

(84)

Since $\gamma >0$ is arbitrary and we have (80), we conclude that R is $\mathrm{v}\left(\delta \right)$-achievable.

□

3.4. General Formula for $\delta =0$

In this subsection, we consider the special case with $\delta =0$. In this case, we can elucidate the relationship between the minimum achievable rates for VL source codes with an asymptotically vanishing decoding error probability and the FL source codes.

We obtain the following corollary from Theorem 3 and Proposition 1:

Corollary 2.

For any general target source $\mathit{X}$,

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\mathit{X}\right)=\underset{\gamma \downarrow 0}{lim}{G}_{\left[\gamma \right]}\left(\mathit{X}\right),\end{array}$$

(85)

where $G_{\left[\gamma \right]}\left(\mathit{X}\right)$ is defined in (37).

It has been shown by Han [2] that any source $\mathit{X}={\left\{X^n\right\}}_{n=1}^{\infty }$ satisfying the uniform integrability (cf. Han [2]) satisfies

$$\begin{array}{c}\hfill \underset{\gamma \downarrow 0}{lim}{G}_{\left[\gamma \right]}\left(\mathit{X}\right)=H\left(\mathit{X}\right):=\underset{n\to \infty }{lim\; sup}\frac{1}{n}H\left(X^n\right),\end{array}$$

(86)

where $H\left(\mathit{X}\right)$ is called the sup-entropy rate. Notice here, in particular, that the finiteness of an alphabet implies the uniform integrability [2]. Thus, we obtain the following corollary:

Corollary 3.

For any finite alphabet target source $\mathit{X}$,

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\mathit{X}\right)=H\left(\mathit{X}\right).\end{array}$$

(87)

Remark 8.

As in the case of FL resolvability and FL source coding problems, $S_{\mathrm{v}}\left(\mathit{X}\right)$ is tightly related to VL source codes with vanishing decoding error probabilities. Denoting by $R_{\mathrm{v}}^{*}\left(\mathit{X}\right)$ the minimum error-vanishing VL -achievable rate for a source $\mathit{X}$, Han [36] has shown that

$$\begin{array}{c}\hfill R_{\mathrm{v}}^{*}\left(\mathit{X}\right)=\underset{\gamma \downarrow 0}{lim}{G}_{\left[\gamma \right]}\left(\mathit{X}\right),\end{array}$$

(88)

and, hence, from Corollary 3, it is concluded that

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\mathit{X}\right)=R_{\mathrm{v}}^{*}\left(\mathit{X}\right).\end{array}$$

(89)

In addition, if a general source$\mathit{X}$ satisfies the uniform integrability and the strong converse property (cf. Han [2]), then equation (86) holds and hence it follows from ([2], Theorem 1.7.1) that

$$\begin{array}{c}\hfill S_{\mathrm{f}}\left(\mathit{X}\right)=S_{\mathrm{v}}\left(\mathit{X}\right)=R_{\mathrm{v}}^{*}\left(\mathit{X}\right)=R_{\mathrm{v}}\left(\mathit{X}\right)=R_{\mathrm{f}}\left(\mathit{X}\right)=H\left(\mathit{X}\right),\end{array}$$

(90)

where $R_{\mathrm{f}}\left(\mathit{X}\right):=R_{\mathrm{f}}\left(0\right|\mathit{X})$ and $R_{\mathrm{v}}\left(\mathit{X}\right)$ denotes the minimum achievable rate of VL source codes with zero error probabilities for all $n=1,2,\cdots$.

Remark 9.

Han and Verdú [6] have discussed the problem of mean-resolvability for the target distribution $P_{X^n}$. In this problem, the coin distribution may be a general source $\tilde{\mathit{X}}={\left\{{\tilde{X}}^n\right\}}_{n=1}^{\infty }$, where ${\tilde{X}}^n$ is a random variable that takes values in ${\mathcal{X}}^n$ with the average length rate $\frac{1}{n}\mathbb{E}\left[L_n\right]$ in (18) replaced with the entropy rate $\frac{1}{n}H\left({\tilde{X}}^n\right)$. Denoting by ${\overline{S}}_{\mathrm{v}}\left(\mathit{X}\right)$ the mean-resolvability, which is defined as the infimum of v-achievable rates for a general source $\mathit{X}$ (with the countably infinite alphabet), we can easily verify that any mean-resolution rate $R>{\overline{S}}_{\mathrm{v}}\left(\mathit{X}\right)$ must satisfy $R\ge {lim}_{\gamma \downarrow 0}{G}_{\left[\gamma \right]}\left(\mathit{X}\right)$ so that $S_{\mathrm{v}}\left(\mathit{X}\right)\le {\overline{S}}_{\mathrm{v}}\left(\mathit{X}\right)$. On the other hand, $S_{\mathrm{v}}\left(\mathit{X}\right)\ge {\overline{S}}_{\mathrm{v}}\left(\mathit{X}\right)$ by definition. Thus, in view of Corollary 2, we have

Corollary 4.

For any general target source $\mathit{X}$,

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\mathit{X}\right)={\overline{S}}_{\mathrm{v}}\left(\mathit{X}\right)=\underset{\gamma \downarrow 0}{lim}{G}_{\left[\gamma \right]}\left(\mathit{X}\right).\end{array}$$

(91)

4. VL Resolvability: Divergence

So far, we have considered the problem of VL resolvability, in which the approximation level is measured by the variational distance between $X^n$ and ${\tilde{X}}^n$. It is sometimes of use to deal with another quantity as an approximation measure. In this section, we use the (unnormalized) divergence as the approximation measure.

4.1. Definitions

In this subsection, we address the following problem.

Definition 4

(VL $\delta$-resolvability: divergence). A resolution rate $R\ge 0$ is said to be VL δ-achievable or simply $v_D\left(\delta \right)$-achievable (under the divergence) with $\delta \ge 0$, if there exists a VL uniform random number $U^{\left(L_n\right)}$ and a deterministic mapping ${\phi }_n:{\mathcal{U}}^{*}\to {\mathcal{X}}^n$ satisfying

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}\mathbb{E}\left[L_n\right]& \le R,\hfill \end{array}$$

(92)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}D({\tilde{X}}^n\left|\right|X^n)& \le \delta ,\hfill \end{array}$$

(93)

where ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$ and $D\left({\tilde{X}}^n\right|\left|X^n\right)$ denotes the divergence between $P_{{\tilde{X}}^n}$ and $P_{X^n}$:

$$\begin{array}{c}\hfill D({\tilde{X}}^n\left|\right|X^n)=\sum _{\mathit{x}\in {\mathcal{X}}^n}{P}_{{\tilde{X}}^n}\left(\mathit{x}\right)log\frac{P_{{\tilde{X}}^n}\left(\mathit{x}\right)}{P_{X^n}\left(\mathit{x}\right)}.\end{array}$$

(94)

The infimum of all ${\mathrm{v}}_D\left(\delta \right)$-achievable rates, i.e.,

$$\begin{array}{c}\hfill S_{\mathrm{v}}^D\left(\delta \right|\mathit{X}):=inf\{R:Ris{\mathrm{v}}_D\left(\delta \right)-achievable\}\end{array}$$

(95)

is called the VL $\delta$-resolvability or simply the ${\mathrm{v}}_D\left(\delta \right)$-resolvability.

Remark 10.

The measure of approximation is now the divergence $D\left({\tilde{X}}^n\right|\left|X^n\right)$ but not its reversed version $D\left(X^n\right|\left|{\tilde{X}}^n\right)$. In the context of resolvability, divergence $D\left({\tilde{X}}^n\right|\left|X^n\right)$ (and its counterpart in the case of channel resolvability) is usually employed as in [6,7,9]. We also use this type of divergence in the subsequent sections.

To establish the general formula for $S_{\mathrm{v}}^D\left(\delta \right|\mathit{X})$, we introduce the following quantity for a general source $\mathit{X}={\left\{X^n\right\}}_{n=1}^{\infty }$. Recall that $\mathcal{P}\left({\mathcal{X}}^n\right)$ denotes the set of all probability distributions on ${\mathcal{X}}^n$. For $\delta \ge 0$, defining the δ-ball using the divergence as

$$\begin{array}{c}\hfill B_{\delta }^D\left(X^n\right)=\left\{P_{V^n}\in \mathcal{P}\left({\mathcal{X}}^n\right):D(V^n\left|\right|X^n)\le \delta \right\},\end{array}$$

(96)

we introduce the following quantity, referred to as the smooth entropy using the divergence:

$$\begin{array}{cc}\hfill H_{\left[\delta \right]}^D\left(X^n\right)& :=\underset{P_{V^n}\in B_{\delta }^D\left(X^n\right)}{inf}H\left(V^n\right),\hfill \end{array}$$

(97)

where $H\left(V^n\right)$ denotes the Shannon entropy of $P_{V^n}$. Obviously, $H_{\left[\delta \right]}^D\left(X^n\right)$ is a nonincreasing function of $\delta$. Now, we define

$$\begin{array}{cc}\hfill H_{\left[\delta \right]}^D\left(\mathit{X}\right)& =\underset{n\to \infty }{lim\; sup}\frac{1}{n}{H}_{\left[\delta \right]}^D\left(X^n\right).\hfill \end{array}$$

(98)

The following lemma is used to derive Corollary 5 of Theorem 4 below in the next subsection.

Lemma 1.

For any general source $\mathit{X}$,

$$\begin{array}{c}\hfill H_{\left[\delta \right]}\left(\mathit{X}\right)\le H_{\left[g\right(\delta \left)\right]}^D\left(\mathit{X}\right)\left(\delta \ge 0\right),\end{array}$$

(99)

where we define $g\left(\delta \right)=2{\delta }^2/lnK$, and

$$\begin{array}{c}\hfill \underset{\delta \downarrow 0}{lim}{G}_{\left[\delta \right]}\left(\mathit{X}\right)=\underset{\delta \downarrow 0}{lim}{H}_{\left[\delta \right]}\left(\mathit{X}\right)=\underset{\delta \downarrow 0}{lim}{H}_{\left[\delta \right]}^D\left(\mathit{X}\right)\le \overline{H}\left(\mathit{X}\right).\end{array}$$

(100)

(Proof) See Appendix E.

4.2. General Formula

Here, we establish another main theorem, which characterizes $S_{\mathrm{v}}^D\left(\delta \right|\mathit{X})$ for all $\delta \ge 0$ in terms of the smooth entropy using the divergence.

Theorem 4.

For any general target source $\mathit{X}$,

$$\begin{array}{c}\hfill S_{\mathrm{v}}^D\left(\delta \right|\mathit{X})=\underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ]}^D\left(\mathit{X}\right)\left(\delta \ge 0\right).\end{array}$$

(101)

Remark 11.

It should be noticed that the approximation measure considered here is not the normalized divergence

$$\begin{array}{c}\hfill \frac{1}{n}D({\phi }_n\left(U^{\left(L_n\right)}\right)\left|\right|X^n),\end{array}$$

(102)

which has been used in the problem of FL δ-resolvability [7]. The achievability scheme given in the proof of the direct part of Theorem 4 can also be used in the case of this relaxed measure. Indeed, denoting the VL δ-resolvability with the normalized divergence by ${\tilde{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{X})$, the general formula for ${\tilde{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{X})$ is given in the same form as (101), if the radius of the δ-ball $B_{\delta }^D\left(X^n\right)$ in the definition of $H_{\left[\delta \right]}^D\left(X^n\right)$ is replaced with the normalized divergence. It generally holds that $S_{\mathrm{v}}^D\left(\delta \right|\mathit{X})\ge {\tilde{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{X})$ for all $\delta \ge 0$ because the normalized divergence is smaller than the unnormalized divergence.

As we have seen in Lemma 1, we generally have $S_{\mathrm{v}}^D\left(g\left(\delta \right)\right|\mathit{X})\ge S_{\mathrm{v}}\left(\delta \right|\mathit{X})$ for any $\delta \in [0,1)$ with $g\left(\delta \right)=2{\delta }^2/lnK$. In particular, in the case that $\delta =0$, we obtain the following corollary of Theorems 3 and 4.

Corollary 5.

For any general target source $\mathit{X}$,

$$\begin{array}{cc}\hfill S_{\mathrm{v}}^D\left(0\right|\mathit{X})& =S_{\mathrm{v}}\left(\mathit{X}\right).\hfill \end{array}$$

(103)

Corollary 5 indicates that the ${\mathrm{v}}_D\left(0\right)$-resolvability $S_{\mathrm{v}}^D\left(0\right|\mathit{X})$ coincides with the $\mathrm{v}$-resolvability $S_{\mathrm{v}}\left(\mathit{X}\right)$ and is also characterized by the r.h.s. of (85). By (88), it also implies that $S_{\mathrm{v}}^D\left(0\right|\mathit{X})=R_{\mathrm{v}}^{*}\left(\mathit{X}\right)$, where $R_{\mathrm{v}}^{*}\left(\mathit{X}\right)$ denotes the minimum error-vanishing achievable rate with VL source codes for $\mathit{X}$.

Proof of Theorem 4.

(1)

Converse Part:

Let R be ${\mathrm{v}}_D\left(\delta \right)$-achievable. Then, there exists $U^{\left(L_n\right)}$ and ${\phi }_n$ satisfying (92) and

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{lim\; sup}{\delta }_n\le \delta ,\hfill \end{array}$$

(104)

where we define ${\delta }_n=D({\tilde{X}}^n\left|\right|X^n)$ with ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$. Equation (104) implies that, for any given $\gamma >0$, it holds that ${\delta }_n\le \delta +\gamma$ for all $n\ge n_0$ with some $n_0>0$, and therefore

$$\begin{array}{c}\hfill H_{[\delta +\gamma ]}^D\left(X^n\right)\le H_{\left[{\delta }_n\right]}^D\left(X^n\right)(\forall n\ge n_0)\end{array}$$

(105)

since $H_{\left[\delta \right]}^D\left(X^n\right)$ is a nonincreasing function of $\delta$. Since $P_{{\tilde{X}}^n}\in B_{{\delta }_n}^D\left(X^n\right)$, we have

$$\begin{array}{c}\hfill H_{\left[{\delta }_n\right]}^D\left(X^n\right)\le H\left({\tilde{X}}^n\right).\end{array}$$

(106)

On the other hand, it follows from (23) that

$$\begin{array}{cc}\hfill H\left({\tilde{X}}^n\right)& \le H\left(U^{\left(L_n\right)}\right)=\mathbb{E}\left[L_n\right]+H\left(L_n\right),\hfill \end{array}$$

(107)

where the inequality is due to the fact that ${\phi }_n$ is a deterministic mapping and ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$.

Combining (105)–(107) yields

$$\begin{array}{cc}\hfill H_{[\delta +\gamma ]}^D\left(\mathit{X}\right)& =\underset{n\to \infty }{lim\; sup}\frac{1}{n}{H}_{[\delta +\gamma ]}^D\left(X^n\right)\hfill \\ \hfill & \le \underset{n\to \infty }{lim\; sup}\frac{1}{n}\mathbb{E}\left[L_n\right]+\underset{n\to \infty }{lim\; sup}\frac{1}{n}H\left(L_n\right)\le R,\hfill \end{array}$$

(108)

where we used (25) and (92) for the last inequality. Since $\gamma >0$ is arbitrary, we have

$$\begin{array}{c}\hfill \underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ]}^D\left(\mathit{X}\right)\le R.\end{array}$$

(109)

(2)

Direct Part:

We modify the achievability scheme in the proof of the direct part of Theorem 3. Although the proof of this part is quite similar to that of the direct part of Theorem 3, we give here the full proof in order to avoid subtle possible confusions. We may assume that $H^{+}:={lim}_{\gamma \downarrow 0}{H}_{[\delta +\gamma ]}^D\left(\mathit{X}\right)$ is finite ($H^{+}<+\infty$). Letting $R=H^{+}+\mu$, where $\mu >0$ is an arbitrary constant, we shall show that R is ${\mathrm{v}}_D\left(\delta \right)$-achievable.

Let $V^n$ be a random variable subject to $P_{V^n}\in B_{\delta +\gamma }^D\left(X^n\right)$ satisfying

$$\begin{array}{c}\hfill H_{[\delta +\gamma ]}^D\left(X^n\right)+\gamma \ge H\left(V^n\right)\end{array}$$

(110)

for any fixed $\gamma \in (0,\frac{1}{2}]$. We can choose a $c_n>0$ so large that

$$\begin{array}{c}\hfill {\gamma }_0:=Pr\{V^n\notin T_n\}\le \gamma \end{array}$$

(111)

where

$$\begin{array}{c}\hfill T_n:=\left\{\mathit{x}\in {\mathcal{X}}^n:\frac{1}{n}log\frac{1}{P_{V^n}\left(\mathit{x}\right)}\le c_n\right\}.\end{array}$$

(112)

We also define

$$\begin{array}{c}\hfill \ell \left(\mathit{x}\right):=⌈log\frac{1}{P_{V^n}\left(\mathit{x}\right)}+n\gamma ⌉for\mathit{x}\in T_n.\end{array}$$

(113)

Letting, for $m=1,2,\dots ,{\beta }_n:=\lceil n(c_n+\gamma )\rceil$,

$$\begin{array}{c}\hfill S_n\left(m\right):=\left\{\mathit{x}\in {\mathcal{X}}^n:\ell \left(\mathit{x}\right)=m\right\},\end{array}$$

(114)

these sets form a partition of $T_n$:

$$\begin{array}{c}\hfill \bigcup _{m=1}^{{\beta }_n}{S}_n\left(m\right)=T_n.\end{array}$$

(115)

We set $L_n$ so that

$$\begin{array}{c}\hfill Pr\{L_n=m\}=\frac{Pr\{V^n\in S_n\left(m\right)\}}{Pr\{V^n\in T_n\}}=\frac{Pr\{V^n\in S_n\left(m\right)\}}{1-{\gamma }_0},\end{array}$$

(116)

which satisfies

$$\begin{array}{c}\hfill \sum _{m=1}^{{\beta }_n}Pr\{L_n=m\}=\frac{Pr\{V^n\in T_n\}}{1-{\gamma }_0}=1,\end{array}$$

(117)

and, hence, the probability distribution of $U^{\left(L_n\right)}$ is given as

$$\begin{array}{c}\hfill P_{U^{\left(L_n\right)}}(\mathit{u},m):=Pr\{U^{\left(L_n\right)}=\mathit{u},L_n=m\}=\frac{Pr\{V^n\in S_n\left(m\right)\}}{(1-{\gamma }_0)K^m}(\forall \mathit{u}\in {\mathcal{U}}^m).\end{array}$$

(118)

(a)

Construction of Mapping ${\phi }_n:{\mathcal{U}}^{*}\to {\mathcal{X}}^n$:

Index the elements in $S_n\left(m\right)$ as ${\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_{|S_n\left(m\right)|}(m=1,2,\dots ,{\beta }_n)$, where it holds that

$$\begin{array}{c}\hfill |S_n\left(m\right)|\le K^{m-n\gamma }\end{array}$$

(119)

(cf. (61)–(63)). For $i=1,2,\dots ,|S_n\left(m\right)|$, define ${\tilde{A}}_i^{\left(m\right)}\subset {\mathcal{U}}^m$ as the set of sequences $\mathit{u}\in {\mathcal{U}}^m$ so that

$$\begin{array}{c}\hfill \sum _{\mathit{u}\in {\tilde{A}}_i^{\left(m\right)}}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)\le \frac{P_{V^n}\left({\mathit{x}}_i\right)}{1-{\gamma }_0}<\sum _{\mathit{u}\in {\tilde{A}}_i^{\left(m\right)}}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)+\frac{Pr\{L_n=m\}}{K^m}\end{array}$$

(120)

and

$$\begin{array}{c}\hfill {\tilde{A}}_i^{\left(m\right)}\cap {\tilde{A}}_j^{\left(m\right)}=\varnothing (i\ne j).\end{array}$$

(121)

If

$$\begin{array}{c}\hfill \sum _{i=1}^{|S_n\left(m\right)|}\sum _{\mathit{u}\in {\tilde{A}}_i^{\left(m\right)}}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)<\frac{1}{1-{\gamma }_0}\sum _{i=1}^{|S_n\left(m\right)|}{P}_{V^n}\left({\mathit{x}}_i\right)=Pr\{L_n=m\},\end{array}$$

(122)

then add a ${\mathit{u}}_i\in {\mathcal{U}}^m\setminus \left({\cup }_j{\tilde{A}}_j^{\left(m\right)}\right)$ to obtain

$$\begin{array}{c}\hfill A_i^{\left(m\right)}={\tilde{A}}_i^{\left(m\right)}\cup \left\{{\mathit{u}}_i\right\}\end{array}$$

(123)

for $i=1,2,\dots$ in order, until it holds that with some $1\le c\le |S_n\left(m\right)|$

$$\begin{array}{c}\hfill \bigcup _{i=1}^c{A}_i^{\left(m\right)}\cup \bigcup _{i=c+1}^{|S_n\left(m\right)|}{\tilde{A}}_i^{\left(m\right)}={\mathcal{U}}^m,\end{array}$$

(124)

where ${\mathit{u}}_1,{\mathit{u}}_2,\cdots$ are selected to be all distinct. Since $|{\mathcal{U}}^m|=K^m$ and

$$\begin{array}{c}\hfill \sum _{\mathit{u}\in {\mathcal{U}}^m}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)=\sum _{\mathit{u}\in {\mathcal{U}}^m}\frac{Pr\{V^n\in S_n\left(m\right)\}}{(1-{\gamma }_0)K^m}=Pr\{L_n=m\},\end{array}$$

(125)

such a $1\le c\le |S_n\left(m\right)|$ always exists. For simplicity, we set for $i=c+1,c+2,\dots ,|S_n\left(m\right)|$

$$\begin{array}{c}\hfill A_i^{\left(m\right)}={\tilde{A}}_i^{\left(m\right)}\end{array}$$

(126)

and for $i=1,2,\dots ,|S_n\left(m\right)|$

$$\begin{array}{c}\hfill {\phi }_n\left(\mathit{u}\right)={\mathit{x}}_{i}for\mathit{u}\in A_i^{\left(m\right)},\end{array}$$

(127)

which defines the random variable ${\tilde{X}}^n$ with values in ${\mathcal{X}}^n$ such that

$$\begin{array}{c}\hfill P_{{\tilde{X}}^n}\left({\mathit{x}}_i\right)=\sum _{\mathit{u}\in A_i^{\left(m\right)}}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)({\mathit{x}}_i\in S_n\left(m\right)),\end{array}$$

(128)

that is, ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$. Notice that, by this construction, we have

$$\begin{array}{c}\hfill \left|P_{{\tilde{X}}^n}\left({\mathit{x}}_i\right)-\frac{P_{V^n}\left({\mathit{x}}_i\right)}{1-{\gamma }_0}\right|\le \frac{Pr\{L_n=m\}}{K^m}=\frac{Pr\{V^n\in S_n\left(m\right)\}}{(1-{\gamma }_0)K^m}\end{array}$$

(129)

for $i=1,2,\dots ,|S_n\left(m\right)|;m=1,2,\dots ,{\beta }_n$, and

$$\begin{array}{c}\hfill Pr\{{\tilde{X}}^n\notin T_n\}=0\mathrm{and}Pr\{V^n\notin T_n\}\le \gamma .\end{array}$$

(130)

(b)

Evaluation of Average Length:

The average length $\mathbb{E}\left[L_n\right]$ is evaluated as follows:

$$\begin{array}{cc}\hfill \mathbb{E}\left[L_n\right]& =\sum _{m=1}^{{\beta }_n}\sum _{\mathit{u}\in {\mathcal{U}}^m}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)·m\hfill \\ \hfill & =\sum _{m=1}^{{\beta }_n}\sum _{i=1}^{|S_n\left(m\right)|}\sum _{\mathit{u}\in A_i^{\left(m\right)}}{P}_{U^{\left(L_n\right)}}(\mathit{u},m)·m\hfill \\ \hfill & =\sum _{m=1}^{{\beta }_n}\sum _{{\mathit{x}}_i\in S_n\left(m\right)}{P}_{{\tilde{X}}^n}\left({\mathit{x}}_i\right)·m,\hfill \end{array}$$

(131)

where we have used ${\mathcal{U}}^m={\bigcup }_{i=1}^{|S_n\left(m\right)|}{A}_i^{\left(m\right)}$ and (128). For ${\mathit{x}}_i\in S_n\left(m\right)$ we obtain from (129) and the right inequality of (130)

$$\begin{array}{cc}\hfill P_{{\tilde{X}}^n}\left({\mathit{x}}_i\right)& \le \frac{P_{V^n}\left({\mathit{x}}_i\right)}{1-{\gamma }_0}+\frac{Pr\{V^n\in S_n\left(m\right)\}}{(1-{\gamma }_0)K^m}\hfill \\ \hfill & =\frac{1}{1-{\gamma }_0}\left(P_{V^n}\left({\mathit{x}}_i\right)+\frac{Pr\{V^n\in S_n\left(m\right)\}}{K^m}\right)\hfill \\ \hfill & \le \frac{1}{1-{\gamma }_0}\left(1+\frac{1}{P_{V^n}\left({\mathit{x}}_i\right)K^m}\right)P_{V^n}\left({\mathit{x}}_i\right)\hfill \\ \hfill & \le \left(1+2\gamma \right)\left(1+\frac{1}{K^{n\gamma }}\right)P_{V^n}\left({\mathit{x}}_i\right),\hfill \end{array}$$

(132)

where, to derive the last inequality, we have used the fact $0\le {\gamma }_0\le \gamma \le \frac{1}{2}$ and

$$\begin{array}{c}\hfill P_{V^n}\left({\mathit{x}}_i\right)\ge K^{-(m-n\gamma )}(\forall {\mathit{x}}_i\in S_n\left(m\right)).\end{array}$$

(133)

It should be noticed that (132) also implies that

$$\begin{array}{c}\hfill P_{{\tilde{X}}^n}\left(\mathit{x}\right)\le \left(1+2\gamma \right)\left(1+\frac{1}{K^{n\gamma }}\right)P_{V^n}\left(\mathit{x}\right)(\forall \mathit{x}\in {\mathcal{X}}^n)\end{array}$$

(134)

since $P_{{\tilde{X}}^n}\left(\mathit{x}\right)=0$ for $\mathit{x}\notin T_n={\bigcup }_{m=1}^{{\beta }_n}{S}_n\left(m\right)$. Plugging the inequality

$$\begin{array}{c}\hfill m\le log\frac{1}{P_{V^n}\left({\mathit{x}}_i\right)}+n\gamma +1(\forall {\mathit{x}}_i\in S_n\left(m\right))\end{array}$$

(135)

and (132) into (131), we obtain

$$\begin{array}{cc}\hfill \mathbb{E}\left[L_n\right]& \le (1+2\gamma )\left(1+\frac{1}{K^{n\gamma }}\right)\hfill \\ \hfill & ·\sum _{m=1}^{{\beta }_n}\sum _{{\mathit{x}}_i\in S_n\left(m\right)}{P}_{V^n}\left({\mathit{x}}_i\right)\left(log\frac{1}{P_{V^n}\left({\mathit{x}}_i\right)}+n\gamma +1\right)\hfill \\ \hfill & \le (1+2\gamma )\left(1+\frac{1}{K^{n\gamma }}\right)\left(H\left(V^n\right)+n\gamma +1\right).\hfill \end{array}$$

(136)

Thus, we obtain from (136)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}\mathbb{E}\left[L_n\right]& \le (1+2\gamma )\left\{\underset{n\to \infty }{lim\; sup}\frac{1}{n}H\left(V^n\right)+\gamma \right\}\hfill \\ \hfill & \le (1+2\gamma )\left\{\underset{n\to \infty }{lim\; sup}\frac{1}{n}{H}_{[\delta +\gamma ]}^D\left(X^n\right)+2\gamma \right\}\hfill \\ \hfill & \le (1+2\gamma )(H^{+}+2\gamma ),\hfill \end{array}$$

(137)

where the second inequality follows from (110). Since we have assumed that $H^{+}$ is finite and $\gamma \in (0,\frac{1}{2}]$ is arbitrary, the r.h.s. of (137) can be made as close to $H^{+}$ as desired. Therefore, for all sufficiently small $\gamma >0$, we obtain

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}\mathbb{E}\left[L_n\right]& \le H^{+}+\mu =R\hfill \end{array}$$

(138)

(c)

Evaluation of Divergence:

The divergence $D\left({\tilde{X}}^n\right|\left|X^n\right)$ can be rewritten as

$$\begin{array}{c}\hfill D({\tilde{X}}^n\left|\right|X^n)=D({\tilde{X}}^n\left|\right|V^n)+\mathbb{E}\left[log\frac{P_{V^n}\left({\tilde{X}}^n\right)}{P_{X^n}\left({\tilde{X}}^n\right)}\right].\end{array}$$

(139)

In view of (132), we obtain

$$\begin{array}{cc}\hfill D\left({\tilde{X}}^n\right|\left|V^n\right)& =\sum _{m=1}^{{\beta }_n}\sum _{\mathit{x}\in S_n\left(m\right)}{P}_{{\tilde{X}}^n}\left(\mathit{x}\right)log\frac{P_{{\tilde{X}}^n}\left(\mathit{x}\right)}{P_{V^n}\left(\mathit{x}\right)}\hfill \\ \hfill & \le \sum _{m=1}^{{\beta }_n}\sum _{\mathit{x}\in S_n\left(m\right)}{P}_{{\tilde{X}}^n}\left(\mathit{x}\right)log\left\{\left(1+2\gamma \right)\left(1+\frac{1}{K^{n\gamma }}\right)\right\}\hfill \\ \hfill & \le \frac{2\gamma }{lnK}+log\left(1+\frac{1}{K^{n\gamma }}\right)\hfill \end{array}$$

(140)

and

$$\begin{array}{cc}\hfill \mathbb{E}\left[log\frac{P_{V^n}\left({\tilde{X}}^n\right)}{P_{X^n}\left({\tilde{X}}^n\right)}\right]& =\sum _{\mathit{x}\in {\mathcal{X}}^n}{P}_{{\tilde{X}}^n}\left(\mathit{x}\right)log\frac{P_{V^n}\left(\mathit{x}\right)}{P_{X^n}\left(\mathit{x}\right)}\hfill \\ \hfill & \le \left(1+2\gamma \right)\left(1+\frac{1}{K^{n\gamma }}\right)D(V^n\left|\right|X^n)\hfill \\ \hfill & \le \left(1+2\gamma \right)(\delta +\gamma )\left(1+\frac{1}{K^{n\gamma }}\right),\hfill \end{array}$$

(141)

where to obtain the last inequality we used the fact that $P_{V^n}\in B_{\delta +\gamma }^D\left(X^n\right)$. Plugging (140) and (141) into (139) yields

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}D({\tilde{X}}^n\left|\right|X^n)& \le \frac{2\gamma }{lnK}+(1+2\gamma )(\delta +\gamma )\hfill \\ \hfill & \le \delta +\gamma (2\delta +5),\hfill \end{array}$$

(142)

where we have used the fact that $\frac{2\gamma }{lnK}\le 3\gamma$ for all $K\ge 2$ and the assumption $0<\gamma \le \frac{1}{2}$ to derive the last inequality. Since $\gamma \in (0,\frac{1}{2}]$ is arbitrary and we have (138), R is ${\mathrm{v}}_D\left(\delta \right)$-achievable.

□

5. Mean and VL Channel Resolvability

So far, we have studied the problem of source resolvability, whereas the problem of channel resolvability has been introduced by Han and Verdú [6] to investigate the capacity of identification codes [11]. In a conventional problem of this kind, a target output distribution $P_{Y^n}$ via a channel $W^n$ due to an input $X^n$ is approximated by encoding the FL uniform random number $U_{M_n}$ as a channel input. In this section, we generalize the problem of such channel resolvability to that in the variable-length setting.

5.1. Definitions

Let $\mathcal{X}$ and $\mathcal{Y}$ be finite or countably infinite alphabets. Let $\mathit{W}={\left\{W^n\right\}}_{n=1}^{\infty }$ be a general channel, where $W^n:{\mathcal{X}}^n\to {\mathcal{Y}}^n$ denotes a stochastic mapping. We denote by $\mathit{Y}={\left\{Y^n\right\}}_{n=1}^{\infty }$ the output process via $\mathit{W}$ due to an input process $\mathit{X}={\left\{X^n\right\}}_{n=1}^{\infty }$, where $X^n$ and $Y^n$ take values in ${\mathcal{X}}^n$ and ${\mathcal{Y}}^n$, respectively. Again, we do not impose any assumptions such as stationarity or ergodicity on either $\mathit{X}$ or $\mathit{W}$. As in the previous sections, we will identify $X^n$ and $Y^n$ with their probability distributions $P_{X^n}$ and $P_{Y^n}$, respectively, and these symbols are used interchangeably.

In this section, we consider several types of problems of approximating a target output distribution $P_{Y^n}$. The first one is the problem of mean-resolvability [6], in which the channel input is allowed to be an arbitrary general source.

Definition 5

(mean $\delta$-channel resolvability: variational distance). Let $\delta \in [0,1)$ be fixed arbitrarily. A resolution rate $R\ge 0$ is said to be mean δ-achievable for $\mathit{X}$ (under the variational distance) if there exists a general source $\tilde{\mathit{X}}={\left\{{\tilde{X}}^n\right\}}_{n=1}^{\infty }$ satisfying

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}H\left({\tilde{X}}^n\right)& \le R,\hfill \end{array}$$

(143)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}d(P_{Y^n},P_{{\tilde{Y}}^n})& \le \delta ,\hfill \end{array}$$

(144)

where ${\tilde{Y}}^n$ denotes the output via $W^n$ due to input ${\tilde{X}}^n$. The infimum of all mean δ-achievable rates for $\mathit{X}$, i.e.,

$$\begin{array}{c}\hfill {\overline{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W}):=inf\{R:Rismean\delta -achievablefor\mathit{X}\}\end{array}$$

(145)

is referred to as the mean δ-resolvability for $\mathit{X}$. We also define the mean δ-resolvability for the worst input as

$$\begin{array}{c}\hfill {\overline{S}}_{\mathrm{v}}\left(\delta \right|\mathit{W}):=\underset{\mathit{X}}{sup}{\overline{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W}).\end{array}$$

(146)

On the other hand, we may also consider the problem of VL channel resolvability. Here, the VL uniform random number $U^{\left(L_n\right)}$ is defined as in the foregoing sections. Consider the problem of approximating the target output distribution $P_{Y^n}$ via $W^n$ due to $X^n$ by using another input ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$ with a deterministic mapping ${\phi }_n:{\mathcal{U}}^{*}\to {\mathcal{X}}^n$.

Definition 6

(VL $\delta$-channel resolvability: variational distance). Let $\delta \in [0,1)$ be fixed arbitrarily. A resolution rate $R\ge 0$ is said to be VL δ-achievable or simply $\mathrm{v}\left(\delta \right)$-achievable for $\mathit{X}$ (under the variational distance) if there exists a VL uniform random number $U^{\left(L_n\right)}$ and a deterministic mapping ${\phi }_n:{\mathcal{U}}^{*}\to {\mathcal{X}}^n$ satisfying

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}\mathbb{E}\left[L_n\right]& \le R,\hfill \end{array}$$

(147)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}d(P_{Y^n},P_{{\tilde{Y}}^n})& \le \delta ,\hfill \end{array}$$

(148)

where $\mathbb{E}[·]$ denotes the expected value and ${\tilde{Y}}^n$ denotes the output via $W^n$ due to input ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$. The infimum of all $v\left(\delta \right)$-achievable rates for $\mathit{X}$, i.e.,

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W}):=inf\{R:Risv\left(\delta \right)-achievablefor\mathit{X}\}\end{array}$$

(149)

is called the VL δ-channel resolvability or simply $v\left(\delta \right)$-channel resolvability for $\mathit{X}$. We also define the VL δ-channel resolvability or simply $v\left(\delta \right)$-channel resolvability for the worst input as

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\delta \right|\mathit{W}):=\underset{\mathit{X}}{sup}{S}_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W}).\end{array}$$

(150)

When $W^n$ is the identity mapping, the problem of channel resolvability reduces to that of source resolvability, which has been investigated in the foregoing sections. In this sense, the problem of channel resolvability is a generalization of the problem of source resolvability.

Similarly to the problem of source resolvability, we may also use the divergence between the target output distribution $P_{Y^n}$ and the approximated output distribution $P_{{\tilde{Y}}^n}$ as an approximation measure.

Definition 7

(mean $\delta$-channel resolvability: divergence). Let $\delta \ge 0$ be fixed arbitrarily. A resolution rate $R\ge 0$ is said to be mean δ-achievable for $\mathit{X}$ (under the divergence) if there exists a general source $\tilde{\mathit{X}}={\left\{{\tilde{X}}^n\right\}}_{n=1}^{\infty }$ satisfying

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}H\left({\tilde{X}}^n\right)& \le R,\hfill \end{array}$$

(151)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}D({\tilde{Y}}^n\left|\right|Y^n)& \le \delta ,\hfill \end{array}$$

(152)

where ${\tilde{Y}}^n$ denotes the output via $W^n$ due to input ${\tilde{X}}^n$. The infimum of all mean δ-achievable rates for $\mathit{X}$, i.e.,

$$\begin{array}{c}\hfill {\overline{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{X},\mathit{W}):=inf\{R:Rismean\delta -achievablefor\mathit{X}\}\end{array}$$

(153)

is referred to as the mean δ-channel resolvability for $\mathit{X}$. We also define the mean δ-channel resolvability for the worst input as

$$\begin{array}{c}\hfill {\overline{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{W}):=\underset{\mathit{X}}{sup}{\overline{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{X},\mathit{W}).\end{array}$$

(154)

Definition 8

(VL $\delta$-channel resolvability: divergence). Let $\delta \ge 0$ be fixed arbitrarily. A resolution rate $R\ge 0$ is said to be VL δ-achievable or simply $v_D\left(\delta \right)$-achievable for $\mathit{X}$ (under the divergence) if there exists a VL uniform random number $U^{\left(L_n\right)}$ and a deterministic mapping ${\phi }_n:{\mathcal{U}}^{*}\to {\mathcal{X}}^n$ satisfying

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}\mathbb{E}\left[L_n\right]& \le R,\hfill \end{array}$$

(155)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}D({\tilde{Y}}^n\left|\right|Y^n)& \le \delta ,\hfill \end{array}$$

(156)

where $\mathbb{E}[·]$ denotes the expected value and ${\tilde{Y}}^n$ denotes the output via $W^n$ due to input ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$. The infimum of all ${\mathrm{v}}_D\left(\delta \right)$-achievable rates for $\mathit{X}$, i.e.,

$$\begin{array}{c}\hfill S_{\mathrm{v}}^D\left(\delta \right|\mathit{X},\mathit{W}):=inf\{R:Ris{\mathrm{v}}_D\left(\delta \right)-achievablefor\mathit{X}\}\end{array}$$

(157)

is called the VL δ-channel resolvability or simply ${\mathrm{v}}_D\left(\delta \right)$-channel resolvability for $\mathit{X}$. We also define the VL δ-channel resolvability or simply ${\mathrm{v}}_D\left(\delta \right)$-channel resolvability for the worst input as

$$\begin{array}{c}\hfill S_{\mathrm{v}}^D\left(\delta \right|\mathit{W}):=\underset{\mathit{X}}{sup}{S}_{\mathrm{v}}^D\left(\delta \right|\mathit{X},\mathit{W}).\end{array}$$

(158)

Remark 12.

Since the outputs of a deterministic mapping ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$ form a general source $\tilde{\mathit{X}}$, it holds that

$$\begin{array}{cc}\hfill {\overline{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W})& \le S_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W})(\delta \in [0,1)),\hfill \end{array}$$

(159)

$$\begin{array}{cc}\hfill {\overline{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{X},\mathit{W})& \le S_{\mathrm{v}}^D\left(\delta \right|\mathit{X},\mathit{W})(\delta \ge 0)\hfill \end{array}$$

(160)

for any general source $\mathit{X}$ and general channel $\mathit{W}$. These relations lead to the analogous relation for the mean/VL δ-channel resolvability for the worst input:

$$\begin{array}{cc}\hfill {\overline{S}}_{\mathrm{v}}\left(\delta \right|\mathit{W})& \le S_{\mathrm{v}}\left(\delta \right|\mathit{W})(\delta \in [0,1)),\hfill \end{array}$$

(161)

$$\begin{array}{cc}\hfill {\overline{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{W})& \le S_{\mathrm{v}}^D\left(\delta \right|\mathit{W})(\delta \ge 0).\hfill \end{array}$$

(162)

5.2. General Formulas

For a given general source $\mathit{X}={\left\{X^n\right\}}_{n=1}^{\infty }$ and a general channel $\mathit{W}={\left\{W^n\right\}}_{n=1}^{\infty }$, let $\mathit{Y}={\left\{Y^n\right\}}_{n=1}^{\infty }$ be the channel output via $\mathit{W}$ due to input $\mathit{X}$. We define

$$\begin{array}{c}\hfill H_{\left[\delta \right],W^n}\left(X^n\right)=\underset{P_{V^n}\in B_{\delta }(X^n,W^n)}{inf}H\left(V^n\right),\end{array}$$

(163)

$$\begin{array}{c}\hfill H_{\left[\delta \right],W^n}^D\left(X^n\right)=\underset{P_{V^n}\in B_{\delta }^D(X^n,W^n)}{inf}H\left(V^n\right)\end{array}$$

(164)

where $H\left(V^n\right)$ denotes the Shannon entropy of $V^n$ and $B_{\delta }(X^n,W^n)$ and $B_{\delta }^D(X^n,W^n)$ are defined as

$$\begin{array}{cc}\hfill B_{\delta }(X^n,W^n)& =\left\{P_{V^n}\in \mathcal{P}\left({\mathcal{X}}^n\right):d(P_{Y^n},P_{Z^n})\le \delta \right\},\hfill \end{array}$$

(165)

$$\begin{array}{cc}\hfill B_{\delta }^D(X^n,W^n)& =\left\{P_{V^n}\in \mathcal{P}\left({\mathcal{X}}^n\right):D(Z^n\left|\right|Y^n)\le \delta \right\},\hfill \end{array}$$

(166)

respectively, with $Z^n$ defined as the output via $W^n$ due to input $V^n$. Both $H_{\left[\delta \right],W^n}\left(X^n\right)$ and $H_{\left[\delta \right],W^n}^D\left(X^n\right)$ are nonincreasing functions of $\delta$. In addition, we define

$$\begin{array}{cc}\hfill H_{\left[\delta \right],\mathit{W}}\left(\mathit{X}\right)& =\underset{n\to \infty }{lim\; sup}\frac{1}{n}{H}_{\left[\delta \right],W^n}\left(X^n\right),\hfill \end{array}$$

(167)

$$\begin{array}{cc}\hfill H_{\left[\delta \right],\mathit{W}}^D\left(\mathit{X}\right)& =\underset{n\to \infty }{lim\; sup}\frac{1}{n}{H}_{\left[\delta \right],W^n}^D\left(X^n\right),\hfill \end{array}$$

(168)

which play an important role in characterizing the mean/VL $\delta$-channel resolvability.

We show the general formulas for the mean/VL $\delta$-channel resolvability.

Theorem 5

(with variational distance). For any input process $\mathit{X}$ and any general channel $\mathit{W}$,

$$\begin{array}{c}\hfill {\overline{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W})=S_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W})=\underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ],\mathit{W}}\left(\mathit{X}\right)(\delta \in [0,1)).\end{array}$$

(169)

In particular,

$$\begin{array}{cc}\hfill {\overline{S}}_{\mathrm{v}}\left(\delta \right|\mathit{W})=S_{\mathrm{v}}\left(\delta \right|\mathit{W})& =\underset{\mathit{X}}{sup}\underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ],\mathit{W}}\left(\mathit{X}\right)(\delta \in [0,1)).\hfill \end{array}$$

(170)

Theorem 6

(with divergence). For any input process $\mathit{X}$ and any general channel $\mathit{W}$,

$$\begin{array}{c}\hfill {\overline{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{X},\mathit{W})=S_{\mathrm{v}}^D\left(\delta \right|\mathit{X},\mathit{W})=\underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ],\mathit{W}}^D\left(\mathit{X}\right)(\delta \ge 0).\end{array}$$

(171)

In particular,

$$\begin{array}{c}\hfill {\overline{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{W})=S_{\mathrm{v}}^D\left(\delta \right|\mathit{W})=\underset{\mathit{X}}{sup}\underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ],\mathit{W}}^D\left(\mathit{X}\right)(\delta \ge 0).\end{array}$$

(172)

Remark 13.

It can be easily verified that the variational distance satisfies

$$\begin{array}{cc}\hfill d(P_{Y^n},P_{Z^n})& \le d(P_{X^n},P_{V^n}),\hfill \end{array}$$

(173)

and, therefore, we have $B_{\delta }\left(X^n\right)\subseteq B_{\delta }(X^n,W^n)$. This relation and formulas (32) and (169) indicate that

$$\begin{array}{c}\hfill S_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W})\le S_{\mathrm{v}}\left(\delta \right|\mathit{X})(\delta \in [0,1))\end{array}$$

(174)

for any given channel $\mathit{W}$. Likewise, it is well known that the divergence satisfies the data processing inequality $D({\tilde{Y}}^n\left|\right|Y^n)\le D({\tilde{X}}^n\left|\right|X^n)$ [33], and formulas (101) and (171) lead to

$$\begin{array}{c}\hfill S_{\mathrm{v}}^D\left(\delta \right|\mathit{X},\mathit{W})\le S_{\mathrm{v}}^D\left(\delta \right|\mathit{X})(\delta \ge 0),\end{array}$$

(175)

regardless of channel $\mathit{W}$.

Remark 14.

It is obvious that Theorems 5 and 6 reduce to Theorems 3 and 4, respectively, when the channel $\mathit{W}$ is the identity mapping. Precisely, for the identity mapping $\mathit{W}=\mathit{I}$, the mean δ-resolvability and the $\mathrm{v}\left(\delta \right)$-channel resolvability for $\mathit{X}$ are given by

$$\begin{array}{c}\hfill {\overline{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X})=S_{\mathrm{v}}\left(\delta \right|\mathit{X})=\underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ]}\left(\mathit{X}\right),\end{array}$$

(176)

where ${\overline{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X})$ denotes the mean δ-resolvability ${\overline{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W})$ for the identity mapping $\mathit{W}$. The analogous relationship holds under the divergence:

$$\begin{array}{c}\hfill {\overline{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{X})=S_{\mathrm{v}}^D\left(\delta \right|\mathit{X})=\underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ]}^D\left(\mathit{X}\right),\end{array}$$

(177)

where ${\overline{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{X})$ denotes the mean δ-resolvability ${\overline{S}}_{\mathrm{v}}^D\left(\delta \right|\mathit{X},\mathit{W})$ for the identity mapping $\mathit{W}=\mathit{I}$. Thus, it turns out that Theorems 5 and 6 are indeed generalizations of Theorems 3 and 4.

Proof of Theorems 5 and 6.

(1)

Converse Part:

Because of the general relationship (159), to prove the converse part of Theorem 5, it suffices to show that

$$\begin{array}{cc}\hfill {\overline{S}}_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W})& \ge \underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ],\mathit{W}}\left(\mathit{X}\right).\hfill \end{array}$$

(178)

Let R be mean $\delta$-achievable for $\mathit{X}$ under the variational distance. Then, there exists a general source $\tilde{\mathit{X}}={\left\{{\tilde{X}}^n\right\}}_{n=1}^{\infty }$ satisfying (143) and

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}{\delta }_n\le \delta ,\end{array}$$

(179)

where ${\delta }_n:=d(P_{Y^n},P_{{\tilde{Y}}^n})$. Fixing $\gamma >0$ arbitrarily, we have ${\delta }_n\le \delta +\gamma$ for all $n\ge n_0$ with some $n_0>0$ and then

$$\begin{array}{c}\hfill H_{[\delta +\gamma ],W^n}\left(X^n\right)\le H_{\left[{\delta }_n\right],W^n}\left(X^n\right)(n\ge n_0)\end{array}$$

(180)

since $H_{\left[\delta \right],W^n}\left(X^n\right)$ is a nonincreasing function of $\delta$. Since $P_{{\tilde{X}}^n}\in B_{{\delta }_n}(X^n,W^n)$, we have $H_{\left[{\delta }_n\right],W^n}\left(X^n\right)\le H\left({\tilde{X}}^n\right)$. Thus, we obtain from (143)

$$\begin{array}{c}\hfill H_{[\delta +\gamma ],\mathit{W}}\left(\mathit{X}\right)\le \underset{n\to \infty }{lim\; sup}\frac{1}{n}H\left({\tilde{X}}^n\right)\le R.\end{array}$$

(181)

Since $\gamma >0$ is an arbitrary constant, this implies (178).

The converse part of Theorem 6 can be proven in an analogous way with due modifications.

(2)

Direct Part:

Because of the general relationship (159), to prove the direct part (achievability) of Theorem 5, it suffices to show that, for any fixed $\gamma >0$, the resolution rate

$$\begin{array}{c}\hfill R=\underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ],\mathit{W}}\left(\mathit{X}\right)+3\gamma \end{array}$$

(182)

is $\mathrm{v}\left(\delta \right)$-achievable for $\mathit{X}$ under the variational distance.

Let $P_{V^n}\in B_{\delta +\gamma }(X^n,W^n)$ be a source satisfying

$$\begin{array}{c}\hfill H\left(V^n\right)\le H_{[\delta +\gamma ],W^n}\left(X^n\right)+\gamma .\end{array}$$

(183)

Then, by the same argument to derive (80) and (82) as developed in the proof of the direct part of Theorem 3, we can construct a VL uniform random number $U^{\left(L_n\right)}$ and a deterministic mapping ${\phi }_n:{\mathcal{U}}^{*}\to {\mathcal{X}}^n$ satisfying

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n}\mathbb{E}\left[L_n\right]\le \underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ],\mathit{W}}\left(\mathit{X}\right)+3\gamma =R\end{array}$$

(184)

and

$$\begin{array}{c}\hfill d(P_{{\tilde{X}}^n},P_{V^n})\le \frac{1}{2}{K}^{-n\gamma }+\gamma ,\end{array}$$

(185)

where ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$. Let $Z^n$ denote the output random variable via $W^n$ due to input $V^n$. Then, letting ${\tilde{Y}}^n$ be the output via channel $W^n$ due to input ${\tilde{X}}^n$, we can evaluate $d(P_{{\tilde{Y}}^n},P_{Z^n})$ as

$$\begin{array}{cc}\hfill d(P_{{\tilde{Y}}^n},P_{Z^n})& =\frac{1}{2}\sum _{\mathit{y}\in {\mathcal{Y}}^n}|P_{{\tilde{Y}}^n}\left(\mathit{y}\right)-P_{Z^n}\left(\mathit{y}\right)|\hfill \\ \hfill & =\frac{1}{2}\sum _{\mathit{y}\in {\mathcal{Y}}^n}\left|\sum _{\mathit{x}\in {\mathcal{X}}^n}W\left(\mathit{y}\right|\mathit{x})\left(P_{{\tilde{X}}^n}\left(\mathit{x}\right)-P_{V^n}\left(\mathit{x}\right)\right)\right|\hfill \\ \hfill & \le \frac{1}{2}\sum _{\mathit{y}\in {\mathcal{Y}}^n}\sum _{\mathit{x}\in {\mathcal{X}}^n}W\left(\mathit{y}\right|\mathit{x})\left|P_{{\tilde{X}}^n}\left(\mathit{x}\right)-P_{V^n}\left(\mathit{x}\right)\right|\hfill \\ \hfill & =d(P_{{\tilde{X}}^n},P_{V^n})\le \frac{1}{2}{K}^{-n\gamma }+\gamma .\hfill \end{array}$$

(186)

Thus, we obtain

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}d(P_{Y^n},P_{{\tilde{Y}}^n})& \le \underset{n\to \infty }{lim\; sup}d(P_{Y^n},P_{Z^n})+\underset{n\to \infty }{lim\; sup}d(P_{{\tilde{Y}}^n},P_{Z^n})\hfill \\ \hfill & \le \delta +2\gamma ,\hfill \end{array}$$

(187)

where we have used the fact $P_{V^n}\in B_{\delta +\gamma }(X^n,W^n)$ to derive the last inequality. Since $\gamma >0$ is an arbitrary constant, we can conclude that R is $\mathrm{v}\left(\delta \right)$-achievable for $\mathit{X}$.

The direct part of Theorem 6 can be proven in the same way as Theorem 4 with due modifications. Fixing $P_{V^n}\in B_{\delta +\gamma }^D(X^n,W^n)$ and using the encoding scheme as developed in the proof of Theorem 4, the evaluation of the average length rate is exactly the same, and we can obtain (138). A key step is to evaluate the divergence $D\left({\tilde{Y}}^n\right|\left|Y^n\right)$, which can be rewritten as

$$\begin{array}{c}\hfill D({\tilde{Y}}^n\left|\right|Y^n)=D({\tilde{Y}}^n\left|\right|Z^n)+\mathbb{E}\left[log\frac{P_{Z^n}\left({\tilde{Y}}^n\right)}{P_{Y^n}\left({\tilde{Y}}^n\right)}\right].\end{array}$$

(188)

The first term on the r.h.s. can be bounded as

$$\begin{array}{cc}\hfill D\left({\tilde{Y}}^n\right|\left|Z^n\right)& \le D({\tilde{X}}^n\left|\right|V^n)\le \frac{2\gamma }{lnK}+log\left(1+\frac{1}{K^{n\gamma }}\right)\hfill \end{array}$$

(189)

as in (140), where the left inequality is due to the data processing inequality. Similarly to the derivation of (141), the second term can be bounded as

$$\begin{array}{cc}\hfill \mathbb{E}\left[log\frac{P_{Z^n}\left({\tilde{Y}}^n\right)}{P_{Y^n}\left({\tilde{Y}}^n\right)}\right]& =\sum _{\mathit{y}\in {\mathcal{Y}}^n}\sum _{\mathit{x}\in {\mathcal{X}}^n}{P}_{{\tilde{X}}^n}\left(\mathit{x}\right)W^n\left(\mathit{y}\right|\mathit{x})log\frac{P_{Z^n}\left(\mathit{y}\right)}{P_{Y^n}\left(\mathit{y}\right)}\hfill \\ \hfill & \le \left(1+2\gamma \right)\left(1+\frac{1}{K^{n\gamma }}\right)\sum _{\mathit{y}\in {\mathcal{Y}}^n}\sum _{\mathit{x}\in {\mathcal{X}}^n}{P}_{V^n}\left(\mathit{x}\right)W^n\left(\mathit{y}\right|\mathit{x})log\frac{P_{Z^n}\left(\mathit{y}\right)}{P_{Y^n}\left(\mathit{y}\right)}\hfill \\ \hfill & =\left(1+2\gamma \right)\left(1+\frac{1}{K^{n\gamma }}\right)D(Z^n\left|\right|Y^n),\hfill \end{array}$$

(190)

where we have used (134). Here, $D\left(Z^n\right|\left|Y^n\right)\le \delta +\gamma$ because $Z^n$ is the output via $W^n$ due to input $V^n\in B_{\delta +\gamma }^D(X^n,W^n)$. The rest of the steps are the same as in the proof of Theorem 4.

□

6. Second-Order VL Channel Resolvability

So far, we have analyzed the first-order VL resolvabilities and established various first-order resolvability theorems. One of the next important steps is the second-order analysis, and so, in this section, we generalize VL resolvabilities in the second-order setting.

6.1. Definitions

We now turn to considering the second-order resolution rates [26,27,29]. First, we consider the VL resolvability based on the variational distance.

Definition 9

(VL $(\delta ,R)$-channel resolvability: variational distance). A second-order resolution rate $L\in (-\infty ,+\infty )$ is said to be VL $(\delta ,R)$-achievable (under the variational distance) for $\mathit{X}$ with $\delta \in [0,1)$ if there exist a VL uniform random number $U^{\left(L_n\right)}$ and a deterministic mapping ${\phi }_n:{\mathcal{U}}^{*}\to {\mathcal{X}}^n$ satisfying

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(\mathbb{E}\left[L_n\right]-nR\right)& \le L,\hfill \end{array}$$

(191)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}d(P_{Y^n},P_{{\tilde{Y}}^n})& \le \delta ,\hfill \end{array}$$

(192)

where ${\tilde{Y}}^n$ denotes the output via $W^n$ due to input ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$. The infimum of all VL $(\delta ,R)$-achievable rates for $\mathit{X}$ is denoted by

$$\begin{array}{c}\hfill T_{\mathrm{v}}(\delta ,R|\mathit{X},\mathit{W}):=inf\{L:LisVL(\delta ,R)-achievablefor\mathit{X}\}.\end{array}$$

(193)

When $\mathit{W}$ is the identity mapping $\mathit{I}$, $T_{\mathrm{v}}(\delta ,R|\mathit{X},\mathit{W})$ is simply denoted by $T_{\mathrm{v}}(\delta ,R|\mathit{X})$ (source resolvability).

Next, we may consider the VL resolvability with the divergence instead of the variational distance.

Definition 10

(VL $(\delta ,R)$-channel resolvability: divergence). A second-order resolution rate $L\in (-\infty ,+\infty )$ is said to be VL $(\delta ,R)$-achievable for $\mathit{X}$ (with the divergence) where $\delta \ge 0$ if there exists a VL uniform random number $U^{\left(L_n\right)}$ and a deterministic mapping ${\phi }_n:{\mathcal{U}}^{*}\to {\mathcal{X}}^n$ satisfying

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(\mathbb{E}\left[L_n\right]-nR\right)& \le L,\hfill \end{array}$$

(194)

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}D({\tilde{Y}}^n\left|\right|Y^n)& \le \delta ,\hfill \end{array}$$

(195)

where ${\tilde{Y}}^n$ denotes the output via $W^n$ due to input ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$. The infimum of all VL $(\delta ,R)$-achievable rates for $\mathit{X}$ is denoted as

$$\begin{array}{c}\hfill T_{\mathrm{v}}^D(\delta ,R|\mathit{X},\mathit{W}):=inf\{L:LisVL(\delta ,R)-achievablefor\mathit{X}\}.\end{array}$$

(196)

When $\mathit{W}$ is the identity mapping $\mathit{I}$, $T_{\mathrm{v}}^D(\delta ,R|\mathit{X},\mathit{W})$ is simply denoted by $T_{\mathrm{v}}^D(\delta ,R|\mathit{X})$ (source resolvability).

Remark 15.

It is easily verified that

$$\begin{array}{c}\hfill T_{\mathrm{v}}(\delta ,R|\mathit{X},\mathit{W})=\left\{\begin{array}{cc}+\infty \hfill & forR<S_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W})\hfill \\ -\infty \hfill & forR>S_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W}).\hfill \end{array}\right.\end{array}$$

(197)

Hence, only the case $R=S_{\mathrm{v}}\left(\delta \right|\mathit{X},\mathit{W})$ is of interest to us. The same remark also applies to $T_{\mathrm{v}}^D(\delta ,R|\mathit{X},\mathit{W})$.

6.2. General Formulas

We establish general formulas for the second-order resolvability. The proofs of the following theorems are given below subsequently to Remark 17.

Theorem 7

(with variational distance). For any input process $\mathit{X}$ and general channel $\mathit{W}$,

$$\begin{array}{c}\hfill T_{\mathrm{v}}(\delta ,R|\mathit{X},\mathit{W})=\underset{\gamma \downarrow 0}{lim}\underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(H_{[\delta +\gamma ],W^n}\left(X^n\right)-nR\right)(\delta \in [0,1),R\ge 0).\end{array}$$

(198)

In particular, in the case where $\mathit{W}$ is the identity mapping $\mathit{I}$,

$$\begin{array}{c}\hfill T_{\mathrm{v}}(\delta ,R|\mathit{X})=\underset{\gamma \downarrow 0}{lim}\underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(H_{[\delta +\gamma ]}\left(X^n\right)-nR\right)(\delta \in [0,1),R\ge 0).\end{array}$$

(199)

Theorem 8

(with divergence). For any input process $\mathit{X}$ and general channel $\mathit{W}$,

$$\begin{array}{c}\hfill T_{\mathrm{v}}^D(\delta ,R|\mathit{X},\mathit{W})=\underset{\gamma \downarrow 0}{lim}\underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(H_{[\delta +\gamma ],W^n}^D\left(X^n\right)-nR\right)(\delta \ge 0,R\ge 0).\end{array}$$

(200)

In particular, in the case where $\mathit{W}$ is the identity mapping $\mathit{I}$,

$$\begin{array}{c}\hfill T_{\mathrm{v}}^D(\delta ,R|\mathit{X})=\underset{\gamma \downarrow 0}{lim}\underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(H_{[\delta +\gamma ]}^D\left(X^n\right)-nR\right)(\delta \ge 0,R\ge 0).\end{array}$$

(201)

Remark 16.

As discussed in Section 5, we may also consider using a general source $\tilde{\mathit{X}}$ as an input to channel $\mathit{W}$, and we can define L to be a mean $(\delta ,R)$-achievable rate for $\mathit{X}$ by replacing (191) and (194) with

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(H\left({\tilde{X}}^n\right)-nR\right)& \le L.\hfill \end{array}$$

(202)

Let ${\overline{T}}_{\mathrm{v}}(\delta ,R|\mathit{X},\mathit{W})$ and ${\overline{T}}_{\mathrm{v}}^D(\delta ,R|\mathit{X},\mathit{W})$ denote the infimum of all mean $(\delta ,R)$-achievable rates for $\mathit{X}$ under the variational distance and the divergence, respectively. Then, it is not difficult to verify that

$$\begin{array}{cc}\hfill {\overline{T}}_{\mathrm{v}}(\delta ,R|\mathit{X},\mathit{W})& =T_{\mathrm{v}}(\delta ,R|\mathit{X},\mathit{W})(\delta \in [0,1)),\hfill \end{array}$$

(203)

$$\begin{array}{cc}\hfill {\overline{T}}_{\mathrm{v}}^D(\delta ,R|\mathit{X},\mathit{W})& =T_{\mathrm{v}}^D(\delta ,R|\mathit{X},\mathit{W})(\delta \ge 0).\hfill \end{array}$$

(204)

Thus, there is no loss in the $(\delta ,R)$-achievable resolution rate even if the channel input $\tilde{\mathit{X}}$ is restricted to be generated by the VL uniform random number $U^{\left(L_n\right)}$.

Remark 17.

As in the first-order case, when the channel $\mathit{W}$ is the identity mapping $\mathit{I}$, $T_{\mathrm{v}}(\delta ,R|\mathit{X})$ coincides with the minimum second-order length rate of the VL source codes. More precisely, we denote by $R_{\mathrm{v}}^{*}(\delta ,R|\mathit{X})$ the minimum second-order length rate of a sequence of VL source codes with the first-order average length rate R and an average error probability asymptotically not exceeding δ. Yagi and Nomura [31] have shown that

$$\begin{array}{c}\hfill R_{\mathrm{v}}^{*}(\delta ,R|\mathit{X})=\underset{\gamma \downarrow 0}{lim}\underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(G_{[\delta +\gamma ]}\left(X^n\right)-nR\right)(\delta \in [0,1),R\ge 0).\end{array}$$

(205)

Modifying the proof of Proposition 1 (cf. Appendix C), we can show that the r.h.s. of (199) coincides with that of (205), and, therefore, it generally holds that

$$\begin{array}{c}\hfill T_{\mathrm{v}}(\delta ,R|\mathit{X})=R_{\mathrm{v}}^{*}(\delta ,R|\mathit{X})(\delta \in [0,1),R\ge 0).\end{array}$$

(206)

As a special case, suppose that $\mathit{X}$ is a stationary and memoryless source X with the finite third absolute moment of $log\frac{1}{P_X\left(X\right)}$. In this case, Kostina et al. [25] have recently given a single-letter characterization for $R_{\mathrm{v}}^{*}(\delta ,R|\mathit{X})$ with $R=H_{\left[\delta \right]}\left(\mathit{X}\right)=(1-\delta )H\left(X\right)$ as

$$\begin{array}{c}\hfill R_{\mathrm{v}}^{*}(\delta ,R|\mathit{X})=-\sqrt{\frac{V\left(X\right)}{2\pi }}{e}^{-\frac{{\left(Q^{-1}\left(\delta \right)\right)}^2}{2}},\end{array}$$

(207)

where $V\left(X\right)$ denotes the variance of $log\frac{1}{P_X\left(X\right)}$ (varentropy) and $Q^{-1}$ is the inverse of the complementary cumulative distribution function of the standard Gaussian distribution. In view of the general relation (206), we can also obtain the single-letter characterization for $T_{\mathrm{v}}(\delta ,R|\mathit{X})$:

$$\begin{array}{c}\hfill T_{\mathrm{v}}(\delta ,R|\mathit{X})=-\sqrt{\frac{V\left(X\right)}{2\pi }}{e}^{-\frac{{\left(Q^{-1}\left(\delta \right)\right)}^2}{2}}.\end{array}$$

(208)

It has not yet been made clear whether we can also have a single-letter formula for $T_{\mathrm{v}}(\delta ,R|\mathit{X},\mathit{W})$ when the channel $\mathit{W}$ is memoryless but not necessarily the identity mapping.

Proof of Theorems 7 and 8.

(1)

Converse Part:

We will show the converse part of Theorem 7. The converse part of Theorem 8 can be proved in an analogous way.

Let L be VL $(\delta ,R)$-achievable for $\mathit{X}$ under the variational distance. Then, there exists $U^{\left(L_n\right)}$ and ${\phi }_n$ satisfying (191) and

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{lim\; sup}{\delta }_n\le \delta ,\hfill \end{array}$$

(209)

where we define ${\delta }_n=d(P_{Y^n},P_{{\tilde{Y}}^n}),$ and ${\tilde{Y}}^n$ is the output via $W^n$ due to input ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$. Equation (209) implies that, for any given $\gamma >0$, it holds that ${\delta }_n\le \delta +\gamma$ for all $n\ge n_0$ with some $n_0>0$, and, therefore,

$$\begin{array}{c}\hfill H_{[\delta +\gamma ],W^n}\left(X^n\right)\le H_{\left[{\delta }_n\right],W^n}\left(X^n\right)(\forall n\ge n_0).\end{array}$$

(210)

Since $P_{{\tilde{X}}^n}\in B_{{\delta }_n}(X^n,W^n)$, we have

$$\begin{array}{c}\hfill H_{\left[{\delta }_n\right],W^n}\left(X^n\right)\le H\left({\tilde{X}}^n\right).\end{array}$$

(211)

On the other hand, it follows from (23) that

$$\begin{array}{cc}\hfill H\left({\tilde{X}}^n\right)& \le H\left(U^{\left(L_n\right)}\right)=\mathbb{E}\left[L_n\right]+H\left(L_n\right),\hfill \end{array}$$

(212)

where the inequality is due to the fact that ${\phi }_n$ is a deterministic mapping and ${\tilde{X}}^n={\phi }_n\left(U^{\left(L_n\right)}\right)$.

Combining (210)–(212) yields

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(H_{[\delta +\gamma ],W^n}\left(X^n\right)-nR\right)\hfill \\ \hfill & \le \underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(H\left({\tilde{X}}^n\right)-nR\right)\hfill \\ \hfill & \le \underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(\mathbb{E}\left[L_n\right]-nR\right)+\underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}H\left(L_n\right)\le L,\hfill \end{array}$$

(213)

where we have used (24) and (191) for the last inequality. Since $\gamma >0$ is arbitrary, we have

$$\begin{array}{c}\hfill \underset{\gamma \downarrow 0}{lim}\underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(H_{[\delta +\gamma ],W^n}\left(X^n\right)-nR\right)\le L.\end{array}$$

(214)

(2)

Direct Part:

We will show the direct part (achievability) of Theorem 7 by modifying the argument of Theorems 3 and 5, whereas the direct part of Theorem 8 can be proved in a similar manner by modifying that of Theorem 4 instead of Theorem 3.

Letting

$$\begin{array}{c}\hfill L=\underset{\gamma \downarrow 0}{lim}\underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(H_{[\delta +\gamma ],W^n}\left(X^n\right)-nR\right)+2\gamma ,\end{array}$$

(215)

where $\gamma >0$ is an arbitrary constant, we shall show that L is VL $(\delta ,R)$-achievable for $\mathit{X}$ under the variational distance.

We use the same achievability scheme as in the proof of Theorem 3 with slightly different parameter settings. For $\gamma >0$, we choose a $c_n>0$ so that

$$\begin{array}{c}\hfill Pr\{V^n\notin T_n\}\le \gamma \end{array}$$

(216)

where $P_{V^n}\in B_{\delta +\gamma }(X^n,W^n)$ with $H_{[\delta +\gamma ],W^n}\left(X^n\right)+\gamma \ge H\left(V^n\right)$ and

$$\begin{array}{c}\hfill T_n:=\left\{\mathit{x}\in {\mathcal{X}}^n:\frac{1}{n}log\frac{1}{P_{V^n}\left(\mathit{x}\right)}\le c_n\right\}.\end{array}$$

(217)

We here define

$$\begin{array}{c}\hfill \ell \left(\mathit{x}\right):=\left\{\begin{array}{cc}\lceil log\frac{1}{P_{V^n}\left(\mathit{x}\right)}+\sqrt{n}\gamma \rceil \hfill & for\mathit{x}\in T_n\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(218)

and ${\beta }_n=\lceil (n{c}_n+\sqrt{n}\gamma )\rceil$. Arguing similarly to the proof of Theorems 3 and 5, we can show that there exist ${\phi }_n:{\mathcal{U}}^{*}\to {\mathcal{X}}^n$ and $U^{\left(L_n\right)}$ such that

$$\begin{array}{c}\hfill d(P_{Y^n},P_{{\tilde{Y}}^n})\le \delta +2\gamma +\frac{1}{2}{K}^{-\sqrt{n}\gamma }\end{array}$$

(219)

and

$$\begin{array}{cc}\hfill \mathbb{E}\left[L_n\right]& \le \left(1+\frac{1}{K^{\sqrt{n}\gamma }}\right)\left(H_{[\delta +\gamma ],W^n}\left(X^n\right)+2\sqrt{n}\gamma +1\right).\hfill \end{array}$$

(220)

Therefore, we obtain

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}d(P_{Y^n},P_{{\tilde{Y}}^n})\le \delta +2\gamma \end{array}$$

(221)

and

$$\begin{array}{cc}\hfill & \underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(\mathbb{E}\left[L_n\right]-nR\right)\hfill \\ \hfill & \le \underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(H_{[\delta +\gamma ],W^n}\left(X^n\right)-nR\right)+2\gamma \hfill \\ \hfill & \le \underset{\gamma \downarrow 0}{lim}\underset{n\to \infty }{lim\; sup}\frac{1}{\sqrt{n}}\left(H_{[\delta +\gamma ],W^n}\left(X^n\right)-nR\right)+2\gamma =L.\hfill \end{array}$$

(222)

Since $\gamma >0$ is arbitrary, L is VL $(\delta ,R)$-achievable for $\mathit{X}$.

□

7. Conclusions

We have investigated the problem of VL source/channel resolvability, in which a given target probability distribution is approximated by transforming VL uniform random numbers.

Table 1. Summary of First-Order Resolvability and Information Quantities.

Approximation Measure	Resolvability	Characterization	Theorem #
Fixed-Length Resolvability
Variational Distance	$S_{\mathrm{f}}\left(\mathit{X}\right)$	$\overline{H}\left(\mathit{X}\right)$	Theorem 1 [6]
	$S_{\mathrm{f}}\left(\delta \mathit{X}\right)$	${\overline{H}}_{\delta }\left(\mathit{X}\right)$	Theorem 2 [7]
Variable-Length Resolvability
Variational Distance	$S_{\mathrm{v}}\left(\delta \mathit{X}\right)$	${\displaystyle \underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ]}\left(\mathit{X}\right)}$	Theorem 3
	$S_{\mathrm{v}}(\delta \mathit{X},\mathit{W})$	${\displaystyle \underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ],\mathit{W}}\left(\mathit{X}\right)}$	Theorem 5
Divergence	$S_{\mathrm{v}}^D\left(\delta \mathit{X}\right)$	${\displaystyle \underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ]}^D\left(\mathit{X}\right)}$	Theorem 4
	$S_{\mathrm{v}}^D(\delta \mathit{X},\mathit{W})$	${\displaystyle \underset{\gamma \downarrow 0}{lim}{H}_{[\delta +\gamma ],\mathit{W}}^D\left(\mathit{X}\right)}$	Theorem 6

summarizes various first-order resolvabilities and their characterizations in terms of information quantities. In this table, the theorem numbers that contain the corresponding characterization are also indicated.

In this paper, we have first analyzed the fundamental limits on the VL $\delta$-source resolvability with the variational distance in Theorem 3. The VL $\delta$-source resolvability is essentially characterized in terms of smooth Shannon entropies. In the proof of the direct part, we have developed a simple method for information spectrum slicing, in which sliced information densities quantized to the same integer are approximated by an FL uniform random number of the same length. Next, we have extended the analysis to the $\delta$-source resolvability under the unnormalized divergence in Theorem 4. The smoothed entropy with the divergence again plays an important role in characterizing the $\delta$-source resolvability.

Then, we have addressed the problem of $\delta$-channel resolvability. It has been revealed in Theorems 5 and 6 that using an arbitrary general source as a coin distribution (mean-resolvability problem) cannot go beyond the fundamental limits of the VL resolvability, in which only VL uniform random numbers are allowed to be a coin distribution. As in the case of source resolvability, we have discussed the $\delta$-channel resolvability under the variational distance and the unnormalized divergence. The second-order channel resolvability has been characterized in Theorems 7 and 8 as well as the first-order case. We notice here that a counterpart of the VL uniform random number is the problem of VL source coding, for which the general treatment, focused on overflow/underflow probabilities, is found in [38]. Indeed, when the variational distance is used as an approximation measure, it turns out that the $\delta$-source resolvability is equal to the minimum achievable rate of VL source codes with an error probability of less than or equal to $\delta$. This is a parallel relationship between FL source resolvability and the minimum achievable rate of FL source codes [6,7]. It is of interest to investigate whether there is a coding problem to which the $\delta$-channel resolvability is closely related.

When $\delta =0$, asymptotically exact approximation is required. In the case where the channel $\mathit{W}$ is the identity mapping $\mathit{I}$, it turned out that the source resolvability under the variational distance and the unnormalized divergence coincides and is given by ${lim}_{\gamma \downarrow 0}{H}_{\left[\gamma \right]}\left(\mathit{X}\right)$, where $\mathit{X}$ is the general target source. This result is analogous to the dual problem of VL intrinsic randomness [5,36], in which the maximum achievable rates of VL uniform random numbers extracted from a given source $\mathit{X}$ are the same under two kinds of approximation measures. It should be emphasized that in the case of VL intrinsic randomness, the use of normalized divergence as an approximation measure results in the same general formula as with the variational distance and the unnormalized divergence, which does not necessarily hold in the case of mean/VL resolvability (cf. Remark 11). It is also noteworthy that whereas only the case of $\delta =0$ has been completely solved for the VL intrinsic randomness, we have also dealt with the case of a $\delta >0$ for the VL source/channel resolvability.

When $\mathit{X}$ is a stationary and memoryless source or is even with a one-point spectrum (cf. Corollary 1), the formulas established reduce to a single-letter characterization for the first- and second-order source resolvability under the variational distance. In the case where the divergence is an approximation measure and/or the channel $\mathit{W}$ is a non-identity mapping, however, it has not yet been made clear whether we can derive a single-letter characterization for the $\delta$-source/channel resolvability. This question remains to be studied.

As noted in Remark 10, the order of arguments in divergence $D\left({\tilde{X}}^n\right|\left|X^n\right)$ is important, and it seems difficult to extend the analyses in this paper to the case with the reversed $D\left(X^n\right|\left|{\tilde{X}}^n\right)$ used as an approximation measure. In the context of intrinsic randomness, the reversed divergence is also discussed [23]. Investigating the problem of source/channel resolvability using such divergence is an interesting research topic.