A New Lower Bound for Noisy Permutation Channels via Divergence Packing ^†

Abstract

Noisy permutation channels are applied in modeling biological storage systems and communication networks. For noisy permutation channels with strictly positive and full-rank square matrices, new achievability bounds are given in this paper, which are tighter than existing bounds. To derive this bound, we use the $ϵ$-packing with Kullback–Leibler divergence as a distance and introduce a novel way to illustrate the overlapping relationship of error events. This new bound shows analytically that for such a matrix W, the logarithm of the achievable code size with a given block n and error probability $ϵ$ is closely approximated by $\ell log\left({\displaystyle \frac{\sqrt{n}}{-{\Phi }^{-1}(ϵ/G)}}\right)+logV\left(W\right)$, where $\ell =\mathsf{rank}\left(W\right)-1$, $G=2\left({\displaystyle \genfrac{}{}{0pt}{}{\ell +1}{2}}\right)$, and $V\left(W\right)$ is a characteristic of the channel referred to as channel volume ratio. Our numerical results show that the new achievability bound significantly improves the lower bound of channel coding. Additionally, the Gaussian approximation can replace the complex computations of the new achievability bound over a wide range of relevant parameters.

1. Introduction

The noisy permutation channel, consisting of a discrete memoryless channel (DMC) and a uniform random permutation block, was introduced in [1] and is a point-to-point communication model that captures the out-of-order arrival of packets. Such channels can be used as models of communication networks and DNA storage systems, where the ordering of the codeword does not carry any information. Previously, several advances have been made to asymptotic bounds, including the binary channels in [2], the capacity of full-rank DMCs in [1], and the converse bounds based on divergence covering [3,4].

The code lengths of suitable codes in communication systems are in the order of thousands or hundreds, invalidating the asymptotic assumptions in classical information theory. We initiate the study of new channel coding bounds to extend the information-theoretic results for noisy permutation channels in finite blocklength analysis. Finite blocklength analysis and finer asymptotics are important branches of research in information theory. Interest in this topic has been growing since the seminal works of [5,6,7]. These works suggest that the channel coding rate in the finite blocklength regime is closely related to the information density [8], i.e., the stochastic measure of the input distribution and channel noise. The second-order approximation of conventional channels involves the variance of the information density, which has been shown to approximate the channel coding rate at short blocklengths.

The codeword positions are randomly permuted in noisy permutation channels. Conventional analysis techniques, specifically the dependence-testing (DT) bound and the random coding union (RCU) bound ([9] Theorems 17 and 18), become inapplicable. Since the messages are mapped to different probability distributions in noisy permutation channels [1], the only statistical information the receiver would use from the received codeword ${\mathsf{Y}}^n$ is which marginal distribution ${\mathsf{Y}}^n$ belongs to. Therefore, the finer asymptotic completely differs from that in conventional channels.

The main contributions of our work are the following:

• We present a new nonasymptotic achievability bound for noisy permutation channels that have strictly positive square matrices W with full rank. The two main ingredients of our proof are the following: the $ϵ$-packing [10,11] with Kullback–Leibler (KL) divergence as a distance, and an analysis for the error events that decouple the union of error events from the message set. Additionally, this new bound is stronger than existing bounds ([1] Equation (36)).
• We show that the finite blocklength achievable code size can be approximated by

  $$log{M}^{*}(n,ϵ)\approx \ell log\left({\displaystyle \frac{\sqrt{n}}{-{\Phi }^{-1}(ϵ/G)}}\right)+logV\left(W\right),$$

  (1)

  where $\ell =\mathsf{rank}\left(W\right)-1$, $G=2\left({\displaystyle \genfrac{}{}{0pt}{}{\ell +1}{2}}\right)$, and $V\left(W\right)$ is the channel volume ratio.
• To complement these results and assist in understanding them, we particularize all these results to typical DMCs, i.e., BSC and BEC permutation channels. Additionally, our Gaussian approximations, through numerical results, lead to tight approximations of the achievable code size for blocklengths n as short as 100 in these cases.

We continue this section with the motivation and application. Section 2 sets up the system model. In Section 3, we provide methods to construct a set of divergence packing centers (message set) and bounds for packing numbers. In Section 4, we present our new achievability bound and particularize this bound to the typical DMCs. Section 5 studies the asymptotic behavior of the achievability bound using Gaussian approximation analysis and applies it to the typical DMCs. In Section 6, we present numerical results. We conclude this paper in Section 7.

1.1. Motivation and Application

The noisy permutation channel models the scenario where codewords undergo reordering, which occurs in communication networks and DNA storage systems. We briefly outline some applications of this channel.

(a)

Communication Networks: First, noisy permutation channels are a suitable model for the multipath routed network in which packets arrive at different delays [12,13]. In such networks, data packets within the same group often take paths of differing lengths, bandwidths, and congestion levels as they traverse the network to the receiver. Consequently, transmission delays exhibit unpredictable variations, causing these packets to arrive at their destination in a potentially different order from their original sending sequence. Moreover, during transmission, data packets may be lost or corrupted due to reasons such as link failures or buffer overflow. Treating all possible packets as the input alphabet, this scenario fits the noisy permutation channel model.

(b)

DNA Storage Systems: The DNA storage systems, known for their high density and reliability over long periods, are another motivation for our research [14,15,16]. Such a system can be seen as an out-of-order communication channel [1,14,17]. The source data is written onto DNA molecules (or codeword strings) consisting of letters from an alphabet of four nucleotides $\{A,C,G,T\}$. Due to physical conditions causing random fragmentation of DNA molecules, long-read sequencing technology, such as nanopore sequencing [18], is employed at the receiver to read entire randomly permuted DNA molecules. In the noisy permutation channel, the DMC matrix models potential errors during the synthesis and storage of DNA molecules, followed by a random permutation block that represents the random permutation of DNA molecules. For a comprehensive overview of DNA storage systems, see [1,14]; studies presenting specific DNA-based storage coding schemes include [17,19,20].

1.2. Notation

We use $\left[n\right]=\{1,\dots ,n\}$, ${\mathbb{Z}}_{\ge a}=\{a,a+1,a+2,\dots \}$ to represent integer intervals. Let $1\{·\}$ denote the indicator function. For a given $\mathcal{X}$ and a random variable $\mathsf{X}\in \mathcal{X}$, we write $\mathsf{X}\sim P_{\mathsf{X}}$ to indicate that the random variable $\mathsf{X}$ follows the distribution $P_{\mathsf{X}}$. Let ${\mathsf{X}}^n=({\mathsf{X}}_1,\dots ,{\mathsf{X}}_n)$ and $x^n=(x_1,\dots ,x_n)$ denote the random vector and its realization in the n-th Cartesian product ${\mathcal{X}}^n$, respectively. A $\left(\right|\mathcal{X}|-1)$-dimensional simplex on ${\mathbb{R}}^{\left|\mathcal{X}\right|}$ is a set of points as follows:

$${\Delta }_{\left|\mathcal{X}\right|-1}=\left\{(p_1,p_2,\dots ,p_{\left|\mathcal{X}\right|})\in {\mathbb{R}}^{\left|\mathcal{X}\right|}|\sum _{x=1}^{\left|\mathcal{X}\right|}{p}_x=1,p_x\ge 0\right\}.$$

(2)

The KL divergence and the total variation are denoted by $D(·\parallel ·)$ and $\mathrm{TV}(·,·)$, respectively. For a matrix A, we use the notation $\mathsf{rank}\left(A\right)$ to represent the rank of matrix A. The probability and mathematical expectation are denoted by $\mathbb{P}[·]$ and $\mathbb{E}[·]$, respectively. The cumulative distribution function of the standard normal distribution is denoted by $\Phi (·)$ and ${\Phi }^{-1}(·)$ is its inverse function.

2. System Model

The code ${\mathcal{C}}_n$ consists of a message set $\mathcal{M}$, an (possibly randomized) encoder function $f_n:\mathcal{M}\to {\mathcal{X}}^n$, and a (possibly randomized) decoder function $g_n:{\mathcal{Y}}^n\to \mathcal{M}\cup \left\{\mathsf{e}\right\}$, where the notation ‘$\mathsf{e}$’ indicates that the decoder chooses “error”. We write $\mathcal{X}$ for the finite input alphabet and $\mathcal{Y}$ for the finite output alphabet. M denotes the achievable code size of code ${\mathcal{C}}_n$.

The input alphabet $\mathcal{X}$ abstracts transmitted codeword symbols in various applications. For instance, in DNA storage applications, $\mathcal{X}$ denotes the alphabet of four nucleotides, while the n-length codeword ${\mathsf{X}}^n\in {\mathcal{X}}^n$ represents the DNA molecule formed by the corresponding n nucleotides. The sender uses encoder $f_n$ to encode the message M into a codeword ${\mathsf{X}}^n$, which is then passed through the DMC W to produce ${\mathsf{Z}}^n\in {\mathcal{Y}}^n$. The DMC is defined by a $\left|\mathcal{X}\right|\times \left|\mathcal{Y}\right|$ matrix W, where $W\left(z\right|x)$ denotes the probability that the output $z\in \mathcal{Y}$ occurs given input $x\in \mathcal{X}$. Finally, ${\mathsf{Z}}^n$ passes through a random permutation block $P_{{\mathsf{Y}}^n|{\mathsf{Z}}^n}$ to generate ${\mathsf{Y}}^n\in {\mathcal{Y}}^n$. The random permutation block $P_{{\mathsf{Y}}^n|{\mathsf{Z}}^n}$ operates as follows. First, a random permutation is denoted as $\sigma :\{1,\dots ,n\}\mapsto \{1,\dots ,n\}$, drawn randomly and uniformly from the symmetric group ${\mathcal{S}}_n$ over $\{1,\dots ,n\}$. Then, ${\mathsf{Y}}^n$ is generated by permuting ${\mathsf{Z}}^n$ according to ${\mathsf{Y}}_i={\mathsf{Z}}_{\sigma \left(i\right)}$ for all $i\in \{1,\dots ,n\}$. The receiver uses decoder $g_n$ to produce the estimate of the message $\widehat{\mathsf{M}}$. We can describe these steps by the following Markov chain:

$$\mathsf{M}\to {\mathsf{X}}^n\to {\mathsf{Z}}^n\to {\mathsf{Y}}^n\to \widehat{\mathsf{M}}.$$

(3)

The channel model of the noisy permutation channel is illustrated in Figure 1. For codewords ${\mathsf{X}}^n$ drawn i.i.d. from $P_{\mathsf{X}}$, the random permutation block does not change the probability distribution of codewords [1], i.e., if ${\mathsf{X}}^n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }{P}_{\mathsf{X}}$, then ${\mathsf{Z}}^n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }{P}_{\mathsf{X}}$ and ${\mathsf{Y}}^n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }{P}_{\mathsf{X}}$.

Figure 1. Illustration of a communication model with a DMC followed by a random permutation.

We say W is strictly positive if all the transition probabilities are greater than 0. We impose the following restrictions on the channel.

Assumption 1.

The channel W is a strictly positive and full-rank square matrix.

For a given code ${\mathcal{C}}_n$, the average error probability of code ${\mathcal{C}}_n$ is

$$\begin{array}{c}\hfill P_e=\mathbb{P}[\mathsf{M}\ne \widehat{\mathsf{M}}].\end{array}$$

(4)

The achievable code size with a given blocklength and probability of error is denoted by $M^{*}(n,ϵ)=max\left\{M|\exists {\mathcal{C}}_n\mathrm{s}.\mathrm{t}.P_e\le ϵ\right\}.$ The code rate of the encoder–decoder pair $(f_n,g_n)$ is denoted by

$$R={\displaystyle \frac{logM}{logn}},$$

(5)

where $log(·)$ is the binary logarithm (with base 2) in this paper. Note that the rate $R(n,ϵ)$ for the noisy permutation channel is not the conventional definition where $R\left(n\right)={\displaystyle \frac{1}{n}}logM$ since the noisy permutation channels will have rate 0 under this definition. The capacity for noisy permutation channels is defined as $C=sup\{R\ge 0:R\mathrm{is}\mathrm{achievable}\}$.

3. Message Set and Divergence Packing

A divergence packing is a set of centers in simplex ${\Delta }_{\left|\mathcal{Y}\right|-1}$ such that the minimum distance between each center is greater than some KL distance. The following definition abides by the packing number.

Definition 1.

The achievability space of the marginal distribution is defined by ${\Delta }_{\left|\mathcal{Y}\right|-1}^{*}=\{P_{\mathsf{X}}\circ W|P_{\mathsf{X}}\in {\Delta }_{\left|\mathcal{X}\right|-1}\}$.

Definition 2.

Let $\{P_1,\dots ,P_M\}\subset {\Delta }_{\left|\mathcal{Y}\right|-1}^{*}$ be the set of divergence packing centers. The divergence packing number on ${\Delta }_{\left|\mathcal{Y}\right|-1}^{*}$ is defined by

$$\begin{array}{c}\hfill N^{*}(r,|\mathcal{Y}\left|\right)=max\left\{M|\exists \{P_1,\dots ,P_M\}\mathrm{s}.\mathrm{t}.\underset{i\ne j}{min}D(P_i\parallel P_j)\ge r\right\},\end{array}$$

(6)

where $r>0$ is the packing radius.

Here, we provide some intuition for using divergence packing in noisy permutation channels. Using the ML decoder, we relate the non-asymptotic channel performance to the likelihood ratio of two distributions (decoding metric). The statistical mean of this decoding metric is the KL divergence, which arises from applying the law of large numbers as the blocklength grows. Thus, using divergence packing can obtain an upper bound of the error probability by lower bounding the distance between each distribution, and the code size can be analyzed asymptotically via the Berry–Esseen bound (see Section 5). These factors motivate us to use the KL divergence in constructing the message set.

Since the messages correspond to different distributions in noisy permutation channels, the message set can be equivalent to the set of marginal distributions at the receiver (e.g., see [1]). In the sequel, we denote the marginal distribution corresponding to message m by $P_m$.

Additionally, in Gaussian approximations, we need the following definition.

Definition 3.

Let $\mathrm{vol}(y,{\Delta }_{\left|\mathcal{Y}\right|-1}^{*})$ and $\mathrm{vol}(y,{\Delta }_{\left|\mathcal{Y}\right|-1})$ be, respectively, the volume of the projection of ${\Delta }_{\left|\mathcal{Y}\right|-1}^{*}$ and ${\Delta }_{\left|\mathcal{Y}\right|-1}$ from ${\mathbb{R}}^{\left|\mathcal{Y}\right|}$ to space ${\mathbb{R}}^{\left|\mathcal{Y}\right|-1}$, in which the y-th dimension is removed. The channel volume ratio is defined as

$$V\left(W\right)=\underset{y\in \mathcal{Y}}{max}{\displaystyle \frac{\mathrm{vol}(y,{\Delta }_{\left|\mathcal{Y}\right|-1}^{*})}{\mathrm{vol}(y,{\Delta }_{\left|\mathcal{Y}\right|-1})}}.$$

(7)

Next, we present several lower bounds on packing numbers. In the two-dimensional case, our construction achieves tighter bounds. For higher dimensions, our primary tool is the volume bound for $ϵ$-packing (e.g., see [8] Theorem 27.3). These results form the foundation for constructing marginal probability distributions in subsequent sections, while also playing a key role in the analysis of Gaussian approximations.

3.1. Binary Case

We first give the lower bound of the packing number in the binary case. Consider ${\Delta }_1^{*}=\left\{(q,1-q)\right|{\delta }_1\le q\le 1-{\delta }_2\}$, where $0<{\delta }_1<1-{\delta }_2<1$. Define

$$\begin{array}{c}\hfill {\Gamma }_{b,2}^{*}=\left\{(q,1-q)|q={\displaystyle \frac{\xi a}{\lfloor 1/b\rfloor }}+{\delta }_1\mathrm{where}a\in {\mathbb{Z}}_{\ge 0}\right\},\end{array}$$

(8)

where $\xi =1-{\delta }_1-{\delta }_2,b>0$.

Then, we have the following result proved in Appendix A.

Proposition 1.

Fix ${\Delta }_1^{*}=\left\{(q,1-q)\right|{\delta }_1\le q\le 1-{\delta }_2\}$, where ${\delta }_1>0$ and ${\delta }_2>0$. We can construct a set of packing centers by (8) with $b={\displaystyle \frac{1}{\xi }}\sqrt{{\displaystyle \frac{r}{2loge}}}$ and $\xi =1-{\delta }_1-{\delta }_2$ such that

$$N^{*}(r,2)=\lfloor 1/r\rfloor +1.$$

(9)

3.2. General Case

Next, we introduce a general method for constructing the set of packing centers and the bounds of their size. Let $b>0$, and we consider the following set:

$$\begin{array}{c}\hfill {\Gamma }_{b,\left|\mathcal{Y}\right|}^{*}=\left\{P\in {\Delta }_{\left|\mathcal{Y}\right|-1}^{*}|\begin{array}{c}P=\left({\displaystyle \frac{a_1}{\lfloor 1/b\rfloor }},\dots ,{\displaystyle \frac{a_{\left|\mathcal{Y}\right|}}{\lfloor 1/b\rfloor }}\right)\mathrm{where}{a}_1,\dots ,a_{\left|\mathcal{Y}\right|}\in {\mathbb{Z}}_{\ge 0}\hfill \end{array}\right\}.\end{array}$$

(10)

The intuition behind constructing this set is that the minimum distance of distributions in this uniform structure can be bounded by the total variation distance. Then, we can obtain the set of divergence packing centers that have a certain radius by applying Pinsker’s inequality.

We have the following lower bound proved in Appendix A.

Theorem 1.

Fix a W that satisfies Assumption 1 and generate ${\Delta }_{\left|\mathcal{Y}\right|-1}^{*}$. We can construct a set of packing centers by (10) with $b=\sqrt{{\displaystyle \frac{r}{2loge}}}$ such that

$$\begin{array}{c}\hfill N^{*}(r,|\mathcal{Y}\left|\right)\ge V\left(W\right){\left({\displaystyle \frac{loge}{8r}}\right)}^{{\displaystyle \frac{\left|\mathcal{Y}\right|-1}{2}}},\end{array}$$

(11)

where $r>0$ is the packing radius.

4. New Bounds on Rate

In this section, we introduce our new bound, which is based on divergence packing and yields the spirit of the RCU bound. The key ingredient is our analysis of error events.

To that end, we introduce some definitions. Suppose we have a set of marginal distributions $\mathcal{M}$, which is constructed by (8) or (10) with any $b>0$. Fixing a $P=(p_1,\dots ,p_{\left|\mathcal{Y}\right|})\in \mathcal{M}$, we are often concerned with the divergence packing centers close to P. To do this, we consider $Q_{a,b}\left(P\right)=(q_1,\dots ,q_{\left|\mathcal{Y}\right|})$, where $a\ne b$, $q_a=p_a+{\displaystyle \frac{K}{\lfloor 1/r\rfloor }},q_b=p_b-{\displaystyle \frac{K}{\lfloor 1/r\rfloor }}$ and $q_i=p_i$ for $i\in \mathcal{Y}\setminus \{a,b\}$. K is a constant that has value $\xi$ or 1 when $\mathcal{M}$ is constructed by (8) or (10), respectively. We define ${\mathcal{R}}_P$ as the neighboring set of P:

$${\mathcal{R}}_P=\left\{Q_{a,b}\left(P\right)|a,b\in \mathcal{Y},a\ne b\right\}\cap \mathcal{M}.$$

(12)

In general, the distribution $Q_{a,b}\left(P\right)$ coincides with distributions in the set $\mathcal{M}$, except near the boundaries of the simplex where $Q_{a,b}\left(P\right)$ may violate the constraints of the probability space. We use the intersection operation in (12) to make sure all elements of ${\mathcal{R}}_P$ remain within the simplex. For convenience, we use $j\in \left[\right|{\mathcal{R}}_P\left|\right]$ to index $Q_j\in {\mathcal{R}}_P$, and we say $Q_j\in {\mathcal{R}}_P$ is the neighboring distribution of P. By counting, we have $\left|{\mathcal{R}}_P\right|\le 2({\displaystyle \genfrac{}{}{0pt}{}{\left|\mathcal{Y}\right|}{2}})$.

For the marginal distribution $P_m$ corresponding to the transmitted message m, we use the log-likelihood ratio to define the following decoding metric:

$$d(m,j,y):=log{\displaystyle \frac{P_m\left(y\right)}{Q_j\left(y\right)}},$$

(13)

where $Q_j\in {\mathcal{R}}_{P_m}$.

Then, the proof of our main result consists of three parts, each detailed in one of the following subsections. In the first subsection, we introduce a lemma. This lemma shows that the message sets constructed by (8) or (10) have an overlapping relationship for error events. In the second subsection, we use this lemma to give an equivalent expression for the error probability. The third subsection contains our main result. Additionally, we particularize this new bound to BSC and BEC permutation channels in the fourth and fifth subsections, respectively.

4.1. Overlapping of Error Events

Intuitively, the rate of decay of $P_e$ is dominated by the rate of decay of the probability of error in distinguishing neighboring messages. In order to use this intuition mathematically, we need to analyze the relationship between error events. The following lemma, proved in Appendix B, does this and can be used for analyzing random coding bounds.

Lemma 1.

Let $\mathcal{M}$ be constructed by (8) or (10) with any $b>0$. Fix a $P\in \mathcal{M}$. Then, for every $\Lambda =({\lambda }_1,\dots ,{\lambda }_{\left|\mathcal{Y}\right|})\in {\Delta }_{\left|\mathcal{Y}\right|-1}$ and $Q=(q_1,\dots ,q_{\left|\mathcal{Y}\right|})\in \mathcal{M}\setminus \left({\mathcal{R}}_P\cap \left\{P\right\}\right)$, if

$$\prod _{i=1}^{\left|\mathcal{Y}\right|}{p}_i^{{\lambda }_i}\le \prod _{i=1}^{\left|\mathcal{Y}\right|}{q}_i^{{\lambda }_i},$$

(14)

there exists a $Q^{*}=(q_1^{*},\dots ,q_{\left|\mathcal{Y}\right|}^{*})\in {\mathcal{R}}_P$ such that

$$\prod _{i=1}^{\left|\mathcal{Y}\right|}{p}_i^{{\lambda }_i}\le \prod _{i=1}^{\left|\mathcal{Y}\right|}{\left(q_i^{*}\right)}^{{\lambda }_i}.$$

(15)

4.2. Equivalent Expression

In this subsection, we give a lemma tailored to our purposes. It follows directly from Lemma 1.

Lemma 2.

For the set of marginal distributions $\mathcal{M}$ constructed by (8) or (10) with any $b>0$, we have

$$\begin{array}{c}\hfill \mathbb{P}\left[\bigcup _{j=1,j\ne m}^{\left|\mathcal{M}\right|}\left\{P_j^n\left({\mathsf{Y}}^n\right)\ge P_m^n\left({\mathsf{Y}}^n\right)\right\}\right]=\mathbb{P}\left[\bigcup _{j=1}^{|{\mathcal{R}}_{P_m}|}\left\{Q_j^n\left({\mathsf{Y}}^n\right)\ge P_m^n\left({\mathsf{Y}}^n\right)\right\}\right],\end{array}$$

(16)

where the sequence $Y^n$ is drawn i.i.d. from $P_m$ and $Q_j\in {\mathcal{R}}_{P_m}$.

Proof. 

Using Lemma 1, for $j\in \left[\right|\mathcal{M}\left|\right]$, if $\left\{P_j^n\left(y^n\right)\ge P_m^n\left(y^n\right)\right\}$ occurs, we obtain $j\in \left[\right|{\mathcal{R}}_{P_m}\left|\right]$ such that $\left\{Q_j^n\left(y^n\right)\ge P_m^n\left(y^n\right)\right\}$ occurs. Then, we observe

$$\mathbb{P}\left[\bigcup _{j=1,j\ne m}^{\left|\mathcal{M}\right|}\left\{P_j^n\left({\mathsf{Y}}^n\right)\ge P_m^n\left({\mathsf{Y}}^n\right)\right\}\right]=\sum _{y^n\in {\mathcal{Y}}^n}{P}_m^n\left(y^n\right)\bigcup _{j=1,j\ne m}^{\left|\mathcal{M}\right|}1\left\{P_j^n\left(y^n\right)\ge P_m^n\left(y^n\right)\right\}$$

(17)

$$=\sum _{y^n\in {\mathcal{Y}}^n}{P}_m^n\left(y^n\right)\bigcup _{j=1}^{|{\mathcal{R}}_{P_m}|}1\left\{Q_j^n\left(y^n\right)\ge P_m^n\left(y^n\right)\right\}.$$

(18)

$$=\mathbb{P}\left[\bigcup _{j=1}^{|{\mathcal{R}}_{P_m}|}\left\{Q_j^n\left({\mathsf{Y}}^n\right)\ge P_m^n\left({\mathsf{Y}}^n\right)\right\}\right],$$

(19)

where in (17) we sum over all possible outputs, and (18) relies on Lemma 1 by setting $\Lambda$ to be the empirical distribution of $y^n$. This completes the proof of (16). □

Remark 1.

If the transmitted message is m, Lemma 2 shows that the union of error events ${\cup }_{j=1,j\ne m}^{\left|\mathcal{M}\right|}\left\{P_j^n\ge P_m^n\right\}$ can be equivalent to a minor union ${\cup }_{j=1}^{|{\mathcal{R}}_{P_m}|}\left\{Q_j^n\ge P_m^n\right\}$ on ${\mathcal{R}}_{P_m}$. Its size depends on the size of the output alphabet.

4.3. Main Result: New Lower Bound

The main result in this section is the following. Please refer to Appendix C for the proof.

Theorem 2.

Fix a W that satisfies Assumption 1 and generate ${\Delta }_{\left|\mathcal{Y}\right|-1}^{*}$. Let the set of marginal distributions $\mathcal{M}$ be constructed by (8) or (10) with any $b>0$. Then, there exists a code ${\mathcal{C}}_n$ (average error probability) with achievable code size $\left|\mathcal{M}\right|$ such that

$$\begin{array}{c}\hfill ϵ\le min\left\{1,{\displaystyle \frac{1}{\left|\mathcal{M}\right|}}\sum _{m=1}^{\left|\mathcal{M}\right|}\sum _{j=1}^{|{\mathcal{R}}_{P_m}|}\sum _{y^n\in {\mathcal{Y}}^n}{P}_m^n\left(y^n\right)·1\left\{\sum _{i=1}^{n}d(m,j,y_i)\le 0\right\}\right\}.\end{array}$$

(20)

Remark 2.

Theorem 2 relies on the message set constructed by (8) or (10). We restrict the channel W to be a full-rank square matrix, which makes ${\Delta }_{\left|\mathcal{Y}\right|-1}^{*}$ an equal-dimensional subspace of ${\Delta }_{\left|\mathcal{Y}\right|-1}$. Therefore, the evenly spaced grid structure on ${\Delta }_{\left|\mathcal{Y}\right|-1}^{*}$ can be constructed by using (8) or (10). Without this condition, we cannot apply (10) unless we make strong assumptions about W.

Remark 3.

Theorem 2 upper bounds the probability of error with the sum of the probabilities of error events on ${\mathcal{R}}_{P_m}$ instead of $\mathcal{M}$, which makes our bound much stronger than [1]. In fact, if we do not use Lemma 2 but instead apply the union bound and the second moment method for TV distance ([21] Lemma 4.2(iii)) in the proof of Theorem 2, we can obtain the existing bound ([1] Equation (36)).

4.4. BSC Permutation Channels

In this subsection, we particularize the nonasymptotic bounds to the BSC, i.e., the DMC matrix is

$$W=\left[\begin{array}{cc}1-\delta & \delta \\ \delta & 1-\delta \end{array}\right],$$

(21)

denoted ${\mathsf{BSC}}_{\delta }$. According to Proposition 1 and Theorem 1, using (8) to construct the set of marginal distributions is better than (10) in the binary case. Therefore, we focus on the former in this subsection. For convenience, we denote $P_m(·)=({\delta }_m,1-{\delta }_m)$. For $i,j\in \left[\right|\mathcal{M}\left|\right]$, let ${\delta }_i<{\delta }_j$ if $i<j$. Then, for a $P_m$, we clearly have

$${\mathcal{R}}_{P_m}=\left\{\begin{array}{cc}\{P_{m-1},P_{m+1}\},\hfill & 2\le m\le \left|\mathcal{M}\right|-1,\hfill \\ \left\{P_2\right\},\hfill & m=1,\hfill \\ \left\{P_{\left|\mathcal{M}\right|-1}\right\},\hfill & m=\left|\mathcal{M}\right|.\hfill \end{array}\right.$$

(22)

Let

$$f_1(n,T_i)=\left\{\begin{array}{cc}{\sum }_{t=0}^{T_i}\left({\displaystyle \genfrac{}{}{0pt}{}{n}{t}}\right){\delta }_i^t{(1-{\delta }_i)}^{n-t},\hfill & i\ge 2,\hfill \\ 0,\hfill & i=1,\hfill \end{array}\right.$$

(23)

and

$$f_2(n,T_i)=\left\{\begin{array}{cc}{\sum }_{t=T_i}^n\left({\displaystyle \genfrac{}{}{0pt}{}{n}{t}}\right){\delta }_i^t{(1-{\delta }_i)}^{n-t},\hfill & i\le \left|\mathcal{M}\right|-1,\hfill \\ 0,\hfill & i=\left|\mathcal{M}\right|.\hfill \end{array}\right.$$

(24)

The following bound is a straightforward generalization of Theorem 2.

Theorem 3

(Achievability). For the BSC permutation channel with crossover probability δ, there exists a code ${\mathcal{C}}_n$ such that

$$\begin{array}{c}\hfill ϵ\le {\displaystyle \frac{{\sum }_{i=1}^{\left|\mathcal{M}\right|}}{\left|\mathcal{M}\right|}}min\left\{1,f_1(n,{\underset{¯}{T}}_i)+f_2(n,{\overline{T}}_i)\right\},\end{array}$$

(25)

where

$${\underset{¯}{T}}_i=⌊{\displaystyle \frac{nlog\frac{1-{\delta }_{i-1}}{1-{\delta }_i}}{log\frac{{\delta }_i(1-{\delta }_{i-1})}{{\delta }_{i-1}(1-{\delta }_i)}}}⌋$$

(26)

and

$${\overline{T}}_i=⌈{\displaystyle \frac{nlog\frac{1-{\delta }_{i+1}}{1-{\delta }_i}}{log\frac{{\delta }_i(1-{\delta }_{i+1})}{{\delta }_{i+1}(1-{\delta }_i)}}}⌉.$$

(27)

The set of marginal distributions is constructed by (8) and for the radius r, we have

$$\left|\mathcal{M}\right|=\lfloor 1/r\rfloor +1.$$

(28)

Proof. 

Let us assume the transmitted message is $m\in \mathcal{M}$, corresponding to the marginal distribution $P_m$. In BSC, we focus on the set (22). Using the same argument in the proof of Lemma 1, the term corresponding to $d(m,m-1,y_i)$ in (20) can be computed as

$$\begin{array}{cc}\hfill \sum _{y^n\in {\mathcal{Y}}^n}{P}_m^n\left(y^n\right)·& 1\left\{\sum _{i=1}^{n}d(m,m-1,y_i)\le 0\right\}=\sum _{t=0}^{{\underset{¯}{T}}_m}\left({\displaystyle \genfrac{}{}{0pt}{}{n}{t}}\right){\delta }_m^t{(1-{\delta }_m)}^{n-t},\hfill \end{array}$$

(29)

where ${\underset{¯}{T}}_m$ follows from (A19). Similarly, the term corresponding to $d(m,m+1,y_i)$ in (20) can be computed as

$$\begin{array}{cc}\hfill \sum _{y^n\in {\mathcal{Y}}^n}{P}_m^n\left(y^n\right)·& 1\left\{\sum _{i=1}^{n}d(m,m+1,y_i)\le 0\right\}=\sum _{t={\overline{T}}_m}^n\left({\displaystyle \genfrac{}{}{0pt}{}{n}{t}}\right){\delta }_m^t{(1-{\delta }_m)}^{n-t},\hfill \end{array}$$

(30)

where ${\overline{T}}_m$ follows from (A19). Equations (29) and (30) are substituted into Theorem 2 to complete the proof. □

4.5. BEC Permutation Channels

The BEC permutation channel with erasure probability $\delta$ consists of input alphabet $\mathcal{X}=\{0,1\}$ and output alphabet $\mathcal{Y}=\{0,e,1\}$, where the conditional distribution is

$$\forall z\in \mathcal{Y},\forall x\in \mathcal{X},W\left(z\right|x)=\left\{\begin{array}{cc}1-\delta ,\hfill & z=x,\hfill \\ \delta ,\hfill & z=e,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(31)

Moreover, we denote such a channel as ${\mathsf{BEC}}_{\delta }$ for convenience.

Next, we have the following achievability bound.

Proposition 2.

For BEC permutation channels with erasure probability $2\delta$, there exists a code ${\mathcal{C}}_n$ such that the average probability of error and the code size satisfy (25) and (28), respectively.

Proof. 

The derivations of this proof follow from ([1] Proposition 6), and we include the details for the sake of completeness. We first note that the BSC matrix satisfies the Doeblin minorization condition (e.g., see [1] Definition 5) with ${\mathbf{1}}^{\top }/2$ and constant $2\delta$. Using ([1] Lemma 6), we find that ${\mathsf{BSC}}_{\delta }$ is a degraded version of ${\mathsf{BEC}}_{2\delta }$. Then, for the encoder and decoder pairs $(f_n,g_n)$ for BSC permutation channels and $(f_n,{\tilde{g}}_n)$ for BEC permutation channels, the average probability of error satisfies [1] Equation (36).

$$P_e(f_n,g_n,{\mathsf{BSC}}_{\delta })=P_e(f_n,{\tilde{g}}_n,{\mathsf{BEC}}_{2\delta }).$$

(32)

Then, the argument of the proof of Theorem 3 is repeated. This completes the proof. □

5. Gaussian Approximation

We turn to the asymptotic analysis of the noisy permutation channel for a given blocklength and average probability of error.

5.1. Auxiliary Lemmata

To establish our Gaussian approximation, we will present two lemmata. The first lemma we will exploit is an important tool in the Gaussian approximation analysis:

Lemma 3

(Berry–Esseen, [22] Chapter XVI.5, Theorem 2). Fix a positive integer n. Let ${\mathsf{Z}}_i$ indexed by $(1,\dots ,n)$ be independent. Then, for any real x and $C_0\le 6$ we have

$$\left|\mathbb{P}\left[\sum _{i=1}^n{\mathsf{Z}}_i<n\left({\mu }_n+x\sqrt{{\displaystyle \frac{V_n}{n}}}\right)\right]-\Phi \left(x\right)\right|\le {\displaystyle \frac{B_n}{\sqrt{n}}},$$

(33)

where

$$\begin{array}{c}\hfill {\mu }_n={\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathbb{E}\left[{\mathsf{Z}}_i\right],V_n={\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathrm{Var}\left[{\mathsf{Z}}_i\right],\end{array}$$

(34)

$$\begin{array}{c}\hfill T_n={\displaystyle \frac{1}{n}}\sum _{i=1}^n\mathbb{E}\left[{\left|{\mathsf{Z}}_i-{\mu }_i\right|}^3\right],B_n=C_0{\displaystyle \frac{T_n}{V_n^{3/2}}}.\end{array}$$

(35)

To develop the Gaussian approximation, we consider the following definitions. The variance and third absolute moment of log-likelihood ratio between two distributions P and Q are defined as $V(P\parallel Q)=\mathbb{E}\left[{\left(log{\displaystyle \frac{P}{Q}}-D(P\parallel Q)\right)}^2\right]$ and $T(P\parallel Q)=\mathbb{E}\left[{\left|log{\displaystyle \frac{P}{Q}}-D(P\parallel Q)\right|}^3\right]$, respectively. Then, the following lemma is concerned with properties of $V(P\parallel Q)$ and $T(P\parallel Q)$, which is proved in Appendix D.

Lemma 4.

Fix a W that satisfies Assumption 1 and generate ${\Delta }_{\left|\mathcal{Y}\right|-1}^{*}$. Let $\mathcal{M}$ constructed by (10) with any $b>0$ be the set of packing centers on ${\Delta }_{\left|\mathcal{Y}\right|-1}^{*}$. If packing radius $r\le {\displaystyle \frac{2loge}{9}}$, for any $P\in \mathcal{M}$ and $Q\in {\mathcal{R}}_P$, we have

$$V(P\parallel Q)=r{F}_0,$$

(36)

where

$$\left({\displaystyle \frac{5}{8p_{max}}}-{\displaystyle \frac{2}{9}}\right)loge\le F_0\le {\displaystyle \frac{5loge}{2p_{min}{(1-p_{max})}^2}},$$

(37)

$p_{min}$ and $p_{max}$ are constants greater than 0. Additionally, we have

$$T(P\parallel Q)\le {\displaystyle \frac{36\sqrt{2}{(loge)}^{3/2}}{p_{min}^2{(1-p_{max})}^3}}{r}_0^{3/2}.$$

(38)

5.2. Main Result: Gaussian Approximation

The main result in this section is the following. Please refer to Appendix E for the proof.

Theorem 4.

Fix W is a strictly positive and full-rank square matrix for noisy permutation channels. Then, there exists a number $N_0\ge 1$, such that

$$log{M}^{*}(n,ϵ)=\ell log\left({\displaystyle \frac{\sqrt{n}}{-{\Phi }^{-1}(ϵ/G)}}\right)+logV\left(W\right)+\theta$$

(39)

is achievable for all $n\ge N_0$, where $\ell =\mathsf{rank}\left(W\right)-1$, $G=2\left({\displaystyle \genfrac{}{}{0pt}{}{\ell +1}{2}}\right)$, $V\left(W\right)$ is the channel volume ratio, and θ is a constant.

This achievable code size (39) is different from the Gaussian approximation of the traditional channel (e.g., see [7]) since our bound is obtained by divergence packing numbers $N^{*}(r,|\mathcal{Y}\left|\right)$. The packing radius is a key ingredient affecting the lower bound of $N^{*}(r,|\mathcal{Y}\left|\right)$ and affects the error probability.

5.3. Approximation of BSC and BEC Permutation Channels

We apply Theorem 4 to obtain the following approximation.

Corollary 1.

For BSC permutation channels with crossover probability δ, there exists a number $N_0\ge 1$, such that

$$log{M}^{*}(n,ϵ)=log\left({\displaystyle \frac{(1-2\delta )\sqrt{n}}{-{\Phi }^{-1}\left(\frac{ϵ}{2}\right)}}\right)+\theta$$

(40)

is achievable for all $n\ge N_0$, where θ is a constant.

Proof. 

For BSC, we have ${\Delta }_1^{*}=\left\{(p,1-p)\right|\delta \le p\le 1-\delta \}$. By using Lagrange’s formula [23], we have $V\left({\mathsf{BSC}}_{\delta }\right)=1-2\delta$. Substituting this into Theorem 4 yields the result. □

Remark 4.

We remark that the Gaussian approximation shows some properties of the code size with a given blocklength n and probability of error ϵ. In BSC permutation channels, while the channel capacity is only related to the rank of the channel matrix, the rate at which the achievable code size approaches the capacity is affected by crossover probability δ.

The approximation of BSC permutation channels can also be derived from Proposition 1. To use the message set constructed by (8), we need the following lemma, which is proved in Appendix D:

Lemma 5.

Fix a W that satisfies Assumption 1 and generate ${\Delta }_1^{*}$. Let $\mathcal{M}$ constructed by (8) with any $b>0$ be the set of packing centers on ${\Delta }_1^{*}$. Then, there exists a packing radius $r_1$, such that for all $r\le r_1$, we have

$$F_{0}r\le V(P\parallel Q)\le F_{1}r$$

(41)

and

$$T(P\parallel Q)\le F_2{r}^{3/2},$$

(42)

where $F_0$, $F_1$ and $F_2$ are positive and finite.

Then, we have the following result.

Proposition 3.

For BSC permutation channels with crossover probability δ, there exists a number $N_0\ge 1$, such that

$$log{M}^{*}(n,ϵ)=log\left(⌈{\displaystyle \frac{(1-2\delta )\sqrt{n}}{-{\Phi }^{-1}\left(\frac{ϵ}{2}\right)}}⌉\right)+\theta$$

(43)

is achievable for all $n\ge N_0$, where θ is a constant.

Proof. 

Instead of using (10), we use (8) with ${\Delta }_1^{*}=\left\{(p,1-p)\right|\delta \le p\le 1-\delta \}$. Repeat the argument of the proof of Theorem 4 replacing Lemma 4 with Lemma 5. Note that for $\mathcal{M}$ constructed by (8) with any $b>0$, we have

$$b={\displaystyle \frac{1}{1-2\delta }}\sqrt{{\displaystyle \frac{r}{2loge}}}.$$

(44)

We use Proposition 1 to continue as follows:

$$\begin{array}{cc}\hfill log{M}^{*}(n,ϵ)& \ge log\left(⌊{\displaystyle \frac{(1-2\delta )\sqrt{n}}{-{\Phi }^{-1}\left(\frac{ϵ}{2}\right)}}⌋+1\right)+log{F}_0\hfill \\ \hfill & \ge log\left(⌈{\displaystyle \frac{(1-2\delta )\sqrt{n}}{-{\Phi }^{-1}\left(\frac{ϵ}{2}\right)}}⌉\right)+log{F}_0,\hfill \end{array}$$

(45)

where $F_0\ge 0$ is a constant. This completes the proof. □

Next, for BEC permutation channels, we have the following approximation.

Proposition 4.

For BEC permutation channels with erasure probability η, there exists a number $N_0\ge 1$, such that

$$log{M}^{*}(n,ϵ)=log\left({\displaystyle \frac{(1-\eta )\sqrt{n}}{-{\Phi }^{-1}\left(\frac{ϵ}{2}\right)}}\right)+\theta$$

(46)

and

$$log{M}^{*}(n,ϵ)=log\left(⌈{\displaystyle \frac{(1-\eta )\sqrt{n}}{-{\Phi }^{-1}\left(\frac{ϵ}{2}\right)}}⌉\right)+\theta$$

(47)

are achievable for all $n\ge N_0$, where θ is a constant.

Proof. 

Through Theorem 2, repeat the argument of the proof of Corollary 1 and Theorem 3, replacing $\delta$ with $\eta /2$. □

6. Numerical Results

In this section, we perform numerical evaluations to illustrate our results. We first validate the precision of our Gaussian approximation across a wide range of parameters. Secondly, we present the performance of bounds of a binary DNA storage system and compare them with existing bounds.

6.1. Precision of the Gaussian Approximation

Here, we give the numerical results. According to Proposition 2, the BEC permutation channel with erasure probability $2\delta$ can be equivalent to the BSC permutation channel with crossover probability $\delta$. Thus, we focus on the numerical results of BSC permutation channels. We use Theorem 3 to compute the non-asymptotic achievability bound. We start searching from $M=2$ until the right side of (25) is less than the error probability $ϵ$. For Gaussian approximation, we use (40) and (43) but omit the remainder term $\theta$. As Figure 2, Figure 3, Figure 4 and Figure 5 show, although the remainder term of the Gaussian approximation is a constant, it is still quite close to the non-asymptotic achievability bound. In fact, for all $n\ge 20$, the difference between (43) and Theorem 3 is within 1 bit in $log{M}^{*}(n,ϵ)$.

Figure 2. Rate–blocklength tradeoff for the BSC with crossover probability $\delta =0.11$ and average block error rate $ϵ={10}^{-3}$.

Figure 3. Rate–blocklength tradeoff for the BSC with crossover probability $\delta =0.11$ and average block error rate $ϵ={10}^{-6}$.

Figure 4. Rate–blocklength tradeoff for the BSC with crossover probability $\delta =0.22$ and average block error rate $ϵ={10}^{-3}$.

Figure 5. Rate–blocklength tradeoff for the BSC with crossover probability $\delta =0.22$ and average block error rate $ϵ={10}^{-6}$.

6.2. Comparison with Existing Bound

Additionally, in the context of DNA storage systems, we consider codewords composed of nucleotides $\{A,C,G,T\}$. For simplicity, A and C are regarded as the symbol 0, and G and T are regarded as the symbol 1 in code constructions, giving a binary alphabet $\{0,1\}$. The synthesis errors and random permutation of DNA molecules are modeled as the BSC Permutation Channel with crossover probability $\delta =0.25$. To reduce the computation complexity, we use approximation (43). Furthermore, we present numerical results for the existing lower bound, namely, Makur’s achievability bound ([1] Equation (36)) for BSC permutation channels. The result shows that our new achievability bound is uniformly better than Makur’s bound. In the setup of Figure 6, our bound quickly approaches half of the capacity ($n\approx 1000$). As the blocklength increases, Makur’s bound reaches $20\%$ of the channel capacity at about $n\approx 1.4\times {10}^5$, shown in Figure 6. This is because we show the overlapping relationship of error events, which reduces the number of error events when applying the union bound.

Figure 6. Rate–blocklength tradeoff for the BSC with crossover probability $\delta =0.25$ and average block error rate $ϵ={10}^{-3}$: example of a DNA storage system.

7. Conclusions and Discussion

In summary, we established a new achievability bound for noisy permutation channels with a strictly positive and full-rank square matrix. The key element is that our analysis indicates that the size of error events in the union is independent of the message set. This allows us to derive a refined asymptotic analysis of the achievable rate. Numerical simulations show that our new achievability bound is stronger than Makur’s achievability bound in [1]. Additionally, our approximation is quite accurate, even though the remainder term is a constant. Finally, the primary future work will generalize the DMC matrix in noisy permutation channels to non-full-rank and non-strictly positive matrices. Other future work may improve asymptotic expansion (e.g., improving the remainder term to $o\left(1\right)$).

Appendix A.1. Proof of Proposition 1

We note that ${\Gamma }_{b,2}^{*}$ constructed by (8) is a subset of ${\Delta }_1^{*}$. Fix $b={\displaystyle \frac{1}{\xi }}\sqrt{{\displaystyle \frac{r}{2loge}}}$. Then, for $P,Q\in {\Gamma }_{b,2}$ we have

$$\underset{P\ne Q}{min}(2loge){\mathrm{TV}}^2(P,Q)\ge r.$$

(A1)

Use Pinsker’s inequality ([8] Theorem 7.10) to obtain $\underset{P\ne Q}{min}D(P\parallel Q)\ge r$. Then, we conclude this proof by realizing that $|{\Gamma }_{b,2}^{*}|=\lfloor 1/b\rfloor +1$.

Appendix A.2. Proof of Theorem 1

To prove Theorem 1, we consider the set

$$\begin{array}{c}\hfill {\Gamma }_{b,\left|\mathcal{Y}\right|}=\left\{P\in {\Delta }_{\left|\mathcal{Y}\right|-1}|\begin{array}{c}\hfill P=\left({\displaystyle \frac{a_1}{\lfloor 1/b\rfloor }},\dots ,{\displaystyle \frac{a_{\left|\mathcal{Y}\right|}}{\lfloor 1/b\rfloor }}\right)\mathrm{where}{a}_1,\dots ,a_{\left|\mathcal{Y}\right|}\in {\mathbb{Z}}_{\ge 0}\end{array}\right\}.\end{array}$$

(A2)

We have the following lemma:

Lemma A1.

Let ${\Gamma }_{b,\left|\mathcal{Y}\right|}$ be constructed by (A2) with any $b>0$. For $P,Q\in {\Gamma }_{b,\left|\mathcal{Y}\right|}$, we have

$$\underset{P\ne Q}{min}\mathrm{TV}(P,Q)\ge b.$$

(A3)

Proof. 

For every $P\in {\Gamma }_{b,\left|\mathcal{Y}\right|}$, we can find a $Q\in {\Gamma }_{b,\left|\mathcal{Y}\right|}$ such that for $m,n\in \mathcal{Y}$ and $m\ne n$, we have $q_m\ne p_m$, $q_n\ne p_n$. For $i\in \mathcal{Y}\setminus \{m,n\}$, we have $q_i=p_i$. Then, we obtain that $\underset{P\ne Q}{min}\mathrm{TV}(P,Q)=\underset{P\ne Q}{min}{\displaystyle \frac{1}{2}}{\sum }_{i\in \mathcal{Y}}|p_i-q_i|={\displaystyle \frac{1}{\lfloor 1/b\rfloor }}\ge b$. □

Fix a radius $b=\sqrt{{\displaystyle \frac{r}{2loge}}}>0$ and fix a ${\Gamma }_{b,\left|\mathcal{Y}\right|}^{*}$ constructed by (10). Note that we have ${\Gamma }_{b,\left|\mathcal{Y}\right|}^{*}\subset {\Gamma }_{b,\left|\mathcal{Y}\right|}$. For any $P,Q\in {\Gamma }_{b,\left|\mathcal{Y}\right|}^{*}$, we have

$$\underset{P\ne Q}{min}D(P\parallel Q)\stackrel{\left(a\right)}{\ge }\underset{P\ne Q}{min}(2loge){\mathrm{TV}}^2(P,Q)\stackrel{\left(b\right)}{\ge }r,$$

(A4)

where $\left(a\right)$ follows from Pinsker’s inequality and $\left(b\right)$ follows from Lemma A1. On the other hand, note that the right inequality of (A4) shows that

$$min\sum _{y\in \mathcal{Y}}|P\left(y\right)-Q\left(y\right)|\ge \sqrt{{\displaystyle \frac{2}{loge}}}\sqrt{r}.$$

(A5)

Hence, a total variation packing constructed by ${\Gamma }_{b,\left|\mathcal{Y}\right|}^{*}$ can be regarded as an ${\ell }_1$-norm packing of ${\Delta }_{\left|\mathcal{Y}\right|-1}^{*}$ with radius $\sqrt{{\displaystyle \frac{2r}{loge}}}$. Let $B_{{\ell }_1}^{\left|\mathcal{Y}\right|-1}$ be the ${\ell }_1$-norm unit ball. The volume bound ([8] Theorem 27.3) of ${\ell }_1$-norm packing provides the lower bound of $|{\Gamma }_{b,\left|\mathcal{Y}\right|}^{*}|$ as

$$\left|{\Gamma }_{b,\left|\mathcal{Y}\right|}^{*}\right|\ge {\left({\displaystyle \frac{loge}{2r}}\right)}^{{\displaystyle \frac{\left|\mathcal{Y}\right|-1}{2}}}{\displaystyle \frac{\mathrm{vol}(y,{\Delta }_{\left|\mathcal{Y}\right|-1}^{*})}{\mathrm{vol}\left(B_{{\ell }_1}^{\left|\mathcal{Y}\right|-1}\right)}}.$$

(A6)

Here, $\mathrm{vol}\left(B_{{\ell }_1}^{\left|\mathcal{Y}\right|-1}\right)$ is the volume of the ${\ell }_1$-norm unit ball; $\mathrm{vol}(y,{\Delta }_{\left|\mathcal{Y}\right|-1}^{*})$ is the volume of the projection of ${\Delta }_{\left|\mathcal{Y}\right|-1}^{*}$ from ${\mathbb{R}}^{\left|\mathcal{Y}\right|}$ to the space ${\mathbb{R}}^{\left|\mathcal{Y}\right|-1}$, consistent with the argument used in the proof of ([24] Proposition 2).

Note that the maximum volume ratio is $V\left(W\right)$ by Definition 3. We continue the bounding as follows:

$$\begin{array}{cc}\hfill |{\Gamma }_{b,\left|\mathcal{Y}\right|}^{*}|& \ge {\left({\displaystyle \frac{loge}{2r}}\right)}^{{\displaystyle \frac{\left|\mathcal{Y}\right|-1}{2}}}\times {\displaystyle \frac{V\left(W\right)\mathrm{vol}(y,{\Delta }_{\left|\mathcal{Y}\right|-1})}{\mathrm{vol}\left(B_{{\ell }_1}^{\left|\mathcal{Y}\right|-1}\right)}},\hfill \end{array}$$

(A7)

$$\begin{array}{cc}\hfill & =V\left(W\right){\left({\displaystyle \frac{loge}{8r}}\right)}^{{\displaystyle \frac{\left|\mathcal{Y}\right|-1}{2}}},\hfill \end{array}$$

(A8)

where

• (A7) holds since this projection can remove any y-th dimension, where $y\in \mathcal{Y}$. Consequently, the lower bound is given by taking the maximum of the volume ratio over $y\in \mathcal{Y}$;
• (A8) follows by using Lagrange’s formula [23] and the volume formula to obtain $\mathrm{vol}(y,{\Delta }_{\left|\mathcal{Y}\right|-1}^{*})={\displaystyle \frac{V\left(W\right)}{\left(\right|\mathcal{Y}|-1)!}}$ and $\mathrm{vol}\left(B_{{\ell }_1}^{\left|\mathcal{Y}\right|-1}\right)={\displaystyle \frac{2^{\left|\mathcal{Y}\right|-1}}{\left(\right|\mathcal{Y}|-1)!}}$, respectively.

Finally, we conclude the proof by realizing that the left inequality of (A4) implies that ${\Gamma }_{b,\left|\mathcal{Y}\right|}^{*}$ gives the set of divergence packing centers. This completes the proof of Theorem 1.

Appendix D.1. Proof of Lemma 4

Since the DMC matrix is strictly positive, each term of marginal distribution P is uniformly bounded away from zero, i.e., there exists a $p_{min}\in (0,1)$ such that $P\left(y\right)\ge p_{min}$ for all $y\in \mathcal{Y}$. Similarly, there exists a $p_{max}\in (0,1)$ such that $P\left(y\right)\le p_{max}$ for all $y\in \mathcal{Y}$. Clearly, we have $p_{min}={min}_{y\in \mathcal{Y}}\left\{P\left(y\right)\right|P\in \mathcal{M}\}$ and $p_{max}={max}_{y\in \mathcal{Y}}\left\{P\left(y\right)\right|P\in {\Delta }_{\left|\mathcal{Y}\right|-1}^{*}\}$. Without loss of generality, we assume $\left|\mathcal{M}\right|\ge 2$. That is, we have $r_0\le {\displaystyle \frac{2loge}{9}}$.

We know that

$$\begin{array}{cc}\hfill P\left(y\right)& ={\displaystyle \frac{a}{1/r-\delta }}\hfill \\ \hfill & ={\displaystyle \frac{a\sqrt{\frac{r_0}{2loge}}}{1-\delta \sqrt{\frac{r_0}{2loge}}}},\hfill \end{array}$$

(A28)

where $\delta \in [0,1)$ and $a\in {\mathbb{Z}}_{\ge 1}$.

We consider each term $P\left(y\right)$ in P separately. If $P\left(y\right)=Q\left(y\right)$, we obviously get zero. If $P\left(y\right)>Q\left(y\right)$, we have $Q\left(y\right)={\displaystyle \frac{(a-1)\sqrt{\frac{r_0}{2loge}}}{1-\delta \sqrt{\frac{r_0}{2loge}}}}$. Here, since the distributions in $\mathcal{M}$ have no zero term, we have $a\in {\mathbb{Z}}_{\ge 2}$. Since $a=P\left(y\right)\left(1-\delta \sqrt{{\displaystyle \frac{r_0}{2loge}}}\right)\sqrt{{\displaystyle \frac{2loge}{r_0}}}$, we have

$$\begin{array}{c}\hfill {\displaystyle \frac{\sqrt{r_{0}loge}}{P\left(y\right)(\sqrt{2}-\delta \sqrt{r_0/loge})}}\le log{\displaystyle \frac{P\left(y\right)}{Q\left(y\right)}}\le {\displaystyle \frac{\sqrt{r_{0}loge}}{P\left(y\right)(\sqrt{2}-\delta \sqrt{r_0/loge})-\sqrt{r_0/loge}}}.\end{array}$$

(A29)

Consequently, we have

$$P\left(y\right){\left(log{\displaystyle \frac{P\left(y\right)}{Q\left(y\right)}}\right)}^2\ge {\displaystyle \frac{r_{0}loge}{2P\left(y\right)}}$$

(A30)

and

$$\begin{array}{c}\hfill P\left(y\right){\left(log{\displaystyle \frac{P\left(y\right)}{Q\left(y\right)}}\right)}^2\le {\displaystyle \frac{r_{0}loge}{2{(a-1)}^{2}P\left(y\right)/a^2}}\le {\displaystyle \frac{2r_{0}loge}{P\left(y\right)}}{\left({\displaystyle \frac{1}{1-\delta \sqrt{\frac{r_0}{2loge}}}}\right)}^2.\end{array}$$

(A31)

If $P\left(y\right)<Q\left(y\right)$, consider $Q\left(y\right)={\displaystyle \frac{(a+1)\sqrt{\frac{r_0}{2loge}}}{1-\delta \sqrt{\frac{r_0}{2loge}}}}$, where $a\in {\mathbb{Z}}_{\ge 1}$. Applying the same argument to obtain

$$\begin{array}{c}\hfill {\displaystyle \frac{\sqrt{r_{0}loge}}{P\left(y\right)(\sqrt{2}-\delta \sqrt{r_0/loge})+\sqrt{r_0/loge}}}\le |log{\displaystyle \frac{P\left(y\right)}{Q\left(y\right)}}|\le {\displaystyle \frac{\sqrt{r_{0}loge}}{P\left(y\right)(\sqrt{2}-\delta \sqrt{r_0/loge})}}.\end{array}$$

(A32)

Consequently, we have

$$\begin{array}{c}\hfill P\left(y\right){\left(log{\displaystyle \frac{P\left(y\right)}{Q\left(y\right)}}\right)}^2\ge {\displaystyle \frac{r_{0}loge}{2{(a+1)}^{2}P\left(y\right)/a^2}}\ge {\displaystyle \frac{r_{0}loge}{8P\left(y\right)}}\end{array}$$

(A33)

and

$$\begin{array}{cc}\hfill P\left(y\right){\left(log{\displaystyle \frac{P\left(y\right)}{Q\left(y\right)}}\right)}^2& \le {\displaystyle \frac{r_{0}loge}{2P\left(y\right)}}{\left({\displaystyle \frac{1}{1-\delta \sqrt{\frac{r_0}{2loge}}}}\right)}^2.\hfill \end{array}$$

(A34)

Then, we see that

$$V(P\parallel Q)\ge -D{(P\parallel Q)}^2+\sum _{y}P\left(y\right){\left(log{\displaystyle \frac{P\left(y\right)}{Q\left(y\right)}}\right)}^2$$

(A35)

$$\ge r_{0}loge\left({\displaystyle \frac{5}{8p_{max}}}-{\displaystyle \frac{r_0}{loge}}\right),$$

(A36)

where

• (A35) simply expand the variance;
• (A36) holds since the definition of ${\mathcal{R}}_P$ ensures that for all $y\in \mathcal{Y}$, we only have two terms that are not zero and can be bounded by (A30) and (A33), respectively.

Finally, we complete the proof of the lower bound by noting that $r_0\le {\displaystyle \frac{2loge}{9}}$.

Note that

$${\left({\displaystyle \frac{1}{1-\delta \sqrt{\frac{r_0}{2loge}}}}\right)}^2\le {\displaystyle \frac{1}{{(1-p_{max})}^2}}.$$

(A37)

Using (A31) and (A34), we obtain the upper bound

$$V(P\parallel Q)\le {\displaystyle \frac{5loge}{2p_{min}{(1-p_{max})}^2}}{r}_0,$$

(A38)

which yields (36).

We now turn to bound $T(P\parallel Q)$. For $P\left(y\right)>Q\left(y\right)$, we have

$$P\left(y\right)|log{\displaystyle \frac{P\left(y\right)}{Q\left(y\right)}}{|}^3\le {\displaystyle \frac{4}{\sqrt{2}}}{\displaystyle \frac{{(r_{0}loge)}^{3/2}}{P^2\left(y\right)}}{\left({\displaystyle \frac{1}{1-\delta \sqrt{\frac{r_0}{2loge}}}}\right)}^3.$$

(A39)

For $P\left(y\right)<Q\left(y\right)$, we have

$$P\left(y\right)|log{\displaystyle \frac{P\left(y\right)}{Q\left(y\right)}}{|}^3\le {\displaystyle \frac{1}{2\sqrt{2}}}{\displaystyle \frac{{(r_{0}loge)}^{3/2}}{P^2\left(y\right)}}{\left({\displaystyle \frac{1}{1-\delta \sqrt{\frac{r_0}{2loge}}}}\right)}^3.$$

(A40)

Note that $D(P\parallel Q)$ can be bounded by the following:

$$D(P\parallel Q)\le {\displaystyle \frac{3\sqrt{2loge}}{2}}\sqrt{r_0}\left({\displaystyle \frac{1}{1-\delta \sqrt{\frac{r_0}{2loge}}}}\right).$$

(A41)

Using the inequality ${|a-b|}^3\le {4\left(\right|a|}^3+{\left|b\right|}^3)$, we obtain

$$T(P\parallel Q)\le {\displaystyle \frac{36\sqrt{2}{(loge)}^{3/2}}{p_{min}^2{(1-p_{max})}^3}}{r}_0^{3/2},$$

(A42)

which establishes (38).

Appendix D.2. Proof of Lemma 5

Then, we prove Lemma 5 for $\mathcal{M}$ constructed by (8). We first note that

$$\begin{array}{cc}\hfill P\left(y\right)& ={\displaystyle \frac{a\xi }{1/r-\delta }}\hfill \\ \hfill & ={\displaystyle \frac{\left(a-\frac{{\delta }_1\delta }{\xi }+F_0\right)\sqrt{\frac{r_0}{2loge}}}{1-\frac{\delta }{\xi }\sqrt{\frac{r_0}{2loge}}}},\hfill \end{array}$$

(A43)

where $\delta \in [0,1]$, $F_0={\delta }_1/\sqrt{{\displaystyle \frac{r_0}{2loge}}}$, and $a\in {\mathbb{Z}}_{\ge 1}$. Note that $F_0-{\displaystyle \frac{{\delta }_1\delta }{\xi }}\ge 0$ holds for small $r_0$. Repeat the proof of Lemma 4, replacing a with $a-{\displaystyle \frac{{\delta }_1\delta }{\xi }}+F_0$ and $\delta$ with ${\displaystyle \frac{\delta }{\xi }}$. Then, there exists a packing radius $r_1$, such that for all $r_0\le r_1$, we have

$$F_0{r}_0\le V(P\parallel Q)\le F_1{r}_0$$

(A44)

and

$$T(P\parallel Q)\le F_2{r}_0^{3/2},$$

(A45)

where $F_0$, $F_1$ and $F_2$ are positive and finite.