On Decoder Ties for the Binary Symmetric Channel with Arbitrarily Distributed Input

Abstract

The error probability of block codes sent under a non-uniform input distribution over the memoryless binary symmetric channel (BSC) and decoded via the maximum a posteriori (MAP) decoding rule is investigated. It is proved that the ratio of the probability of MAP decoder ties to the probability of error grows most linearly in blocklength when no MAP decoding ties occur, thus showing that decoder ties do not affect the code’s error exponent. This result generalizes a similar recent result shown for the case of block codes transmitted over the BSC under a uniform input distribution.

1. Introduction

Consider the classical channel coding context, where we send a block code through the memoryless binary symmetric channel (BSC) with crossover probability $0<p<1/2$. Given a sequence of binary codes ${\left\{{\mathcal{C}}_n\right\}}_{n\ge 1}$ with n being the blocklength, we denote the sequence of corresponding minimal probabilities of decoding error under maximum a posteriori (MAP) decoding by ${\left\{a_n\right\}}_{n\ge 1}$. The following result was recently shown in [1] when the channel input selects codewords from ${\mathcal{C}}_n$ according to a uniform distribution.

Theorem 1

([1]). For any sequence of codes ${\left\{{\mathcal{C}}_n\right\}}_{n\ge 1}$ of blocklength n and size $|{\mathcal{C}}_n|=M$ with ${\mathcal{C}}_n\subseteq {\{0,1\}}^n$, sent over the BSC with crossover probability $0<p<1/2$ under a uniform channel input distribution over ${\mathcal{C}}_n$, its minimum probability of decoding error $a_n$ satisfies

$$b_n\le a_n\le \left(1+\frac{(1-p)}{p}n\right)b_n,$$

(1)

where

$$\begin{array}{cc}\hfill b_n& =P_{X^n,Y^n}\left\{(x^n,y^n)\in {\mathcal{X}}^n\times {\mathcal{Y}}^n:P_{X^n|Y^n}\left(x^n\right|y^n)<\underset{u^n\in {\mathcal{C}}_n\backslash \left\{x^n\right\}}{max}{P}_{X^n|Y^n}\left(u^n\right|y^n)\right\},\hfill \end{array}$$

(2)

where $P_{X^n,Y^n}$ is the joint input–output distribution that $X^n=(X_1,X_2,\dots ,X_n)\in {\mathcal{X}}^n\triangleq {\{0,1\}}^n$ is sent over the BSC (via n uses) and $Y^n=(Y_1,Y_2,\dots ,Y_n)\in {\mathcal{Y}}^n\triangleq {\{0,1\}}^n$ is received.

Noting that $b_n$ in (2) is the probability that a decoding error occurs without inducing decoder ties (which occur when two or more codewords in ${\mathcal{C}}_n$ are identified by the decoder as the estimated transmitted codeword; i.e., when more than one codeword in ${\mathcal{C}}_n$ maximize $P_{X^n|Y^n}(·|y^n)$ for a given received word $y^n$), the above result in (1) directly implies that decoder ties do not affect the error exponent of $a_n$. The error exponent or reliability function of a block coding communication system represents the largest rate of exponential decay of the system’s probability of decoding error as the coding blocklength grows to infinity (e.g., see [2,3,4,5,6,7,8,9,10,11,12,13,14]).

It is known that uniformly distributed data achieves the largest entropy rate and leaves no room for data compression. Thus, ideally compressed data should exhibit uniform distribution for all blocklengths n. However, this setting is often impractical due to the sub-optimality of the implemented data compression schemes. Instead, we generally have non-uniformly distributed data after compression in the form of residual redundancy such as in speech or image coding (e.g., [15,16]). Furthermore, one may have a compressed source that can be divided into several groups, within each of which the symbols are equally probable. Decoder ties can thus occur with respect to two (or more) codewords corresponding to symbols within the same group.

In this paper, we consider a non-uniform prior distribution over ${\mathcal{C}}_n$ and prove that decoder ties, under optimal MAP decoding, still have linear and hence sub-exponential impact on the error probability $a_n$, thus extending Theorem 1 established for the case of a uniform prior distribution over ${\mathcal{C}}_n$. Since our problem falls within the general framework of joint source-channel coding for point-to-point communication systems, we refer the reader to [14,15,16,17,18,19,20,21] (Section 4.6) and the references therein for theoretical studies on this subject as well as practical designs that outperform separate source and channel coding under complexity or delay constraints.

The proof technique used in [1] to show (1) above is based on the observation that there are two types of decoding errors. One is that the received tuple at the channel output induces no decoder ties but the corresponding decoder decision is wrong. The other is that the received tuple at the channel output causes a decoder tie, but the decoder picks the wrong codeword. As a result, the MAP error probability $a_n$ can be upper bounded by the sum of two terms, $b_n$ and ${\delta }_n$, where $b_n$ is the probability of the first type of decoding errors as given in (2), and ${\delta }_n$ is the probability of decoder ties regardless of whether the tie breaker misses the correct codeword or not. Under the assumption that the channel input is uniformly distributed over block code ${\mathcal{C}}_n$ for each blocklength n and an arbitrary sequence of codes ${\left\{{\mathcal{C}}_n\right\}}_{n\ge 1}$, it was shown in [1] that flipping a properly selected bit component of the channel output that causes a decoder tie can produce a unique channel output that leads to the first type of decoding errors. An analysis of this bit-flipping manipulation shows that the ratio ${\delta }_n/b_n$ grows at most linearly in n and hence yields the upper bound in (1). However, this flipping technique no longer works when non-uniform channel inputs are considered. To tackle this problem, we judiciously separate the channel output tuples that induce decoder ties into two groups, one group consisting of output tuples that do not fulfill the above flipping manipulation property and the other group composed of the remaining output tuples (i.e., the complement group). We then show that the probability of the former group is upper bounded by that of the latter group, and therefore ${\delta }_n/b_n$ remains growing at most linearly in blocklength n under arbitrary channel input statistics. Note that the group that fails the flipping property is an empty set when channel input is uniformly distributed over ${\mathcal{C}}_n$, thereby making the result of Theorem 1 a special case of the extended result in this paper. The rest of the paper is organized as follows. Section 2 presents the main result and highlights the key steps of the proof to facilitate its understanding. The proof is then provided in full detail, along with illustrative examples, in Section 3 and Appendix A and Appendix B. Finally, conclusions and future directions are given in Section 4.

Throughout the paper, we denote $\left[M\right]\triangleq \{1,2,\dots ,M\}$ for positive integer M and set $d(x^n,y^n|\mathcal{S})$ to be the Hamming distance between n-tuples $x^n=(x_1,x_2,\dots ,x_n)$ and $y^n=(y_1,y_2,\dots ,y_n)$ with the indices of the tuples restricted to $\mathcal{S}\subseteq \left[n\right]$. By convention, we set $d(x^n,y^n|\mathcal{S})=0$ when $\mathcal{S}=\varnothing$ and use $d(x^n,y^n)$ to represent $d(x^n,y^n|\left[n\right])$.

2. Main Result

Consider a binary code ${\mathcal{C}}_n\subseteq {\{0,1\}}^n$ with fixed blocklength n and size M to be used over the memoryless BSC with crossover probability $0<p<\frac{1}{2}$. Denote the prior probability on ${\mathcal{C}}_n$ by $P_{X^n}$ and hence $P_{X^n}\left({\mathcal{C}}_n\right)=1$. Without loss of generality, we assume that all codewords in ${\mathcal{C}}_n$ occur with positive probability, i.e., $P_{X^n}\left(x^n\right)>0$ for all $x^n\in {\mathcal{C}}_n$; hence, ${\mathcal{C}}_n$ is the support of $P_{X^n}$.

It is known the minimal probability of decoding error is achieved by the MAP decoder, which upon the reception of the channel output $y^n\in {\{0,1\}}^n$ estimates the codeword $x^n\in {\mathcal{C}}_n$ according to

$$\begin{array}{c}\hfill e\left(y^n\right)=arg\underset{u^n\in {\mathcal{C}}_n}{max}{P}_{X^n|Y^n}\left(u^n\right|y^n),\end{array}$$

(3)

where $P_{X^n|Y^n}$ is the posterior conditional distribution of $X^n$ given $Y^n$. We can see from (3) that if more than one $u^n\in {\mathcal{C}}_n$ achieves the maximum value of $P_{X^n|Y^n}\left(u^n\right|y^n)$ for a given $y^n$, a decoder tie occurs, in which case the set of these $u^n$, denoted conveniently as $\left\{e\left(y^n\right)\right\}$, contains more than one element. As a result, an erroneous MAP decision is made if one of the two situations occurs: $i)$ the transmitted codeword does not belong to $\left\{e\left(y^n\right)\right\}$; $ii)$ the transmitted codeword belong to $\left\{e\left(y^n\right)\right\}$ and $\left|\right\{e\left(y^n\right)\left\}\right|>1$, but the tie breaker picks the wrong one from $\left\{e\left(y^n\right)\right\}$. By conveniently denoting

$${\mathcal{C}}_n=\{c_1,c_2,\cdots ,c_M\},$$

(4)

the probability of the first situation acts as a lower bound $b_n$ for $a_n$ (i.e., $b_n\le a_n$), where $b_n$ is given in (2) and can be written as

$$\begin{array}{ccc}\hfill b_n& =& \sum _{i=1}^M{P}_{X^n}\left(c_i\right)P_{Y^n|X^n}\left(\left\{y^n\in {\{0,1\}}^n:P_{X^n|Y^n}\left(c_i\right|y^n)<\underset{r\in \left[M\right]\backslash \left\{i\right\}}{max}{P}_{X^n|Y^n}\left(c_r\right|y^n)\right\}\right).\hfill \end{array}$$

(5)

It is shown in [22] that $b_n$ exactly equals the generalized Poor–Verdú (lower) bound [23,24] as its tilting parameter approaches infinity. The probability of the second situation is bounded above by the probability that the transmitted codeword belongs to $\left\{e\left(y^n\right)\right\}$ and $\left|\right\{e\left(y^n\right)\left\}\right|>1$, disregarding whether the tie breaker picks the wrong codeword or not, and this upper bound can be expressed as

$$\begin{array}{ccc}\hfill {\delta }_n& \triangleq & \sum _{i=1}^M{P}_{X^n}\left(c_i\right)P_{Y^n|X^n}\left(\left\{y^n\in {\{0,1\}}^n:P_{X^n|Y^n}\left(c_i\right|y^n)=\underset{r\in \left[M\right]\backslash \left\{i\right\}}{max}{P}_{X^n|Y^n}\left(c_r\right|y^n)\right\}\right).\hfill \end{array}$$

(6)

We thus have

$$b_n\le a_n\le b_n+{\delta }_n.$$

(7)

By proving the inequality

$${\delta }_n\le 2qn{b}_n,$$

(8)

where

$$q\triangleq \frac{1-p}{p}>1,$$

(9)

we have our main result as follows.

Theorem 2.

For any sequence of binary codes ${\left\{{\mathcal{C}}_n\right\}}_{n\ge 1}$ and prior probabilities ${\left\{P_{X^n}\right\}}_{n\ge 1}$ used over the BSC, we have

$$b_n\le a_n\le \left(1+2qn\right)b_n.$$

(10)

Remark 1.

Theorem 2 implies that the relative deviation of $a_n$ from $b_n$ is at most linear in the blocklength n and the impact of decoder ties in (6) to $a_n$ is only sub-exponential. Consequently, $a_n$ and $b_n$ must have the same error exponent. Note also that the upper bound in (10) differs from the result in Theorem 1 by an additional multiplicative factor of 2 in the $qn$ term. As explained in the introduction section, this is a consequence of the fact that the probability of the group of channel output tuples that cause decoder ties but fail the flipping manipulation property is upper bounded by that of the remaining tie-inducing channel outputs. The full technical details are provided in Section 3.2. Finally, we emphasize that Theorem 2 holds for arbitrary binary codes, including “bad” codes for which high probability codewords have small Hamming distance between them. Hence, tightening the upper bound in (10) by restricting the analysis for “sufficiently good" codes, in the sense that their most likely codewords sit “sufficiently” far apart in ${\{0,1\}}^n$, is an interesting future direction.

List of Main Symbols: Before providing an overview of the main steps of the proof of Theorem 2 (which is presented in full detail in the next section), we describe in

Table 1. Summary of the main symbols used in this paper.

Symbol	Description	Defined in
$\left[M\right]$	A shorthand for $\{1,2,\dots ,M\}$
${\mathcal{C}}_n$	The code $\left\{c_1,c_2,\dots ,c_n\right\}$ with $c_1$ being the all-zero codeword
$d(u^n,v^n|\mathcal{S})$	The Hamming distance between the portions of $u^n$ and $v^n$ with indices in $\mathcal{S}$
All terms below are functions of ${\mathcal{C}}_n$ (this dependence is not explicitly shown to simplify notation)
${\mathcal{T}}_i$	The set of channel outputs $y^n$ inducing a decoder tie when $c_i$ is sent	(12)
${\mathcal{N}}_i$	The set of channel outputs $y^n$ leading to a tie-free decoder decision error when $c_i$ is sent	(15)
${\mathcal{I}}_i\left(y^n\right)$	The set $\{m\in \left[M\right]\backslash \left\{i\right\}:y^n\in {\mathcal{T}}_m\}$ for $y^n\in {\mathcal{T}}_i$	(21)
${\mathcal{S}}_{i,j}$	The set of indices for which the components of $c_i$ and $c_j$ differ
${\ell }_{i,j}$	The size of ${\mathcal{S}}_{i,j}$, i.e., $|{\mathcal{S}}_{i,j}|$
${\mathcal{T}}_{j|i}$	The subset of ${\mathcal{T}}_i$ consisting of channel outputs $y^n$ such that j is the minimal	(22a)
	number r in ${\mathcal{I}}_i\left(y^n\right)$ satisfying $d(c_i,y^n|{\mathcal{S}}_{i,r})<{\ell }_{i,r}$
${\mathcal{N}}_{j|i}$	The subset of ${\mathcal{N}}_i$ consisting of channel outputs $y^n$ that satisfy $P_{X^n,Y^n}(c_i,y^n)·q=$	(22b)
	$P_{X^n,Y^n}(c_j,y^n)·\frac{1}{q}$ and that are not included in ${\mathcal{N}}_{r|i}$ for $r\in [j-1]\subset \left\{i\right\}$
${\mathcal{T}}_{j|i}$	The subset of ${\mathcal{T}}_i\backslash \left({\bigcup }_{h\in \left[M\right]\backslash \left\{i\right\}}{\mathcal{T}}_{h|i}\right)$ consisting of channel outputs $y^n$	(23)
	such that j is the minimal number in ${\mathcal{I}}_i\left(y^n\right)$
${\mathcal{S}}_{1,j}^{\left(m\right)}$	The subset of ${\mathcal{S}}_{1,j}$ defined according to whether each index in ${\mathcal{S}}_{1,j}$ is in each	(43)
	of ${\mathcal{S}}_{1,2}$, …, ${\mathcal{S}}_{1,j-1}$, ${\mathcal{S}}_{1,j+1}$, …, ${\mathcal{S}}_{1,M}$
${\mathcal{S}}_{1,j}^{\left(m\right)}$	The union of ${\mathcal{S}}_{1,j}^{\left(1\right)}$, ${\mathcal{S}}_{1,j}^{\left(2\right)}$, …, ${\mathcal{S}}_{1,j}^{\left(m\right)}$	(48)
${\ell }_{1,j}^{\left(m\right)}$	The size of ${\mathcal{S}}_{1,j}^{\left(m\right)}$, i.e., $|{\mathcal{S}}_{1,j}^{\left(m\right)}|$
${\eta }_k$	The mapping from $k\in \{0,1,\dots ,{\ell }_j-1\}$ to $\left[2^{M-2}\right]$ used for partitioning ${\mathcal{T}}_{j|1}$ into ${\ell }_{1,j}$	(49)
	subsets ${\left\{{\mathcal{T}}_{j|1}\left(k\right)\right\}}_{0\le k<{\ell }_{1,j}}$
${\mathcal{T}}_{j|1}\left(k\right)$	The kth partition of ${\mathcal{T}}_{j|1}$ for $k=0$, 1, …, ${\ell }_{1,j}-1$	(52a)
${\mathcal{N}}_{j|1}\left(k\right)$	The kth subset of ${\mathcal{N}}_{j|1}$ for $k=0$, 1, …, ${\ell }_{1,j}-1$	(52b)
${\mathcal{U}}_{j|1}\left(k\right)$	The set of representative elements in ${\mathcal{T}}_{j|1}\left(k\right)$ for partitioning ${\mathcal{T}}_{j|1}\left(k\right)$
${\mathcal{T}}_{j|1}(u^n;k)$	The subset of ${\mathcal{T}}_{j|1}\left(k\right)$ associated with $u^n\in {\mathcal{U}}_{j|1}\left(k\right)$	(55a)
${\mathcal{N}}_{j|1}(u^n;k)$	The subset of ${\mathcal{N}}_{j|1}\left(k\right)$ associated with $u^n\in {\mathcal{U}}_{j|1}\left(k\right)$	(55b)

the main symbols used in the paper and indicate the equation where they are first introduced. We emphasize that sets ${\mathcal{T}}_{j|i}$, ${\mathcal{N}}_{j|i}$ and ${\mathcal{S}}_{1,j}^{\left(m\right)}$ are defined differently from their counterparts in [1] that use the same notation.

We also visually illustrate in Figure 1 some of the main sets defined in Table 1 under the setting of Example 1, which is presented in Section 3 below for a non-uniformly distributed binary code with $M=4$ codewords and blocklength $n=4$ given by ${\mathcal{C}}_4=\{c_1,c_2,c_3,c_4\}=\{0000,0101,0110,0111\}$. More specifically, we only show the non-empty component subsets in ${\mathcal{Y}}^n={\{0,1\}}^4$ corresponding to codewords $c_1$ and $c_2$; refer to

Table A2. List of ${\mathcal{T}}_i$, ${\mathcal{N}}_i$, ${\mathcal{T}}_{j|i}$, ${\mathcal{T}}_{j|i}$ and ${\mathcal{N}}_{j|i}$ for $i\in \left[4\right]$ and $j\in \left[4\right]\backslash \left\{i\right\}$ in Example 1.

${\mathcal{T}}_1$	$\left\{0101,0110,0111,1101,1110,1111\right\}$			${\mathcal{N}}_1$	∅
${\mathcal{T}}_2$	$\left\{0101,0111,1101,1111\right\}$			${\mathcal{N}}_2$	${\{0,1\}}^4\backslash {\mathcal{T}}_2$
${\mathcal{T}}_3$	$\left\{0110,0111,1110,1111\right\}$			${\mathcal{N}}_3$	${\{0,1\}}^4\backslash {\mathcal{T}}_3$
${\mathcal{T}}_4$	∅			${\mathcal{N}}_4$	${\{0,1\}}^4$
${\mathcal{T}}_{2|1}$	∅	${\mathcal{T}}_{2|1}$	$\{0101,0111,1101,1111\}$	${\mathcal{N}}_{2|1}$	∅
${\mathcal{T}}_{3|1}$	∅	${\mathcal{T}}_{3|1}$	$\{0110,1110\}$	${\mathcal{N}}_{3|1}$	∅
${\mathcal{T}}_{4|1}$	∅	${\mathcal{T}}_{4|1}$	∅	${\mathcal{N}}_{4|1}$	∅
${\mathcal{T}}_{1|2}$	$\{0101,0111,1101,1111\}$	${\mathcal{T}}_{1|2}$	∅	${\mathcal{N}}_{1|2}$	${\mathcal{N}}_2\backslash \{0000,0010,1000,1010\}$
${\mathcal{T}}_{3|2}$	∅	${\mathcal{T}}_{3|2}$	∅	${\mathcal{N}}_{3|2}$	$\{0010,1010\}$
${\mathcal{T}}_{4|2}$	∅	${\mathcal{T}}_{4|2}$	∅	${\mathcal{N}}_{1|2}$	∅
${\mathcal{T}}_{1|3}$	$\{0110,0111,1110,1111\}$	${\mathcal{T}}_{1|3}$	∅	${\mathcal{N}}_{1|3}$	${\mathcal{N}}_3\backslash \{0000,0001,1000,1001\}$
${\mathcal{T}}_{2|3}$	∅	${\mathcal{T}}_{2|3}$	∅	${\mathcal{N}}_{2|3}$	$\{0001,1001\}$
${\mathcal{T}}_{4|3}$	∅	${\mathcal{T}}_{4|3}$	∅	${\mathcal{N}}_{4|3}$	∅
${\mathcal{T}}_{1|4}$	∅	${\mathcal{T}}_{1|4}$	∅	${\mathcal{N}}_{1|4}$	∅
${\mathcal{T}}_{2|4}$	∅	${\mathcal{T}}_{2|4}$	∅	${\mathcal{N}}_{2|4}$	∅
${\mathcal{T}}_{3|4}$	∅	${\mathcal{T}}_{3|4}$	∅	${\mathcal{N}}_{3|4}$	∅

in Appendix A for a detailed listing of all component subsets in ${\{0,1\}}^4$ (including empty ones).

Overview of the Proof: Given that codeword $c_i$ is sent over the channel, $i\in \left[M\right]$, let ${\mathcal{T}}_i$ denote the set of output tuples $y^n$ that result in MAP decoding ties:

$$\begin{array}{ccc}{\mathcal{T}}_i\hfill & \triangleq \hfill & \left\{y^n\in {\{0,1\}}^n:P_{X^n|Y^n}\left(c_i\right|y^n)=\underset{r\in \left[M\right]\backslash \left\{i\right\}}{max}{P}_{X^n|Y^n}\left(c_r\right|y^n)\right\}\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hspace{1em}\hfill & =\hfill & \left\{y^n\in {\{0,1\}}^n:P_{X^n,Y^n}(c_i,y^n)=\underset{r\in \left[M\right]\backslash \left\{i\right\}}{max}{P}_{X^n,Y^n}(c_r,y^n)\right\},\hfill \end{array}$$

(12)

where (12) holds because $P_{X^n|Y^n}\left(x^n\right|y^n)=\frac{P_{X^n,Y^n}(x^n,y^n)}{P_{Y^n}\left(y^n\right)}$. Then, ${\delta }_n$ in (6) can be rewritten as

$${\delta }_n=\sum _{i\in \left[M\right]}{P}_{X^n}\left(c_i\right)P_{Y^n|X^n}\left({\mathcal{T}}_i|c_i\right)=\sum _{i\in \left[M\right]}{P}_{X^n,Y^n}(c_i,{\mathcal{T}}_i).$$

(13)

Similarly, let ${\mathcal{N}}_i$ denote the set of output tuples $y^n$ which guarantee a tie-free MAP decoding error when $c_i$ is transmitted over the channel:

$$\begin{array}{ccc}{\mathcal{N}}_i\hfill & \triangleq \hfill & \left\{y^n\in {\{0,1\}}^n:P_{X^n|Y^n}\left(c_i\right|y^n)<\underset{r\in \left[M\right]\backslash \left\{i\right\}}{max}{P}_{X^n|Y^n}\left(c_r\right|y^n)\right\}\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hspace{1em}\hfill & =& \left\{y^n\in {\{0,1\}}^n:P_{X^n,Y^n}(c_i,y^n)<\underset{r\in \left[M\right]\backslash \left\{i\right\}}{max}{P}_{X^n,Y^n}(c_r,y^n)\right\}.\hfill \end{array}$$

(15)

Hence, $b_n$ in (5) can be rewritten as:

$$b_n=\sum _{i\in \left[M\right]}{P}_{X^n}\left(c_i\right)P_{Y^n|X^n}\left({\mathcal{N}}_i\right|c_i)=\sum _{i\in \left[M\right]}{P}_{X^n,Y^n}(c_i,{\mathcal{N}}_i).$$

(16)

Note if ${\delta }_n=0$, then (7) is tight and (10) holds trivially; so, without loss of generality, we assume in the proof that ${\delta }_n>0$, which implies that there exists at least one non-empty ${\mathcal{T}}_i$ for $i\in \left[M\right]$. Then, according to (13) and (16), we have that

$$\begin{array}{ccc}\hfill \frac{{\delta }_n}{b_n}& =& \frac{{\sum }_{i\in \left[M\right]}{P}_{X^n,Y^n}(c_i,{\mathcal{T}}_i)}{{\sum }_{i\in \left[M\right]}{P}_{X^n,Y^n}(c_i,{\mathcal{N}}_i)}\le \frac{{\sum }_{i\in \left[M\right]:{\mathcal{T}}_i\ne \varnothing }{P}_{X^n,Y^n}(c_i,{\mathcal{T}}_i)}{{\sum }_{i\in \left[M\right]:{\mathcal{T}}_i\ne \varnothing }{P}_{X^n,Y^n}(c_i,{\mathcal{N}}_i)}.\hfill \end{array}$$

(17)

We can upper-bound (17) by

$$\frac{{\sum }_{i\in \left[M\right]:{\mathcal{T}}_i\ne \varnothing }{P}_{X^n,Y^n}(c_i,{\mathcal{T}}_i)}{{\sum }_{i\in \left[M\right]:{\mathcal{T}}_i\ne \varnothing }{P}_{X^n,Y^n}(c_i,{\mathcal{N}}_i)}\le \underset{1\in \left[M\right]:{\mathcal{T}}_i\ne \varnothing }{max}\frac{P_{X^n,Y^n}(c_i,{\mathcal{T}}_i)}{P_{X^n,Y^n}(c_i,{\mathcal{N}}_i)},$$

(18)

where for convenience we will refer to an inequality of the form given in (18) as the ratio-sum inequality. As a result, Theorem 2 holds if we can substantiate that $2qn$ is an upper bound for (18). To this end, we will find a proper partition of ${\mathcal{T}}_i$ and an equal number of disjoint subsets of ${\mathcal{N}}_i$, of which the individual probabilities can be evaluated. For ease of notation, we denote the individual probabilities corresponding to the K-partition of ${\mathcal{T}}_i$ and K disjoint subsets of ${\mathcal{N}}_i$ by ${\left\{{\alpha }_k\right\}}_{k=1}^K$ and ${\left\{{\beta }_k\right\}}_{k=1}^K$, respectively. Then, we obtain that

$$\begin{array}{c}\hfill \frac{P_{X^n,Y^n}(c_i,{\mathcal{T}}_i)}{P_{X^n,Y^n}(c_i,{\mathcal{N}}_i)}\le \frac{{\sum }_{k=1}^K{\alpha }_k}{{\sum }_{k=1}^K{\beta }_k}.\end{array}$$

(19)

By showing that each individual ratio ${\alpha }_k/{\beta }_k$, $k\in \left[K\right]$ is bounded above by $2qn$, the ratio-sum inequality can again be applied to complete the proof.

3. Proof of Theorem 2

In [1], where a uniformly distributed prior probability $P_{X^n}$ over ${\mathcal{C}}_n$ is assumed, one can flip a properly selected bit in the output $y^n\in {\mathcal{T}}_i$ to convert it to a corresponding element in ${\mathcal{N}}_i$. In light of this connection, one can evaluate the ratio $\frac{P_{Y^n|X^n}\left({\mathcal{T}}_i\right|c_i)}{P_{Y^n|X^n}\left({\mathcal{N}}_i\right|c_i)}$. This approach, however, no longer works when a non-uniformly distributed prior probability is considered. Therefore, we have to devise a more judicious approach to extend the result in [1] for a general prior probability.

3.1. A Partition of Non-Empty ${\mathcal{T}}_i$ and Corresponding Disjoint Subsets of ${\mathcal{N}}_i$

In this section, instead of finding a disjoint covering of the set of decoder ties ${\mathcal{T}}_i$ as in [1], we establish a proper partition of ${\mathcal{T}}_i$ from Definitions 1 and 2. This is one of the key differences from the techniques used in [1]. Example 1 is given after Proposition 1 to illustrate Definitions 1 and 2.

Given $y^n\in {\mathcal{T}}_i$ defined in (12), there exists at least one $m\in \left[M\right]\backslash \left\{i\right\}$ such that

$$\begin{array}{c}\hfill P_{X^n,Y^n}(c_i,y^n)=P_{X^n,Y^n}(c_m,y^n)=\underset{r\in \left[M\right]\backslash \left\{i\right\}}{max}{P}_{X^n,Y^n}(c_r,y^n).\end{array}$$

(20)

We collect the indices m that satisfy (20) in ${\mathcal{I}}_i\left(y^n\right)$ as follows:

$${\mathcal{I}}_i\left(y^n\right)\triangleq \left\{h\in \left[M\right]\backslash \left\{i\right\}:P_{X^n,Y^n}(c_i,y^n)=P_{X^n,Y^n}(c_h,y^n)=\underset{r\in \left[M\right]\backslash \left\{i\right\}}{max}{P}_{X^n,Y^n}(c_r,y^n)\right\}.$$

(21)

Remark 2.

First, we note that ${\mathcal{I}}_i\left(y^n\right)$ is not empty as long as $y^n\in {\mathcal{T}}_i$. Also, for any $y^n\in {\mathcal{T}}_i$, we can infer from (21) that $h\in {\mathcal{I}}_i\left(y^n\right)$ if and only if $y^n\in {\mathcal{T}}_h$.

In Definitions 1 and 2 that follow, we will assign each $y^n\in {\mathcal{T}}_i$ to a subset indexed by $j\in {\mathcal{I}}_i\left(y^n\right)$. These subsets will form a partition of ${\mathcal{T}}_i$ as stated in Proposition 1.

Definition 1.

For $j\in \left[M\right]\backslash \left\{i\right\}$, denoting by ${\mathcal{S}}_{i,j}$ the set of indices where the bit components of $c_i$ and $c_j$ differ, we define

$$\{\begin{array}{}\mathrm{(22a)}& {\mathcal{T}}_{j|i}\triangleq \left\{y^n\in {\mathcal{T}}_i:j={\displaystyle \underset{r\in {\mathcal{I}}_i\left(y^n\right):d(c_i,y^n|{\mathcal{S}}_{i,r})<\left|{\mathcal{S}}_{i,r}\right|}{min}}r\right\};& {\mathcal{N}}_{j|i}\triangleq \{y^n\in {\mathcal{N}}_i:P_{X^n,Y^n}(c_i,y^n)·q=P_{X^n,Y^n}(c_j,y^n)·\frac{1}{q}\mathrm{(22b)}& \mathit{and}{P}_{X^n,Y^n}(c_j,y^n)\ne P_{X^n,Y^n}(c_r,y^n)\mathrm{for}r\in [j-1]\backslash \left\{i\right\}\}.\end{array}$$

Since there may exist $y^n\in {\mathcal{T}}_i$ satisfying $d(c_i,y^n|{\mathcal{S}}_{i,r})=\left|{\mathcal{S}}_{i,r}\right|$ for all $r\in {\mathcal{I}}_i\left(y^n\right)$, the collection of all elements in ${\bigcup }_{j\in \left[M\right]\backslash \left\{i\right\}}{\mathcal{T}}_{j|i}$ may not exhaust the elements in ${\mathcal{T}}_i$ (see Example 1). We thus go on to collect the remaining elements in ${\mathcal{T}}_i\backslash {\bigcup }_{j\in \left[M\right]\backslash \left\{i\right\}}{\mathcal{T}}_{j|i}$ as follows.

Definition 2.

Define for $j\in \left[M\right]\backslash \left\{i\right\}$,

$$\begin{array}{ccc}\hfill {\mathcal{T}}_{j|i}& \triangleq & \left\{y^n\in {\mathcal{T}}_i\backslash \left(\bigcup _{h\in \left[M\right]\backslash \left\{i\right\}}{\mathcal{T}}_{h|i}\right):j=\underset{r\in {\mathcal{I}}_i\left(y^n\right)}{min}r\right\}.\hfill \end{array}$$

(23)

With the sets defined in Definitions 1 and 2, a partition of ${\mathcal{T}}_i$ and disjoint subsets of ${\mathcal{N}}_i$ are constructed as proven in the following proposition.

Proposition 1.

For non-empty ${\mathcal{T}}_i$, the following two properties hold.

(i) 

The collection ${\{{\mathcal{T}}_{j|i}\bigcup {\mathcal{T}}_{j|i}\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$ forms a (disjoint) partition of ${\mathcal{T}}_i$.

(ii) 

${\left\{{\mathcal{N}}_{j|i}\right\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$ is a collection of disjoint subsets of ${\mathcal{N}}_i$.

Before proving Proposition 1, we provide the following example to illustrate the above sets.

Example 1.

This example illustrates the necessity of introducing ${\mathcal{T}}_{j|i}$ as a companion to ${\mathcal{T}}_{j|i}$. Suppose ${\mathcal{C}}_4=\{c_1,c_2,c_3,c_4\}=\{0000,0101,0110,0111\}$. Let $P_{X^4}\left(c_1\right)=\frac{q^2}{2+q^2+q^{-2}}$, $P_{X^4}\left(c_2\right)=P_{X^4}\left(c_3\right)=\frac{1}{2+q^2+q^{-2}}$ and $P_{X^4}\left(c_4\right)=\frac{q^{-2}}{2+q^2+q^{-2}}$. Then, $y^4=0111$ satisfies

$$\underset{\frac{q^2}{(2+q^2+q^{-2})}{p}^4{q}^1}{\underbrace{P_{X^4,Y^4}(c_1,y^4)}}=\underset{r\in \left[4\right]\backslash \left\{1\right\}}{max}{P}_{X^4,Y^4}(c_r,y^4)=\underset{\frac{1}{(2+q^2+q^{-2})}{p}^4{q}^3}{\underbrace{P_{X^4,Y^4}(c_2,y^n)}}=\underset{\frac{1}{(2+q^2+q^{-2})}{p}^4{q}^3}{\underbrace{P_{X^4,Y^4}(c_3,y^4)}}>\underset{\frac{q^{-2}}{(2+q^2+q^{-2})}{p}^4{q}^4}{\underbrace{P_{X^4,Y^4}(c_4,y^4)}},$$

(24)

where the probabilities $P_{X^n,Y^n}(x^n,y^n)$ are written in the form

$$P_{X^n,Y^n}(x^n,y^n)=P_{X^n}\left(x^n\right)P_{Y^n|X^n}\left(y^n\right|x^n)=P_{X^n}\left(x^n\right)p^n{q}^{n-d(x^n,y^n)}.$$

(25)

Note that the first equality in (24) indicates $0111\in {\mathcal{T}}_1$ and the last two equalities and the right-most inequality jointly imply ${\mathcal{I}}_1\left(0111\right)=\{2,3\}$. In light of Proposition 1, this 0111 must lie in one and only one of ${\{{\mathcal{T}}_{j|1}\bigcup {\mathcal{T}}_{j|1}\}}_{j\in \left[4\right]\backslash \left\{1\right\}}$ as shown in

Table A1. Measures used in Example 1.

	$\mathit{d}(0000,{\mathit{y}}^4)-2$	$\mathit{d}(0101,{\mathit{y}}^4)$	$\mathit{d}(0110,{\mathit{y}}^4)$	$\mathit{d}(0111,{\mathit{y}}^4)+2$	${\mathcal{I}}_1\left({\mathit{y}}^4\right)$	${\mathcal{I}}_2\left({\mathit{y}}^4\right)$	${\mathcal{I}}_3\left({\mathit{y}}^4\right)$	${\mathcal{I}}_4\left({\mathit{y}}^4\right)$
$y^4=0000$	$-2$	2	2	5	∅	∅	∅	∅
$y^4=0001$	$-1$	1	3	4	∅	∅	∅	∅
$y^4=0010$	$-1$	3	1	4	∅	∅	∅	∅
$y^4=0100$	$-1$	1	1	4	∅	∅	∅	∅
$y^4=1000$	$-1$	3	3	6	∅	∅	∅	∅
$y^4=0011$	0	2	2	3	∅	∅	∅	∅
$y^4=0101$	0	0	2	3	$\left\{2\right\}$	$\left\{1\right\}$	∅	∅
$y^4=0110$	0	2	0	3	$\left\{3\right\}$	∅	$\left\{1\right\}$	∅
$y^4=1001$	0	2	4	5	∅	∅	∅	∅
$y^4=1010$	0	4	2	5	∅	∅	∅	∅
$y^4=1100$	0	2	2	5	∅	∅	∅	∅
$y^4=0111$	1	1	1	2	$\{2,3\}$	$\{1,3\}$	$\{1,2\}$	∅
$y^4=1011$	1	3	3	4	∅	∅	∅	∅
$y^4=1101$	1	1	3	4	$\left\{2\right\}$	$\left\{1\right\}$	∅	∅
$y^4=1110$	1	3	1	4	$\left\{3\right\}$	∅	$\left\{1\right\}$	∅
$y^4=1111$	2	2	2	3	$\{2,3\}$	$\{1,3\}$	$\{1,2\}$	∅

and Table A2 of Appendix A. Since there exist no integers h in ${\mathcal{I}}_1\left(0111\right)$ fulfilling $d(c_1,0111|{\mathcal{S}}_{1,h})<\left|{\mathcal{S}}_{1,h}\right|$, this 0111 belongs to ${\mathcal{T}}_{j|1}$ with $j={min}_{r\in {\mathcal{I}}_1\left(0111\right)}=2$. Recall that in [1], an element in ${\mathcal{N}}_{j|1}$ can be obtained if flipping a zero of $y^n\in {\mathcal{T}}_{j|1}$ can make it further away from $c_1$ but closer to $c_j$. However, for $y^4=0111$ in this example if we flip the only zero to one, it gets further away from both $c_1$ and $c_h$ for any $h=2,3,4$. Therefore, the bit-flipping manipulation fails.

With $y^4=0111$, we also have

$$\underset{\frac{1}{(2+q^2+q^{-2})}{p}^4{q}^3}{\underbrace{P_{X^4,Y^4}(c_2,y^4)}}=\underset{r\in \left[4\right]\backslash \left\{2\right\}}{max}{P}_{X^4,Y^4}(c_r,y^4)=\underset{\frac{q^2}{(2+q^2+q^{-2})}{p}^{4}q}{\underbrace{P_{X^4,Y^4}(c_1,y^4)}}=\underset{\frac{1}{(2+q^2+q^{-2})}{p}^4{q}^3}{\underbrace{P_{X^4,Y^4}(c_3,y^n)}}>\underset{\frac{1}{(2+q^2+q^{-2})}{p}^4{q}^4}{\underbrace{P_{X^4,Y^4}(c_4,y^4)}},$$

(26)

where the first equality indicates $0111\in {\mathcal{T}}_2$ and the remaining parts in (26) jointly imply that ${\mathcal{I}}_2\left(0111\right)=\{1,3\}$. Proposition 1 then states that this 0111 lies in one and only one of $\{{\mathcal{T}}_{j|2}$$\bigcup {\mathcal{T}}_{j|2}{\}}_{j\in \left[4\right]\backslash \left\{2\right\}}$. Since $1\in {\mathcal{I}}_2\left(0111\right)=\{1,3\}$ and $d(c_2,0111|{\mathcal{S}}_{2,1})=0<\left|{\mathcal{S}}_{2,1}\right|=2$, we have $0111\in {\mathcal{T}}_{1|2}$ according to (22a). Thus, we can flip a bit in 0111 to get further away from $c_2$ and closer to $c_j$ simultaneously. More specifically, the bit-flipping manipulation produces either 0110 or 0011, which lies in ${\mathcal{N}}_{1|2}$ as $y^4=0111$ is in ${\mathcal{T}}_{1|2}$. Therefore, we can associate the element in ${\mathcal{T}}_{1|2}$ with an element in ${\mathcal{N}}_{1|2}$ via a single flipping operation. For completeness, a full list of the sets ${\mathcal{T}}_i$, ${\mathcal{N}}_i$, ${\mathcal{T}}_{j|i}$, ${\mathcal{T}}_{j|i}$ and ${\mathcal{N}}_{j|i}$ for $i\in \left[4\right]$ and $j\in \left[4\right]\backslash \left\{i\right\}$, is given in Appendix A.

Proof of Proposition 1.

First, we note that by the definitions in (22a) and (23), ${\left\{{\mathcal{T}}_{j|i}\right\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$ are disjoint and so is ${\left\{{\mathcal{T}}_{j|i}\right\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$. Additionally, (23) implies ${\mathcal{T}}_{j|i}\bigcap {\mathcal{T}}_{h|i}=\varnothing$ for arbitrary $j,h\in \left[M\right]\backslash \left\{i\right\}$. Furthermore, according to Definitions 1 and 2, for any $y^n\in {\mathcal{T}}_i$, we have either $y^n\in {\mathcal{T}}_{h|i}$ or $y^n\in {\mathcal{T}}_{h|i}$ for some $h\in \left[M\right]\backslash \left\{i\right\}$. Consequently, ${\{{\mathcal{T}}_{j|i}\bigcup {\mathcal{T}}_{j|i}\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$ forms a partition of ${\mathcal{T}}_i$.

On the other hand, the inequality in (22b) prevents multiple inclusions of an element from the previous collections. Therefore, ${\left\{{\mathcal{N}}_{j|i}\right\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$ are a collection of disjoint subsets of ${\mathcal{N}}_i$. □

Remark 3.

When channel inputs are uniformly distributed as considered in [1], it follows that

$${\mathcal{I}}_i\left(y^n\right)=\left\{h\in \left[M\right]\backslash \left\{i\right\}:d(c_i,y^n)=d(c_h,y^n)=\underset{r\in \left[M\right]\backslash \left\{i\right\}}{max}d(c_r,y^n)\right\},$$

(27)

and $d(c_i,y^n|{\mathcal{S}}_{i,j})=\frac{|{\mathcal{S}}_{i,j}|}{2}<\left|{\mathcal{S}}_{i,j}\right|$ for every $j\in {\mathcal{I}}_i\left(y^n\right)$. Therefore, (22a) is reduced to

$$\begin{array}{ccc}\hfill {\mathcal{T}}_{j|i}& =& \left\{y^n\in {\mathcal{T}}_i:j=\underset{r\in {\mathcal{I}}_i\left(y^n\right)}{min}r\right\},\hfill \end{array}$$

(28)

and

$$\begin{array}{c}\hfill {\mathcal{T}}_{j|i}=\varnothing .\end{array}$$

(29)

We then have the following two remarks. First, we note that the ${\mathcal{T}}_{j|i}$ newly defined via (22a) and reduced to (28) in the regime considered in [1] is more restrictive than the ${\mathcal{T}}_{j|i}$ introduced in [1] (Equation (16a)). As a consequence, ${\left\{{\mathcal{T}}_{j|i}\right\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$ forms a partition of ${\mathcal{T}}_i$ in this paper while those introduced in [1] (Equation (16a)) are a disjoint covering of ${\mathcal{T}}_i$ under uniform channel inputs. Second, (29) shows that [1] does not need to consider a companion ${\mathcal{T}}_{j|i}$ to ${\mathcal{T}}_{j|i}$, but this paper does.

Based on Proposition 1, we continue the derivation from (17) and obtain:

$$\begin{array}{ccc}\hfill \frac{{\delta }_n}{b_n}& \le & \frac{{\sum }_{i\in \left[M\right]}{P}_{X^n,Y^n}\left(c_i,{\bigcup }_{j\in \left[M\right]\backslash \left\{i\right\}}({\mathcal{T}}_{j|i}\bigcup {\mathcal{T}}_{j|i})\right)}{{\sum }_{i\in \left[M\right]}{P}_{X^n,Y^n}\left(c_i,{\bigcup }_{j\in \left[M\right]\backslash \left\{i\right\}}{\mathcal{N}}_{j|i}\right)}\hfill \end{array}$$

(30)

$$\begin{array}{ccc}& =& \frac{{\sum }_{i\in \left[M\right]}{\sum }_{j\in \left[M\right]\backslash \left\{i\right\}}{P}_{X^n,Y^n}\left(c_i,{\mathcal{T}}_{j|i}\right)+{\sum }_{i\in \left[M\right]}{\sum }_{j\in \left[M\right]\backslash \left\{i\right\}}{P}_{X^n,Y^n}\left(c_i,{\mathcal{T}}_{j|i}\right)}{{\sum }_{i\in \left[M\right]}{\sum }_{j\in \left[M\right]\backslash \left\{i\right\}}{P}_{X^n,Y^n}\left(c_i,{\mathcal{N}}_{j|i}\right)},\end{array}$$

(31)

where (31) holds because ${\left\{{\mathcal{T}}_{j|i}\right\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$ and ${\left\{{\mathcal{T}}_{j|i}\right\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$ are disjoint, and the same applies to $\{{\mathcal{N}}_{j|i}$${\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$. An additional upper bound for (31) requires the verification of the inequality:

$$\sum _{i\in \left[M\right]}\sum _{j\in \left[M\right]\backslash \left\{i\right\}}{P}_{X^n,Y^n}\left(c_i,{\mathcal{T}}_{j|i}\right)\le \sum _{j\in \left[M\right]}\sum _{i\in \left[M\right]\backslash \left\{j\right\}}{P}_{X^n,Y^n}\left(c_j,{\mathcal{T}}_{i|j}\right),$$

(32)

which is an immediate consequence of the proposition to be proven in the next section (Proposition 2), stating that

$$\begin{array}{ccc}\hfill y^n& \in & {\mathcal{T}}_{j|i}\mathrm{and}h\in {\mathcal{I}}_i\left(y^n\right)\Rightarrow y^n\in {\mathcal{T}}_{\ell |h}\mathrm{for}\mathrm{some}\hfill \\ & & \ell \in {\mathcal{I}}_h\left(y^n\right)\mathrm{and}{P}_{X^n,Y^n}(c_i,y^n)=P_{X^n,Y^n}(c_h,y^n).\hfill \end{array}$$

(33)

3.2. Verification of (32)

Recall that the main technique used in [1] is to associate every element in ${\mathcal{T}}_i$ with a corresponding element in ${\mathcal{N}}_i$ via the bit-flipping manipulation. By this bit-flipping association, the probability ratio of the elements and corresponding elements respectively in ${\mathcal{T}}_i$ and ${\mathcal{N}}_i$ can be evaluated. However, as Example 1 indicates, for an element in ${\mathcal{T}}_{j|i}$, the bit-flipping association no longer works. This reveals the challenge of generalizing the results in [1] from uniform channel inputs to arbitrarily distributed channel inputs. A solution is to subdivide the elements in ${\mathcal{T}}_i$ into two groups ${\left\{{\mathcal{T}}_{j|i}\right\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$ and ${\left\{{\mathcal{T}}_{j|i}\right\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$, where the bit-flipping association to ${\left\{{\mathcal{N}}_{j|i}\right\}}_{j\in \left[M\right]\backslash \left\{i\right\}}$ works for the former group but not for the latter. The inequality in (32) can then be used to exclude the latter group with an upper bound:

$$\begin{array}{ccc}\hfill \frac{{\delta }_n}{b_n}& \le & \frac{{\sum }_{i\in \left[M\right]}{\sum }_{j\in \left[M\right]\backslash \left\{i\right\}}{P}_{X^n,Y^n}\left(c_i,{\mathcal{T}}_{j|i}\right)+{\sum }_{i\in \left[M\right]}{\sum }_{j\in \left[M\right]\backslash \left\{i\right\}}{P}_{X^n,Y^n}\left(c_i,{\mathcal{T}}_{j|i}\right)}{{\sum }_{i\in \left[M\right]}{\sum }_{j\in \left[M\right]\backslash \left\{i\right\}}{P}_{X^n,Y^n}\left(c_i,{\mathcal{N}}_{j|i}\right)}\hfill \end{array}$$

(34)

$$\begin{array}{ccc}& \le \hfill & 2\frac{{\sum }_{i\in \left[M\right]}{\sum }_{j\in \left[M\right]\backslash \left\{i\right\}}{P}_{X^n,Y^n}\left(c_i,{\mathcal{T}}_{j|i}\right)}{{\sum }_{i\in \left[M\right]}{\sum }_{j\in \left[M\right]\backslash \left\{i\right\}}{P}_{X^n,Y^n}\left(c_i,{\mathcal{N}}_{j|i}\right)}.\hfill \end{array}$$

(35)

Since uniform channel inputs as considered in [1] guarantee (29), it can be seen from (35) that the multiplicative factor of 2 can be reduced to 1 as observed in Remark 1. For general arbitrary channel inputs, we have the factor of 2 since the set ${\mathcal{T}}_{j|i}$ may not be empty. The validity of (32) can be confirmed by the next proposition.

Proposition 2.

Suppose $y^n\in {\mathcal{T}}_{j|i}$. Then, for every $h\in {\mathcal{I}}_i\left(y^n\right)$, we have

$$\begin{array}{c}\hfill y^n\in {\mathcal{T}}_{\ell |h}\mathrm{for}\mathrm{some}\ell \in {\mathcal{I}}_h\left(y^n\right)\mathrm{and}{P}_{X^n,Y^n}(c_i,y^n)=P_{X^n,Y^n}(c_h,y^n).\end{array}$$

(36)

Proof. 

Suppose $y^n\in {\mathcal{T}}_{j|i}$. Then, $d(c_i,y^n|{\mathcal{S}}_{i,h})=\left|{\mathcal{S}}_{i,h}\right|$ for every $h\in {\mathcal{I}}_i\left(y^n\right)$. We therefore have:

$$P_{X^n,Y^n}(c_i,y^n)=P_{X^n,Y^n}(c_h,y^n)=\underset{r\in \left[M\right]\backslash \left\{i\right\}}{max}{P}_{X^n,Y^n}(c_r,y^n).$$

(37)

We can rewrite (37) as

$$P_{X^n,Y^n}(c_h,y^n)=P_{X^n,Y^n}(c_i,y^n)=\underset{r\in \left[M\right]\backslash \left\{h\right\}}{max}{P}_{X^n,Y^n}(c_r,y^n),$$

(38)

implying $y^n\in {\mathcal{T}}_h$ and $i\in {\mathcal{I}}_h\left(y^n\right)$. Noting that $d(c_h,y^n|S_{h,i})=0<\left|{\mathcal{S}}_{h,i}\right|$ because $d(c_i,y^n|S_{i,h})=\left|{\mathcal{S}}_{i,h}\right|$ and $S_{h,i}=S_{i,h}$, we conclude that the smallest integer $\ell \in {\mathcal{I}}_h\left(y^n\right)$ satisfying $d(c_h,y^n|S_{h,\ell })<\left|{\mathcal{S}}_{h,\ell }\right|$ exists, and therefore $y^n\in {\mathcal{T}}_{\ell |h}$. □

Remark 4.

Two observations can be made based on Proposition 2. First, Proposition 2 indicates that every $y^n\in {\mathcal{T}}_{j|i}$ must appear at least once in the sum ${\sum }_{h\in \left[M\right]}$${\sum }_{\ell \in \left[M\right]\backslash \left\{h\right\}}$ $P_{X^n,Y^n}\left(c_h,{\mathcal{T}}_{\ell |h}\right)$, contributing the same probability mass $P_{X^n,Y^n}(c_h,y^n)$ as $P_{X^n,Y^n}(c_i,y^n)$. Second, Proposition 2 also implies that every $y^n\in {\mathcal{T}}_{j|i}$ cannot be contained in $\left({\bigcup }_{h\in \left[M\right]}{\bigcup }_{r\in \left[M\right]\backslash \left\{h\right\}}{\mathcal{T}}_{r|h}\right)\backslash {\mathcal{T}}_{j|i}$. This observation can be substantiated as follows. For every $h\in {\mathcal{I}}_i\left(y^n\right)$, Proposition 2 implies $y^n\in {\mathcal{T}}_{\ell |h}$ for some $\ell \in {\mathcal{I}}_h\left(y^n\right)$ and hence Definition 2 immediately gives $y^n\notin {\mathcal{T}}_{r|h}$ for all $r\in \left[M\right]\backslash \left\{h\right\}$. For $h\notin {\mathcal{I}}_i\left(y^n\right)$, we have $y^n\notin {\mathcal{T}}_h$ and therefore $y^n\notin {\mathcal{T}}_{r|h}\subseteq {\mathcal{T}}_h$ for all $r\in \left[M\right]\backslash \left\{h\right\}$ as pointed out in Remark 2. As a result, every $y^n\in {\mathcal{T}}_{j|i}$ appears exactly once in the sum ${\sum }_{h\in \left[M\right]}$${\sum }_{r\in \left[M\right]\backslash \left\{h\right\}}{P}_{X^n,Y^n}\left(c_h,{\mathcal{T}}_{r|h}\right)$. Combining the two observations leads to:

$$\begin{array}{ccc}\hfill \sum _{i\in \left[M\right]}\sum _{j\in \left[M\right]\backslash \left\{i\right\}}\sum _{y^n\in {\mathcal{T}}_{j|i}}{P}_{X^n,Y^n}\left(c_i,y^n\right)& \le & \sum _{j\in \left[M\right]}\sum _{\ell \in \left[M\right]\backslash \left\{j\right\}}\sum _{y^n\in {\mathcal{T}}_{\ell |j}}{P}_{X^n,Y^n}\left(c_j,y^n\right).\hfill \end{array}$$

(39)

To flesh out the above inequality, we give the next example.

Example 2.

Proceeding from Example 1, we observe from Table A1 and Table A2 in Appendix A that 0111 is contained in ${\mathcal{T}}_{2|1}$, ${\mathcal{T}}_{1|2}$ and ${\mathcal{T}}_{1|3}$. Hence, it appears once in the sum ${\sum }_{j\in \left[4\right]}{\sum }_{i\in \left[4\right]\backslash \left\{j\right\}}$ $P_{X^n,Y^n}\left(c_j,{\mathcal{T}}_{i|j}\right)$ while it contributes twice in the sum ${\sum }_{j\in \left[4\right]}{\sum }_{i\in \left[4\right]\backslash \left\{j\right\}}{P}_{X^n,Y^n}\left(c_j,{\mathcal{T}}_{i|j}\right)$. We then confirm from (A35) that:

$$\begin{array}{ccc}\hfill \sum _{i\in \left[4\right]}\sum _{j\in \left[4\right]\backslash \left\{i\right\}}{P}_{X^4,Y^4}\left(c_i,{\mathcal{T}}_{j|i}\right)& \ge & \sum _{i\in \left[4\right]}\sum _{j\in \left[4\right]\backslash \left\{i\right\}}{P}_{X^4,Y^4}\left(c_i,{\mathcal{T}}_{j|i}\right).\hfill \end{array}$$

(40)

We continue the derivation from (35) and obtain

$$\begin{array}{ccc}\hfill \frac{{\delta }_n}{b_n}& \le & 2\frac{{\sum }_{i\in \left[M\right]}{\sum }_{j\in \left[M\right]\backslash \left\{i\right\}:{\mathcal{T}}_{j|i}\ne \varnothing }{P}_{X^n,Y^n}\left(c_i,{\mathcal{T}}_{j|i}\right)}{{\sum }_{i\in \left[M\right]}{\sum }_{j\in \left[M\right]\backslash \left\{i\right\}:{\mathcal{T}}_{j|i}\ne \varnothing }{P}_{X^n,Y^n}\left(c_i,{\mathcal{N}}_{j|i}\right)}\hfill \end{array}$$

(41)

$$\begin{array}{ccc}& \le & 2\underset{i\in \left[M\right]\mathrm{and}j\in \left[M\right]\backslash \left\{i\right\}:{\mathcal{T}}_{j|i}\ne \varnothing }{max}\frac{P_{X^n,Y^n}\left(c_i,{\mathcal{T}}_{j|i}\right)}{P_{X^n,Y^n}\left(c_i,{\mathcal{N}}_{j|i}\right)},\hfill \end{array}$$

(42)

where we add the restriction ${\mathcal{T}}_{j|i}\ne \varnothing$ in (41) to exclude the cases of zero dividing by zero in (42), and (42) follows the ratio-sum inequality in (18).

In the next section, we introduce a number of delicate decompositions of non-empty ${\mathcal{T}}_{j|i}$ and an equal number of disjoint subsets of ${\mathcal{N}}_{j|i}$ to facilitate the bit-flipping association of the pairs.

3.3. Atomic Decomposition of Non-Empty ${\mathcal{T}}_{j|i}$ and the Corresponding Disjoint Subsets of ${\mathcal{N}}_{j|i}$

To simplify the exposition, we assume without loss of generality that $c_1$ is the all-zero codeword (It is known that we can simultaneously flip the same position of all codewords to yield a new code of equal performance over the BSC. Thus, via a number of flipping manipulations, we can transform any code to a code of equal performance with the first codeword being all-zero.) Below we present the proof for $i=1$. The proof for general $i>1$ follows annalagously.

Since $c_1$ is the all-zero codeword, ${\mathcal{S}}_{1,j}$ is the set containing the indices of the non-zero components of $c_j$. To facilitate the investigation of the structure of $c_j$ relative to the remaining codewords ${\left\{c_r\right\}}_{r\in \left[M\right]\backslash \{1,j\}}$, we first partition ${\mathcal{S}}_{1,j}$ into $2^{M-2}$ subsets according to whether each index in ${\mathcal{S}}_{1,j}$ is in ${\mathcal{S}}_{1,2}$, …, ${\mathcal{S}}_{1,j-1}$, ${\mathcal{S}}_{1,j+1}$, …, ${\mathcal{S}}_{1,M}$ or not, as follows:

$${\mathcal{S}}_{1,j}^{\left(m\right)}\triangleq \left(\bigcap _{r=2}^{j-1}{\mathcal{S}}_{r;{\lambda }_r}\right)\bigcap \left(\bigcap _{r=j+1}^M{\mathcal{S}}_{r;{\lambda }_r}\right)\bigcap {\mathcal{S}}_{1,j}\mathrm{for}m\triangleq 1+\sum _{r=2}^{j-1}{\lambda }_r·2^{r-2}+\sum _{r=j+1}^M{\lambda }_r·2^{r-3},$$

(43)

where ${\mathcal{S}}_{r;1}\triangleq {\mathcal{S}}_{1,r}$ and ${\mathcal{S}}_{r;0}\triangleq \left[n\right]\backslash {\mathcal{S}}_{1,r}={\mathcal{S}}_{1,r}^{\mathrm{c}}$, and each ${\lambda }_r\in \{0,1\}$. An example of the partition is given below.

Example 3.

Suppose ${\mathcal{C}}_4=\{00000,11001,01111,01101\}$. For $j=3$ and ${\mathcal{S}}_{1,j}=\{2,3,4,5\}$, we obtain $2^{4-2}=4$ subsets as

$$\begin{array}{c}\hfill {\mathcal{S}}_{1,3}^{\left(m\right)}=\left\{\begin{array}{cc}{\mathcal{S}}_{1,3}^{\left(1\right)}=S_{1,2}^{\mathrm{c}}\bigcap {\mathcal{S}}_{1,4}^{\mathrm{c}}\bigcap {\mathcal{S}}_{1,3}=\left\{4\right\},\hfill & \mathit{if}({\lambda }_4,{\lambda }_2)=(0,0);\hfill \\ {\mathcal{S}}_{1,3}^{\left(2\right)}=S_{1,2}\bigcap {\mathcal{S}}_{1,4}^{\mathrm{c}}\bigcap {\mathcal{S}}_{1,3}=\varnothing ,\hfill & \mathit{if}({\lambda }_4,{\lambda }_2)=(0,1);\hfill \\ {\mathcal{S}}_{1,3}^{\left(3\right)}=S_{1,2}^{\mathrm{c}}\bigcap {\mathcal{S}}_{1,4}\bigcap {\mathcal{S}}_{1,3}=\left\{3\right\},\hfill & \mathit{if}({\lambda }_4,{\lambda }_2)=(1,0);\hfill \\ {\mathcal{S}}_{1,3}^{\left(4\right)}=S_{1,2}\bigcap {\mathcal{S}}_{1,4}\bigcap {\mathcal{S}}_{1,3}=\{2,5\},\hfill & \mathit{if}({\lambda }_4,{\lambda }_2)=(1,1).\hfill \end{array}\right.\end{array}$$

(44)

As $c_1$ is the all-zero codeword, the components of $c_r$ with indices in ${\mathcal{S}}_{1,j}^{\left(m\right)}$ can now be unambiguously identified and must all equal ${\lambda }_r$. As a result,

$$d\left(c_1,c_r|{\mathcal{S}}_{1,j}^{\left(m\right)}\right)=\left\{\begin{array}{cc}\left|{\mathcal{S}}_{1,j}^{\left(m\right)}\right|,\hfill & {\lambda }_r=1;\hfill \\ 0,\hfill & {\lambda }_r=0.\hfill \end{array}\right.$$

(45)

Example 4.

Proceeding from Example 3, we have

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}d\left(c_1,c_2|{\mathcal{S}}_{1,3}^{\left(1\right)}\right)=0\hfill & \mathit{because}{\lambda }_2=0;\hfill \\ d\left(c_1,c_2|{\mathcal{S}}_{1,3}^{\left(2\right)}\right)=\left|{\mathcal{S}}_{1,3}^{\left(2\right)}\right|=0\hfill & \mathit{because}{\lambda }_2=1;\hfill \\ d\left(c_1,c_2|{\mathcal{S}}_{1,3}^{\left(3\right)}\right)=0\hfill & \mathit{because}{\lambda }_2=0;\hfill \\ d\left(c_1,c_2|{\mathcal{S}}_{1,3}^{\left(4\right)}\right)=\left|{\mathcal{S}}_{1,3}^{\left(4\right)}\right|=2\hfill & \mathit{because}{\lambda }_2=1,\hfill \end{array}\right.\end{array}$$

(46)

and

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}d\left(c_1,c_4|{\mathcal{S}}_{1,3}^{\left(1\right)}\right)=0\hfill & \mathit{because}{\lambda }_4=0;\hfill \\ d\left(c_1,c_4|{\mathcal{S}}_{1,3}^{\left(2\right)}\right)=0\hfill & \mathit{because}{\lambda }_4=0;\hfill \\ d\left(c_1,c_4|{\mathcal{S}}_{1,3}^{\left(3\right)}\right)=\left|{\mathcal{S}}_{1,3}^{\left(3\right)}\right|=1\hfill & \mathit{because}{\lambda }_4=1;\hfill \\ d\left(c_1,c_4|{\mathcal{S}}_{1,3}^{\left(4\right)}\right)=\left|{\mathcal{S}}_{1,3}^{\left(4\right)}\right|=2\hfill & \mathit{because}{\lambda }_4=1.\hfill \end{array}\right.\end{array}$$

(47)

It should be emphasized that ${\mathcal{S}}_{1,j}^{\left(m\right)}$ in this paper is defined differently from that in [1]. While the one defined in [1] partitions ${\mathcal{S}}_{1,j}$ only according to codewords with indices less than j, the one defined in this paper considers all other $M-2$ codewords in the partition manipulation, and hence the order of codewords becomes irrelevant.

Next, to decompose ${\mathcal{T}}_{j|1}$, we further define a sequence of incremental sets:

$${\mathcal{S}}_{1,j}^{\left(m\right)}\triangleq \bigcup _{h=1}^m{\mathcal{S}}_{1,j}^{\left(h\right)},m\in \left[2^{M-2}\right],$$

(48)

and set ${\mathcal{S}}_{1,j}^{\left(0\right)}\triangleq \varnothing$. Let ${\ell }_{1,j}\triangleq \left|{\mathcal{S}}_{1,j}\right|$ and ${\ell }_{1,j}^{\left(m\right)}\triangleq \left|{\mathcal{S}}_{1,j}^{\left(m\right)}\right|$ respectively denote the sizes of ${\mathcal{S}}_{1,j}$ and ${\mathcal{S}}_{1,j}^{\left(m\right)}$ and note that $0={\ell }_{1,j}^{\left(0\right)}\le {\ell }_{1,j}^{\left(1\right)}\le {\ell }_{1,j}^{\left(2\right)}\le \cdots \le {\ell }_{1,j}^{\left(2^{M-2}\right)}={\ell }_{1,j}$.

The idea behind the partition of ${\mathcal{T}}_{j|1}$ into ${\ell }_{1,j}$ subsets, indexed by $k\in [{\ell }_{1,j}-1]\bigcup \left\{0\right\}$, is as follows. Pick one $y^n\in {\mathcal{T}}_{j|1}$. We start by examining whether $d(c_1,y^n|{\mathcal{S}}_{1,j}^{\left(1\right)})$ is strickly less than ${\ell }_{1,j}^{\left(1\right)}-1$. If the answer is negative, we continue examining whether $d(c_1,y^n|{\mathcal{S}}_{1,j}^{\left(2\right)})$ is strictly less than ${\ell }_{1,j}^{\left(2\right)}-1$. Proceed until we reach the smallest m such that $d(c_1,y^n|{\mathcal{S}}_{1,j}^{\left(m\right)})<{\ell }_{1,j}^{\left(m\right)}-1$ holds. Setting k to be equal to $k=d(c_1,y^n|{\mathcal{S}}_{1,j}^{\left(m\right)})$, we assign this $y^n$ to the subset ${\mathcal{T}}_{j|1}\left(k\right)$. Notably, there exists no such number $m\in \left[2^{M-2}\right]$ that satisfies $d(c_1,y^n|{\mathcal{S}}_{1,j}^{\left(m\right)})<{\ell }_{1,j}^{\left(m\right)}-1$ if and only if $d(c_1,y^n|{\mathcal{S}}_{1,j})={\ell }_{1,j}-1$; in this case, we find the smallest m satisfying ${\mathcal{S}}_{1,j}^{\left(m\right)}={\mathcal{S}}_{1,j}$ and assign this element to ${\mathcal{T}}_{j|1}({\ell }_{1,j}-1)$ as $d(c_1,y^n|{\mathcal{S}}_{1,j}^{\left(m\right)})={\ell }_{1,j}-1$. For ease of describing the above algorithmic partition process, we introduce a mapping from $k\in [{\ell }_{1,j}-1]\bigcup \left\{0\right\}$ to $m\in \left[2^{M-2}\right]$ as follows:

$${\eta }_k\triangleq \left\{\begin{array}{cc}min\left\{m\in \left[2^{M-2}\right]:k<{\ell }_{1,j}^{\left(m\right)}-1\right\},\hfill & 0\le k<{\ell }_{1,j}-1;\hfill \\ min\left\{m\in \left[2^{M-2}\right]:k={\ell }_{1,j}^{\left(m\right)}-1\right\},\hfill & k={\ell }_{1,j}-1.\hfill \end{array}\right.$$

(49)

We can see that for $0\le k<{\ell }_{1,j}-1$, we have ${\ell }_{1,j}^{({\eta }_k-1)}-1\le k<{\ell }_{1,j}^{\left({\eta }_k\right)}-1$. Therefore, if $y^n\in {\mathcal{T}}_{j|1}$ is assigned to ${\mathcal{T}}_{j|1}\left(k\right)$ for some $k<{\ell }_{1,j}-1$, we must have

$${\ell }_{1,j}^{({\eta }_k-1)}-1\le d\left(c_1,y^n|{\mathcal{S}}_{1,j}^{({\eta }_k-1)}\right)\le d\left(c_1,y^n|{\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}\right)=k<{\ell }_{1,j}^{\left({\eta }_k\right)}-1.$$

(50)

On the other hand, if $y^n\in {\mathcal{T}}_{j|1}$ is collected in ${\mathcal{T}}_{j|1}({\ell }_{1,j}-1)$, then ${\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}={\mathcal{S}}_{1,j}$ and

$${\ell }_{1,j}^{({\eta }_k-1)}-1\le d\left(c_1,y^n|{\mathcal{S}}_{1,j}^{({\eta }_k-1)}\right)\le d\left(c_1,y^n|{\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}\right)={\ell }_{1,j}-1.$$

(51)

A formal definition of ${\mathcal{T}}_{j|1}\left(k\right)$ is given next, where the corresponding subsets ${\mathcal{N}}_{j|1}\left(k\right)$ of ${\mathcal{N}}_{j|1}$ are also introduced.

Definition 3.

Define for $k=0$, 1, …, ${\ell }_{1,j}-1$,

$$\{\begin{array}{}\mathrm{(52a)}& {\mathcal{T}}_{j|1}\left(k\right)\hfill & \triangleq & \left\{y^n\in {\mathcal{T}}_{j|1}:{\ell }_{1,j}^{({\eta }_k-1)}-1\le d\left(c_1,y^n|{\mathcal{S}}_{1,j}^{({\eta }_k-1)}\right)\mathrm{and}d\left(c_1,y^n|{\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}\right)=k\right\};\hfill \mathrm{(52b)}& {\mathcal{N}}_{j|1}\left(k\right)\hfill & \triangleq & \left\{y^n\in {\mathcal{N}}_{j|1}:{\ell }_{1,j}^{({\eta }_k-1)}=d\left(c_1,y^n|{\mathcal{S}}_{1,j}^{({\eta }_k-1)}\right)\mathrm{and}d\left(c_1,y^n|{\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}\right)=k+1\right\},\hfill \end{array}$$

where ${\eta }_k$ is defined in (49).

With Definition 3, we have the following proposition.

Proposition 3.

For non-empty ${\mathcal{T}}_{j|1}$, the following two properties hold.

(i) 

${\left\{{\mathcal{T}}_{j|1}\left(k\right)\right\}}_{k\in [{\ell }_{1,j}-1]\bigcup \left\{0\right\}}$ forms a partition of ${\mathcal{T}}_{j|1}$;

(ii) 

${\left\{{\mathcal{N}}_{j|1}\left(k\right)\right\}}_{k\in [{\ell }_{1,j}-1]\bigcup \left\{0\right\}}$ is a collection of disjoint subsets of ${\mathcal{N}}_{j|1}$.

Proof. 

It can be seen from the definitions of ${\left\{{\mathcal{T}}_{j|1}\left(k\right)\right\}}_{k\in [{\ell }_{1,j}-1]\bigcup \left\{0\right\}}$ and ${\left\{{\mathcal{N}}_{j|1}\left(k\right)\right\}}_{k\in [{\ell }_{1,j}-1]\bigcup \left\{0\right\}}$ that they are collections of mutually disjoint subsets of ${\mathcal{T}}_{j|1}$ and ${\mathcal{N}}_{j|1}$, respectively. It remains to argue that every element in ${\mathcal{T}}_{j|1}$ belongs to ${\mathcal{T}}_{j|1}\left(k\right)$ for some $k\in [{\ell }_{1,j}-1]\bigcup \left\{0\right\}$. Noting that the element $y^n$ in ${\mathcal{T}}_{j|1}$ satisfies $d(c_1,y^n|{\mathcal{S}}_{1,j})\le {\ell }_{1,j}-1$, we differentiate two cases: $d(c_1,y^n|{\mathcal{S}}_{1,j})\le {\ell }_{1,j}-2$ and $d(c_1,y^n|{\mathcal{S}}_{1,j})={\ell }_{1,j}-1$. For the former case, $d(c_1,y^n|{\mathcal{S}}_{1,j}^{\left(m\right)})<{\ell }_{1,j}^{\left(m\right)}-1$ must hold for $m={\eta }_k$; hence, this $y^n$ will be contained in ${\mathcal{T}}_{j|1}\left(k\right)$. For the latter case, $y^n$ will be included in ${\mathcal{T}}_{j|1}({\ell }_{1,j}-1)$. The proof is thus completed. □

In light of Proposition 3, we can apply the ratio-sum inequality to obtain

$$\begin{array}{ccc}\hfill \frac{P_{X^n,Y^n}(c_1,{\mathcal{T}}_{j|1})}{P_{X^n,Y^n}(c_1,{\mathcal{N}}_{j|1})}& \le & \frac{{\sum }_{k=0:{\mathcal{T}}_{j|1}\left(k\right)\ne \varnothing }^{{\ell }_{1,j-1}}{P}_{X^n,Y^n}\left(c_1,{\mathcal{T}}_{j|1}\left(k\right)\right)}{{\sum }_{k=0:{\mathcal{T}}_{j|1}\left(k\right)\ne \varnothing }^{{\ell }_{1,j-1}}{P}_{X^n,Y^n}\left(c_1,{\mathcal{N}}_{j|1}\left(k\right)\right)}\hfill \end{array}$$

(53)

$$\begin{array}{ccc}& \le & \underset{k\in [{\ell }_{1,j}-1]\bigcup \left\{0\right\}:{\mathcal{T}}_{j|1}\left(k\right)\ne \varnothing }{max}\frac{P_{X^n,Y^n}\left(c_1,{\mathcal{T}}_{j|1}\left(k\right)\right)}{P_{X^n,Y^n}\left(c_1,{\mathcal{N}}_{j|1}\left(k\right)\right)}.\hfill \end{array}$$

(54)

We continue to construct a fine partition of ${\mathcal{T}}_{j|1}\left(k\right)$ and the corresponding disjoint subsets of ${\mathcal{N}}_{j|1}\left(k\right)$ in Proposition 4 after giving the next definition.

Definition 4.

Define for $u^n\in {\mathcal{T}}_{j|1}\left(k\right)$,

$$\{\begin{array}{}\mathrm{(55a)}& {\mathcal{T}}_{j|1}(u^n;k)\triangleq \left\{y^n\in {\mathcal{T}}_{j|1}\left(k\right):d\left(u^n,y^n|{\left({\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}\right)}^{\mathrm{c}}\right)=0\right\};\mathrm{(55b)}& {\mathcal{N}}_{j|1}(u^n;k)\triangleq \left\{y^n\in {\mathcal{N}}_{j|1}\left(k\right):d\left(u^n,y^n|{\left({\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}\right)}^{\mathrm{c}}\right)=0\right\},\end{array}$$

where ${\eta }_k$ is given in (49).

Note from Definition 4 that for one element $u^n$ in non-empty ${\mathcal{T}}_{j|1}\left(k\right)$, we can find a group of elements that have identical bit components to $u^n$ with indices in ${\left({\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}\right)}^{\mathrm{c}}$. We denote this group as ${\mathcal{T}}_{j|1}(u^n;k)$. We continue this grouping manipulation until all elements in ${\mathcal{T}}_{j|1}\left(k\right)$ are exhausted as summarized below.

Proposition 4.

For non-empty ${\mathcal{T}}_{j|1}\left(k\right)$, there exists a representative subset ${\mathcal{U}}_{j|1}\left(k\right)\subseteq {\mathcal{T}}_{j|1}\left(k\right)$ such that the following two properties hold.

(i) 

${\left\{{\mathcal{T}}_{j|1}(u^n;k)\right\}}_{u^n\in {\mathcal{U}}_{j|1}\left(k\right)}$ forms a (non-empty) partition of ${\mathcal{T}}_{j|1}\left(k\right)$;

(ii) 

${\left\{{\mathcal{N}}_{j|1}(u^n;k)\right\}}_{u^n\in {\mathcal{U}}_{j|1}\left(k\right)}$ is a collection of (non-empty) disjoint subsets of ${\mathcal{N}}_{j|1}\left(k\right)$.

Since the above proposition can be self-validated via its sequential selection manipulation of each $u^n$ from ${\mathcal{T}}_{j|1}\left(k\right)$, we omit the proof. Interested readers can find the details in [1] (Section III-C).

From Proposition 4, using again the ratio-sum inequality, we obtain that for non-empty ${\mathcal{T}}_{j|1}\left(k\right)$,

$$\begin{array}{ccc}\hfill \frac{P_{X^n,Y^n}\left(c_1,{\mathcal{T}}_{j|1}\left(k\right)\right)}{P_{X^n,Y^n}\left(c_1,{\mathcal{N}}_{j|1}\left(k\right)\right)}& \le & \frac{{\sum }_{u^n\in {\mathcal{U}}_{j|1}\left(k\right)}{P}_{X^n,Y^n}\left(c_1,{\mathcal{T}}_{j|1}(u^n;k)\right)}{{\sum }_{u^n\in {\mathcal{U}}_{j|1}\left(k\right)}{P}_{X^n,Y^n}\left(c_1,{\mathcal{N}}_{j|1}(u^n;k)\right)}\hfill \end{array}$$

(56)

$$\begin{array}{ccc}& \le & \underset{u^n\in {\mathcal{U}}_{j|1}\left(k\right)}{max}\frac{P_{X^n,Y^n}\left(c_1,{\mathcal{T}}_{j|1}(u^n;k)\right)}{P_{X^n,Y^n}\left(c_1,{\mathcal{N}}_{j|1}(u^n;k)\right)}.\hfill \end{array}$$

(57)

Noting that the above result can be similarly conducted for general $i>1$, we combine (42), (54) and (57) to conclude that

$$\begin{array}{c}\hfill \frac{{\delta }_n}{b_n}\le 2\underset{i\in \left[M\right]\mathrm{and}j\in \left[M\right]\backslash \left\{i\right\}:{\mathcal{T}}_{j|i}\ne \varnothing }{max}\underset{k\in [{\ell }_{i,j}-1]\bigcup \left\{0\right\}:{\mathcal{T}}_{j|i}\left(k\right)\ne \varnothing }{max}\underset{u^n\in {\mathcal{U}}_{j|i}\left(k\right)}{max}\frac{P_{X^n,Y^n}\left(c_i,{\mathcal{T}}_{j|i}(u^n;k)\right)}{P_{X^n,Y^n}\left(c_i,{\mathcal{N}}_{j|i}(u^n;k)\right)}.\end{array}$$

(58)

The final task is to evaluate $P_{X^n,Y^n}\left(c_i,{\mathcal{T}}_{j|i}(u^n;k)\right)/P_{X^n,Y^n}\left(c_i,{\mathcal{N}}_{j|i}(u^n;k)\right)$ in order to characterize a linear upper bound for ${\delta }_n/b_n$.

3.4. Characterization of a Linear Upper Bound for ${\delta }_n/b_n$

We again focus on $i=1$ with $c_1$ being the all-zero codeword for simplicity. The definitions of ${\mathcal{T}}_{j|1}(u^n;k)$ in (55a) and ${\mathcal{N}}_{j|1}(u^n;k)$ in (55b) indicate that when dealing with the ratio $P_{X^n,Y^n}(c_1,{\mathcal{T}}_{j|1}(u^n;k))/P_{X^n,Y^n}(c_1,{\mathcal{N}}_{j|1}(u^n;k))$, we only need to consider those bits with indices in ${\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}$ because the remaining bits of all tuples in ${\mathcal{T}}_{j|1}(u^n;k)$ and ${\mathcal{N}}_{j|1}(u^n;k)$ have identical values as $u^n$. Note that all $|{\mathcal{T}}_{j|1}(u^n;k)|$ elements in ${\mathcal{T}}_{j|1}(u^n;k)$ have exactly k ones with indices in ${\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}$, and all $|{\mathcal{N}}_{j|1}(u^n;k)|$ elements in ${\mathcal{N}}_{j|1}(u^n;k)$ have exactly $k+1$ ones with indices in ${\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}$, we can immediately infer that

$$\frac{P_{X^n,Y^n}(c_1,{\mathcal{T}}_{j|1}(u^n;k))}{P_{X^n,Y^n}(c_1,{\mathcal{N}}_{j|1}(u^n;k))}=\frac{P_{X^n}\left(c_1\right)·P_{Y^n|c_1}\left({\mathcal{T}}_{j|1}(u^n;k)\right|c_1)}{P_{X^n}\left(c_1\right)·P_{Y^n|c_1}\left({\mathcal{N}}_{j|1}(u^n;k)\right|c_1)}=\frac{(1-p)}{p}·\frac{|{\mathcal{T}}_{j|1}(u^n;k)|}{|{\mathcal{N}}_{j|1}(u^n;k)|}.$$

(59)

The cardinalities of ${\mathcal{T}}_{j|1}(u^n;k)$ and ${\mathcal{N}}_{j|1}(u^n;k)$ then decide the ratio in (59) as verified in the next proposition, based on which the proof of Theorem 2 can be completed from (58).

Proposition 5.

For $u^n\in {\mathcal{T}}_{j|1}\left(k\right)$, we have

$$\frac{P_{X^n,Y^n}(c_1,{\mathcal{T}}_{j|1}(u^n;k))}{P_{X^n,Y^n}(c_1,{\mathcal{N}}_{j|1}(u^n;k))}\le \frac{(1-p)}{p}n.$$

(60)

Proof. 

Recall from (22a), (52a) and (55a) that $y^n\in {\mathcal{T}}_{j|1}(u^n;k)\subseteq {\mathcal{T}}_{j|1}\left(k\right)\subseteq {\mathcal{T}}_{j|1}$ if and only if

$$\{\begin{array}{}\mathrm{(61a)}& P_{X^n,Y^n}(c_1,y^n)=P_{X^n,Y^n}(c_j,y^n)=\underset{h\in \left[M\right]\backslash \left\{1\right\}}{max}{P}_{X^n,Y^n}(x_{\left(h\right)}^n,y^n)\mathrm{and}d(c_1,y^n|{\mathcal{S}}_{1,j})<\left|{\mathcal{S}}_{i,j}\right|;\mathrm{(61b)}& {\ell }_{1,j}^{({\eta }_k-1)}-1\le d\left(c_1,y^n|{\mathcal{S}}_{1,j}^{({\eta }_k-1)}\right)\mathrm{and}d\left(c_1,y^n|{\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}\right)=k;\mathrm{(61c)}& d\left(u^n,y^n|{\left({\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}\right)}^{\mathrm{c}}\right)=0.\end{array}$$

Thus, the number of elements in ${\mathcal{T}}_{j|1}(u^n;k)$ is exactly the number of channel outputs $y^n$ fulfilling the above three conditions. We then examine the number of $y^n$ satisfying (61b) and (61c). Noting that these $y^n$ have either ${\ell }_{1,j}^{({\eta }_k-1)}-1$ ones or ${\ell }_{1,j}^{({\eta }_k-1)}$ ones with indices in ${\mathcal{S}}_{1,j}^{({\eta }_k-1)}$, we know that there are at most

$$\begin{array}{c}\hfill \left(\genfrac{}{}{0pt}{}{{\ell }_{1,j}^{({\eta }_k-1)}}{{\ell }_{1,j}^{({\eta }_k-1)}-1}\right)\left(\genfrac{}{}{0pt}{}{{\ell }_j^{\left({\eta }_k\right)}-{\ell }_{1,j}^{({\eta }_k-1)}}{k-({\ell }_{1,j}^{({\eta }_k-1)}-1)}\right)+\left(\genfrac{}{}{0pt}{}{{\ell }_{1,j}^{({\eta }_k-1)}}{{\ell }_{1,j}^{({\eta }_k-1)}}\right)\left(\genfrac{}{}{0pt}{}{{\ell }_j^{\left({\eta }_k\right)}-{\ell }_{1,j}^{({\eta }_k-1)}}{k-{\ell }_{1,j}^{({\eta }_k-1)}}\right)\end{array}$$

(62)

of $y^n$ tuples satisfying (61b) and (61c). Disregarding (61a), we get that the number of elements in ${\mathcal{T}}_{j|1}(u^n;k)$ is upper-bounded by (62).

On the other hand, from (22b), (52b) and (55b), we obtain that $w^n\in {\mathcal{N}}_{j|1}(u^n;k)\subseteq {\mathcal{N}}_{j|1}\left(k\right)\subseteq {\mathcal{N}}_{j|1}$ if and only if

$$\{\begin{array}{}\mathrm{(63a)}& P_{X^n,Y^n}(c_1,w^n)·q^2=P_{X^n,Y^n}(c_j,w^n);\mathrm{(63b)}& P_{X^n,Y^n}(c_1,w^n)·q^2\ne P_{X^n,Y^n}(c_r,w^n)\mathrm{for}r\in [j-1]\backslash \left\{1\right\};\mathrm{(63c)}& {\ell }_{1,j}^{({\eta }_k-1)}=d\left(c_1,w^n|{\mathcal{S}}_{1,j}^{({\eta }_k-1)}\right)\mathrm{and}d\left(c_1,w^n|{\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}\right)=k+1;\mathrm{(63d)}& d\left(u^n,w^n|{\left({\mathcal{S}}_{1,j}^{\left({\eta }_k\right)}\right)}^{\mathrm{c}}\right)=0.\end{array}$$

We then claim that any $w^n$ satisfying (63c) and (63d) directly validate (63a) and (63b). Note that the validity of the claim, which we prove in Appendix B, immediately implies that the number of elements in ${\mathcal{N}}_{j|1}(u^n;k)$ can be determined by (63c) and (63d), and hence

$$|{\mathcal{N}}_{j|1}(u^n;k)|=\left(\genfrac{}{}{0pt}{}{{\ell }_j^{\left({\eta }_k\right)}-{\ell }_{1,j}^{({\eta }_k-1)}}{k+1-{\ell }_{1,j}^{({\eta }_k-1)}}\right).$$

(64)

Under this claim, (62) and (64) result in

$$\begin{array}{ccc}\hfill \frac{|{\mathcal{T}}_{j|1}(u^n;k)|}{|{\mathcal{N}}_{j|1}(u^n;k)|}& \le & \frac{\left(\genfrac{}{}{0pt}{}{{\ell }_{1,j}^{({\eta }_k-1)}}{{\ell }_{1,j}^{({\eta }_k-1)}-1}\right)\left(\genfrac{}{}{0pt}{}{{\ell }_j^{\left({\eta }_k\right)}-{\ell }_{1,j}^{({\eta }_k-1)}}{k-({\ell }_{1,j}^{({\eta }_k-1)}-1)}\right)+\left(\genfrac{}{}{0pt}{}{{\ell }_{1,j}^{({\eta }_k-1)}}{{\ell }_{1,j}^{({\eta }_k-1)}}\right)\left(\genfrac{}{}{0pt}{}{{\ell }_j^{\left({\eta }_k\right)}-{\ell }_{1,j}^{({\eta }_k-1)}}{k-{\ell }_{1,j}^{({\eta }_k-1)}}\right)}{\left(\genfrac{}{}{0pt}{}{{\ell }_j^{\left({\eta }_k\right)}-{\ell }_{1,j}^{({\eta }_k-1)}}{k+1-{\ell }_{1,j}^{({\eta }_k-1)}}\right)}\hfill \end{array}$$

(65)

$$\begin{array}{ccc}& =& {\ell }_{1,j}^{({\eta }_k-1)}+\frac{k+1-{\ell }_{1,j}^{({\eta }_k-1)}}{{\ell }_j^{\left({\eta }_k\right)}-k}\hfill \end{array}$$

(66)

$$\begin{array}{ccc}& \le & {\ell }_{1,j}^{({\eta }_k-1)}+\frac{{\ell }_j^{\left({\eta }_k\right)}-{\ell }_{1,j}^{({\eta }_k-1)}}{1}\hfill \end{array}$$

(67)

$$\begin{array}{ccc}& \le & n,\hfill \end{array}$$

(68)

where (67) holds because $k\le {\ell }_j^{\left({\eta }_k\right)}-1$ by (49), and (68) follows from ${\ell }_{1,j}^{\left({\eta }_k\right)}\le {\ell }_{1,j}\le n$. The proof of the proposition is thus completed by (59) and (68). □

4. Conclusions

In this paper, we analyzed the error probability of block codes sent over the memoryless BSC under an arbitrary (not necessarily uniform) input distribution and used in conjunction with (optimal) MAP decoding. We showed that decoder ties do not affect the error exponent of the probability of error, thus extending a similar result recently established in [1] for uniformly distributed channel inputs. This result was obtained by proving that the relative deviation of the error probability from the probability of error grows no more than linearly in blocklength when no MAP decoding ties occur, directly implying that decoder ties have only a sub-exponential effect on the error probability as blocklength grows without bound. Future work includes further extending this result for more general channels used under arbitrary input statistics, such as non-binary symmetric channels (Note that the result of Theorem 1 can be extended for non-binary (q-ary, $q>2$) codes sent over q-ary symmetric memoryless channels under a uniform input distribution; see [25] (Theorem 2).) and binary non-symmetric channels. Studying how to sharpen the upper bound derived in (10) for “sufficiently good” codes as highlighted in Remark 1 and for codes with small blocklengths are other worthwhile future directions.