Trade-offs between Error Exponents and Excess-Rate Exponents of Typical Slepian–Wolf Codes

Abstract

Typical random codes (TRCs) in a communication scenario of source coding with side information in the decoder is the main subject of this work. We study the semi-deterministic code ensemble, which is a certain variant of the ordinary random binning code ensemble. In this code ensemble, the relatively small type classes of the source are deterministically partitioned into the available bins in a one-to-one manner. As a consequence, the error probability decreases dramatically. The random binning error exponent and the error exponent of the TRCs are derived and proved to be equal to one another in a few important special cases. We show that the performance under optimal decoding can be attained also by certain universal decoders, e.g., the stochastic likelihood decoder with an empirical entropy metric. Moreover, we discuss the trade-offs between the error exponent and the excess-rate exponent for the typical random semi-deterministic code and characterize its optimal rate function. We show that for any pair of correlated information sources, both error and excess-rate probabilities exponential vanish when the blocklength tends to infinity.

1. Introduction

As is well known, the random coding error exponent is defined by

$$\begin{array}{c}\hfill {\mathsf{E}}_r\left(R\right)=\underset{n\to \infty }{lim}\left\{-\frac{1}{n}log\mathbb{E}\left[P_e\left({\mathcal{C}}_n\right)\right]\right\},\end{array}$$

(1)

where R is the coding rate, $P_e\left({\mathcal{C}}_n\right)$ is the error probability of a codebook ${\mathcal{C}}_n$, and the expectation is with respect to (w.r.t.) the randomness of ${\mathcal{C}}_n$ across the ensemble of codes. The error exponent of the typical random code (TRC) is defined as [1]

$$\begin{array}{c}\hfill {\mathsf{E}}_{trc}\left(R\right)=\underset{n\to \infty }{lim}\left\{-\frac{1}{n}\mathbb{E}\left[log{P}_e\left({\mathcal{C}}_n\right)\right]\right\}.\end{array}$$

(2)

We believe that the error exponent of the TRC is the more relevant performance metric, as it captures the most likely error exponent of a randomly selected code, as opposed to the random coding error exponent, which is dominated by the relatively poor codes of the ensemble, rather than the channel noise, at relatively low coding rates. In addition, since in random coding analysis, the code is selected at random and remains fixed, it seems reasonable to study the performance of the chosen code itself instead of directly considering the ensemble performance.

To the best of our knowledge, not much is known about TRCs. In [2], Barg and Forney considered TRCs with independently and identically distributed codewords, along with typical linear codes, for the special case of the binary symmetric channel with maximum likelihood (ML) decoding. It was also shown that at a certain range of low rates, $E_{trc}\left(R\right)$ lies between $E_r\left(R\right)$ and the expurgated exponent, $E_{ex}\left(R\right)$. In [3] Nazari et al. provided bounds on the error exponents of TRCs for both discrete memoryless channels (DMC) and multiple-access channels. In a recent article by Merhav [1], an exact single-letter expression has been derived for the error exponent of typical, random, fixed composition codes, over DMCs, and a wide class of (stochastic) decoders, collectively referred to as the generalized likelihood decoder (GLD). Later, Merhav studied error exponents of TRCs for the colored Gaussian channel [4], typical random trellis codes [5], and a Lagrange dual lower bound for the TRC exponent [6]. Large deviations around the TRC exponent were studied in [7].

While originally defined for pure channel coding [1,2,3], the notion of TRCs has natural analogues in other settings as well, such as source coding with side information in the decoder [8]. Typical random Slepian–Wolf (SW) code of a certain variant of the ordinary variable-rate random binning code ensemble is the main theme of this work. The random coding error exponent of SW coding, based on fixed-rate (FR) random binning, was first addressed by Gallager in [9], and improved later on by the expurgated bound in [10] and [11]. Variable-rate (VR) SW coding received less attention in the literature; VR codes under average rate constraint have been studied in [12] and proved to outperform FR codes in terms of error exponents. Optimum trade-offs between the error exponent and the excess-rate exponent in VR coding were analyzed in [13]. Sphere-packing upper bounds for source coding with side information in the FR and VR regimes were studied in [9] and [12], respectively. More works where exponential error bounds in source coding have been studied are [14,15,16,17,18].

It turns out that both the FR and VR ensembles suffer from an intrinsic deficiency, caused by statistical fluctuations in the sizes of the bins that are populated by the relatively small type classes of the source. This fundamental problem of the ordinary ensembles is alleviated in some variant of the ordinary VR ensemble-the semi-deterministic (SD) code ensemble, which has already been proposed and studied in its FR version in [18]. In the SD code ensemble, for source type classes which are exponentially larger than the space of the available bins, we just randomly assign each source sequence into one of the bins, as is done in ordinary random binning. Otherwise, for relatively small type classes, we deterministically order each source sequence into a different bin, which provides a one-to-one mapping. This way, all these relatively small source type classes do not contribute to the probability of error. The main results concerning the SD code are the following:

• The random binning error exponent and the error exponent of the TRC are derived in Theorems 1 and 2, respectively, and proved in Theorem 3 to be equal to one another in a few important special cases, which include the matched likelihood decoder, the MAP decoder, and the universal minimum entropy decoder. To the best of our knowledge, this phenomenon has not been seen elsewhere before, since the TRC exponent usually improves upon the random coding exponent. As a byproduct, we are able to provide a relatively simple expression for the TRC exponent.
• We prove in Theorem 4 that the error exponent of the TRC under MAP decoding is also attained by two universal decoders: the minimum entropy decoder and the stochastic entropy decoder, which is a GLD with an empirical conditional entropy metric. As far as we know, this result is first of its kind in source coding; in other scenarios, the random coding bound is attained also by universal decoders, but here, we find that the TRC exponent is also universally achievable. Moreover, while the likelihood decoder and the MAP decoder have similar error exponents [19], here we prove a similar result, but for two universal decoders (one stochastic and one deterministic) that share the same metric.
• We discuss the trade-offs between the error exponent and the excess-rate exponent for a typical random SD code, similarly to [13], but with a different notion of the excess-rate event, which takes into account the available side information. In Theorem 5, we provide an expression for the optimal rate function that guarantees a required level for the error exponent of the typical random SD code. Analogously, Theorem 6 proposes an expression for the optimal rate function that guarantees a required level for the excess-rate exponent. Furthermore, we show that for any pair of correlated information sources, the typical random SD code attains both exponentially vanishing error and excess-rate probabilities.

The remaining part of the paper is organized as follows. In Section 2, we establish notation conventions. In Section 3, we formalize the model, the coding technique, the main objectives of this work, and we review some background. In Section 4, we provide the main results concerning error exponents and universal decoding in the SD ensemble, and in Section 5, we discuss the trade-offs between the error exponent and the excess-rate exponent.

2. Notation Conventions

Throughout the paper, random variables will be denoted by capital letters, realizations will be denoted by the corresponding lower case letters, and their alphabets will be denoted by calligraphic letters. Random vectors and their realizations will be denoted, respectively, by boldface capital and lower case letters. Their alphabets will be superscripted by their dimensions. Sources and channels will be subscripted by the names of the relevant random variables/vectors and their conditionings, whenever applicable, following the standard notation conventions, e.g., $Q_U$, $Q_{V|U}$, and so on. When there is no room for ambiguity, these subscripts will be omitted. For a generic joint distribution $Q_{UV}=\{Q_{UV}(u,v),u\in \mathcal{U},v\in \mathcal{V}\}$, which will often be abbreviated by Q, information measures will be denoted in the conventional manner, but with a subscript Q; that is, $H_Q\left(U\right)$ is the marginal entropy of U, $H_Q\left(U\right|V)$ is the conditional entropy of U given V, and $I_Q(U;V)=H_Q\left(U\right)-H_Q\left(U\right|V)$ is the mutual information between U and V. The Kullback–Leibler divergence between two probability distributions, $Q_{UV}$ and $P_{UV}$, is defined as

$$\begin{array}{c}\hfill D(Q_{UV}\parallel P_{UV})=\sum _{(u,v)\in \mathcal{U}\times \mathcal{V}}{Q}_{UV}(u,v)log\frac{Q_{UV}(u,v)}{P_{UV}(u,v)},\end{array}$$

(3)

where logarithms, here and throughout the sequel, are understood to be taken to the natural base. The probability of an event $\mathcal{E}$ will be denoted by $\mathbb{P}\left\{\mathcal{E}\right\}$, and the expectation operator w.r.t. a probability distribution Q will be denoted by ${\mathbb{E}}_Q[·]$, where the subscript will often be omitted. For two positive sequences, $\left\{a_n\right\}$ and $\left\{b_n\right\}$, the notation $a_n\doteq b_n$ will stand for equality in the exponential scale, that is, ${lim}_{n\to \infty }(1/n)log\left(a_n/b_n\right)=0$. Similarly, $a_n\stackrel{·}{\le }{b}_n$ means that ${lim\; sup}_{n\to \infty }(1/n)log\left(a_n/b_n\right)\le 0$, and so on. The indicator function of an event $\mathcal{A}$ will be denoted by $𝟙\left\{\mathcal{A}\right\}$. The notation ${\left[t\right]}_{+}$ will stand for $max\{0,t\}$.

The empirical distribution of a sequence $\mathit{u}\in {\mathcal{U}}^n$, which will be denoted by ${\widehat{P}}_{\mathit{u}}$, is the vector of relative frequencies, ${\widehat{P}}_{\mathit{u}}\left(u\right)$, of each symbol $u\in \mathcal{U}$ in $\mathit{u}$. The type class of $\mathit{u}\in {\mathcal{U}}^n$, denoted $\mathcal{T}\left(\mathit{u}\right)$, is the set of all vectors ${\mathit{u}}^{\prime }$ with ${\widehat{P}}_{{\mathit{u}}^{\prime }}={\widehat{P}}_{\mathit{u}}$. When we wish to emphasize the dependence of the type class on the empirical distribution $\widehat{P}$, we will denote it by $\mathcal{T}\left(\widehat{P}\right)$. The set of all types of vectors of length n over $\mathcal{U}$ will be denoted by ${\mathcal{P}}_n\left(\mathcal{U}\right)$, and the set of all possible types over $\mathcal{U}$ will be denoted by $\mathcal{P}\left(\mathcal{U}\right)={\bigcup }_{n=1}^{\infty }{\mathcal{P}}_n\left(\mathcal{U}\right)$. Information measures associated with empirical distributions will be denoted with ‘hats’ and will be subscripted by the sequences from which they are induced. For example, the entropy associated with ${\widehat{P}}_{\mathit{u}}$, which is the empirical entropy of $\mathit{u}$, will be denoted by ${\widehat{H}}_{\mathit{u}}\left(U\right)$. Similar conventions will apply to the joint empirical distribution, the joint type class, the conditional empirical distributions and the conditional type classes associated with pairs (and multiples) of sequences of length n. Accordingly, ${\widehat{P}}_{\mathit{u}\mathit{v}}$ would be the joint empirical distribution of $(\mathit{u},\mathit{v})={\left\{(u_i,v_i)\right\}}_{i=1}^n$, $\mathcal{T}\left({\widehat{P}}_{\mathit{u}\mathit{v}}\right)$ will denote the joint type class of $(\mathit{u},\mathit{v})$, $\mathcal{T}\left({\widehat{P}}_{\mathit{u}|\mathit{v}}\right|\mathit{v})$ will stand for the conditional type class of $\mathit{u}$ given $\mathit{v}$, ${\widehat{H}}_{\mathit{u}\mathit{v}}\left(U\right|V)$ will be the empirical conditional entropy, and so on. Likewise, when we wish to emphasize the dependence of empirical information measures upon a given empirical distribution Q, we denote them using the subscript Q, as described above.

3. Problem Formulation and Background

3.1. Problem Formulation

Let $(\mathit{U},\mathit{V})={\left\{(U_t,V_t)\right\}}_{t=1}^n$ be n independent copies of a pair of random variables, $(U,V)\sim P_{UV}$, taking on values in finite alphabets, $\mathcal{U}$ and $\mathcal{V}$, respectively. The vector $\mathit{U}$ will designate the source vector to be encoded and the vector $\mathit{V}$ will serve as correlated side information available to the decoder. In ordinary VR binning, the coding rate is not fixed for every $\mathit{u}\in {\mathcal{U}}^n$, but depends on its empirical distribution. Let us denote a rate function by $R(·)$, which is a given continuous function from the probability simplex of $\mathcal{U}$ to the set of nonnegative reals. In that manner, for every type $Q_U\in {\mathcal{P}}_n\left(\mathcal{U}\right)$, all source sequences in $\mathcal{T}\left(Q_U\right)$ are randomly partitioned into $e^{nR\left(Q_U\right)}$ bins. Every source sequence is encoded by its bin index, denoted by $\mathcal{B}\left(\mathit{u}\right)$, along with a header that indicates its type index, which requires only a negligible extra rate when n is large enough.

The SD code ensemble is a refinement of the ordinary VR code: for types with $H_Q\left(U\right)\ge R\left(Q_U\right)$, i.e., type classes which are exponentially larger than the space of available bins, we just randomly assign each source sequence into one out of the $e^{nR\left(Q_U\right)}$ bins. For the other types, we deterministically order each member of $\mathcal{T}\left(Q_U\right)$ into a different bin. This way, all type classes with $H_Q\left(U\right)<R\left(Q_U\right)$ do not contribute to the probability of error. The entire binning code of source sequences of blocklength n, i.e., the set ${\left\{\mathcal{B}\left(\mathit{u}\right)\right\}}_{\mathit{u}\in {\mathcal{U}}^n}$, is denoted by ${\mathcal{B}}_n$. A sequence of SW codes, ${\left\{{\mathcal{B}}_n\right\}}_{n\ge 1}$, indexed by the block length n, will be denoted by $\mathcal{B}$.

The decoder estimates $\mathit{u}$ based on the bin index $\mathcal{B}\left(\mathit{u}\right)$, the type index $\mathcal{T}\left(\mathit{u}\right)$, and the side information sequence $\mathit{v}$, which is a realization of $\mathit{V}$. The optimal (MAP) decoder estimates $\mathit{u}$ according to

$$\begin{array}{c}\hfill \widehat{\mathit{u}}=\underset{{\mathit{u}}^{\prime }\in \mathcal{B}\left(\mathit{u}\right)\cap \mathcal{T}\left(\mathit{u}\right)}{arg\; max}P({\mathit{u}}^{\prime },\mathit{v}).\end{array}$$

(4)

As in [1,20], we consider here the GLD. The GLD estimates $\mathit{u}$ stochastically, using the bin index $\mathcal{B}\left(\mathit{u}\right)$, the type index $\mathcal{T}\left(\mathit{u}\right)$, and the side information sequence $\mathit{v}$, according to the following posterior distribution

$$\begin{array}{c}\hfill \mathbb{P}\left\{\widehat{\mathit{U}}={\mathit{u}}^{\prime }|\mathit{v},\mathcal{B}\left(\mathit{u}\right),\mathcal{T}\left(\mathit{u}\right)\right\}=\frac{exp\left\{nf\left({\widehat{P}}_{{\mathit{u}}^{\prime }\mathit{v}}\right)\right\}}{{\sum }_{\tilde{\mathit{u}}\in \mathcal{B}\left(\mathit{u}\right)\cap \mathcal{T}\left(\mathit{u}\right)}exp\left\{nf\left({\widehat{P}}_{\tilde{\mathit{u}}\mathit{v}}\right)\right\}},\end{array}$$

(5)

where ${\widehat{P}}_{\mathit{u}\mathit{v}}$ is the empirical distribution of $(\mathit{u},\mathit{v})$ and $f(·)$ is a given continuous, real valued functional of this empirical distribution. The GLD provides a unified framework which covers several important special cases, e.g., matched decoding, mismatched decoding, MAP decoding, and universal decoding (similarly to the $\alpha$-decoders described in [11]). A more detailed discussion is given in [20].

The probability of error is the probability of the event $\{\widehat{\mathit{U}}\ne \mathit{U}\}$. For a given binning code ${\mathcal{B}}_n$, the probability of error is given by

$$\begin{array}{c}\hfill P_e\left({\mathcal{B}}_n\right)=\sum _{\mathit{u},\mathit{v}}P(\mathit{u},\mathit{v})·𝟙\left\{{\widehat{H}}_{\mathit{u}}\left(U\right)\ge R\left({\widehat{P}}_{\mathit{u}}\right)\right\}·\frac{{\sum }_{{\mathit{u}}^{\prime }\in \mathcal{B}\left(\mathit{u}\right)\cap \mathcal{T}\left(\mathit{u}\right),{\mathit{u}}^{\prime }\ne \mathit{u}}exp\left\{nf\left({\widehat{P}}_{{\mathit{u}}^{\prime }\mathit{v}}\right)\right\}}{{\sum }_{\tilde{\mathit{u}}\in \mathcal{B}\left(\mathit{u}\right)\cap \mathcal{T}\left(\mathit{u}\right)}exp\left\{nf\left({\widehat{P}}_{\tilde{\mathit{u}}\mathit{v}}\right)\right\}}.\end{array}$$

(6)

For a given rate function, we derive the random binning exponent of this ensemble, which is defined by

$$\begin{array}{c}\hfill {\mathsf{E}}_r\left(R(·)\right)=\underset{n\to \infty }{lim}\left\{-\frac{log\mathbb{E}\left[P_e\left({\mathcal{B}}_n\right)\right]}{n}\right\},\end{array}$$

(7)

and compare it to the TRC exponent, which is

$$\begin{array}{c}\hfill {\mathsf{E}}_{trc}\left(R(·)\right)=\underset{n\to \infty }{lim}\left\{-\frac{\mathbb{E}[log{P}_e\left({\mathcal{B}}_n\right)]}{n}\right\}.\end{array}$$

(8)

Although it is unclear that the limits in (7) and (8) exist a priori, it will be evident from the analyses in Appendix A and Appendix B, respectively.

One way to define the excess-rate probability is as $\mathbb{P}\{R\left({\widehat{P}}_{\mathit{U}}\right)\ge R\}$, where R is some target rate [13]. Due to the availability of side information in the decoder, it makes sense to require a target rate which depends on the pair $(\mathit{u},\mathit{v})$. Since the lowest possible compression rate in this setting is given by $H_P\left(U\right|V)$ [8], given $\mathit{U}=\mathit{u}$ and $\mathit{V}=\mathit{v}$, it is reasonable to adopt ${\widehat{H}}_{\mathit{u}\mathit{v}}\left(U\right|V)$ as a reference rate. Hence, an alternative definition of the excess-rate probability of a code ${\mathcal{B}}_n$, is as $p_{er}({\mathcal{B}}_n,R(·),\Delta )=\mathbb{P}\{R\left({\widehat{P}}_{\mathit{U}}\right)\ge {\widehat{H}}_{\mathit{U}\mathit{V}}\left(U\right|V)+\Delta \}$, where $\Delta >0$ is a redundancy threshold. (Note that the entire analysis remains intact if we allow a more general redundancy threshold as $\Delta =\Delta \left({\widehat{P}}_{\mathit{u}\mathit{v}}\right)$. This covers other alternatives for the excess-rate probability, e.g., $\mathbb{P}\{R\left({\widehat{P}}_{\mathit{U}}\right)\ge R\}$ or $\mathbb{P}\{R\left({\widehat{P}}_{\mathit{U}}\right)\ge \alpha {\widehat{H}}_{\mathit{U}}\left(U\right)\}$, $\alpha \in (0,1)$.) Accordingly, the excess-rate exponent function, achieved by a sequence of codes $\mathcal{B}$, is defined as

$$\begin{array}{c}\hfill {\mathsf{E}}_{er}(\mathcal{B},R(·),\Delta )=\underset{n\to \infty }{lim\; inf}-\frac{1}{n}log{p}_{er}({\mathcal{B}}_n,R(·),\Delta ).\end{array}$$

(9)

The main mission is to characterize the optimal trade-off between the error exponent and the excess-rate exponent for the typical random SD code, and the optimal rate function that attains a prescribed value for the error exponent of the typical random SD code.

3.2. Background

In pure channel coding, Merhav [1] has derived a single-letter expression for the error exponent of the typical random fixed composition code:

$$\begin{array}{c}\hfill {\mathsf{E}}_{trc}(R,Q_X)=\underset{n\to \infty }{lim}\left\{-\frac{1}{n}\mathbb{E}\left[log{P}_e\left({\mathcal{C}}_n\right)\right]\right\}.\end{array}$$

(10)

In order to present the main result of [1], we define first a few quantities. Consider a DMC, $W=\left\{W\right(y\left|x\right),x\in \mathcal{X},y\in \mathcal{Y}\}$, where $\mathcal{X}$ and $\mathcal{Y}$ are the finite input/output alphabets. Define

$$\begin{array}{c}\hfill \alpha (R,Q_Y)=\underset{\{Q_{\tilde{X}|Y}:I_Q(\tilde{X};Y)\le R,Q_{\tilde{X}}=Q_X\}}{max}\{g\left(Q_{\tilde{X}Y}\right)-I_Q(\tilde{X};Y)\}+R,\end{array}$$

(11)

where the function $g(·)$, which is the decoding metric, is a continuous function that maps joint probability distributions over $\mathcal{X}\times \mathcal{Y}$ to real numbers. Additionally define

$$\begin{array}{cc}\hfill \Gamma (Q_{X{X}^{\prime }},R)& =\underset{Q_{Y|X{X}^{\prime }}}{min}\left\{D\right(Q_{Y|X}\parallel W|Q_X)+I_Q(X^{\prime };Y|X)\hfill \\ \hfill & +{[max\{g\left(Q_{XY}\right),\alpha (R,Q_Y)\}-g\left(Q_{X^{\prime }Y}\right)]}_{+}\},\hfill \end{array}$$

(12)

where $D(Q_{Y|X}\parallel W|Q_X)$ is the conditional divergence between $Q_{Y|X}$ and W, averaged by $Q_X$. A brief intuitive explanation on the term $\Gamma (Q_{X{X}^{\prime }},R)$ can be found in [7] (Section 4.1). Having defined the above quantities, the error exponent of the TRC is given by [1]

$$\begin{array}{c}\hfill {\mathsf{E}}_{trc}(R,Q_X)=\underset{\{Q_{X^{\prime }|X}:I_Q(X;X^{\prime })\le 2R,Q_{X^{\prime }}=Q_X\}}{min}\{\Gamma (Q_{X{X}^{\prime }},R)+I_Q(X;X^{\prime })-R\}.\end{array}$$

(13)

Returning to the SW model, several articles have been written on error exponents for the FR and the VR codes. Here, we mention only those results that are directly relevant to the current work. The random binning and expurgated bounds of the FR ensemble in the SW model are given, respectively, by [11] (Section VI, Theorem 2), [10](Appendix I, Theorem 1)

$$\begin{array}{cc}\hfill E_r^{fr}\left(R\right)& =\underset{Q_U}{min}\left\{D(Q_U\parallel P_U)+E_r(Q_U,P_{V|U},H_Q\left(U\right)-R)\right\},\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill E_{ex}^{fr}\left(R\right)& =\underset{Q_U}{min}\left\{D(Q_U\parallel P_U)+E_{ex}(Q_U,P_{V|U},H_Q\left(U\right)-R)\right\},\hfill \end{array}$$

(15)

where $E_r(Q_U,P_{V|U},S)$ and $E_{ex}(Q_U,P_{V|U},S)$ are, respectively, the random coding and expurgated bounds associated with the channel $P_{V|U}$ w.r.t. the ensemble of fixed composition code of rate S, whose composition is $Q_U$. The exponent function $E_r(Q_U,P_{V|U},S)$ is given by

$$\begin{array}{c}\hfill E_r(Q_U,P_{V|U},S)=\underset{Q_{V|U}}{min}\left\{D\right(Q_{V|U}\parallel P_{V|U}|Q_U)+{[I_Q(U;V)-S]}_{+}\},\end{array}$$

(16)

and $E_{ex}(Q_U,P_{V|U},S)$ is given by

$$\begin{array}{c}\hfill E_{ex}(Q_U,P_{V|U},S)=\underset{\{Q_{U^{\prime }|U}:I_Q(U;U^{\prime })\le S,Q_{U^{\prime }}=Q_U\}}{min}\{{\mathbb{E}}_{Q_{U{U}^{\prime }}}\left[d_{P_{V|U}}(U,U^{\prime })\right]+I_Q(U;U^{\prime })-S\},\end{array}$$

(17)

where

$$\begin{array}{c}\hfill d_{P_{V|U}}(u,u^{\prime })=-log\left[\sum _{v\in \mathcal{V}}\sqrt{P_{V|U}\left(v\right|u)P_{V|U}\left(v\right|u^{\prime })}\right].\end{array}$$

(18)

The exact error exponent of VR random binning is given by [13] (Equation (34)):

$$\begin{array}{c}\hfill E_r^{vr}\left(R(·)\right)=\underset{Q_{UV}}{min}\left\{D(Q_{UV}\parallel P_{UV})+{[R\left(Q_U\right)-H_Q\left(U\right|V)]}_{+}\right\}.\end{array}$$

(19)

4. Error Exponents and Universal Decoding

To present some of the results, we need a few more definitions. The minimum conditional entropy (MCE) decoder estimates $\mathit{u}$, using the bin index $\mathcal{B}\left(\mathit{u}\right)$, the type index $\mathcal{T}\left(\mathit{u}\right)$, and the side information vector $\mathit{v}$, according to

$$\begin{array}{c}\hfill \widehat{\mathit{u}}=\underset{{\mathit{u}}^{\prime }\in \mathcal{B}\left(\mathit{u}\right)\cap \mathcal{T}\left(\mathit{u}\right)}{arg\; min}{\widehat{H}}_{{\mathit{u}}^{\prime }\mathit{v}}\left(U\right|V).\end{array}$$

(20)

The stochastic conditional entropy (SCE) decoder is a special case of the GLD with the decoding metric $f\left({\widehat{P}}_{\mathit{u}\mathit{v}}\right)=-{\widehat{H}}_{\mathit{u}\mathit{v}}\left(U\right|V)$; i.e., it estimates $\mathit{u}$ according to the following posterior distribution

$$\begin{array}{c}\hfill \mathbb{P}\left\{\widehat{\mathit{U}}={\mathit{u}}^{\prime }|\mathit{v},\mathcal{B}\left(\mathit{u}\right),\mathcal{T}\left(\mathit{u}\right)\right\}=\frac{exp\{-n{\widehat{H}}_{{\mathit{u}}^{\prime }\mathit{v}}\left(U\right|V)\}}{{\sum }_{\tilde{\mathit{u}}\in \mathcal{B}\left(\mathit{u}\right)\cap \mathcal{T}\left(\mathit{u}\right)}exp\{-n{\widehat{H}}_{\tilde{\mathit{u}}\mathit{v}}\left(U\right|V)\}}.\end{array}$$

(21)

First, we present random binning error exponents, which are modifications of (19) to this ensemble. Define the expression

$$\begin{array}{c}\hfill E(Q_{UV},R(·))=\underset{Q_{U^{\prime }|V}}{min}{\left[R\left(Q_U\right)-H_Q\left(U^{\prime }\right|V)+{[f\left(Q_{UV}\right)-f\left(Q_{U^{\prime }V}\right)]}_{+}\right]}_{+}\end{array}$$

(22)

and the exponent functions:

$$\begin{array}{cc}\hfill E_{r,GLD}\left(R(·)\right)& =\underset{\{Q_{UV}:H_Q\left(U\right)\ge R\left(Q_U\right)\}}{min}\left\{D(Q_{UV}\parallel P_{UV})+E(Q_{UV},R(·))\right\},\hfill \end{array}$$

(23)

and

$$\begin{array}{c}\hfill E_{r,MAP}\left(R(·)\right)=\underset{\{Q_{UV}:H_Q\left(U\right)\ge R\left(Q_U\right)\}}{min}\left\{D(Q_{UV}\parallel P_{UV})+{[R\left(Q_U\right)-H_Q\left(U\right|V)]}_{+}\right\}.\end{array}$$

(24)

The following result is proved in Appendix A.

Theorem 1.

Let $R(·)$ be a given rate function. Then, for the SD ensemble,

• ${\mathsf{E}}_r\left(R(·)\right)=E_{r,GLD}\left(R(·)\right)$ for the GLD;
• ${\mathsf{E}}_r\left(R(·)\right)=E_{r,MAP}\left(R(·)\right)$ for the MAP and MCE decoders.

As a matter of fact, a special case of the second part of Theorem 1 has already been proved in [18] for the FR regime, while here, we prove a stronger result, according to which, the MCE decoder attains the same random binning error exponent as the MAP decoder, in the VR coding regime too. The first part of Theorem 1 is completely new; it proposes a single letter expression for the random binning error exponent, for a wide family of stochastic and deterministic decoders. Additionally, note that an analogous result to the first part of Theorem 1 has been proved in [20]. Comparing the expressions in (19) and (24), namely, the random binning error exponents of the ordinary VR and the SD VR ensembles, respectively, we find that they differ at relatively high coding rates, since these minimization problems share the same objective but (24) also has the constraint $H_Q\left(U\right)\ge R\left(Q_U\right)$. The origin of this constraint is the deterministic coding of the relatively small type classes.

Next, we provide a single-letter expression for the error exponent of the TRCs in this ensemble. We define

$$\begin{array}{cc}\hfill \gamma (R(·),Q_U,Q_V)& =\underset{\left\{\begin{array}{c}{Q}_{\tilde{U}|V}:Q_{\tilde{U}}=Q_U,\\ H_Q\left(\tilde{U}\right|V)\ge R\left(Q_{\tilde{U}}\right)\end{array}\right\}}{max}\{f\left(Q_{\tilde{U}V}\right)+H_Q\left(\tilde{U}\right|V)\}-R\left(Q_{\tilde{U}}\right)\hfill \end{array}$$

(25)

and

$$\begin{array}{cc}\hfill \Psi (R(·),Q_{U{U}^{\prime }V})& ={\left[max\{f\left(Q_{UV}\right),\gamma (R(·),Q_U,Q_V)\}-f\left(Q_{U^{\prime }V}\right)\right]}_{+}.\hfill \end{array}$$

(26)

Furthermore, define

$$\begin{array}{c}\hfill \Lambda (Q_{U{U}^{\prime }},R\left(Q_U\right))=\underset{Q_{V|U{U}^{\prime }}}{min}\left\{\Psi (R\left(Q_U\right),Q_{U{U}^{\prime }V})-H_Q\left(V\right|U,U^{\prime })-{\mathbb{E}}_Q[logP\left(V\right|U)]\right\},\end{array}$$

(27)

and the following exponent function:

$$\begin{array}{c}\hfill E_{trc,GLD}\left(R(·)\right)=\underset{\left\{\begin{array}{c}{Q}_{U{U}^{\prime }}:Q_{U^{\prime }}=Q_U,\\ H_Q\left(U\right)\ge R\left(Q_U\right)\end{array}\right\}}{min}\left\{\Lambda (Q_{U{U}^{\prime }},R\left(Q_U\right))-{\mathbb{E}}_Q[logP\left(U\right)]-H_Q(U,U^{\prime })+R\left(Q_U\right)\right\}.\end{array}$$

(28)

Then, the following theorem is proved in Appendix B.

Theorem 2.

Let $R(·)$ be a given rate function. Then, for the SD ensemble and the GLD,

$$\begin{array}{c}\hfill {\mathsf{E}}_{trc}\left(R(·)\right)=E_{trc,GLD}\left(R(·)\right).\end{array}$$

(29)

As explained before, an analogous result has already been proved in pure channel coding [1], and one can find a high degree of similarity between the expressions in (25)–(28) and the expressions in Section 3.2. While in channel coding, the coding rate is fixed, here, on the other hand, we allow the rate to depend on the type class of the source. In order to optimize the rate function, we constrain the problem by introducing the excess-rate exponent (9), which is the exponential rate of decay of the probability that the compression rate will be higher than some predefined level. A detailed discussion on optimal rate functions and optimal trade-offs between these two exponents can be found in Section 5.

The definition of the error exponent of the TRC as in (8) should not be taken for granted. The reason for that is the following. It turns out that the definition in (8) and the value of $-\frac{1}{n}log{P}_e\left({\mathcal{B}}_n\right)$ for the highly probable codes in the ensemble may not be the same, and they coincide if and only if the ensemble does not contain both zero error probability codes and positive error probability codes. For example, the FR ensemble in SW coding contains the one-to-one code (which obviously attains $P_e\left({\mathcal{B}}_n\right)=0$) as long as $R\ge log\left|\mathcal{U}\right|$, but it is definitely not a typical code, at least when ordinary random binning is considered. Hence, in this case, we conclude that $-\frac{1}{n}\mathbb{E}[log{P}_e\left({\mathcal{B}}_n\right)]=\infty$, while the value of $-\frac{1}{n}log{P}_e\left({\mathcal{B}}_n\right)$ for the highly probable codes is still finite. As for the SD code ensemble, the definition in (8) indeed provides the error exponent of the highly probable codes in the ensemble, which is explained by the following reasoning. For any given rate function such that $R\left(Q_U\right)<H_Q\left(U\right)$ for at least one type class, all the type classes with $R\left(Q_U\right)<H_Q\left(U\right)$ are encoded by random binning; thus, all the codes in the ensemble have a strictly positive error probability, which implies that the value of $-\frac{1}{n}log{P}_e\left({\mathcal{B}}_n\right)$ concentrates around the error exponent of the TRC, as defined in (8).

The proof of Theorem 2 follows exactly the same lines as the proof of ([1] (Theorem 1)), except for one main modification: when we introduce the type class enumerator $N\left(Q_{U{U}^{\prime }}\right)$ (see below) and sum over joint types, the summation set becomes $\{Q_{U{U}^{\prime }}:Q_{U^{\prime }}=Q_U,H_Q\left(U\right)\ge R\left(Q_U\right)\}$, where the constraint $H_Q\left(U\right)\ge R\left(Q_U\right)$ is due to the indicator function in (6). Afterwards, the analysis of the type class enumerator yields the constraint $H_Q(U,U^{\prime })\ge R\left(Q_U\right)$, which becomes redundant and thus omitted. This constraint is analogous to the constraint $I_Q(X;X^{\prime })\le 2R$ in the minimization of (13). The origin of $H_Q(U,U^{\prime })\ge R\left(Q_U\right)$ is the following. Define

$$\begin{array}{c}\hfill N\left(Q_{U{U}^{\prime }}\right)=\sum _{(\mathit{u},{\mathit{u}}^{\prime })\in \mathcal{T}\left(Q_{U{U}^{\prime }}\right)}𝟙\left\{\mathcal{B}\left({\mathit{u}}^{\prime }\right)=\mathcal{B}\left(\mathit{u}\right)\right\},\end{array}$$

(30)

which enumerate pairs of source sequences. Then, one of the main steps in the proof of Theorem 2 is deriving the high probability value of $N\left(Q_{U{U}^{\prime }}\right)$, which is 0 if $H_Q(U,U^{\prime })<R\left(Q_U\right)$ (a relatively small set of source pair and relatively large number of bins) and $e^{n[H_Q(U,U^{\prime })-R\left(Q_U\right)]}$ for $H_Q(U,U^{\prime })\ge R\left(Q_U\right)$ (a large set of source sequence pair and a small number of bins). One should note that the analysis of $N\left(Q_{U{U}^{\prime }}\right)$ is not trivial, since it is not a binomial random variable; i.e., the enumerator $N\left(Q_{U{U}^{\prime }}\right)$ is given by the sum of dependent binary random variables. For a sum N of independent binary random variables, ordinary tools from large deviation theory (e.g., the Chernoff bound) can be invoked for assessing the exponential moments $\mathbb{E}\left[N^s\right]$, $s\ge 0$, or the large deviation rate function of $\mathbb{P}\{N\ge e^{n\sigma }\}$, $\sigma \in \mathrm{I}\mathrm{R}$. For sums of dependent binary random variables, such as $N\left(Q_{U{U}^{\prime }}\right)$ in the current problem, this can no longer be done by the same techniques, and it requires more advanced tools (see, e.g., [1,4,5,6]).

It is possible to compare (23) and (28) analytically in the special cases of the matched or the mismatched likelihood decoders and the MCE decoder. In the following theorem, the choice $f\left(Q_{UV}\right)=\beta {\mathbb{E}}_Q[log\tilde{P}(U,V)]$, where $\tilde{P}(U,V)$ is a possibly different source distribution than $P(U,V)$, corresponds to a family of stochastic mismatched decoders. We have the following result, the proof of which is given in Appendix D.

Theorem 3.

Consider the SD ensemble and a given rate function $R(·)$. Then,

• For a GLD with the decoding metric $f\left(Q\right)=\beta {\mathbb{E}}_Q[log\tilde{P}(U,V)]$, for a given $\beta >0$,

  $$\begin{array}{c}\hfill E_{trc,GLD}\left(R(·)\right)=E_{r,GLD}\left(R(·)\right).\end{array}$$

  (31)
• For the MCE decoder,

  $$\begin{array}{c}\hfill E_{trc,MCE}\left(R(·)\right)=E_{r,MCE}\left(R(·)\right).\end{array}$$

  (32)

This result is quite surprising at first glance, since one expects the error exponent of the TRC to be strictly better than the random binning error exponent, as in ordinary channel coding at relatively low coding rates [1,2]. This phenomenon is due to the fact that part of the source type classes are deterministically partitioned into bins in a one-to-one fashion, and hence do not affect the probability of error (notice that the constraint $H_Q\left(U\right)\ge R\left(Q_U\right)$ appears in both the random binning and the TRC exponents, while in the latter, it makes the original constraint $H_Q(U,U^{\prime })\ge R\left(Q_U\right)$ redundant). In the cases of FR or ordinary VR binning, these relatively “thin” type classes dominated the error probability at relatively high binning rates, but now, by encoding them deterministically into the bins; other mechanisms dominate the error event, such as the channel noise (between $\mathit{U}$ and $\mathit{V}$) or the random binning of the type classes with $H_Q\left(U\right)\ge R\left(Q_U\right)$. The result of the second part of Theorem 3 is also nontrivial, since it establishes an equality between the error exponent of the TRC and the random binning error exponent, but now for a universal decoder.

Concerning universal decoding, it is already known [21] (Exercise 3.1.6), [13] that the random binning error exponents under optimal MAP decoding in both the FR and VR codes, given by (14) and (19), respectively, are also attained by the MCE decoder. Furthermore, a similar result for the SD ensemble has been proved here in Theorem 1. The natural question that arises is whether the error exponent of the TRC is also universally attainable. The following result, which is proved in Appendix E, provides a positive answer to this question.

Theorem 4.

Consider the SD ensemble and a given rate function $R(·)$. Then, the error exponents of the TRC under the MAP, the MCE, and the SCE decoders are all equal; i.e.,

$$\begin{array}{c}\hfill E_{trc,MAP}\left(R(·)\right)=E_{trc,MCE}\left(R(·)\right)=E_{trc,SCE}\left(R(·)\right).\end{array}$$

(33)

Theorem 4 asserts that the error exponent of the typical random SD code is not affected if the optimal MAP decoder is replaced by a certain universal decoder, which must not even be deterministic. While the left hand equality in (33) follows immediately from the results of Theorems 1 and 3, the right hand equality in (33) is far less trivial, since the SCE decoder is both universal and stochastic, and hence, its TRC exponent is expected to be inferior w.r.t. the TRC exponent under MAP decoding, but nevertheless, they turn out to be equal. Comparing to channel coding, it has been recently proved in [22] that the error exponent of the typical random fixed composition code (given in (13)) is the same for the ML and the maximum mutual information decoder, but on the other hand, numerical evidence shows that a GLD which is based on an empirical mutual information metric attains a strictly lower exponent.

5. Optimal Trade-off Functions

In this section, we study the optimal trade-off between the threshold $\Delta$, the error exponent of the TRC, and the excess-rate exponent. Since both exponents depend on the rate function, we wish to characterize rate functions that are optimal w.r.t. this trade-off. Since a single-letter characterization of the error exponent of the TRC has already been given in (28), we next provide a single-letter expression for the excess-rate exponent. Define the following exponent function:

$$\begin{array}{c}\hfill E_{er}(R(·),\Delta )=\underset{\{Q_{UV}:R\left(Q_U\right)\ge H_Q\left(U\right|V)+\Delta \}}{min}D(Q_{UV}\parallel P_{UV}).\end{array}$$

(34)

Then, we have the following.

Proposition 1.

Fix $\Delta >0$ and let $R(·)$ be any rate function. Then,

$$\begin{array}{c}\hfill {\mathsf{E}}_{er}(\mathcal{B},R(·),\Delta )=E_{er}(R(·),\Delta ).\end{array}$$

(35)

Proof. 

The excess-rate probability is given by:

$$\begin{array}{cc}\hfill & \mathbb{P}\{R\left({\widehat{P}}_{\mathit{U}}\right)\ge {\widehat{H}}_{\mathit{U}\mathit{V}}\left(U\right|V)+\Delta \}\hfill \end{array}$$

$$\begin{array}{cc}\hfill & =\sum _{Q_{UV}}𝟙\{R\left(Q_U\right)\ge H_Q\left(U\right|V)+\Delta \}·\mathbb{P}\{(\mathit{U},\mathit{V})\in \mathcal{T}\left(Q_{UV}\right)\}\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill & \doteq \sum _{\{Q_{UV}:R\left(Q_U\right)\ge H_Q\left(U\right|V)+\Delta \}}exp\left\{-nD(Q_{UV}\parallel P_{UV})\right\}\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill & \doteq exp\left\{-n·\underset{\{Q_{UV}:R\left(Q_U\right)\ge H_Q\left(U\right|V)+\Delta \}}{min}D(Q_{UV}\parallel P_{UV})\right\},\hfill \end{array}$$

(38)

which proves the desired result. □

Since Proposition 1 is proved by the method of types [21], we conclude that the excess-rate event is dominated by one specific type class $\mathcal{T}\left(Q_{UV}\right)$, whose respective rate $R\left(Q_U\right)$ has been chosen too large w.r.t. the value of $H_Q\left(U\right|V)+\Delta$. One extreme case is when the rate function is given by $H_Q\left(U\right)$, which obviously provides a one-to-one mapping, since the size of each $\mathcal{T}\left(Q_U\right)$ is upper-bounded by $e^{n{H}_Q\left(U\right)}$. In this case, the probability of error is zero, while the excess-rate probability is one, at least when $\Delta$ is not too large. In Section 5.2, we prove that the optimal rate function is indeed upper-bounded by $H_Q\left(U\right)$, but can also be strictly smaller, especially when the requirement on the error exponent is not too stringent.

One way to explore the trade-off between the error exponent of the TRC and the excess-rate exponent, which will be presented in Section 5.1, is to require the excess-rate exponent to exceed some value ${\mathsf{E}}_r>0$, then solve $E_{er}(R(·),\Delta )\ge {\mathsf{E}}_r$ for an optimal rate function $R^{*}\left(Q_U\right)$, and then to substitute this optimal rate function back into the error exponents in (24) and (28) to give expressions for the optimal trade-off function $E_e({\mathsf{E}}_r,\Delta )$. In Section 5.2, we present an alternative option to characterize this trade-off, which is to require the error exponent of the TRC to exceed some value ${\mathsf{E}}_e>0$, to solve $E_e\left(R(·)\right)\ge {\mathsf{E}}_e$ in order to extract an optimal rate function, and then to substitute it back into the excess-rate exponent in (34) to provide an expression for the optimal trade-off function $E_{er}({\mathsf{E}}_e,\Delta )$.

5.1. Constrained Excess-Rate Exponent

Relying on the exponent function in (34), the following theorem proposes a rate function, whose optimality is proved in Appendix F.

Theorem 5.

Let ${\mathsf{E}}_r>0$ be fixed. Then, the constraint $E_{er}(R(·),\Delta )\ge {\mathsf{E}}_r$ implies that

$$\begin{array}{c}\hfill R\left(Q_U\right)\le J(Q_U,{\mathsf{E}}_r,\Delta )\stackrel{▵}{=}\underset{\{Q_{V|U}:D(Q_{UV}\parallel P_{UV})\le {\mathsf{E}}_r\}}{min}\left\{H_Q\left(U\right|V)+\Delta \right\}.\end{array}$$

(39)

This means that we have a dichotomy between two kinds of source types. Each type class that is associated with an empirical distribution that is relatively close to the source distribution, i.e., when $D(Q_{UV}\parallel P_{UV})\le {\mathsf{E}}_r$ for some $Q_{V|U}$, is partitioned into $e^{nJ(Q_U,{\mathsf{E}}_r,\Delta )}$ bins, and the rest of the type classes, those that are relatively distant from $P_U$, are encoded by a one-to-one mapping. Two extreme cases should be considered here. First, when ${\mathsf{E}}_r$ is relatively small, then only the types closest to $P_U$ are encoded with a rate approximately $H_P\left(U\right|V)+\Delta$, which can be made arbitrarily close to the SW limit [8], and each a–typical source sequence is allocated with $n·{log}_2\left|\mathcal{U}\right|$ bits. This coding scheme is the one related to VR coding with an average rate constraint, like the one discussed in [12]. Second, when ${\mathsf{E}}_r$ is extremely large, then each type class is encoded to $exp\{n\Delta \}$ bins, which is equivalent to FR coding.

Following the first part of Theorem 3, let us denote the error exponent of the TRC under MAP decoding by $E_e(·)$. Upon substituting the optimal rate function of Theorem 5 back into (24) and (28) and using the fact that $E_e(·)$ is monotonically increasing, we find that the optimal trade-off function for the typical random SD code is given by

$$\begin{array}{c}\hfill E_e({\mathsf{E}}_r,\Delta )=\underset{\left\{Q_{UV}:H_Q\left(U\right)\ge J\left(Q_U\right)\right\}}{min}\left\{D(Q_{UV}\parallel P_{UV})+{[J\left(Q_U\right)-H_Q\left(U\right|V)]}_{+}\right\},\end{array}$$

(40)

or, alternatively,

$$\begin{array}{cc}\hfill E_e({\mathsf{E}}_r,\Delta )& =\underset{\left\{\begin{array}{c}{Q}_{U{U}^{\prime }}:Q_{U^{\prime }}=Q_U,\\ H_Q\left(U\right)\ge J\left(Q_U\right)\end{array}\right\}}{min}\left\{\Lambda (Q_{U{U}^{\prime }},J\left(Q_U\right))-{\mathbb{E}}_Q[logP\left(U\right)]-H_Q(U,U^{\prime })+J\left(Q_U\right)\right\},\hfill \end{array}$$

(41)

where $J\left(Q_U\right)=J(Q_U,{\mathsf{E}}_r,\Delta )$ is given in (39). The dependence of $E_e({\mathsf{E}}_r,\Delta )$ on ${\mathsf{E}}_r$ is as follows. Let $Q_{U{U}^{\prime }}^{*}(\Delta )$ and $Q_{V|U}^{*}$ be the respective minimizers of the problems which are similar to (39) and (41), except that the constraint $D(Q_{UV}\parallel P_{UV})\le {\mathsf{E}}_r$ is removed from (39). Furthermore, let $Q_U^{*}(\Delta )$ be the marginal distribution of $Q_{U{U}^{\prime }}^{*}(\Delta )$. Now, when ${\mathsf{E}}_r$ is sufficiently large, i.e., when ${\mathsf{E}}_r\ge D(Q_U^{*}(\Delta )\times Q_{V|U}^{*}\parallel P_{UV})$, $E_e({\mathsf{E}}_r,\Delta )$ reaches a plateau and is the lowest possible. It follows from the fact that the stringent requirement on the excess-rate forces the encoder to encode each type class $Q_U$ to its target rate $\Delta$, thus all of them affect the error event. Otherwise, when ${\mathsf{E}}_r<D(Q_U^{*}(\Delta )\times Q_{V|U}^{*}\parallel P_{UV})$, the constraint $D(Q_{UV}\parallel P_{UV})\le {\mathsf{E}}_r$ is active and $E_e({\mathsf{E}}_r,\Delta )$ is a monotonically nonincreasing function of ${\mathsf{E}}_r$. The reason for that is the fact that as ${\mathsf{E}}_r$ decreases, more and more type classes are encoded with $n·{log}_2\left|\mathcal{U}\right|$ bits, and hence do not contribute to the error event. When ${\mathsf{E}}_r=0$, necessarily $Q_U=P_U$, only the typical set is encoded, and $E_e(0,\Delta )$ is the highest possible. In this case, $J\left(Q_U\right)=H_P\left(U\right|V)+\Delta$ and the constraint set in (41) becomes empty when $\Delta >I_P(U;V)$, and then $E_e(0,\Delta )=\infty$.

5.2. Constrained Error Exponent

Based on (24), the following theorem proposes a rate function, whose optimality is proved in Appendix G.

Theorem 6.

Let ${\mathsf{E}}_e>0$ be fixed. Then, the constraint $E_e\left(R(·)\right)\ge {\mathsf{E}}_e$ implies that

$$\begin{array}{c}\hfill R\left(Q_U\right)\ge \Omega (Q_U,{\mathsf{E}}_e)\stackrel{▵}{=}min\left\{H_Q\left(U\right),G(Q_U,{\mathsf{E}}_e)\right\},\end{array}$$

(42)

where,

$$\begin{array}{c}\hfill G(Q_U,{\mathsf{E}}_e)=\underset{\{Q_{V|U}:D(Q_{UV}\parallel P_{UV})\le {\mathsf{E}}_e\}}{max}\{H_Q\left(U\right|V)+{\mathsf{E}}_e-D(Q_{UV}\parallel P_{UV})\}.\end{array}$$

(43)

The dependence of $G(Q_U,{\mathsf{E}}_e)$ on ${\mathsf{E}}_e$ is as follows. For any given $Q_U$, let ${\tilde{Q}}_{V|U}$ be the minimizer of $D(Q_{UV}\parallel P_{UV})$. Then, as long as ${\mathsf{E}}_e<D(Q_U\times {\tilde{Q}}_{V|U}\parallel P_{UV})$, the constraint set in (43) is empty, and $R\left(Q_U\right)$ can vanish, which practically means that in this range, the entire type class $\mathcal{T}\left(Q_U\right)$ can be totally ignored, while still achieving $P_e\approx e^{-n{\mathsf{E}}_e}$. Only for the unique type $Q_U=P_U$, $G(P_U,{\mathsf{E}}_e)>0$ for all ${\mathsf{E}}_e\ge 0$, and specifically, we find that $G(P_U,0)=H_P\left(U\right|V)$. Furthermore, let $Q_{V|U}^{*}$ be the maximizer in the unconstrained problem

$$\begin{array}{c}\hfill \underset{Q_{V|U}}{max}\left\{H_Q\left(U\right|V)-D(Q_{UV}\parallel P_{UV})\right\}.\end{array}$$

(44)

Then, as long as ${\mathsf{E}}_e\in [D(Q_U\times {\tilde{Q}}_{V|U}\parallel P_{UV}),D(Q_U\times Q_{V|U}^{*}\parallel P_{UV}))$, $G(Q_U,{\mathsf{E}}_e)$ is a monotonically nondecreasing function of ${\mathsf{E}}_e$. When ${\mathsf{E}}_e\ge D(Q_U\times Q_{V|U}^{*}\parallel P_{UV})$, the maximization in (43) reaches its unconstrained optimum, and $G(Q_U,{\mathsf{E}}_e)$ increases without bound in an affine fashion as ${\mathsf{E}}_e+H_{Q^{*}}\left(U\right|V)-D(Q_U\times Q_{V|U}^{*}\parallel P_{UV})$. As can be seen in (42), $\Omega (Q_U,{\mathsf{E}}_e)$ finally reaches a plateau at the level of $H_Q\left(U\right)$.

Upon substituting $\Omega (Q_U,{\mathsf{E}}_e)$ back into (34) and using the fact that $E_{er}(·,\Delta )$ is monotonically nonincreasing, we find that the trade-off function is given by

$$\begin{array}{c}\hfill E_{er}({\mathsf{E}}_e,\Delta )=\underset{\{Q_{UV}:\Omega (Q_U,{\mathsf{E}}_e)\ge H_Q\left(U\right|V)+\Delta \}}{min}D(Q_{UV}\parallel P_{UV}).\end{array}$$

(45)

Since $\Omega (Q_U,{\mathsf{E}}_e)$ is monotonically nondecreasing in ${\mathsf{E}}_e$ for every $Q_U$, $E_{er}({\mathsf{E}}_e,\Delta )$ is monotonically nonincreasing in ${\mathsf{E}}_e$, which is not very surprising. The dependence of $E_{er}({\mathsf{E}}_e,\Delta )$ on ${\mathsf{E}}_e$ and $\Delta$ is as follows. At ${\mathsf{E}}_e=0$, notice that $\Omega (Q_U,0)=-\infty$ for any $Q_U\ne P_U$ while $\Omega (P_U,0)=H_P\left(U\right|V)$. Thus, $E_{er}(0,\Delta )=0$ as long as $\Delta =0$, and it follows from the monotonicity that $E_{er}({\mathsf{E}}_e,0)=0$ everywhere. Otherwise, if $\Delta >0$, $\{Q_{UV}:\Omega (Q_U,{\mathsf{E}}_e)\ge H_Q\left(U\right|V)+\Delta \}$ is empty as long as ${\mathsf{E}}_e<{\mathsf{E}}_e^{*}(\Delta )$, where an expression for ${\mathsf{E}}_e^{*}(\Delta )$ can be found by solving

$$\begin{array}{c}\hfill \underset{Q_{UV}}{max}\{\Omega (Q_U,{\mathsf{E}}_e)-H_Q\left(U\right|V)\}\le \Delta ,\end{array}$$

(46)

and then $E_{er}({\mathsf{E}}_e,\Delta )=\infty$ in this range. In the other extreme case of a very large ${\mathsf{E}}_e$, $\Omega (Q_U,{\mathsf{E}}_e)$ reaches a plateau at a level of $H_Q\left(U\right)$. Then, if $\Delta \le H_P\left(U\right)-H_P\left(U\right|V)=I_P(U;V)$, $E_{er}({\mathsf{E}}_e,\Delta )$ reaches zero for a sufficiently large ${\mathsf{E}}_e$. Else, if $\Delta >I_P(U;V)$, $E_{er}({\mathsf{E}}_e,\Delta )$ reaches a strictly positive plateau, given by

$$\begin{array}{c}\hfill \underset{\{Q_{UV}:I_Q(U;V)\ge \Delta \}}{min}D(Q_{UV}\parallel P_{UV}),\end{array}$$

(47)

which is a monotonically nondecreasing function of $\Delta$. Particularly, it means that in this range, the typical random SD code attains both an exponentially vanishing excess-rate probability and $P_e\approx 0$.

It is interesting to relate this to the expurgated bound of the FR code in the SW model, which is given by (15). Comparing $E_{ex}^{fr}\left(R\right)$ and $E_e(\infty ,\Delta )$ analytically is rather difficult. Thus, we examined these two exponent functions numerically. Consider the case of a double binary source with alphabets $\mathcal{U}=\mathcal{V}=\{0,1\}$, and joint probabilities given by $P_{UV}(0,0)=0.75$, $P_{UV}(0,1)=0.1$, $P_{UV}(1,0)=0$, and $P_{UV}(1,1)=0.15$. We already mentioned before, that in the special case of ${\mathsf{E}}_r=\infty$, the rate function is given by the threshold $\Delta$, hence we choose $\Delta =R$ in order to have a fair comparison. Graphs of the functions $E_{ex}^{fr}\left(R\right)$ and $E_e(\infty ,R)$ are presented in Figure 1.

Figure 1. Graphs of the functions $E_{ex}^{fr}\left(R\right)$ and $E_e(\infty ,R)$.

As can be seen in Figure 1, both $E_{ex}^{fr}\left(R\right)$ and $E_e(\infty ,R)$ tend to infinity as R tends to $log2\approx 0.693$. For relatively high binning rates, $E_{ex}^{fr}\left(R\right)$ is strictly higher than $E_e(\infty ,R)$, which can be explained in the following way: Referring to the analogy between SW coding and channel coding, one can think of each bin as containing a channel code. In general, a channel code behaves well if it does not contain pairs of relatively “close” codewords. Since we randomly assign the source vectors into the bins (even if the populations of the bins are totally equal, which can be attained by randomly partitioning each type class into $exp\left\{nR\right\}$ subsets), it is reasonable to assume that some bins will contain relatively bad codebooks. On the other hand, in the expurgated SW code [11], each type class $\mathcal{T}\left(Q_U\right)$ is partitioned into $exp\left\{nR\right\}$ “balanced” subsets in some sense (referring to the enumerators $N\left(Q_{U{U}^{\prime }}\right)$ in (30), they are equally populated in all of the bins), such that the codebooks contained in the bins have approximately equal error probabilities. Moreover, we conclude from (15) that each bin contains a codebook with a quality of an expurgated channel code. This code is certainly better than the TRCs in the SD ensemble.

In channel coding, it is known [23] that the random Gilbert–Varshamov ensemble has an exact random coding error exponent which is as high as the maximum between (16) and (17). In SW source coding, on the other hand, it seems to be a more challenging problem to define an ensemble, such that the error exponent of its TRCs is as high as $E_{ex}^{fr}\left(R\right)$ of (15). Since the gap between $E_{ex}^{fr}\left(R\right)$ and $E_e(\infty ,R)$ is not necessarily very significant, as can be seen in Figure 1, we conclude that the SD ensemble may be more attractive because the amount of computations needed for drawing a code from it are much lower than the amount of computations required for having an expurgated SW code. In addition, it is important to note that the probability of drawing a SD code with an exponent much lower than $E_e(\infty ,R)$ decays exponentially fast, in analogy to the result in pure channel coding [7].