Rate and Nearly-Lossless State over the Gilbert–Elliott Channel

Abstract

The capacity of the Gilbert–Elliott channel is calculated for a setting in which the state sequence is revealed to the encoder and is, along with the transmitted message, to be conveyed to the receiver with a vanishing symbol error rate. Said capacity does not depend on whether the state sequence is provided to the encoder strictly causally, causally, or noncausally. It can be achieved using a Block-Markov coding scheme with backward decoding.

1. Introduction

The Gilbert–Elliott channel [1,2,3,4] is a binary-input, binary-output finite-state channel [5,6,7,8], which was proposed as a simple model for digital communications with bursty errors. Its state sequence is a stationary two-state Markov process whose Time-i state $S_i$ takes values in the set $\mathcal{S}=\{0,1\}$. When the Time-i state $S_i$ is 0, the Time-i channel output $Y_i$ is the result of feeding the Time-i input $x_i$ through a Binary Symmetric Channel (BSC) of crossover probability ${ϵ}_0$; when $S_i$ is 1, the output $Y_i$ is the result of feeding $x_i$ through a BSC of crossover probability ${ϵ}_1$. Alternatively, we can describe the output sequence as being the componentwise mod-2 addition of the input sequence with a noise sequence $\left\{Z_i\right\}$, where the latter is a binary two-state Hidden Markov process.

The capacity of the Gilbert–Elliott channel is achieved by having the input sequence comprise Independent and Identically Distributed (IID) random bits. It is given, in bits, by $1-H\left(\left\{Z_i\right\}\right)$, where $H\left(\left\{Z_i\right\}\right)$ denotes the entropy rate of the noise $\left\{Z_i\right\}$ [3]. If the state sequence is revealed to the decoder, then capacity is still achieved by the above input distribution, but it now equals the weighted average of the capacities of the BSCs, namely, ${\pi }_0(1-H_{\mathrm{b}}\left({ϵ}_0\right))+{\pi }_1(1-H_{\mathrm{b}}\left({ϵ}_1\right))$. Here, ${\pi }_1=1-{\pi }_0$ denotes the probability that $S_i$ is 1, and $H_{\mathrm{b}}(·)$ is the binary entropy function. Throughout this paper, logarithms are to base 2, and information is measured in bits.

Here, we study the case where the state sequence is revealed not to the decoder but to the encoder. We do not, however, seek the Shannon capacity but rather the “rate-and-nearly-lossless-state” capacity, where the decoder wishes to recover not only the transmitted message but also the state sequence. We thus require that the state sequence be conveyed to the decoder with a vanishing symbol error rate. This requirement is weaker than the requirement that the probability of the receiver correctly recovering the entire state sequence tends to one. Neither do we consider a general (nonzero) distortion constraint, as studied for memoryless channels in [9,10].

We solve for this capacity and show that it does not depend on whether the state sequence is provided to the encoder strictly causally, causally, or noncausally. Moreover, it can be achieved using a Block-Markov coding scheme with backward decoding.

2. The Gilbert–Elliott Channel

The Gilbert–Elliott channel has binary input, output, and state alphabets: $\mathcal{X}=\mathcal{Y}=\mathcal{S}=\{0,1\}$. The evolution of the state is unaffected by the channel inputs: irrespective of the channel inputs, it forms a stationary time-homogeneous Markov chain of kernel

$$Pr[S_i=s_i|S^{i-1}=s^{i-1}]=Pr[S_i=s_i|S_{i-1}=s_{i-1}]$$

(1)

and stationary distribution

$$P_S=\left({\pi }_0,{\pi }_1\right)$$

(2)

by which we mean that

$$P_S\left(1\right)=1-P_S\left(0\right)={\pi }_1=1-{\pi }_0.$$

(3)

Above and throughout, we use $A^i$ to denote $A_1,\dots ,A_i$, and we use $A_j^i$ to denote $A_j,\dots ,A_i$. We denote the entropy rate of the state sequence $H\left(\left\{S_i\right\}\right)$. It is given explicitly by [11]

$$H\left(\left\{S_i\right\}\right)=H\left(S_2\right|S_1).$$

(4)

Given $S=0$, the channel from X to Y is a BSC with crossover probability ${ϵ}_0$, whereas, given $S=1$, it is a BSC with crossover probability ${ϵ}_1$. Here, ${ϵ}_0,{ϵ}_1\in [0,1]$ are arbitrary known constants. Thus, if we define

$$W(1-x|x,0)=1-W\left(x\right|x,0)={ϵ}_0$$

(5)

and

$$W(1-x|x,1)=1-W\left(x\right|x,1)={ϵ}_1,$$

(6)

then we can express the behavior of the channel given the state as

$$Pr[Y_i=y_i|X^i=x^i,Y^{i-1}=y^{i-1},S^i=s^i]=W\left(y_i\right|x_i,s_i).$$

(7)

3. The Rate-and-Nearly-Lossless-State Capacity

When discussing rate-R blocklength-n communications over the Gilbert–Elliott channel, we consider the message set $\mathcal{M}=\{1,\dots ,2^{nR}\}$ with $2^{nR}$ messages, one of which is to be conveyed to the receiver. The latter observes the output sequence $Y^n$ and attempts to recover the transmitted message m and the state sequence $S^n$. The decoder is thus specified by a function

$$\varphi :{\mathcal{Y}}^n\to \mathcal{M}\times {\mathcal{S}}^n,y^n\mapsto (\widehat{m},{\widehat{s}}^n).$$

(8)

As to the encoder, its structure depends on the manner in which the state information is revealed to it. In the strictly causal setting, the Time-i channel input may depend, not only on the transmitted message, but also on the past states; it is therefore denoted $X_i(m,S^{i-1})$. The encoder is thus specified by n functions

$$f_i:\mathcal{M}\times {\mathcal{S}}^{i-1}\to \mathcal{X},i=1,\dots ,n,$$

(9)

with $X_i(m,S^{i-1})$ being $f_i(m,S^{i-1})$. In the causal case, the Time-i channel input is denoted $X_i(m,S^i)$, and the encoder is specified by n functions

$$f_i:\mathcal{M}\times {\mathcal{S}}^i\to \mathcal{X},i=1,\dots ,n,$$

(10)

with $X_i(m,S^i)$ being $f_i(m,S^i)$. Finally, in the noncausal case, the Time-i channel input is denoted $X_i(m,S^n)$, and the encoder is specified by one function

$$f:\mathcal{M}\times {\mathcal{S}}^n\to {\mathcal{X}}^n,$$

(11)

with $X_i(m,S^n)$ being the i-th component of $f(m,S^n)$.

We refer to an encoder/decoder pair as a “coding scheme”. The probability of error associated with a given coding scheme and a given message m is $Pr[\widehat{M}\ne m]$ calculated when the scheme’s encoder is used to transmit Message m, and the scheme’s decoder is used by the receiver. The average probability of error associated with a coding scheme is the arithmetic average of the probabilities of error associated with the different messages. It can be expressed as

$$P_{\mathrm{e}}\triangleq Pr[\widehat{M}\ne M]$$

(12)

when the transmitted message M is drawn equiprobably from $\mathcal{M}$.

The symbol error rate, or (expected) Hamming distortion, in reconstructing the state sequence is

$$\beta \triangleq {\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}Pr[{\widehat{S}}_i\ne S_i]$$

(13)

again computed when the transmitted message M is drawn equiprobably from $\mathcal{M}$.

In all cases, we say that a rate R is achievable if there exists a sequence of coding schemes indexed by the blocklength for which $P_{\mathrm{e}}$ and $\beta$ both tend to zero. The capacity is defined as the supremum of the achievable rates, with the understanding that, if no positive rate is achievable, then capacity is zero.

Our main result is the following theorem:

Theorem 1.

Irrespective of whether the state information is provided to the encoder strictly causally, causally, or noncausally, if

$$1-\left({\pi }_0{H}_{\mathrm{b}}\left({ϵ}_0\right)+{\pi }_1{H}_{\mathrm{b}}\left({ϵ}_1\right)\right)-H\left(\left\{S_i\right\}\right)$$

(14)

is positive, then it equals the rate-and-nearly-lossless-state capacity of the Gilbert–Elliott channel; else, said capacity is zero.

The proof is provided in the next section. Expression (14) can be interpreted as the result of subtracting the optimal average description length of the state sequence from the capacity of the channel when the receiver is cognizant of the state.

4. Proof of Theorem 1

4.1. Converse

We prove the converse part of the theorem under the assumption that the state sequence is provided to the encoder noncausally. Let M be uniform over $\mathcal{M}$. We have

$$\begin{array}{}\mathrm{(15)}\hfill & \hfill H\left(S^n\right|M,Y^n)& \le \hfill & H\left(S^n\right|{\widehat{S}}^n)\hfill \mathrm{(16)}\hfill & & \le \hfill & {\displaystyle \sum _{i=1}^n}H\left(S_i\right|{\widehat{S}}_i)\hfill \mathrm{(17)}\hfill & & \le \hfill & {\displaystyle \sum _{i=1}^n}{H}_{\mathrm{b}}\left(Pr[\widehat{S_i}\ne S_i]\right)\hfill \mathrm{(18)}\hfill & & \le \hfill & n{H}_{\mathrm{b}}\left(\beta \right).\hfill \end{array}$$

Here, (15) holds because ${\widehat{S}}^n$ is a function of $Y^n$ and hence also of $(M,Y^n)$; (17) because ${\widehat{S}}_i$ is binary and because conditioning reduces entropy; and (18) because $H_{\mathrm{b}}(·)$ is concave and by recalling the definition of $\beta$ in (13). Note that $H_{\mathrm{b}}\left(\beta \right)$ tends to zero when $\beta$ tends to zero.

Since the decoder needs to recover M with high probability, Fano’s inequality implies the existence of some sequence $\left\{{ϵ}_n\right\}$ tending to zero with the blocklength such that

$$n(R-{ϵ}_n)\le I(M;Y^n).$$

(19)

From these two inequalities, we obtain

$$\begin{array}{}\mathrm{(20)}& \hfill n(R-{ϵ}_n-H_{\mathrm{b}}\left(\beta \right))& \le \hfill & I(M;Y^n)-H\left(S^n\right|M,Y^n)\hfill \mathrm{(21)}& & =\hfill & I(M;Y^n,S^n)-I(M;S^n|Y^n)-H\left(S^n\right|M,Y^n)\hfill \mathrm{(22)}& & =\hfill & I(M;Y^n,S^n)-H\left(S^n\right|Y^n)\hfill \mathrm{(23)}& & =\hfill & I(M;Y^n|S^n)-H\left(S^n\right|Y^n)\hfill \mathrm{(24)}& & =\hfill & H\left(Y^n\right|S^n)-H\left(S^n\right|Y^n)-H\left(Y^n\right|S^n,M)\hfill \mathrm{(25)}& & =\hfill & \left(H\left(Y^n\right)-I(Y^n;S^n)\right)-\left(H\left(S^n\right)-I(Y^n;S^n)\right)-H\left(Y^n\right|S^n,M)\hfill \mathrm{(26)}& & =\hfill & H\left(Y^n\right)-H\left(S^n\right)-H\left(Y^n\right|S^n,M).\hfill \end{array}$$

Note that, above, we only used the chain rule and the fact that $M\perp \perp S^n$ (i.e., that M and $S^n$ are independent). The three terms in the last line can be bounded or simplified as follows:

$$\begin{array}{}\mathrm{(27)}& \hfill H\left(Y^n\right)& \le \hfill & n\hfill \mathrm{(28)}& \hfill H\left(S^n\right)& =\hfill & H\left(S_1\right)+(n-1)H\left(\left\{S_i\right\}\right)\hfill \mathrm{(29)}& \hfill H\left(Y^n\right|S^n,M)& =\hfill & H\left(Y^n\right|S^n,M,X^n)\hfill \mathrm{(30)}& & =\hfill & H\left(Y^n\right|S^n,X^n)\hfill \mathrm{(31)}& & =\hfill & {\displaystyle \sum _{i=1}^n}H\left(Y_i\right|S_i,X_i)\hfill \mathrm{(32)}& & =\hfill & n\left({\pi }_0{H}_{\mathrm{b}}\left({ϵ}_0\right)+{\pi }_1{H}_{\mathrm{b}}\left({ϵ}_1\right)\right).\hfill \end{array}$$

The converse now follows from (26), (27), (28), and (32).

4.2. Direct Part

For the direct part of the theorem, we assume that the state sequence is revealed to the encoder strictly causally and consider a Block-Markov coding scheme with backward decoding. Consider b blocks, each of k channel uses. Label the length-k typical state sequences $1,\dots ,2^{k(H\left(\left\{S_i\right\}\right)+ϵ)}$. For each block, randomly generate a codebook with $2^{k\tilde{R}}$ independent codewords (with $\tilde{R}$ to be specified later), each having k independent Bernoulli$(1/2)$ components. In the first block, use the entire codebook to send a message of $k\tilde{R}$ bits. As to the transmission in Block i for $i\ge 2$, first check whether or not the state sequence in the previous block, namely, ${\left(s^k\right)}_{i-1}$, was typical. If it was, set ${\ell }_{i-1}$ to be the label that is assigned to it; else, set ${\ell }_{i-1}=1$. Use the generated codebook to send both ${\ell }_{i-1}$ and $m_i$, the latter consisting of $k\left(\tilde{R}-(H\left(\left\{S_i\right\}\right)+ϵ)\right)$ information bits, so that the total number of bits matches the size of the codebook.

After completing the transmission in all b blocks, add one extra block to transmit ${\ell }_b$. To this end, the transmitter—ignoring any information about the states in this extra block—uses the Gilbert–Elliott channel to send ${\ell }_b$ (which we view as a message) and nothing else. The length of the extra block may be larger than k, but—as long as the Shannon capacity of our channel is positive—this will not affect the overall rate when we choose b to be very large. We will address this caveat shortly. However, first, we discuss the decoding.

To decode, we begin with the last bock to decode ${\ell }_b$ and thus recover ${\left(s^k\right)}_b$. This task will be accomplished successfully provided that the last extra block is sufficiently long. With the state information ${\left(s^k\right)}_b$ at hand, we then decode both ${\ell }_{b-1}$ and $m_b$ from Block b. Both can be decoded correctly with high probability (as k grows large) provided that

$$\tilde{R}<I(X_i;Y_i|S_i)=1-\left({\pi }_0{H}_{\mathrm{b}}\left({ϵ}_0\right)+{\pi }_1{H}_{\mathrm{b}}\left({ϵ}_1\right)\right).$$

(33)

We continue this procedure backwards: By the time we get to decoding Block i, we will have already reliably recovered ${\left(s^k\right)}_i$ in Block $(i+1)$, so we could use it as decoder-side information. The overall information rate—when b tends to infinity—approaches

$$\tilde{R}-H\left(\left\{S_i\right\}\right)-ϵ,$$

(34)

which can indeed be made arbitrarily close to (14).

We have now established that, provided that $\tilde{R}$ is appropriately chosen, as k grows large, the probability of the decoder correctly decoding both the message and $s^{kb}$ tends to one. It only remains to note that this also means that the symbol error rate $\beta$ is small. Indeed, having ${\widehat{S}}^{kb}=S^{kb}$ with high probability guarantees that the symbol error rate in the first b blocks is close to zero, whereas the influence of the extra block (whose symbol error rate will typically be high, since we do not attempt to decode the states there) becomes negligible when we choose b to be large.

We now return to the caveat regarding the Shannon capacity and the extra block. We need to show that the Shannon capacity is positive whenever (14) is positive. (If (14) is not positive, there is no need for a direct part.) Recall that the Shannon capacity of the Gilbert–Elliott channel without any state information is $1-H\left(\left\{Z_i\right\}\right)$, with $\left\{Z_i\right\}$ denoting the noise random variables, which form a Hidden Markov process. This Shannon capacity is positive unless $H\left(\left\{Z_i\right\}\right)=1$ bit. Our concern regarding the caveat is thus only when $H\left(\left\{Z_i\right\}\right)$ is 1.

In Appendix A, we show the following:

Proposition 1.

If $H\left(\left\{Z_i\right\}\right)=1$, then

1.

$\left\{Z_i\right\}$ are IID Bernoulli$(1/2)$, and

2.

either $\left\{S_i\right\}$ are IID or ${ϵ}_0={ϵ}_1=1/2$.

If $\left\{Z_i\right\}$ are IID Bernoulli$(1/2)$ and $\left\{S_i\right\}$ are IID, then (14) cannot be positive because, in this case, it equals

$$H\left(Z_i\right)-H\left(Z_i\right|S_i)-H\left(S_i\right)=-H\left(S_i\right|Z_i).$$

(35)

If ${ϵ}_0={ϵ}_1=1/2$, then (14) is also (trivially) not positive. Hence, indeed, the Shannon capacity is positive whenever (14) is positive.