Repair Rates for Multiple Descriptions on Distributed Storage ^†

Abstract

In a traditional distributed storage system, a source can be restored perfectly when a certain subset of servers is contacted. The coding is independent of the contents of the source. This paper considers instead a lossy source coding version of this problem where the more servers that are contacted, the higher the quality of the restored source. An example could be video stored on distributed storage. In information theory, this is called the multiple description problem, where the distortion depends on the number of descriptions received. The problem considered in this paper is how to restore the system operation when one of the servers fail and a new server replaces it, that is, repair. The requirement is that the distortions in the restored system should be no more than in the original system. The question is how many extra bits are needed for repair. We find an achievable rate and show that this is optimal in certain cases. One conclusion is that it is necessary to design the multiple description codes with repair in mind; just using an existing multiple description code results in unnecessary high repair rates.

1. Introduction

In distributed storage systems, data is divided into multiple segments that are then stored on separate servers. In a typical setup [1], data is divided into k segments that are stored on n servers using an $(n,k)$ maximum distance separable (MDS) code. If a user is able to contact any set of k servers, the data can be reconstructed. Notice that in this setup, if the user is able to contact less than k servers, it can retrieve no information, while on the other hand, there is no advantage in being able to contact more than k servers. One could instead want the quality of the reconstructed data to depend on how many servers a user is able to contact. An example could be video: it is common that the quality of streamed video depends on the network connection. In the context of distributed storage, the quality would now be dependent on the number of servers possible to connect, which could be constrained by network connection, physical location, delay, or cost. In information theory, this is known as multiple description coding [2,3]. Originally, multiple description coding was aimed at packet transmission networks, where some packets may be lost, but it can be directly applied to the distributed storage problem. We will accordingly call the systems we consider multiple description distributed storage.

A central issue in distributed storage is how to repair the system when one or more of the servers fail or become unavailable and are replaced by new servers [1]. In traditional distributed storage, this is also solved by the MDS code: if one server fails, the repair can be done by contacting k surviving servers, reconstruct the source, and then generating a new coded segment. The problem we consider in this paper is how repair can be done for multiple description distributed storage. The paper [1] and many following papers also consider how much network traffic is required for repair. However, in this paper we will only consider the amount of additional data needed to be stored for repair to be possible. The amount of network traffic is a topic for future research.

In general, the quality of reconstruction could be dependent not only on the number of servers connected, but which servers. However, to simplify the problem, we only consider the symmetric scenario where the quality only depends on the number of servers. This is the symmetric multiple description problem considered in [4]. A multiple description coding system with repair is specified as follows: when a subset $J\subset \{1,\dots ,n\}$ of servers is contacted, a source X should be restored with a distortion at most $D_J$. If one (or multiple) of the servers fails, we should be able to set up a replacement server with enough information so that the whole region $D_J,J\subset \{1,\dots ,n\}$ is restored. We consider two scenarios:

• There is a special (highly reliable) repair server that does not participate in the usual operation of the system, but only comes into action if another server fails. The repair server can contact all other (non-failed) servers and use their information combined with its own information to restore the failed server (collaborative repair).
• The repair information is stored in a distributed fashion among the n servers (distributed repair).

For simplicity, in this paper we only consider failure of a single server.

A straightforward solution is to separate the source coding problem (multiple description) and the repair problem. Any existing code for multiple description can then be used, and repair can be done using minimum distance separable (MDS) erasure codes as in traditional distributed storage [1]. We will use this as a baseline. For case 1 above, the repair server can simply store the xor (sum modulo 2) of the bits on the operational servers. When one server fails, the xor together with the bits from the surviving servers can restore the failed server. Thus, if each operational server stores $lR$ bits, the repair server also needs to store $lR$ bits. For distributed repair, the xor can replaced with an $(n,n-1)$ erasure code. Therefore in addition to the $lR$ bits for operation, each server needs to store $\frac{lR}{n-1}$ bits for repair. It should be clear that these rates are also optimal with separation: even if the system knows in advance which server will fail, it cannot store less information. We can consider this as a separate source channel coding solution, with multiple description being source coding and the repair being channel coding. It is known that in many information theory problems, joint source–channel coding is superior to separation. This is then the question we consider here: can we find a better joint source–channel coding solution that can beat the above rates? We will see that for some cases of desired distortion, separation is in fact optimal, while in other cases, joint source–channel coding provides much better rates.

The problem of repair of multiple description has been considered in some previous papers. In [5], the authors consider a problem like 1. above, but they do not give a single letter description of rate-distortion regions. In [6], the authors consider practical codes for repairing. In the current paper we aim to provide single letter expression for achievable rate-distortion regions, and in some cases the actual rate-distortion region. This paper is an extended version of our conference paper [7] with proof of the general achievable rate and specialization to the two level case, where we can prove optimality in certain cases.

2. Problem Description

In the following, we use the term repair node for the special repair server and operational nodes to denote the other servers. We let $I_k=\{1,\dots ,k\}$ and $X_{I_k}=[X_1,\dots ,X_k]$, with the definition $I_0=\varnothing$ and $X_{I_0}=\left[\right]$ (e.g., $H\left(Y\right|X_{I_0})=H\left(Y\right)$). For variables with multiple indices, $X_{I_k,I_j}$ denotes a matrix of variables, i.e, the collection $\{X_{11},X_{12},\dots ,X_{1j},X_{21},\dots ,\dots ,X_{k1},$$X_{k2},\dots ,X_{kj}\}$, and $X_{k{I}_j}$ denotes a row.

We consider a symmetric multiple description problem as in [4]. We have an i.i.d. (independent identically distributed) source X that takes values in a finite alphabet $\mathcal{X}$ and needs to be restored in the finite alphabet $\widehat{\mathcal{X}}$; this can be generalized to a continuous alphabet Gaussian source through usual quantization arguments [3]. Let $J\subset I_n$. We are given a collection of distortion measures ${\tilde{d}}_{\left|J\right|}:\mathcal{X}\times \widehat{\mathcal{X}}\to {\mathbb{R}}^{+}$, and define

$$d_{\left|J\right|}(x^l,{\widehat{x}}^l)=\frac{1}{l}\sum _{i=1}^l{\tilde{d}}_{\left|J\right|}(x_i,{\widehat{x}}_i)$$

The required maximum distortion $D_J$ is then a function of $\left|J\right|$ and the distortion measures $d_{\left|J\right|}$ only.

2.1. Distributed Repair

We will first define the distributed repair problem. For a source sequence $x^l$ of length l, each node stores $l{R}_t$ bits. There are n encoding functions $f_i:{\mathcal{X}}^l\to \{1,\dots ,2^{l{R}_t}\}$, $2^{n-1}-1$ decoding functions $g_J:{\{1,\dots ,2^{l{R}_t}\}}^{\left|J\right|}\to {\widehat{\mathcal{X}}}^l$, $J\subset I_n,1\le \left|J\right|\le n-1$, and n repair functions $h_i:{\{1,\dots ,2^{l{R}_t}\}}^{n-1}\to \{1,\dots ,2^{l{R}_t}\}$. We define the error probability of repair as

$$P_r^{\left(l\right)}=\underset{i=1,\dots ,n}{max}P\left(h_i\left(f_{I_n-\left\{i\right\}}\left(x^l\right)\right)\ne f_i\left(x^l\right)\right).$$

Here, $f_{I_n-\left\{i\right\}}\left(x^l\right)$ is the length $n-1$ list obtained by removing the i-th component from $(f_1(\left(x^l\right),f_2\left(x^l\right),\dots ,f_n\left(x^l\right))$. We now say that an a tuple $(R_t,D_1,\dots ,D_{n-1})$ is achievable if there exists a sequence of $(2^{l{R}_t},l)$ codes with

$$\begin{array}{cc}\hfill \forall m<n:\underset{l\to \infty }{lim}\underset{J:\left|J\right|=m}{max}E\left[d_{\left|J\right|}(x^l,g_J\left(f_J\left(x^l\right)\right))\right]& \le D_m\hfill \\ \hfill \underset{l\to \infty }{lim}{P}_r^{\left(l\right)}& =0\hfill \end{array}$$

(1)

We call this exact repair. The repaired node is required to be an exact copy of the failed node, except that we allow a certain, vanishing, and error rate. Notice that the randomness in the system is purely due to the source $x^l$. Thus, for a given sequence $x^l$, either all failures can be repaired exactly, and if they can be repaired once, they can be repaired infinitely many times; or, some failures can never be repaired. The probability of the source sequences that are not repairable should be vanishingly small.

An alternative problem formulation, which we call functional repair, is to allow approximate repair, where the only requirement is that after repair the distortion constraint is satisfied. In that case, one would have to carefully consider repeated repair. In this paper, we will only consider exact repair for coding schemes. It should be noted that in the cases where we have tight converses (the two node case [7], Theorem 3 in some scenarios), the converses are actually for functional repair; thus, functional repair might not decrease rates.

2.2. Collaborate Repair

For collaborate repair with a dedicated repair node, each node stores $lR$ bits and the repair node $l{R}_r$ bits. There are now n encoding functions $f_i:{\mathcal{X}}^l\to \{1,\dots ,2^{lR}\}$ and additionally a repair encoder $f_r:{\mathcal{X}}^l\to \{1,\dots ,2^{l{R}_r}\}$, $2^n-1$ decoding functions $g_J:{\{1,\dots ,2^{l{R}_t}\}}^{\left|J\right|}\to {\widehat{\mathcal{X}}}^l$, $J\subset I_n,1\le \left|J\right|\le n$, and n repair functions $h_i:{\{1,\dots ,2^{lR}\}}^{n-1}\times \{1,\dots ,2^{l{R}_r}\}\to \{1,\dots ,2^{lR}\}$. We define the error probability of repair as

$$P_r^{\left(l\right)}=\underset{i=1,\dots ,n}{max}P\left(h_i(f_{I_n-\left\{i\right\}}\left(x^l\right),f_r\left(x^l\right))\ne f_i\left(x^l\right)\right)$$

We now say that an a tuple $(R,R_r,D_1,\dots ,D_n)$ is achievable if there exists a sequence of $(2^{lR},2^{l{R}_r},l)$ codes with

$$\begin{array}{cc}\hfill \forall m\le n:\underset{l\to \infty }{lim}\underset{J:\left|J\right|=m}{max}E\left[d_{\left|J\right|}(x^l,g_J\left(f_J\left(x^l\right)\right))\right]& \le D_m\hfill \\ \hfill \underset{l\to \infty }{lim}{P}_r^{\left(l\right)}& =0\hfill \end{array}$$

(2)

3. Achievable Rate

The rate-distortion region for multiple description coding is only known in a few cases; among those are the two node Gaussian case first studied in [2], and the two level case studied in [8,9]. There are, therefore, many different achievable schemes for multiple description coding, e.g., [4,10,11,12], and we have to design repairs for each specific method. In this paper, we will consider the Puri Pradhan Ramchandran (PPR) scheme [4,13], as this is specifically aimed at the symmetric case and is well-suited for repair. It is optimal in certain cases [8,9], but not always [11].

The coding method in [4] is based on source-channel erasure codes (SCEC) from [13]. An $(n,k)$-SCEC is similar to an $(n,k)$-MDS erasure code: if any k of n packets are received, the transmitted message can be recovered with a certain distortion. However, with an $(n,k)$-SCEC if $m>k$ packets are received, the message can be recovered with decreasing distortion with m. Using a concatenation of $(n,1),(n,2),\dots ,(n,n)$ SCEC, [4] obtained the following result

Proposition 1

(PPR [4]). For any symmetric probability distribution $p({\mathbf{y}}_{I_{n-1},I_n},y_n|x)$ the lower convex closure of $(R,D_1,\dots ,D_n)$ is achievable, where $E[d_{\left|J\right|}(X,g_J\left({\mathbf{Y}}_{I_{\left|J\right|}J}\right)]\le D_{\left|J\right|},|J|\le n$ and

$$\begin{array}{cc}\hfill R& \ge \sum _{k=1}^{n-1}\frac{1}{k}H\left({\mathbf{Y}}_{k{I}_k}\right|{\mathbf{Y}}_{I_{k-1},I_k})\hfill \\ \hfill & +\frac{1}{n}I(Y_n;X|{\mathbf{Y}}_{I_{n-1}{I}_n})-\frac{1}{n}H({\mathbf{Y}}_{I_{n-1}{I}_n},Y_n|X)\hfill \end{array}$$

A probability distribution $p({\mathbf{y}}_{I_{n-1},I_n},y_{n-1}|x)$ is symmetric if for all $1\le r_i\le n,i\in I_{n-}$ the joint distribution of $Y_{n-1}$ and all $(r_1+r_2+\cdots +r_{n-1})$ random variables where any $r_i$ are chosen from the ith layer, conditioned on X are the same.

We first notice that for collaborative repair, reconstruction from n nodes does not make sense: since we can repair the last node from $n-1$ nodes, there can be no gain for a user to access all n nodes. The performance is therefore specified by $(D_1,D_2,\dots ,D_{n-1})$. As a baseline, we thus consider the standard PPR scheme where we use at most $n-1$ nodes for the reconstruction. Now, in layer $n-1$, we just need a single common message (in standard PPR that happens at layer n). This message can be encoded using an $(n,n-1)$ MDS erasure code. We then get the following rate, which we state without proof as it is a simple modification of PPR:

Proposition 2.

For any symmetric probability distribution $p({\mathbf{y}}_{I_{n-2},I_n},y_{n-1}|x)$ the lower convex closure of $(R,D_1,\dots ,D_{n-1})$ is achievable, where $E[d_{\left|J\right|}(X,g_J\left({\mathbf{Y}}_{I_{\left|J\right|}J}\right)]\le D_{\left|J\right|},|J|\le n-1$, the following rate is achievable with n nodes and using at most $(n-1)$ nodes for reconstruction

$$\begin{array}{cc}\hfill R& \ge \sum _{k=1}^{n-2}\frac{1}{k}H\left({\mathbf{Y}}_{k{I}_k}\right|{\mathbf{Y}}_{I_{k-1},I_k})\hfill \\ \hfill & +\frac{1}{n-1}I(Y_{n-1};X|{\mathbf{Y}}_{I_{n-2}{I}_{n-1}})-\frac{1}{n}H\left({\mathbf{Y}}_{I_{n-2}{I}_n}\right|X)\hfill \end{array}$$

Notice that one should not think of this as an ‘improved’ PPR scheme; rather it is the PPR scheme adapted to the special case here, where at most $n-1$ nodes are used for reconstruction.

For our repair coding scheme, we amend the PPR scheme, specifically from Proposition 2. We still use an $(n,k)$-SCEC at layers $k\le n-2$, but add a common message ($U_k$) at each layer $k\le n-2$. At layer 1, this is a true common message that is duplicated to all nodes. At layers $k>1$ this is a message stored with an $(n,k)$-MDS code. Common messages were shown to be necessary to achieve optimality for the two-node case in [7]. We also use binning for repair of correlated quantizations. A system schematic for a specific case can be seen in Figure 1 below. The addition of common messages strictly decreases the rate for repair in some cases, see Section 5.

Figure 1. Two layer repair. See text for explanation.

The following is the main result of the paper, an achievable repair rate; this rate can be compared to the rate in Proposition 2. As above, we call a probability distribution $p({\mathbf{y}}_{I_{n-2},I_n},{\mathbf{u}}_{I_{n-2}},y_{n-1}|x)$ symmetric if for all $1\le r_i\le n-1,i\in I_{n-2}$ and all $k\in I_{n-2}$ the joint distribution of $Y_{n-1},U_k$ and all $(r_1+r_2+\cdots +r_{n-2})$ random variables where any $r_i$ are chosen from the ith layer, conditioned on X are the same.

Theorem 1

(Distributed repair). For any symmetric probability distribution $p({\mathbf{y}}_{I_{n-2},I_n},{\mathbf{u}}_{I_{n-2}},$$y_{n-1}\left|x\right)$ the lower convex closure of $(R+R_r,D_1,\dots ,D_{n-1})$ is achievable, where $E[d_{\left|J\right|}(X,g_J({\mathbf{Y}}_{I_{\left|J\right|}J},$${\mathbf{U}}_{I_{\left|J\right|}}\left)\right]\le D_{\left|J\right|},\left|J\right|\le n-1$ and the information needed to encode operational information is

$$\begin{array}{c}{\displaystyle R>\sum _{k=1}^{n-2}\frac{1}{k}H\left({\mathbf{Y}}_{k{I}_k}\right|{\mathbf{U}}_{I_k},{\mathbf{Y}}_{I_{k-1}{I}_k})-\frac{1}{n}H\left({\mathbf{Y}}_{k{I}_n}\right|X,{\mathbf{U}}_{I_k},{\mathbf{Y}}_{I_{k-1}{I}_n})}\hfill \\ +\frac{1}{n-1}I(Y_{n-1};X|{\mathbf{Y}}_{I_{n-2}{I}_{n-1}},{\mathbf{U}}_{I_{n-2}})\hfill \\ +\sum _{k=1}^{n-2}\frac{1}{k}(H\left(U_k\right|{\mathbf{Y}}_{I_{k-1}{I}_k},{\mathbf{U}}_{I_{k-1}})-H\left(U_k\right|X,{\mathbf{Y}}_{I_{k-1}{I}_n},{\mathbf{U}}_{I_{k-1}}))\hfill \end{array}$$

with additional information needed to encode repair information

$$\begin{array}{cc}\hfill R_r& >\frac{1}{n-1}\sum _{k=1}^{n-2}\left[H\left(Y_{kn}\right|{\mathbf{U}}_{I_k},{\mathbf{Y}}_{k{I}_{n-1}}{\mathbf{Y}}_{I_{k-1}{I}_n})\phantom{-\frac{1}{n}H\left({\mathbf{Y}}_{k{I}_n}\right|X,{\mathbf{Y}}_{k-1I_n},{\mathbf{U}}_{I_k})}\right.\hfill \\ \hfill & {\left.-\frac{1}{n}H\left({\mathbf{Y}}_{k{I}_n}\right|X,{\mathbf{Y}}_{k-1I_n},{\mathbf{U}}_{I_k})\right]}^{+}\hfill \end{array}$$

with ${\left[x\right]}^{+}=max\{0,x\}$

Proof. 

There is a formal proof in Appendix A—the purpose here is to outline how the coding is done and how the rate expressions are obtained, without a deep knowledge of [4].

Consider at first layer 1. We generate a codebook ${\mathcal{C}}_{u1}$ by picking $2^{l{R}_{u1}^{\prime }}$ elements uniformly randomly with replacement from the typical set according to the distribution $p_{U_1}\left(u_1\right)$. We also generate n independent random codebooks ${\mathcal{C}}_{1I_n}$ drawn from the typical set according to $p_{Y_{11}}\left(y_{11}\right)$ with $2^{l{R}_1^{\prime }}$ codewords. We need to be able to find a codeword in ${\mathcal{C}}_{u1}$ that is jointly typical with $x^l$ with high probability, which, from standard rate distortion, is the case if

$$\begin{array}{cc}\hfill R_{u1}=R_{u1}^{\prime }& >H\left(U_1\right)-H\left(U_1\right|X)=I(X;U_1)\hfill \end{array}$$

This codeword is stored in all the nodes. We now need to be able to find n codewords from ${\mathcal{C}}_{1I_n}$ that are jointly typical with $x^l$ and the chosen codeword $u_1^l\in {\mathcal{C}}_{u1}$. There are about $2^{nlH\left(Y_{11}\right)}$ (marginally) typical sequences, and about $2^{lH(Y_{11},\dots ,Y_{1n}|U_1,X)}$ that are jointly typical with a given $x^l$ and $u_1^l$ (see, e.g., [14] (Section 15.2)); the probability that a given codeword combination in ${\mathcal{C}}_{1I_n}$ is jointly typical, therefore it is about $2^{l(H(Y_{11},\dots ,Y_{1n}|U_1,X)-nH\left(Y_{11}\right))}$. The probability that no codeword is jointly typical then is about

$${\left(1-2^{l(H(Y_{11},\dots ,Y_{1n}|U_1,X)-nH\left(Y_{11}\right))}\right)}^{2^{nl{R}_1^{\prime }}}\le exp\left(-2^{l(n{R}_1^{\prime }-(nH\left(Y_{11}\right)-H(Y_{11},\dots ,Y_{1n}|U_1,X)))}\right)$$

The inequality is standard in rate distortion, see [3,14]. Thus, if

$$\begin{array}{cc}\hfill n{R}_1^{\prime }& >nH\left(Y_{11}\right)-H(Y_{11},\dots ,Y_{1n}|U_1,X)\hfill \end{array}$$

(3)

there is a high probability that at least one of the $2^{nl{R}_1^{\prime }}$ codeword combinations is jointly typical.

The codewords in ${\mathcal{C}}_{1j}$ are randomly binned into $2^{l{R}_1}$ bins. At the time of decoding, the common codeword $u_1^l\in {\mathcal{C}}_{u1}$ is available as well as the bin number i for the codeword $y_{ij}^l\in {\mathcal{C}}_{1j}$. The decoder looks for a codeword in bin i that is typical with $u_1^l$. There is always one, the actual codeword, but if there is more than one, the decoding results in error. The probability that a random codeword in ${\mathcal{C}}_{1j}$ is jointly typical with $u_1^l$ is about $2^{l(H\left(Y_{11}\right|U_1)-H\left(Y_{11}\right))}$ as above, while there are about $2^{l(R_1^{\prime }-R_1)}$ codewords in each bin. By the union bound, the probability that there is at least one random codeword in the bin jointly typical is approximately upper bounded by $2^{l(R_1^{\prime }-R_1)}{2}^{-l(H\left(Y_{11}\right)-H\left(Y_{11}\right|U_1))}$. Thus, if

$$R_1^{\prime }-R_1<H\left(Y_{11}\right)-H\left(Y_{11}\right|U_1)$$

(4)

there is only one such codeword with high probability. Combining (3) and (4) we get

$$R_1>H\left(Y_{11}\right|U_1)-\frac{1}{n}H(Y_{i1},\dots ,Y_{in}|U_1,X)$$

At layer $k<n-1$ we similarly generate a random codebook ${\mathcal{C}}_{uk}$ with $2^{l{R}_{uk}^{\prime }}$ typical elements according to the marginal distribution $p_{U_k}\left(u_k\right)$ and n independent random codebooks ${\mathcal{C}}_{k{I}_n}$ according to the distribution $p_{Y_{k1}}\left(y_{k1}\right)$ with $2^{l{R}_k^{\prime }}$ codewords. We need to be able to find a codeword in ${\mathcal{C}}_{uk}$ that is jointly typical with $x^l$ and all the codewords chosen in the previous layers. This is possible if

$$\begin{array}{cc}\hfill R_{uk}^{\prime }& >H\left(U_k\right)-H\left(U_k\right|X,{\mathbf{Y}}_{I_{k-1}{I}_n},{\mathbf{U}}_{I_{k-1}})\hfill \end{array}$$

with the same argument as for (3). We also need to be able to find an n-tuple of codewords from ${\mathcal{C}}_{k{I}_n}$ that are jointly typical with all prior codewords and $x^l$, which is possible with high probability if (again as in (3))

$$\begin{array}{cc}\hfill n{R}_k^{\prime }& >nH\left(Y_{k1}\right)-H\left({\mathbf{Y}}_{k{I}_n}\right|X,{\mathbf{Y}}_{k-1I_n},{\mathbf{U}}_{I_k})\hfill \end{array}$$

For ${\mathcal{C}}_{uk}$, we generate n independent binning partitions each with $2^{l{R}_{uk}}$ elements. The bin number in the i-th partition is stored in the i-th node. When the decoder has access to k nodes, say nodes $1,\dots ,k$ it needs to be able to be able to find a unique codeword in the k bins jointly typical with codewords from previous layers. The probability that a random selected codeword is jointly typical is about $2^{l(H\left(U_k\right|{\mathbf{Y}}_{I_{k-1}{I}_k},{\mathbf{U}}_{I_{k-1}})-H\left(U_k\right))}$, as above. There are about $2^{l{R}_{uk}^{\prime }}{2}^{-lk{R}_{uk}}$ in each combined bin. Therefore, if

$$k{R}_{uk}>R_{uk}^{\prime }+H\left(U_k\right|{\mathbf{Y}}_{I_{k-1}{I}_k},{\mathbf{U}}_{I_{k-1}})-H\left(U_k\right)$$

or

$$R_{uk}>\frac{1}{k}(H\left(U_k\right|{\mathbf{Y}}_{I_{k-1}{I}_k},{\mathbf{U}}_{I_{k-1}})-H\left(U_k\right|X,{\mathbf{Y}}_{I_{k-1}{I}_n},{\mathbf{U}}_{I_{k-1}}))$$

(5)

with high probability there is only one jointly typical codeword in the combined bin. It also needs to find a single codeword in the k bins for ${\mathcal{C}}_{k{I}_k}$ s that are jointly typical with $({\mathbf{U}}_{I_k},{\mathbf{Y}}_{I_{k-1}{I}_k})$. The probability that a random codeword is jointly typical is about $2^{l(H\left({\mathbf{Y}}_{k{I}_k}\right|{\mathbf{U}}_{I_k},{\mathbf{Y}}_{I_{k-1}{I}_k})-kH\left(Y_{k1}\right))}$, while the number of codewords in the k joint bins is about $2^{lk{R}_k^{\prime }}{2}^{-lk{R}_k}$. With high probability there is only one such if

$$k(R_k^{\prime }-R_k)<kH\left(Y_{k1}\right)-H\left({\mathbf{Y}}_{k{I}_k}\right|{\mathbf{U}}_{I_k},{\mathbf{Y}}_{I_{k-1}{I}_k})$$

or

$$R_k>\frac{1}{k}H\left({\mathbf{Y}}_{k{I}_k}\right|{\mathbf{U}}_{I_k},{\mathbf{Y}}_{I_{k-1}{I}_k})-\frac{1}{n}H\left({\mathbf{Y}}_{k{I}_n}\right|X,{\mathbf{U}}_{I_k},{\mathbf{Y}}_{I_{k-1}{I}_n})$$

(as in [13] this can be repeated for any collection of k nodes).

At layer $n-1$ only a single codebook is generated, and this is binned into n independent partitions. Upon receipt, in analogy with (5), this can be found uniquely with high probability if

$$\begin{array}{cc}\hfill R_{n-1}& >\frac{1}{n-1}H\left(Y_{n-1}\right|{\mathbf{Y}}_{I_{n-2}{I}_{n-1}},{\mathbf{U}}_{I_{n-2}})\hfill \\ \hfill & -\frac{1}{n-1}H\left(Y_{n-1}\right|X,{\mathbf{Y}}_{I_{n-2}{I}_n},{\mathbf{U}}_{I_{n-2}})\hfill \end{array}$$

For repair, the joint $2^{nl{R}_k^{\prime }}$ codewords in ${\mathcal{C}}_{k1}\times \cdots \times {\mathcal{C}}_{kn}$ at layer $k<n-1$ are binned into $2^{l{R}_{rk}}$ bins. The single bin number of the n chosen codewords is encoded with an $(n,n-1)$ MDS erasure code.

Now, suppose node n is lost, and needs to be recovered. The repair node works from the bottom up. So, suppose the previous $k-1$ layers have been recovered, that is, ${\mathbf{y}}_{I_{k-1}{I}_b}^l,{\mathbf{u}}_{I_{k-1}}^l$ are known without error. First $u_k^l$ is recovered, which can be done since $n-1\ge k$ nodes are used. It can also decode the codewords in ${\mathcal{C}}_{k{I}_{n-1}}$. It restores the bin number of the repair codeword from the erasure code. There are approximately $2^{l(n{R}_k^{\prime }-R_{rk})}$ codewords in the bin, but since it knows the codewords in ${\mathcal{C}}_{k{I}_{n-1}}$, there are only about $2^{l(R_k^{\prime }-R_{rk})}$ valid ones. It searches in the bin for valid codewords jointly typical with ${\mathbf{y}}_{k{I}_{n-1}}^l,{\mathbf{y}}_{I_{k-1}{I}_n}^l,{\mathbf{u}}_{I_k}^l$. With high probability, there is only one such if

$$R_k^{\prime }-R_{rk}<H\left(Y_{kn}\right)-H\left(Y_{kn}\right|{\mathbf{U}}_{I_k},{\mathbf{Y}}_{k{I}_{n-1}}{\mathbf{Y}}_{I_{k-1}{I}_n})$$

(The right hand side could be negative. This means that the lost codeword can be recovered from the surviving ones without extra repair information. Then we just put $R_{rk}=0$.) Then

$$\begin{array}{cc}\hfill R_{rk}& >H\left(Y_{kn}\right|{\mathbf{U}}_{I_k},{\mathbf{Y}}_{k{I}_{n-1}}{\mathbf{Y}}_{I_{k-1}{I}_n})\hfill \\ \hfill & -\frac{1}{n}H\left({\mathbf{Y}}_{k{I}_n}\right|X,{\mathbf{Y}}_{k-1I_n},{\mathbf{U}}_{I_k})\hfill \end{array}$$

(6)

There is at least one codeword in the bin, namely the correct one. Thus, if there is no error (more than one codeword), the repair is exact, as required from the exact repairability condition in Section 2. □

The above result can easily be adapted to the case of a repair node that collaborates with the operational nodes. There are only two differences:

• The repair node can restore operation of the full n node distortion region. Therefore, the terminal single common codeword is not at layer $n-1$, but at layer n. At the same time, the repair node now has to store repair information for this last codeword.
• For distributed repair, distributions are chosen to minimize $R+R_r$. For collaborative repair, distributions are chosen to minimize R, and $R_r$ is then as given for those distributions.

With this in mind, we get

Theorem 2

(Collaborative repair). For any symmetric probability distribution $p({\mathbf{y}}_{I_{n-1},I_n},{\mathbf{u}}_{I_{n-1}},$$y_n\left|x\right)$ the lower convex closure of $(R,D_1,\dots ,D_n)$ is achievable, where $E[d_{\left|J\right|}(X,g_J({\mathbf{Y}}_{I_{\left|J\right|}J},{\mathbf{U}}_{I_{\left|J\right|}})]\le D_{\left|J\right|},|J|\le n$ and

$$\begin{array}{c}{\displaystyle R>\sum _{k=1}^{n-1}\frac{1}{k}H\left({\mathbf{Y}}_{k{I}_k}\right|{\mathbf{U}}_{I_k},{\mathbf{Y}}_{I_{k-1}{I}_k})-\frac{1}{n}H\left({\mathbf{Y}}_{k{I}_n}\right|X,{\mathbf{U}}_{I_k},{\mathbf{Y}}_{I_{k-1}{I}_n})}\hfill \\ +\frac{1}{n}I(Y_n;X|{\mathbf{Y}}_{I_{n-1}{I}_n},{\mathbf{U}}_{I_{n-1}})\hfill \\ +\sum _{k=1}^{n-1}\frac{1}{k}(H\left(U_k\right|{\mathbf{Y}}_{I_{k-1}{I}_k},{\mathbf{U}}_{I_{k-1}})-H\left(U_k\right|X,{\mathbf{Y}}_{I_{k-1}{I}_n},{\mathbf{U}}_{I_{k-1}}))\hfill \end{array}$$

The additional information the repair node has to store is

$$\begin{array}{cc}\hfill R_r& >\sum _{k=1}^{n-1}\left[H\left(Y_{kn}\right|{\mathbf{U}}_{I_k},{\mathbf{Y}}_{k{I}_{n-1}}{\mathbf{Y}}_{I_{k-1}{I}_n})\phantom{-\frac{1}{n}H\left({\mathbf{Y}}_{k{I}_n}\right|X,{\mathbf{Y}}_{k-1I_n},{\mathbf{U}}_{I_k})}\right.\hfill \\ \hfill & {\left.-\frac{1}{n}H\left({\mathbf{Y}}_{k{I}_n}\right|X,{\mathbf{Y}}_{k-1I_n},{\mathbf{U}}_{I_k})\right]}^{+}\hfill \\ \hfill & +\frac{1}{n}H\left(Y_n\right|{\mathbf{Y}}_{I_{n-1}{I}_n},{\mathbf{U}}_{I_{n-1}})-\frac{1}{n}H\left(Y_n\right|{\mathbf{Y}}_{I_{n-1}{I}_n},{\mathbf{U}}_{I_{n-1}},X)\hfill \end{array}$$

The proof is nearly identical to the proof of Theorem 1, so it will be omitted.

4. The Two Level Case

In [9], the authors considered the situation when there were only two cases of node access: Either we have access to all n nodes, or we have access to a given number $k<n$ nodes; there are two levels of distortion: $(D_k,D_n)$. Importantly, they were able to derive the exact capacity region for this case for Gaussian sources, one of the few cases when this known except for the original EC case [2]. This makes it an interesting case to consider for repair: at least we can upper bound the number of bits needed for repair by the achievable rate in Section 3. The paper [9] considered the vector Gaussian case, but we restrict ourselves to the scalar Gaussian case.

To fit into the framework of [9], we need to consider the case when there is a repair node, Theorem 2. In that case, the scheme is as shown on Figure 1. The $U_k$ represents a common codeword that is stored jointly on the operational nodes with an $(n,k)$ MDS. If one server fails, this can be restored without additional information from the repair as $k\le n-1$. $Y_{k1},\dots ,Y_{kL}$ represent individual codewords using SCEC (source-channel erasure code) codes from [4,13]; here, the repair is accomplished using correlation and a bin index, similar to the two node case. Finally, $Y_n$ represents resolution information, which can be repaired due to the $(n+1,n)$ MDS code.

The explicit rate constraints from Theorem 2 are

$$\begin{array}{c}{\displaystyle R>\frac{1}{k}H\left({\mathbf{Y}}_{k{I}_k}\right|U_k)}\hfill \\ +\frac{1}{n}H\left(Y_n\right|{\mathbf{Y}}_{k{I}_n},U_k)-\frac{1}{n}H({\mathbf{Y}}_{k{I}_n},Y_n|X,U_k)\hfill \\ +\frac{1}{k}(H\left(U_k\right)-H\left(U_k\right|X))\hfill \end{array}$$

with

$$\begin{array}{cc}\hfill R_r& >{\left[H\left(Y_{kn}\right|U_k,{\mathbf{Y}}_{k{I}_{n-1}})-\frac{1}{n}H\left({\mathbf{Y}}_{k{I}_n}\right|X,U_k)\right]}^{+}\hfill \\ \hfill & +\frac{1}{n}H\left(Y_n\right|{\mathbf{Y}}_{k{I}_n},U_k)-\frac{1}{n}H\left(Y_n\right|{\mathbf{Y}}_{k{I}_n},U_k,X)\hfill \end{array}$$

We consider an iid Gaussian source with $x_i\sim N(0,1)$ with a quadratic distortion function: ${\tilde{d}}_{\left|J\right|}(x_i,{\widehat{x}}_i)={(x_i-{\widehat{x}}_i)}^2$. For this situation, we can calculate the achievable repair rate explicitly. We recall that the problem setup is that R is fixed to the optimum rate from [9]. We then obtain:

Theorem 3.

In the Gaussian two level case, we have the following bounds on the repair rate:

1. 

For $k{(D_k^{-1}-1)}^{-1}-n{(D_n^{-1}-1)}^{-1}\le 0$ a common message is used and achieves

$$R_r\le \frac{1}{2}log\left(\frac{D_k(n-k)}{D_k(n-k-1)+D_n}\right)$$

For $k=n-1$ the upper bound is tight.

2. 

For $0<k{(D_k^{-1}-1)}^{-1}-n{(D_n^{-1}-1)}^{-1}\le n-k$ no common message is used and

$$R_r\le \frac{1}{2}log\left(\frac{(D_k-1)n(n-k){\left(\frac{k(D_k-D_n)}{(D_k-1)D_n(k-n)}\right)}^{1/n}}{k(-D_{k}n+D_n+n-1)+(D_k-1)(n-1)n}\right)$$

For $k=n-1$ the upper bound is tight.

3. 

For $k{(D_k^{-1}-1)}^{-1}-n{(D_n^{-1}-1)}^{-1}>n-k$ no common message is used and the exact repair rate is

$$R_r=R=\frac{1}{2n}log\left(\frac{1}{D_n}\right)$$

for all k and n.

We will discuss some implications of this result. The converse is provided by the bound (A8) $(n-1)R+R_r\ge \frac{1}{2}log\left(\frac{1}{D_n}\right)$, which is simply the requirement that the repair node together with the surviving nodes should be able to restore the source with distortion $D_n$. This is clearly also a converse for functional repair, which could indicate that relaxing to functional repair cannot decrease rates. For $k=n-1$, the theorem provides the exact repair rate; without using common messages, we could not have achieved the bound. We can compare with separate repair and multiple description coding, as mentioned in the introduction. For case 3, the theorem separation is optimal, but for the other cases $R_r<R$. For example, for $n=10,k=5,D_k=0.5,D_n=0.48$, we get $R=0.06,R_r=0.02$ for case 1.

5. Example Gaussian Case

Figure 2 shows typical numerical results. All curves are for two levels of constraints, $(D_1,D_2)$, but variable number of nodes. First, from the bottom, we have the curve for the optimum region for the two node problem according to EC [2,3]. Notice that this is achieved without any refinement information, using only correlation between the base layer random variables; refinement information is only required for $D_1>\frac{1}{2}$ and $D_2<2D_1-1$. Second, we have the curves for the three node problem, but where we use at most two nodes for reconstruction, either using [4] (Section V) directly (ignoring the $D_3$ constraint), or using Theorem 1 without repair. It can be noticed that using Proposition 2 gives a slight improvement; this is not due to the common message, but due to the fact that PPR uses $n-1$ codewords in the last layer, while the modified PPR uses only one. For the 4 node case, we use (4,1)-SCEC and (4,2)-SCEC successively, as well as (4,1)-MDS common message and (4,2)-MDS common message. Therefore, we have 2 variables $U_1$ and $U_2$ for common messages, and $Y_{1i}$ and $Y_{2i}$ for SCEC, where i = 1, 2, 3, 4. As a result, it is noted that the overall rate of the 4 node system improves over that of the 3 node system, whereas the overall rate of the 2 node system improves over that of the 3 node system where common message and SCEC were used only once. We see that a common message gives a clear improvement.

Figure 2. Plots of R or $R+R_r$ for two levels of constraints $(D_1,D_2)$ and variable number of nodes.

6. Conclusions

The paper has derived achievable rates for repair of multiple description distributed storage, which in some cases is optimal. Our solution shows that joint repair and multiple description coding beats separate coding in many cases. It also shows that it is sub-optimal for repair to just take a standard multiple description code and add repair information. Rather, the multiple description code has to be designed with repair in mind. In this paper, we do this by adding common messages.

This paper is only a first step in solving repair of multiple description distributed storage. For one thing, we have assumed that the repair bandwidth is unlimited. When the required repair bandwidth is also of concern as in [1], an entirely new set of constraints comes into play. We will consider this in a later paper.

Appendix A.1. Codebook Generation

The codebooks ${\mathcal{C}}_{I_{n-2}{I}_n}$ are generated and binned exactly as in [4]. The difference from [4] is that there is no n-th layer, and that at layer $n-1$ there is only one codebook ${\mathcal{C}}_{n-1}$. The codebook ${\mathcal{C}}_{n-1}$ of size $2^{l{R}_{n-1}^{\prime }}$ is generated like ${\mathcal{C}}_n$ in [4], but then stored on the nodes with an $(n,n-1)$ MDS code.

We also generate $n-2$ common codebooks ${\mathcal{C}}_{u{I}_{n-2}}$ by drawing $2^{l{R}_{uk}^{\prime }}$ codewords $\left({\mathbf{u}}_k^{\left(l\right)}\left(1\right),\dots ,{\mathbf{u}}_k^{\left(l\right)}\left(2^{l{R}_{u_k}^{\prime }}\right)\right)$ independently with replacement over the set $T_{ϵ}^l\left(U_k\right)$ according to a uniform distribution. The indices for ${\mathcal{C}}_{uk},k=2,\dots ,,n-2$ are next binned. Let ${\xi }_{uk}=2^{l(R_{uk}^{\prime }-R_{uk}+{\gamma }_{uk})}$ for some ${\gamma }_k>0$ and make $2^{l{R}_{uk}}$ bins. For each bin, select ${\xi }_{uk}$ numbers from the set $\{1,\dots ,2^{l{R}_{uk}^{\prime }}\}$, uniformly and with replacement. They are finally coded with an $(n,k)$ MDS erasure code.

We finally generate $(n-1)$ repair codebooks through binning. First, if

$$\begin{array}{cc}\hfill 0& >H\left(Y_{kn}\right|{\mathbf{U}}_{I_k},{\mathbf{Y}}_{k{I}_{n-1}}{\mathbf{Y}}_{I_{k-1}{I}_n})\hfill \\ \hfill & -\frac{1}{n}H\left({\mathbf{Y}}_{k{I}_n}\right|X,{\mathbf{Y}}_{k-1I_n},{\mathbf{U}}_{I_k})\hfill \end{array}$$

(A1)

it turns out, as will be seen later, that the lost codeword can be recovered from the remaining ones with high probability. In that case, we set $R_{rk}=0$ and store no extra repair information. For consistency, we think of there being one bin at layer k containing all codewords. Otherwise, we let ${\xi }_{rk}=2^{l(n{R}_k^{\prime }-R_{rk}+{\gamma }_{rk})}$ for some ${\gamma }_{rk}>0$ and make $2^{l{R}_{rk}}$ bins. For each bin, select ${\xi }_{rk}$ vectors from the set ${\{1,\dots ,2^{l{R}_k^{\prime }}\}}^n$, uniformly and with replacement. The bin indices are further coded with an $(n,n-1)$ MDS erasure code.

Appendix A.2. Encoding

Given a source codeword ${\mathbf{x}}^{\left(l\right)}\in {\mathcal{X}}^l$, we find codewords so that

$$\left({\mathbf{x}}^{\left(l\right)},{\mathbf{u}}_{I_{n-2}}^{\left(l\right)}\left({\mathbf{V}}_{I_{n-2}}\right),{\mathbf{y}}_{I_{n-2}{I}_n}^{\left(l\right)}.\ast \left({\mathbf{Q}}_{I_{n-2}{I}_m}\right),{\mathbf{y}}_{n-1}^{\left(l\right)}\left(Q_{n-1}\right)\right)$$

are jointly typical. The binning of ${\mathbf{Q}}_{I_{n-2}{I}_m}$ and $Q_{n-1}$ are done exactly as in [4] to obtain bin indices $B_{I_{n-2}{I}_n},B_{n-1}$. The bin index $B_{n-1}$ is further coded with the $(n,n-1)$ MDS code. For $V_k$, we find the smallest bin index $B_{uk}$ that contains $V_k$ (if $V_k$ is in no bin, $B_{uk}=0$), and this is further coded with the $(n,k)$ MDS code.

For repair, for those $k\in I_{n-1}$ where repair information is needed, we find the smallest bin index $W_k$ so that ${\mathbf{Q}}_{k{I}_n}$ is in the corresponding bin; if no bin contains $W_k$, we put $W_k=0$. These are then coded with the $(n,n-1)$ MDS code.

Appendix A.3. Decoding

We assume node $1,2,\dots ,j$ are available. The bin indices $B_{u{I}_{j^{\prime }}}$ are decoded from the MDS code, where $j^{\prime }=min\{j,n-2\}$. The decoding now is similar to [4], except that there is also a common codeword. Consider decoding at layer $k\in \{2,\dots ,j^{\prime }\}$. First, we find an index $V_k$ in bin $B_{uk}$ so that

$$\left({\mathbf{y}}_{I_{k-1}{I}_j}^{\left(l\right)},{\mathbf{u}}_k^{\left(l\right)}\left(V_k\right),{\mathbf{u}}_{I_{k-1}}^{\left(l\right)}\right)\in T_{ϵ}^l\left({\mathbf{Y}}_{I_{k-1}{I}_j},{\mathbf{U}}_{I_k}\right)$$

Next, for any size k subset $S\subset I_j$, the decoder looks in bins ${\mathbf{B}}_{kS}$ for codewords ${\mathbf{y}}_{kS}^{\left(l\right)}$ so that

$$\left({\mathbf{y}}_{kS}^{\left(l\right)},{\mathbf{y}}_{I_{k-1}{I}_j}^{\left(l\right)},{\mathbf{u}}_{I_k}^{\left(l\right)}\right)\in T_{ϵ}^l\left({\mathbf{Y}}_{k{I}_k},{\mathbf{Y}}_{I_{k-1}{I}_j},{\mathbf{U}}_{I_k}\right)$$

If $j=n-1$, $B_{n-1}$ is first recovered from the MDS code. Then, the above procedure is repeated (there is no $U_{n-1}$).

The reconstructions of ${\widehat{\mathbf{x}}}^{\left(l\right)}$ are standard as in [4].

Appendix A.4. Repair

Without loss of generality and to simplify notation, we can assume that node n fails. The repair is done layer by layer. At layer 1, we copy $V_1$ from any node to the replacement node n. Next, from the $(n-1)$ surviving nodes we decode the repair bin index $W_1$ from the MDS code; if there is no extra repair information, we put $W_1=1$. We know ${\mathbf{Q}}_{1I_{n-1}}$ from the surviving nodes. In bin $W_1$, we look for an index $Q_{1n}$ so that the corresponding codeword $({\mathbf{y}}^{\left(l\right)}.\ast \left({\mathbf{Q}}_{1I_n}\right),{\mathbf{u}}_1^{\left(l\right)}\left(V_1\right))\in T_{ϵ}^l({\mathbf{Y}}_{1I_n},U_1)$; if there is more than one, there is a repair error. We then store the recovered $Q_{1n}$ in the replacement node n.

The following layers proceed in almost the same way. However, now to recover the common message $V_k$ we arbitrarily choose k of the surviving nodes and decode $V_k$ just as with usual operation. The decoded $V_k$ is then encoded with the exact same MDS code and we store the corresponding codeword on the replacement node n. We next find an index $Q_{kn}$ in bin $W_k$ so that $({\mathbf{y}}_{I_k{I}_n}^{\left(l\right)}.\ast \left({\mathbf{Q}}_{I_k{I}_n}\right),{\mathbf{u}}_{I_k}^{\left(l\right)}\left(V_{I_k}\right))\in T_{ϵ}^l({\mathbf{Y}}_{I_k{I}_n},{\mathbf{U}}_{I_k})$.

On the last layer, we simply decode $Q_{n-1}$ from the surviving nodes as usual, and then we re-encode with the same MDS code, and store the recovered bin index on the new node n.

We notice that this repair is exact: the information on the restored node is exactly the same as on the failed node, except if a repair error happens.

Appendix A.5. Analysis of Decoding Error

We have some slightly modified error events compared to [4] and some additional ones. We find it necessary to write these down explicitly

• $E_0$: ${\mathbf{x}}^{\left(l\right)}\notin T_{ϵ}^l\left(X\right)$.
• $E_1$: There exists no indices so that

  $$\begin{array}{c}\hfill \left({\mathbf{x}}^{\left(l\right)},{\mathbf{u}}_{I_{n-2}}^{\left(l\right)}\left({\mathbf{V}}_{I_{n-2}}\right),{\mathbf{y}}_{I_{n-2}{I}_n}^{\left(l\right)}.\ast \left({\mathbf{Q}}_{I_{n-2}{I}_m}\right),\right.\\ \hfill \left.{\mathbf{y}}_{n-1}^{\left(l\right)}\left(Q_{n-1}\right)\right)\\ \hfill \in T_{ϵ}^l(X,U_{I_{n-2}},Y_{I_{n-2}{I}_n},Y_{n-1})\end{array}$$
• $E_2$: Not all the indices $({\mathbf{B}}_{2I_n},\dots ,{\mathbf{B}}_{(n-2)I_n},B_{n-1})$ are greater than zero.
• $E_3$: For some subset $S\subset I_n$ with $\left|S\right|=k\in \{2,\dots ,n-2\}$ there exists some other ${\mathbf{Q}}_{kS}^{\prime }$ in bins ${\mathbf{B}}_{kS}$ so that (We use a slightly different notation for $E_3$ compared to [4], which we think is clearer.

  $$\left({\mathbf{y}}_{kS}^{\left(l\right)}\left({\mathbf{Q}}_{kS}^{\prime }\right),{\mathbf{y}}_{I_{k-1}{I}_j}^{\left(l\right)},{\mathbf{u}}_{I_k}^{\left(l\right)}\right)\in T_{ϵ}^l\left(Y_{k{I}_k},Y_{I_{k-1}{I}_k},U_{I_k}\right)$$
• $E_4$: Not all the indices $B_{uk}$ are greater than zero.
• $E_5$: For some $2\le k\le n-2$ there exist another index $V_k^{\prime }\ne V_k$ in bin $B_{uk}$ so that

  $$\left({\mathbf{y}}_{I_{k-1}{I}_j}^{\left(l\right)},{\mathbf{u}}_k^{\left(l\right)}\left(V_k^{\prime }\right),{\mathbf{u}}_{I_{k-1}}^{\left(l\right)}\right)\in T_{ϵ}^l\left(Y_{I_{k-1}{I}_k},U_{I_k}\right)$$

  (A2)
• $E_6$: There is a decoding error in the $(n,k)$ MDS erasure code for $B_{uk}$.
• $E_7$: There is a decoding error in the $(n,n-1)$ MDS erasure code for $B_{n-1}$.

First by Remark A1, $P\left(E_6\right),P\left(E_7\right)\to 0$ as long as the rates before the MDS is scaled appropriately.

As in [4] we have $P\left(E_0\right)\to 0$ as $l\to \infty$. For $E_1$ as in [4] we define $E_{1i}$ as an encoding error on layer i given that the previous layers have been encoded correctly and in addition, here, that ${\mathbf{u}}_i^{\left(l\right)}$ has been encoded correctly. Then, as in [4], we find that $P\left(E_{1i}\right)\to 0$ if

$$\begin{array}{cc}\hfill n{R}_1^{\prime }& >nH\left(Y_{11}\right)-H\left({\mathbf{Y}}_{1I_n}\right|X,U_1)\hfill \\ \hfill n{R}_i^{\prime }& >nH\left(Y_{i1}\right)-H\left({\mathbf{Y}}_{i{I}_n}\right|X,{\mathbf{Y}}_{I_{i-1}{I}_n},{\mathbf{U}}_{I_{i-1}})\hfill \\ \hfill n{R}_{n-1}^{\prime }& >I(Y_{n-1};X,{\mathbf{Y}}_{I_{n-2}{I}_n}{\mathbf{U}}_{I_{n-2}})\hfill \end{array}$$

(A3)

with the difference being the addition of the $U_{\ast }$ variables. Similarly, we can define $E_{1i}^u$ as an encoding error of ${\mathbf{u}}_i^{\left(l\right)}$ given that the previous layers have been encoded correctly, and we similarly have that $P\left(E_{1i}^u\right)\to 0$ if

$$\begin{array}{cc}\hfill R_{u1}^{\prime }& >H\left(U_1\right)-H\left(U_1\right|X)\hfill \\ \hfill R_{ui}^{\prime }& >H\left(U_i\right)-H\left(U_i\right|X,{\mathbf{Y}}_{I_{i-1}{I}_n},{\mathbf{U}}_{I_{i-1}})\hfill \end{array}$$

(A4)

The proof that $P\left(E_2\right)\to 0$ is unchanged from [4], and the proof that $P\left(E_4\right)\to 0$ is similar.

The proof that $P\left(E_3\right)\to 0$ is similar to [4], except that at the time of decoding at layer k the decoder has access to ${\mathbf{u}}_{I_k}^{\left(l\right)}$. The relevant probability of decoding error at layer k is therefore $P\left(E_{3k}\right|{\mathbf{E}}_{3I_{k-1}}^c,E_2^c,E_4^c,{\mathbf{E}}_{5I_k}^c,E_6^c,E_7^c)$, and since we search for codewords in $T_{ϵ}^l\left(Y_{k{I}_k},Y_{I_{k-1}{I}_j},U_{I_k}\right)$, the condition for this error probability converging to zero is

$$R_k>R_k^{\prime }-H\left(Y_{k1}\right)+\frac{1}{k}H\left({\mathbf{Y}}_{k{I}_k}\right|{\mathbf{U}}_{I_k},{\mathbf{Y}}_{I_{k-1}{I}_k})$$

(A5)

instead of [4] (A17).

To prove that $P\left(E_5\right)\to 0$, we let $E_{5k}$ be the decoding error on layer k, and then bound $P_{5k}=P\left(E_{5k}\right|{\mathbf{E}}_{3I_{k-1}}^c,E_2^c,E_4^c,{\mathbf{E}}_{5I_{k-1}}^c,E_6^c,E_7^c)$. If we pick a random codeword ${\mathbf{u}}_k^{\left(l\right)}\in T_{ϵ}^l\left(U_k\right)$, the probability that this is jointly typical, i.e., the event (A2), is

$$P\le 2^{-l(I(U_k;Y_{I_{k-1}}{U}_{I_{k-1}})-\delta \left(ϵ\right))}$$

There are ${\xi }_{uk}=2^{l(R_{uk}^{\prime }-R_{uk}+{\gamma }_{uk})}$ elements in each bin, and therefore,

$$P_{5k}\le {\xi }_{uk}P$$

if we let ${\gamma }_{uk}>\delta \left(ϵ\right)$, we have $P_{5k}\to 0$ if

$$R_{uk}^{\prime }-R_{uk}<I(U_k;Y_{I_{k-1}}{U}_{I_{k-1}})$$

Together with (A4), this gives (5).

Appendix A.6. Analysis of Repair Error

If $E_4-E_7$, from above happen, there is also a repair error. Notice that at time of repair, we have access to $n-1$ nodes, and we can therefore use decoding for $n-1$ nodes, and in that case we have proven that ${\sum }_{i=4}^{7}P\left(E_i\right)\to 0$ as $l\to \infty$. We have the following additional repair error events:

• $E_{r1}$: Some $W_k=0$ for $k\in I_{n-2}$.
• $E_{r2}$: For $k\in I_{n-2}$, there exists another bin index $Q_{kn}^{\prime }$ in bin $W_k$ so that

  $$\begin{array}{c}\hfill ({\mathbf{y}}_{k{I}_n}^{\left(l\right)}\left({\mathbf{Q}}_{k{I}_n}^{\prime }\right),{\mathbf{y}}_{I_{k-1}{I}_n}^{\left(l\right)}.\ast \left({\mathbf{Q}}_{I_{k-1}{I}_n}\right),{\mathbf{u}}_{I_k}^{\left(l\right)}\left(V_{I_k}\right))\\ \hfill \in T_{ϵ}^l(Y_{I_k{I}_n},U_{I_k})\end{array}$$
• $E_{r3}$: For $k\in I_{n-2}$, there is a decoding error in the $(n,n-1)$ MDS erasure code for $W_k$.

Appendix A.6.1. Bounding E_r1

In total, for all bins, we pick $N=2^{l{R}_{rk}}{\xi }_{rk}=2^{l(n{R}_k^{\prime }+{\gamma }_{rk})}$ elements with replacement from a set of size $2^{nl{R}_{rk}^{\prime }}$. The probability that a particular element was never picked is then $P\left(E_{r1}\right)={\left(1-2^{-nl{R}_{rk}^{\prime }}\right)}^N$ and

$$\begin{array}{cc}\hfill logP\left(E_{r1}\right)& =Nlog\left(1-2^{-nl{R}_{rk}^{\prime }}\right)\le -N{2}^{-nl{R}_{rk}^{\prime }}\hfill \\ \hfill & =-2^{l{\gamma }_{rk}}\to -\infty \mathrm{as}l\to \infty \hfill \end{array}$$

Appendix A.6.2. Bounding E_r2

First, we will argue that if (A1) is satisfied, we can predict ${\mathbf{y}}_{kn}^{\left(l\right)}$ with probability approaching one. We can state this as follows: if we pick a random ${\mathbf{y}}_{kn}^{\left(l\right)}\in T_{ϵ}^l\left(Y_{kn}\right)$, what is the probability P that

$$\begin{array}{c}\hfill ({\mathbf{y}}_{kn}^{\left(l\right)},{\mathbf{y}}_{k{I}_{n-1}}^{\left(l\right)}\left({\mathbf{Q}}_{k{I}_{n-1}}\right),{\mathbf{y}}_{I_{k-1}{I}_n}^{\left(l\right)}.\ast \left({\mathbf{Q}}_{I_{k-1}{I}_n}\right),{\mathbf{u}}_{I_k}^{\left(l\right)}\left(V_{I_k}\right))\\ \hfill \in T_{ϵ}^l(Y_{I_k{I}_n},U_{I_k})\end{array}$$

(A6)

This is actually a standard channel coding problem, so we get

$$P\le 2^{-l(I(Y_{kn};Y_{k{I}_{n-1}},Y_{i{I}_{n-1}},U_{I_k})-\delta \left(ϵ\right))}$$

(A7)

Since the codebook ${\mathcal{C}}_{uk}$ has $2^{l{R}_k^{\prime }}$ elements, we then have

$$P\left(E_{r2k}\right)\le 2^{l{R}_k^{\prime }}P$$

Thus, $P\left(E_{r2k}\right)\to 0$ as $l\to \infty$ if

$$R_k^{\prime }<H\left(Y_{kn}\right)-H\left(Y_{kn}\right|Y_{k{I}_{n-1}},Y_{i{I}_{n-1}},U_{I_k})-\delta \left(ϵ\right)$$

Now in consideration of (A5) there is no gain from making $R_k^{\prime }$ larger than needed. Thus, $R_k^{\prime }$ is chosen arbitrarily close to the limit given by (A3), and we therefore have $P\left(E_{r2k}\right)\to 0$ if

$$\begin{array}{cc}\hfill & H\left(Y_{k1}\right)-\frac{1}{n}H\left({\mathbf{Y}}_{k{I}_n}\right|X,{\mathbf{Y}}_{I_{k-1}{I}_n},{\mathbf{U}}_{I_{k-1}})\hfill \\ \hfill & <H\left(Y_{kn}\right)-H\left(Y_{kn}\right|Y_{k{I}_{n-1}},Y_{i{I}_{n-1}},U_{I_k})-\delta \left(ϵ\right)\hfill \end{array}$$

which is (A1).

Now, turn to the case when (A1) is not satisfied. We look for vectors $(Q_{k1}^{\prime },Q_{k2}^{\prime },\dots ,Q_{kn}^{\prime })\in {\{1,\dots ,2^{l{R}_k^{\prime }}\}}^n$ that

• Are in the bin indicated by $W_k$.
• Has $Q_{ki}^{\prime }=Q_{ki}$, $i\le n-1$.
• Are jointly typical, i.e., satisfy (A6).

For condition 3, (A7) is still valid. Each bin contains ${\xi }_{rk}=2^{l(n{R}_k^{\prime }-R_{rk}+{\gamma }_{rk})}$ vectors. Each of these has probability $P_2=2^{-l(n-1)R_{lk}}$ of satisfying conditions 2. Therefore,

$$P\left(E_{r2k}\right)\le {\xi }_{rk}{P}_{2}P=2^{l(R_k^{\prime }-R_{rk}+{\gamma }_{rk})}P$$

if we choose ${\gamma }_{rk}>\delta \left(ϵ\right)$ we have $P\left(E_{r2k}\right)\to 0$ as $l\to \infty$ if

$$R_k^{\prime }-R_{rk}<H\left(Y_{kn}\right)-H\left(Y_{kn}\right|Y_{k{I}_{n-1}},Y_{i{I}_{n-1}},U_{I_k})$$

Which together with (A3) and the argument above leads to (6).