Side Information Design in Zero-Error Coding for Computing

Abstract

We investigate the zero-error coding for computing problems with encoder side information. An encoder provides access to a source X and is furnished with side information $g(Y)$. It communicates with a decoder that possesses side information Y and aims to retrieve $f(X,Y)$ with zero probability of error, where f and g are assumed to be deterministic functions. In previous work, we determined a condition that yields an analytic expression for the optimal rate $R^{*}(g)$; in particular, it covers the case where $P_{X,Y}$ is full support. In this article, we review this result and study the side information design problem, which consists of finding the best trade-offs between the quality of the encoder’s side information $g(Y)$ and $R^{*}(g)$. We construct two greedy algorithms that give an achievable set of points in the side information design problem, based on partition refining and coarsening. One of them runs in polynomial time.

1. Introduction

1.1. Zero-Error Coding for Computing

The problem of Figure 1 is a zero-error setting that relates to Orlitsky and Roche’s coding for computing problems from [1]. This coding problem appears in video compression [2,3], where $X^n$ models a set of images known at the encoder. The decoder does not always want to retrieve each whole image. Instead, the decoder receives, for each image $X_t,t\le n$, a request $Y_t$ to retrieve information $f(X_t,Y_t)$. This information can, for instance, be a detection: cat, dog, car, bike; or a scene recognition: street/city/mountain, etc. The encoder does not know the decoder’s exact request but has prior information about it (e.g., type of request), which is modeled by ${(g(Y_t))}_{t\le n}$. This problem also relates to the zero-error Slepian–Wolf open problem, which corresponds to the special case, where g is constant and $f(X,Y)=X$.

Figure 1. Zero-error coding for computing with side information at the encoder.

Similar schemes to the one depicted in Figure 1 have already been studied, but they differ from the one we are studying in two ways. First, they consider that no side information is available to the encoder. Second, and more importantly, they consider different coding constraints: the lossless case is studied by Orlitsky and Roche in [1], the lossy case by Yamamoto in [4], and the zero-error “unrestricted inputs” case by Shayevitz in [5]. The latter results can be used as bounds for our problem depicted in Figure 1, but do not exactly characterize its optimal rate.

Numerous extensions of the problem depicted in Figure 1 have been studied recently. The distributed context, for instance, has an additional encoder that encodes Y before transmitting it to the decoder. Achievability schemes have been proposed for this setting by Krithivasan and Pradhan in [6] using abelian groups; by Basu et al. in [7] using hypergraphs for the case with maximum distortion criterion; and by Malak and Médard in [8] using hyperplane separations for the continuous lossless case.

Another related context is the network setting, where the function of source random variables from source nodes has to be retrieved at the sink node of a given network. For tree networks, the feasible rate region is characterized by Feizi and Médard in [9] for networks of depth one, and by Sefidgaran and Tchamkerten in [10] under a Markov source distribution hypothesis. In [11], Ravi and Dey consider a bidirectional relay with zero-error “unrestricted inputs” and characterize the rate region for a specific class of functions. In [12], Guang et al. study zero-error function computation on acyclic networks with limited capacities, and give an inner bound based on network cut-sets. For both distributed and network settings, the zero-error coding for computing problems with encoder side information remains open.

In a previous work [13], we determined a condition that we called “pairwise shared side information” such that, if satisfied, the optimal rate $R^{*}(g)$ has a single-letter expression. This covers many cases of interest, in particular the case where $P_{X,Y}$ is full support for any functions $f,g$. For the sake of completeness, we review this result. Moreover, we propose an alternative and more interpretable expression for this pairwise shared side information. More precisely, we show that the instances where the “pairwise shared side information” condition is satisfied correspond to the worst possible optimal rates in an auxiliary zero-error Slepian–Wolf problem.

1.2. Encoder’s Side Information Design

In the zero-error coding for computing problems with encoder side information, it can be observed that a “coarse” encoder side information (e.g., if g constant) yields a high optimal rate $R^{*}(g)$, whereas a “fine” encoder side information (e.g., $g=Id$) yields a low optimal rate $R^{*}(g)$. The side information design problem consists of determining the best trade-offs between the optimal rate $R^{*}(g)$ and the quality of the encoder’s side information, which is measured by its entropy $H(g(Y))$. This expression describes the optimal rate of a zero-error code that transmits the quantized version of Y via the g function. The best trade-offs correspond to the Pareto front of the achievable set, i.e., whose corner-points cannot be obtained by a time sharing between other coding strategies. In short, we aim at determining the Pareto front of the convex hull of the achievable pairs $\left(H(g(Y)),R^{*}(g)\right)$.

In this article, we propose a greedy algorithm that gives an achievable set of points in the side information design problem, when $P_{X,Y}$ is full support. Studying our problem with the latter hypothesis is interesting because, unlike the case of the Slepian–Wolf problem, it does not necessarily correspond to a worst-case scenario. Recall indeed, that, when $P_{X,Y}$ is full support, the Slepian–Wolf encoder does not benefit from the side information available at the decoder and needs to send X. In our problem instead, if the retrieval function $f(X,Y)=Y$, since the decoder already has access to Y, no information needs to be sent by the encoder and the optimal rate is 0. Finally, the proposed algorithm relies on our results with “pairwise shared side information”, which gives the optimal rate for all functions g and performs a greedy partition coarsening when choosing the next achievable point. Moreover, it runs in polynomial time.

This paper is organized as follows. In Section 2, we formally present the zero-error coding for computing problems and the encoder’s side information design problem. In Section 3, we give our theoretic results on the zero-error coding for computing problems, including the “pairwise shared side information” condition. In Section 4, we present our greedy algorithms for the encoder’s side information design problem.

2. Formal Presentation of the Problem

We denote sequences by $x^n=(x_1,\dots ,x_n)$. The set of probability distributions over $\mathcal{X}$ is denoted by $\Delta (\mathcal{X})$. The distribution of X is denoted by $P_X\in \Delta (\mathcal{X})$ and its support is denoted by $\mathrm{supp}{P}_X$. Given the sequence length $n\in {\mathbb{N}}^{★}$, we denote by ${\Delta }_n(\mathcal{X})\subset \Delta (\mathcal{X})$ the set of empirical distributions of sequences from ${\mathcal{X}}^n$. We denote by ${\{0,1\}}^{*}$ the set of binary words. The collection of subsets of a set $\mathcal{Y}$ is denoted by $\mathcal{P}(\mathcal{Y})$.

Definition 1.

The zero-error source-coding problem of Figure 1 is described by the following:

-

Four finite sets $\mathcal{U}$, $\mathcal{X}$, $\mathcal{Y}$, $\mathcal{Z}$ and a source distribution $P_{X,Y}\in \Delta (\mathcal{X}\times \mathcal{Y})$.

-

For all $n\in {\mathbb{N}}^{★}$, $(X^n,Y^n)$ is the random sequence of n copies of $(X,Y)$, drawn in an i.i.d. fashion using $P_{X,Y}$.

-

Two deterministic functions

$$\begin{array}{cc}\hfill & f:\mathcal{X}\times \mathcal{Y}\to \mathcal{U},\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & g:\mathcal{Y}\to \mathcal{Z}.\hfill \end{array}$$

(2)

-

An encoder that knows $X^n$ and ${\left(g(Y_t)\right)}_{t\le n}$ sends binary strings over a noiseless channel to a decoder that knows $Y^n$ and that wants to retrieve ${\left(f(X_t,Y_t)\right)}_{t\le n}$ without error.

A coding scheme in this setting is described by:

-

A time horizon $n\in {\mathbb{N}}^{★}$ and an encoding function ${\varphi }_e:{\mathcal{X}}^n\times {\mathcal{Z}}^n\to {\{0,1\}}^{*}$ such that $Im{\varphi }_e$ is prefix-free.

-

A decoding function ${\varphi }_d:{\mathcal{Y}}^n\times {\{0,1\}}^{*}\to {\mathcal{U}}^n$.

-

The rate is the average length of the codeword per source symbol,

i.e., $R\doteq \frac{1}{n}\mathbb{E}\left[\ell \circ {\varphi }_e\left(X^n,{(g(Y_t))}_{t\le n}\right)\right]$, where ℓ denotes the codeword length function.

-

n, ${\varphi }_e$, ${\varphi }_d$ must satisfy the zero-error property:

$$\begin{array}{c}\hfill \mathbb{P}\left({\varphi }_d\left(Y^n,{\varphi }_e\left(X^n,{(g(Y_t))}_{t\le n}\right)\right)\ne {\left(f(X_t,Y_t)\right)}_{t\le n}\right)=0.\end{array}$$

(3)

The minimal rate under the zero-error constraint is defined by

$$\begin{array}{c}\hfill R^{*}(g)\doteq \underset{\begin{array}{c}n,{\varphi }_e,{\varphi }_d\\ \mathit{zero}-\mathit{error}\end{array}}{inf}\frac{1}{n}\mathbb{E}\left[\ell \circ {\varphi }_e\left(X^n,{(g(Y_t))}_{t\le n}\right)\right].\end{array}$$

(4)

The definition of the Pareto front that we give below is adapted to the encoder’s side information design problem and allows us to describe the best trade-off between the quality of the encoder side information and the rate to compute the function $f(X,Y)$ at the decoder. In other works, the definition of a Pareto front may differ depending on the minimization/maximization problem considered and on the number of variables to be optimized.

Definition 2

(Pareto front). Let $\mathcal{S}\subset {\mathbb{R}}_{+}^2$ be a set, the Pareto front of $\mathcal{S}$ is defined by

$$\begin{array}{c}\hfill Par(\mathcal{S})\doteq \left\{x\in \mathcal{S}|\forall x^{\prime }\in \mathcal{S}\setminus \left\{x\right\},x_1^{\prime }>x_1\mathrm{or}{x}_2^{\prime }>x_2\right\}.\end{array}$$

(5)

Definition 3.

The side information design problem in Figure 1 consists of determining the Pareto front of the achievable pairs $(H(g(Y)),R^{*}(g))$:

$$\begin{array}{c}\hfill \mathcal{F}\doteq Par\left(Conv\left\{\left(H(g(Y)),R^{*}(g)\right)|g:\mathcal{Y}\to \mathcal{Z}\right\}\right),\end{array}$$

(6)

where Conv denotes the convex hull.

In our zero-error setup, all alphabets are finite. Therefore, the Pareto front of the convex hull in (6) is computed on a finite set of points, which correspond to the best trade-offs for the encoder’s side information.

3. Theoretic Results

Determining the optimal rate in the zero-error coding for computing problems, with or without encoder side information, is an open problem. In a previous contribution [13], we determined a condition that, when satisfied, yields an analytic expression for the optimal rate. Interestingly, this condition is general as it does not depend on the function f to be retrieved at the decoder.

3.1. General Case

We first build the characteristic graph $G_{\left[n\right]}$, which is a probabilistic graph that captures the zero-error encoding constraints on a given number n of source uses. It differs from the graphs used in [5], as we do not need a cartesian representation of these graphs to study the optimal rates. Furthermore, it has a vertex for each possible realization of $\left(X^n,{\left(g(Y_t)\right)}_{t\le n}\right)$ known at the encoder, instead of ${\mathcal{X}}^n$ as in the zero-error Slepian–Wolf problem [14].

Definition 4

(Characteristic graph $G_{\left[n\right]}$). The characteristic graph $G_{\left[n\right]}$ is defined by the following:

-

${\mathcal{X}}^n\times {\mathcal{Z}}^n$ as a set of vertices with distribution $P_{X,g(Y)}^n$.

-

$(x^n,z^n)(x^{\prime n},z^{\prime n})$ are adjacent if $z^n=z^{\prime n}$ and there exists $y^n\in g^{-1}(z^n)$ such that

$$\begin{array}{cc}\hfill & \forall t\le n,P_{X,Y}(x_t,y_t)P_{X,Y}(x_t^{\prime },y_t)>0,\hfill \end{array}$$

(7)

$$\mathit{and} \exists t\le n,f(x_t,y_t)\ne f(x_t^{\prime },y_t);$$

(8)

where $g^{-1}(z^n)=\left\{y^n\in {\mathcal{Y}}^n|{\left(g(y_t)\right)}_{t\le n}=z^n\right\}$.

The characteristic graph $G_{\left[n\right]}$ is designed with the same core idea as in [15]: $(x^n,z^n)$ and $(x^{\prime n},z^{\prime n})$ are adjacent if there exists a side information symbol $y^n$ compatible with the observation of the encoder (i.e., $z^n=z^{\prime n}$ and $y^n\in g^{-1}(z^n)$), such that $f(x^n,y^n)\ne f(x^{\prime n},y^n)$. In order to prevent erroneous decodings, the encoder must map adjacent pairs of sequences to different codewords; hence the use of graph colorings, defined below.

Definition 5

(Coloring, independent subset). Let $G=(\mathcal{V},\mathcal{E},P_V)$ be a probabilistic graph. A subset $\mathcal{S}\subseteq \mathcal{V}$ is independent if $x{x}^{\prime }\notin \mathcal{E}$ for all $x,x^{\prime }\in \mathcal{S}$. Let $\mathcal{C}$ be a finite set (the set of colors), a mapping $c:\mathcal{V}\to \mathcal{C}$ is a coloring if $c^{-1}(i)$ is an independent subset for all $i\in \mathcal{C}$.

The chromatic entropy of $G_{\left[n\right]}$ gives the best rate of n-shot zero-error encoding functions, as in [14].

Definition 6

(Chromatic entropy $H_{\chi }$). The chromatic entropy of a probabilistic graph $G=(\mathcal{V},\mathcal{E},P_V)$ is defined by

$$\begin{array}{c}\hfill H_{\chi }(G)=inf\left\{H\left(c(V)\right)|c\mathit{is}a\mathit{coloring}\mathit{of}G\right\}.\end{array}$$

(9)

Theorem 1

(Optimal rate). The optimal rate is written as follows:

$$R^{*}(g)=\underset{n\to \infty }{lim}\frac{1}{n}{H}_{\chi }(G_{\left[n\right]}).$$

(10)

Proof. 

By construction, the following holds: for all encoding functions ${\varphi }_e$, ${\varphi }_e$ is a coloring of $G_{\left[n\right]}$ if and only if there exists a decoding function ${\varphi }_d$ such that $(n,{\varphi }_e,{\varphi }_d)$ satisfies the zero-error property. Thus, the best achievable rate is written as follows:

$$\begin{array}{cc}\hfill R^{*}(g)=& \underset{n}{inf}\underset{{\varphi }_e\hspace{0.17em}\mathrm{coloring}\hspace{0.17em}\mathrm{of}\hspace{0.17em}{G}_{\left[n\right]}}{inf}H\left({\varphi }_e\left(X^n,{\left(g(Y_t)\right)}_{t\le n}\right)\right)\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill =& \underset{n\to \infty }{lim}\frac{1}{n}{H}_{\chi }(G_{\left[n\right]}).\hfill \end{array}$$

(12)

where (12) comes from Fekete’s Lemma and from the definition of $H_{\chi }$.    □

A general single-letter expression for $R^{*}(g)$ is missing due to the lack of the intrinsic structure of $G_{\left[n\right]}$. In Section 3.2, we introduce a hypothesis that gives structure to $G_{\left[n\right]}$ and allows us to derive a single-letter expression for $R^{*}(g)$.

3.2. Pairwise Shared Side Information

Definition 7.

The distribution $P_{X,Y}$ and the function g satisfy the “pairwise shared side information” condition if

$$\begin{array}{c}\hfill \forall z\in \mathcal{Z},\forall x,x^{\prime }\in \mathcal{X},\exists y\in g^{-1}(z),P_{XY}(x,y)P_{XY}(x^{\prime },y)>0,\end{array}$$

(13)

where $Im(g)$ is the image of the function g. This means that for all z output of g, every pair $(x,x^{\prime })$ “shares” at least one side information symbol $y\in g^{-1}(z)$.

Note that any full-support distribution $P_{X,Y}$ satisfies the “pairwise shared side information” hypothesis. In Theorem 2, we give an interpretation of the “pairwise shared side information” condition in terms of the optimal rate in an auxiliary zero-error Slepian–Wolf problem.

Theorem 2.

The tuple $(P_{X,Y},g)$ satisfies the condition “pairwise shared side information” (13)

 ⟺ $R^{*}(g)=H(X|g(Y))$ in the case $f(X,Y)=X$, and for all $z\in \mathcal{Z}$, $P_{X|g(Y)=z}$ is full support.

The proof of Theorem 2 is given in Appendix A.1.

Definition 8

(AND, OR product). Let $G_1=({\mathcal{V}}_1,{\mathcal{E}}_1,P_{V_1})$, $G_2=({\mathcal{V}}_2,{\mathcal{E}}_2,P_{V_2})$ be two probabilistic graphs; their AND (resp. OR) product denoted by $G_1\wedge G_2$ (resp. $G_1\vee G_2$) is defined by the following: ${\mathcal{V}}_1\times {\mathcal{V}}_2$ as a set of vertices, $P_{V_1}{P}_{V_2}$ as probability distribution on the vertices, and $(v_1{v}_2),(v_1^{\prime }{v}_2^{\prime })$ are adjacent if

$$\begin{array}{c}{v}_1{v}_1^{\prime }\in {\mathcal{E}}_1\mathit{AND}{v}_2{v}_2^{\prime }\in {\mathcal{E}}_2,\\ \mathit{resp}.(v_1{v}_1^{\prime }\in {\mathcal{E}}_1\mathit{and}{v}_1\ne v_1^{\prime })\mathit{OR}(v_2{v}_2^{\prime }\in {\mathcal{E}}_2\mathit{and}{v}_2\ne v_2^{\prime });\end{array}$$

(14)

with the convention that all vertices are self-adjacent. We denote by $G_1^{\wedge n}$ (resp. $G_1^{\vee n}$) the n-th AND (resp. OR) power.

AND and OR powers significantly differ in terms of existing single-letter expression for the associated asymptotic chromatic entropy. Indeed, in the zero-error Slepian–Wolf problem in [14], the optimal rate ${lim}_{n\to \infty }\frac{1}{n}{H}_{\chi }(G^{\wedge n})$, which relies on an AND power, does not have a single-letter expression. Instead, closed-form expressions for OR powers of graphs exist. More precisely, as recalled in Proposition 1, ${lim}_{n\to \infty }\frac{1}{n}{H}_{\chi }(G^{\vee n})$ admits a single-letter expression called the Körner graph entropy, introduced in [16], and defined below. This observation is key for us to derive a single-letter expression for our problem. More precisely, by using a convex combination of Körner graph entropies, we provide a single-letter expression in Theorem 3 for the optimal rate $R^{*}(g)$.

Definition 9

(Körner graph entropy $H_{\kappa }$). For all $G=(\mathcal{V},\mathcal{E},P_V)$, let $\Gamma (G)$ be the collection of independent sets of vertices in G. The Körner graph entropy of G is defined by

$$\begin{array}{c}\hfill H_{\kappa }(G)=\underset{V\in W\in \Gamma (G)}{min}I(W;V),\end{array}$$

(15)

where the minimum is taken over all distributions $P_{W|V}\in \Delta {(\mathcal{W})}^{\mathcal{V}}$, with $\mathcal{W}=\Gamma (G)$ and the constraint that the random vertex V belongs to the random set W with probability one.

Below, we recall that the limit of the normalized chromatic entropy of the OR product of graphs admits a closed-form expression given by the Körner graph entropy $H_{\kappa }$. Moreover, the Körner graph entropy of OR products of graphs is simply the sum of the individual Körner graph entropies.

Proposition 1

(Properties of $H_{\kappa }$). Theorem 5 in [14] for all probabilistic graphs G and $G^{\prime }$,

$$\begin{array}{cc}\hfill & H_{\kappa }(G)=\underset{n\to \infty }{lim}\frac{1}{n}{H}_{\chi }(G^{\vee n}),\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & H_{\kappa }(G\vee G^{\prime })=H_{\kappa }(G)+H_{\kappa }(G^{\prime }).\hfill \end{array}$$

(17)

Definition 10

(Auxiliary graph $G_z^f$). For all $z\in \mathcal{Z}$, we define the auxiliary graph $G_z^f$ by

-

$\mathcal{X}$ as set of vertices with distribution $P_{X|g(Y)=z}$;

-

$x{x}^{\prime }$ are adjacent if $f(x,y)\ne f(x^{\prime },y)$ for some $y\in g^{-1}(z)\cap supp{P}_{Y|X=x}\cap supp{P}_{Y|X=x^{\prime }}$.

Theorem 3

(Pairwise shared side information). If $P_{X,Y}$ and g satisfy (13), the optimal rate is written as follows:

$$R^{*}(g)=\sum _{z\in \mathcal{Z}}{P}_{g(Y)}(z)H_{\kappa }(G_z^f).$$

(18)

The proof is in Appendix A.2, the keypoint is the particular structure of $G_{\left[n\right]}$: a disjointed union of OR products.

Remark 1.

The “pairwise shared side information” assumption (13) implies that the adjacency condition (7) is satisfied, which makes $G_{\left[n\right]}$ a disjoint union of OR products. Moreover, Körner graph entropies appear in the final expression for $R^{*}(g)$, even if $G_{\left[n\right]}$ is not an n-th OR power.

Now, consider the case where $P_{X,Y}$ is full support. This is a sufficient condition to have (13). The optimal rate in this setting is derived from Theorem 3, which leads to the analytic expression in Theorem 4.

Theorem 4

(Optimal rate when $P_{X,Y}$ is full support). When $P_{X,Y}$ is full support, the optimal rate is written as follows:

$$\begin{array}{cc}\hfill R^{*}(g)=& H\left(j(X,g(Y))|g(Y)\right),\hfill \end{array}$$

(19)

where the function j returns a word in ${\mathcal{U}}^{*}$, defined by

$$\begin{array}{cc}\hfill j:& \mathcal{X}\times \mathcal{Z}\to {\mathcal{U}}^{*}\hfill \\ \hfill & (x,z)\mapsto {\left(f(x,y^{\prime })\right)}_{y^{\prime }\in g^{-1}(z)}.\hfill \end{array}$$

(20)

Proof. 

By Theorem 3, $R^{*}(g)={\sum }_{z\in \mathcal{Z}}{P}_{g(Y)}(z)H_{\kappa }(G_z^f)$. It can be shown that $G_z^f$ is complete multipartite for all z as $P_{X,Y}$ is full support; and it satisfies $H_{\kappa }(G_z^f)=H\left(j(X,g(Y))|g(Y)=z\right)$.    □

3.3. Example

In this example, the “pairwise shared side information” assumption is satisfied and $R^{*}(g)$ is strictly less than a conditional Huffman coding of X knowing $g(Y)$; and also strictly less than the optimal rate without exploiting $g(Y)$ at the encoder.

Consider the probability distribution and function outcomes depicted in Figure 2, with $\mathcal{U}=\{a,b,c\}$, $\mathcal{X}=\{0,\dots ,3\}$, $\mathcal{Y}=\{0,\dots ,7\}$, and $\mathcal{Z}=\{0,1\}$. Let us show that the “pairwise shared side information” assumption is satisfied. The source symbols $0,1,2\in \mathcal{X}$ share the side information symbol 0 (resp. 5) when $g(Y)=0$ (resp. $g(Y)=1$). The source symbol $3\in \mathcal{X}$ shares the side information symbols $1,2,3$ with the source symbols $0,1,2$, respectively, when $g(Y)=0$, and the source symbol 3 shares the side information symbol 5 with all other source symbols when $g(Y)=1$.

Figure 2. An example of $P_{X,Y}$ and g that satisfies (13), along with the outcomes $f(X,Y)$. The elements outside $supp{P}_{X,Y}$ are denoted by *.

Since the “pairwise shared side information” assumption is satisfied, we can use Theorem 3; the optimal rate is written as follows:

$$\begin{array}{c}\hfill R^{*}(g)=P_{g(Y)}(0)H_{\kappa }(G_0^f)+P_{g(Y)}(1)H_{\kappa }(G_1^f).\end{array}$$

(21)

First, we need to determine the probabilistic graphs $G_0^f$ and $G_1^f$. In $G_0^f$, the vertex 0 is adjacent to 2 and 3, as $f(0,0)\ne f(2,0)$ and $f(0,1)\ne f(3,1)$. The vertex 1 is also adjacent to 2 and 3 as $f(1,0)\ne f(2,0)$ and $f(1,2)\ne f(3,2)$. Furthermore $P_{X|g(Y)=0}$ is uniform, hence $G_0^f=(C_4,Unif(\mathcal{X}))$ where $C_4$ is the cycle graph with 4 vertices.

In $G_1^f$, the vertices 1, 2, 3 are pairwise adjacent as $f(1,5)$, $f(2,5)$ and $f(3,5)$ are pairwise different; and 0 is adjacent to 1, 2, and 3 because of the different function outputs generated by $Y=4$ and $Y=5$. Thus, $G_1^f=(K_4,P_{X|g(Y)=1})$ with $P_{X|g(Y)=1}=(\frac{1}{4},\frac{3}{8},\frac{1}{8},\frac{1}{4})$, and $K_4$ is the complete graph with 4 vertices.

Now, let us determine $H_{\kappa }(G_0^f)$ and $H_{\kappa }(G_1^f)$. On the one hand,

$$\begin{array}{cc}\hfill H_{\kappa }(G_0^f)& =H(V_0)-\underset{V_0\in W\in \Gamma (G_0^f)}{max}H(V_0|W)\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill & =2-1=1,\hfill \end{array}$$

(23)

with $V_0\sim P_{X|g(Y)=0}=Unif(\mathcal{X})$; and where $H(V_0|W)$ in (22) is maximized by taking $W=\{0,1\}$ when $V\in \{0,1\}$, and $W=\{2,3\}$ otherwise.

On the other hand,

$$\begin{array}{cc}\hfill H_{\kappa }(G_1^f)& =\underset{V_1\in W\in \Gamma (G_1^f)}{min}I(W;V_1)\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & =H(V_1)\approx 1.906,\hfill \end{array}$$

(25)

with $V_1\sim P_{X|g(Y)=1}$; where (25) follows from $\Gamma (G_1^f)=\{\left\{0\right\},\dots ,\left\{3\right\}\}$, as $G_1^f$ is complete. Hence, $R^{*}(g)\approx 1.362$.

The rate that we would obtain by transmitting X knowing $g(Y)$ at both encoder and decoder with a conditional Huffman algorithm is written as $R_{\mathrm{Huff}}=H(X|g(Y))\approx 1.962$.

The rate that we would obtain without exploiting $g(Y)$ at the encoder is $R_{\mathrm{No}g}=H(X)\approx 1.985$ because of the different function outputs generated by $Y=4$ and $Y=5$.

Finally, $H(f(X,Y)|Y)\approx 0.875$.

In this example, we have

$$\begin{array}{c}\hfill H(X)=R_{\mathrm{No}g}>R_{\mathrm{Huff}}>R^{*}(g)>H(f(X,Y)|Y).\end{array}$$

(26)

This illustrates the impact of the side information at the encoder in this setting, as we can observe a large gap between the optimal rate $R^{*}(g)$ and $R_{\mathrm{No}g}$.

4. Optimization of the Encoder Side Information

4.1. Preliminary Results on Partitions

In order to optimize the function g in the encoder side information, we propose a new equivalent characterization of the function g in the form of a partition of the set $\mathcal{Y}$. The equivalence is shown in Proposition 2 below.

Proposition 2.

For all $g:\mathcal{Y}\to \mathcal{Z}$, the collection of subsets ${(g^{-1}(z))}_{z\in \mathcal{Z}}$ is a partition of $\mathcal{Y}$.

Conversely, if $\mathcal{A}\subset \mathcal{P}(\mathcal{Y})$ is a partition of $\mathcal{Y}$, then there exists a mapping $g_{\mathcal{A}}:\mathcal{Y}\to \mathcal{Z}$ such that $\forall z\in Im{g}_{\mathcal{A}},\exists A_z\in \mathcal{A},A_z=g_{\mathcal{A}}^{-1}(z)$.

Proof. 

The direct part results directly from the fact that g is a function. For the converse part, we take $\mathcal{Z}$ such that $\left|\mathcal{Z}\right|=\left|\mathcal{A}\right|$ and we define $g_{\mathcal{A}}:\mathcal{Y}\to \mathcal{Z}$ by $g_{\mathcal{A}}(y)=z$, where $z\in \mathcal{Z}$ is the unique index such that $y\in A_z$. The property $\forall z\in Im{g}_{\mathcal{A}},\exists A_z\in \mathcal{A},A_z=g_{\mathcal{A}}^{-1}(z)$ is therefore satisfied.    □

Now, let us define coarser and finer partitions, with the corresponding notions of merging and splitting. These operations on partitions are the core idea of our greedy algorithms; as illustrated in Proposition 2, the partitions of $\mathcal{Y}$ correspond to functions $g:\mathcal{Y}\to \mathcal{Z}$ for the encoder’s side information. Therefore, obtaining a partition from another means finding another function $g:\mathcal{Y}\to \mathcal{Z}$ for the encoder’s side information.

Definition 11

(Coarser, Finer). Let $\mathcal{A},\mathcal{B}\subset \mathcal{P}(\mathcal{Y})$ be two partitions of the finite set $\mathcal{Y}$. We say that $\mathcal{A}$ is coarser than $\mathcal{B}$ if

$$\begin{array}{c}\hfill \forall B\in \mathcal{B},\exists A\in \mathcal{A},B\subset A.\end{array}$$

(27)

If so, we also say that $\mathcal{B}$ is finer than $\mathcal{A}$.

Example 1.

Let $\mathcal{Y}=\{1,2,3,4\}$, the partition $\mathcal{A}=\left\{\left\{1\right\},\{2,3,4\}\right\}$ is coarser than $\mathcal{B}=\left\{\left\{1\right\},\left\{2\right\},\{3,4\}\right\}$.

Definition 12

(Merging, Splitting). A merging is an operation that maps a partition $\mathcal{A}=\{A_1,\dots ,A_i,\dots ,A_j,\dots ,A_m\}$ to the partition ${\mathcal{A}}^{\prime }=\{A_1,\dots ,A_i\cup A_j,\dots ,A_m\}$. A splitting in an operation that maps a partition $\mathcal{A}=\{A_1,\dots ,A_i,\dots ,A_m\}$ to the partition ${\mathcal{A}}^{\prime }=\{A_1,\dots ,A_i^{(1)},A_i^{(2)},\dots ,A_m\}$, where $\{A_i^{(1)},A_i^{(2)}\}$ form a partition of the subset $A_i$.

We also define the set of partitions $Merge(\mathcal{A})$ (resp. $Split(\mathcal{A})$), which correspond to all partitions that can be obtained with a merging (resp. splitting) of $\mathcal{A}$:

$$\begin{array}{cc}\hfill & Merge(\mathcal{A})\doteq \left\{m(\mathcal{A})|m\mathit{is}\mathrm{a}\mathit{merging}\right\};\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill & Split(\mathcal{A})\doteq \left\{s(\mathcal{A})|s\mathit{is}\mathrm{a}\mathit{splitting}\right\}.\hfill \end{array}$$

(29)

Proposition 3.

If $\mathcal{A}$ is coarser (resp. finer) than $\mathcal{B}$, then $\mathcal{A}$ can be obtained from $\mathcal{B}$ by performing a finite number of mergings (resp. splittings).

4.2. Greedy Algorithms Based on Partition Coarsening and Refining

In this Section, we assume $P_{X,Y}$ to be full support.

With Proposition 2, we know that determining the Pareto front by a brute force approach would at least require to enumerate the partitions of $\mathcal{Y}$. Therefore, the complexity of this approach is exponential in $\left|\mathcal{Y}\right|$. In the following we describe the greedy Algorithms 1 and 2 that give an achievable set for the encoder’s side information design problem; one of them has a polynomial complexity. Then we give an example where the Pareto front coincides with the boundary of the convex hull of the achievable rate region obtained by both greedy algorithms.

Algorithm 1 Greedy coarsening algorithm

1: $\mathcal{A}\leftarrow \left\{\left\{1\right\},\dots \left\{\right|\mathcal{Y}\left|\right\}\right\}$       // $\mathcal{A}$ starts by being the finest partition of $\mathcal{Y}$, i.e., $g_{\mathcal{A}}=Id$. 2: Front$\leftarrow [\mathcal{A},\mathrm{ndef},\dots ,\mathrm{ndef}]$       // Will contain the list of the $\left|\mathcal{Y}\right|$ partitions chosen during the execution 3:   4: for  $i\in \{1,\dots ,|\mathcal{Y}|-1\}$ do 5:     // Maximize over $\mathcal{B}$ merging of $\mathcal{A}$ the slope between $\left(H(g_{\mathcal{B}}(Y)),R^{*}(g_{\mathcal{B}})\right)$ and $\left(H(g_{\mathcal{A}}(Y)),R^{*}(g_{\mathcal{A}})\right)$. 6:     $\mathcal{A}\leftarrow {argmax}_{\mathcal{B}\in Merge(\mathcal{A})}\frac{R^{*}(g_{\mathcal{B}})-R^{*}(g_{\mathcal{A}})}{H(g_{\mathcal{B}}(Y))-H(g_{\mathcal{A}}(Y))}$ 7:     Front$\left[i\right]\leftarrow \mathcal{A}$ 8:   9: return Front       // $\mathcal{A}=\left\{\{1,\dots ,|\mathcal{Y}\left|\right\}\right\}$ at this point

Algorithm 2 Greedy refining algorithm

1: $\mathcal{A}\leftarrow \left\{\{1,\dots ,|\mathcal{Y}\left|\right\}\right\}$       // $\mathcal{A}$ starts by being the coarsest partition of $\mathcal{Y}$, i.e., $g_{\mathcal{A}}=Id$. 2: Front $\leftarrow [\mathcal{A},\mathrm{ndef},\dots ,\mathrm{ndef}]$       // Will contain the list of the $\left|\mathcal{Y}\right|$ partitions chosen during the execution 3:   4: for $i\in \{1,\dots ,|\mathcal{Y}|-1\}$ do 5:     // Minimize over $\mathcal{B}$ splitting of $\mathcal{A}$ the slope between $\left(H(g_{\mathcal{A}}(Y)),R^{*}(g_{\mathcal{A}})\right)$ and $\left(H(g_{\mathcal{B}}(Y)),R^{*}(g_{\mathcal{B}})\right)$. 6:     $\mathcal{A}\leftarrow {argmin}_{\mathcal{B}\in Split(\mathcal{A})}\frac{R^{*}(g_{\mathcal{B}})-R^{*}(g_{\mathcal{A}})}{H(g_{\mathcal{B}}(Y))-H(g_{\mathcal{A}}(Y))}$ 7:     Front$\left[i\right]\leftarrow \mathcal{A}$ 8:   9: return Front      // $\mathcal{A}=\left\{\left\{1\right\},\dots \left\{\right|\mathcal{Y}\left|\right\}\right\}$ at this point

In these a argmin (resp. argmax) means any minimizer (resp. maximizer) of the specified quantity; and the function $g_{\mathcal{A}}:\mathcal{Y}\to \mathcal{Z}$ is a function for the encoder’s side information corresponding to the partition $\mathcal{A}$, whose existence is given by Proposition 2.

The coarsening (resp. refining) algorithm starts by computing its first achievable point $\left(H(g_{\mathcal{A}}(Y)),R^{*}(g_{\mathcal{A}})\right)$ with $\mathcal{A}$ being the finest (resp. coarsest) partition: it evaluates $R^{*}(g_{\mathcal{A}})$, with $g_{\mathcal{A}}=Id$ (resp. $g_{\mathcal{A}}$ constant); and $H(g_{\mathcal{A}}(Y))=H(Y)$ (resp. $H(g_{\mathcal{A}}(Y))=0$). Then, at each iteration, the next achievable point will be computed by using a merging (resp. splitting) of the current partition $\mathcal{A}$. The next partition will be a coarser (resp. finer) partition chosen from $Merge(\mathcal{A})$ (resp. $Split(\mathcal{A})$), following a greedy approach. Since we want to achieve good trade-offs between $H(g_{\mathcal{A}}(Y))$ and $R^{*}(g_{\mathcal{A}})$, we want to decrease $H(g(Y))$ (resp. $R^{*}(g_{\mathcal{A}})$) as much as possible while increasing the other quantity as less as possible. We do so by maximizing over $\mathcal{B}\in Merge(\mathcal{A})$ the negative ratio

$$\begin{array}{c}\hfill \frac{R^{*}(g_{\mathcal{B}})-R^{*}(g_{\mathcal{A}})}{H(g_{\mathcal{B}}(Y))-H(g_{\mathcal{A}}(Y))},\end{array}$$

(30)

resp. minimizing over $\mathcal{B}\in Split(\mathcal{A})$ the negative ratio

$$\begin{array}{c}\hfill \frac{R^{*}(g_{\mathcal{B}})-R^{*}(g_{\mathcal{A}})}{H(g_{\mathcal{B}}(Y))-H(g_{\mathcal{A}}(Y))};\end{array}$$

(31)

hence the use of slope maximization (resp. minimization) in the algorithm. At the end, the set of achievable points computed by the algorithm is returned.

In Figure 3, we show rate pairs associated with all possible partitions of $\mathcal{Y}$: a point corresponds to a partition of $\mathcal{Y}$, its position gives the associated rates $\left(H(g(Y)),R^{*}(g)\right)$. Two points are linked if their corresponding partitions $\mathcal{A},\mathcal{B}$ satisfy $\mathcal{A}\in Merge(\mathcal{B})$ or $\mathcal{A}\in Split(\mathcal{B})$. The obtained graph is the Hasse diagram for the partial order “coarser than”. Note that due to symmetries in the chosen example, several points associated with different partitions may overlap. In Figure 4, (resp. Figure 5), we give an illustration of the trajectory of the greedy coarsening (resp. refining) algorithm.

Figure 3. An illustration of the rate pairs associated with all partitions of $\mathcal{Y}$. The Pareto front is the broken line corresponding to the partitions $p_1$–$p_2$–$p_3$–$p_4$–$p_5$; with $p_1=\left\{\{1,2,3,4\}\right\}$, $p_2=\left\{\{1,2,4\},\left\{3\right\}\right\}$, $p_3=\left\{\{1,2\},\{3,4\}\right\}$, $p_4=\left\{\{1,2\},\left\{3\right\},\left\{4\right\}\right\}$, $p_5=\left\{\left\{1\right\},\left\{2\right\},\left\{3\right\},\left\{4\right\}\right\}$.

Figure 4. An illustration of the trajectory of the coarsening greedy algorithm (blue), with the Pareto front of the achievable rates (dashed red).

Figure 5. An illustration of the trajectory of the refining greedy algorithm (green), with the Pareto front of the achievable rates (dashed red).

Figure 3, Figure 4 and Figure 5 are obtained with the following problem data:

$$\begin{array}{c}\hfill P_{X,Y}=Unif(\mathcal{X}\times \mathcal{Y})f(·,·)=\left(\begin{array}{cccc}0& 0& 0& 1\\ 0& 0& 1& 1\\ 1& 1& 0& 0\\ 1& 1& 1& 1\end{array}\right).\end{array}$$

(32)

As stated in Theorem 5, the complexity of the coarsening greedy algorithm is polynomial since $|Merge(\mathcal{A})|$ is quadratic in $\left|\mathcal{Y}\right|$ and the evaluation of $R^{*}(g)$ can be conducted in polynomial time. This polynomial complexity property is not satisfied by the refining greedy algorithm, as $|Split(\mathcal{A})|$ is exponential in $\left|\mathcal{Y}\right|$.

Theorem 5.

The coarsening greedy algorithm runs in polynomial time in $\left|\mathcal{Y}\right|$. The refining greedy algorithm runs in exponential time in $\left|\mathcal{Y}\right|$.

Proof. 

The number of points evaluated by the coarsening (resp. refining) greedy algorithm is $O(|\mathcal{Y}{|}^3)$ (resp. $O(2^{\left|\mathcal{Y}\right|})$): $O(|\mathcal{Y}|)$ mergings (resp. splittings) are made; and for each of these mergings, all points from $Merge(\mathcal{A})$ (resp. $Split(\mathcal{A})$) are evaluated; they are, at most, $(\genfrac{}{}{0pt}{}{\left|\mathcal{Y}\right|}{2})={O(|\mathcal{Y}|}^2)$ (resp. $O(2^{\left|\mathcal{Y}\right|})$, in the worst case $\mathcal{A}=\left\{\{1,\dots ,|\mathcal{Y}\left|\right\}\right\}$). Since the expression $R^{*}(g)=H\left(j(X,g(Y))|g(Y)\right)$ from Theorem 4 allows for an evaluation of $R^{*}(g)$ in polynomial time in $\left|\mathcal{Y}\right|$, the coarsening (resp. refining) greedy algorithm has a polynomial (resp. exponential) time complexity.    □

Appendix A.1. Proof of Theorem 2

Consider the particular case $f(X,Y)=X$ of Figure 1. The optimal rate in this particular case equals the optimal rate $R^{*}(g)$ in the following auxiliary problem, depicted in Figure A1: $(X,g(Y))$ as source available at the encoder and to be retrieved by the decoder which knows Y (thus, expecting it to retrieve $g(Y)$ in addition to X does not change the optimal rate).

Figure A1. An auxiliary zero-error Slepian–Wolf problem.

Definition A1

(Characteristic graph for the zero-error Slepian–Wolf problem). Let $\mathcal{X},\mathcal{Y}$ be two finite sets and $P_{Y|X}$ be a conditional distribution from $\Delta {(\mathcal{Y})}^{\left|\mathcal{X}\right|}$. The characteristic graph associated with $P_{Y|X}$ is defined by the following:

-

$\mathcal{X}$ as set of vertices;

-

$x,x^{\prime }\in \mathcal{X}$ are adjacent if $P_{Y|X}(y|x)P_{Y|X}(y|x^{\prime })>0$ for some $y\in \mathcal{Y}$.

This auxiliary problem is a particular instance of the zero-error Slepian–Wolf problem; its optimal rate is written as $\overline{H}(G)$, where $\overline{H}(G)$ is the complementary graph entropy [17] and G is the characteristic graph in the Slepian–Wolf problem, defined in Definition A1, for the pair $\left((X,g(Y)),Y\right)$. The graph G has $\mathcal{X}\times \mathcal{Z}$ as a set of vertices, and $(x,z)$ is adjacent to $(x^{\prime },z^{\prime })$ if there exists a side information symbol $y\in \mathcal{Y}$ such that $P_{X,Y,g(Y)}(x,y,z)P_{X,Y,g(Y)}(x^{\prime },y,z^{\prime })>0$. It can be observed that the vertices $(x,z)$ and $(x^{\prime },z^{\prime })$ such that $z\ne z^{\prime }$ are not adjacent in G. The graph G is therefore a disjoint union indexed by $\mathcal{Z}$:

$$\begin{array}{cc}\hfill & G={⨆}_{z\in \mathcal{Z}}^{P_{g(Y)}}{G}_z;\hfill \end{array}$$

(A1)

$$\begin{array}{cc}\hfill & R^{*}(g)=\overline{H}(G)=\overline{H}\left({⨆}_{z\in \mathcal{Z}}^{P_{g(Y)}}{G}_z\right);\hfill \end{array}$$

(A2)

where for all $z\in \mathcal{Z}$, $G_z$ is the characteristic graph defined in Definition A1 for the pair $(X_z^{\prime },Y_z^{\prime })\sim P_{X,Y|g(Y)=z}$.

$(\Rightarrow )$ Assume that g and $P_{X,Y}$ satisfy the “pairwise shared side information” condition. It directly follows that $P_{X|g(Y)=z}$ is full support for all $z\in \mathcal{Z}$. Let $z\in \mathcal{Z}$, and let $(x,z),(x^{\prime },z)$ be any two vertices of $G_z$. By construction, there exists $y\in g^{-1}(z)$ such that $P_{X,Y}(x,y)P_{X,Y}(x^{\prime },y)>0$; hence, $P_{X,Y,g(Y)}(x,y,z)P_{X,Y,g(Y)}(x^{\prime },y,z)>0$, and $(x,z),(x^{\prime },z)$ are adjacent in $G_z$. Each graph $G_z$ is therefore complete and perfect; the graph $G={⨆}_{z\in \mathcal{Z}}^{P_{g(Y)}}{G}_z$ is a disjoint union of perfect graphs and is therefore also perfect. We have the following:

$$\begin{array}{cc}\hfill R^{*}(g)& =\overline{H}\left({⨆}_{z\in \mathcal{Z}}^{P_{g(Y)}}{G}_z\right)\hfill \end{array}$$

(A3)

$$\begin{array}{cc}\hfill & =H_{\kappa }\left({⨆}_{z\in \mathcal{Z}}^{P_{g(Y)}}{G}_z\right)\hfill \end{array}$$

(A4)

$$\begin{array}{cc}\hfill & =\sum _{z\in \mathcal{Z}}{P}_{g(Y)}(z)H_{\kappa }(G_z)\hfill \end{array}$$

(A5)

$$\begin{array}{cc}\hfill & =\sum _{z\in \mathcal{Z}}{P}_{g(Y)}(z)H(P_{X|g(Y)=z})\hfill \end{array}$$

(A6)

$$\begin{array}{cc}\hfill & =H(X|g(Y));\hfill \end{array}$$

(A7)

where (A3) comes from (A2); (A4) and (A5) follow from Corollary 12 in [18] used on the perfect graph ${⨆}_{z\in \mathcal{Z}}^{P_{g(Y)}}{G}_z$; and (A6) holds as the independent subsets of the complete graph $G_z$ are singletons containing one of its vertices.

$(\Leftarrow )$ Conversely, assume that $P_{X|g(Y)=z}$ is full support for all $z\in \mathcal{Z}$, and $R^{*}(g)=H(X|g(Y))$.

Assume, ad absurdum, that at least one of the ${(G_z)}_{z\in \mathcal{Z}}$ is not complete; then, there exists a coloring of that graph that maps two different vertices to the same color. Thus, there exists $z\in \mathcal{Z}$ such that

$$\begin{array}{c}\hfill \overline{H}(G_z)<H(P_{X|g(Y)=z}),\end{array}$$

(A8)

as $P_{X|g(Y)=z}$ is full support. We have

$$\begin{array}{cc}\hfill H(X|g(Y))& =R^{*}(g)\hfill \end{array}$$

(A9)

$$\begin{array}{cc}\hfill & =\overline{H}\left({⨆}_{z\in \mathcal{Z}}^{P_{g(Y)}}{G}_z\right)\hfill \end{array}$$

(A10)

$$\begin{array}{cc}\hfill & \le \sum _{z\in \mathcal{Z}}{P}_{g(Y)}(z)\overline{H}(G_z)\hfill \end{array}$$

(A11)

$$\begin{array}{cc}\hfill & <H(X|g(Y));\hfill \end{array}$$

(A12)

where (A10) comes from (A2), (A11) results from Theorem 2 in [17], and (A12) follows from (A8). We arrive at a contradiction, and hence all the graphs ${(G_z)}_{z\in \mathcal{Z}}$ are complete: for all $z\in \mathcal{Z}$ and $x,x^{\prime }\in \mathcal{X}$, there exists a side information symbol $y\in \mathcal{Y}$ such that $P_{X,Y,g(Y)}(x,y,z)P_{X,Y,g(Y)}(x^{\prime },y,z)>0$; hence, $y\in g^{-1}(z)$ and satisfies $P_{X,Y}(x,y)P_{X,Y}(x^{\prime },y)>0$. The condition “pairwise shared side information” is satisfied by $P_{X,Y},g$.

Appendix A.2. Proof of Theorem 3

Let us specify the adjacency condition in $G_{\left[n\right]}$ under assumption (13). Two vertices are adjacent if they satisfy (7) and (8); however, (7) is always satisfied under (13). Thus, $(x^n,z^n)(x^{\prime n},z^n)$ are adjacent if $z^n=z^{\prime n}$ and

$$\begin{array}{c}\hfill \exists y^n\in g^{-1}(z^n),\exists t\le n,f(x_t,y_t)\ne f(x_t^{\prime },y_t).\end{array}$$

(A13)

It can be observed that condition (A13) is the adjacency condition of an OR product of adequate graphs; more precisely,

$$\begin{array}{c}\hfill G_{\left[n\right]}=\underset{z^n\in {\mathcal{Z}}^n}{⨆}\underset{t\le n}{\bigvee }{G}_{z_t}^f.\end{array}$$

(A14)

Although $G_{\left[n\right]}$ cannot be expressed as an n-th OR power, we will show that its chromatic entropy asymptotically coincides with that of an appropriate OR power: we now search for an asymptotic equivalent of $H_{\chi }(G_{\left[n\right]})$.

Definition A2.

${\mathcal{S}}_n$ is the set of colorings of $G_{\left[n\right]}$ that can be written as $(x^n,z^n)\mapsto (T_{z^n},\tilde{c}(x^n,z^n))$ for some mapping $\tilde{c}:{\mathcal{X}}^n\times {\mathcal{Z}}^n\to \tilde{\mathcal{C}}$; where $T_{z^n}$ denotes the type of $z^n$.

In the following, we define $Z^n\doteq {\left(g(Y_t)\right)}_{t\le n}$. Now, we need several Lemmas. Lemma A1 states that the optimal coloring $c(x^n,z^n)$ of $G_{\left[n\right]}$ has the type of $z^n$ as a prefix at a negligible rate cost. Lemma A3 gives an asymptotic formula for the minimal entropy of the colorings from ${\mathcal{S}}_n$.

Lemma A1.

The following asymptotic comparison holds as follows:

$$\begin{array}{c}\hfill H_{\chi }(G_{\left[n\right]})=\underset{\begin{array}{c}c\mathit{coloring}\mathit{of}{G}_{\left[n\right]}\\ s.t.c\in {\mathcal{S}}_n\end{array}}{inf}H(c(X^n,Z^n))+O(logn).\end{array}$$

(A15)

Definition A3

(Isomorphic probabilistic graphs). Let $G_1=({\mathcal{V}}_1,{\mathcal{E}}_1,P_{V_1})$ and $G_2=({\mathcal{V}}_2,{\mathcal{E}}_2,P_{V_2})$ be two probabilistic graphs. We say that $G_1$ is isomorphic to $G_2$ (denoted by $G_1\simeq G_2$) if there exists an isomorphism between them, i.e., a bijection $\psi :{\mathcal{V}}_1\to {\mathcal{V}}_2$ such that

-

For all $v_1,v_1^{\prime }\in {\mathcal{V}}_1$, $v_1{v}_1^{\prime }\in {\mathcal{E}}_1⟺\psi (v_1)\psi (v_1^{\prime })\in {\mathcal{E}}_2$;

-

For all $v_1\in {\mathcal{V}}_1$, $P_{V_1}(v_1)=P_{V_2}\left(\psi (v_1)\right)$.

Lemma A2.

Let $\mathcal{B}$ be a finite set, let $P_B\in \Delta (\mathcal{B})$ and let ${(G_b)}_{b\in \mathcal{B}}$ be a family of isomorphic probabilistic graphs, then $H_{\chi }\left({⨆}_{b^{\prime }\in \mathcal{B}}^{P_B}{G}_{b^{\prime }}\right)=H_{\chi }(G_b)$ for all $b\in \mathcal{B}$.

Lemma A3.

The following asymptotic comparison holds as follows:

$$\begin{array}{c}\hfill \underset{\begin{array}{c}c\mathit{coloring}\mathit{of}{G}_{\left[n\right]}\\ s.t.c\in {\mathcal{S}}_n\end{array}}{inf}H(c(X^n,Z^n))=n\sum _{z\in \mathcal{Z}}{P}_{g(Y)}(z)H_{\kappa }(G_z^f)+o(n).\end{array}$$

(A16)

The proof of Lemma A1 is given in Appendix A.3 and its keypoint is the asymptotically negligible entropy of the prefix $T_{Z^n}$ of the colorings of ${\mathcal{S}}_n$. The proof of Lemma A2 is given in Appendix A.5. The proof of Lemma A3 is given in Appendix A.4 and relies on the decomposition $G_{\left[n\right]}={⨆}_{Q_n\in {\Delta }_n(\mathcal{Z})}{G}_{\left[n\right]}^{Q_n}$, where $G_{\left[n\right]}^{Q_n}$ is the subgraph induced by the vertices $(x^n,z^n)$ such that the type of $z^n$ is $Q_n$. We show that $G_{\left[n\right]}^{Q_n}$ is a disjoint union of isomorphic graphs whose chromatic entropy is given by Lemma A2 and (17): $|H_{\chi }(G_{\left[n\right]}^{Q_n})-n{\sum }_{z\in \mathcal{Z}}{Q}_n(z)H_{\kappa }(G_z^f)|\le n{ϵ}_n$. Finally, uniform convergence arguments enable us draw a conclusion.

Now, let us combine these results together as follows:

$$\begin{array}{cc}\hfill R^{*}(g)& =\frac{1}{n}{H}_{\chi }(G_{\left[n\right]})+o(1)\hfill \end{array}$$

(A17)

$$\begin{array}{cc}\hfill & =\frac{1}{n}\underset{\begin{array}{c}c\mathrm{coloring}\mathrm{of}{G}_{\left[n\right]}\\ \mathrm{s}.\mathrm{t}.c\in {\mathcal{S}}_n\end{array}}{inf}H(c(X^n,Z^n))+o(1)\hfill \end{array}$$

(A18)

$$\begin{array}{cc}\hfill & =\sum _{z\in \mathcal{Z}}{P}_{g(Y)}(z)H_{\kappa }(G_z^f)+o(1),\hfill \end{array}$$

(A19)

where (A17) comes from Theorem 1, (A18) comes from Lemma A1, and (A19) comes from Lemma A3. The proof of Theorem 3 is complete.

Appendix A.3. Proof of Lemma A1

Let $c_n^{*}$ be the coloring of $G_{\left[n\right]}$ with minimal entropy. Then, we have the following:

$$\begin{array}{cc}\hfill H_{\chi }(G_{\left[n\right]})& =\underset{c\mathrm{coloring}\mathrm{of}{G}_{\left[n\right]}}{inf}H(c(X^n,Z^n))\hfill \end{array}$$

(A20)

$$\begin{array}{cc}\hfill & \le \underset{\begin{array}{c}c\mathrm{coloring}\mathrm{of}{G}_{\left[n\right]}\\ \mathrm{s}.\mathrm{t}.c\in {\mathcal{S}}_n\end{array}}{inf}H(c(X^n,Z^n))\hfill \end{array}$$

(A21)

$$\begin{array}{cc}\hfill & =\underset{\begin{array}{c}c:(x^n,z^n)\\ \mapsto (T_{z^n},\tilde{c}(x^n,z^n))\end{array}}{inf}H(T_{Z^n},\tilde{c}(X^n,Z^n))\hfill \end{array}$$

(A22)

$$\begin{array}{cc}\hfill & \le H(T_{Z^n})+H(c_n^{*}(X^n,Z^n))\hfill \end{array}$$

(A23)

$$\begin{array}{cc}\hfill & =H_{\chi }(G_{\left[n\right]})+O(logn),\hfill \end{array}$$

(A24)

where (A22) comes from Definition A2; (A23) comes from the subadditivity of the entropy, $(x^n,z^n)\mapsto (T_{z^n},c_n^{*}(x^n,z^n))$ is a coloring of $G_{\left[n\right]}$ that belongs to ${\mathcal{S}}_n$, and (A24) comes from $H(T_{Z^n})=O(logn)$, as $log|{\Delta }_n(\mathcal{Z})|=O(logn)$. The desired equality comes from the bounds $H_{\chi }(G_{\left[n\right]})$ and $H_{\chi }(G_{\left[n\right]})+O(logn)$ on (A21).

Appendix A.4. Proof of Lemma A3

For all $Q_n\in {\Delta }_n(\mathcal{Z})$, let

$$\begin{array}{c}\hfill G_{\left[n\right]}^{Q_n}=\underset{\begin{array}{c}{z}^n\in {\mathcal{Z}}^n\\ T_{z^n}=Q_n\end{array}}{⨆}\underset{t\le n}{\bigvee }{G}_{z_t}^f,\end{array}$$

(A25)

with the probability distribution induced by $P_{X,Z}^n$. This graph is formed of the connected components of $G_{\left[n\right]}$ whose corresponding $z^n$ has type $Q_n$. We need to find an equivalent for $H_{\chi }(G_{\left[n\right]}^{Q_n})$. Since $G_{\left[n\right]}^{Q_n}$ is a disjoint union of isomorphic graphs, we can use Lemma A2 as follows:

$$\begin{array}{cc}\hfill & H_{\chi }(G_{\left[n\right]}^{Q_n})=H_{\chi }\left(\underset{z\in \mathcal{Z}}{\bigvee }{(G_z^f)}^{\vee n{Q}_n(z)}\right).\hfill \end{array}$$

(A26)

On one hand,

$$\begin{array}{cc}\hfill H_{\chi }\left(\underset{z\in \mathcal{Z}}{\bigvee }{(G_z^f)}^{\vee n{Q}_n(z)}\right)& \ge H_{\kappa }\left(\underset{z\in \mathcal{Z}}{\bigvee }{(G_z^f)}^{\vee n{Q}_n(z)}\right)\hfill \end{array}$$

(A27)

$$\begin{array}{cc}\hfill & =n\sum _{z\in \mathcal{Z}}{Q}_n(z)H_{\kappa }(G_z^f),\hfill \end{array}$$

(A28)

where (A27) comes from $H_{\kappa }\le H_{\chi }$ Lemma 14 in [14], (A28) comes from (17). On the other hand,

$$\begin{array}{cc}\hfill H_{\chi }\left(\underset{z\in \mathcal{Z}}{\bigvee }{(G_z^f)}^{\vee n{Q}_n(z)}\right)& \le \sum _{z\in \mathcal{Z}}{Q}_n(z)H_{\chi }({(G_z^f)}^{\vee n})\hfill \end{array}$$

(A29)

$$\begin{array}{cc}\hfill & =n\sum _{z\in \mathcal{Z}}{Q}_n(z)H_{\kappa }(G_z^f)+n{ϵ}_n,\hfill \end{array}$$

(A30)

where ${ϵ}_n\doteq {max}_z\frac{1}{n}{H}_{\chi }({(G_z^f)}^{\vee n})-H_{\kappa }(G_z^f)$ is a quantity that does not depend on $Q_n$ and satisfies ${lim}_{n\to \infty }{ϵ}_n=0$; (A29) comes from the subadditivity of $H_{\chi }$. Combining Equations (A26), (A28), and (A30) yields

$$\begin{array}{c}\hfill \left|H_{\chi }(G_{\left[n\right]}^{Q_n})-n\sum _{z\in \mathcal{Z}}{Q}_n(z)H_{\kappa }(G_z^f)\right|\le n{ϵ}_n.\end{array}$$

(A31)

Now, we have an equivalent for $H_{\chi }(G_{\left[n\right]}^{Q_n})$.

$$\begin{array}{cc}\hfill & \underset{\begin{array}{c}c\mathrm{coloring}\mathrm{of}{G}_{\left[n\right]}\\ \mathrm{s}.\mathrm{t}.c\in {\mathcal{S}}_n\end{array}}{inf}H(c(X^n,Z^n))\hfill \end{array}$$

(A32)

$$\begin{array}{cc}\hfill =& \underset{\begin{array}{c}c:(x^n,z^n)\\ \mapsto (T_{z^n},\tilde{c}(x^n,z^n))\end{array}}{inf}H(\tilde{c}(X^n,Z^n)|T_{Z^n})+H(T_{Z^n})\hfill \end{array}$$

(A33)

$$\begin{array}{cc}\hfill =& \underset{\begin{array}{c}c:(x^n,z^n)\\ \mapsto (T_{z^n},\tilde{c}(x^n,z^n))\end{array}}{inf}\sum _{Q_n\in {\Delta }_n(\mathcal{Z})}{P}_{T_{Z^n}}(Q_n)H(\tilde{c}(X^n,Z^n)|T_{Z^n}=Q_n)+O(logn)\hfill \end{array}$$

(A34)

$$\begin{array}{cc}\hfill =& \sum _{Q_n\in {\Delta }_n(\mathcal{Z})}{P}_{T_{Z^n}}(Q_n)\underset{c_{Q_n}\mathrm{coloring}\mathrm{of}{G}_{\left[n\right]}^{Q_n}}{inf}H(c_{Q_n}(X^n,Z^n)|T_{Z^n}=Q_n)+O(logn)\hfill \end{array}$$

(A35)

$$\begin{array}{cc}\hfill =& \sum _{Q_n\in {\Delta }_n(\mathcal{Z})}{P}_{T_{Z^n}}(Q_n)H_{\chi }(G_{\left[n\right]}^{Q_n})+O(logn)\hfill \end{array}$$

(A36)

$$\begin{array}{cc}\hfill =& \sum _{Q_n\in {\Delta }_n(\mathcal{Z})}{P}_{T_{Z^n}}(Q_n)\left(n\sum _{z\in \mathcal{Z}}{Q}_n(z)H_{\kappa }(G_z^f)\pm n{ϵ}_n\right)+O(logn)\hfill \end{array}$$

(A37)

$$\begin{array}{cc}\hfill =& n\sum _{Q_n\in {\Delta }_n(\mathcal{Z})}{2}^{-nD(Q_n\parallel P_{g(Y)})+o(n)}\left(\sum _{z\in \mathcal{Z}}{Q}_n(z)H_{\kappa }(G_z^f)\right)\pm n{ϵ}_n+O(logn)\hfill \end{array}$$

(A38)

$$\begin{array}{cc}\hfill =& n\sum _{z\in \mathcal{Z}}{P}_{g(Y)}(z)H_{\kappa }(G_z^f)+o(n),\hfill \end{array}$$

(A39)

where (A34) comes from $H(T_{Z^n})=O(logn)$, as $log|{\Delta }_n(\mathcal{Z})|=O(logn)$; (A35) follows from the fact that the entropy of $\tilde{c}$ can be minimized independently on each $G_{\left[n\right]}^{Q_n}$; (A36) follows from the definition of $G_{\left[n\right]}^{Q_n}$; (A37) comes from (A31); (A38) comes from Lemma 2.6 in [19] and the fact that ${ϵ}_n$ does not depend on $Q_n$.

Appendix A.5. Proof of Lemma A2

Let ${({\tilde{G}}_i)}_{i\le N}$ be isomorphic probabilistic graphs and G such that $G={⨆}_i{\tilde{G}}_i$. Let $c_1^{*}:{\mathcal{V}}_1\to \mathcal{C}$ be the coloring of ${\tilde{G}}_1$ with minimal entropy, and let $c^{*}$ be the coloring of G defined by

$$\begin{array}{cc}\hfill c^{*}:& \mathcal{V}\to \mathcal{C}\hfill \end{array}$$

(A40)

$$\begin{array}{cc}\hfill & vs.\mapsto c_1^{*}\circ {\psi }_{i_v\to 1}(v),\hfill \end{array}$$

(A41)

where $i_v$ is the unique integer such that $v\in {\mathcal{V}}_{i_v}$ and ${\psi }_{i_v\to 1}:{\mathcal{V}}_{i_v}\to {\mathcal{V}}_1$ is an isomorphism between ${\tilde{G}}_{i_v}$ and ${\tilde{G}}_1$. In other words, $c^{*}$ applies the same coloring pattern $c_1^{*}$ on each connected component of G. We have

$$H_{\chi }(G)\le H(c^{*}(V))$$

(A42)

$$\begin{array}{cc}\hfill & =h\left({\sum }_{j\le N}{P}_{i_V}(j)P_{c^{*}(V_j)}\right)\hfill \end{array}$$

(A43)

$$\begin{array}{cc}\hfill & =h\left({\sum }_{j\le N}{P}_{i_V}(j)P_{c_1^{*}(V_1)}\right)\hfill \end{array}$$

(A44)

$$\begin{array}{cc}\hfill & =H(c_1^{*}(V_1))\hfill \end{array}$$

(A45)

$$\begin{array}{cc}\hfill & =H_{\chi }({\tilde{G}}_1),\hfill \end{array}$$

(A46)

where h denotes the entropy of a distribution; (A44) comes from the definition of $c^{*}$; and (A46) comes from the definition of $c_1^{*}$.

Now, let us prove the upper bound on $H_{\chi }({\tilde{G}}_1)$. Let c be a coloring of G, and let $i^{*}\doteq {argmin}_{i}H(c(V_i))$ (i.e., $i^{*}$ is the index of the connected component for which the entropy of the coloring induced by c is minimal). We have

$$H(c(V))=h\left({\sum }_{j\le N}{P}_{i_V}(j)P_{c(V_j)}\right)$$

(A47)

$$\begin{array}{cc}\hfill & \ge {\sum }_{j\le N}{P}_{i_V}(j)h(P_{c(V_j)})\hfill \end{array}$$

(A48)

$$\begin{array}{cc}\hfill & \ge {\sum }_{j\le N}{P}_{i_V}(j)H(c(V_{i^{*}}))\hfill \end{array}$$

(A49)

$$\begin{array}{cc}\hfill & \ge H_{\chi }({\tilde{G}}_{i^{*}}),\hfill \end{array}$$

(A50)

$$\begin{array}{cc}\hfill & =H_{\chi }({\tilde{G}}_1),\hfill \end{array}$$

(A51)

where (A48) follows from the concavity of h; (A49) follows from the definition of $i^{*}$; (A50) comes from the fact that c induces a coloring of ${\tilde{G}}_{i^{*}}$; (A51) comes from the fact that ${\tilde{G}}_1$ and ${\tilde{G}}_{i^{*}}$ are isomorphic. Now, we can combine the bounds (A46) and (A51): for all coloring c of G we have

$$\begin{array}{c}\hfill H_{\chi }(G)\le H_{\chi }({\tilde{G}}_1)\le H(c(V)),\end{array}$$

(A52)

which yields the desired equality when taking the infimum over c.