Universality of Logarithmic Loss in Fixed-Length Lossy Compression ^†

Abstract

We established a universality of logarithmic loss over a finite alphabet as a distortion criterion in fixed-length lossy compression. For any fixed-length lossy-compression problem under an arbitrary distortion criterion, we show that there is an equivalent lossy-compression problem under logarithmic loss. The equivalence is in the strong sense that we show that finding good schemes in corresponding lossy compression under logarithmic loss is essentially equivalent to finding good schemes in the original problem. This equivalence relation also provides an algebraic structure in the reconstruction alphabet, which allows us to use known techniques in the clustering literature. Furthermore, our result naturally suggests a new clustering algorithm in the categorical data-clustering problem.

1. Introduction

Logarithmic loss is a unique distortion measure in the sense that it allows a “soft” estimation (or reconstruction) of the source. Although logarithmic loss plays a crucial role in learning theory, not much work has been published regarding lossy compression until recently. A few exceptions are a line of work on multiterminal source coding [1,2,3], the single-shot approach to lossy source coding under logarithmic loss [4], and several universal properties of logarithmic loss in information theory [5,6,7]. In [4], Shkel and Verdú focused on the lossy-compression problem when the distortion measure is given by logarithmic loss. On the other hand, Jiao et al. justified logarithmic loss by showing it is the only loss function that satisfies a natural data-processing requirement [5]. Painsky and Wornell provided a universal property of logarithmic loss in the context of classification. In [7], No focused on the universal property of logarithmic loss in the successive refinement problem. We would also like to point out that the information bottleneck method [8,9,10,11] is related to lossy compression under logarithmic loss. Indeed, it is equivalent to the noisy lossy-compression problem under logarithmic loss [12].

In this paper, we present a new universal property of logarithmic loss in fixed-length lossy-compression problems. Consider an arbitrary fixed-length lossy-compression problem, where source and reconstruction alphabets $𝒳$ and $\widehat{\mathcal{X}}$ are discrete. Suppose arbitrary distortion measure $d:\mathcal{X}\times \widehat{\mathcal{X}}$ is given. Then, we show that there exists a corresponding fixed-length lossy-compression problem where the source alphabet remains the same, but the reconstruction alphabet is a set of distributions on $𝒳$, and the distortion measure is logarithmic loss. This implies that there is a correspondence between any fixed-length lossy-compression problem under an arbitrary distortion measure and that under logarithmic loss. The correspondence is in the following strong sense:

• optimal schemes for the two problems are the same; and
• a good scheme for one problem is also a good scheme for the other.

We are more precise about the “optimal” and “goodness” of the scheme in later sections. This finding essentially implies that it is enough to consider the lossy-compression problem under logarithmic loss.

The above correspondence provides new insights into the fixed-length lossy-compression problem. In general, the reconstruction alphabet in the lossy-compression problem does not have any well-defined operations. However, in the corresponding lossy compression under logarithmic loss, reconstruction symbols are probability distributions that have their own algebraic structure. Thus, under the corresponding setting, we can apply various techniques, such as the information geometric approach, clustering with Bregman divergence, and relaxation of the optimization problem. Furthermore, the equivalence relation suggests a new algorithm in the categorical data-clustering problem, where data are not in the continuous space.

The remainder of the paper is organized as follows. In Section 2, we revisit some of the known results of logarithmic loss and fixed-length lossy compression. Section 3 is dedicated to the equivalence between lossy compression under arbitrary distortion measures and that under logarithmic loss. In Section 4, we present the geometric interpretation of our result. We provide the log-convex relaxation of lossy compression and connection to the clustering problems in Section 5. Finally, we conclude in Section 6.

Notation: Uppercase X denotes a random variable, where $𝒳$ denotes a set of alphabet. On the other hand, lowercase x denotes a specific possible realization of random variable X, i.e., $x\in \mathcal{X}$. Similarly, $X^n$ denotes an n-dimensional random vector $(X_1,X_2,\dots ,X_n)$ while lowercase $x^n$ denotes a realization of $X^n$. The absolute value of function $\left|f\right|$ denotes a size of image of function $f:\mathcal{X}\to \mathcal{Y}$, i.e., $\left|\right\{f\left(x\right):x\in \mathcal{X}\left\}\right|$. If it was clear from the context, we used ${\sum }_x$ instead of ${\sum }_{x\in \mathcal{X}}$. We used a natural logarithm and nats instead of bits.

2. Preliminaries

2.1. Logarithmic Loss

Suppose $𝒳$ is a finite set of discrete symbols, and $\mathcal{M}\left(\mathcal{X}\right)$ is the set of probability measures on $𝒳$. For $x\in \mathcal{X}$ and $q\in \mathcal{M}\left(\mathcal{X}\right)$, the definition of logarithmic loss $\ell :\mathcal{X}\times \mathcal{M}\left(\mathcal{X}\right)[0,\infty ]$ is given by

$$\begin{array}{c}\hfill \ell (x,q)=log\frac{1}{q\left(x\right)}.\end{array}$$

2.2. Fixed-Length Lossy Compression

In this section, we briefly introduce the basic settings of the fixed-length lossy-compression problem [13]. In a fixed-length lossy-compression setting, we have a source X with finite alphabet $\mathcal{X}=\{1,\dots ,r\}$ and source distribution $p_X$. An encoder $f:\mathcal{X}\to \{1,\dots ,M\}$ maps the source symbol to one of M messages. On the other side, a decoder $g:\{1,\dots ,M\}\to \widehat{\mathcal{X}}$ maps the message to actual reconstruction $\widehat{X}$, where the reconstruction alphabet is also finite $\widehat{\mathcal{X}}=\{1,\dots ,s\}$. Let $d:\mathcal{X}\times \widehat{\mathcal{X}}\to [0,\infty )$ be a distortion measure between source and reconstruction.

First, we can define the code that the expected distortion is lower than a given distortion level.

Definition 1

(Average distortion criterion). An $(M,D)$ code is a pair of an encoder f with $\left|f\right|\le M$ and a decoder g, such that

$$\begin{array}{c}\hfill \mathbb{E}\left[d\right(X,g\left(f\right(X\left)\right)\left)\right]\le D.\end{array}$$

The minimum number of codewords required to achieve average distortion not exceeding D is defined by

$$\begin{array}{c}\hfill M^{\star }\left(D\right)=min\{M:\exists (M,D)code\}.\end{array}$$

Similarly, we can define the minimum achievable average distortion given number of codewords M.

$$\begin{array}{c}\hfill D^{\star }\left(M\right)=min\{D:\exists (M,D)code\}.\end{array}$$

One may consider a stronger criterion that restricts the probability of exceeding a given distortion level.

Definition 2

(Excess distortion criterion). An $(M,D,ϵ)$ code is a pair of an encoder f with $\left|f\right|\le M$ and a decoder g such that

$$\begin{array}{c}\hfill \mathit{Pr}\left[d(X,g(f\left(X\right)\left)\right)>D\right]\le ϵ.\end{array}$$

The minimum number of codewords required to achieve excess distortion probability ϵ, and distortion D is defined by

$$\begin{array}{c}\hfill M^{\star }(D,ϵ)=min\{M:\exists (M,D,ϵ)code\}.\end{array}$$

Similarly, we can define the minimum achievable excess distortion probability given target distortion D and number of codewords M.

$$\begin{array}{c}\hfill {ϵ}^{\star }(M,ϵ)=min\{ϵ:\exists (M,D,ϵ)code\}.\end{array}$$

Given target distortion D and $p_X$, the information rate-distortion function is defined by

$$\begin{array}{c}\hfill R\left(D\right)=\underset{p_{\widehat{X}|X}:\mathbb{E}\left[d(X,\widehat{X})\right]\le D}{inf}I(X;\widehat{X})\end{array}$$

(1)

We make the following benign assumptions:

• There exists a unique rate-distortion function achieving conditional distribution $p_{\widehat{X}|X}^{\star }$.
• We assume that $p_{\widehat{X}}^{\star }\left(\widehat{x}\right)>0$ for all $\widehat{x}$ ∈ $\widehat{\mathcal{X}}$ since we can always discard the reconstruction symbol with zero probability.
• If $d(x,{\widehat{x}}_1)=d(x,{\widehat{x}}_2)$ for all $x\in \mathcal{X}$, then ${\widehat{x}}_1={\widehat{x}}_2$. (If $d(x,{\widehat{x}}_1)=d(x,{\widehat{x}}_2)$ for all x, then, there is no difference between ${\widehat{x}}_1$ and ${\widehat{x}}_2$ in terms of loss. Thus, we can always discard ${\widehat{x}}_2$ without loss of generality.)

2.3. D-Tilted Information

Define the information density of joint distribution $p_{X,\widehat{X}}$ by

$$\begin{array}{c}\hfill {\mathsf{𝚤}}_{X;\widehat{X}}(x;\widehat{x})=log\frac{p_{X,\widehat{X}}(x,\widehat{x})}{p_X\left(x\right)p_{\widehat{X}}\left(\widehat{x}\right)}.\end{array}$$

Then, we are ready to define D-tilted information that plays a key role in fixed-length lossy compression.

Definition 3

([13] (Definition 6)). The D-tilted information in $x\in \mathcal{X}$ is defined as

$$\begin{array}{c}\hfill {\mathsf{𝚥}}_X(x,D)=log\frac{1}{\mathbb{E}[exp\left({\lambda }^{\star }D-{\lambda }^{\star }d(x,{\widehat{X}}^{\star })\right)]}\end{array}$$

where the expectation is with respect to the marginal distribution of ${\widehat{X}}^{\star }$ and ${\lambda }^{\star }=-R^{\prime }\left(D\right)$.

Note that ${\widehat{X}}^{\star }$ is a random variable that has a marginal distribution of $p_X\times p_{\widehat{X}|X}^{\star }$, and $R^{\prime }\left(D\right)$ is the first derivative of rate-distortion function $R\left(D\right)$.

Theorem 1

([14] (Lemma 1.4)). For all $\widehat{x}$ ∈ $\widehat{\mathcal{X}}$,

$$\begin{array}{c}\hfill {\mathsf{𝚥}}_X(x,D)={\mathsf{𝚤}}_{X;{\widehat{X}}^{\star }}(x;\widehat{x})+{\lambda }^{\star }d(x,\widehat{x})-{\lambda }^{\star }D;\end{array}$$

(2)

therefore, we have

$$\begin{array}{c}\hfill R\left(D\right)=\mathbb{E}\left[\mathsf{𝚥}(X,D)\right].\end{array}$$

Let $p_{X|\widehat{X}}^{\star }$ be the induced conditional probability from $p_{\widehat{X}|X}^{\star }$. Then, (2) can equivalently be expressed as

$$\begin{array}{c}log\frac{1}{p_{X|\widehat{X}}^{\star }\left(x\right|\widehat{x})}\hfill \\ =log\frac{1}{p_X\left(x\right)}-{\mathsf{𝚥}}_X(x,D)+{\lambda }^{\star }d(x,\widehat{x})-{\lambda }^{\star }D.\hfill \end{array}$$

(3)

The following lemma shows that $p_{X|\widehat{X}}^{\star }(·|\widehat{x})$ are all distinct.

Lemma 1

([7] (Lemma 2)). For all ${\widehat{x}}_1\ne {\widehat{x}}_2$, there exists $x\in \mathcal{X}$ such that $p_{X|\widehat{X}}^{\star }\left(x\right|{\widehat{x}}_1)\ne p_{X|\widehat{X}}^{\star }\left(x\right|{\widehat{x}}_2)$.

3. One-to-One Correspondence Between General Distortion and Logarithmic Loss

3.1. Main Results

Consider fixed-length lossy compression under arbitrary distortion $d(·,·)$, as described in Section 2.2. We have a source X with finite alphabet $\mathcal{X}=\{1,\dots ,r\}$, source distribution $p_X$, and finite reconstruction alphabet $\widehat{\mathcal{X}}=\{1,\dots ,s\}$. For a fixed number of messages M, let $f^{\star }$ and $g^{\star }$ be the encoder and decoder that achieve optimal average distortion $D^{\star }\left(M\right)$, i.e.,

$$\begin{array}{c}\hfill \mathbb{E}\left[d(X,g^{\star }\left(f^{\star }\left(X\right)\right))\right]=D^{\star }\left(M\right).\end{array}$$

Let $p_{\widehat{X}|X}^{\star }$ denote the rate-distortion function achieving conditional distribution at distortion $D=D^{\star }\left(M\right)$. In other words, $p_X\times p_{\widehat{X}|X}^{\star }$ achieves the infimum in

$$\begin{array}{c}\hfill R\left(D^{\star }\left(M\right)\right)=\underset{p_{\widehat{X}|X}:\mathbb{E}\left[d(X,\widehat{X})\right]\le D^{\star }\left(M\right)}{inf}I(X;\widehat{X}).\end{array}$$

(4)

Note that $R\left(D^{\star }\left(M\right)\right)$ may be strictly smaller than $logM$ in general since $R(·)$ is an information rate-distortion function that does not characterize the best achievable performance for the “one-shot” setting in which $D^{\star }\left(M\right)$ is defined.

Now, we define the corresponding fixed-length lossy-compression problem under logarithmic loss. In the corresponding problem, source alphabet $\mathcal{X}=\{1,\dots ,r\}$, source distribution $p_X$, and number of messages M remain the same. However, we have a different reconstruction alphabet $\mathcal{Y}=\{p_{X|\widehat{X}}^{\star }(·|\widehat{x}):\widehat{x}\in \widehat{\mathcal{X}}\}\subset \mathcal{M}\left(\mathcal{X}\right)$ where $p^{\star }$ pertains to the achiever of the infimum in Equation (4) associated with the original loss function. Recall that $\mathcal{M}\left(\mathcal{X}\right)$ is the set of all probability measures on $𝒳$. Let the distortion of the corresponding problem be the logarithmic loss.

We now further connect the encoding and decoding schemes between the two problems. Suppose $f:\to \mathcal{X}\{1,\dots ,M\}$ and $g:\{1,\dots ,M\}\to \widehat{\mathcal{X}}$ are an encoder and decoder pair in the original problem. When f and g are given in the original problem, we define the corresponding encoder and decoder in the corresponding problem as follows. We let the encoder be the same $f_{\ell }=f$, and define the decoder $g_{\ell }:\{1,\dots ,M\}\to \mathcal{Y}$ by

$$\begin{array}{c}\hfill g_{\ell }\left(m\right)=p_{X|\widehat{X}}^{\star }(·|g\left(m\right)).\end{array}$$

Then, $f_{\ell }$ and $g_{\ell }$ are a valid encoder and decoder pair for the corresponding fixed-length lossy-compression problem under logarithmic loss. Conversely, given $f_{\ell }$ and $g_{\ell }$, we can find corresponding f and g because Lemma 1 guarantees that $p_{X|\widehat{X}}(·|\widehat{x})$ are distinct.

The following result shows the relation between the corresponding schemes.

Theorem 2.

For any encoder–decoder pair $(f_{\ell },g_{\ell })$ for the corresponding fixed-length lossy-compression problem under logarithmic loss, we have

$$\begin{array}{c}\mathbb{E}\left[\ell (X,g_{\ell }\left(f_{\ell }\left(X\right)\right))\right]\hfill \\ =H\left(X\right|{\widehat{X}}^{\star })+{\lambda }^{\star }\left(\mathbb{E}\left[d(X,g(f\left(X\right)\left)\right)\right]-D^{\star }\left(M\right)\right)\hfill \\ \ge H\left(X\right|{\widehat{X}}^{\star })\hfill \end{array}$$

where $(f,g)$ is the corresponding encoder–decoder pair for the original lossy-compression problem. Note that $H\left(X\right|{\widehat{X}}^{\star })$ and the expectations are with respect to distribution $p_X\times p_{\widehat{X}|X}^{\star }$. Moreover, equality holds if and only if $f_{\ell }=f^{\star }$ and $g_{\ell }\left(m\right)=p_{X|\widehat{X}}^{\star }(·|g^{\star }\left(m\right))$.

Proof. 

We have

$$\begin{array}{c}\mathbb{E}\left[\ell (X,g_{\ell }\left(f_{\ell }\left(X\right)\right))\right]\hfill \\ =\mathbb{E}\left[\ell \left(X,p_{X|\widehat{X}}^{\star }(·|g\left(f\left(X\right)\right))\right)\right]\hfill \\ =\mathbb{E}\left[log\frac{1}{p_{X|\widehat{X}}^{\star }\left(X\right|g\left(f\left(X\right)\right))}\right].\hfill \end{array}$$

Then, Equation (3) implies that

$$\begin{array}{c}\mathbb{E}\left[\ell (X,g_{\ell }\left(f_{\ell }\left(X\right)\right))\right]\hfill \\ =\mathbb{E}\left[log\frac{1}{p_X\left(X\right)}-{\mathsf{𝚥}}_X(X,D^{\star }\left(M\right))\right]\hfill \\ +\mathbb{E}\left[{\lambda }^{\star }d(X,g\left(f\left(X\right)\right))-{\lambda }^{\star }{D}^{\star }\left(M\right)\right]\hfill \\ =H\left(X\right|{\widehat{X}}^{\star })+{\lambda }^{\star }\left(\mathbb{E}[d(X,g(f\left(X\right)\left)\right)-D^{\star }\left(M\right)\right)]\hfill \end{array}$$

(5)

$$\begin{array}{c}\ge H\left(X\right|{\widehat{X}}^{\star })\hfill \end{array}$$

(6)

where Equation (5) is because $\mathbb{E}\left[{}_{\mathsf{𝚥}X}(X,D^{\star }\left(M\right))\right]=R\left(D^{\star }\left(M\right)\right)=I(X;{\widehat{X}}^{\star })$ with respect to distribution $p_X\times p_{\widehat{X}|X}^{\star }$. Inequality (6) is because $D^{\star }\left(M\right)$ is the minimum achievable average distortion with M codewords. Equality holds if and only if $\mathbb{E}\left[d(X,g(f\left(X\right)\left)\right)\right]=D^{\star }\left(M\right)$, which can be achieved by the optimal scheme for the original lossy-compression problem. In other words, the equality holds if

$$\begin{array}{cc}\hfill f_{\ell }^{\star }=& f^{\star }\hfill \\ \hfill g_{\ell }^{\star }\left(m\right)=& p_{X|\widehat{X}}^{\star }(·|g^{\star }\left(m\right)).\hfill \end{array}$$

 □

In the above theorem, distortion $D^{\star }\left(M\right)$ plays a critical role, which is the minimal achievable distortion in the one-shot setting. We also used $p_{X|\widehat{X}}^{\star }$ in the corresponding problem, which is the rate-distortion-achieving conditional distribution. This might be confusing since the rate-distortion function provides the optimal rate in the asymptotic setting. However, recall that the minimal mutual information between X and $\widehat{X}$ in Equation (1) is the “information” rate-distortion function. The “information” rate-distortion function is equal to the optimum rate in the asymptotic case if the source is independent and identically distributed.

On the other hand, we viewed the “information” rate-distortion function differently. We considered the one-shot setting where source X and reconstruction $\widehat{X}$ are single variables. Given number of messages M, the minimal achievable distortion is given by $D^{\star }\left(M\right)$. Under this setting, we focused on minimal mutual information between X and $\widehat{X}$ when the distortion between X and $\widehat{X}$ is restricted by $D^{\star }\left(M\right)$. Our theorem implies that minimal achieving distribution $p_{X|\widehat{X}}^{\star }$ provides the corresponding one-shot lossy-compression problem under logarithmic loss.

Remark 1.

In the corresponding fixed-length lossy-compression problem under logarithmic loss, the minimal achievable average distortion given number of codewords M is

$$\begin{array}{c}\hfill D_{\ell }^{\star }\left(M\right)=H\left(X\right|{\widehat{X}}^{\star })\end{array}$$

where the conditional entropy is with respect to distribution $p_X\times p_{\widehat{X}|X}^{\star }$.

Remark 2.

From now on, we denote the original lossy-compression problem under given distortion measure $d(·,·)$ with reconstruction alphabet $\widehat{\mathcal{X}}$ by “original problem”. On the other hand, we denote the corresponding lossy-compression problem under logarithmic loss with reconstruction alphabet $\mathcal{Y}$ by “corresponding problem”.

3.2. Example: Memoryless Bernoulli Source with Hamming Distortion Measure

In this section, we consider the memoryless Bernoulli source under Hamming distortion measure as an example of the above equivalence. Let $X=U^n$ be a memoryless Bernoulli source with probability $\alpha$, where $\mathcal{X}={\mathcal{U}}^n={\{0,1\}}^n$, and reconstruction $\widehat{X}=V^n$ is also an n-dimensional binary vector where $\widehat{\mathcal{X}}={\mathcal{V}}^n={\{0,1\}}^n$. Note that block length n is fixed, so the problem is in the one-shot setting. Distortion measure d is separable Hamming distortion, i.e.,

$$\begin{array}{c}\hfill d(X,\widehat{X})=d_H(U^n,V^n)=\frac{1}{n}\sum _{i=1}^n{d}_H(U_i,V_i)\end{array}$$

where $d_H(u,v)=1$ if $u\ne v$ and $d_H(u,v)=0$ if $u=v$. Let M be the number of messages. Then, we are interested in optimal encoding and decoding schemes that achieve distortion $D=D^{\star }\left(M\right)$.

In this scenario, the information rate-distortion function is not hard to compute [15]:

$$\begin{array}{cc}\hfill R\left(D\right)=& \underset{p_{\widehat{X}|X}:\mathbb{E}\left[d(X,\widehat{X})\right]\le D}{inf}I(X;\widehat{X})\hfill \\ \hfill =& \underset{p_{U^n|V^n}:\mathbb{E}\left[d_H(U^n,V^n)\right]\le D}{inf}I(U^n;V^n)\hfill \\ \hfill =& n\underset{p_{U|V}:\mathbb{E}\left[d_H(U,V)\right]\le D}{inf}I(U;V)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill =& n(h_2\left(\alpha \right)-h_2\left(D\right)),\hfill \end{array}$$

(8)

where $h_2(·)$ is the binary entropy function. Let $p_{U|V}^{\star }$ be the distribution that achieves the infimum in Equation (7). We have an analytic formula for rate-distortion-achieving distribution $p_{X|\widehat{X}}^{\star }$. For $x=u^n$ and $=v^n$, we have

$$\begin{array}{cc}\hfill p_{X|\widehat{X}}^{\star }\left(x\right|\widehat{x})=& \prod _{i=1}^n{p}_{U|V}^{\star }\left(u_i\right|v_i)\hfill \\ \hfill =& \prod _{i=1}^n{D}^{d_H(u_i,v_i)}{(1-D)}^{1-d_H(u_i,v_i)}\hfill \\ \hfill =& {(1-D)}^n{\left(\frac{D}{1-D}\right)}^{n{d}_H(u^n,v^n)}\hfill \\ \hfill =& {(1-D)}^n{\left(\frac{D}{1-D}\right)}^{nd(x,\widehat{x})}.\hfill \end{array}$$

Then, the corresponding problem is the rate-distortion problem under logarithmic loss where the set of reconstruction symbols is

$$\begin{array}{cc}\hfill \mathcal{Y}=& \{p_{X|\widehat{X}}^{\star }(·|\widehat{x}):\widehat{x}\in {\mathcal{V}}^n\}.\hfill \end{array}$$

Remark 3.

We can rewrite Equation (3) in this case.

$$\begin{array}{cc}\hfill \ell (x,p_{X|\widehat{X}}^{\star }(·|\widehat{x}))=& log\frac{1}{p_{X|\widehat{X}}^{\star }\left(x\right|\widehat{x})}\hfill \\ \hfill =& nlog\frac{1}{1-D}+nd(x,\widehat{x})log\frac{1-D}{D}.\hfill \end{array}$$

The above equation explicitly shows the correspondence between logarithmic loss and the original distortion measure.

3.3. Discussion

3.3.1. One-to-One Correspondence

Theorem 2 implies that, for any fixed-length lossy-compression problem, we can find an equivalent problem under logarithmic loss where optimal encoding schemes are the same. Thus, without loss of generality, we can restrict our attention to the problem under logarithmic loss with reconstruction alphabet $\mathcal{Y}=\{q^{\left(1\right)},\dots ,q^{\left(s\right)}\}$ for some $q^{\left(1\right)},\dots ,q^{\left(s\right)}\in \mathcal{M}\left(\mathcal{X}\right)$.

3.3.2. Scheme Suboptimality

Suppose f and g are a suboptimal encoder and decoder for the original fixed-length lossy-compression problem. Then, the theorem implies

$$\begin{array}{c}\mathbb{E}\left[\ell (X,g_{\ell }\left(X\right))-H\left(X\right|{\widehat{X}}^{\star })\right]\hfill \\ ={\lambda }^{\star }\left(\mathbb{E}\left[d(X,g(f\left(X\right)\left)\right)\right]-D^{\star }\left(M\right)\right).\hfill \end{array}$$

(9)

The left-hand side of Equation (9) is the cost of suboptimality for the corresponding lossy-compression problem. On the other hand, the right-hand side is proportional to the cost of suboptimality for the original problem. In Section 3.3.1, we discussed that the optimal schemes of the two problems coincide. Equation (9) shows stronger equivalence in which costs of suboptimalities are linearly related. This implies that a good code for one problem is also good for the other.

3.3.3. Operations on the Reconstruction Alphabet

In general, reconstruction alphabet $\widehat{\mathcal{X}}$ does not have an algebraic structure. However, in the corresponding rate-distortion problem, the reconstruction alphabet is the set of probability measures where we have natural operations such as convex combinations of elements or projection to a convex hull. We discuss such operations closer in Section 5.

3.4. Exact Performance of Optimal Scheme

In the previous section, we showed that there is a corresponding lossy-compression problem under logarithmic loss that shares the same optimal coding scheme. In this section, we investigate the exact performance of the optimal scheme for the fixed-length lossy-compression problem under logarithmic loss, when the reconstruction alphabet is the set of all probability measures on $𝒳$, i.e., $\mathcal{M}\left(\mathcal{X}\right)$. (Recently, Shkel and Verdu [4] independently proposed similar results. The result was also presented in our conference version of the paper [16].) We also characterize minimal average distortion $D^{\star }\left(M\right)$ when we have a fixed number of messages M. Note that this is a single-letter version of ([2], [Lemma 1]). Although the optimal scheme associated with $\mathcal{M}\left(\mathcal{X}\right)$ may differ from the optimal scheme with restricted reconstruction alphabets $\mathcal{Y}$, it provides an insight, as we show in Section 4. In this section, we restrict our attention to deterministic schemes. However, it is not hard to show that the same result holds even if we allow a stochastic encoder and decoder.

Let an encoder and a decoder be $f:\mathcal{X}\to \{1,\dots ,M\}$ and $g:\{1,\dots ,M\}\to \mathcal{M}\left(\mathcal{X}\right)$ where $g\left(m\right)=q^{\left(m\right)}\in \mathcal{M}\left(\mathcal{X}\right)$. Then, we have

$$\begin{array}{c}\mathbb{E}\left[\ell (X,g(f\left(X\right)\left)\right)\right]\hfill \\ =\sum _{x\in \mathcal{X}}{p}_X\left(x\right)log\frac{1}{q^{\left(f\right(x\left)\right)}\left(x\right)}\hfill \\ =H\left(X\right)+\sum _{m=1}^M\sum _{x\in f^{-1}\left(m\right)}{p}_X\left(x\right)log\frac{p_X\left(x\right)}{q^{\left(m\right)}\left(x\right)}\hfill \\ =H\left(X\right)+\sum _{m=1}^M{u}_{m}log{u}_m\hfill \\ +\sum _{m=1}^M{u}_m\sum _{x\in f^{-1}\left(m\right)}\frac{p_X\left(x\right)}{u_m}log\frac{p_X\left(x\right)/u_m}{q^{\left(m\right)}\left(x\right)},\hfill \end{array}$$

where $f^{-1}\left(m\right)=\{x\in \mathcal{X}:f\left(x\right)=m\}$ and $u_m={\sum }_{x\in f^{-1}\left(m\right)}{p}_X\left(x\right)$. Since $p_{X\left|f\right(X)}\left(x\right|m)=\frac{p_X\left(x\right)}{u_m}$ for all $x\in f^{-1}\left(m\right)$, we have

$$\begin{array}{c}\mathbb{E}\left[\ell (X,g(f\left(X\right)\left)\right)\right]\hfill \\ =H\left(X\right)-H\left(f\right(X\left)\right)\hfill \\ +\sum _{m=1}^M{u}_{m}D\left(p_{X\left|f\right(X)}(·|m)\Vert q^{\left(m\right)}\right)\hfill \\ \ge H\left(X\right)-H\left(f\right(X\left)\right).\hfill \end{array}$$

Equality can be achieved by choosing $q^{\left(m\right)}=p_{X\left|f\right(X)}(·|m)$, which can be done no matter what f is. Thus, we have

$$\begin{array}{c}\hfill D^{\star }\left(M\right)=H\left(X\right)-\underset{f:\left|f\right|\le M}{max}H\left(f\left(X\right)\right).\end{array}$$

This implies that the optimal encoder is function f that maximizes $H\left(f\right(X\left)\right)$, and the optimal decoder is given by $g\left(m\right)=p_{X\left|f\right(X)}(·|m)$. The above result provides a trivial lower bound:

$$\begin{array}{c}\hfill D^{\star }\left(M\right)\ge H\left(X\right)-logM.\end{array}$$

The optimal scheme under an excess distortion criterion is given in Appendix A.

4. Geometrical Interpretation

In this section, we present another geometrical interpretation of the decoder in lossy-compression problems. Consider the original lossy-compression problem with discrete reconstruction alphabet $\widehat{\mathcal{X}}$ and distortion measure $d(·,·)$. Suppose encoding function f is given that may or may not be optimal, where $\left|f\right|=M$. Let $A_m=f^{-1}\left(m\right)=\{x\in \mathcal{X}:f\left(x\right)=m\}$, which is the set of source symbols that are mapped to message m. Then, optimal reconstruction $g\left(m\right)$ is given by

$$\begin{array}{c}\hfill g\left(m\right)=\underset{\widehat{x}\in \widehat{\mathcal{X}}}{argmin}\mathbb{E}\left[d(X,\widehat{x})|X\in A_m\right].\end{array}$$

(10)

Now, consider the corresponding lossy-compression problem under logarithmic loss. Recall that the set of reconstruction alphabets is given by

$$\begin{array}{c}\hfill \mathcal{Y}=\{p_{X|\widehat{X}}^{\star }(·|\widehat{x}):\widehat{x}\in \widehat{\mathcal{X}}\}\end{array}$$

where $\mathcal{Y}\subset \mathcal{M}\left(\mathcal{X}\right)$. As we have seen in Section 3.4, the optimal reconstruction is $g_{\ell }^E\left(m\right)=p_{X\left|f\right(X)}(·|m)$ if we have extended set of reconstruction alphabet $\mathcal{M}\left(\mathcal{X}\right)$. Thus, it is natural to find the probability distribution in $𝒴$, which is the nearest distribution from $g_{\ell }^E\left(m\right)$. We propose Kullback–Leibler divergence to measure the distance between probability distributions. In other words, we want to find ${\tilde{g}}_{\ell }\left(m\right)\in \mathcal{Y}$, such that

$$\begin{array}{c}\hfill {\tilde{g}}_{\ell }\left(m\right)=\underset{q\in \mathcal{Y}}{argmin}D(g_{\ell }^E\left(m\right)\parallel q).\end{array}$$

(11)

This can be viewed as projecting the optimal solution from extended set $\mathcal{M}\left(\mathcal{X}\right)$ to original feasible set $𝒴$. Since $q\in \mathcal{Y}$, there exists $\widehat{x}$ ∈ $\widehat{\mathcal{X}}$, such that $q(·)=p_{X|\widehat{X}}^{\star }(·|\widehat{x})$. Then, the above Kullback–Leibler divergence is given by

$$\begin{array}{c}D(p_{X\left|f\right(X)}(·|m)\parallel p_{X|\widehat{X}}^{\star }(·|\widehat{x}))\hfill \\ =\sum _{x\in A_m}{p}_{X\left|f\right(X)}\left(x\right|m)log\frac{p_{X\left|f\right(X)}\left(x\right|m)}{p_{X|\widehat{X}}^{\star }\left(x\right|\widehat{x})}\hfill \\ =\sum _{x\in A_m}\frac{p_X\left(x\right)}{Pr\left[X\in A_m\right]}log\frac{p_X\left(x\right)}{Pr\left[X\in A_m\right]p_{X|\widehat{X}}^{\star }\left(x\right|\widehat{x})}\hfill \\ =log\frac{1}{Pr\left[X\in A_m\right]}+\sum _{x\in A_m}\frac{p_X\left(x\right)}{Pr\left[X\in A_m\right]}log\frac{p_X\left(x\right)}{p_{X|\widehat{X}}^{\star }\left(x\right|\widehat{x})}\hfill \\ =log\frac{1}{Pr\left[X\in A_m\right]}+\sum _{x\in A_m}\frac{p_X\left(x\right)}{Pr\left[X\in A_m\right]}\left(-\mathsf{𝚥}(x,D)+{\lambda }^{\star }d(x,\widehat{x})-{\lambda }^{\star }D\right),\hfill \end{array}$$

where the last equality is from Equation (2). Note that $d(x,\widehat{x})$ is the only term that is a function of $\widehat{x}$, and ${\lambda }^{\star }$ is positive. Thus, if $q(·)=p_{X|\widehat{X}}(·|\widehat{x})$ achieves the minimum in Equation (11), then $\widehat{x}$ minimizes the following:

$$\begin{array}{c}\hfill \sum _{x\in A_m}\frac{p_X\left(x\right)}{Pr\left[X\in A_m\right]}d(x,\widehat{x})=\mathbb{E}\left[d(X,\widehat{x})|X\in A_m\right].\end{array}$$

(12)

Since Equation (12) coincides with Equation (10), we have

$$\begin{array}{c}\hfill {\tilde{g}}_{\ell }\left(m\right)=p_{X|\widehat{X}}^{\star }(·|g\left(m\right)).\end{array}$$

(13)

Remark 4.

In Section 3, we directly defined $g_{\ell }\left(m\right)=p_{X|\widehat{X}}^{\star }(·|g\left(m\right))$. However, we obtained ${\tilde{g}}_{\ell }\left(m\right)$ via the following two-step procedure:

• extend the reconstruction set from $𝒴$ to $\mathcal{M}\left(\mathcal{X}\right)$, then characterize optimal decoding functions $g_{\ell }^E\left(m\right)\in \mathcal{M}\left(\mathcal{X}\right)$; and
• find the measure ${\tilde{g}}_{\ell }\left(m\right)\in \mathcal{Y}$ that is closest to $g_{\ell }^E\left(m\right)$.

The above result (13) implies that ${\tilde{g}}_{\ell }\left(m\right)=g_{\ell }\left(m\right)$.

5. Log-Convex Relaxation

In the previous section, we obtained the optimal reconstruction symbol from the extended reconstruction alphabet, and projected it to the feasible set. In this section, instead of direct projection to $𝒴$, we propose another slight extension of $𝒴$, namely, log-convex hull. As we show in the following sections, the log-convex hull has interesting properties.

5.1. rI-Projection

Before defining the log-convex hull, we need to define the log-convex combination of probability distributions. Let p and q be probability distributions in $\mathcal{M}\left(\mathcal{X}\right)$. For $0<t<1$, the log-convex combination of p and q is given by

$$\begin{array}{c}\hfill \overline{p^t{q}^{1-t}}\left(x\right)=\frac{p{\left(x\right)}^{t}q{\left(x\right)}^{1-t}}{{\sum }_{\tilde{x}}p{\left(\tilde{x}\right)}^{t}q{\left(\tilde{x}\right)}^{1-t}}.\end{array}$$

(14)

It is clear to see that $log\overline{p^t{q}^{1-t}}$ is a convex combination of $logp\left(x\right)$ and $logq\left(x\right)$ with a normalizing constant. We can now define log-convex hull logconv$\left(\mathcal{Y}\right)$ that is a set of log-convex combination of probability measures in set $𝒴$. More precisely,

$$\begin{array}{c}\hfill \mathrm{logconv}\left(\mathcal{Y}\right)=\left\{q^{\left(r\right)}\in \mathcal{M}\left(\mathcal{X}\right):q^{\left(r\right)}\left(x\right)=\frac{1}{c\left(r\right)}exp\left(\sum _{\widehat{x}}r\left(\widehat{x}\right)log{p}_{X|\widehat{X}}^{\star }\left(x\right|\widehat{x})\right)\right\}\end{array}$$

where r is a weight vector (i.e., $r\in \mathcal{M}\left(\mathcal{X}\right)$), and $c\left(r\right)$ is a normalizing constant. By definition, logconv$\left(\mathcal{Y}\right)$ is log-convex since it contains all log-convex combinations of probability distributions in $𝒴$.

Instead of having projection of $p_{X\left|f\right(X)}(·|m)$ to $𝒴$, we consider the projection to logconv$\left(\mathcal{Y}\right)$. Since logconv$\left(\mathcal{Y}\right)$ is log-convex, ([17], [Theorem 1]) implies that there exists unique probability distribution $q_m^{\star }\in \mathrm{logconv}\left(\mathcal{Y}\right)$ that achieves the following minimum.

$$\begin{array}{c}\hfill \underset{q\in \mathrm{logconv}\left(\mathcal{Y}\right)}{min}D\left(p_{X\left|f\right(X)}(·|m)\parallel q\right).\end{array}$$

Projection $q_m^{\star }$ is called an rI-projection of $p_{X\left|f\right(X)}(·|m)$ to logconv$\left(\mathcal{Y}\right)$. Let $r_m^{\star }$ be the corresponding weights, i.e.,

$$\begin{array}{c}\hfill q_m^{\star }=q^{\left(r_m^{\star }\right)}.\end{array}$$

Csiszár and Matúš ([17], [Theorem 1]) showed that the rI-projection satisfies the following inequality for all $\widehat{x}$ ∈ $\widehat{\mathcal{X}}$.

$$\begin{array}{c}\hfill D(p_{X\left|f\right(X)}(·|m)\parallel p_{X|\widehat{X}}^{\star }(·|\widehat{x}))\ge D(p_{X\left|f\right(X)}(·|m)\parallel q_m^{\star })+D(q_m^{\star }\parallel p_{X|\widehat{X}}^{\star }(·|\widehat{x})).\end{array}$$

(15)

On the other hand, the log-convex combination of probability measures $q^{\left(r\right)}$ is called the geometric mean of probability measures [18]. The author also provided geometric compensation identity, which is given by

$$\begin{array}{cc}\hfill \sum _{\widehat{x}}r\left(\widehat{x}\right)D(p_{X\left|f\right(X)}(·|m)\parallel p_{X|\widehat{X}}^{\star }(·|\widehat{x}))=& D(p_{X\left|f\right(X)}(·|m)\parallel q^{\left(r\right)})+\sum _{\widehat{x}}r\left(\widehat{x}\right)D(q^{\left(r\right)}\parallel p_{X|\widehat{X}}^{\star }(·|\widehat{x})).\hfill \end{array}$$

(16)

The above result holds for any $r\in \mathcal{M}\left(\mathcal{X}\right)$; therefore, Equation (16) also holds when $q^{\left(r\right)}=q_m^{*}$. Together with Inequality (15), we get the following result. For all $\widehat{x}$ ∈ $\widehat{\mathcal{X}}$, if $r_m^{\star }\left(\widehat{x}\right)\ne 0$, then

$$\begin{array}{cc}\hfill D(p_{X\left|f\right(X)}(·|m)\parallel p_{X|\widehat{X}}^{\star }(·|\widehat{x}))=& D(p_{X\left|f\right(X)}(·|m)\parallel q_m^{\star })+D(q_m^{\star }\parallel p_{X|\widehat{X}}^{\star }(·|\widehat{x})).\hfill \end{array}$$

Remark 5.

The above result is similar to the projection to polytope in Euclidean space. Suppose vectors $v_1,v_2,\dots ,v_n$ form a polytope, and consider the projection from a vector w to the polytope. Let h be a projection. Then, h is a convex combination of $v_i$’s. Thus, there exist coefficients ${\left\{a_i\right\}}_{1\le i\le n}$, such that

$$\begin{array}{c}\hfill h=\sum _{i=1}^n{a}_i{v}_i\end{array}$$

where ${\sum }_i{a}_i=1$, and $a_i\ge 0$ for all i. Let $E=\{1\le i\le n:a_i\ne 0\}$ be the set of indices of nonzero coefficients. Then, projection h is on the plane generated by ${\left\{v_i\right\}}_{i\in E}$. Thus, two vectors $w-h$ and $h-v_i$ are orthogonal for all $i\in E$. Then, Pythagorean theorem implies that, for all i, we have either $a_i=0$ or

$$\begin{array}{c}\hfill \parallel w-v_i{\parallel }^2={\parallel w-h\parallel }^2+{\parallel h-v_i\parallel }^2.\end{array}$$

5.2. Optimization

As we saw in the previous section, we want to find $q\in \mathrm{logconv}\left(\mathcal{Y}\right)$ that minimizes $D\left(p_{X\left|f\right(X)}(·|m)\parallel q\right)$. Note that

$$\begin{array}{cc}\hfill D\left(p_{X\left|f\right(X)}(·|m)\parallel q^{\left(r\right)}\right)=& \sum _{x\in \mathcal{X}}{p}_{X\left|f\right(X)}\left(x\right|m)log\frac{p_{X\left|f\right(X)}\left(x\right|m)}{q^{\left(r\right)}\left(x\right)}\hfill \\ \hfill =& \sum _{x\in \mathcal{X}}{p}_{X\left|f\right(X)}\left(x\right|m)log{p}_{X\left|f\right(X)}\left(x\right|m)+\sum _{x\in \mathcal{X}}{p}_{X\left|f\right(X)}\left(x\right|m)log\frac{1}{q^{\left(r\right)}\left(x\right)}.\hfill \end{array}$$

Since the first term is not a function of $q^{\left(r\right)}$, it is enough to consider the second term. By the definition of $q^{\left(r\right)}$, we have

$$\begin{array}{cc}\hfill \sum _{x\in \mathcal{X}}{p}_{X\left|f\right(X)}\left(x\right|m)log\frac{1}{q^{\left(r\right)}\left(x\right)}=& -\sum _{x\in \mathcal{X}}{p}_{X\left|f\right(X)}\left(x\right|m)\sum _{\widehat{x}}r\left(\widehat{x}\right)log{p}_{X|\widehat{X}}^{\star }\left(x\right|\widehat{x})+logc\left(r\right)\hfill \\ \hfill =& -\sum _{x\in \mathcal{X}}{p}_{X\left|f\right(X)}\left(x\right|m)\sum _{\widehat{x}}r\left(\widehat{x}\right)log{p}_{X|\widehat{X}}^{\star }\left(x\right|\widehat{x})\hfill \\ & +log\left(\sum _{x^{\prime }}exp\left(\sum _{\widehat{x}}r\left(\widehat{x}\right)log{p}_{X|\widehat{X}}^{\star }\left(x^{\prime }\right|\widehat{x})\right)\right).\hfill \end{array}$$

Thus, minimizing $D(p_{X\left|f\right(X)}(·|m)\parallel q)$ is equivalent to solving the following optimization problem.

$$\begin{array}{cc}\hfill \underset{r\in \mathcal{M}\left(\widehat{\mathcal{X}}\right)}{min}& -\sum _{\widehat{x}}r\left(\widehat{x}\right)\sum _x{p}_{X\left|f\right(X)}\left(x\right|m)log{p}_{X|\widehat{X}}^{\star }\left(x\right|\widehat{x})+log\left(\sum _{x^{\prime }}exp\left(\sum _{\widehat{x}}r\left(\widehat{x}\right)log{p}_{X|\widehat{X}}^{\star }\left(x^{\prime }\right|\widehat{x})\right)\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& r\left(\widehat{x}\right)\ge 0\hfill \\ \hfill & \sum _{\widehat{x}}r\left(\widehat{x}\right)=1.\hfill \end{array}$$

Since the objective function is a convex function of $r\left(\widehat{x}\right)$, the above problem is a convex optimization problem that can be efficiently solved.

5.3. Relaxation in Clustering

In the corresponding lossy-compression problem under logarithmic loss, reconstruction symbols are probability measures that have a natural algebraic structure, as we discussed in Section 3.3.3. In this section, we present the benefits of such a property when we apply some known techniques from the clustering literature.

Lossy compression is closely related to the clustering problem [19,20,21]. Many works focused on the application of k-means clustering to a lossy-compression problem [22,23,24], which is an extension of the Lloyd max algorithm [25,26]. However, k-means clustering is only available when there exists a well-defined operation in $\widehat{\mathcal{X}}$ (e.g., $\widehat{\mathcal{X}}={\mathbb{R}}^n$). This is because k-means clustering requires computing the mean of data points, which is the center of each cluster. In general lossy-compression problems, reconstruction alphabet $\widehat{\mathcal{X}}$ may not have such an operation. In such cases, we may have to apply k-medoidlike clustering [27], where the center of each cluster has to be a data point. The k-medoidlike algorithm in the context of lossy compression is shown in Algorithm 1.

Algorithm 1k-medoidlike clustering in lossy compression.

Randomly initialize ${\widehat{x}}_1,\dots ,{\widehat{x}}_M\in \widehat{\mathcal{X}}$  repeat   Set $A_m\leftarrow \varnothing$ for all $1\le m\le M$.   for $x\in \mathcal{X}$ do    $A_m\leftarrow A_m\cup \left\{x\right\}$ where $m=\underset{m^{\prime }}{\mathrm{argmin}}d(x,{\widehat{x}}_{m^{\prime }})$   end for   for $m=1$ to M do    ${\widehat{x}}_m\leftarrow \underset{\widehat{x}\in \widehat{\mathcal{X}}}{\mathrm{argmin}}{\sum }_{x\in A_m}{p}_X\left(x\right)d(x,\widehat{x})$   end for  until converge

On the other hand, in the corresponding problem, the reconstruction alphabet is the set of probability distributions where operations such as log-convex combinations are well-defined. This allows us to propose a k-meanslike clustering algorithm, as shown in Algorithm 2.

Algorithm 2k-meanslike clustering in lossy compression.

Randomly initialize $r_1,\dots ,r_M\in \mathcal{M}\left(\widehat{\mathcal{X}}\right)$  repeat   Set $A_m\leftarrow \varnothing$ for all $1\le m\le M$   for $x\in \mathcal{X}$ do    $A_m\leftarrow A_m\cup \left\{x\right\}$ where $m=\underset{m^{\prime }}{argmin}log\frac{1}{q^{\left(r_{m^{\prime }}\right)}\left(x\right)}$    Set $f\left(x\right)=m$ where $x\in A_m$   end for   for $m=1$ to M do    $r_m\leftarrow \underset{r\in \mathcal{M}\left(\widehat{\mathcal{X}}\right)}{argmin}D(p_{X\left|f\right(X)}(·|m)\parallel q^{\left(r\right)})$   end for  until converge

The main idea of the above algorithm is that log-convex combination $q_m^{\star }$ behaves like center of cluster $A_m$. In the clustering literature, there are many known variations of k-means clustering [28,29]. The above result shows that we can borrow those techniques and apply them to the lossy-compression problem even without any algebraic structures on the reconstruction alphabet.

5.4. Application to General Clustering Problems

The idea of the previous section can be applied to an actual clustering problem. We mainly focus on clustering categorical data where data points are not in continuous space [30,31,32,33,34]. Since operations such as mean are not well-defined in this case, it is hard to apply known data-clustering algorithms in continuous space. The key idea is that the equivalence relation with logarithmic loss allows the algebraic structure on any set. More precisely, we can transform any clustering problem to the clustering problem in continuous space and apply known techniques such as variations of k-means.

A more rigorous definition of the problem is given below. Assume that we have a finite set of data points $𝒳$, and each data point has its weight $p_X\left(x\right)$. We normalize the weights so that ${\sum }_x{p}_X\left(x\right)=1$, and the weights may or may not be uniform. The distance between two points are given by measure $d:\mathcal{X}\times \mathcal{X}\to [0,\infty )$. Suppose we want to partition the data points into M clusters.

If we let $\widehat{\mathcal{X}}$ = $𝒳$, then the clustering problem turns out to be a lossy-compression problem under distortion measure $d(·,·)$, where the number of messages is M. Let $D=D^{\star }\left(M\right)$ be the optimal achievable distortion, and $p_{\widehat{X}|X}^{\star }$ be the distribution that achieves rate-distortion function $R\left(D\right)$ as defined in Equation (4). Then, we can find the corresponding lossy-compression problem under logarithmic loss. Finally, we can apply clustering algorithms in continuous space such as k-means to the corresponding problem. For example, Algorithm 2 can be applied to the corresponding problem.

Remark 6.

Note that it is hard to have an exact analytic formula for $D^{\star }\left(M\right)$ or $p_{|X\widehat{X}}^{\star }$. However, as we mentioned in Section 3.3.2, we do not have to find an optimal scheme under the exact problem formulation. If we can provide a good scheme of the corresponding problem with $D\approx D^{\star }\left(M\right)$, that should be a good enough scheme in the original problem.

6. Conclusions

To conclude our discussion, we summarize our main contributions. We showed that for any fixed-length lossy-compression problem under an arbitrary distortion measure, there exists a corresponding lossy-compression problem under logarithmic loss where optimal schemes coincide. We also proved that a good scheme for one lossy-compression problem is also good for another problem. This equivalence provides an algebraic structure on any reconstruction alphabet that allows using various optimization techniques in lossy-compression problems, such as log-convex relaxation. Furthermore, our results naturally suggest a k-meanslike clustering algorithm in categorical data-clustering problems.