Soft Classification in a Composite Source Model

Abstract

A composite source model consists of an intrinsic state and an extrinsic observation. The fundamental performance limit of reproducing the intrinsic state is characterized by the indirect rate–distortion function. In a remote classification application, a source encoder encodes the extrinsic observation (e.g., image) into bits, and a source decoder plays the role of a classifier that reproduces the intrinsic state (e.g., label of image). In this work, we characterize the general structure of the optimal transition probability distribution, achieving the indirect rate–distortion function. This optimal solution can be interpreted as a “soft classifier”, which generalizes the conventionally adopted “classify-then-compress” scheme. We then apply the soft classification to aid the lossy compression of the extrinsic observation of a composite source. This leads to a coding scheme that exploits the soft classifier to guide reproduction, outperforming existing coding schemes without classification or with hard classification.

1. Introduction

In [1,2], information transmission with additional noise is studied. In this setting, the encoder has access only to a noise-corrupted version of the original signal, while the goal is to minimize the expected distortion between the original signal and the final output. Under the mean-squared error (MSE) distortion criterion, the optimality of an estimate-then-compress scheme is proved. This indirect compression model shares similarities with the semantic communication problem, which has recently attracted much attention. A common model for studying semantic communication is the composite source model [3], which consists of an intrinsic state and an extrinsic observation. The intrinsic state corresponds to the semantic aspect of information, while the encoder observes only the extrinsic observation. In classification tasks, the states are discrete, and a scheme similar to the one mentioned above is the hard-classify-then-compress (HCTC) scheme, i.e., first performing optimal classification based on the observation, then compressing the classification result.

As will be shown in this paper, in general, however, this HCTC scheme is suboptimal due to the inherent “ambiguity” of the states. In a typical composite source, some observations correspond to multiple states with similar posterior probabilities. In this case, losslessly transmitting the classification result offers limited performance gains. The HCTC scheme, however, compresses the classification results indifferently, without accounting for the “ambiguity” of the observations. In [4,5], a soft classification scheme is investigated in a binary classification model. Unlike the HCTC scheme, the soft classification scheme directly compresses the observation and leverages the “ambiguity” of the source. In this paper, the study is further conducted and generalized to multi-class classification.

The value of the knowledge of semantic information in reconstructing observations has also received much attention. In most cases, a codebook precisely matched to the source statistics is hard to obtain. In contrast, the Gaussian codebook is well studied and commonly used. In this work, the role of the classification results obtained through the soft classification scheme in aiding the reconstruction of observations is investigated. We propose a classification-aided-compression (CAC) scheme and study its rate–distortion properties. An achievable upper bound proposed in [6], where the codebook precisely matched to the source statistics is available, is used for comparison. Our work compares the rate–distortion performance achievable using Gaussian codebooks, assisted by different classifiers when the matched codebook is unavailable, with the performance achievable when the matched codebook is available. Numerical results show the benefit of soft classification in improving the rate–distortion performance when only Gaussian codebooks are available, especially at high rates.

1.1. Related Works

In his landmark work [7,8], Shannon excluded the semantic aspects of information from the framework of his classical rate–distortion theory, asserting that the semantic information is irrelevant to the engineering problem. This viewpoint that communication systems should focus on symbol transmission has since become a foundational paradigm in source coding. However, in many practical scenarios, the encoder has only indirect access to the target signal (e.g., through observations corrupted by channel noise or measurement errors). Such indirect source coding problems have been studied in [1,2], but largely limited to simple cases involving AWGN.

To enhance the performance of communication systems in such scenarios, there has been ongoing consideration of how to leverage the semantic aspects of information in communication [9,10,11,12]. These works aim to establish a universal, task-independent definition of semantic information. In recent years, with the development of 5G and the growing interest in post-5G technologies, some researchers have taken the semantic aspects of information into account when designing communication systems [13,14,15,16], to enable new applications in emerging scenarios. Compared to the former works, in these studies, the semantic aspects of information are associated with specific goals. Our work aligns with the latter. Task-oriented compression has been investigated in several works, with quantization being the primary method [17,18,19,20]. In such works, classification [21,22], detection [23], and inferring a latent variable from compressed data [24] are common tasks.

A commonly used model for studying semantic communication is the composite source model [3], which consists of an intrinsic state and multiple sub-sources. Different values of the intrinsic state correspond to different sub-sources and output symbol distributions. The source coding problem for composite sources is studied in [25]. The rate–distortion problems of composite sources under various distortion metrics and constraints are investigated in [4,26], where the composite source model is also connected to the semantic aspects of information. Such rate–distortion problems involving an intrinsic state or noisy observations are referred to as “indirect rate–distortion problems” [1,2,3,27,28]. Two cases have been studied in [4]. In the first case, the state and the observation obey a joint Gaussian distribution, which is further explored in [27]. The second case involves binary classification, and the corresponding solution implies a soft classification scheme. In our work, this case is extended to multi-class classification settings.

The concept of semantic information and the composite source model has also drawn attention in real-world signal processing tasks such as image and speech processing. In certain machine vision tasks, only specific semantic features are required, and the bitrate needed to represent these features is often much lower than that required to encode the entire image. In [29], a scalable image compression scheme is proposed, which simultaneously satisfies the requirements of both human vision and machine vision tasks. The composite source model has also been used to characterize image and speech signals [30,31,32,33]. A widely used model in speech signal processing, the Hidden Markov Model (HMM), is essentially a type of composite source model [34]. In image classification tasks, the label of an image can be viewed as the intrinsic state, with the image generated by its corresponding sub-source. The classification task thus becomes equivalent to inferring the intrinsic state from the image [35]. Other tasks such as recognition [36] and anomaly detection [37] can be modeled in a similar manner. Recently developed generative models, such as the variational auto-encoder [38] and the diffusion model [39], are also related to the composite source model. Taking the variational auto-encoder as an example, different latent variables correspond to different output distributions, similar to how different intrinsic states correspond to different observation distributions in the composite source model. Beyond signal processing, composite sources have also been applied to other practical applications. In [40], a multi-source or composite source information fusion method is studied in the context of the failure mode and effects analysis (FEMA) problem, where assessments from different experts are treated as different sources. The model is also well-suited for capturing ambiguity in classification tasks, where observations do not correspond to unique labels but instead map to multiple possible states with different probabilities. In such scenarios, soft classification is employed [41,42,43].

Despite the numerous studies on semantic information and composite sources, several important issues remain to be addressed. For the composite source classification, studies such as [4,5] have only investigated the case of symmetric binary classification. Nevertheless, many practical tasks involve more general scenarios, such as multi-class classification. In these cases, the soft classification scheme that achieves the classification rate–distortion function is difficult to obtain. In [44], the authors propose a classification-based approach to reconstruct sparse sources, indicating the value of classification when the codebook matched to the source statistics is unavailable. However, only hard classification is considered therein. To the best of our knowledge, no existing work has considered the tradeoff between the classification rate and the effectiveness of classification in reducing the reconstruction rate under this kind of scenarios.

1.2. Contribution and Organization of Paper

Our main contributions include:

• We characterize the classification rate–distortion function for general multi-class composite sources under classification distortion constraint only.
• Based on this setting, we study two schemes: the HCTC scheme and the soft classification scheme. We convert the rate–distortion performance of the HCTC scheme into the rate–distortion function of a discrete source. We also identify sufficient conditions for the rate–distortion optimality of both schemes and analyze the rate–distortion properties of the soft classification scheme. Our analysis shows that, by allowing only a small additional distortion compared to the minimum achievable distortion, the upper bound of the rate of the soft classification scheme can decrease rapidly. Through numerical results, we compare the performance of the two schemes and show that each has different strengths and weaknesses under different scenarios.
• When the reconstruction distortion is constrained, we derive an achievable upper bound of the reconstruction rate–distortion function and propose the CAC scheme using only Gaussian codebooks. Numerical results show that, with proper classification before compression, the rate–distortion performance of the CAC scheme can approach that of the scheme with a matched codebook. We also find that, under high-resolution conditions, the total bitrate of the CAC scheme can be minimized by separately optimizing the classifier and each sub-encoder.

The remainder of the paper is organized as follows.

Section 2 introduces and describes the composite source model and formulates the problem as an indirect rate–distortion problem. In Section 3, the classification rate–distortion function is characterized. We further study two achievable upper bounds of the rate–distortion function and, based on these, propose two classification schemes. Section 4 derives an upper bound of the rate–distortion function under reconstruction distortion constraint, and based on this, proposes the CAC scheme. Section 5 presents the numerical results. For cases with only the classification distortion constraint, we compare the performance of the two schemes. For cases with the reconstruction distortion constraint, we evaluate the performance of the proposed CAC scheme under classifiers of different levels. Finally, Section 6 concludes the paper.

2. Problem Formulation

The composite source consists of two parts: the intrinsic state S and the extrinsic observation X. Within the composite source, different states correspond to different sub-sources of observation X, each with a distinct distribution.

In the composite source coding, the encoder and decoder do not have access to the state S, while the observation X is available to the encoder [4]. The state can only be inferred from the extrinsic observation. Assume that the source has L states, with prior probabilities $P_1,\dots ,P_L$.

For state S, the conditional distribution of the observation X is denoted as $P(X|S)$. In the following work, we assume that the state S is a discrete label. Let $\mathcal{X}$ and $\mathcal{S}$ denote the alphabets of X and S, respectively. The conditional probability density function (pdf) corresponding to the t-th state is denoted as $f_t(x)$. The pdf of the entire source is denoted by $f(x)$, given by,

$$f(x)=\sum _{t=1}^L{P}_t{f}_t(x).$$

(1)

Let the block length be denoted by n. When encoding a sequence of source observations $X^n$, it is first mapped to an index $\mathcal{W}$ in the codebook. Here $\mathcal{W}$ takes value from $\{1,2,\cdots ,2^{nR}\}$, where R is the rate. On the decoder side, $\mathcal{W}$ is mapped to either the reconstruction of the states (classification result) $\widehat{S}$ or the reconstruction of the observation $\widehat{X}$. The model for this coding problem is illustrated in Figure 1. In all of the schemes described in this work, we assume $n\to +\infty$.

In different scenarios, the primary concern varies. In some cases, the emphasis is on classification distortion [21,22]. In some other cases, reconstruction distortion is the focus [44]. Consider two distortion measures: classification distortion measure $d_s$ and the reconstruction distortion measure $d_o$. Let $D_s$ and $D_o$ denote the distortion constraints on classification and reconstruction, respectively. The rate–distortion problem can then be formulated as an optimization problem that minimizes the mutual information between X and $\widehat{S}$ (or $\widehat{X}$), subject to the corresponding distortion constraint [4]:

$$\begin{array}{cc}\hfill R(D_s)& =minI(X;\widehat{S}),\hfill \\ \hfill s.t.\mathbb{E}\{{\widehat{d}}_s(X,\widehat{S})\}& \le D_s,\hfill \end{array}$$

(2)

or

$$\begin{array}{cc}\hfill R(D_o)& =minI(X;\widehat{X}),\hfill \\ \hfill s.t.\mathbb{E}\{d_o(X,\widehat{X})\}& \le D_o,\hfill \end{array}$$

(3)

where $S,X,\widehat{S}$, and $\widehat{X}$ form a Markov chain $S\leftrightarrow X\leftrightarrow \widehat{S}/\widehat{X}$ and ${\widehat{d}}_s(x,\widehat{s})=\mathbb{E}\{d_s(S,\widehat{s})|X=x\}$. Equation (2) describes the rate–distortion formulation for tasks concerned solely with classification distortion, while (3) corresponds to scenarios where reconstruction distortion is the main focus. In this work, we adopt the Hamming distortion metric to characterize classification distortion, i.e., $d_S(s,\widehat{s})=\mathbb{1}(s\ne \widehat{s})$ and use the MSE to measure the reconstruction distortion.

In the following sections, the rate–distortion function is referred to as the classification rate–distortion function when only classification distortion is constrained, and as the reconstruction rate–distortion function when only reconstruction distortion is constrained.

3. Rate–Distortion Analysis for Classification in Composite Sources

3.1. Classification Rate–Distortion Function

In this section, we consider the case where only the classification distortion is constrained. The corresponding optimization problem is formulated by (2).

Let the transition probability from $X=x$ to $\widehat{S}=t$ be denoted as $g_t(x)$, i.e., $g_t(x)=P(\widehat{S}=t|X=x)$. The expected distortion can be expressed as

$$\begin{array}{cc}\hfill \mathbb{E}\{{\widehat{d}}_S(X,\widehat{S})\}& =\mathbb{E}\{d_S(S,\widehat{S})\}\hfill \\ \hfill & =\sum _{t=1}^L{P}_t\mathbb{E}\{d_S(t,\widehat{S})|S=t\}\hfill \\ \hfill & =\sum _{t=1}^L{P}_t{\int }_{\mathcal{X}}{f}_t(x)\sum _{u=1}^L{g}_u(x)\mathbb{1}(t\ne u)dx\hfill \\ \hfill & =1-\sum _{t=1}^L{P}_t{\int }_{\mathcal{X}}{f}_t(x)\sum _{u=1}^L{g}_u(x)\mathbb{1}(t=u)dx\hfill \\ \hfill & =1-{\int }_{\mathcal{X}}\sum _{t=1}^L{P}_t{f}_t(x)g_t(x)dx,\hfill \end{array}$$

(4)

while the rate (mutual information) is given by [45]

$$I(X;\widehat{S})=H(\widehat{S})-H(\widehat{S}|X).$$

(5)

The two terms $H(\widehat{S})$ and $H(\widehat{S}|X)$ in the equation above can also be expressed with $g_t(x)$. Therefore, the optimization problem can be solved by optimizing the transition probability $g_t(x)$. As a result, the classification rate–distortion function can be characterized based on the optimization result.

Theorem 1. 

The classification rate–distortion function of a composite source is given by

$$R(D_s)=\sum _{t=1}^L{\int }_{\mathcal{X}}f(x)g_t(x){log}_2{\displaystyle \frac{g_t(x)}{{\int }_{\mathcal{X}}f(x)g_t(x)dx}}dx,$$

(6)

where

$$g_t(x)={\displaystyle \frac{P(\widehat{S}=t)e^{\frac{\lambda P_t{f}_t(x)}{f(x)}}}{{\displaystyle \sum _{w=1}^L}P(\widehat{S}=w)e^{\frac{\lambda P_w{f}_w(x)}{f(x)}}}},$$

(7)

$$P(\widehat{S}=t)={\int }_{\mathcal{X}}f(x)g_t(x)dx,$$

(8)

and the parameter λ is chosen such that

$$1-{\int }_{\mathcal{X}}\sum _{t=1}^L{P}_t{f}_t(x)g_t(x)dx=D_s.$$

(9)

Proof. 

See Appendix A. □

Remark 1. 

In Theorem 1, $g_t(x)$ represents the transition probability $P(\widehat{S}=t|X=x)$. The theorem shows that, for a fixed $X=x$, the transition probability from $X=x$ to $\widehat{S}=t$ satisfies

$$g_t(x)\propto P(\widehat{S}=t)e^{\lambda P(S=t|X=x)}.$$

(10)

The result aligns with the intuition that, for a fixed $X=x$, the transition probability $g_t(x)$ increases with the posterior probability $P(S=t|X=x)$.

The form of $g_t(x)$ implies a soft classification scheme. In this scheme, as λ increases, $g_t(x)$ approaches a step function, indicating that the classification becomes “harder”. At the same time, the rate increases and the distortion decreases. When $\lambda \to +\infty$, we have

$$g_t(x)\to \left\{\begin{array}{ccc}\hfill 1& & t=t^{\ast }(x)\hfill \\ \hfill 0& & otherwise,\hfill \end{array}\right.$$

(11)

for all x where the most probable state $t^{\ast }(x)=arg\underset{t}{max}P(S=t\mid X=x)$ is unique. In this case, the classification reduces to hard classification. Thus, at the far left of the RD curve, the rate approaches the entropy of the hard classification results.

Conversely, as λ decreases towards 0, the functions $g_t(x)$ for $t=[1:L]$ become smoother, making the classification more “random”. As a result, the distortion increases while the rate decreases. By adjusting λ, the functions $g_t(x)$ can be tuned to satisfy the distortion constraint.

Equation (7) also involves the marginal probability of $\widehat{S}$, assigning larger weights to the states with higher marginal probabilities. This helps reduce the first term $H(\widehat{S})$ in (5).

3.2. Two Classification Schemes

Next, two achievable upper bounds of the classification rate–distortion function are introduced and analyzed, along with the corresponding classification–compression schemes.

3.2.1. Hard-Classify-Then-Compress

The first scheme is called the hard-classify-then-compress (HCTC) scheme. In this scheme, a hard classification is first performed on the observation X, and the resulting classification outcome is denoted as $\overline{S}$. This hard classification result $\overline{S}$ is then compressed into $\widehat{S}$.

The procedure of the HCTC scheme is illustrated in Figure 2. As shown in the figure, the following Markov chain is formed:

$$S⟷X⟷\overline{S}⟷\widehat{S}.$$

(12)

Define

$$\begin{array}{cc}\hfill {\tilde{d}}_s(\overline{s},\widehat{s})& ={\mathbb{E}}_S\{d_s(S,\widehat{s})|\overline{S}=\overline{s}\}\hfill \\ \hfill & ={\mathbb{E}}_S\{\mathbb{1}(S\ne \widehat{s})|\overline{S}=\overline{s}\}\hfill \\ \hfill & =1-{\mathbb{E}}_S\{\mathbb{1}(S=\widehat{s})|\overline{S}=\overline{s}\}\hfill \\ \hfill & =1-P(S=\widehat{s}|\overline{S}=\overline{s}).\hfill \end{array}$$

(13)

Since the joint distribution of S and $\overline{S}$ is known, $\tilde{d}(\overline{s},\widehat{s})$ can be calculated for each pair $(\overline{S},\widehat{S})$.

Theorem 2. 

The rate–distortion behavior of the HCTC scheme is equivalent to the rate–distortion function of the hard classification result $\overline{S}$ using the distortion metric defined in (13).

Proof. 

See Appendix B. □

Remark 2. 

Theorem 2 shows that the rate–distortion performance of the HCTC scheme can be reduced to the rate–distortion function of a discrete source, enabling numerical computation using the Blahut–Arimoto (BA) algorithm [46] and its variants such as [47].

Corollary 1. 

A sufficient condition for the optimality of the HCTC scheme is as follows: $\forall x_1,x_2\in {\mathcal{X}}_u, t=[1:L]$, $P(S=t|X=x_1)=P(S=t|X=x_2)$, where ${\mathcal{X}}_u=\{x|P(S=u|X=x)\ge P(S=i|X=x),\forall i=[1:L]\}$.

Proof. 

See Appendix C. □

Remark 3. 

The condition in Corollary 1 implies that all the observations X leading to the same hard classification result convey identical information about the intrinsic state S. Consequently, compressing the hard classification result $\overline{S}$ introduces no additional distortion compared to directly compressing the original observation X.

3.2.2. Symmetric Cases and Soft Classification

Based on Theorem 1, the classification rate–distortion function can be computed by alternatively optimizing $g_t(x)$ and $P(\widehat{S}=t),t=[1:L]$, in a manner similar to the BA algorithm [48].

When $\lambda >0$, if all $P(\widehat{S}=t),t=1,2,\cdots$ are initialized as ${\displaystyle \frac{1}{L}}$ and remain unchanged after the first iteration, then in all subsequent iterations, we have

$$\begin{array}{cc}\hfill g_t{(x)}^{(i)}& ={\displaystyle \frac{P{(\widehat{S}=t)}^{(i-1)}{e}^{\frac{\lambda P_t{f}_t(x)}{f(x)}}}{{\displaystyle \sum _{w=1}^L}P{(\widehat{S}=w)}^{(i-1)}{e}^{\frac{\lambda P_w{f}_w(x)}{f(x)}}}}\hfill \\ \hfill P{(\widehat{S}=t)}^{(i)}& ={\displaystyle \frac{1}{L}},\forall t\in [1:L],\hfill \end{array}$$

(14)

thus achieving convergence. For such composite sources, the optimal transition functions are given by

$$g_t(x)={\displaystyle \frac{e^{\frac{\lambda P_t{f}_t(x)}{f(x)}}}{{\displaystyle \sum _{w=1}^L}{e}^{\frac{\lambda P_w{f}_w(x)}{f(x)}}}},\forall t\in [1:L].$$

(15)

Equation (15) shows that $g_t(x)$ takes the form of a softmax function with coefficient $\lambda$ applied to the conditional probability $P(S=t|X=x)$. Unlike (7), this expression does not involve the marginal probability of $\widehat{S}$. Consequently, $g_t(x)$ can be directly derived from the joint distribution of S and X using (15), without the need to solve any nonlinear equations. This significantly simplifies the computation.

Lemma 1. 

The optimal transition probability functions take the form of (15), if and only if

$${\int }_{\mathcal{X}}f(x){\displaystyle \frac{e^{\frac{\lambda P_t{f}_t(x)}{f(x)}}}{{\displaystyle \sum _{w=1}^L}P{e}^{\frac{\lambda P_w{f}_w(x)}{f(x)}}}}dx={\displaystyle \frac{1}{L}},\forall t\in [1:L].$$

(16)

Proposition 1. 

One type of composite source that satisfies the condition in Lemma 1 is as follows (assume the source has L states):

• The number of states is finite, and the observation space satisfies $\mathcal{X}\subseteq {\mathbb{R}}^K$.
• The probability distributions of the sub-sources are distinct.
• $P_t={\displaystyle \frac{1}{L}},\forall t\in [1:L]$.
• Let ${\mu }_t$ denote the mean of the t-th sub-source. There exists a point ${\mu }_0$ such that $\parallel {\mu }_t-{\mu }_0{\parallel }_2^2$ is equal for all t. For simplicity and without loss of generality, assume ${\mu }_0=0$ in the following discussion.
• For any two sub-sources indexed by a and b, there exists an orthonormal matrix H such that

  (1) 

  ${\mu }_b=H{\mu }_a$.

  (2) 

  $\forall t\in [1:L],\exists u\in [1:L],s.t.$

  $$f_u(Hx)=f_t(x),\forall x\in \mathcal{X},$$

  (17)

  specifically,

  $$f_b(Hx)=f_a(x),\forall x\in \mathcal{X}$$

  (18)

  (3) 

  If $x\in \mathcal{X}$, $Hx\in \mathcal{X}$.

Proof. 

See Appendix D. □

Unfortunately, many practical scenarios do not satisfy the conditions in Proposition 1. Nevertheless, (15) can still be employed to obtain an achievable upper bound of the classification rate–distortion function in cases where the rate–distortion function itself is difficult to solve. Moreover, (15) induces a simplified soft classification scheme, in which the conditional probabilities $P(\widehat{S}=t|X=x)$ follow the transition functions $g_t(x)$ in (15). In the remainder of this paper, the soft classification scheme refers to this simplified version.

Assume a length-n sequence of observation $X^n$ (with n sufficiently large). The steps of the soft-classification are as follows:

• Calculate the transition probabilities from observation X to the classification result $\widehat{S}$ using (20).
• Calculate the marginal distribution $P(\widehat{S})$ and the mutual information $I(X;\widehat{S})$.
• $\forall R>I(X;\widehat{S})$, randomly generate a codebook $\mathcal{C}$ containing $2^{nR}$ i.i.d. sequences ${\widehat{S}}^n$ drawn according to $P(\widehat{S})$. Each sequence is a codeword, indexed by $\mathcal{W}\in \{1,2,\cdots ,2^{nR}\}$.
• When encoding, select the codeword that is distortion typical [45] with $X^n$. If there is more than one such ${\widehat{S}}^n$, choose the one with the smallest index $\mathcal{W}$. If no such codeword exists for a given $X^n$, encode it using $\mathcal{W}=1$.
• At the decoder, recover the sequence ${\widehat{S}}^n$ from the received index $\mathcal{W}$ using the codebook $\mathcal{C}$. Due to the properties of the distortion-typical set, the rate–distortion pair $(R,D)$ is achievable for any $R>I(X;\widehat{S})$.

Theorem 3. 

An achievable upper bound on the classification rate–distortion function is given by

$$\begin{array}{cc}\hfill R(D_s)& \le I({\tilde{g}}_t)\hfill \\ \hfill & \le logL+{\int }_{\mathcal{X}}f(x)\sum _{t=1}^L{\tilde{g}}_t(x){log}_2{\tilde{g}}_t(x)dx,\hfill \end{array}$$

(19)

where

$${\tilde{g}}_t(x)={\displaystyle \frac{e^{\frac{s{P}_t{f}_t(x)}{f(x)}}}{{\displaystyle \sum _{w=1}^L}{e}^{\frac{s{P}_w{f}_w(x)}{f(x)}}}},$$

(20)

and s is set to satisfy

$$1-{\int }_{\mathcal{X}}\sum _{t=1}^L{P}_t{f}_t(x){\tilde{g}}_t(x)dx=D_s.$$

(21)

$$I({\tilde{g}}_t)=\sum _{t=1}^L{\int }_{\mathcal{X}}f(x){\tilde{g}}_t(x){log}_2{\displaystyle \frac{{\tilde{g}}_t(x)}{\int f(x){\tilde{g}}_t(x)dx}}dx$$

(22)

is the rate required by this scheme. Its upper bound, denoted as $R_{ub}({\tilde{g}}_t)$, is given in the last line of (19).

Proposition 2. 

The slope of the upper bound $R_{ub}({\tilde{g}}_t)$ is $-s{log}_{2}e$, and s varies from 0 to $+\infty$, with the classification changing from “soft” to “hard”. As $s\to +\infty$, $\forall x$ with unique $t^{\ast }$, the classification becomes the hard classification, corresponding to the optimal Bayesian classification. Conversely, When $s=0$, the classifier performs classification completely randomly.

Proof. 

See Appendix E. □

Remark 4. 

Denote the rate of the soft classifier as $R_s$. Let $R_s(s)$ and $D_s(s)$ denote the rate and distortion, respectively, of the classifier parameterized by a fixed s. The corresponding upper bound on the rate is denoted as $R_{ub}(s)$.

Proposition 2 can be expressed as

$${\displaystyle \frac{d{R}_{ub}}{d{D}_s}}=-s{log}_{2}e.$$

(23)

Consider the upper bound $R_{ub}$. Let $D_{s,min}$ denote the minimum achievable distortion, attained by the Bayesian optimal classifier, (i.e., the hard classification). Now suppose a small additional distortion ${\mathsf{\Delta }}_D$ beyond $D_{s,min}$ is acceptable. In this scenario, since the classifier is almost a hard one, we have $s\to +\infty$. When the distortion increases by ${\mathsf{\Delta }}_D$, $R_{ub}$ can be reduced by ${\delta }_{R_{ub}}\approx s{log}_{2}e{\mathsf{\Delta }}_D$. This indicates that even a slight relaxation in the distortion constraint can significantly reduce the rate upper bound of the soft classification scheme.

Equation (20) gives the form of transition probability from the observation X to the classification result $\widehat{S}$. Consider the following example of a four-class classification problem. Assume a composite source with state S and observation X. When $S=t$, $X\sim \mathcal{N}({\mu }_t,I)$, where I is the identity matrix and ${\mu }_1=(1,0),{\mu }_2=(0,1),{\mu }_3=(-1,0),{\mu }_4=(0,-1)$. Figure 3a–c illustrate ${\tilde{g}}_1(x)$ for $s=0,1$, and 100, respectively.

When $s=0$, the transition probability ${\tilde{g}}_1(x)$ is 0.25 for all $X=x$, indicating completely random classification. Comparing the cases where $s=1$ and $s=100$, the latter case is very close to the hard classification since s is very large. As shown in Figure 3c, the transition probability function is almost 0-1 binary.

In the HCTC scheme, the classification results are compressed without considering the posterior probabilities $P(S=t|X=x)$ for different x. Such compression can cause significant distortion since the classification results of X with “clear” states are also compressed. In contrast, in the soft classification scheme (Figure 3b), for $X=x$ with “ambiguous” states (e.g., the point $(0,0)$), $H(\widehat{S}|X=x)$ is significantly greater than 0. Consequently, the rate allocated to these classification results is much lower than that allocated to X with more certain states. Therefore, compression mainly occurs in these “ambiguous” regions. In these regions, the optimal classification already introduces non-negligible distortion. As a result, saving bits for these observations does not lead to significant additional distortion.

When the classification distortion constraint is tight, the rate that can be saved is limited. Therefore, the soft classification scheme, by concentrating bit savings primarily on X with “ambiguous” states, can achieve a smaller distortion than the HCTC scheme.

As shown in Corollary 1, when the “ambiguity” of observations with the same hard classification result is identical, the performance of the HCTC scheme is no worse than that of the soft classification scheme. When the prior probabilities of the states are different and the classification distortion constraint is loose, the performance of the HCTC scheme can even surpass that of the soft classification. In the soft classification scheme, as $s\to 0$, ${\tilde{g}}_t(x)\to {\displaystyle \frac{1}{L}},\forall t=[1:L],x\in \mathcal{X}$. Thus, when $R=0$, $D=1-{\displaystyle \frac{1}{L}}$, the classification becomes completely random, regardless of the prior distribution of the states. In contrast, the HCTC scheme classifies all observations into the class with the highest prior probability, resulting in a smaller distortion compared to the soft classification scheme.

It is also noteworthy that, unlike deep neural networks with the output of the softmax results, our soft classification scheme does not output the posterior distribution of the states. In practical applications, the observations are first compressed into an index $\mathcal{W}$. Then the classification results are decoded based on $\mathcal{W}$. Therefore, the soft classification scheme directly outputs a reconstruction of the state. Its difference from the hard classification is that, when the rate is constrained, by incorporating the source statistics and applying stronger compression to the “ambiguous” observations, the soft classification can reduce the expected distortion.

4. Classification-Aided Reconstruction of Composite Sources

In this section, we consider the case where the reconstruction distortion is constrained. The metric of the reconstruction distortion is the MSE distortion.

Consider an achievable upper bound of $R(D_o)$. This problem can be formulated as the optimization problem (3). One approach is to perform direct lossy compression on the observation. Nevertheless, for most sources, the matched codebook is difficult to obtain. In contrast, for Gaussian sources, which are widely adopted in signal processing tasks, the codebook is easy to obtain. Therefore, in such cases, the codebook of a Gaussian source with the same variance (covariance matrix), can be used as a substitute. However, some sources have large variances, leading to poor rate–distortion performance when compressed directly. In [44], the magnitude classifying quantization (MCQ) scheme is proposed. In this scheme, the observation is first classified according to its amplitude. Then the observations are compressed by sub-encoders corresponding to the classification result. The study shows that classification changes the variance of the input to each sub-encoder. Hence when using Gaussian codebooks, although an additional rate is introduced for classification, the overall rate of the MCQ scheme can still be lower than that of direct compression in some cases.

In this section, the soft classification introduced in Section 3.2.2 is adopted, and the classification-aided-compression (CAC) scheme is proposed. Based on this scheme, an achievable upper bound of $R(D_o)$ is also derived.

Another upper bound of $R(D_o)$ is as follows:

$$\begin{array}{cc}\hfill R(D_o)& \le I(X;\widehat{X})\hfill \\ \hfill & \le I(X;\widehat{S},\widehat{X})\hfill \\ \hfill & =I(X;\widehat{S})+I(X;\widehat{X}|\widehat{S}).\hfill \end{array}$$

(24)

Inequality (24) implies the CAC scheme to encode the observation of an L-state composite source X under the reconstruction distortion $D_o$. The first term represents classification, while the second term represents compression using the corresponding encoder according to the classification result.

The specific steps of the scheme are as follows:

• Select a distortion level D and perform the soft classification. Denote the result as $\widehat{S}$.
• Encode X using the corresponding Gaussian encoder based on the soft classification result.

Figure 4 illustrates the steps of the CAC scheme.

An upper bound of $R(D_o)$ can be derived from the scheme described above.

Theorem 4. 

An achievable upper bound of $R(D_o)$ is

$$R(D_o)\le \underset{D\in [D_{s,min},D_{s,max}]}{min}\{R_s(D)+\sum _{u=1}^{L}P(\widehat{S}=u)R_{{\tilde{X}}_u,g}\},$$

(25)

where $D_{s,min}$ and $D_{s,max}$ are the minimum and maximum classification distortions that can be achieved by the soft classifier, $R_s(D)$ is the rate at the classification distortion D when using the soft classification scheme in Section 3.2.2, $P(\widehat{S}=u)$ is obtained from (8), and

$$R_{{\tilde{X}}_u,g}={\displaystyle \frac{1}{2}}\sum _{i=1}^K{log}_2{\displaystyle \frac{{\sigma }_{ui}^2}{D_{ui}}}.$$

(26)

In (26), K denotes the dimension of X. ${\sigma }_{ui}^2$ is the i-th eigenvalue of ${\mathsf{\Sigma }}_u$, where

$${\mathsf{\Sigma }}_u={\displaystyle \frac{{\int }_{\mathcal{X}}f(x)g_{D,u}(x)(x-{\gamma }_u){(x-{\gamma }_u)}^{T}dx}{{\int }_{\mathcal{X}}f(x)g_{D,u}(x)dx}},$$

(27)

$${\gamma }_u={\displaystyle \frac{{\int }_{\mathcal{X}}xf(x)g_{D,u}(x)dx}{{\int }_{\mathcal{X}}f(x)g_{D,u}(x)dx}},$$

(28)

and $g_{D,u}(x)$ is the transition probability function of the soft classification scheme that satisfies (21) with the right-hand side replaced by D. For $D_{ui}$, we have

$$D_{ui}=min\{\alpha ,{\sigma }_{ui}^2\},$$

(29)

where α is chosen to satisfy

$$\sum _{u=1}^L\sum _{i=1}^{K}P(\widehat{S}=u)D_{ui}=D_o.$$

(30)

Proof. 

See Appendix F. □

For the CAC scheme, in the compression step, X is compressed using a Gaussian encoder. Define

$$R_{ub}(D_o)\triangleq R_s(D)+\sum _{u=1}^{L}P(\widehat{S}=u)R_{{\tilde{X}}_u,g},$$

(31)

where $D\in [D_{s,min},D_{s,max}]$. We have

$$\begin{array}{cc}\hfill R_{ub}(D_o)& =R_s(D)+{\displaystyle \frac{1}{2}}\sum _{u=1}^{L}P(\widehat{S}=u)\sum _{i=1}^K{log}_2{\displaystyle \frac{{\sigma }_{ui}^2}{D_{ui}}}\hfill \\ \hfill & =R_s(D)+{\displaystyle \frac{1}{2}}\sum _{u=1}^{L}P(\widehat{S}=u){log}_2(det({\mathsf{\Sigma }}_u))-{\displaystyle \frac{1}{2}}\sum _{u=1}^L\sum _{i=1}^{K}P(\widehat{S}=u){log}_2{D}_{ui}.\hfill \end{array}$$

(32)

Consider two special cases where $s=0$ and $s\to +\infty$. When $s=0$, (20) becomes ${\tilde{g}}_t=1/L$. Thus $P(\widehat{S}=u)=1/L,\forall u=[1:L]$. Hence $\forall u=[1:L]$, the pdf of X given the condition that $\widehat{S}=u$ is

$$\begin{array}{cc}\hfill f(x|\widehat{S}=u)& ={\displaystyle \frac{P(X=x,\widehat{S}=u)}{P(\widehat{S}=u)}}\hfill \\ \hfill & ={\displaystyle \frac{f(x)\frac{1}{L}}{\frac{1}{L}}}\hfill \\ \hfill & =f(x).\hfill \end{array}$$

(33)

From (33), we can see that when $s=0$, the input to each sub-encoder obeys the same distribution as the composite source. It is also evident that, in this case, the rate required for the classification part is 0. Hence when $s=0$, the CAC scheme reduces to the direct compression. As $s\to +\infty$, the classification becomes hard classification. In this case, the scheme is similar to the MCQ method proposed in [44], where a pre-compression classification based on amplitude is performed.

In the compression step, X is first subtracted by the mean of the corresponding sub-encoder input, resulting in a zero-mean vector for encoding. Then, orthogonal transformations are applied separately to the vectors to be encoded by each sub-encoder, ensuring their components are mutually uncorrelated. After the transformation, the distortion allocation $D_{ui}$ to each component of different classification results follows the classical reverse water-filling strategy. The only difference is that each $D_{ui}$ is weighted by its corresponding probability $P(\widehat{S}=u)$.

When $D_o\le K{\sigma }_{ui}^2,\forall u=[1:L],i=[1:K]$, according to (29), all $D_{ui}$ are equal. Hence, $D_{ui}=\alpha$ and (32) can be rewritten as

$$\begin{array}{cc}\hfill R_{ub}(D_o)& =R_s(D)+{\displaystyle \frac{1}{2}}\sum _{u=1}^{L}P(\widehat{S}=u){log}_2(det({\mathsf{\Sigma }}_u))-{\displaystyle \frac{1}{2}}\sum _{u=1}^L\sum _{i=1}^{K}P(\widehat{S}=u){log}_2\alpha \hfill \\ \hfill & =R_s(D)+{\displaystyle \frac{1}{2}}\sum _{u=1}^{L}P(\widehat{S}=u){log}_2(det({\mathsf{\Sigma }}_u))-{\displaystyle \frac{K}{2}}{log}_2\alpha .\hfill \end{array}$$

(34)

In (34), the third term depends solely on $\alpha$. According to (30), we have

$$D_{ui}=\alpha ={\displaystyle \frac{D_o}{K}},$$

(35)

Hence the third term in (34) is solely determined by $D_o$. Denote $D_T=K{\sigma }^{\ast 2}$, where ${\sigma }^{\ast 2}$ denotes the minimum eigenvalue among all ${\mathsf{\Sigma }}_u$ across classifiers with $s\ge 0$ (with s being the coefficient in (20) that controls the softness of the classifier). It can also be observed that in (34), the first two terms depend only on the classification operation and are independent of $D_o$. Thus, when $D_o\le D_T$, the effects of the classifier and $D_o$ on $R_{ub}$ can be completely decoupled, which is convenient for further analysis. Therefore, the optimal classifier can be obtained by optimizing s to minimize the first two terms when $D_o\le D_T$.

In the high-resolution case, $D_o$ is very small, thus satisfying the condition above. Denote the first two terms in (34) as $RC(s)$, i.e., $RC(s)=R_s(D)+{\displaystyle \frac{1}{2}}{\sum }_{u=1}^{L}P(\widehat{S}=u){log}_2(det({\mathsf{\Sigma }}_u))$. Let $D_o(s,R)$ denote the reconstruction distortion achieved by the CAC scheme when the coefficient of the classifier is s and the total rate is R. From (34), we have

$$\begin{array}{cc}\hfill D_o(s,R)& =K{2}^{{\displaystyle \frac{2}{K}}(RC(s)-R)}\hfill \\ \hfill & =K{2}^{{\displaystyle \frac{2}{K}}RC(s)}{2}^{-{\displaystyle \frac{2}{K}}R}.\hfill \end{array}$$

(36)

Hence in the high-resolution case, the achievable distortion of this scheme exhibits an exponential decay with respect to the rate R. By selecting a proper classifier, the coefficient of the function can be minimized, thus minimizing the reconstruction distortion.

As the reconstruction distortion constraint becomes more relaxed, the reverse water-filling scheme allows for completely omitting the bits used to reconstruct components corresponding to small eigenvalues in the covariance matrices of each classification result.

5. Numerical Result

In this section, we present and compare numerical results of the schemes developed in the previous sections.

5.1. Soft Classification and HCTC

This section presents and compares the performance of the schemes proposed in Section 3.2 under various settings. The procedures of the two schemes are detailed in Section 3.2.1 and Section 3.2.2, respectively.

First, we consider the Gaussian mixture (GM) source model with equal prior probabilities for all states. The distributions of the composite sources in the first seven settings are given as follows. For each setting, let ${\mu }_t$ denote the mean of the t-th state, and ${\mathsf{\Sigma }}_t$ its covariance matrix.

In the first two settings, the sources are 4-state GM sources. In both settings, ${\mu }_1=(1,0)$, ${\mu }_2=(0,1)$, ${\mu }_3=(-1,0)$, ${\mu }_4=(0,-1)$. In the first setting, ${\mathsf{\Sigma }}_t={\mathbf{I}}_{2\times 2},t=1,2,3,4$. In the second setting, ${\mathsf{\Sigma }}_t=2{\mathbf{I}}_{2\times 2},t=1,2,3,4$.

In the next two settings, the sources remain 4-state GM sources, and the covariance matrices of all states are again identity matrices. In the third setting, the mean points of the states are ${\mu }_1=(1,1)$, ${\mu }_2=(-1,1)$, ${\mu }_3=(-1,-1)$, ${\mu }_4=(1,-1)$. In the fourth setting, the mean points of the states are ${\mu }_1=(3,3)$, ${\mu }_2=(-3,3)$, ${\mu }_3=(-3,-3)$, ${\mu }_4=(3,-3)$.

These four settings perfectly satisfy the conditions in Proposition 1. Hence in these settings, the rate–distortion function $R(D_s)$ attains the value of the upper bound $R_{ub}({\tilde{g}}_t)$.

In the fifth setting, the number of states is increased to 9, with the mean points of the different states illustrated in Figure 5a. Furthermore, the number of states is further increased to 16 and 32 as shown in Figure 5b and Figure 5c, respectively. In these three settings, the covariance matrix of each state remains the identity matrix. The performance of the soft classification scheme and the HCTC scheme is compared under these seven settings as depicted in Figure 6.

As shown in Figure 6, the soft classification scheme consistently outperforms the HCTC scheme in all seven settings. In some cases, such as the sources depicted in Figure 6e–g, the conditions in Proposition 1 are not satisfied, indicating that the transition probability functions in (15) are not optimal. In Figure 6e,f, the performance of both schemes is compared with the rate–distortion function obtained from the constrained BA (CBA) algorithm [47]. In these cases, even with the suboptimal transition probability functions (20), the soft classification scheme still outperforms the HCTC scheme, and its performance is close to the rate–distortion function, especially when the distortion is near the minimum. As analyzed in Section 3.2.2, the soft classification scheme can better exploit the “ambiguity” in a composite source, thus reducing the rate without significantly increasing the distortion.

It is also noteworthy that in some sources, the gap between the two schemes is significant, while in some others, it is not. Figure 7 shows how the ratio of the bits required for the soft classification and the HCTC schemes changes as the extra classification distortion allowed beyond the Bayesian minimum classification distortion increases in different sources. In this figure, $D_s$ denotes the distortion constraint, and $D_{s,min}$ denotes the Bayesian minimum distortion. The classification distortion constraint $D_s$ takes the value from $D_{s,min}$ to 0.5 for all sources. The second setting is not shown in this figure since in this setting, the Bayesian minimum distortion is greater than 0.5.

As shown in Figure 7, the ratio of the bits required by the two schemes decreases dramatically as the distortion constraint $D_s$ increases slightly compared to the Bayesian minimum distortion in all sources, except for the fourth setting. In the fourth setting, the distances between the states are large relative to their variances, resulting in a very small probability of “ambiguity” for the state corresponding to the observation. As mentioned in Section 3.2.2, the soft classification scheme primarily exploits the “ambiguity” of the states. Hence in this setting, there is no significant difference between the two schemes.

Now, we compare the first, third, and fourth settings. All three settings have four states, and the covariance matrices of the states are identity matrices. As shown in Figure 7, in the sources where the states are closer to each other, the soft classification scheme achieves more significant bit savings. This still holds when comparing the fourth, fifth, and sixth settings, where the number of states varies. In these three settings, the outermost mean points are located at the same positions. Thus, as the number of states increases, the distances between the states decrease, and the performance gap between the two schemes becomes larger.

An interesting finding emerges from the comparison among the third, sixth, and seventh settings. Although the minimum distances between the mean points are the same across these three sources, the ratio of the bits required by the two schemes approaches one as the number of states increases. A straightforward explanation is that adding more states in this manner does not introduce significant additional “ambiguity” to the source, since, for most observations, only a few (at most 4) states have non-negligible posterior probabilities. Nevertheless, as the number of states increases, both schemes require more bits. Thus the relative gap between the two schemes narrows.

In some other cases, however, the degree of “ambiguity” is consistent across all parts of the source. In such scenarios, the advantages of soft classification become less evident. Consider a 4-class composite source where the observation X is also discrete when $S=i(i=[1:4]),P(X=i)=0.7,P(X=j)=0.1,\forall j=[1:4],j\ne i$.

Figure 8 shows the rate–distortion performance of the HCTC scheme and soft classification scheme in this discrete composite source. Figure 8a illustrates the cases where the prior probabilities of the states are all equal. The result shows that the performance of the two schemes is identical. Figure 8b shows the case where the prior probabilities of the states are unequal, with $P(S=1)=0.4,P(S=2)=P(S=3)=P(S=4)=0.2$. In this setting, the performance of the soft classification scheme is inferior to that of the HCTC scheme, which aligns with the analysis presented in Section 3.2.2.

In Figure 9, the performance of the soft classification scheme, the HCTC scheme, and the rate–distortion function (obtained via the CBA algorithm) is compared in a four-component composite source, where the sub-sources are the same as those in the first setting. The difference lies in the prior probabilities of the states: $P(S=1)=0.4,P(S=2)=0.2,P(S=3)=0.2,P(S=4)=0.2$. This setting clearly illustrates the strengths and weaknesses of the two schemes. When the distortion constraint is very loose, the HCTC scheme performs better since the soft classification tends to classify observations with an almost uniform distribution, while the HCTC scheme tends to classify the observations into the state with higher prior probability, leading to lower distortion. However, when the distortion constraint is tight, the soft classification outperforms the HCTC scheme by better saving bits in the “ambiguous” regions without significantly increasing the distortion.

5.2. Upper Bound of Reconstruction Rate–Distortion Function

In this part, the achievable upper bounds of $R(D_o)$ and their properties are shown in five cases.

The first setting is the same as the four-class Gaussian mixture source in Section 5.1. In the second setting, the distances between different sub-sources are increased, with ${\mu }_1=(3,0)$, ${\mu }_2=(0,3)$, ${\mu }_3=(-3,0)$, ${\mu }_4=(0,-3)$. The covariance matrices of all sub-sources remain identity matrices. The third setting considers a two-component composite source, where X is a scalar. Specifically, $X\sim \mathcal{N}(0,1)$ when $S=1$, and $X\sim \mathcal{N}(0,25)$ when $S=2$. The fourth setting is also a one-dimensional two-component GM source, where $X\sim \mathcal{N}(-1,1)$ when $S=1$, and $X\sim \mathcal{N}(1,25)$ when $S=2$. The fifth setting involves a four-component two-dimensional GM source. When $S=t$, $X\sim \mathcal{N}(0,{\mathsf{\Sigma }}_t)$. We have

$${\mathsf{\Sigma }}_1=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right),{\mathsf{\Sigma }}_2=\left(\begin{array}{cc}1& 0\\ 0& 25\end{array}\right),{\mathsf{\Sigma }}_3=\left(\begin{array}{cc}25& 0\\ 0& 1\end{array}\right),{\mathsf{\Sigma }}_4=\left(\begin{array}{cc}5& 0\\ 0& 5\end{array}\right).$$

(37)

In all of these settings, the distribution of the state S is uniform.

As mentioned in Section 4, the influence of $D_s$ and $D_o$ on $R_{ub}$ can be decoupled, and the optimal classifier can be obtained by minimizing $RC(s)$ when the reconstruction distortion constraint satisfies $D_o\le D_T$. The function $RC(s)$ in the first three settings is shown in Figure 10.

As s increases, the classification rate $R_s(D)$ also increases. Nevertheless, changing s also changes the conditional covariance matrices ${\mathsf{\Sigma }}_u$ corresponding to each classification result. Thus, a tradeoff arises between the classification rate and the determinants of these covariance matrices.

In the first setting, the distributions of different states are close to each other, leading to a small determinant for the overall covariance matrix. Consequently, classification does not significantly reduce the determinants of the conditional covariance matrices. On the contrary, it introduces additional code length. Since the classification rate dominates in $RC(s)$, the value of $RC(s)$ increases as the classification becomes harder.

In other cases where the determinant of the initial covariance matrix is large, such as in the second setting, applying a harder classification before compression can make the conditional distributions of the observation corresponding to each classification result more “concentrated”, thus reducing the determinants. Therefore, harder classification can help save bits as illustrated in Figure 10b.

In some cases, however, proper soft classification can help save bits as shown in Figure 10c. Specifically, in the third setting, it can be explained as follows. The weighted average of the input variances of the two sub-encoders, weighted by the probabilities of the classification results, remains unchanged after classification. Since the logarithmic function is concave, the corresponding weighted average of the log-variances of the sub-encoders after classification is smaller than that of the entire composite source, thereby reducing $RC(s)$. Nevertheless, when hard classification is applied, the weighted log-variance cannot be further reduced significantly, while the classification rate $R_s$ increases.

The curves of the achievable upper bounds $R_{ub}(D_o)$ in the first two settings are shown in Figure 11. The performance is compared with the achievable upper bound $R_1(D_o)$ of the rate–distortion function proposed in [6]. The bound $R_1(D_o)$ characterizes the achievable rate–distortion pairs when a codebook matched to the source statistics is available and is asymptotically tight at high rates. In addition, the CAC scheme with a hard classifier is similar to the MCQ scheme in [44].

The results align with the $RC(s)$ curve in Figure 10. At high rates, classifiers with smaller $RC(s)$ values lead to lower distortion. In the first setting, direct compression performs better, while in the second setting, the CAC scheme with a hard classifier achieves better performance. In Figure 11b, the CAC scheme with a properly chosen classifier performs closely to the $R_1(D_o)$ bound at high rates. This indicates that when the codebook matched to the source statistics is unavailable and only Gaussian codebooks can be used, classification before compression can reduce the variance and thus lower the rate in this setting. Moreover, the performance of this scheme can approach that of using a matched codebook.

For a fixed classifier, when $D_o>D_T$, a series of inflection points appear as $D_o$ increases. When $D_o>{\sum }_{u=1}^{L}P(\widehat{S}=u)tr({\mathsf{\Sigma }}_u)$, reconstructing every observation as the mean of the input to the corresponding sub-encoder satisfies the constraint. In this case, no coding is needed for reconstruction, and the overall code length equals the classification code length. Therefore, when $D_o$ is sufficiently large, a “softer” classifier can save bits. It can also be observed from Figure 11 that in these two settings, the “harder” the classifier, the smaller the values of $D_o$ corresponding to the turning points. This is because in both cases, a “harder” classifier results in smaller eigenvalues and trace values of the covariance matrices.

Figure 11b further shows that for some composite sources, the optimal classifier varies with the reconstruction distortion constraint. Denote the optimal s as $s^{\ast }$. At high rates, $s^{\ast }=10$, whereas when $D_o=3.2$, $s^{\ast }=4$.

Next, the high-rate scenario is studied in the last three settings.

Figure 12 compares the distortion–rate performance of three schemes—direct compression, CAC with a soft classifier, and CAC with a hard classifier—along with $D_1(R)$, the distortion–rate function corresponding to $R_1(D_o)$, under high rate conditions in the last three settings. It can be observed that, with a properly chosen soft classifier, the CAC scheme significantly outperforms the other two schemes in all settings. In the third setting, the CAC scheme with a soft classifier can achieve approximately a 15% reduction in distortion at the same rate compared to both direct compression and the CAC scheme with a hard classifier. In the fourth and fifth settings, the distortion reductions are approximately 21% and 25%, respectively. These numerical results are consistent with the analysis in Section 4, which shows that, under high-rate conditions and with a fixed classifier, the achievable distortion of the three schemes decays exponentially with the rate R. Under this condition, when the rate is the same for all three schemes, the ratio of the distortions remains constant, and this constant does not vary with the rate. Therefore, optimizing s in the classifier effectively minimizes the coefficient of the exponential function in (36), thereby minimizing the distortion. The comparison between the performance of the CAC scheme with a soft classifier and the $D_1(R)$ curve further demonstrates that, with a properly selected classifier and Gaussian codebooks, the CAC scheme can approach the rate–distortion performance of the scheme using codebook matched to the source statistics.

6. Conclusions

We have generalized the classification problem in [4,5] into the multi-class classification scenario. In the case where only the classification distortion is constrained, we have characterized the rate–distortion function and proposed two classification schemes based on two upper bounds. We have also studied the scenario where the reconstruction distortion is constrained and evaluated the performance of the CAC scheme. In our future work, we plan to explore methods for implementing soft classification or compression methods that asymptotically approach the information-theoretic performance limits. In addition, investigating data-driven methods for estimating the rate–distortion behaviors of composite sources under both constraints is also an interesting direction for future research.